Glusterfs Crypt Ftruncate Extend File Size Trace Code Flow

ftruncate expend file fop

• finodelk (whole file)
• fgetxattr(get trusted.glusterfs.crypt.att.size value)
• readv(rmw readv head block)
• writev(write head block)
• writev(write full block loop )
• readv(rmw readv tail block)
• writev(write tail block)
• fsetxattr(update new file size)
• finodelk(unlock file) 

finodelk: lock whole file

1. Allocate GF_FOP_FTRUNCATE local datastruct
2. Init local xattr dict
3. Get inode info
4. Set crypt algo info
5. Set lock info (the lock whole file)
• lock.l_len    = 0;
• lock.l_start  = 0;
• lock.l_type   = F_WRLCK;
• lock.l_whence = SEEK_SET;
6. 呼叫STACK_WIND finodelk去加鎖整個檔案

Doxygen簡易使用(畫出c struct關係圖)

安裝Doxygen(ubuntu 14.04):

• # sudo apt-get install doxygen

安裝web server(因為doxygen檔案以html形式呈現):

• # sudo apt-get install apache2

  # sudo apt-get install graphviz

使用Doxygen:

1. # sudo doxygen -g DOXYGEN_CONFIG_FILE

2. # sudo vim DOXYGEN_CONFIG_FILE

3. # sudo doxygen DOXYGEN_CONFIG_FILE

    

修改DOXYGEN_CONFIG_FILE內容

• PROJECT_NAME           = "glusterfs"
• PROJECT_NUMBER         = 3.8.3
• OUTPUT_DIRECTORY       = /var/www         //產生文件的folder
• CREATE_SUBDIRS         = YES
• ALLOW_UNICODE_NAMES    = YES
• OUTPUT_LANGUAGE        = English
• FULL_PATH_NAMES        = YES
• INPUT                  = /root/glusterfs  //source code的folder
• RECURSIVE              = YES
• HAVE_DOT               = YES
• CALL_GRAPH             = YES
• CALLER_GRAPH           = YES

GlusterFS Timer Mechanism

GlusterFS context(glusterfs_ctx) has a timer data structure(gf_timer_registert_t). 

gf_timer_registert_t {
    pthread_t          th,
    char               fin,    // finish
    _gf_timer          stale,  // stale event list
    _gf_timer          active, // active event list
    pthread_mutex_t    lock   
}

There are

• pthread
• active list
• stale list
• pthread lok

in glusterfs context timer. active and stale list date structure is _gf_timer, it is also gf timer event.

_gf_timer {
    _gf_timer         *next, *prev, // pointer to next and previous
    timespec          at,           // time to execute the event
    gf_timer_cbk_t    callbk,       // execute funcation
    void              data,         // event data for callbk
    xlator_t          *xl           // event translator
}

glusterfs_ctx timer

glusterfs_ctx will create a pthread, and pthread will execute gf_timer_proc() function.
The thread will get the timer event from active list and check the timestamp of event. if timestamp > current time, it will move the timer event from active list to stale list, and execute the callback function of timer event.

 

gfid introduction

trusted.gfid is inode number, A uuid that's unique to each file across the entire cluster. 

在brick中每一個檔案(include dir)都有一個gfid, 而在.glusterfs的folder中,會有一個對應的檔案, 
利用gfid的前兩個digit來分folder. 
ex. file1的gfid = 0xb8d5abfc-35f1-4716-8da8-824f186c74e8 
在.glusterfs下面就會有b8/d5/b8d5abfc-35f1-4716-8da8-824f186c74e8 
brick 的/目錄的gfid是0x00000000000000000000000000000001 
若是folder的話在.glusterfs會指向parent目錄的folder連結 
ex. 
在根目錄下有一個c folder. (/c)
gfid=0x07c8896ce3404eb69690f744618ea65b 
對應的gfid file = .glusterfs/07/c8/07c8896c-e340-4eb6-9690-f744618ea65b 
他會link(symlink)到parent的目錄下的目錄 
像以下這樣 
07c8896c-e340-4eb6-9690-f744618ea65b -> ../../00/00/00000000-0000-0000-0000-000000000001/c/

在brick下查看attribute的指令:
# getattr getfattr -m . -d -e hex $brick-folder

GlusterFS IO Flow

GDB with GlsuterFS

使用GDB去追glusterfs daemon有兩種方式. 

• gdb模式下去launch glusterd daemon
• attach glusterfs deamon

以attach的方式attach glusterfs daemon:

假設glusterd的pid是22579

# gdb 

# (gdb) attach 22579

因為通常glusterfs daemon launch 起來後, 會進入epoll dispatch. 所以我們直接在event_dispatch_epool_handler下break point

# (gdb) b event_dispatch_epoll_handler