Client Node Tools and Diagnostics

 

Information on debugging Network stability and SAN/Storage stability applies to the SNFS Client nodes as well.

 

 

 

SNFS NSS traces

To collect debug tracing from fsmpm the directory "debug" must exists in the StorNext directory on the same level as the "bin" directory. The output of the tracing is placed in "debug/nssdbg.out".

Create a file called verbose in $CVFS_ROOT/debug/verbose.

Usually the default value of NSS_DEBUG_MBR|NSS_DEBUG_VOTE is sufficient and that can be accomplished just by creating the file with no content.  

If the "debug" directory exists and the "verbose" files does not exists the NSS_DEBUG_VOTE flag is used for tracing. (debug directory almost always present, so this is default.)
If no value is placed into "verbose" the following flag combination is or'ed together: NSS_DEBUG_MBR|NSS_DEBUG_VOTE.
A value of "-1" placed into "verbose" will use all the available flags for tracing or each individual flag value or any combination of the or'ed values can be placed in the file to customize what tracing is required.

The following values are defined as debug tracing flags in fsmpm.

A value in the file can be used as the debug value.
#define NSS_DEBUG_GENERAL 0x01
#define NSS_DEBUG_INPUT 0x02
#define NSS_DEBUG_OUTPUT 0x04
#define NSS_DEBUG_MBR 0x08
#define NSS_DEBUG_VOTE 0x10
#define NSS_DEBUG_LDAP_CRED 0x20

The values below are only valid in Stornext 3.1 and above:

#define NSS_PROXY_DEBUG 0x40
#define NSS_FOREIGN_DEBUG 0x80

The values below are only valid in StorNext 4.0 and above:

#define NSS_HAMON_RESET_DEBUG 0x100

The values below are only valid in StorNext 4.3.2 and above:

#define NSS_HELPER_DEBUG 0x200

Note: with Xsan being merged into the base OS in Lion, $CVFS_ROOT has been broken into three things, CVFS_ROOT, CVFS_LOGROOT, and CVFS_CONFIG. For a StorNext install, CVFS_LOGROOT is the same as CVFSROOT, and CVFS_CONFIG is $CVFS_ROOT/config.

For an Apple install, they are all different. The main reasons are: config needs to be under /Library/Preferences, so we put it in /Library/Preferences/Xsan, and our read-only data (/bin, /man) and our writable data (/data, /debug) need to be in different places.

 

This file will need to be /Library/Logs/Xsan/debug/verbose for an Apple install.

 

SNFS cvdb perf traces

The StorNext File System client has the ability to dump out I/O performance statistics. These statistics can be gathered through the cvdb tool. The command

 

The cvdb and cvdbset
commands are found on Windows at:

    C:\Program Files\StorNext\bin

and on Linux and Unix at:

    /usr/cvfs/bin


# cd /tmp/place_for_storing_cvdbout_trace_files
# cvdbset - all
# cvdbset perf  :vnops
# cvdbset -r 32M
# cvdb -e
# cvdb -g -C -F

Run for duration of the test then  <ctrl-C>

# cvdbset -d
# cvdbset - all

 

The output of cvdb perf tracing will show performance for each application I/O request, as well as the resulting device I/O requests that SNFS will make.

 

Performance output for requests into the file system

 

    PERF: VFS Write EofDmaAlgn 41 MB/s offs 0x0 len 0x400000 mics 95618 ino 0x5

 

Performance output for requests sent from SNFS to individual disk devices

    PERF: Device Write 41 MB/s IOs 2 exts 1 offs 0x0 len 0x400000 mics 95589 ino 0x5

 

The output fields seen in the logs are as follows:

    offs: offset into file

    len: length of I/O

    mics: microseconds for I/O request to complete

    ino: inode number of file being accessed

 

 

See the SNFS Tuning Guide document for additional details on SNFS cleint perf tracing.