How to Run CVFSCK (DRAFT)

CVFSCK

When working with the DXi’s there are times that it's necessary to run a file system check on the snfs filesystem, which is where the DXi blockpool (data & metadata) resides. CVFSCK checks the integrity and structure of the underlying file system and attempts to correct any issues it finds.

You’ll want to run through the file system check procedure (cvfsck)

- if you can’t mount /snfs

- if you notice inconsistent behavior in the /snfs filesystem, like not being able to access certain files, finding unexpected data in files or missing files altogether

- If blockpool corruption is detected and cleaned up but persists

- If there was a messy hardware issues (controllers/disks/chassis)

- If there was a system crash

CVFSCK Procedure for vol0 (/SNFS) - see note at the end for DXi8500 systems

Remove Processwatcher:

# rm /var/DXi/processwatcher

rm: remove regular empty file `/var/DXi/processwatcher'? y

Stop services:

# service heartbeat stop

Stopping High-Availability services:

[ OK ]

Start SNFS:

# /etc/init.d/cvfs start

Initiating start of ADIC DSM component

Initializing StorNext Filesystem (SNFS)

Loading SNFS modules

net.core.rmem_max = 1048576

Starting /usr/cvfs/bin/fsmpm.

net.core.rmem_max = 1048575

Starting /usr/cvfs/bin/cvfsd...

Mounting SNFS filesystems

/dev/cvfsctl1_vol0 on /snfs type cvfs (rw,noatime,cachebufsize=256k,nfsopen=yes,timeout=600,sparse=yes)

SNFS Initialized [ OK ]

Unmount SNFS:

# umount /snfs

Stop SNFS via cvadmin - 2 options here:

# /usr/cvfs/bin/cvadmin -e "stop vol0;"

# /usr/cvfs/bin/cvadmin

StorNext File System Administrator

Enter command(s)

For command help, enter "help" or "?".

List FSS

File System Services (* indicates service is in control of FS):

1>*vol0[0] located on localhost.localdomain:35665 (pid 24356)

Select FSM "vol0"

Created : Mon Apr 20 13:24:21 2009

Active Connections: 0

Fs Block Size : 64K

Msg Buffer Size : 4K

Disk Devices : 18

Stripe Groups : 6

Fs Blocks : 271699584 (16.19 TB)

Fs Blocks Free : 268484095 (16.00 TB) (98%)

snadmin (vol0) > stop vol0

Stop FSS "vol0"

FSS 'vol0' on localhost.localdomain stop initiated.

FSS 'vol0' stopped.

Select FSM "none"

snadmin> exit

Run the cvfsck –j command to replay the journal, it doesn’t take long and subsequent commands will be more accurate:

# /usr/cvfs/bin/cvfsck -jv vol0

Run cvfsck in no write verbose mode (see what s broken):

NOTE: This command is read only so if you see something broken contact your escalation point before trying to fix the problem. Once you tell CVFSCK to repair the issues detected, there’s no going back.

# /usr/cvfs/bin/cvfsck -nv vol0

You can look at the output on the screen or in the Log file that has the output from cvfsck –nv:

NOTE: This is just an example, your log file name will have a different date/time stamp

# more /usr/cvfs/data/vol0/trace/cvfsck-04_29_2009-10_37_33

Once you have determined you want cvfsck to try and fix SNFS issues run this:

# /usr/cvfs/bin/cvfsck -v vol0

Check log file that has the output from cvfsck –v and make sure it corrected what it found:

# more /usr/cvfs/data/vol0/trace/cvfsck-04_29_2009-10_50_36

Now run cvfsck –nv again to make sure SNFS is clean and no more issues are found:

# /usr/cvfs/bin/cvfsck -nv vol0

Reboot the node:

# sync; sync; sync; reboot

====================================================================================

NOTE: DXi 8500

If you dealing with an DXi 8500 there are a few adjustments that you need to make as the 8500’s have 2 file systems to deal with (blockpool, bulkdata) not just one (snfs) like the other systems. You should substitute these commands below for the corresponding commands in the above procedure.

For DXi8500

# umount /snfs /bp

# /usr/cvfs/bin/cvadmin -F vol0 -e "stop bulkdata;"

# /usr/cvfs/bin/cvadmin -F vol0 -e "stop blockpool;"

# /usr/cvfs/bin/cvfsck -jv bulkdata

# /usr/cvfs/bin/cvfsck -jv blockpool

# /usr/cvfs/bin/cvfsck -nv bulkdata

# /usr/cvfs/bin/cvfsck -nv blockpool

# /usr/cvfs/bin/cvfsck -v bulkdata

# /usr/cvfs/bin/cvfsck -v blockpool

NOTE: Clobber Free List

This is an option that is needed occasionally to clear the free inode list. There’s no need to run it unless directed to or unless you’re sure the free inode list is corrupted.

# /usr/cvfs/bin/cvfsck -C vol0

If it is determined that this needs to be run, you should run it after

/usr/cvfs/bin/cvfsck -nv vol0

And before

# /usr/cvfs/bin/cvfsck -v vol0

From the Man page

-C Clobber a corrupted free inode list. No data will be lost in this process, but metadata usage

may be increased after running this command. Only run this command if cvfsck cannot repair

free-list inconsistencies.