Quantum| SNFS: DXi Unable to Start Due to Configuration Changes |
During a DXi system capacity expansion, the StorNext File System (SNFS) configuration file is changed to reflect new additional data storage disks. However, we have had instances where these configuration files have been misconfigured by the scripts, either during an expansion or, in some cases, while the system was experiencing RAID controller issues. This topic explains how to determine if StorNext is not starting due to one of these events and how to (hopefully) recover from said event.
When a SNFS is initially created, the cvmkfs process uses the configuration file (/usr/cvfs/config/[fs_name].cfg) to lay out the metadata information and the data stripe groups. When a change is made to the file system, cvupdatefs attempts to write the information from the newly edited [fs_name].cfg file to update the metadata and stripe group information and copy the old configuration file into a [fs_name].cfg.bak file for future reference. However, a failure to update the metadata stripe group with this new information can leave the edited, incorrect configuration file on the system. This means that when the StorNext attempts to start the file system, the current configuration file does not match the metadata information on the file system, leaving the DXi in an unbootable state...the SNFS never mounts and the DXi software never starts.
When the StorNext File System attempts to start a file system with a configuration file that does not match the current configuration in the metadata, the following error occurs (in both the /var/log/messages file and the /usr/cvfs/data/[fs_name]/log/cvlog file):
bulkdata cvlog:
[1002 23:41:16] 0x2b53a98a5200 (**FATAL**) File system Icb check failed! ICB Check Error: [StripeGroup count mismatch]
messages:
Oct 2 23:41:16 xa-drx-vtl-2 fsm[17399]: StorNext FSS 'bulkdata[0]': File system Icb check failed! ICB Check Error: [StripeGroup count mismatch]
If those messages are present in the current logs, stop rebooting the system. Further reboots and attempted reconfigurations have the chance of overwriting the [fs_name].cfg.bak file with the newer, incorrect configuration making recovery more difficult. If this has happened already, you might need to look in a previous collect to get the proper configuration file.
You need to determine the difference in the current, non-functioning configuration file and the previous successful starts. Here is the configuration banner printed in the cvlog when the file system starts:
[1002 22:45:26] 0x2afe4d8f6200 (Info) Revision 4.2.0 Build 27947 Branch lcrs_4.2.0_RC35698
[1002 22:45:26] 0x2afe4d8f6200 (Info) Built for Linux 2.6.18-8.el5 x86_64
[1002 22:45:26] 0x2afe4d8f6200 (Info) Created on Thu Sep 20 12:14:49 MDT 2012
[1002 22:45:26] 0x2afe4d8f6200 (Info) Built in /home/mlund/nightly/VM-0-RedHat50AS-26x86-64-SP0/sn/buildinfo
[1002 22:45:26] 0x2afe4d8f6200 (Info)
Configuration:
DiskTypes-7
Disks-12
StripeGroups-12
MaxConnections-8
ThreadPoolSize-128
StripeAlignSize-0
FsBlockSize-65536
BufferCacheSize-32M
InodeCacheSize-65536
RestoreJournal-Disabled
RestoreJournalDir-None
Since you know there is a stripe group mismatch, and the DXi currently uses one disk per stripe group, I recommend grepping the word "Disks-" from the cvlog file. On my most current case, I received the following output:
[kjhall@srscratch log]$ grep "Disks-" cvlog
Disks-8
Disks-8
Disks-12
Disks-12
Disks-12
Disks-12
Disks-12
Disks-12
Disks-12
Disks-10
Disks-10
Disks-10
Disks-10
No timestamps, but you can see the initial configuration had 8 disks, which was expanded to 12 (normal). After six or seven successful bootups, the system was "shrunk" from 12 to 10 disks, which was caused by the faulty RAID controller and the automatic scripts that attempted to "grow" the file system (PTR 36388) while the controller was malfunctioning. An SNFS cannot be "shrunk" or reduced in number of disks in the configuration file. Here is the same system, with the first instance of "Disks-10":
[1002 23:41:16] 0x2b53a98a5200 (Info) Revision 4.2.0 Build 27947 Branch lcrs_4.2.0_RC35698
[1002 23:41:16] 0x2b53a98a5200 (Info) Built for Linux 2.6.18-8.el5 x86_64
[1002 23:41:16] 0x2b53a98a5200 (Info) Created on Thu Sep 20 12:14:49 MDT 2012
[1002 23:41:16] 0x2b53a98a5200 (Info) Built in /home/mlund/nightly/VM-0-RedHat50AS-26x86-64-SP0/sn/buildinfo
[1002 23:41:16] 0x2b53a98a5200 (Info)
Configuration:
DiskTypes-7
Disks-10
StripeGroups-10
MaxConnections-8
ThreadPoolSize-128
StripeAlignSize-0
FsBlockSize-65536
BufferCacheSize-32M
InodeCacheSize-65536
RestoreJournal-Disabled
RestoreJournalDir-None
(Notice that the timestamps between the previous, successful boot and the one above. Using that information, I was able to narrow the search for the cause of the problem to the hour between 22:45 (last boot with 12 disks) and 23:41 (first boot with 10 disks) on October 2nd.)
Looking at the last good boot in the logs, you can see the LUNs and their associated labels in the /usr/cvfs/debug/nssdbg.out file:
[1002 22:45:26] 0x2b247154d200 NOTICE PortMapper: CVFS Volume Cvfs_BDSS_JRN on device: /dev/sdb (blk 0x810 raw 0x810) con: 0 lun: 0 state: 0x4 inquiry [DELL PERC H710 3.13] controller # 'default' serial # '6B8CA3A0E5ADBD0019B3398B0EC3538F' Size: 2078687 Sector Size: 512
[1002 22:45:26] 0x2b247154d200 NOTICE PortMapper: CVFS Volume Cvfs_BDSS_MD1 on device: /dev/sdf (blk 0x850 raw 0x850) con: 0 lun: 0 state: 0x4 inquiry [DELL PERC H710 3.13] controller # 'default' serial # '6B8CA3A0E5ADBD0019B339B010F6E3B6' Size: 563591135 Sector Size: 512
[1002 22:45:26] 0x2b247154d200 NOTICE PortMapper: CVFS Volume Cvfs_BDSS_MD2 on device: /dev/sdh (blk 0x870 raw 0x870) con: 0 lun: 0 state: 0x4 inquiry [DELL PERC H710 3.13] controller # 'default' serial # '6B8CA3A0E5ADBD0019B339BE11C3AE96' Size: 563591135 Sector Size: 512
[1002 22:45:26] 0x2b247154d200 NOTICE PortMapper: CVFS Volume Cvfs_BDSS_MD3 on device: /dev/sdj (blk 0x890 raw 0x890) con: 0 lun: 0 state: 0x4 inquiry [DELL PERC H710 3.13] controller # 'default' serial # '6B8CA3A0E5ADBD0019B339CB1293331C' Size: 563591135 Sector Size: 512
[1002 22:45:26] 0x2b247154d200 NOTICE PortMapper: CVFS Volume Cvfs_BDSS_MD4 on device: /dev/sdl (blk 0x8b0 raw 0x8b0) con: 0 lun: 0 state: 0x4 inquiry [DELL PERC H710 3.13] controller # 'default' serial # '6B8CA3A0E5ADBD0019B339D913644330' Size: 563591135 Sector Size: 512
[1002 22:45:26] 0x2b247154d200 NOTICE PortMapper: CVFS Volume Cvfs_BDSS_MD5 on device: /dev/sdn (blk 0x8d0 raw 0x8d0) con: 0 lun: 0 state: 0x4 inquiry [DELL PERC H710 3.13] controller # 'default' serial # '6B8CA3A0E5ADBD0019B339E71437F040' Size: 563591135 Sector Size: 512
[1002 22:45:26] 0x2b247154d200 NOTICE PortMapper: CVFS Volume Cvfs_BDSS_MD6 on device: /dev/sdp (blk 0x8f0 raw 0x8f0) con: 0 lun: 0 state: 0x4 inquiry [DELL PERC H710 3.13] controller # 'default' serial # '6B8CA3A0E5ADBD0019B339F5150D1AA3' Size: 563591135 Sector Size: 512
[1002 22:45:26] 0x2b247154d200 NOTICE PortMapper: CVFS Volume Cvfs_Array_1_TRAY_99_VOL_1_00 on device: /dev/sdq (blk 0x4100 raw 0x4100) con: 2 lun: 0 state: 0x204 inquiry [LSI VirtualDisk 0784] controller # '50080E53E2722008' serial # '60080E50003E27220000025C5214F1FA' Size: 25232914399 Sector Size: 512
[1002 22:45:26] 0x2b247154d200 NOTICE PortMapper: CVFS Volume Cvfs_Array_1_TRAY_99_VOL_2_02 on device: /dev/sdr (blk 0x4110 raw 0x4110) con: 2 lun: 1 state: 0x204 inquiry [LSI VirtualDisk 0784] controller # '50080E53E2722008' serial # '60080E50003E27220000025F5214F200' Size: 25232914399 Sector Size: 512
[1002 22:45:26] 0x2b247154d200 NOTICE PortMapper: CVFS Volume Cvfs_Array_2_TRAY_99_VOL_1_01 on device: /dev/sdu (blk 0x4140 raw 0x4140) con: 2 lun: 0 state: 0x204 inquiry [LSI VirtualDisk 0784] controller # '50080E53E246E008' serial # '60080E50003E246E000002715213A5FC' Size: 25232914399 Sector Size: 512
[1002 22:45:26] 0x2b58b03dc200 NOTICE PortMapper: CVFS Volume Cvfs_Array_1_TRAY_0_VOL_1_04 on device: /dev/sds (blk 0x4120 raw 0x4120) con: 2 lun: 2 state: 0x204 inquiry [LSI VirtualDisk 0784] controller # '50080E53E2722008' serial # '60080E50003E2722000003385214F523' Size: 25232914399 Sector Size: 512
[1002 22:45:26] 0x2b58b03dc200 NOTICE PortMapper: CVFS Volume Cvfs_Array_1_TRAY_0_VOL_2_05 on device: /dev/sdt (blk 0x4130 raw 0x4130) con: 2 lun: 3 state: 0x204 inquiry [LSI VirtualDisk 0784] controller # '50080E53E2722008' serial # '60080E50003E27220000033A5214F52B' Size: 25232914399 Sector Size: 512
Now you can match these LUNs with what is supposed to be in the configuration file. The current config file shows:
[kjhall@srscratch snfs-info]$ grep Node bulkdata.cfg
Node Cvfs_BDSS_MD1 0
Node Cvfs_BDSS_MD2 0
Node Cvfs_BDSS_MD3 0
Node Cvfs_BDSS_MD4 0
Node Cvfs_BDSS_MD5 0
Node Cvfs_BDSS_MD6 0
Node Cvfs_BDSS_JRN 0
Node Cvfs_Array_1_TRAY_99_VOL_1_00 0
Node Cvfs_Array_2_TRAY_99_VOL_1_01 0
Node Cvfs_Array_1_TRAY_99_VOL_2_02 0
While the older .bak file:
[kjhall@srscratch snfs-info]$ grep Node bulkdata.cfg.bak
Node Cvfs_BDSS_MD1 0
Node Cvfs_BDSS_MD2 0
Node Cvfs_BDSS_MD3 0
Node Cvfs_BDSS_MD4 0
Node Cvfs_BDSS_MD5 0
Node Cvfs_BDSS_MD6 0
Node Cvfs_BDSS_JRN 0
Node Cvfs_Array_1_TRAY_99_VOL_1_00 0
Node Cvfs_Array_2_TRAY_99_VOL_1_01 0
Node Cvfs_Array_1_TRAY_99_VOL_2_02 0
Node Cvfs_Array_1_TRAY_0_VOL_1_04 0
Node Cvfs_Array_1_TRAY_0_VOL_2_05 0
Notice that the [fs_name].cfg.bak has all the correct LUNs, quantity and labels, while the current configuration file lacks two of them. This was the reason the file system would not start and was easily corrected by copying the old configuration back to be the current configuration.
You can test it by starting StorNext (service cvfs start), then mounting the file system (mount /snfs). A successful mount means the current file system configuration file matches the configuration information in the metadata stripe group.
This page was generated by the BrainKeeper Enterprise Wiki, © 2018 |