Linux SNMS and SNFS Installations with the HaShared File System

The HaShared file system is required for SNMS and GUI-supported installations. The shared file system holds operational information for those components, which must be protected against split-brain corruption. The additional complexity this entails is simplified and automated by the HA Manager Subsystem.

Touch Files that Control StorNext and HA

Environment variables defined in the /usr/adic/.profile and
/usr/adic/.cshrc files reference touch files that are used in StorNext to track state transitions. Some of these are unique to HA configurations. The variables and their values are as follows:

1. ACTIVE_SNFS_SERVER=/usr/adic/install/.active_snfs_server

The file is created following activation of the HaShared file system to designate the local node as the Primary server.

2. HA_STARTING_UP=/usr/cvfs/install/.ha_starting_up

The file is created at the start of the 'service cvfs start' script, and removed at the end of the activation script for the HaShared file system. If it is found before it is created, that causes the creation of the HA_IDLE_FAILED_STARTUP file as described in the next item.

3. HA_IDLE_FAILED_STARTUP=/usr/cvfs/install/.ha_idle_failed_startup

When the HA_STARTING_UP file exists as the 'service cvfs start' starts, it is assumed that the previous attempt to start failed before completing, possibly because an HA reset occurred. The HA_IDLE_FAILED_STARTUP file is created to block future attempts to start, and the current script exits. This avoids an infinitely looping series of startup attempts, and allows an administrator to log in and correct problems. The HA Manager reports the existence of this file as a mode, and offers the clear command for removing the file.

4. SNSM_HA_CONFIGURED=/usr/adic/install/.snsm_ha_configured

The file is created by the cnvt2ha.sh script to indicate that the system has been converted to HA. Its existence allows the snhamgr_daemon to run.

5. START_SNFS_ONLY=/usr/adic/install/.start_snfs_only

The file is created by running one of the following commands: '/usr/adic/bin/adic_control startonly snfs' or '/usr/cvfs/bin/DSM_control startonly'.

Its existence indicates to the snactivated script that Storage Manager components are not to be started. The file is removed by using any of the following commands: 'DSM_control stop', 'service cvfs stop', or 'adic_control stop snfs'.

Starting a Single StorNext HA Server for Production

The command service cvfs start sets in motion a sequence of events that result in the starting of all the Storage Manager components.

Note: The individual Storage Manager component scripts should not be run by hand. There are safeguards in the control scripts to preserve the HA protections against split-brain scenario in any case, but StorNext can get into certain states that are tricky to reconcile if component scripts are used in the wrong sequence. The shared file system can make that reconciliation more difficult.

The cvfs script (indirectly) starts the DSM_control script, which starts the FSMPM, waits for it, and then repeatedly attempts to mount all of the cvfs type file systems. The FSMPM reads the FSM configuration files and the fsmlist file. It starts the HaShared and HaUnmanaged FSMs in the fsmlist, but delays starting the HaManaged FSMs. The sub state of the delayed FSMs can be displayed with the fsmlist command in the cvadmin tool. Meanwhile, the mounts taking place because of the action of DSM_control are triggering elections that are activating the locally started FSMs if they are not already being serviced by active FSMs on the peer server.

When an FSM completes activation, it runs the snactivated script. The script for the HaShared file system creates the ACTIVE_SNFS_SERVER file, and then calls snhamgr --primary to set the Primary status for this server. That induces the FSMPM to start the HaManaged FSMs. The HaShared activation script waits a limited time for all of the managed file systems to be mounted, and then it calls 'adic control start' to start the other Storage Manager components. Finally, the HaShared activation script removes the startup-failure-detection touch file.

While all this is happening, the DSM_control script is monitoring progress and reporting statuses of the mounts and the component startups. It will wait a limited time for completion. When it finishes and exits all the nested scripts and returns to the user, all of the Storage Manager components should be up. But if it times out before that, the background activities should continue bringing up Storage Manager. The results of this can be observed a few moments later.

Starting and Stopping the StorNext HA Cluster

When starting or stopping StorNext HA, it is always helpful to first get the cluster state from the HA Manager as follows:

snhamgr status

The status output indicates whether one or both servers are stopped, if they are in non-default modes, and if either server has Primary status. The typical first step in stopping an HA cluster is to stop the secondary server and to lock it. This allows the other server to be put in config or single mode to operate with HA monitoring turned off. Then, that server can be stopped without incurring an HA reset. These steps are automated in the following cluster command:

snhamgr stop

When starting the cluster into production, both servers must be in default mode. The first server to start is likely to have its HaShared FSM activated, which will result in that server becoming Primary. The redundant server becomes Secondary when it starts operation, and its FSM processes wait in Standby until they are elected to usurp control of their file systems. These steps are automated in the following cluster command, which starts the local server, if necessary, to become Primary, followed by starting the Secondary server:

snhamgr start

StorNext HA also has the ability to stop a Primary server while it is in default mode without incurring an HA reset in most cases. It does this as follows:

  1. Stop Storage Manager processes, including the database
  2. Unmount all CVFS file systems on the local server other than the HaShared file system
  3. Stop all FSMs on the local server other than the HaShared FSM
  4. Unmount the HaShared file system
  5. Stop the FSMPM
  6. Stop the HaShared FSM

FSMs are elected and activate on the peer server as they are stopped on the local server.

An HA reset can occur if step 4 fails. (That is, if the HaShared file system cannot be unmounted for any reason.) This is the method for protecting Storage Manager management data against split-brain-scenario corruption. All of the at-risk data is contained on the shared file system, so the unmount operation ensures that the local server cannot modify the data.

Configuration Changes

When making configuration changes that require an FSM to be stopped, the primary HA server must be placed in config mode. When configuration or software changes are made, a StorNext HA cluster may need to be placed in config mode with only one server running. This avoids the possibility of an HA reset being induced by the arbitrary starts and stops of FSMs and other components as changes are made.

Many file system configuration changes require an active FSM (file system manager process for each file system) to be stopped and restarted on the primary HA node. This can trigger a standby FSM on the secondary HA node to take over control of the file system, likely using the old/original configuration.

Examples of changes that require an HA downgrade:

  • Removing file systems (and adding new file systems).
  • Changing name servers (fsnameservers file).
  • File system configuration file changes "config/*.cfg" and "config/*.cfgx".
    • Stripe group changes.
    • Adding stripe groups.
    • Marking stripe groups OFF.
    • Adding LUNs to stripe groups (bandwidth expansion).
  • Changes in the GUI panels under Configuration > File Systems > Edit > file-system-name that match a parameter found in *.cfgx cause an FSM restart.
  • Associating and disassociating affinities with a stripe group requires a *.cfgx change. Using cvaffinity to connect an affinity with a file or directory does not change *.cfgx.
  • Additional guidance can be found within snfs_config(5) and snfs.cfgx(5) in the MAN Pages Reference Guide.

Note: License changes do not cause an FSM restart, nor do Policy class changes.

Production Single-Server Operation

During extended outages of one server, it might not be productive to incur an HA reset since there is no standby FSM to fail over to. However, reconfiguring the remaining server to non-HA mode is not practical. The single mode solves this dilemma.

Single mode can be entered from default mode, and default mode can be entered from single mode without stopping StorNext. This makes it easy to decommission and replace a failed server. Here are the steps for doing this:

  1. If necessary, power down the decommissioning server.
  2. On the working server, run the following two commands in order:
snhamgr peerdown
snhamgr mode=single
  1. Replace the decommissioned server
  2. Acquire licenses for the new server
  3. Replace those license file entries on the working server
  4. Install StorNext on the new server, but do not configure it except for copying in the /usr/cvfs/config/fsnameservers file from the working server.
  5. On the working server, verify the /usr/cvfs/config/ha_peer file contains the IP address of the new server; update if required.
  6. On the working server, run the following two commands in order:
snhamgr mode=default
snhamgr peerup
  1. Run the conversion script on the new server as follows:
/usr/adic/util/cnvt2ha.sh secondary <shared fs> <peer IP address>

For additional information, see Single (Singleton) Mode and Replacing a HA System.

Non-production Operation

There is a method for starting the SNFS file systems without starting the Storage Manager management components in the rare case that this is needed. The following two commands accomplish the same goal:

  • adic_control startonly snfs
  • DSM_control startonly

These commands create the /usr/adic/install/start_snfs_only touch file, which signals to the snactivated.pl script not to start the management components. The file exists until StorNext is stopped, and has its effect whenever the FSMs activate.