Replace an MDC in an HA Environment

This section describes how to replace the secondary HA server. This procedure may also be used to upgrade the operating system of the MDC (for example, if you upgrade from RHEL5 to RHEL6). If you need to replace both the primary and secondary MDCs, then run through this procedure once, fail over so that NEW secondary MDC becomes primary, and then run through this procedure a second time.

Before beginning this procedure make sure you have obtained the proper licenses required for the new HA MDC. The current license should be sufficient if you are just upgrading the OS.

Note: This procedure requires a certain level of technical expertise. Do not attempt performing this procedure unless you are confident you can complete the steps successfully. If you are unsure about your ability to complete these steps, contact the Quantum Technical Assistance Center for help.

Note: If you need to replace the system that is currently the primary MDC, stop StorNext on the primary. This will cause a failover to the secondary system.

  1. If both HA MDCs are currently up and running, make sure the system you want to replace is designated as the secondary MDC. This can be accomplished by running service cvfs stop on the designated machine.
  2. Run a manual backup to tape or to Object Storage from the StorNext GUI.
  3. Make sure all store/retrieve requests have finished.
  4. If you are using the Distributed Data Mover (DDM) feature, note the value of the DISTRIBUTED_MOVING parameter (either All or Threshold) in /usr/adic/TSM/config/fs_sysparm (or fs_sysparm_override). Use a text editor to set the DISTRIBUTED_MOVING value to None. Use the adic_control restart TSM command to put this change into effect.
  5. Unmount all file systems from all clients, and then stop the SNFS processes on each client machine. On the Linux platform, do this by running service cvfs stop.
  6. Uninstall StorNext from the secondary server, but retain the log files. Do this by running the command install.stornext -remove.
  7. Power down the uninstalled secondary server.

  1. Set the primary node to config mode and the peer node to peerdown mode by running the following commands:

    snhamgr peerdown

    snhamgr mode=config

  2. Check the StorNext HA Manager (snhamgr) status by running the command snhamgr status. The status should look similar to this:

    LocalMode=config

    LocalStatus=primary

    RemoteMode=peerdown

    RemoteStatus=unknown

  3. Change the /usr/cvfs/config/ha_peer file on the primary MDC to the new MDC IP address.
  4. If the /usr/cvfs/config/fsnameservers file includes the old MDC IP address, replace it with the new MDC IP address on the primary MDC and all the clients.
  5. In the primary MDC’s /usr/cvfs/config/license.dat file, remove all the old MDC licenses by commenting out the lines you want removed. Keep only the primary MDC licenses.
  6. Push those changes to the synchronization mirror directory by running this command: /usr/adic/util/syncha.sh -primary
  7. (Optional) Upgrade the Operating System of the new secondary server at this point.
  8. Install StorNext on the NEW secondary server by running this command: install.stornext

  9. Put the new licenses on the NEW secondary servers into /usr/cvfs/config/license.dat. The StorNext GUI can be run on the secondary to enter the licenses.

    Note: You must restart the StorNext GUI after you create or edit the license.dat file.

  10. Validate that the ha_smith_interval value is identical on both nodes, or HA will not work.

    Caution: On StorNext appliances, the ha_smith_interval value is not set to the default. The /usr/cvfs/config/ha_smith_interval file is used to set the ha_smith_interval=20, which is in seconds. By default, the file is not present and the default value is 5 seconds. If a single MDC controller is recovered by removing and installing StorNext, the values will not match.

  11. Set the primary node to default mode and the peer node to peerup mode by running the following commands:

    Note: This causes StorNext services to stop.

    snhamgr mode=default

    snhamgr peerup

  12. Start the StorNext service on the primary MDC.

    systemctl start cvfs
  13. In the StorNext GUI, go to the Tools> High Availability > Convert screen and convert the secondary MDC to HA.
  1. After the conversion is complete, check the snhamgr status on both MDCs. Run the cvadmin command to verify that all file systems are listed correctly.
  2. Perform a system backup by running the snbackup command. This process may take substantial time depending on the number of managed files in the system.
  3. Start and mount StorNext file systems on the clients, and then verify that all clients have full access
  4. Conduct a failover to confirm that the secondary MDC has converted correctly. Confirm this by testing access to all file systems, moving files to/from tapes, and reviewing GUI configuration information.
  5. If you conducted a failover to the secondary server, fail back to the original primary server.
  6. Verify that all clients still have full access.
  7. If you are using the DDM feature and if you use the secondary server as a DDM mover, make sure the file systems are mounted.
  8. If you are using DDM, edit fs_sysparm or fs_sysparm_override to use your preferred DDM mode, (All or Threshold).
  9. Use the command adic_control restart TSM to put this change into effect.
  10. (Optional) If you need to replace both MDCs, fail the primary MDC over to the NEW secondary and then repeat this procedure.