Upgrade an HA System

This section describes how to upgrade a High Availability (HA) system using primary and secondary MDC nodes.

Note: This section is not intended for single-node MDCs. If you are performing an upgrade on a single MDC node system using RPM files, see Install a StorNext File System Only Using an RPM.

Before you begin this or any other upgrade procedure, verify that you have the necessary licenses required to upgrade. If you do not have the required licenses, the upgrade may fail and you will be unable to continue.

The upgrade procedure requires running commands from the command line. You should also have a solid understanding of the HA feature in StorNext and a general familiarity with the StorNext GUI.

Note: You should not attempt to perform this procedure unless you have carefully read the upgrade steps and are confident you can complete the steps successfully.

Note: If you use scripts in your environment to upgrade - removing StorNext from the secondary node first, then upgrading the primary node, they can still be used to perform the HA upgrade.

Note: If you are uncertain about your ability to perform the upgrade unassisted, please contact Quantum Professional Services for assistance in upgrading either remotely or on site.

Caution: If you decide to perform the upgrade yourself and you encounter an error during the upgrade which is not caused by a bug in the upgrade installer, Quantum’s Profession Services group must be consulted to come to your location to correct the issue, and you will be billed for their services. Note that Professional Services representatives cannot travel without a purchase order, and normal scheduling applies. Please consider this before you decide to perform an HA upgrade on your own.

This procedure describes how to upgrade a StorNext HA system on the Linux operating system. See the StorNext 6 Compatibility Guide for more information about valid upgrade paths.

As always, before you begin the upgrade make sure you have obtained the proper licenses for both nodes for the targeted build. Refer to The License Check Option if you are not sure you have the necessary licenses for StorNext.

  1. It is recommended that you run and complete a StorNext backup prior to starting the upgrade. If you have not completed a backup, you can initiate one from the StorNext GUI.
  2. Make sure all store and retrieve requests have finished.
  3. If you are using DDM, disable it on all the hosts by using the StorNext GUI or the fsddmconfig command. Storage Manager requires that the MDC and DDM clients are running the same version of the software. Refer to the section on Distributed Data Mover in the StorNext User’s Guide for information on DDM.
  1. Login to command line of the secondary MDC node.
  2. Identify the correct installation directory for your operating system and hardware platform, and then change to that directory. For example, for Red Hat Linux 7 running on an x86 64-bit platform, change to the RedHat7_x86_64 directory.

  3. Upgrade StorNext on the secondary MDC node by running the following command:

    ./install.stornext
  1. After the upgrade on the secondary node completes, log in to command line of the primary MDC node.

  2. Upgrade StorNext on the primary MDC node by running the following command:

    ./install.stornext

Note: Upgrading the primary MDC node triggers a failover of control to the secondary MDC node. At this point, the secondary node becomes the new primary node. Client connectivity can be checked to ensure that the new primary node is functioning properly.

Note: The new primary MDC node converts the old StorNext file system to the new StorNext file system in the background. During this process, it is possible the old primary MDC running the older version of StorNext will still try to access the new StorNext file system. Because the file system is converting, the old primary StorNext log may show ICB error messages in the log.

Note: Also, while the file system is being converted, stores and retrieves will not be available until the conversion is completed. It is possible that this process could take hours.

During the upgrade of the node originally operating as the primary, the node originally operating as the secondary node becomes the primary. After the database updates are complete and the upgrade of the original primary completes, a failover of the node currently operating as the primary should be triggered so it once again is operating as the secondary, and to reinstate the original primary node to operate as primary again. To initiate the graceful failover of an HA pair, perform the following procedure.

  1. Open an SSH connection to the node operating as the primary.
  2. Log in to the command line of the primary node as root.

  3. Confirm that the node is operating as the primary by entering the following at the command line:

    snhamgr -m status

  4. Verify the output is (bold used for clarification):

    :default:primary:default:running:

  5. On the node operating as the primary, initiate an HA failover to the node operating as the secondary.

    service cvfs stop
  6. Wait until the secondary node becomes the primary, and leave your SSH connection to this node open. (Time may vary.)
  7. Open an SSH connection to the node now operating as the primary.
  8. Log in to the command line of the primary node as root.

  9. Confirm that the node is operating as the primary by entering the following at the command line:

    snhamgr -m status

  10. Verify the output is:

    :default:primary:default:stopped:

  11. From the SSH connection to the node now operating as the secondary, enter the following:

    service cvfs start

  12. Confirm that the node is operating as the secondary by entering the following at the command line:

    snhamgr -m status

  13. Verify the output is:

    :default:running:default:primary:
  14. Repeat if desired to fail over to the original system operating as the primary.
  15. Verify that all clients have full access.
  16. Test access to all file systems.

If you are using the DDM feature, follow these steps:

  1. Upgrade the DDM hosts to same version of StorNext as the MDC.
  2. Enble the DDM host using the fsddmconfig command or the StorNext GUI.

To upgrade a Windows HA server to the latest version of StorNext, perform the following procedure:

  1. On the secondary MDC, navigate to Start Menu > Programs > StorNext File System > Stop and Remove to stop the File System Managers (FSMs).
  2. Upgrade the secondary MDC to the new version of StorNext.
  3. Click Services > Start to start the FSMs.

  4. From the command line, type the following for all the file systems to fail over the FSMs.

    cvadminfail <fs_name>
  5. Now the upgraded MDC should now be the primary MDC.
  6. Repeat Step 1 through Step 3 on the other (now secondary) MDC.
  7. You can initiate a failover of the MDCs to verify HA is correctly failing over if desired.

Beginning with StorNext 6.3, StorNext automatically monitors the used capacity of the HA shared file system and generates a RAS event when the file system reaches a capacity of 85% or higher.

Note: StorNext continues to generate a RAS event once an hour until the used capacity level falls below 85%.

The percent at which a RAS event is generated is controlled by the HA shared file system's configuration file parameter fsCapacityThreshold.

Note: If the fsCapacityThreshold parameter is not present in the configuration file or if it is set to zero (0), then the default of 85% is used.

To effectively disable this RAS event for the HA shared file system, set the fsCapacityThreshold parameter to 100.

Do the following to change the fsCapacityThreshold parameter for the HA shared file system:

  • Run the following command to set the secondary MDC to the locked state; this shuts down the cvfs services on the MDC:

    snhamgr mode=locked

    1. Run the following command to set the primary MDC to the config state:

      snhamgr mode=config

    2. Edit the configuration file to change the fsCapacityThreshold parameter to a value between 0 and 100 inclusive:

      sncfgedit /usr/cvfs/config/<ha_shared_fs_name>.cfgx

    3. Run the following command to return the primary MDC to the default state; this shuts down the cvfs services on the MDC:

      snhamgr mode=default

    4. Run the following command to restart the cvfs services:

      service cvfs start

    5. Run the following command and wait for the MDC status to become primary; this is displayed by the LocalStatus parameter.

      snhamgr status

      Below is an example of the output; note the LocalStatus=primary line.

      LocalMode=default

      LocalStatus=primary

      RemoteMode=locked

      RemoteStatus=stopped

    1. Run the following command to return the secondary MDC to the default state:

      snhamgr mode=default

    2. Run the following command to restart the cvfs services:

      service cvfs start

    After you complete the procedure, StorNext begins to monitor the used capacity of the HA shared file system using the updated fsCapacityThreshold value.