Convert to a High Availability (HA) System

Follow these steps to configure HA:

1. Using the StorNext GUI, on the Tools menu, click High Availability, and then click Convert. The High Availability page appears.

   Note: If the system has previously been converted to HA mode, continue to Step 6.

   

2. On the Shared File System option, select the shared file system you want to use for the HA pair.

   Caution: Once you convert a file system to HA you cannot undo the operation, so choose carefully when selecting the file system.

   Note: The file system should not have quotas enabled. Enabling quotas on the file system can interfere with the proper functionality of Storage Manager and the HA infrastructure.

   Caution: On the Tools > High Availability page, it is critical that you assign the Metadata network IP address in the MDC Address field and not the Management network IP address.

   Caution: Double-check that the Shared File System option is set to the shared file system on the metadata array for the metadata appliance/HA pair and NO OTHER shared file system on the list before applying changes. The file system selection on this page may only be done once, so make sure you select the shared file system on the metadata array.

   Note: A virtual IP (VIP) may also be set at this time. Use the IP addresses assigned to the port used for the Management network on each MDC server for the VIP values. Do not use the IP addresses assigned for the ports used for the Metadata network.

3. For the MDC Address, select one IP address to be placed in the ha_peer file for use in administrative operations between the MDC servers in the HA pair.
4. For non-Lattus systems ONLY: If your HA cluster also runs the blockpool, select Enable and then enter the virtual IP address and virtual netmask. Ask your network administrator for the vIP address and netmask.

5. Click Convert to convert the primary MDC server to HA.

   Note: Once the primary MDC server has been converted, the Status displays Converted.

6. Enter the IP address of the secondary MDC server in the System Name field along with a port number on the same LAN as the primary MDC server.

7. (If the system is configured with a DNS entry, you may choose to use that name in place of the IP address in this field.) Click Scan Host. The system should resolve the secondary MDC server - the MDC Address will auto-fill with a value. If you do not already have licenses for the secondary system in the license file, you will be required to switch to the license page to import them before continuing. The information comes from the license.dat file

   Note: .Until you have performed the scan, you cannot import the license file for the secondary system using the StorNext import function. After you have performed the scan you can import licenses for the secondary. Following the conversion to HA, the license file contains licenses and is present on the servers.

8. Click Convert to convert the secondary MDC server.

   Note: Both MDC servers will reboot, which can take 30 minutes or more per server to complete.

   

9. Storage Manager may need to be started following the HA conversion if the system was in config mode at the time that HA conversion was initiated. To restart the Storage Manager components, click Start button in the Storage Manager panel of the Tools > System Control page.

There are some indications within the GUI that the system is being upgraded. Here are some notes about this visual feedback:

• After the StorNext upgrade has completed, and the HA configuration has been done, the GUI for the secondary MDC server provides a message stating it is not the primary MDC server and a link to launch the primary MDC server.

  Caution: Do not log in to the GUI of the secondary MDC server at any point during the upgrade/HA conversion process. System configuration and licensing for the system could be compromised.

• When you are able to log into the primary system/server, after accepting the EULA, the system will automatically display the Tools > System Control page. Click Start to restart the Storage Manager components.
• Wait until the system icons for both MDC servers of the system as well as File System and Storage Manager are green, which indicates normal operation.

If you are using the DDM feature, perform the following:

1. If you use the secondary MDC server as a DDM mover, make sure the file systems are mounted.
2. Edit the fs_sysparm or the fs_sysparm_override files to use your preferred DDM mode: All or Threshold. Use the adic_control restart TSM command to put this change into effect.

After an initial conversion of your system to HA, your may want to verify HA failover. To initiate the failover of a system that has just been converted to HA:

1. Open an SSH connection to the MDC server operating as the primary.

2. Log in to the command line of the primary MDC server, using the following credentials:

   User: stornext

   Password: (You (the customer) should have the password for the StorNext account)

3. Type sudo rootsh to gain root user access. Enter the password again.

4. Confirm that the MDC server is operating as the primary by entering the following at the command prompt:

   snhamgr -m status

5. Verify the output is (bold used for clarification):

   :default:primary:default:running:

6. Initiate an HA failover on the primary MDC server to the secondary MDC server.

   Caution: Only use the snhamgr force smith or cvadmin -e hamon smith commands to initiate a system failover the first time the system is converted to HA and should only be done when the system is operating in a non-production environment. These commands cause a HARD RESET of the MDC server, and will impact StorNext operations. For initiating a failover any other time, see Initiate a Graceful System Failover for the graceful HA failover procedure.

   1. Issue the following command to stop StorNext services on the primary MDC server:

      adic_control stop

   2. Then, issue the following command to initiate the failover:

      snhamgr force smith

   Note: Restarting StorNext more than once every three minutes can trigger a startup delay of three minutes. The delay slows down looping HA resets so that an administrator can correct any configuration problems that are causing the resets.

7. Wait 3 minutes until the MDC server previously operating as the secondary becomes the primary.
8. Open an SSH connection to the MDC server now operating as the primary.
9. Log in to the command line of the primary MDC server.

10. Confirm that the MDC server is operating as the primary by entering the following at the command prompt:

    snhamgr -m status

11. Verify the output is:

    :default:primary:default:running:

12. Repeat if desired to fail over to the original system operating as the primary.

    Note: When the command force smith is used on an MDC server, the system will reboot, and it may take a significant amount of time for the MDC server to come back online, so plan for this additional delay. Wait until the MDC server reboot has completed before initiating another fail over to the MDC server originally operating as the primary.

13. Once the failover has completed, restart SNFS services on any clients that were stopped earlier.
14. Mount the SNFS file systems on each client machine, if needed.
15. Verify that all clients have full access.
16. Verify access to all file systems and move files to/from disk and tape.

During the upgrade of the server originally operating as the primary, the server originally operating as the secondary server becomes the primary. After the database updates are complete and the upgrade of the original primary completes, a failover of the server currently operating as the primary should be triggered so it once again is operating as the secondary, and to reinstate the original primary server to operate as primary again. During StorNext 5.x or later upgrades, the Metadata Appliance fails over once. Because of this single failover, the MDC server originally operating as primary is set as the secondary after the upgrade. This procedure provides the steps necessary to set the MDC servers back to the system state prior to the upgrade for this scenario.

To initiate the graceful failover of an HA pair:

1. Open an SSH connection to the MDC server operating as the primary.

2. Log in to the command line of the primary MDC server, using the following credentials:

   User: stornext

   Password: (You (the customer) should have the password for the StorNext account)

3. Type sudo rootsh to gain root user access.
4. Enter the password a second time.

5. Confirm that the MDC server is operating as the primary by entering the following at the command line:

   snhamgr -m status

6. Verify the output is (bold used for clarification):

   :default:primary:default:running:

7. On the MDC server operating as the primary, initiate an HA failover to the MDC server operating as the secondary:

   service cvfs stop
8. Wait until the secondary MDC server becomes the primary, and leave your SSH connection to this server open. The time may vary.
9. Open an SSH connection to the MDC server now operating as the primary.

10. Log in to the command line of the primary MDC server, using the following credentials:

    User: stornext

    Password: (You (the customer) should have the password for the StorNext account)

11. Type sudo rootsh to gain root user access.
12. Enter the password a second time.

13. Confirm that the MDC server is operating as the primary by entering the following at the command line:

    snhamgr -m status

14. Verify the output is:

    :default:primary:default:stopped:

15. From the SSH connection to the MDC server now operating as the secondary, enter the following:

    service cvfs start

16. Confirm that the MDC server is operating as the secondary by entering the following at the command line:

    snhamgr -m status

17. Verify the output is:

    :default:running:default:primary:
18. Repeat if desired to fail over to the original system operating as the primary.
19. Verify that all clients have full access.
20. Test access to all file systems.