Quantum| MD appliance Firmware Upgrade Flow |
Obtaining, uploading, and verifing an Upgrade:
The first step in upgrading a Stornext Appliance is to obtain the file for the upgrade. You can locate these files on csweb.quantum.com. The files are generally large enough that they will have multiple parts. Here is an example of the csweb listing of firmware files for 5.3 as seen from CSWEB.
As you can see we provide the .iso files as well as the .fw files. The ISO files are for new installs of the 5.3 code were as the .fw files are for upgrade from a current code version. It is important that customer correlate the .fw files with the checksum generated before uprade to ensure the file hasn't been corrupted or the correct files was acquired.
When providing the .fw files for upgrade you can send the link to the drop box directly to the customer, this eliminates the need to transfer files to the gps.quantum.com ftp site for the customer to download.
After the .fw files have been aquired they are uploaded to the the system VIA the Stornext. This is done by going to Tools>Firmware Upgrade.
While uploading the .fw images from the GUI, you can see the progress bar update the user as to what percentage the upload is at.
In the background we're upload the file to the shared file system as seen here.
[root@upm440 upload]# pwd;ls -alh
/usr/adic/HAM/shared/upload
total 532M
drwxrwxr-x 2 root root 26 Dec 7 2015 .
drwxrwxrwx 17 root root 19 Sep 4 2014 ..
-rw-rw-r-- 1 root root 479M Dec 7 2015 QTM-DXiSNA-upd-5.3.0.OS6-16202-16195.1of2.fw
From a SSH session into the MDC BASH shell you can watch the progress as well, the watch command can be useful. This example refreshes the ls -al command in the /usr/adic/HAM/shared/upload directory every two second so you can watch the byte count of the file incriment.
Every 2.0s: ls -al Mon Dec 7 11:43:29 2015
total 675840
drwxrwxr-x 2 root root 26 Dec 7 2015 .
drwxrwxrwx 17 root root 19 Sep 4 2014 ..
-rw-rw-r-- 1 root root 662404012 Dec 7 2015 QTM-DXiSNA-upd-5.3.0.OS6-16202-16195.1of2.fw
Once you have the files uploaded into the GUI they will display the .fw files.
There is a Validate option in the GUI that will check if the upgrade is allowed. You can check the radio button in the GUI and click 'Validate'.
To look at what is being verified you can switch back the SSH session and look at /var/log/DXi/upgrades. You'll see a log file Node*.upgradeouput.log. The first thing the validation looks at is the check sum.
##From Node2.upgradeoutput.log
Dec 7 14:06:52: ******************* Verify_New_Image *********************
Dec 7 14:06:52: Checking firmware Image checksum.
Dec 7 14:06:59: Firmware Image checksum is OK.
This confirms the file passed the checksum verfication. This can be manually tested by running md5sum on the .fw file as well.
[root@downm440 upgrades]# md5sum /usr/adic/HAM/shared/upload/QTM-DXiSNA-upd-5.3.0.OS6-16202-16195.1of2.fw
ed34aabf82bdc189267526f5a49d7ba5 /usr/adic/HAM/shared/upload/QTM-DXiSNA-upd-5.3.0.OS6-16202-16195.1of2.fw
As you can see the md5sum ouput does match the file provided on csweb.
[root@downm440 upgrades]# cat /tmp/QTM-DXiSNA-upd-5.3.0.OS6-16202-16195.1of2.fw
ed34aabf82bdc189267526f5a49d7ba5 QTM-DXiSNA-upd-5.3.0.OS6-16202-16195.1of2.fw
Next the validation check if the version we're attempting to upgrade to is allowed, many times a customer must step the upgrade to a version between the current code level and the desired revision.
##From Node2.upgradeoutput.log shows start of VersionCheck.sh
Dec 07 14:05:59: ******************* VersionCheck.sh *********************
..........
Result: Allowed
Dec 07 14:05:59: Upgrade allowed.
Here i show an example of a code revision not supported, this M440 is at 5.1.0 and I'm validating a 4.7.1 code revision. The validation fails.
A quick look at the upgradeoutput.log show the validation failing.
Dec 7 14:30:44: ******************* Verify_New_Image *********************
Dec 7 14:30:44: Checking firmware Image checksum.
Dec 7 14:30:47: Firmware Image checksum is OK.
Dec 7 14:30:56: Image Version mismatch, aborting Upgrade.
Starting the Install:
The validation feature of the GUI is a bit redundant since we actually run Verify_New_Image and VersionCheck.sh again. The difference is that we run the install script which checks the snhamgr status and starts the upgrade on the secondary.
You start the upgrade by selecting the radio button on the 1 of 2 firmware file, no need to do both, the upgrade will run from here.
Go to the /var/log/DXi/upgrade directory and tail out the Node1.upgradeoutput.log.
You see that both nodes log entries for Install_New_Image and Verify_New_Image.
Aug 11 11:11:45: ******************* Install_New_Image *********************
Aug 11 11:11:45: Initiating upgrade from node 1
Aug 11 11:11:45: ******************* Verify_New_Image *********************
Aug 11 11:11:45: Checking firmware Image checksum.
Aug 11 11:11:52: Firmware Image checksum is OK.
VersionCheck.sh is now run again, we see the local files /opt/DXi/base_version and /opt/DXi/version_info are checked, notice the base version identfies this as a Stornext System, the version info is specfic to the M-series appliances.
Using '/opt/DXi/base_version'
'/opt/DXi/base_version' line: 0: Major=5
'/opt/DXi/base_version' line: 1: Minor=1
'/opt/DXi/base_version' line: 2: Patch=1
'/opt/DXi/base_version' line: 3: LCR=
'/opt/DXi/base_version' line: 4: Port=
'/opt/DXi/base_version' line: 5: Build=15404
'/opt/DXi/base_version' line: 6: Version=5.1.1-15404
'/opt/DXi/base_version' line: 7: Platform=DXiSNA
Using '/opt/DXi/version_info'
'/opt/DXi/version_info' line: 0: Major=5
'/opt/DXi/version_info' line: 1: Minor=1
'/opt/DXi/version_info' line: 2: Patch=1
'/opt/DXi/version_info' line: 3: LCR=
'/opt/DXi/version_info' line: 4: Port=
'/opt/DXi/version_info' line: 5: Build=15331
'/opt/DXi/version_info' line: 6: Version=5.1.1-15331
'/opt/DXi/version_info' line: 7: Platform=M660
Next base-version_info is checked. Notice this is a file in the PWD of the upgrade scripts. This file represents what version we are attempting to upgrade to.
Using './base-version_info'
'./base-version_info' line: 0: Major=5
'./base-version_info' line: 1: Minor=2
'./base-version_info' line: 2: Patch=1
'./base-version_info' line: 3: LCR=
'./base-version_info' line: 4: Port=
'./base-version_info' line: 5: Build=15725
'./base-version_info' line: 6: Version=5.2.1.OS6-15725
'./base-version_info' line: 7: Platform=DXiSNA
Then we check another file that came with the .fw bundle. This file is specific to the model we're upgrading. These will be checked against the /opt/DXi/base_version and /opt/DXi/version_info files later.
Using VersionTable file './VersionTable.conf.M660'
VersionTable line 11:Base: 4 3 2 * * * * DXiSNA
VersionTable line 12:Apps: 4 3 * * * * * M660
VersionTable line 15:Base: 4 3 3 * * * * DXiSNA
VersionTable line 16:Apps: 4 3 * * * * * M660
VersionTable line 19:Base: 4 6 * * * * * DXiSNA
VersionTable line 20:Apps: 4 6 * * * * * M660
VersionTable line 23:Base: 4 7 * * * * * DXiSNA
VersionTable line 24:Apps: 4 7 * * * * * M660
VersionTable line 27:Base: 5 0 * * * * * DXiSNA
VersionTable line 28:Apps: 5 0 * * * * * M660
VersionTable line 31:Base: 5 0 * * * * * DXiSNA
VersionTable line 32:Apps: 4 7 * * * * * M660
VersionTable line 35:Base: 5 1 * * * * * DXiSNA
VersionTable line 36:Apps: 5 1 * * * * * M660
VersionTable line 39:Base: 5 2 * * * * * DXiSNA
VersionTable line 40:Apps: 5 2 * * * * * M660
A third file from the .fw bundle is checked.
Using './version_info.M660'
'./version_info.M660' line: 0: Major=5
'./version_info.M660' line: 1: Minor=2
'./version_info.M660' line: 2: Patch=1
'./version_info.M660' line: 3: LCR=
'./version_info.M660' line: 4: Port=
'./version_info.M660' line: 5: Build=15725
'./version_info.M660' line: 6: Version=5.2.1-15725
'./version_info.M660' line: 7: Platform=M660
The script then checks to see what version we're at, you see this reflected as 'no match found'. For example here we are checking the VersionTable.conf.M660 files first entry "VersionTable line 11:Base: 4 3 2 * * * * DXiSNA" doesn't match the /opt/DXi/base_version file "/opt/DXi/base_version" (Ins-Base) or ./version_info.M660 (Upg-Base).
No Match Found for:
Type Major Minor Patch LCR Port Build Version Platform
Tab-Base 4 3 2 * * * * DXiSNA
Ins-Base 5 1 1 15404 5.1.1-15404 DXiSNA
Upg-Base 5 2 1 15725 5.2.1.OS6-15725 DXiSNA
No Match Found for:
Type Major Minor Patch LCR Port Build Version Platform
Tab-Base 4 3 3 * * * * DXiSNA
Ins-Base 5 1 1 15404 5.1.1-15404 DXiSNA
Upg-Base 5 2 1 15725 5.2.1.OS6-15725 DXiSNA
However when we do find a match we're allowed to continue with the upgrade.
Match Found for:
Type Major Minor Patch LCR Port Build Version Platform
Tab-Base 5 1 * * * * * DXiSNA
Ins-Base 5 1 1 15404 5.1.1-15404 DXiSNA
Upg-Base 5 2 1 15725 5.2.1.OS6-15725 DXiSNA
Match Found for:
Type Major Minor Patch LCR Port Build Version Platform
Tab-Apps 5 1 * * * * * M660
Ins-Apps 5 1 1 15331 5.1.1-15331 M660
Upg-Apps 5 2 1 15725 5.2.1-15725 M660
The effectivly stop us from upgrading from a version that isn't allowed. There are several reasons for an upgrade to be denied.
[root@upm440 5.3]# cat VersionCheck.sh | grep denied
secho "Missing '$f' file, no version info available, Upgrade denied."
secho "Cannot convert old Base version info. Upgrade denied."
secho "Cannot convert old Apps version info. Upgrade denied."
secho "Upgrade denied on $Series. No version.$Series file found in upgrade"
secho "Upgrade denied on $Series. No VersionTable.$Series file found in upgrade"
secho "Upgrade denied."
Next the Extractparts.sh is ran. This copies files that we need for the upgrade into the shared file system.
Aug 11 11:12:48: ******************* ExtractParts.sh *********************
Aug 11 11:12:48: Activated image /usr/adic/HAM/shared/upload/QTM-DXiSNA-upd-5.2.1.OS6-15725-15662.1of2(1).fw in upgrade path /usr/adic/HAM/shared/upload/new-firmware.20432
Aug 11 11:12:48: Skipping our already extracted image file /usr/adic/HAM/shared/upload/QTM-DXiSNA-upd-5.2.1.OS6-15725-15662.1of2(1).fw
Aug 11 11:12:52: Found PART 2 of required 2 in matching upgrade /usr/adic/HAM/shared/upload/QTM-DXiSNA-upd-5.2.1.OS6-15725-15662.2of2(1).fw
Aug 11 11:12:52: Found the following matching FW files to extract and apply:
/usr/adic/HAM/shared/upload/QTM-DXiSNA-upd-5.2.1.OS6-15725-15662.2of2(1).fw
Aug 11 11:12:52: Extracting /usr/adic/HAM/shared/upload/QTM-DXiSNA-upd-5.2.1.OS6-15725-15662.2of2(1).fw to /usr/adic/HAM/shared/upload/new-firmware.20432
Now we run the VerifyAPIVersion script, if we match here the install script starts looking at the array communications.
Aug 11 11:13:08: ******************* VerifyAPIVersion *********************
Aug 11 11:13:14: API versions between the platform and StorNext software match!
Aug 11 11:13:14: Platform API version is 1 StorNext API version is 1
Aug 11 11:13:14: Copying the upgrade script to local storage on all nodes.
Starting SMagent
SMagent started.
Waiting for SMagent to complete startup...
Found array Qarray1!
Checking that controller A is online...
Controller A is online
Checking that controller B is online...
Controller B is online
Aug 11 11:12:37: ha_smith_interval is already set to 20
Aug 11 11:12:37:
Aug 11 11:12:37: ############################################
Aug 11 11:12:37: Starting Upgrade/Install at Tue Aug 11 11:12:37 PDT 2015
Aug 11 11:12:37:
The PreUpgradeScript runs next.
Aug 11 11:12:37: ******************* PreUpgradeChk *********************
Aug 11 11:12:46: Passed PreUpgradeChk.DXiSNA script
At this point if you're looking primary node you see logging like such as the install script starts
Aug 11 11:13:14: Removing RPMS from /usr/adic/HAM/shared/upload/new-firmware.8305
Aug 11 11:13:33: Enabling boot time upgrade script on node 1
Aug 11 11:13:44: Checking StorNext bundle upgrade to see if it is valid for this node
- The maintenance license status is: Good
Upgrade from version 5.2.0(51843A) to version 5.2.1(55370A) is allowed
Aug 11 11:13:49: StorNext bundle upgrade is valid for this node
Aug 11 11:13:49: Waiting for shutdown to complete
Aug 11 11:23:11:
Aug 11 11:23:11: ############################################
However if you're looking at the secondary node you'll see we wait for 3 hrs for the secondary to complete.
Aug 11 11:14:04: ******************* PreUpgradeChk *********************
Aug 11 11:14:05: Passed PreUpgradeChk.DXiSNA script
Aug 11 11:14:51: Waiting up to 10800 seconds for the secondary node to complete the upgrade <<<Primary Waiting on the secondary.
Now we start the Upgrade.<Version> script. We start SMagent.
Checking if SMagent is running
Starting SMagent
SMagent started.
This is typical logging from starting SMagent seen in the log...
Checking device /dev/sdi (/dev/sg10) : Activating
Checking device /dev/sdct (/dev/sg100) : Skipping
Checking device /dev/sdcu (/dev/sg101) : Skipping
Checking device /dev/sdcv (/dev/sg102) : Skipping
Checking device /dev/sdcw (/dev/sg103) : Activating
After the array connectivity is established we start the upgrade.
Aug 11 12:12:29: Starting Upgrade/Install at Tue Aug 11 12:12:29 PDT 2015
RPM checksums are checked and installed.
Starting Upgrade
Aug 11 12:12:30: Verifying RPM checksums
Verifying RPM checksums
Aug 11 12:12:31: RPM checksums are OK
RPM checksums are OK
Aug 11 12:12:31: Starting RPM installations
Starting RPM installations
Aug 11 12:12:31: Installing Base
Installing Base
UpgradeBase.sh is started and rpms are handled.
Aug 11 12:12:31: ***************** UpgradeBase.sh ******************
***************** UpgradeBase.sh ******************
Aug 11 12:12:31: Starting RPM installations
Starting RPM installations
Removing RPM ixgbe-
After the RPMs having been cleaned up and installed you start to see progress entries in the log as such.
1) Upgrade Pre-Install 5.2.1(55370A) Working \
Stop Tomcat To do
Stop /usr/adic/bin/adic_control To do
Stop Dsm To do
2) Upgrade perl 5.2.1(55370A) To do
3) Upgrade docs 5.2.1(55370A) To do
4) Upgrade mysql 5.2.1(55370A) To do
5) Upgrade database 5.2.1(55370A) To do
6) Upgrade SRVCLOG 5.2.1(55370A) To do
7) Upgrade PSE 5.2.1(55370A) To do
8) Upgrade MSM 5.2.1(55370A) To do
9) Upgrade TSM 5.2.1(55370A) To do
10) Upgrade java 5.2.1(55370A) To do
11) Upgrade tomcat 5.2.1(55370A) To do
12) Upgrade gui 5.2.1(55370A) To do
13) Upgrade DSM 5.2.1(55370A) To do
14) Upgrade Post-Install 5.2.1(55370A) To do
There are many steps to the main script upgrade. You can check the point in which the upgrade made it, if there is a failure by grepping for checkpoint.
[root@downm660 upgrades]# grep checkpoint Node1.upgradeoutput.log
Nov 2 09:42:07: Starting upgrade checkpoint VerifyRPMs
Starting upgrade checkpoint VerifyRPMs
Nov 2 09:42:09: Finished upgrade checkpoint VerifyRPMs
Finished upgrade checkpoint VerifyRPMs
Nov 2 09:42:09: Starting upgrade checkpoint RemoveRPMs
Starting upgrade checkpoint RemoveRPMs
Nov 2 09:42:27: Finished upgrade checkpoint RemoveRPMs
Finished upgrade checkpoint RemoveRPMs
If you see a checkpoint that did start but didn't finish it's a good place to start looking for the failure. Try to understand from the logging which script was running, where it failed and why, much like the above example from the VersionCheck.sh. As you come across helpful tips please feel free to add them to the Useful Notes For Support during firmware upgrades wiki section.
This page was generated by the BrainKeeper Enterprise Wiki, © 2018 |