StorNext MDC ntp time between nodes mismatched after reboot, forcing 2nd reboot

METHODOLOGY TO CORRECTLY CHECK AND  SYNC TIME VIA NTP TO TIME SERVER ON PRIMARY AND SECONDARY MDCS:

 

From: Jeff Syme 

Sent: Wednesday, February 17, 2016 4:47 PM
To: DL-AMER-SPS
Subject: Setting Time on MDC with NTP support: ntpd, ntpdc, chkconfig | grep ntp, ntp.conf, hwclock, ntpdate, service ntpd start, systemctl

 

Team,

 

Problem:

StorNext 4.7.1 MDC time between  nodes mismatched after reboot, forcing 2nd reboot.

ASSETS:

Definitions:

Network Time Protocol (NTP) is a networking protocol for clock synchronization between computer systems over packet-switched, variable-latency data networks. In operation since before 1985, NTP is one of the oldest Internet protocols in current use. This is usually an administrative task outside the scope of StorNext support, until it isn’t.

Remediation Steps:

 

Option1: GUI to Primary MDC – network - time

/// You can simply force a time synch in the GUI and it will immidiately reboot. Know this in advance when setting your customer’s expectations

 

Option 2: CLI /ssh to both MDCs

/// Check your surroundings, begin your efforts on the Secondary MDC, NOT running Storage Manager.

 

 

[root@upm660 config]# ps -ef | grep ntp

[root@upm660 config]#

/// Ntp is not running as a process

 

[root@upm660 config]# chkconfig | grep ntp

ntpd            0:off   1:off   2:off   3:off    4:off    5:off    6:off

ntpdate         0:off   1:off   2:off   3:off   4:off   5:off   6:off

/// ntp is not configured to start on boot

 

/// configure ntpd to start on boot

[root@upm660 config]# chkconfig ntpd on

[root@upm660 config]# chkconfig | grep ntp

ntpd            0:off   1:off   2:on    3:on    4:on    5:on    6:off

ntpdate         0:off   1:off   2:off   3:off   4:off   5:off   6:off

[root@upm660 config]#

 

/// Attempt to ping the customer’s defined time server  section called --- OUR TIMESERVERS -----

[root@upm660 config]# vi /etc/ntp.conf

 

///Here is evidence of the German Krakow’s good file remarks “who and when” edits

# --- OUR TIMESERVERS -----

# time.nist.gov

# server 192.43.244.18                                               # <<== No longer works 02/16/16 G.K.

server 128.138.141.172

restrict 28.138.141.172 mask 255.255.255.255 nomodify notrap noquery

 

#stratum2.ord1.publicntp.net                                         # <<== No longer works 02/16/16 G.K.

#server 208.66.174.71                                                # <<== No longer works 02/16/16 G.K.

#restrict 208.66.174.71 mask 255.255.255.255 nomodify notrap noquery

 

#clock.redhat.com                                                    # <<== No longer works 02/16/16 G.K.

#server 66.187.233.4                                                 # <<== No longer works 02/16/16 G.K.

#restrict 66.187.233.4 mask 255.255.255.255 nomodify notrap noquery

# ?

# server 75.13.24.211                                                # <<== No longer works 02/16/16 G.K.

# restrict 75.13.24.211 mask 255.255.255.255 nomodify notrap noquery

/// :q  to exit without saving

/// :qw!  to exit with saving

 

[root@upm660 etc]# ping 128.138.141.172

PING 128.138.141.172 (128.138.141.172) 56(84) bytes of data.

64 bytes from 128.138.141.172: icmp_seq=1 ttl=53 time=4.80 ms

^C

--- 128.138.141.172 ping statistics ---

3 packets transmitted, 3 received, 0% packet loss, time 2502ms

rtt min/avg/max/mdev = 4.101/4.345/4.804/0.324 ms

[root@upm660 etc]#

/// results are good! the NTP is reachable. Now find a secondary address as a backup.

/// Ask for an internal resources when possible.

 

///Compare the OS system clock with your timezone:

[root@upm660 etc]# date

Wed Feb 17 15:24:57 MST 2016

[root@upm660 etc]#

 

/// Now check the Hardware clock to see if it is sycnh

[root@upm660 etc]# hwclock

Wed 17 Feb 2016 03:30:32 PM MST  -0.187721 seconds

[root@upm660 etc]#

/// results are good! Should an update be needed, update the hardware clock with hwclock -w

[root@upm660 etc]# hwclock -w

[root@upm660 etc]#

 

/// to tell if we can reach it and what the offsets are issue # ntpdate <preferred NTP server IP>

[root@upm660 etc]# ntpdate 128.138.141.172

17 Feb 15:42:13 ntpdate[3374]: step time server 128.138.141.172 offset 3.731942 sec

[root@upm660 etc]#

 

[root@upm660 etc]# service ntpd start

Starting ntpd:                                             [  OK  ]

[root@upm660 etc]#

 

 

/// Check your work:

root@upm660 etc]# hwclock

Wed 17 Feb 2016 03:30:32 PM MST  -0.187721 seconds

[root@upm660 etc]#

 

///After you’ve re-written the /etc/ntp.conf simply copy it to the other MDC via HAM shared between M-series Appliances and yes overwrite

[root@upm660 etc]# cp /etc/ntp.conf

[root@downm660 etc]# cp /usr/adic/HAM/shared/ntp.conf /etc/

[root@downm660 etc]# rm /usr/adic/HAM/shared/ntp.conf

 

///Now check to ensure both MDC are fully operationl connected and in-synch. Make sure your NTP service has  a non-zero value

[root@upm660 etc]# ntpdc

ntpdc> peer

     remote           local      st poll reach  delay   offset    disp

=======================================================================

=LOCAL(0)        127.0.0.1       10   64  360 0.00000  0.000000 0.43832

*utcnist2.colora 10.20.232.72     1   64  377 0.00359  0.025754 0.03848

^10.17.21.255    10.17.21.1      16   64    0 0.00000  0.000000 4.00000

ntpdc>

/// q to quit

Now on the other MDCx / same commands…

 

/// The time has been adjusted… consider if you need to restart Storage Manager

 

 

# systemctl is for red Hat 7 Centos 7

 

 

Best regards,


Quantum
Jeff D. Syme StorNext - Software Product Support (SPS)  sndocs & sn5docs & lattusdocs & Xcellisdocs

Staffed M-F:8AM-5PM MT O: 719.536.6411 | Jeff.Syme@Quantum.com | Quantum.com

”Success is not final, failure is not fatal; it is the courage to continue that counts.” – Winston Churchill

 



This page was generated by the BrainKeeper Enterprise Wiki, © 2018