Overview

Example SR: SR3726066   Product / Software Version: Lattus 4.x and earlier software versions   Problem Description: When an ungraceful shutdown of Lattus storage node occurs, there can be PID (Process IDentifier) files that do not get cleaned up properly. This results in the application not starting automatically and events getting generated.

 

 

Lattus events that occur while in this status:

 

• Application applicationsserver is HALTED
  • Event lists the affected node
• Machine agent has status ‘HALTED’
  • Event lists the affected node
• Job ‘Restarting application applicationserver’ failed on machines [‘BRDNLCN01-mgmt’]
  • Event list the management node because that is where the “Job” failed.
  • The job failed attempting to restart the applicaitonserver for node.

This issue has been seen on other applications but mostly on the applicationserver. A manual clean up is required to resolve the issue.

 

Existing Bugs:

 

• Quantum bugzilla CR48066
• HGST confluence BDY036

Signature that the node was recently rebooted but the old PID file remains (commands were run on Aug 1st):

 

root@storage076:~# uptime
08:23:07 up 2 days, 13:37,  1 user,  load average: 0.03, 0.22, 0.24

root@storage076:/opt/qbase3/var/pid# ls -al
-rw-------  1 root root    4 May  9 16:01 applicationserver.pid
-rw-r--r--  1 root root    4 Jul 29 18:46 monitor_agent.pid
-rw-r--r--  1 root root    5 Jul 29 18:46 rsync.pid
 

 

---

Procedure

To remove the old pid and restart the applications, do the following:

 

# rm /opt/qbase3/var/pid/applicaitonserver.pid
# qshell -c “q.manage.servers.all.stop()”
# qshell -c “q.manage.servers.all.start()”

 

 

---