Most-Often Used Logs: Tsunami Log and Messages Log

Tsunami Log Overview

The tsunami log includes information about services and processes such as truncation, tape states, and space reclamation. Among the things you can find information about in this log are:

I/O issues (device driver issues
Network problems
ERROR statements related to the component you are having problems with
Space reclamation (bpgc, which stands for blockpool garbage collection) status
VTL mount and un-mount info (tape mount/unmount)
TL State information
hwmon (which stands for hardware monitor) status
qbfsd (which stands for Quantum blockpool file system daemon) status

Space Reclamation (bpgc) Status Example

INFO - 11/23/10-13:10:27 - bpgc ReconcileThread.cpp(1773) [bpgc] determineCandidates() - {1082132832} Determining the blockbool deletion candidates

INFO - 11/23/10-13:10:27 - bpgc FileUtil.cpp(201) [bpgc] sortFile() - Sorting file /data/hurricane/bpgc/blockpooltags

Mount/Dismount Example

INFO - 11/29/10-16:22:13 - Vmm.ThreadMethods ThreadMethods.cc(320) [vmm] mountMedium() - Mount of barcode <EA0005> requested from reqId 2033341285

INFO - 11/29/10-16:22:13 - Vmc.CommandGeneric MoveMedium.cc(214) [vmcVL02CX1029BVA00256] ExecuteCommand() - MoveMedium::ExecuteCommand: requestId 2033

341285: Generic Success

INFO - 11/29/10-16:22:13 - VTDMessenger vtdMessenger.cpp(267) [vtdVD15CX1029BVA00256] VTD_MountHandler() - Mount: barcode: EA0005; pathToTape: /Q/part

itions/VL02CX1029BVA00256/Carts/EA0005_CX1029BVA00256_262688; writeProtect: false

INFO - 11/29/10-16:22:13 - Vmm.MediumWorker MediumWorker.cc(80) [vmm] runCmd() - Locking medium EA0005

INFO - 11/29/10-16:22:13 - Vmm.BaseCmd VTDMountMediumCmd.cc(132) [vmm] doExecute() - Sending VTD_MountMsg for mount of barcode <EA0005> pathToTape </Q

/partitions/VL02CX1029BVA00256/Carts/EA0005_CX1029BVA00256_262688> to drive <VD15CX1029BVA00256>

INFO - 11/29/10-16:22:13 - Vmm.DbLoader VmmDbMediumLoader.cc(535) [vmm] update() - Updated barcode: <EA0005>, state: </Q/partitions/VL02CX1029BVA00256

/Carts/EA0005_CX1029BVA00256_262688 - 385>, vmc <VL02CX1029BVA00256> in the database for reqId 2033341285

INFO - 11/29/10-16:22:13 - Vmm.MediumWorker MediumWorker.cc(87) [vmm] runCmd() - Unlocking medium EA0005

INFO - 11/29/10-16:22:15 - Vmm.ThreadMethods ThreadMethods.cc(349) [vmm] dismountMedium() - Dismount of barcode <EA0005> requested by reqId 2033341345

INFO - 11/29/10-16:22:15 - Vmm.MediumWorker MediumWorker.cc(80) [vmm] runCmd() - Locking medium EA0005

INFO - 11/29/10-16:22:15 - Vmm.BaseCmd VTDDismountMediumCmd.cc(90) [vmm] doExecute() - Sending VTD_DismountMsg for dismount of drive <VD15CX1029BVA002

56>

INFO - 11/29/10-16:22:15 - VTDMessenger vtdMessenger.cpp(396) [vtdVD15CX1029BVA00256] VTD_DismountHandler() - Dismount: EA0005 forceUnload: false

INFO - 11/29/10-16:22:15 - Vmm.BaseCmd VMMDismountCmd.cc(68) [vmm] doExecute() - VTDDismountMediumCmd Drive <VD15CX1029BVA00256> Barcode <EA0005> Phys

ical dismount <0> Virtual force unload <0> Status <0 - Generic Success>

INFO - 11/29/10-16:22:15 - Vmm.DbLoader VmmDbMediumLoader.cc(535) [vmm] update() - Updated barcode: <EA0005>, state: </Q/partitions/VL02CX1029BVA00256

/Carts/EA0005_CX1029BVA00256_262688 - 384>, vmc <VL02CX1029BVA00256> in the database for reqId 2033341345

INFO - 11/29/10-16:22:15 - Vmm.BaseCmd VMMDismountCmd.cc(85) [vmm] doExecute() - Medium.unsetMounted Barcode <EA0005> Status <0 - Generic Success>

INFO - 11/29/10-16:22:15 - Vmm.MediumWorker MediumWorker.cc(87) [vmm] runCmd() - Unlocking medium EA0005

INFO - 11/29/10-16:22:15 - Vmc.CommandGeneric MoveMedium.cc(101) [vmcVL02CX1029BVA00256] ExecuteCommand() - MoveMedium::ExecuteCommand: requestId 2033

341345: Generic Success

Location

In the system diagnostics file: scratch\collect\node1-collection\app-info\tsunami.log
On the system: /var/log/DXi/tsunami.log or you can access the symbolic link/shortcut under /hurricane/tsunami.log

Messages Log Overview

The messages log includes information about the operating system and StorNext filesystem issues. It can be useful for finding information on:

Core symptoms, memory problems, and system level issues.
Panics and/or core dump info.
General boot sequence for systems not bringing up FSM.
Disk Automatic Event Notifications (AEN) errors. An AEN is a message issued by the 3ware RAID controller when an error is detected or when an action is completed.

Kernel Panic Example

# grep KDUMP /var/log/messages

Jul 02 11:43:48 Flames KDUMP: CRIT : KERNEL PANIC/OOPS/CRASH has occurred

Jul 02 11:43:50 Flames KDUMP: INFO : *********begin stack traces ********

Jul 02 11:44:26 Flames KDUMP: PID: 30096 TASK: ffff8107a9d0e140 CPU: 2 COMMAND: "bash"

AEN Example

Sep 10 00:07:40 labdxi1 3wcache.sh: Warning: non-optimal cache settings for /c2/u0
Sep 10 00:05:13 labdxi1 kernel: 3w-9xxx: scsi2: AEN: INFO (0x04:0x0029): Verify started:unit=0.

Sep 10 00:07:41 labdxi1 3wcache.sh: /c2/u0 Read Cache = Intelligent

Sep 10 00:07:42 labdxi1 3wcache.sh: /c2/u0 Write Cache = off

Sep 10 00:07:42 labdxi1 3wcache.sh: Unable to correct cache settings due to BBU status:

Sep 10 00:07:42 labdxi1 3wcache.sh: /c2/bbu On No Failed OK OK 0 xx-xxx-xxxx

Sep 10 00:07:42 labdxi1 3wcache.sh: Warning: non-optimal cache settings for /c2/u1

Sep 10 00:07:42 labdxi1 3wcache.sh: /c2/u1 Read Cache = Intelligent

Sep 10 00:07:43 labdxi1 3wcache.sh: /c2/u1 Write Cache = off

Sep 10 00:07:43 labdxi1 3wcache.sh: Unable to correct cache settings due to BBU status:

Sep 10 00:07:43 labdxi1 3wcache.sh: /c2/bbu On No Failed OK OK 0 xx-xxx-xxxx

Sep 10 00:10:33 labdxi1 3wcache.sh: Warning: non-optimal cache settings for /c2/u0

Sep 10 00:10:34 labdxi1 3wcache.sh: /c2/u0 Read Cache = Intelligent

Sep 10 00:10:35 labdxi1 3wcache.sh: /c2/u0 Write Cache = off

Sep 10 00:10:35 labdxi1 3wcache.sh: Unable to correct cache settings due to BBU status:

Sep 10 00:10:35 labdxi1 3wcache.sh: /c2/bbu On No Failed OK OK 0 xx-xxx-xxxx

Sep 10 00:10:35 labdxi1 3wcache.sh: Warning: non-optimal cache settings for /c2/u1

Sep 10 00:10:36 labdxi1 3wcache.sh: /c2/u1 Read Cache = Intelligent

Sep 10 00:10:36 labdxi1 3wcache.sh: /c2/u1 Write Cache = off

Sep 10 00:10:36 labdxi1 3wcache.sh: Unable to correct cache settings due to BBU status:

Sep 10 00:10:36 labdxi1 3wcache.sh: /c2/bbu On No Failed OK OK 0 xx-xxx-xxxx
Sep 10 00:12:33 labdxi1 kernel: 3w-9xxx: scsi1: AEN: WARNING (0x04:0x0023): Sector repair completed:encl=0, slot=4, LBA=0x1EBAA8A.

Location

In System Diagnostics: scratch\collect\node1-collection\os-info\messages
On the system: /var/log/messages

Potential Gotchas - Spurious Shutdown Errors

One specific way that the tsunuami log (and perhaps the also the messages log) can be misleading is that during shutdown sequences, you will see errors that are due to various processes being stopped, while others continue. If you are focused in on finding ERROR statements, this could lead you to think you’ve spotted a problem, when it's actually only a brief anomaly.

What's Next?

The Collect.txt Log >