kdump analysis with Galaxy 3.0 (draft)

The crash command output depends also of the timezone setting where you analyze the kdump.

The collect log for the affected system shows in collect.txt:
###15:01:44### -Date SysconfigClock- 'cat /etc/sysconfig/clock':
ZONE="Europe/Paris"

If you do
# export TZ=Europe/Paris
crash  shows:

     KERNEL: vmlinux-2.6.32-431.11.2.el6.Q_3.x86_64
    DUMPFILE: vmcore  [PARTIAL DUMP]
        CPUS: 40
        DATE: Fri Nov 21 01:22:36 2014
      UPTIME: 08:25:19
LOAD AVERAGE: 1.07, 0.92, 0.79
       TASKS: 3004
    NODENAME: vtldxi6902ha01

This is the right timestamp in the local time context.
This appears to be the timestamp for the KDUMP when the mini-kernel was woke up to handle the kernel Oops. It was logged in the messages file at 01:31 on Nov but the actual time that the kernel Oops was around 16:22 on Nov 20:
 
Nov 21 01:31:43 vtldxi6902ha01 KDUMP: INFO : kcdump: KDUMP = 127.0.0.1-2014-11-20-16:22:45
 
  SYSTEM MAP: System.map-2.6.32-431.11.2.el6.Q_3.x86_64
DEBUG KERNEL: vmlinux-2.6.32-431.11.2.el6.Q_3.x86_64 (2.6.32-431.11.2.el6.Q_3.x86_64)
    DUMPFILE: vmcore  [PARTIAL DUMP]
        CPUS: 40
        DATE: Thu Nov 20 17:22:36 2014
      UPTIME: 08:25:19
LOAD AVERAGE: 1.07, 0.92, 0.79
       TASKS: 3004
    NODENAME: vtldxi6902ha01
     RELEASE: 2.6.32-431.11.2.el6.Q_3.x86_64
     VERSION: #1 SMP Thu Apr 3 15:42:03 PDT 2014
     MACHINE: x86_64  (2499 Mhz)
      MEMORY: 256 GB
       PANIC: "Oops: 0002 [#1] SMP " (check log for details)
         PID: 14846
     COMMAND: "pidof"
        TASK: ffff883367f69540  [THREAD_INFO: ffff883146f18000]
         CPU: 23
       STATE: TASK_RUNNING (PANIC)
 
From the crash sys command the timestamp shows 17:22 on Nov 20.

Why the one hour difference?

Peter answered this:
The starting kernel has no clue about timezone offsets at that time as it would need to read /etc/localtime to find out.

And from Daniel:
The collect log for the affected system shows in collect.txt:
###15:01:44### -Date SysconfigClock- 'cat /etc/sysconfig/clock':
ZONE="Europe/Paris"

If you do
# export TZ=Europe/Paris

The crash sys command should now show the correct timestamp.
 

 
 
The stacktrace of the KDUMP in the messages file timestamp is when it was logged:
 
Nov 21 01:49:05 vtldxi6902ha01 KDUMP: INFO :     *********begin  stack traces ********
Nov 21 01:49:20 vtldxi6902ha01 KDUMP:          PID: 14846  TASK: ffff883367f69540  CPU: 23  COMMAND: "pidof"
Nov 21 01:49:20 vtldxi6902ha01 KDUMP:          #0 [ffff883146f19680] machine_kexec at ffffffff81038f3b
Nov 21 01:49:20 vtldxi6902ha01 KDUMP:          #1 [ffff883146f196e0] crash_kexec at ffffffff810c5d82
Nov 21 01:49:20 vtldxi6902ha01 KDUMP:          #2 [ffff883146f197b0] oops_end at ffffffff8152bae0
Nov 21 01:49:20 vtldxi6902ha01 KDUMP:          #3 [ffff883146f197e0] no_context at ffffffff8104a00b
Nov 21 01:49:20 vtldxi6902ha01 KDUMP:          #4 [ffff883146f19830] __bad_area_nosemaphore at ffffffff8104a295
Nov 21 01:49:20 vtldxi6902ha01 KDUMP:          #5 [ffff883146f19880] bad_area_nosemaphore at ffffffff8104a363
Nov 21 01:49:20 vtldxi6902ha01 KDUMP:          #6 [ffff883146f19890] __do_page_fault at ffffffff8104aabf
Nov 21 01:49:20 vtldxi6902ha01 KDUMP:          #7 [ffff883146f199b0] do_page_fault at ffffffff8152da2e
Nov 21 01:49:20 vtldxi6902ha01 KDUMP:          #8 [ffff883146f199e0] page_fault at ffffffff8152ade5
Nov 21 01:49:20 vtldxi6902ha01 KDUMP:          [exception RIP: __mem_cgroup_commit_charge+0xa7]
Nov 21 01:49:20 vtldxi6902ha01 KDUMP:          RIP: ffffffff8117c897  RSP: ffff883146f19a98  RFLAGS: 00010206
Nov 21 01:49:20 vtldxi6902ha01 KDUMP:          RAX: 0000000000000017  RBX: ffffea00e08d1870  RCX: 0000000000000001
Nov 21 01:49:21 vtldxi6902ha01 KDUMP:          RDX: 00000000000005c0  RSI: ffff884060705020  RDI: ffffc900180c6000
Nov 21 01:49:21 vtldxi6902ha01 KDUMP:          RBP: ffff883146f19a98   R8: ffffc900180c6000   R9: 0000000000001000
Nov 21 01:49:21 vtldxi6902ha01 KDUMP:          R10: 0000000000013560  R11: 0000000000000000  R12: ffff884060705020
Nov 21 01:49:21 vtldxi6902ha01 KDUMP:          R13: 0000000000001000  R14: 00000000000000d0  R15: ffff884050fe8800
Nov 21 01:49:21 vtldxi6902ha01 KDUMP:          ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018
Nov 21 01:49:21 vtldxi6902ha01 KDUMP:          #9 [ffff883146f19aa0] mem_cgroup_charge_common at ffffffff8117c9a5
Nov 21 01:49:21 vtldxi6902ha01 KDUMP:          #10 [ffff883146f19b10] mem_cgroup_newpage_charge at ffffffff8117ca18
Nov 21 01:49:21 vtldxi6902ha01 KDUMP:          #11 [ffff883146f19b20] handle_pte_fault at ffffffff8114acde
Nov 21 01:49:21 vtldxi6902ha01 KDUMP:          #12 [ffff883146f19c00] handle_mm_fault at ffffffff8114b29a
Nov 21 01:49:21 vtldxi6902ha01 KDUMP:          #13 [ffff883146f19c70] __do_page_fault at ffffffff8104a8d8
Nov 21 01:49:21 vtldxi6902ha01 KDUMP:          #14 [ffff883146f19d90] do_page_fault at ffffffff8152da2e
Nov 21 01:49:21 vtldxi6902ha01 KDUMP:          #15 [ffff883146f19dc0] page_fault at ffffffff8152ade5
Nov 21 01:49:21 vtldxi6902ha01 KDUMP:          [exception RIP: copy_user_enhanced_fast_string+0x6]
Nov 21 01:49:21 vtldxi6902ha01 KDUMP:          RIP: ffffffff8128d686  RSP: ffff883146f19e70  RFLAGS: 00010206
Nov 21 01:49:21 vtldxi6902ha01 KDUMP:          RAX: ffff883146f18000  RBX: ffff883146f19f48  RCX: 0000000000000042
Nov 21 01:49:21 vtldxi6902ha01 KDUMP:          RDX: 0000000000000042  RSI: ffff8840309fc000  RDI: 00007f2263416000
Nov 21 01:49:21 vtldxi6902ha01 KDUMP:          RBP: ffff883146f19e98   R8: 0000000000000042   R9: ffff883146f19db8
Nov 21 01:49:21 vtldxi6902ha01 KDUMP:          R10: ffff883146f19db8  R11: 0000000000000007  R12: 0000000000000000
Nov 21 01:49:21 vtldxi6902ha01 KDUMP:          R13: 0000000000000042  R14: ffff883146f19f48  R15: 0000000000000400
Nov 21 01:49:21 vtldxi6902ha01 KDUMP:          ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018
Nov 21 01:49:21 vtldxi6902ha01 KDUMP:          #16 [ffff883146f19e70] simple_read_from_buffer at ffffffff811b15ce
Nov 21 01:49:21 vtldxi6902ha01 KDUMP:          #17 [ffff883146f19ea0] proc_info_read at ffffffff811f5fe5
Nov 21 01:49:21 vtldxi6902ha01 KDUMP:          #18 [ffff883146f19ef0] vfs_read at ffffffff81189775
Nov 21 01:49:21 vtldxi6902ha01 KDUMP:          #19 [ffff883146f19f30] sys_read at ffffffff811898b1
Nov 21 01:49:21 vtldxi6902ha01 KDUMP:          #20 [ffff883146f19f80] system_call_fastpath at ffffffff8100b072
Nov 21 01:49:21 vtldxi6902ha01 KDUMP:          RIP: 000000315aedb730  RSP: 00007fff9f0a7a50  RFLAGS: 00010206
Nov 21 01:49:21 vtldxi6902ha01 KDUMP:          RAX: 0000000000000000  RBX: ffffffff8100b072  RCX: 0000000002023720
Nov 21 01:49:21 vtldxi6902ha01 KDUMP:          RDX: 0000000000000400  RSI: 00007f2263416000  RDI: 0000000000000004
Nov 21 01:49:21 vtldxi6902ha01 KDUMP:          RBP: 00007fff9f0a7b50   R8: 00000000ffffffff   R9: 0000000000000000
Nov 21 01:49:21 vtldxi6902ha01 KDUMP:          R10: 0000000000000022  R11: 0000000000000246  R12: 00007fff9f0a7b50
Nov 21 01:49:21 vtldxi6902ha01 KDUMP:          R13: 0000000000000000  R14: 0000000002023730  R15: 00000000000000ff
Nov 21 01:49:21 vtldxi6902ha01 KDUMP:          ORIG_RAX: 0000000000000000  CS: 0033  SS: 002b
Nov 21 01:49:21 vtldxi6902ha01 KDUMP: INFO :     *********end  stack traces ********
Nov 21 01:49:21 vtldxi6902ha01 srvclogcli: E0000(1)<1232508046>:SRVCLOG RCOMP: 1 RINST: UNKNOWN VCOMP: 1 VINST: KDUMP VPINST: UNKNOWN EVENT: 8 TEXT: Kernel panic/oops/crash has happened  - kdump saved to /snfs/Kdumps/kdump_AV1433BVW00029_2014-11-21_01h48m46s  Ticket creation time: 11/21 01:49:21 CET
 
 

 



This page was generated by the BrainKeeper Enterprise Wiki, © 2018