kdump analysis with Galaxy 3.0 (draft) |
The crash command output depends also of the timezone setting where you analyze the kdump.
The collect log for the affected system shows in collect.txt:
###15:01:44### -Date SysconfigClock- 'cat /etc/sysconfig/clock':
ZONE="Europe/Paris"
If you do
# export TZ=Europe/Paris
crash shows:
KERNEL: vmlinux-2.6.32-431.11.2.el6.Q_3.x86_64
DUMPFILE: vmcore [PARTIAL DUMP]
CPUS: 40
DATE: Fri Nov 21 01:22:36 2014
UPTIME: 08:25:19
LOAD AVERAGE: 1.07, 0.92, 0.79
TASKS: 3004
NODENAME: vtldxi6902ha01
This is the right timestamp in the local time context.
This appears to be the timestamp for the KDUMP when the mini-kernel was woke up to handle the kernel Oops. It was logged in the messages file at 01:31 on Nov but the actual time that the kernel Oops was around 16:22 on Nov 20:
Nov 21 01:31:43 vtldxi6902ha01 KDUMP: INFO : kcdump: KDUMP = 127.0.0.1-2014-11-20-16:22:45
SYSTEM MAP: System.map-2.6.32-431.11.2.el6.Q_3.x86_64
DEBUG KERNEL: vmlinux-2.6.32-431.11.2.el6.Q_3.x86_64 (2.6.32-431.11.2.el6.Q_3.x86_64)
DUMPFILE: vmcore [PARTIAL DUMP]
CPUS: 40
DATE: Thu Nov 20 17:22:36 2014
UPTIME: 08:25:19
LOAD AVERAGE: 1.07, 0.92, 0.79
TASKS: 3004
NODENAME: vtldxi6902ha01
RELEASE: 2.6.32-431.11.2.el6.Q_3.x86_64
VERSION: #1 SMP Thu Apr 3 15:42:03 PDT 2014
MACHINE: x86_64 (2499 Mhz)
MEMORY: 256 GB
PANIC: "Oops: 0002 [#1] SMP " (check log for details)
PID: 14846
COMMAND: "pidof"
TASK: ffff883367f69540 [THREAD_INFO: ffff883146f18000]
CPU: 23
STATE: TASK_RUNNING (PANIC)
From the crash sys command the timestamp shows 17:22 on Nov 20.
Why the one hour difference?
Peter answered this:
The starting kernel has no clue about timezone offsets at that time as it would need to read /etc/localtime to find out.
And from Daniel:
The collect log for the affected system shows in collect.txt:
###15:01:44### -Date SysconfigClock- 'cat /etc/sysconfig/clock':
ZONE="Europe/Paris"
If you do
# export TZ=Europe/Paris
The crash sys command should now show the correct timestamp.
The stacktrace of the KDUMP in the messages file timestamp is when it was logged:
Nov 21 01:49:05 vtldxi6902ha01 KDUMP: INFO : *********begin stack traces ********
Nov 21 01:49:20 vtldxi6902ha01 KDUMP: PID: 14846 TASK: ffff883367f69540 CPU: 23 COMMAND: "pidof"
Nov 21 01:49:20 vtldxi6902ha01 KDUMP: #0 [ffff883146f19680] machine_kexec at ffffffff81038f3b
Nov 21 01:49:20 vtldxi6902ha01 KDUMP: #1 [ffff883146f196e0] crash_kexec at ffffffff810c5d82
Nov 21 01:49:20 vtldxi6902ha01 KDUMP: #2 [ffff883146f197b0] oops_end at ffffffff8152bae0
Nov 21 01:49:20 vtldxi6902ha01 KDUMP: #3 [ffff883146f197e0] no_context at ffffffff8104a00b
Nov 21 01:49:20 vtldxi6902ha01 KDUMP: #4 [ffff883146f19830] __bad_area_nosemaphore at ffffffff8104a295
Nov 21 01:49:20 vtldxi6902ha01 KDUMP: #5 [ffff883146f19880] bad_area_nosemaphore at ffffffff8104a363
Nov 21 01:49:20 vtldxi6902ha01 KDUMP: #6 [ffff883146f19890] __do_page_fault at ffffffff8104aabf
Nov 21 01:49:20 vtldxi6902ha01 KDUMP: #7 [ffff883146f199b0] do_page_fault at ffffffff8152da2e
Nov 21 01:49:20 vtldxi6902ha01 KDUMP: #8 [ffff883146f199e0] page_fault at ffffffff8152ade5
Nov 21 01:49:20 vtldxi6902ha01 KDUMP: [exception RIP: __mem_cgroup_commit_charge+0xa7]
Nov 21 01:49:20 vtldxi6902ha01 KDUMP: RIP: ffffffff8117c897 RSP: ffff883146f19a98 RFLAGS: 00010206
Nov 21 01:49:20 vtldxi6902ha01 KDUMP: RAX: 0000000000000017 RBX: ffffea00e08d1870 RCX: 0000000000000001
Nov 21 01:49:21 vtldxi6902ha01 KDUMP: RDX: 00000000000005c0 RSI: ffff884060705020 RDI: ffffc900180c6000
Nov 21 01:49:21 vtldxi6902ha01 KDUMP: RBP: ffff883146f19a98 R8: ffffc900180c6000 R9: 0000000000001000
Nov 21 01:49:21 vtldxi6902ha01 KDUMP: R10: 0000000000013560 R11: 0000000000000000 R12: ffff884060705020
Nov 21 01:49:21 vtldxi6902ha01 KDUMP: R13: 0000000000001000 R14: 00000000000000d0 R15: ffff884050fe8800
Nov 21 01:49:21 vtldxi6902ha01 KDUMP: ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
Nov 21 01:49:21 vtldxi6902ha01 KDUMP: #9 [ffff883146f19aa0] mem_cgroup_charge_common at ffffffff8117c9a5
Nov 21 01:49:21 vtldxi6902ha01 KDUMP: #10 [ffff883146f19b10] mem_cgroup_newpage_charge at ffffffff8117ca18
Nov 21 01:49:21 vtldxi6902ha01 KDUMP: #11 [ffff883146f19b20] handle_pte_fault at ffffffff8114acde
Nov 21 01:49:21 vtldxi6902ha01 KDUMP: #12 [ffff883146f19c00] handle_mm_fault at ffffffff8114b29a
Nov 21 01:49:21 vtldxi6902ha01 KDUMP: #13 [ffff883146f19c70] __do_page_fault at ffffffff8104a8d8
Nov 21 01:49:21 vtldxi6902ha01 KDUMP: #14 [ffff883146f19d90] do_page_fault at ffffffff8152da2e
Nov 21 01:49:21 vtldxi6902ha01 KDUMP: #15 [ffff883146f19dc0] page_fault at ffffffff8152ade5
Nov 21 01:49:21 vtldxi6902ha01 KDUMP: [exception RIP: copy_user_enhanced_fast_string+0x6]
Nov 21 01:49:21 vtldxi6902ha01 KDUMP: RIP: ffffffff8128d686 RSP: ffff883146f19e70 RFLAGS: 00010206
Nov 21 01:49:21 vtldxi6902ha01 KDUMP: RAX: ffff883146f18000 RBX: ffff883146f19f48 RCX: 0000000000000042
Nov 21 01:49:21 vtldxi6902ha01 KDUMP: RDX: 0000000000000042 RSI: ffff8840309fc000 RDI: 00007f2263416000
Nov 21 01:49:21 vtldxi6902ha01 KDUMP: RBP: ffff883146f19e98 R8: 0000000000000042 R9: ffff883146f19db8
Nov 21 01:49:21 vtldxi6902ha01 KDUMP: R10: ffff883146f19db8 R11: 0000000000000007 R12: 0000000000000000
Nov 21 01:49:21 vtldxi6902ha01 KDUMP: R13: 0000000000000042 R14: ffff883146f19f48 R15: 0000000000000400
Nov 21 01:49:21 vtldxi6902ha01 KDUMP: ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
Nov 21 01:49:21 vtldxi6902ha01 KDUMP: #16 [ffff883146f19e70] simple_read_from_buffer at ffffffff811b15ce
Nov 21 01:49:21 vtldxi6902ha01 KDUMP: #17 [ffff883146f19ea0] proc_info_read at ffffffff811f5fe5
Nov 21 01:49:21 vtldxi6902ha01 KDUMP: #18 [ffff883146f19ef0] vfs_read at ffffffff81189775
Nov 21 01:49:21 vtldxi6902ha01 KDUMP: #19 [ffff883146f19f30] sys_read at ffffffff811898b1
Nov 21 01:49:21 vtldxi6902ha01 KDUMP: #20 [ffff883146f19f80] system_call_fastpath at ffffffff8100b072
Nov 21 01:49:21 vtldxi6902ha01 KDUMP: RIP: 000000315aedb730 RSP: 00007fff9f0a7a50 RFLAGS: 00010206
Nov 21 01:49:21 vtldxi6902ha01 KDUMP: RAX: 0000000000000000 RBX: ffffffff8100b072 RCX: 0000000002023720
Nov 21 01:49:21 vtldxi6902ha01 KDUMP: RDX: 0000000000000400 RSI: 00007f2263416000 RDI: 0000000000000004
Nov 21 01:49:21 vtldxi6902ha01 KDUMP: RBP: 00007fff9f0a7b50 R8: 00000000ffffffff R9: 0000000000000000
Nov 21 01:49:21 vtldxi6902ha01 KDUMP: R10: 0000000000000022 R11: 0000000000000246 R12: 00007fff9f0a7b50
Nov 21 01:49:21 vtldxi6902ha01 KDUMP: R13: 0000000000000000 R14: 0000000002023730 R15: 00000000000000ff
Nov 21 01:49:21 vtldxi6902ha01 KDUMP: ORIG_RAX: 0000000000000000 CS: 0033 SS: 002b
Nov 21 01:49:21 vtldxi6902ha01 KDUMP: INFO : *********end stack traces ********
Nov 21 01:49:21 vtldxi6902ha01 srvclogcli: E0000(1)<1232508046>:SRVCLOG RCOMP: 1 RINST: UNKNOWN VCOMP: 1 VINST: KDUMP VPINST: UNKNOWN EVENT: 8 TEXT: Kernel panic/oops/crash has happened - kdump saved to /snfs/Kdumps/kdump_AV1433BVW00029_2014-11-21_01h48m46s Ticket creation time: 11/21 01:49:21 CET
This page was generated by the BrainKeeper Enterprise Wiki, © 2018 |