Collect and Transfer a Core Dump File

 

Depending on the size of the core dump file and customers network connections, there are a few ways to gather the Core dump and the associated files.
Any of these methods will require the files (Core file, Core daemon) to be compressed with gzip (or other compression method not covered here) and the md5sum of the files gathered before they are transferred.    

You will also need to gather the DXi system collect, storage array log and if applicable the DSET log from the system. (See "Step 2: Collect Support Data" in the DXi-Series Status and Log Collection Get Adobe reader job aid for instructions).

 


 

 

  1. The core file(s) generated by the DXi are automatically placed in the /scratch/core directory. Log into the DXi via ssh or thru the serial connection and change to the directory where the cores are located (If a core file in this directory has '-deleted' at the end of the file it is no longer on the system). Gather the directory listing of the /scratch/core directory as this may be needed later.

# cd /scratch/core

 

# ls -lah


Example of core file:      -rw-------   1 root root 23G Sep 14 13:49 generatesystemc-1316004544-21790-reported

 

 

Example of a deleted core file:      -rw-------   1 root root 0b Sep 14 13:49 generatesystemc-1316004544-21790-reported-deleted

 

  1. The backtrace may be in the collect logs, but if it isn't, then it might be good to get the backtrace at this point. If the backtrace is not located in the core_backtrace.out file in the collect log or it not being heavily used you should be able to get the backtrace at this time.  Please make note of the ‘Caution’ following this step. Instructions on how to gather the backtrace are located in the  Gather Data on a Core Dump topic Step 6. After gathering the backtrace, come back to this step and continue with this procedure to gather the data that would be needed if the issue needs to be escalated. 

 Caution: Using gdb to gather the backtrace on a live customers DXi can use up the available memory and cause the DXi to have problems and possibly crash. This should be performed on another Quantum internal DXi system.

 

  1. Copy the core file out of the /scratch/core directory. 

# mkdir /scratch/qtmtmp             (This directory may have already been created)

 

# cp <nameofcorefile-reported> /scratch/qtmtmp


 Note: Creating a new file in the /scratch/core directory will result in a new RAS ticket for a core. If a new file is created or copied into this directory, the copied or created file will be renamed to <filename>-reported and a RAS ticket will be generated. The core file must be moved to another directory prior to being zipped up. 

 

  1. Collect the md5sum of the core file prior to zipping up the core file.

               # md5sum <nameofcorefile-reported> > /scratch/qtmtmp/md5sum-<nameofcorefile-reported>.txt

 

  1. gzip the core file to compress the size and make it easier to transfer. 

 # cd /scratch/qtmtmp


# gzip -9 <nameofcorefile-reported>


 Note: The -9 option gives you the best compression rate with gzip. 

 

  1. Collect the md5sum of the core file after being compressed.

# md5sum <nameofcorefile-reported>.gz > md5sum-<nameofcorefile-reported>-gz.txt

 

  1. Locate the daemon that cored. This can give you many files and daemons, but usually the ones needed are in the /opt/DXi/ or /etc/init.d/ directories. The output from this command will give you the path of the daemon that cored. (The cored_daemon will be the process that cored i.e. winbindd, replicationd, ostd, etc. and is found in the name of the core file.)  Look in the backtrace for a line that says Core was generated by.

                    [Thread debugging using libthread_db enabled]
        Core was generated by `/hurricane/ostd'.
        Program terminated with signal 6, Aborted.
        #0 0x00002b8a785e5265 in raise () from /lib64/libc.so.6

 

 Note: The Locate command should not be used and may be installed on the DXi, running updatedb can cause heavy load and disturb operations. locate and updatedb has been removed and will be removed in future releases.

 

 
             Example:
# find / -name ostd
/usr/lib64/perl5/5.8.5/x86_64-linux-thread-multi/CORE/nostdio.h
/usr/share/mysql/charsets/geostd8.xml
/var/lock/subsys/ostd
/opt/DXi/adic/perl/lib/5.8.3/x86_64-linux/CORE/nostdio.h
/opt/DXi/adic/TSM/util/fsechostderr
/opt/DXi/ostd
#
 
  1. Copy the core_daemon file to the /scratch/qtmtmp directory.   
# cp <path/cored_daemon> /scratch/qtmtmp
  1. gzip the cored_daemon file to compress the size and make it easier to transfer. 
 # cd /scratch/qtmtmp

# gzip -9 <cored_daemon> 
  1. Collect the md5sum of the core file.
# md5sum <cored_daemon>.gz > md5sum-<cored_daemon>.txt
 
  1.  If the customer allows ftp from the DXi and the files aren’t too large to be ftp’ed due to customer bandwidth constraints, FTP the zipped core file, zipped cored_daemon and the md5sum files to the Quantum ftp site gps.quantum.com.  If you are unable to ftp the files see the later 'Alternatively' section about accessing the files via scp, web browser or dispatching a technician to gather the core files.

             # ftp gps.quantum.com

 

Logon as user: anonymous

Password: Your email address


# ftp> cd incoming/SR<ServiceRequestNumber>

 

 Note: If you haven't previously made the SR directory you can use the mkdir command to create the directory (i.e. mkdir /incoming/SR<ServiceRequestNumber>.

 

Change to Binary mode.


# ftp> bin


# ftp> hash                      (Optional command prints #'s as it is uploading.)


# ftp> put <nameofcorefile-reported>.gz

 

# ftp> put md5sum-<nameofcorefile-reported>.txt

 

# ftp> put md5sum-<nameofcorefile-reported>-gz.txt

 

# ftp> put <cored_daemon>.gz

 

# ftp> put md5sum-<cored_daemon>.txt

 

(You should also upload the DXi collect files at this time if they have not been uploaded.)

 

# ftp> quit

 

 

  1. After you have verified that the files have downloaded properly to the ftp site and are they are valid by verifying the md5sum is the same as it was before the transfer, you will need to remove the files from the /scratch/qtmtmp directory.

# rm /scratch/qtmtmp/<nameofcorefile-reported>.gz

# rm /scratch/qtmtmp/md5sum-<nameofcorefile-reported>.txt


Alternatively:  If you cannot ftp the core file directly from the DXi due to ftp being blocked, upload the files from the DXi to a customer PC. You can do this with SCP or via the GUI, the customer can then transfer them to our ftp site from a computer that allows FTP access. To do this, complete either of the following steps:

 

SCP:

 

 

1.     Open up a utility on the customers computer that supports SCP ( For example WinSCP).
2.     Enter the IP and the root credentials to access the DXi.
3.     Browse to the /scratch/qtmtmp directory and copy the files to the customer’s computer. 
4.     After you have verified that the files have downloaded properly to the ftp site and are they are valid, you will need to remove the files from the /scratch/qtmtmp directory.

 

WebGUI:

 

  1. Copy the files to /hurricane/www/html. 

cp /scratch/qtmtmp/<nameofcorefile-reported>.gz /hurricane/www/html

cp /scratch/qtmtmp/md5sum-<nameofcorefile-reported>.txt /hurricane/www/html

 

  1. After you have copied the files, you will need to remove them from the /scratch/qtmtmp directory.

    # rm /scratch/qtmtmp/<nameofcorefile-reported>.gz

# rm /scratch/qtmtmp/md5sum-<nameofcorefile-reported>.txt

 

  1. Verify that the file has proper permission to allow downloading.

# chmod 775  /hurricane/www/html/<nameofcorefile-reported>.gz

# chmod 775  /hurricane/www/html/md5sum-<nameofcorefile-reported>.txt

 

  1. Go to the DXi GUI using the following:

http://<IPofDXi>/<nameofcorefile-reported>.gz

http://<IPofDXi>/md5sum-<nameofcorefile-reported>.txt

 

  1. This will prompt you to save the core file to the local computer.
  1. Remove files from the directory when done.


    # rm /hurricane/www/html/nameofcorefile-reported.gz

# rm /hurricane/www/html/md5sum-<nameofcorefile-reported>.txt

 

Tech dispatch:

 

1.     Dispatch a technician with a laptop and Ethernet cable to connect directly to the DXi via an ssh session to download the files to his laptop via the SCP method.
 
-or-
 
2.     Dispatch a technician with part number 9-01851-01 (FRU USB FLASH DRIVE 64GB DXI6500 (CE TOOL)) to place the file on for return to Quantum. The procedures for attaching a USB drive to the DXI can be found in the document from PTR 18835 - How to gather kdump.

 

 


What's Next?

Gather Data on a Core Dump

Notes

 locate should not be used et all.

even it may still be installed on the unit, running updatedb can cause heavy load and disturb operations.

locate/updatedb is being removed in actual and future releases.

 

;-)

Note by Peter Catania on 08/29/2012 09:14 AM


This page was generated by the BrainKeeper Enterprise Wiki, © 2018