Core Files: How Timcores from cron.hourly Handles Core Files (DRAFT)

Overview

This topic provides an overview of trimcores and how to identify core files that were created from an event within collect log or on system via CLI. In addition, after a core file has been detected, this topic describes how you can identify the different phases of trimcores activities by viewing the collect log or via CLI on systems. Some best practices are also covered on what NOT to perform within the /scratch/core directory.

About Trimcores

After a core file has been detected, trimcores from cron.hourly will change the status of the current state. The purpose for trimcores is to notify the customer via a RAS ticket on the GUI and through an email notification (if configured). Trimcores also manage the lifecycle of the core file then deletes the content at 30 days so /scratch/core does no run out of disk space.

Detailed Information

The script for trimcores is located under the cron.hourly directory on the DXi:

# ls -al /etc/cron.hourly/trimcores

-rwxr-xr-x 1 root root 5172 Jun 10 13:24 /etc/cron.hourly/trimcores

A. Identify core files that were created from an event within collect log or on system via CLI:

There are two different ways to identify a core file that took place on a DXi from a collect log. Examples below:

1. $ grep trimcores scratch/collect/node1-collection/app-info/srvcLog.hist

Aug 21 12:01:32 2013 66 trimcores 66 UNKNOWN UNKNOWN 1123331452 51 New core file:-rw------- 1 root root 661803008 Aug 21 11:00 /scratch/core/replicationd-1377104426-10016 Ticket creation time: 08/21 12:01:32 MDT

2. $ grep replicationd-1377104426-10016 scratch/collect/node1-collection/collect.txt

54004 -rw------- 1 root root 661803008 Aug 21 11:00 replicationd-1377104426-10016-reported

Note: If reviewing the collect.txt file within the collect log for core file history, then search for the following:

Location of collect.txt in collect log:

$ ls -al scratch/collect/node1-collection/collect.txt

Within the collect.txt search for the following entry from your preferred viewing method:

Cores- 'ls -lRas /scratch/core'

Thre are two different ways to identify a core file that took place on a DXi product from the CLI on the system. Examples:

1. # grep trimcores /usr/adic/SRVCLOG/logs/srvcLog.hist

# ls -al /scratch/core | grep replicationd-1377104426-10016

-rw-r--r-- 1 root root 627 Aug 21 11:00 replicationd-1377104426-10016-reported

Note: If reviewing core file history from the CLI on the system, then execute the following command:

# ls -al /scratch/core

B. Observer the different phases of trimcores activities by viewing the collect log or via the CLI on system. If you are viewing the collect log, this would be in the collect.txt. If via CLI, this is ls –al /scratch/core:

1. This indicates a core file was created and trimcores from cron.hourly has not recorded the event:

replicationd-1377104426-10016

2. This indicates a core file was identified by trimcores from cron.hourly and was recorded. Now the customer will see from a RAS ticket in the GUI and email notification (if configured):

replicationd-1377104426-10016-reported

3. At 30 days the content of the core file will be deleted to free up space under /scratch/core. The core file will keep the history and the core cannot be processed for additional analysis:

replicationd-1377104426-10016-reported-deleted

Note: All core files need to be copied to a different location for analysis within 30 days of the event if not gathered remotely. Good locations to copy the core files for safe storage without contents being deleted are /scratch (if space is permitted) or /snfs/tmp.

C. Best practices not to perform within the /scratch/core directory with preferred options:

1. Do not run gzip, zip or create files in the /scratch/core directory. The reason is that trimcores will run on an hourly basis, and will generate new RAS tickets and possible email notification, which can cause concern for false events.

2. To prevent the aforementioned issue, the preferred options would be with UNIX copy commands or creating new files via CLI. Two examples are below:

# cp /scratch/core/replicationd-1377104426-10016-reported /scratch

# cp /scratch/core/replicationd-1377104426-10016-reported /snfs/tmp

Note: Now you can run gzip, zip or create files under /scratch or /snfs/tmp without interfering trimcores activities.

Additional Note:

Again, core files contents will be deleted at 30 days under /scratch/core and cannot be analyzed for RCA.