Working with logrotate in vmPRO

Many of the processes in vmPRO are managed using logrotate. Here is a good reference for how logrotate works in general: https://library.linode.com/linux-tools/utilities/logrotate

Non-vmPRO Process Logging

Non-vmPRO process logging, such as messages or boot.log,is still managed by rsyslogger, as seen in /etc/rsyslog.conf:

#### GLOBAL DIRECTIVES #### # Use default timestamp format $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat # File syncing capability is disabled by default. This feature is usually not required, # not useful and an extreme performance hit #$ActionFileEnableSync on # Include all config files in /etc/rsyslog.d/ $IncludeConfig /etc/rsyslog.d/*.conf

#### RULES ####

# Log all kernel messages to the console. # Logging much else clutters up the screen. #kern.* /dev/console # Log anything (except mail) of level info or higher. # Don't log private authentication messages! *.info;mail.none;authpriv.none;cron.none /var/log/messages# The authpriv file has restricted access. authpriv.* /var/log/secure # Log all the mail messages in one place. mail.* -/var/log/maillog # Log cron stuff cron.* /var/log/cron # Everybody gets emergency messages *.emerg * # Save news errors of level crit and higher in a special file. uucp,news.crit /var/log/spooler # Save boot messages also to boot.log local7.* /var/log/boot.log

# ### begin forwarding rule ###

# The statement between the begin ... end define a SINGLE forwarding # rule. They belong together, do NOT split them. If you create multiple # forwarding rules, duplicate the whole block! # Remote Logging (we use TCP for reliable delivery) # # An on-disk queue is created for this action. If the remote host is # down, messages are spooled to disk and sent when it is up again. #$WorkDirectory /var/lib/rsyslog # where to place spool files #$ActionQueueFileName fwdRule1 # unique name prefix for spool files #$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as possible) #$ActionQueueSaveOnShutdown on # save messages to disk on shutdown #$ActionQueueType LinkedList # run asynchronously #$ActionResumeRetryCount -1 # infinite retries if host is down # remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional #*.* @@remote-host:514 # ### end of the forwarding rule ###

Providing Logs to Support

Sometimes you will need to send logs to Support, which are hard to capture before they roll over. To increase the chances that a support bundle capture will capture everything needed in the logs, you can modify the size of the logs and how many are retained:

Drop the pan-shell once an ssh session into the appliance is established. Use the ‘shell-escape’ command.
Navigate to the /etc/ directory and view the main config file:

bash-4.1# cat /etc/logrotate.conf

# see "man logrotate" for details # rotate log files weekly weekly # keep 4 weeks worth of backlogs rotate 4 # create new (empty) log files after rotating old ones create # use date as a suffix of the rotated file nodateext # uncomment this if you want your log files compressed #compress # RPM packages drop log rotation information into this directory| include /etc/logrotate.d # no packages own wtmp and btmp -- we'll rotate them here /var/log/wtmp { monthly create 0664 root utmp minsize 1M rotate 1 }

/var/log/btmp { missingok monthly create 0600 root utmp rotate 1 }

# system-specific logs may be also be configured here.

In the /etc/logrotate.d directory, you will see the individual config for each process:

A. bash-4.1# ls

controller datastore_fs dracut files_fs import_fs iscsiuiolog ls_bitmap_fs postgresql recovery_fs recovery_fs_files samba syslog vm_proxy_fs yum

There are all symbolic links.

B. bash-4.1# ls -alh

total 28K drwxr-xr-x. 2 root root 4.0K Jul 5 14:10 . drwxr-xr-x. 70 root root 4.0K Jul 5 11:06 .. lrwxrwxrwx 1 root root 59 Jun 20 16:57 controller -> /usr/local/pancetera/python/controller/controller.logrotate lrwxrwxrwx 1 root root 47 Jun 20 16:57 datastore_fs -> /usr/local/pancetera/bin/datastore_fs.logrotate -rw-r--r-- 1 root root 103 Oct 2 2012 dracut lrwxrwxrwx 1 root root 43 Jun 20 16:57 files_fs -> /usr/local/pancetera/bin/files_fs.logrotate lrwxrwxrwx 1 root root 44 Jun 20 16:57 import_fs -> /usr/local/pancetera/bin/import_fs.logrotate -rw-r--r-- 1 root root 173 Jun 22 2012 iscsiuiolog lrwxrwxrwx 1 root root 47 Jun 20 16:57 ls_bitmap_fs -> /usr/local/pancetera/bin/ls_bitmap_fs.logrotate lrwxrwxrwx 1 root root 59 Jun 20 16:57 postgresql -> /usr/local/pancetera/python/controller/postgresql.logrotate lrwxrwxrwx 1 root root 46 Jun 20 16:57 recovery_fs -> /usr/local/pancetera/bin/recovery_fs.logrotate lrwxrwxrwx 1 root root 52 Jun 20 16:57 recovery_fs_files -> /usr/local/pancetera/bin/recovery_fs_files.logrotate -rw-r--r-- 1 root root 477 Feb 13 17:26 samba -rw-r--r-- 1 root root 238 Jun 20 16:57 syslog lrwxrwxrwx 1 root root 46 Jun 20 16:57 vm_proxy_fs -> /usr/local/pancetera/bin/vm_proxy_fs.logrotate -rw-r--r-- 1 root root 100 Jun 22 2012 yum

bash-4.1#

You can modify the process logging rotation. Here, we see that the top process (controller), while the bottom process (vm_proxy_fs) retains its default values.

bash-4.1# cat controller
/var/log/controller {
    rotate 10
    copytruncate
    missingok
    size 6000k
}
bash-4.1# cat vm_proxy_fs
/var/log/vm_proxy_fs {
    rotate 5
    copytruncate
    missingok
    size 3000k
    postrotate
        /usr/bin/killall -HUP vm_proxy_fs 2> /dev/null || true
    endscript
}
bash-4.1#

Logrotate doesn’t run as a daemon. Instead, it is managed by an hourly cron job, as seen here. This means that you don’t have to cycle a service when making config file changes — this will sort itself out hourly.

bash-4.1# cat /etc/cron.hourly/logrotate
#!/bin/sh
/usr/sbin/logrotate /etc/logrotate.conf >/dev/null 2>&1
EXITVALUE=$?
if [ $EXITVALUE != 0 ]; then
/usr/bin/logger -t logrotate "ALERT exited abnormally with [$EXITVALUE]"
fi
exit 0
bash-4.1#

Be careful not to fill up /var -- doing so will cause problems in any Linux distribution. Always return log rotate to its default values when you are done troubleshooting.

Truncating Logs

When /var is full vmPRO will generate a message alluding to the mout point being low on space. To determin which log is full you can look at the disk usage, pipe to sort then pipe to more to find which logs are taking up the most space. This is the command i choose to use, there are many like it, but this one is mine. Example below.

bash-4.1# du -akD /var/log | sort -nr | more

1024044 /var/log

919092 /var/log/controller

10256 /var/log/smartmotion.1

8756 /var/log/smartmotion

4212 /var/log/messages.2

3876 /var/log/controller.6

Once determined which file is hogging the directory it is best practice not to rm the logs, you will anger logrotate for a number of reasons. You should instead use the truncat command. Example below. Syntax 'truncate -option file'. Here we using the -s option to specify the file to 0 bytes.

bash-4.1# truncate -s 0 controller

bash-4.1# ls -alh |grep controller

-rw-r--r-- 1 root root 0 Jul 5 16:12 controller

-rw-r--r-- 1 root root 36 Jul 1 23:00 controller-uuid-2a2bea9ee24c11e187f90050568a6300.txt

-rw-r--r-- 1 root root 3.4M Jul 5 16:09 controller.1

-rw-r--r-- 1 root root 3.8M Jul 5 16:01 controller.2

-rw-r--r-- 1 root root 0 Jul 5 15:01 controller.3

-rw-r--r-- 1 root root 3.2M Jul 5 14:01 controller.4

-rw-r--r-- 1 root root 3.1M Jul 5 13:01 controller.5

-rw-r--r-- 1 root root 2.9M Jul 5 12:01 controller.6