SoFunction
Updated on 2025-03-02

Linux log polling scheme

Introduction to logrotate

Generally, logs are a very important part of any troubleshooting process, but these logs grow over time. In this case, we need to manually perform log cleaning to reclaim space, which is a tedious management task. To solve this problem, we can configure the logrotate program in Linux, which can automatically perform rotation, compression, deletion and mail out log files.

We can configure the logrotate program so that each log file can be processed daily, weekly, monthly, or when it becomes too large.

logrotate software is a log management tool used to cut logs, delete old log files, and create new log files, which plays a "dumping role" and can save disk space for the system.. Generally, the centos system has been installed.

logrotate is run based on crontab. When running, logrotate will call the configuration file /etc/. You can place a custom configuration file in the /etc/ directory to override the default value.

logrotate version view

root@test 10:14:51:/usr# logrotate --version
logrotate 3.11.0

logrotate configuration file

  • Order:/usr/sbin/logrotate
  • Configuration file:/etc/

This is the main configuration file for logrotate. logrotate is still there/etc//The configuration of a specific service is stored in it.

Make sure the line below is included /etc/ middle，To read the specific service log configuration。
include  /etc/`

logrotate history:

/var/lib/logrotate/

Log rollback principle

When the log reaches a certain size, we classify the logs, keep a backup of the previous logs, and then create a file of the same name to save the new log.

Important logrotate options:

compress                                   passgzip Compress the log after dump
nocompress                                Don't do itgzipCompression processing
copytruncate                              Used for log files that are still being opened，Backup and truncate the current log；It's a way to copy first and then clear，There is a time difference between copying and clearing，Some log data may be lost。
nocopytruncate                           Backup log files are not truncated
create mode owner group             Specify the properties for creating a new file during rotation，likecreate 0777 nobody nobody
nocreate                                    No new log files are created
delaycompress                           andcompress When using it together，The dumped log file is compressed until the next dump
nodelaycompress                        cover delaycompress Options，Dumping and compressing。
missingok                                 like果日志丢失，Keep scrolling the next log without an error
errors address                           The error message during special storage is sent to the specifiedEmail address
ifempty                                    Even if the log file is empty, it will be rotated，This islogrotate的defaultOptions。
notifempty                               When the log file is empty，No rotation
mail address                             Send the dumped log file to the specifiedE-mail address
nomail                                     No log files are sent when dumping
olddir directory                         The dumped log file is placed in the specified directory，必须and当前日志文件exist同一个文件系统
noolddir                                   转储后的日志文件and当前日志文件放exist同一个目录下
sharedscripts                           runpostrotatescript，作用是exist所有日志都轮转后统一执行一次script。like果没有配置这个，那么每个日志轮转后都会执行一次script
prerotate                                 existlogrotateInstructions that need to be executed before dumping，例like修改文件的属性等动作；Must be in line independently
postrotate                               existlogrotateThe instructions that need to be executed after dumping，例like重新启动 (kill -HUP) A service！Must be in line independently
daily                                       Specify the dump cycle to be daily
weekly                                    Specify the dump cycle to be weekly
monthly                                  Specify the dump cycle to be monthly
rotate count                            Specifies the number of dumps before log file deletion，0 There is no backup，5 Retain5 A backup
dateext                                  Use the current date as the naming format
dateformat .%s                       Cooperatedateextuse，紧跟exist下一行出现，Define file name after file cutting，必须Cooperatedateextuse，Only supported %Y %m %d %s These four parameters
size(orminsize) log-size            Dumping when the log file reaches the specified size，log-sizeCan specifybytes(default)andKB (sizek)orMB(sizem).
The log file >= log-size Dump it when it is。 The following is the legal format：（No other formats have tried the upper and lower case）
size = 5 or size 5 （>= 5 Dump in just one byte）
size = 100k or size 100k
size = 100M or size 100M

sshd log rollback actual combat

Modify the configuration file

[root@ localhost ]# vim /etc/
/var/log/ {         # Specify the log file to be cut    missingok               # If the file is lost, no error will be reported    monthly                 # Rotate once a month    create 0664 root utmp   # Set the owner and group of this file    minsize 10M             # Files exceed 10M and rollback    rotate 2                # After the log is split, two copies of historical data are retained}

Restart rsyslog

[root@ localhost ]# systemctl restart rsyslog

Forced cutting

[root@ localhost ]# logrotate –vf /etc/
[root@ localhost ]# cd /var/log/
[root@ localhost log]# ll  
-rw------- 1 root root 0 2019-11-29 14:00 
# After cutting，The source file size becomes0

Log cutting can be cut according to the file size and time.

Let’s briefly explain it using the system log/var/log/message as cutting:

  • After the first execution of rotation (rotation), the original messages will become messages.1, and an empty messages will be created for the system to store logs;
  • After the second execution, messages.1 will become messages.2, and messages will become messages.1, causing an empty message to store the logs!
  • If only three logs are reserved (that is, three rotations), then when the third time is executed, the messages.3 file will be deleted and replaced by the newer saved logs in the future! That is, the latest logs will be saved.
  • How many times the logs are rotated is determined based on the rotation parameter in the configuration file.

Test and run logrotate

If you can't wait for cron to automatically perform log rotation, if you want to manually force the log to be cut, you need to add the -f parameter; however, it is best to verify it through the Debug option before formal execution, which is also very important for debugging.

/usr/sbin/logrotate -f /etc//nginx
/usr/sbin/logrotate -d -f /etc//nginx

The specific logrotate command format is as follows:

logrotate [OPTION...] <configfile>
-d, --debug ：debug model，Test the configuration file for errors。
-f, --force ：Forced dump file。
-m, --mail=command ：After compressing the log，Send logs to the specified mailbox。
-s, --state=statefile ：Use the specified status file。
-v, --verbose ：Show dump process。

debug mode

It will not actually perform rotate or compress operations, but will print out the entire execution process and the called scripts and other detailed information.

logrotate -d <configfile>

verbose mode

The operation will be performed and the details will be printed out.

logrotate -v <configfile>

force enforcement

If the time does not meet the requirements, logrotate will not be executed. If you want to execute immediately and view the results, you will use the enforced execution mode.

logrotate -f <configfile>

Add crontab schedule task

implementcrontab -eEnter vim mode and edit.

[root@root local]# crontab -e

#Digit cutting at 23:59 every day59 23 * * * /usr/sbin/logrotate -s /tmp/ /etc//tomcat

explain:

59 23 * * * cron expression
/usr/sbin/logrotate Start logrotate
-s /tmp/ Save execution status
/etc//tomcat timing task, provided that there is a tomcat file

Tomcat file content:

/soft/tomcat/logs/{
    minsize 50M
    rotate 20
    compress
    copytruncate
    notifempty
}

crontab commonly used commands

View all timing tasks crontab -l

[root@root local]# crontab -l
.....(Omitted)
59 23 * * * /usr/sbin/logrotate -f /etc/logrotate_mytime/nginx_rotate
.....(Omitted)

Edit timing tasks crontab -e

[root@root local]# crontab -e # Enter vim mode.....(Omitted)
59 23 * * * /usr/sbin/logrotate -f /etc/logrotate_mytime/nginx_rotate
.....(Omitted)

Logrotate configuration for common applications

/var/log/nginx/*.log {   # You can specify multiple paths    daily                      # Log polling cycle, weekly, monthly, yearly    rotate 30                  # Save data for 30 days, delete if it exceeds it    size 100M                 # When it exceeds 100M, the units K, M, G, the priority is higher than daily    
    compress                   # Compress after cutting, can also be nocompress    delaycompress              # Compress the last log file during cutting    
    dateext                    # Add date suffix when cutting log files    dateformat -%Y-%m-%d
    
    missingok                  # If there is no log file, you will not report an error    notifempty                 # When the log is empty, the default is ifempty    create 640 nginx nginx     # Create log files using this mode    
    sharedscripts              # After cutting all files, execute the following script only once.    postrotate
        if [ -f /run/ ]; then
            kill -USR1 `cat /run/`
        fi
    endscript
}

logrotate default run time

logrotate is run based on cron, so this time is controlled by cron. You can query the configuration file of cron./etc/crontab 。

The configuration file for the old CentOS cron is/etc/crontab, the new version of CentOS is changed to/etc/anacrontab 。

Summarize

The above is personal experience. I hope you can give you a reference and I hope you can support me more.