During the daily checking service, when I went in from the portainer to look at the container logs, I found a red exclamation mark in the upper right corner: Unable to retrieve container logs.
Because there has never been such a problem before, I first went to the server to view the logs using the command docker logs -f containerID. I found that the logs could not move, so I still stayed on the log records at a certain time.
I thought about it, it shouldn't be a problem with the service log printing. I searched it first and found that it did not match my problem. Because logs can sometimes be collected and displayed, some logs cannot be used, and there should be problems with the log engine set by docker.
I originally wanted to complete an EFK, but I felt that the log volume is not large enough now, so I did not modify the docker log engine, and it is still the default journald
[root@ad-official xiaoxiao]# docker info|grep Logging WARNING: You're not using the default seccomp profile Logging Driver: journald
There is a description in the official document of journald:
man ... RateLimitInterval=, RateLimitBurst= Configures the rate limiting that is applied to all messages generated on the system. If, in the time interval defined by RateLimitInterval=, more messages than specified in RateLimitBurst= are logged by a service, all further messages within the interval are dropped until the interval is over. A message about the number of dropped messages is generated. This rate limiting is applied per-service, so that two services which log do not interfere with each other's limits. Defaults to 1000 messages in 30s. The time specification for RateLimitInterval= may be specified in the following units: "s", "min", "h", "ms", "us". To turn off any kind of rate limiting, set either value to 0. ...
This is written here. Only 1,000 logs can be received within 30 seconds. You can understand it by seeing this. Because a while ago, a single-day log file size was released in docker, the logs from other services were also affected. A large number of logs were discarded by journald, so we can modify the configuration and there is no problem.
Open the /etc/systemd/ file, modify RateLimitBurst from the default 1000 to 5000, and adjust it according to your current log output:
[root@ad-official log]# cat /etc/systemd/ # This file is part of systemd. # # systemd is free software; you can redistribute it and/or modify it # under the terms of the GNU Lesser General Public License as published by # the Free Software Foundation; either version 2.1 of the License, or # (at your option) any later version. # # Entries in this file show the compile time defaults. # You can change settings by editing this file. # Defaults can be restored by simply deleting this file. # # See (5) for details. [Journal] #Storage=auto #Compress=yes #Seal=yes #SplitMode=uid #SyncIntervalSec=5m #RateLimitInterval=30s RateLimitBurst=5000 #SystemMaxUse= #SystemKeepFree= #SystemMaxFileSize= #RuntimeMaxUse= #RuntimeKeepFree= #RuntimeMaxFileSize= #MaxRetentionSec= #MaxFileSec=1month ForwardToSyslog=no #ForwardToKMsg=no #ForwardToConsole=no ForwardToWall=no #TTYPath=/dev/console #MaxLevelStore=debug #MaxLevelSyslog=debug #MaxLevelKMsg=notice #MaxLevelConsole=info #MaxLevelWall=emerg #LineMax=48K
By the way, set ForwardToSyslog and ForwardToWall to no, because the default is yes, which will cause us to clean the journal log files, and the Syslog is not cleared, and the disk will slowly be filled.
Then restart journald and you can restore normal use:systemctl restart
The above is all the content of this article. I hope it will be helpful to everyone's study and I hope everyone will support me more.