SoFunction
Updated on 2025-03-04

The reason why Linux viewing program is Killed and its analysis

Linux viewing program is Killed reason

1. View information and output the recently killed process

View the short message list by Killed egrep -i -r 'killed process' /var/log

[root@ecs-bcb3 ~]# egrep -i -r 'killed process' /var/log 
/var/log/messages:Jul 12 15:00:24 ecs-bcb3 kernel: Killed process 26399 (python3), UID 1002, total-vm:27455044kB, anon-rss:24933916kB, file-rss:712kB, shmem-rss:0kB
/var/log/messages:Jul 12 15:00:24 ecs-bcb3 kernel: Killed process 29412 (python3), UID 1002, total-vm:27455044kB, anon-rss:24934124kB, file-rss:780kB, shmem-rss:0kB
/var/log/messages:Jul 12 15:18:39 ecs-bcb3 kernel: Killed process 25017 (chromedriver), UID 1002, total-vm:218888kB, anon-rss:2312kB, file-rss:184kB, shmem-rss:0kB
/var/log/messages:Jul 12 15:18:39 ecs-bcb3 kernel: Killed process 30252 (chromedriver), UID 1002, total-vm:201472kB, anon-rss:2032kB, file-rss:104kB, shmem-rss:0kB

View the list of Killed details dmesg | egrep -i -B100 'killed process'

[root@ecs-bcb3 ~]# dmesg | egrep -i -B100 'killed process'
[19095552.852409] [31573]  1002 31573   131893     2522     166        0             0 chrome
[19095552.852585] [31597]  1002 31597   144656     4247     168        0           200 chrome
[19095552.852756] [31599]  1002 31599   148317     2766     183        0             0 chrome

[19095552.869403] [26287]  1004 26287  1975489   186919     547        0             0 java
[19095552.869548] [27054]   999 27054     1154       23       8        0             0 sh
[19095552.869700] [27058]   999 27058     1110       16       8        0             0 df
[19095552.869842] Out of memory: Kill process 7429 (python3) score 357 or sacrifice child
[19095552.869991] Killed process 7429 (python3), UID 1002, total-vm:26420076kB, anon-rss:23508276kB, file-rss:920kB, shmem-rss:0kB
...

View the short message list of Killed journalctl -xb | egrep -i 'killed process'

12moon 21 15:33:18 ecs-kms kernel: Killed process 2580 (chrome), UID 0, total-vm:862648kB, anon-rss:53044kB, file-rss:156kB, shmem-rss:2692kB
12moon 21 15:33:18 ecs-kms kernel: Killed process 3000 (chrome), UID 0, total-vm:862136kB, anon-rss:49508kB, file-rss:16kB, shmem-rss:2508kB
12moon 21 15:33:18 ecs-kms kernel: Killed process 3165 (chrome), UID 0, total-vm:859320kB, anon-rss:48372kB, file-rss:112kB, shmem-rss:2492kB
...

2. Set kill priority

  • 2.1 Turn off oom completely
sysctl vm.overcommit_memory=2
echo “vm.overcommit_memory=2” >> /etc/
  • 2.2 Restart after Out Of Memory appears
sysctl vm.panic_on_oom=1
sysctl =X
echo “vm.panic_on_oom=1” >> /etc/
echo “=X” >> /etc/
  • 2.3 Prioritize the kill process
sudo echo 10>/proc/[PID]/oom_adj
  • 2.4 Try not to kill the process
sudo echo -15>/proc/[PID]/oom_adj
  • 2.5 Can't kill this process
sudo echo -17>/proc/[PID]/oom_adj

Remark:

  • The value of oom_adj is between -16 and +15. The higher the value, the higher the priority of being killed.
  • When this value is -17, the system will not kill the process with the specified pid, while -16~15 will cause the /proc/[pid]/oom_adj value of the process to increase exponentially, that is, the possibility of them being killed is exponentially increased.
  • For the process init (process number 1), no matter how much the value is set, it will not be killed.

3. If the above command is denied access

Modify the command into the following method

bash -c "echo '10' | tee /proc/[PID]/oom_adj"
bash -c "echo '-15' | tee /proc/[PID]/oom_adj"
bash -c "echo '-17' | tee /proc/[PID]/oom_adj"

Summarize

The above is personal experience. I hope you can give you a reference and I hope you can support me more.