OOM killing MySql every 14, 15 days at the exact time

barbarian86

• November 20, 2023 02:24

Hello, i have a strange situation with OOM killer. Every 14 or 15 days at exactly 14:10, Cpanel kills mysql with out of memory message. I checked logs, cron jobs, everything. Nothing happens at that particular time frame. I have 128 GB od ram, running MySql 5.7. I did memory usage statistics, it was never near fully used. this is from the last cpanel email for OOM: This is the current state for example when it is under heavy load: Server info: cPanel Version 110.0.15 OS CentOS v7.9.2009 STANDARD standard What can cause this?

• 

  cPRex Jurassic Moderator

  • November 20, 2023 11:14

  Hey there! Can you post the subject line of the email, just so I can confirm the behavior? You can run this command to see the specific entries on the server for the process killed: grep -Ei "(out of memory|oom)" /var/log/messages* -A 1 | grep -i MySQL If MySQL does show up there, you can check the logs around that time in /var/log/messages to see if there is other useful data.

  0
• 

  barbarian86

  • November 20, 2023 11:24

  Hey there! Can you post the subject line of the email, just so I can confirm the behavior? You can run this command to see the specific entries on the server for the process killed: grep -Ei "(out of memory|oom)" /var/log/messages* -A 1 | grep -i MySQL If MySQL does show up there, you can check the logs around that time in /var/log/messages to see if there is other useful data.

  
  Hey, subject is this [nameofthe serrver] FAILED ?: mysql (ipadress) When i run suggester command i do get mysql: /var/log/messages-20231105:Oct 31 13:07:39 srv1 kernel: mysqld invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0 /var/log/messages-20231105-Oct 31 13:07:39 srv1 kernel: mysqld cpuset=/ mems_allowed=0 /var/log/messages-20231105:Oct 31 13:07:40 srv1 kernel: Out of memory: Kill process 67206 (mysqld) score 829 or sacrifice child /var/log/messages-20231105-Oct 31 13:07:40 srv1 kernel: Killed process 67206 (mysqld) total-vm:115121312kB, anon-rss:108236568kB, file-rss:0kB, shmem-rss:0kB /var/log/messages-20231112:Nov 6 13:16:10 srv1 kernel: Out of memory: Kill process 16240 (mysqld) score 824 or sacrifice child /var/log/messages-20231112-Nov 6 13:16:10 srv1 kernel: Killed process 16240 (mysqld) total-vm:114444340kB, anon-rss:107619544kB, file-rss:0kB, shmem-rss:0kB /var/log/messages-20231112:Nov 10 13:05:00 srv1 kernel: Out of memory: Kill process 21006 (mysqld) score 795 or sacrifice child /var/log/messages-20231112-Nov 10 13:05:00 srv1 kernel: Killed process 21006 (mysqld) total-vm:110775540kB, anon-rss:103933828kB, file-rss:0kB, shmem-rss:0kB /var/log/messages-20231119:Nov 16 13:07:08 srv1 kernel: Out of memory: Kill process 39551 (mysqld) score 907 or sacrifice child /var/log/messages-20231119-Nov 16 13:07:08 srv1 kernel: Killed process 39551 (mysqld) total-vm:125966100kB, anon-rss:118924860kB, file-rss:0kB, shmem-rss:0kB You will see it started to happen every sixth day or so, it was 14 before that. When i grep journal or log, all i get is Out of memory: Kill process 39551 (mysqld) score 907 or sacrifice child. Nothing before, nothing after. Strange this is i don't even get an email before this to warn me about high memory usage, as i configured.

  0

Please sign in to leave a comment.