weird out of memory cgroups issue

sahostking

February 05, 2016 13:19

Mysql seems to randomly go down on 2 servers we have with 10Gb. We can increase memory or any resources if required but don't think that will help. We already a few GB a week ago. Same thing I see this in logs the whole day? free memory on server is 7GB which is weird. But for no apprent reason mysql just dies, then shortly thereafter apache. Wonder if its related to this?


Feb  5 20:00:02 lin02 pam_lve[5494]: (CRON) CAN'T FORK (child_process): Cannot allocate memory
Feb  5 20:00:02 lin02 pam_lve[5448]: (CRON) CAN'T FORK (child_process): Cannot allocate memory
Feb  5 20:00:02 lin02 pam_lve[5501]: (CRON) CAN'T FORK (child_process): Cannot allocate memory
Feb  5 20:00:02 lin02 pam_lve[5438]: (CRON) CAN'T FORK (child_process): Cannot allocate memory
Feb  5 20:00:02 lin02 pam_lve[5495]: (CRON) CAN'T FORK (child_process): Cannot allocate memory
Feb  5 20:00:02 lin02 pam_lve[5457]: (CRON) CAN'T FORK (child_process): Cannot allocate memory
Feb  5 20:00:02 lin02 pam_lve[5447]: (CRON) CAN'T FORK (child_process): Cannot allocate memory
Feb  5 20:00:02 lin02 pam_lve[5432]: (CRON) CAN'T FORK (child_process): Cannot allocate memory
Feb  5 20:00:02 lin02 pam_lve[5475]: (CRON) CAN'T FORK (child_process): Cannot allocate memory
Feb  5 20:00:02 lin02 pam_lve[5431]: (CRON) CAN'T FORK (child_process): Cannot allocate memory
Feb  5 20:00:02 lin02 pam_lve[5471]: (CRON) CAN'T FORK (child_process): Cannot allocate memory
Feb  5 20:00:02 lin02 pam_lve[5395]: (CRON) CAN'T FORK (child_process): Cannot allocate memory
Feb  5 20:12:19 lin02 kernel: memory: usage 1048576kB, limit 1048576kB, failcnt 4118
Feb  5 20:12:19 lin02 kernel: memory+swap: usage 1048576kB, limit 1048576kB, failcnt 0
Feb  5 20:12:19 lin02 kernel: Memory cgroup out of memory: Kill process 84642 (php) score 78 or sacrifice child
Feb  5 20:12:21 lin02 kernel: memory: usage 1048576kB, limit 1048576kB, failcnt 4882
Feb  5 20:12:21 lin02 kernel: memory+swap: usage 1048576kB, limit 1048576kB, failcnt 0
Feb  5 20:12:21 lin02 kernel: Memory cgroup out of memory: Kill process 84415 (php) score 80 or sacrifice child
Feb  5 20:12:22 lin02 kernel: memory: usage 1048576kB, limit 1048576kB, failcnt 5503
Feb  5 20:12:22 lin02 kernel: memory+swap: usage 1048576kB, limit 1048576kB, failcnt 0
Feb  5 20:12:22 lin02 kernel: Memory cgroup out of memory: Kill process 84412 (php) score 71 or sacrifice child
Feb  5 20:12:23 lin02 kernel: memory: usage 1048576kB, limit 1048576kB, failcnt 6148
Feb  5 20:12:23 lin02 kernel: memory+swap: usage 1048576kB, limit 1048576kB, failcnt 0
Feb  5 20:12:23 lin02 kernel: Memory cgroup out of memory: Kill process 84417 (php) score 80 or sacrifice child
Feb  5 20:12:24 lin02 kernel: memory: usage 1048576kB, limit 1048576kB, failcnt 6535
Feb  5 20:12:24 lin02 kernel: memory+swap: usage 1048576kB, limit 1048576kB, failcnt 0
Feb  5 20:12:24 lin02 kernel: Memory cgroup out of memory: Kill process 84468 (php) score 70 or sacrifice child

Comments

2 comments

sahostking

February 06, 2016 12:14
Hi I tried to disable cgroups as per error logs, in adding cgroup_disable=memory into grub.conf line it works great and server stays on without omkiller or reboot but Cloudlinux LVE does not want to start because if it. I'm thinking its a requirement somehow. If I take cgroups_disable out then it works again fine ad lve starts up. Feb 6 14:26:11 lin02 kernel: init_memory_mapping: [mem 0x00000000-0x000fffff] Feb 6 14:26:11 lin02 kernel: init_memory_mapping: [mem 0x43fe00000-0x43fffffff] Feb 6 14:26:11 lin02 kernel: init_memory_mapping: [mem 0x43c000000-0x43fdfffff] Feb 6 14:26:11 lin02 kernel: init_memory_mapping: [mem 0x400000000-0x43bffffff] Feb 6 14:26:11 lin02 kernel: init_memory_mapping: [mem 0x00100000-0xbffdffff] Feb 6 14:26:11 lin02 kernel: init_memory_mapping: [mem 0x100000000-0x3ffffffff] Feb 6 14:26:11 lin02 kernel: Reserving 161MB of memory at 688MB for crashkernel (System RAM: 16383MB) Feb 6 14:26:11 lin02 kernel: Early memory node ranges Feb 6 14:26:11 lin02 kernel: PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] Feb 6 14:26:11 lin02 kernel: PM: Registered nosave memory: [mem 0x000a0000-0x000effff] Feb 6 14:26:11 lin02 kernel: PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] Feb 6 14:26:11 lin02 kernel: PM: Registered nosave memory: [mem 0xbffe0000-0xbfffffff] Feb 6 14:26:11 lin02 kernel: PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] Feb 6 14:26:11 lin02 kernel: PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] Feb 6 14:26:11 lin02 kernel: PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] Feb 6 14:26:11 lin02 kernel: PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] Feb 6 14:26:11 lin02 kernel: please try 'cgroup_disable=memory' option if you don't want memory cgroups Feb 6 14:26:11 lin02 kernel: Initializing cgroup subsys memory Feb 6 14:26:11 lin02 kernel: Freeing initrd memory: 16756k freed Feb 6 14:26:11 lin02 kernel: Non-volatile memory driver v1.3 Feb 6 14:26:11 lin02 kernel: crash memory driver: version 1.1 Feb 6 14:26:11 lin02 kernel: Freeing unused kernel memory: 1636k freed Feb 6 14:26:11 lin02 kernel: [TTM] Zone kernel: Available graphics memory: 8131870 kiB Feb 6 14:26:11 lin02 kernel: [TTM] Zone dma32: Available graphics memory: 2097152 kiB Feb 6 14:30:02 lin02 pam_lve[5739]: (CRON) CAN'T FORK (child_process): Cannot allocate memory Feb 6 14:30:04 lin02 crond: /usr/sbin/sendmail: Cannot allocate memory Feb 6 14:30:04 lin02 crond: /usr/sbin/sendmail: Cannot allocate memory Feb 6 14:30:04 lin02 crond: /usr/sbin/sendmail: Cannot allocate memory Feb 6 14:30:04 lin02 crond: /usr/sbin/sendmail: Cannot allocate memory Feb 6 14:30:04 lin02 crond: /usr/sbin/sendmail: Cannot allocate memory Feb 6 14:30:04 lin02 crond: /usr/sbin/sendmail: Cannot allocate memory Feb 6 15:00:02 lin02 pam_lve[18335]: (CRON) CAN'T FORK (child_process): Cannot allocate memory Feb 6 15:00:02 lin02 pam_lve[18333]: (CRON) CAN'T FORK (child_process): Cannot allocate memory Feb 6 15:00:02 lin02 pam_lve[18312]: (CRON) CAN'T FORK (child_process): Cannot allocate memory Feb 6 15:00:02 lin02 pam_lve[18310]: (CRON) CAN'T FORK (child_process): Cannot allocate memory Memory is not an issue as at all times 4GB is available for the 10Gb. I've even increased it now to 16GB from 10GB lets see
0
cPanelMichael

February 08, 2016 17:42
Hello :) Are you utilizing the MySQL Governor application with Cloud Linux? Also, this thread is a few years old but it may include information that's helpful to you: Application Resource Limits In Cloudlinux Thank you.
0

Please sign in to leave a comment.

Comments

Didn't find what you were looking for?