High CPU load and lots of '/usr/sbin/httpd -k start processes'
Almost everyday my server 'hangs' for about 2 hours, because there are 20+ processes '/usr/sbin/httpd -k start', owned by nobody. If I kill these processes, everything returns to normal. I checked all the logs and also the Apache connections to see if a specific site is causing higher than normal traffic, but that's not the case.
I've searched Google and these forums, but I can't find anyone with the exact same problems.
This is my configuration:
Does this problem sound familiair to someone?
/etc/redhat-release:CentOS release 6.8 (Final)
/usr/local/cpanel/version:11.60.0.26
/var/cpanel/envtype:kvm
CPANEL=release
Server version: Apache/2.4.23 (cPanel)
Server built: Nov 8 2016 16:57:01
ea-php-cli Copyright 2016 cPanel, Inc.
PHP 5.6.28 (cli) (built: Nov 14 2016 15:20:37)
Copyright (c) 1997-2016 The PHP Group
Zend Engine v2.6.0, Copyright (c) 1998-2016 Zend Technologies
with the ionCube PHP Loader v4.7.5, Copyright (c) 2002-2014, by ionCube Ltd., and
with Zend Guard Loader v3.3, Copyright (c) 1998-2014, by Zend Technologies
with Zend OPcache v7.0.4-dev, Copyright (c) 1999-2015, by Zend Technologies
mysql Ver 15.1 Distrib 10.0.28-MariaDB, for Linux (x86_64) using readline 5.1
Does this problem sound familiair to someone?
-
Hello, It's normal to see "/usr/sbin/httpd -k start" processes owned by the "nobody" user. That's the standard username Apache runs as. The following thread is a good place to start if you want to determine what's causing the high load average: Troubleshooting high server loads on Linux servers Thank you. 0 -
Hello, It's normal to see "/usr/sbin/httpd -k start" processes owned by the "nobody" user. That's the standard username Apache runs as. The following thread is a good place to start if you want to determine what's causing the high load average: Troubleshooting high server loads on Linux servers Thank you.
Hi Michael, Thanks for your answer. I already found that thread before posting this question. I've tried the solutions mentioned there, but with no luck. Can I somehow see what these processes are doing? It now occurs multiple times each day and everytime I have to manually stop multiple processes (30+) in order to make my server responsive again. Killing those processes doesn't seem to have any negative impact on the running sites. It looks as if these httpd processes are running something outside of the normal websites. Is there someway I can check this?0 -
What sites do you host is it Wordpress 0 -
Different platforms, WordPress (about 75%), Magento and some CakePHP sites. 0 -
What sites do you host is it Wordpress
Are you sure it's not xmlrpc attacks ?0 -
Are you sure it's not xmlrpc attacks ?
I think I have found the cause of the problems. I've disabled the ModSecurity OWASP package about 36 hours ago and since then I've seen no high loads anymore. If this is still the case tomorrow, I will enable them again to see if the problem returns. If so, what can be wrong with this?0 -
I would not disable ModSecurity 0 -
Hey all, Just wondering if there was any cause established with this as I'm having the same issue. Very few visitors on the server, multiple "/usr/sbin/httpd -k start" tasks showing under Process Manager, high load on server (10-15 on a 6 CPU server). If I restart Apache, it's fine. Seems to happen most/every morning, not exactly at the same time. Curiously (per FabianNL's comment), I also recently added the OWASP vendor ruleset to ModSecurity (during a recent cPanel/WHM update - 64.0.17, I believe). I've tried disabling the automatic updates for the ruleset and this hasn't helped, so next step might be disabling the ruleset altogether, but of course I'd rather not... Any further info? Thanks! Peter. 0 -
Very few visitors on the server, multiple "/usr/sbin/httpd -k start" tasks showing under Process Manager, high load on server (10-15 on a 6 CPU server). If I restart Apache, it's fine.
Hello, You may want to try some of the investigation tips referenced on the following thread to help determine why the server load is high: Troubleshooting high server loads on Linux servers There's a script you can setup that will run on a cron job and allow you to see results from when the server load is high. Thank you.0 -
Hi CpanelMichael I've just experienced this - after an update to 64.0.24, with OWASP Ruleset 3, my server because completely unstable. I was seeing load averages over 15mintes as high and 6.8 and over 1 minute as high as 17 at times!... and on a 6 core VPS - with over 400 customers - meant lots of disgruntled users. Upon reading this forum (and after trying virtually everything, cPanel support, CloudLinux Support) I've disabled OWASP core ruleset, and v3.0 on my server - lo and behold my server load is back around 0.48 0.67 1.95 As much as I dislike disabling Mod Security - high load and unhappy customers are worse to deal with, esp when you have a server loading with 6-7 nobody processes using 100% cpu time - all the while websites are unreachable, no Webmail access for customers - basically a server admin's worst nightmare - stretched over a number of days Any ideas on what the next step is now from here - as I've identified the ruleset to be the cause of load on the server 0 -
We stopped using cpanel rules a long time ago, your much better off with Comodo rules. 0 -
Upon reading this forum (and after trying virtually everything, cPanel support, CloudLinux Support) I've disabled OWASP core ruleset, and v3.0 on my server - lo and behold my server load is back around 0.48 0.67 1.95
Could you verify the size of the /var/cpanel/secdatadir/ip.pag file on the system? EX:du -sh /var/cpanel/secdatadir/ip.pag
Thanks.0 -
Could you verify the size of the /var/cpanel/secdatadir/ip.pag file on the system? EX:
du -sh /var/cpanel/secdatadir/ip.pag
Thanks.
Hi cPpanelMichael, I had checked the size of the file as I saw when I would check the apache status httpd status I'd get a lot of L's for logging WLLLL..LLLLLL..LLLLLL - something like that When I checked the file size, it was 26mb if memory serves. I even cleared it out and tailed the file as it grew to around 400kb, then quit as the load situation was not improving. I took the suggestion offered by popeye, and after disabling a few rules that did not apply - my server has been performing as expected 47451 mind the low uptime is due to the recent kernel updates Regards0 -
I am posting my solution with the hope that it can help somebody (or myself) in the future. I was experiencing the same issue on a 24 core, 64GB RAM server. The load would suddenly rise to more than 100 within few minutes after I manually restart apache. After spending two weeks troubleshooting following the link posted by cPanelMichael and support from cpanel staff the issue was not resolved. I even installed nginx (engintron) and even after paying for optimisation ( I thought I was missing something) the issue was still not resolved. I chanced upon ip.pag file on a post, which turned out to be responsible for the high load The Fix (without disabling OWASP) After monitoring the /usr/bin/httpd processes with htop, I noticed that /tmp/ip.pag (about 368MB) was appearing in a lot of the files opened by the process. However when I googled that file, the results were referring to /var/cpanel/secdatadir/ip.pag but the size was 0 and evidently it wasn't being used. What fixed the issue for me was using the script described here ModSecurity SDBM Utility - EasyApache 4 - cPanel Documentation I had to replace references to /var/cpanel/secdatadir with /tmp. /usr/sbin/modsec-sdbm-util -D /tmp -v -n /tmp/ip.pag &&\ rm /tmp/ip.pag &&\ rm /tmp/ip.dir &&\ mv /tmp/new_db.pag /tmp/ip.pag &&\ mv /tmp/new_db.dir /tmp/ip.dir
After the script executed, the load stabilized and dropped to normal below 10. If you use cloudlinux, consider mod_lsapi also as the php handler. Without mod_lsapi my normal load was above 10, now it is below 10 after using mod_lsapi.0
Please sign in to leave a comment.
Comments
14 comments