High load on server caused by PHP-FPM
I have VPS with 4 cores, but the load average reach 300 !
In Process Manager i see a lot of php-fpm: pool mywebsite_com
The problem is: I don't know which file cause problems or am I under attack.
The question is: How can I know what is the problem?
-
I have VPS with 4 cores, but the load average reach 300 ! In Process Manager i see a lot of php-fpm: pool mywebsite_com The problem is: I don't know which file cause problems or am I under attack. The question is: How can I know what is the problem?
Enable slow PHP logging, it can in most cases narrow down the problematic scripts. Modify: /opt/cpanel/ea-php*/root/etc/php-fpm.d/domain.conf You can leave the asterisk next to php if you don't know the PHP version and just replace domain.com example if you use nano:nano /opt/cpanel/ea-php*/root/etc/php-fpm.d/bob.com.conf
then at the bottom of the config, add:request_slowlog_timeout = 10 slowlog = /home/bobcom/logs/bob.com.php.slow.log
replace bobcom with the cpanel user and bob.com with your actual domain. then restart php-fpm/scripts/restartsrv_apache_php_fpm
Now once a script has been serving a single request for more than 10 seconds, it will trace the process and output a bunch of hopefully useful info for you into that log file. Secondly, you can tail the access log file to see your traffic:tail -f /home/bobcom/access-logs/bob.com*
See anything suspicious? Example if running a WordPress site you might see lots of hits to xmlrpc.php and/or wp-login.php Otherwise, it might just be your site's dynamic usage, again if its WordPress this can in most cases be resolved by simply installing a caching plugin. Let us know what you find0 -
What is the RAM of your server? It should be atleast 1 GB. And by the way, browscap also causes high memory load. And about 5 seconds increase on page load. Do not restart your server if you have browscap.ini and have less than 1GB or 1GB of RAM.
8 GB of ram, it was working very well and fast. But now sometimes the load reach 300 I think there is a problem in some php files in my website, but i can't see it process manager because PHP-FPM When I disable PHP-FPM to see the file I get error 500 in my website.0 -
When I disable PHP-FPM to see the file I get error 500 in my website.
Did you check your error logs? If running PHP-FPM PHP errors get logged to:/home/user/logs/domain.com.php.error.log
when PHP-FPM disabled, it will be in an error_log file where the script is throwing the error, you can search for all error_log files:find /home/user/public_html/ -type f -name error_log0 -
Enable slow PHP logging, it can in most cases narrow down the problematic scripts. Modify: /opt/cpanel/ea-php*/root/etc/php-fpm.d/domain.conf You can leave the asterisk next to php if you don't know the PHP version and just replace domain.com example if you use nano:
nano /opt/cpanel/ea-php*/root/etc/php-fpm.d/bob.com.conf
then at the bottom of the config, add:request_slowlog_timeout = 10 slowlog = /home/bobcom/logs/bob.com.php.slow.log
replace bobcom with the cpanel user and bob.com with your actual domain. then restart php-fpm/scripts/restartsrv_apache_php_fpm
Now once a script has been serving a single request for more than 10 seconds, it will trace the process and output a bunch of hopefully useful info for you into that log file. Secondly, you can tail the access log file to see your traffic:tail -f /home/bobcom/access-logs/bob.com*
See anything suspicious? Example if running a WordPress site you might see lots of hits to xmlrpc.php and/or wp-login.php Otherwise, it might just be your site's dynamic usage, again if its WordPress this can in most cases be resolved by simply installing a caching plugin. Let us know what you find
Thank you very much, I added the code in the file. In the same file i see this: pm = ondemand pm.max_children = 200 pm.max_requests = 1000 pm.max_spare_servers = 5 pm.min_spare_servers = 1 pm.process_idle_timeout = 1 pm.start_servers = 0 Is it good settings?0 -
When I entered this code /scripts/restartsrv_apache_php_fpm I got Service Error (XID 5c2fjf) The "apache_php_fpm" service failed to restart because the restart script exited with an error: apache_php_fpm has failed. Contact your system administrator if the service does not automagically recover. 0 -
[root@ ~]# /scripts/restartsrv_apache_php_fpm apache_php_fpm restarted successfully. Now it works I will see what will happen This is the first time i see the load reach 300. maybe it is ddos? Now Load Averages: 2.55 1.76 1.22 which is normal 0 -
Something was entered in incorrectly then. My guess is the path you entered in for slowlog is wrong. Make sure its the path that exists already for your cPanel accounts logs folder. 0 -
This is the first time i see the load reach 300. maybe it is ddos?
Impossible for me to say, we have no information to go off of other than your load is hitting 300. Did you check the access logs yet?0 -
Yes you are right .. I didn't change bob ::) 0 -
Did you check the access logs yet?
I see it now, but it just view the last hour and everything seems normal The problem was at this morning0 -
Use 'cat' to print out the entire access log, then scroll up to the time the issue was occuring. You can pipe it through less so you can use page up / page down to scroll through. cat access.log | less0 -
Use 'cat' to print out the entire access log, then scroll up to the time the issue was occuring. You can pipe it through less so you can use page up / page down to scroll through.
cat access.log | less
The first log at [07/Oct/2019:12:12:30 +0000] But the problem was at 10:00am Very nice codes thank you Jcats, I will use it when it happens again I think i will immediately know what is the problem if i disable php-fpm so I can view file name in Process manager. But when I disable it i got error 500 in my website0 -
I disabled PHP-FPM now and I got [Mon Oct 07 15:37:44.059871 2019] [:error] [pid 19248:tid 47142920025856] [client 151.255.14.183:20502] SoftException in Application.cpp:267: File "/home/xx/public_html/xx.php" is writeable by group [Mon Oct 07 15:37:44.059930 2019] [core:error] [pid 19248:tid 47142920025856] [client 151.255.14.183:20502] End of script outputbefore headers: xx.php [Mon Oct 07 15:37:44.115858 2019] [:error] [pid 19278:tid 47142938937088] 0 -
My message was deleted not sure why, guess your SOL. 0 -
My message was deleted not sure why, guess your SOL.
Do you know how to fix it?0 -
My message was deleted not sure why
An average user not sure what they're doing could get into trouble setting permissions on all files.0 -
The reason for the high load can be anything - not exactly FPM. High number indicates there are many active processes, so if PHP processes is locked in memory for some reason - it will add extra number there. The reason for the lock is not always obvious, although something you can clearly see and do. You should limit number of PHP-FPM processes. Slow log is a good stuff, but you should examine Apache status and see, if there is some IP or URL, that repeats - very often spammers or hijackers attack wordpress login scripts and that can easily generate this kind of load. Sometimes broken MySQL table can give troubles, especially on the old sites. If this has happenned just once - then most likely it was some sort of attack on some script. 0 -
The reason for the high load can be anything - not exactly FPM. High number indicates there are many active processes, so if PHP processes is locked in memory for some reason - it will add extra number there. The reason for the lock is not always obvious, although something you can clearly see and do. You should limit number of PHP-FPM processes. Slow log is a good stuff, but you should examine Apache status and see, if there is some IP or URL, that repeats - very often spammers or hijackers attack wordpress login scripts and that can easily generate this kind of load. Sometimes broken MySQL table can give troubles, especially on the old sites. If this has happenned just once - then most likely it was some sort of attack on some script.
Yes you are right. It was brute-force attack. I installed mod_evasive and I have added some security codes in PHP and everything is good now. Thank you all for your help. I will save this page for future using. Peace.0 -
Thank you @Jcats for all the advice in this thread! @ziadmm0 I'm glad to see you were able to find a solution that resolved the issue - please don't hesitate to let us know if you have any further questions or concerns or if you run into any further issues! 0 -
Jcats solution worked for me. Thanks for the walk-though on this, my server load has decreased significantly. My only issue is I cannot dive into the cause deeper because my access-logs folder does not have any files associated with my domains, or any files period, and when I try to create a new log file, I get an error. ERROR: Could not create file "example.com.log" in /home/examplecom/var/log/apache2/domlogs/examplecom: Directory /home/examplecom/var/log/apache2/domlogs/examplecom does not exist. All of my error logs are contained within the logs folder at the root of my file manager. When I try to access the php.slow.log file, it's permissions are set to 0600 and I am unable to change them or view the file. I am the root user. 0 -
@MikePav - could you create a ticket with our support team so we can look into that issue? There could be several different things going on with Apache and logging that could explain that behavior. 0 -
@MikePav - could you create a ticket with our support team so we can look into that issue? There could be several different things going on with Apache and logging that could explain that behavior.
I have an open ticket at the moment already, #94408397, which I am about to update.0 -
Thanks for that reply - it seems there is a bit more going on with that system than just PHP-FPM, as you also mentioned a DoS attack and some other issues on the AWS side. For public hosting, especially where you don't personally control all the sites, it might be a good idea to look into a tool like CloudLinux as that would let you limit the resources for individual domains on the server. That can be done manually for domains that are causing problems, or inside of an account package so you can assign plans with various levels of resource usage to your clients. For more details about the CloudLinux software, which can be purchased through the cPanel store, their Introduction page is a great place to start: 0
Please sign in to leave a comment.
Comments
24 comments