PHP-FPM fails to restart
We've been experiencing an issue lately with Apache PHP FPM failing to restart. It was consistently happening every Wednesday at a specific time. Since it was so regular I figured it was likely being caused by a cron job. After looking into that, it was apparent that it coincided with when the upcp process ran with a root cron job, so I tried moving the hour of that cronjob to make sure it was that and it moved with it as well.
I've also tried restarting it from the command line using:
and with the restart services in WHM. All of these fail in the same way and it seems the only solution to get the service to recover is reboot the entire server. The error always appears to be this: Failed at step NAMESPACE spawning /opt/cpanel/ea-php{VERSION}/root/usr/sbin/php-fpm: Cannot allocate memory Version is either 73 or 81, the only two versions of php actively being used by our users, sometimes the error for both versions are reported, but not always. I can't seem to find a way to increase the memory for this process or affect any change to allow it to smoothly restart. Any ideas? Is this a common issue? Is there anyway the resources allowed to be used for this?
/scripts/restartsrv_apache_php_fpmand with the restart services in WHM. All of these fail in the same way and it seems the only solution to get the service to recover is reboot the entire server. The error always appears to be this: Failed at step NAMESPACE spawning /opt/cpanel/ea-php{VERSION}/root/usr/sbin/php-fpm: Cannot allocate memory Version is either 73 or 81, the only two versions of php actively being used by our users, sometimes the error for both versions are reported, but not always. I can't seem to find a way to increase the memory for this process or affect any change to allow it to smoothly restart. Any ideas? Is this a common issue? Is there anyway the resources allowed to be used for this?
-
Hey there! When that issue happens, do you see anything interesting in /var/log/messages on the server? Usually when I see that error it's an issue with overall memory on the server, not memory being allocated to a specific process. 0 -
Hi @cPRex! Running this: /scripts/restartsrv_apache_php_fpm & tail -f /var/log/messages returns the following: [1] 1289 Jun 15 10:36:34 {hostname omitted} kernel: Firewall: *TCP_IN Blocked* IN=venet0 OUT= MAC= SRC=94.102.61.54 DST={ip omitted} LEN=44 TOS=0x00 PREC=0x00 TTL=235 ID=54321 PROTO=TCP SPT=52318 DPT=8453 WINDOW=65535 RES=0x00 SYN URGP=0 Jun 15 10:36:35 {hostname omitted} kernel: Firewall: *TCP_IN Blocked* IN=venet0 OUT= MAC= SRC=194.26.135.247 DST={ip omitted} LEN=40 TOS=0x00 PREC=0x00 TTL=234 ID=8953 PROTO=TCP SPT=52988 DPT=31972 WINDOW=1024 RES=0x00 SYN URGP=0 Jun 15 10:36:36 {hostname omitted} kernel: Firewall: *UDP_IN Blocked* IN=venet0 OUT= MAC= SRC=68.61.190.236 DST={ip omitted} LEN=556 TOS=0x00 PREC=0x00 TTL=109 ID=28317 PROTO=UDP SPT=51985 DPT=161 LEN=536 Jun 15 10:36:43 {hostname omitted} kernel: Firewall: *UDP_IN Blocked* IN=venet0 OUT= MAC= SRC=68.61.190.236 DST={ip omitted} LEN=106 TOS=0x00 PREC=0x00 TTL=109 ID=28318 PROTO=UDP SPT=63297 DPT=161 LEN=86 Jun 15 10:37:01 {hostname omitted} systemd: Created slice User Slice of root. Jun 15 10:37:01 {hostname omitted} systemd: Started Session 1859138 of user root. Jun 15 10:37:01 {hostname omitted} systemd: Removed slice User Slice of root. Jun 15 10:37:21 {hostname omitted} systemd: Created slice User Slice of root. Jun 15 10:37:21 {hostname omitted} systemd-logind: New session 1859205 of user root. Jun 15 10:37:21 {hostname omitted} systemd: Started Session 1859205 of user root. Jun 15 10:37:24 {hostname omitted} systemd: Cannot add dependency job for unit systemd-vconsole-setup.service, ignoring: Unit is masked. Jun 15 10:37:24 {hostname omitted} systemd: Cannot add dependency job for unit proc-sys-fs-binfmt_misc.automount, ignoring: Unit is masked. Jun 15 10:37:24 {hostname omitted} systemd: Stopping The PHP FastCGI Process Manager... Jun 15 10:37:24 {hostname omitted} systemd: Stopped The PHP FastCGI Process Manager. Jun 15 10:37:24 {hostname omitted} systemd: Starting The PHP FastCGI Process Manager... Jun 15 10:37:24 {hostname omitted} systemd: Failed at step NAMESPACE spawning /opt/cpanel/ea-php81/root/usr/sbin/php-fpm: Cannot allocate memory Jun 15 10:37:24 {hostname omitted} systemd: ea-php81-php-fpm.service: main process exited, code=exited, status=226/NAMESPACE Jun 15 10:37:24 {hostname omitted} systemd: Failed to start The PHP FastCGI Process Manager. Jun 15 10:37:24 {hostname omitted} systemd: Unit ea-php81-php-fpm.service entered failed state. Jun 15 10:37:24 {hostname omitted} systemd: ea-php81-php-fpm.service failed. [ERROR] Failed to perform action "restart" for php-fpm "81": The "/usr/bin/systemctl restart ea-php81-php-fpm" command (process 1291) reported error number 1 when it ended. Job for ea-php81-php-fpm.service failed because the control process exited with error code. See "systemctl status ea-php81-php-fpm.service" and "journalctl -xe" for details. : ? ea-php81-php-fpm.service - The PHP FastCGI Process Manager Loaded: loaded (/usr/lib/systemd/system/ea-php81-php-fpm.service; enabled; vendor preset: disabled) Active: failed (Result: exit-code) since Thu 2023-06-15 10:37:24 EDT; 82ms ago Process: 1295 ExecStart=/opt/cpanel/ea-php81/root/usr/sbin/php-fpm --nodaemonize (code=exited, status=226/NAMESPACE) Main PID: 1295 (code=exited, status=226/NAMESPACE) Jun 15 10:37:24 {hostname omitted} systemd[1]: Starting The PHP FastCGI Process Manager... Jun 15 10:37:24 {hostname omitted} systemd[1295]: Failed at step NAMESPACE spawning /opt/cpanel/ea-php81/root/usr/sbin/php-fpm: Cannot allocate memory Jun 15 10:37:24 {hostname omitted} systemd[1]: ea-php81-php-fpm.service: main process exited, code=exited, status=226/NAMESPACE Jun 15 10:37:24 {hostname omitted} systemd[1]: Failed to start The PHP FastCGI Process Manager. Jun 15 10:37:24 {hostname omitted} systemd[1]: Unit ea-php81-php-fpm.service entered failed state. Jun 15 10:37:24 {hostname omitted} systemd[1]: ea-php81-php-fpm.service failed. at /usr/local/cpanel/Cpanel/PHPFPM/Controller.pm line 140. Jun 15 10:37:24 {hostname omitted} systemd: Cannot add dependency job for unit systemd-vconsole-setup.service, ignoring: Unit is masked. Jun 15 10:37:24 {hostname omitted} systemd: Cannot add dependency job for unit proc-sys-fs-binfmt_misc.automount, ignoring: Unit is masked. Jun 15 10:37:24 {hostname omitted} systemd: Stopping The PHP FastCGI Process Manager... Jun 15 10:37:24 {hostname omitted} systemd: Stopped The PHP FastCGI Process Manager. Jun 15 10:37:24 {hostname omitted} systemd: Starting The PHP FastCGI Process Manager... Jun 15 10:37:24 {hostname omitted} systemd: Started The PHP FastCGI Process Manager. Cpanel::Exception::Services::RestartError Service Status Found 2 versions of PHP-FPM: 81 73 Service Error (XID mye92t) The "apache_php_fpm" service failed to restart because the restart script exited with an error: apache_php_fpm has failed. Contact your system administrator if the service does not automagically recover. Jun 15 10:37:25 {hostname omitted} kernel: Firewall: *UDP_IN Blocked* IN=venet0 OUT= MAC= SRC=68.61.190.236 DST={ip omitted} LEN=106 TOS=0x00 PREC=0x00 TTL=109 ID=28326 PROTO=UDP SPT=63297 DPT=161 LEN=86 0 -
Jun 15 10:37:24 {hostname omitted} systemd: ea-php81-php-fpm.service failed. [ERROR] Failed to perform action "restart" for php-fpm "81": The "/usr/bin/systemctl restart ea-php81-php-fpm" command (process 1291) reported error number 1 when it ended. Job for ea-php81-php-fpm.service failed because the control process exited with error code. See "systemctl status ea-php81-php-fpm.service" and "journalctl -xe" for details. : ? ea-php81-php-fpm.service - The PHP FastCGI Process Manager Loaded: loaded (/usr/lib/systemd/system/ea-php81-php-fpm.service; enabled; vendor preset: disabled) Is the useful part. You'd have to run journalctl -xe after it fails to get more details. 0 -
Yup, that's the next step! Let us know what that shows! 0 -
Here is the output from that: Jun 16 10:56:25 {hostname omitted} systemd[1]: Starting The PHP FastCGI Process Manager... -- Subject: Unit ea-php73-php-fpm.service has begun start-up -- Defined-By: systemd -- Support: systemd-devel Info Page -- -- Unit ea-php73-php-fpm.service has begun starting up. Jun 16 10:56:25 {hostname omitted} systemd[29202]: Failed at step NAMESPACE spawning /opt/cpanel/ea-php73/root/usr/sbin/php-fpm: Cannot allocate memory -- Subject: Process /opt/cpanel/ea-php73/root/usr/sbin/php-fpm could not be executed -- Defined-By: systemd -- Support: systemd-devel Info Page -- -- The process /opt/cpanel/ea-php73/root/usr/sbin/php-fpm could not be executed and failed. -- -- The error number returned by this process is 12. Jun 16 10:56:25 {hostname omitted} systemd[1]: ea-php73-php-fpm.service: main process exited, code=exited, status=226/NAMESPACE Jun 16 10:56:25 {hostname omitted} systemd[1]: Failed to start The PHP FastCGI Process Manager. -- Subject: Unit ea-php73-php-fpm.service has failed -- Defined-By: systemd -- Support: systemd-devel Info Page -- -- Unit ea-php73-php-fpm.service has failed. -- -- The result is failed. Any ideas? 0 -
I don't see anything specifically helpful from that output. It might be worth reaching out to your hosting provider to have them check the server, because that message seems to indicate that the system as a whole thinks it is low on memory. We do note that PHP-FPM is best run on systems where there is at least 2G of free RAM, as it does take a bit to get that service turned on and running. 0 -
We will reach out to the hosting provider, but for what it's worth we have about 33Gb of ram on this server, so I think it that shouldn't be an issue. I think part of our problem is the memory limit for php fpm itself isn't set high enough, but I don't know where to/if you can adjust that. 0 -
If you manually run upcp --force are you able to reproduce it? Are you swapping to disk due to low available memory when the issue occurs? I would start by profiling memory usage when upcp runs to see what the culprit is. Regardless of how much memory is physically in the server, availability is the key 0 -
upcp --force did not reproduce the error, upcp on Wednesday nights only seems to trigger the issue though. Any other time you try to restart the PHP FPM service on the server it happens so perhaps PHP FPM doesn't always restart with upcp? We are not swapping the disk due to my knowledge. top didn't indicate any spike in memory usage when using Restart Services > PHP-FPM service for Apache I ran the following to see about shared memory just normally, not when restarting PHP FPM echo $((`cat /proc/sys/kernel/shmmax` / 1024 / 1024))Mb -16Mb That is obviously very low compared to what I'd expect, can I increase that? 0
Please sign in to leave a comment.
Comments
9 comments