PHP-FPM configuration settings
once a day at random times I get notified : The service "apache_php_fpm" appears to be down.
it goes down then recovers in 4 minutes.
I really have no idea where to start and need some help to get this stable, thanks to anyone in advance.
fresh new server just migrated all clients from another cpanel server. server has 32cores 400gb ssd RAID 1 hardware - 96GB Ram - CLOUDLINUX 7.3 x86_64 standard "cPanel & WHM 64.0 (build 33) about 100 sites all running php-fpm with all default pool settings mostly php5.6 and a few php 7.
more error report info below and this image: [removed]
Memory Information
Used 4.85 GB
Available 89.36 GB
Installed 94.21 GB
Load Information 2.06 1.80 1.48
Service Check Raw Output
The subprocess "/usr/local/cpanel/scripts/restartsrv_apache_php_fpm" reported error number 9 when it ended.
-
Does this affect any web site?. I think this is cPanel daemon's internal php using php-fpm and don't affect the php-fpm service used by the hosted domains . 0 -
thanks for the reply. I am not sure how to determine if it is site or cpanel, would there be a specific log I could review? It is only down for 4 minutes or less according to the email report and I have yet to check sites in real time while down to see if they are non responsive since the window is so small and seems random. 0 -
Hello, Could you let us know the output from /usr/local/apache/logs/error_log when this happens? Ensure to replace real domain names and IP addresses with examples, and post the output in CODE tags. Thank you. 0 -
If you let me know how I can run a command to get the entire log as a downloaded file I can post privately or personal message. I grabbed a few entries that seem problematic. I am getting thousands of these entries [Mon Jul 24 08:46:48.395089 2017] [proxy_fcgi:error] [pid 6920:tid 139924917442304] [client 0.0.0.0:50723] AH01071: Got error 'Primary script unknown\n'
[Mon Jul 24 08:47:05.680480 2017] [ssl:error] [pid 6920:tid 139924791551744] AH02032: Hostname 0.0.0.0 provided via SNI and hostname www.domain.com provided via HTTP have no compatible SSL setup
[Mon Jul 24 09:05:55.062773 2017] [mpm_worker:notice] [pid 3142:tid 139925245261952] AH00297: SIGUSR1 received. Doing graceful restart [Mon Jul 24 09:05:55.319296 2017] [hostinglimits:notice] [pid 3142:tid 139925245261952] mod_hostinglimits: use Min UID 0 [Mon Jul 24 09:05:55.319331 2017] [hostinglimits:notice] [pid 3142:tid 139925245261952] mod_hostinglimits: version 1.0-32. LVE mechanism enabled [Mon Jul 24 09:05:55.319340 2017] [hostinglimits:notice] [pid 3142:tid 139925245261952] mod_hostinglimits: found apr extention version 3 [Mon Jul 24 09:05:55.319355 2017] [hostinglimits:notice] [pid 3142:tid 139925245261952] mod_hostinglimits: apr_lve_environment_init_group_minuid check ok [Mon Jul 24 09:05:55.374054 2017] [ssl:warn] [pid 3142:tid 139925245261952] AH01909: domain.com:443:0 server certificate does NOT include an ID which matches the server name [Mon Jul 24 09:05:55.375679 2017] [ssl:warn] [pid 3142:tid 139925245261952] AH01909: domain.com:443:0 server certificate does NOT include an ID which matches the server name [Mon Jul 24 09:05:55.408460 2017] [mpm_worker:notice] [pid 3142:tid 139925245261952] AH00292: Apache/2.4.25 (cPanel) OpenSSL/1.0.2k mod_bwlimited/1.4 configured -- resuming normal operations [Mon Jul 24 09:05:55.408505 2017] [core:notice] [pid 3142:tid 139925245261952] AH00094: Command line: '/usr/sbin/httpd'
[Mon Jul 24 09:16:44.073968 2017] [core:error] [pid 15769:tid 139924799944448] [client 0.0.0.0:37320] AH00124: Request exceeded the limit of 10 internal redirects due to probable configuration error. Use 'LimitInternalRecursion' to increase the limit if necessary. Use 'LogLevel debug' to get a backtrace. [Mon Jul 24 09:16:44.145855 2017] [core:error] [pid 16271:tid 139925022525184] [client 0.0.0.0:37322] AH00124: Request exceeded the limit of 10 internal redirects due to probable configuration error. Use 'LimitInternalRecursion' to increase the limit if necessary. Use 'LogLevel debug' to get a backtrace. [Mon Jul 24 09:16:44.217821 2017] [core:error] [pid 15717:tid 139924774766336] [client 0.0.0.0:37324] AH00124: Request exceeded the limit of 10 internal redirects due to probable configuration error. Use 'LimitInternalRecursion' to increase the limit if necessary. Use 'LogLevel debug' to get a backtrace. [Mon Jul 24 09:16:45.390174 2017] [core:error] [pid 15769:tid 139924883871488] [client 0.0.0.0:37332] AH00124: Request exceeded the limit of 10 internal redirects due to probable configuration error. Use 'LimitInternalRecursion' to increase the limit if necessary. Use 'LogLevel debug' to get a backtrace.0 -
Same thing started happening to me about two weeks ago. I get the fail email then few ins later the restart email for php fpm around same time. 0 -
Hello, Feel free to open a support ticket using the link in my signature so we can take a closer look at an affected system to see what's happening. Thank you. 0 -
once a day at random times I get notified : The service "apache_php_fpm" appears to be down. it goes down then recovers in 4 minutes.
Yes, This same issue happened on my server.0 -
@cPanelMichael I submitted a ticket ID is: 8748347. please let me know when you are able to take a look at this. the crashes are increasing daily from only one to now two or three per day. 0 -
Hello, To update, internal case CPANEL-13411 is open to address an issue where the service check for apache_php_fpm hangs when a domain reaches the max_children PHP-FPM limit. This leads to an error message like this during the service check: apache_php_fpm [[check command:-][check command output: The subprocess "/usr/local/cpanel/scripts/restartsrv_apache_php_fpm" reported error number 9 when it ended.][socket connect:N/A][fail count:1]Restarting apache_php_fpm.... [notify:failed service:apache_php_fpm]]...Done Service Check Finished
I'll monitor this case and update this thread with more information as it becomes available. In the meantime, the temporary workaround is to search the PHP-FPM error logs (/opt/cpanel/ea-php$$/root/usr/var/log/php-fpm/error.log) for any domain name's reaching the max_children limit. The offending entries will look like this:ERROR: unable to read what child say: Bad file descriptor (9) NOTICE: [pool domain_tld] child 1234 exited with code 0 after 25.12345 seconds from start
Once you find which domain names are reaching the limit, you can increase the max_children value for them using the "Pool Options" feature in "WHM >> MultiPHP Manager". Thank you.0 -
@cPanelMichael Does this issue still are not fixed? I have set Max Children to 15 from 5, Does this enough valve for Max Children? 0 -
@cPanelMichael Does this issue still are not fixed? I have set Max Children to 15 from 5, Does this enough valve for Max Children?
There's no update to report on CPANEL-13411 at this time. I'll continue to monitor the case and update this thread with new information as it becomes available. Increasing the "Max Children" limit for the offending domain name from "5" to "15" should act as a workaround, yes. However, you may want to monitor the PHP-FPM error log after making the change to verify the error messages no longer appear. Thank you.0 -
Hi, I am checking my PHP-FPM error logs and I get no "error". Thousands of lines like these, all the same. Is this ok? [15-Dec-2017 21:30:50] NOTICE: [pool domain_com] child 13438 exited with code 0 after 78.066207 seconds from start [15-Dec-2017 21:30:50] NOTICE: [pool domain_com] child 13655 started [15-Dec-2017 21:30:57] NOTICE: [pool domain_com] child 13444 exited with code 0 after 78.084127 seconds from start [15-Dec-2017 21:30:57] NOTICE: [pool domain_com] child 13657 started [15-Dec-2017 21:31:26] NOTICE: [pool domain_com] child 13595 exited with code 0 after 79.547892 seconds from start [15-Dec-2017 21:31:26] NOTICE: [pool domain_com] child 13682 started 0 -
Hello @PabloC, Those are just notices and not actual errors. That should not lead to any actual problems. Thank you. 0 -
Increasing the "Max Children" limit for the offending domain name from "5" to "15" should act as a workaround, yes.
Is there truly no global way to change the pm.max_children value for all users under cPanel's implementation of FPM? Having to resort to making a manual change to thousands of hosting accounts one-at-a-time over multiple servers is ludicrous. Although mass updates of php_admin type entries are possible via... /var/cpanel/ApachePHPFPM/system_pool_defaults.yaml ...such as... php_admin_flag_allow_url_fopen: { name: 'php_admin_flag[allow_url_fopen]', value: Off } ...entries such as the following are not processed by /scripts/php_fpm_config --rebuild... pm_max_children: { name: 'pm.max_children', value: 15 } cPanel's documentation suggests otherwise at Configurations Values of PHP-FPM - Version 68 Documentation - cPanel Documentation but the documentation is either untested or out-of-date. Am I missing a trick here? Best regards, LBJ0 -
Hello @LBJ, It looks like the entry you added for the max children value is incorrect. You should be able to simply add the following line to the /var/cpanel/ApachePHPFPM/system_pool_defaults.yaml file: pm_max_children: 20
Then, run the following command:/usr/local/cpanel/scripts/php_fpm_config --rebuild
Additionally, I encourage you to vote for the following feature request: PHP-FPM Enhanced Configuration Thank you.0 -
You should be able to simply add the following line to the /var/cpanel/ApachePHPFPM/system_pool_defaults.yaml file:
pm_max_children: 20
Then, run the following command:/usr/local/cpanel/scripts/php_fpm_config --rebuild
Nope. That won't do the trick with all underscores in the key. Even though that's not in line with your own documentation, we did try that. The closest you can get is with a rebuild after adding... pm.max_children: 20 to /var/cpanel/ApachePHPFPM/system_pool_defaults.yaml However, that still only gives you a conflicting directive as per... ... pm = ondemand pm.max_children = 20 pm.max_children = 5 pm.max_requests = 20 pm.max_spare_servers = 5 pm.min_spare_servers = 1 ... The lower directive takes control and you still end up with 5 as the max_children value. The error.log continues to show it that way too. I'd really encourage you to set up a test system and try this all out for yourself. I think you'd be amazed at the size of the void between what's supposed to happen according to cPanel documentation with FPM and what really does. Best regards, LBJ0 -
Hi @LBJ, I tested that modification before sending my previous response and it worked well. Here's exactly how the /var/cpanel/ApachePHPFPM/system_pool_defaults.yaml looks on the test machine: # cat /var/cpanel/ApachePHPFPM/system_pool_defaults.yaml pm_max_children: 20
Upon running the "/usr/local/cpanel/scripts/php_fpm_config --rebuild" command, here's the output when viewing a user's PHP-FPM configuration file:# grep children /opt/cpanel/ea-php56/root/etc/php-fpm.d/example.tld.conf pm.max_children = 20
I just changed "pm_max_children: 20" to "pm_max_children: 25", ran the "/usr/local/cpanel/scripts/php_fpm_config --rebuild" command again, and it now shows 25:# grep children /opt/cpanel/ea-php56/root/etc/php-fpm.d/example.tld.conf pm.max_children = 25
It also shows the correct value under "Pool Options" in "WHM >> MultiPHP Manager". Are you sure there are no other entries in your /var/cpanel/ApachePHPFPM/system_pool_defaults.yaml file, or no other PHP-FPM custom configurations are enabled? Thank you.0 -
OK. We're now entering the realm of magic. :) That simply doesn't work on any one of our servers, especially not with all underscores as you've entered. Here's what happens if we follow your example... # grep children /opt/cpanel/ea-php71/root/etc/php-fpm.d/dummy.com.conf pm.max_children = 5 # cat /var/cpanel/ApachePHPFPM/system_pool_defaults.yaml pm_max_children: 20 php_admin_value_disable_functions: { name: 'php_admin_value[disable_functions]', value: exec,system,shell_exec,passthru,popen,proc_open,proc_close,proc_get_status,proc_nice,proc_terminate,show_source } php_admin_flag_allow_url_fopen: { name: 'php_admin_flag[allow_url_fopen]', value: Off } php_admin_value_error_reporting: { name: 'php_admin_value[error_reporting]', value: E_ALL & ~E_NOTICE & ~E_DEPRECATED & ~E_STRICT } # /scripts/php_fpm_config --rebuild # grep children /opt/cpanel/ea-php71/root/etc/php-fpm.d/dummy.com.conf pm.max_children = 5 WHM also shows 5 ---- Now if we go with pm.max_children: 20 instead of pm_max_children: 20 --- # grep children /opt/cpanel/ea-php71/root/etc/php-fpm.d/dummy.com.conf pm.max_children = 5 # cat /var/cpanel/ApachePHPFPM/system_pool_defaults.yaml pm.max_children: 20 php_admin_value_disable_functions: { name: 'php_admin_value[disable_functions]', value: exec,system,shell_exec,passthru,popen,proc_open,proc_close,proc_get_status,proc_nice,proc_terminate,show_source } php_admin_flag_allow_url_fopen: { name: 'php_admin_flag[allow_url_fopen]', value: Off } php_admin_value_error_reporting: { name: 'php_admin_value[error_reporting]', value: E_ALL & ~E_NOTICE & ~E_DEPRECATED & ~E_STRICT } # /scripts/php_fpm_config --rebuild # grep children /opt/cpanel/ea-php71/root/etc/php-fpm.d/dummy.com.conf pm.max_children = 20 pm.max_children = 5 WHM still shows 5 and the error.log reports accordingly. Best regards, LBJ Edit performed to mask domain name. 0 -
...and just in case having other directives in the defaults yaml somehow makes a difference... # grep children /opt/cpanel/ea-php71/root/etc/php-fpm.d/dummy.com.conf pm.max_children = 5 # cat /var/cpanel/ApachePHPFPM/system_pool_defaults.yaml pm_max_children: 20 # /scripts/php_fpm_config --rebuild # grep children /opt/cpanel/ea-php71/root/etc/php-fpm.d/dummy.com.conf pm.max_children = 5 ---- # grep children /opt/cpanel/ea-php71/root/etc/php-fpm.d/dummy.com.conf pm.max_children = 5 # cat /var/cpanel/ApachePHPFPM/system_pool_defaults.yaml pm.max_children: 20 # /scripts/php_fpm_config --rebuild # grep children /opt/cpanel/ea-php71/root/etc/php-fpm.d/dummy.com.conf pm.max_children = 20 pm.max_children = 5 Best regards, LBJ 0 -
I've now set up a test VPS to check this further. On the test VPS, we're seeing the same result as we do on all our dedicated servers. Adding an entry such as pm_max_children: 20 to the yaml, and then rebuilding, does not have the same desired result on any of our v68.0.21 boxes as you're demonstrating above. The test box is obviously a completely different architecture to our live servers, and its only similarity is that it runs the same cPanel version and basic software configuration. I'm going to raise a support ticket with cPanel to clarify this. Best regards, LBJ 0 -
Your Support Request ID is: 9158379 G'day Team, We've had a discussion with cPanelMichael via the cPanel forum regarding an issue on all our servers where the process of configuring PHP-FPM values is not working as designed on any of our boxes. For background information, please follow the thread onwards from... php-fpm fails once a day When we add an entry exactly as cPanelMichael specifies like... pm_max_children: 20 ...to the file... /var/cpanel/ApachePHPFPM/system_pool_defaults.yaml ...and then run... /scripts/php_fpm_config --rebuild ...we do not see the configuration move through as cPanelMicahel advises that it should to the WHM interface and... /opt/cpanel/ea-php71/root/etc/php-fpm.d/dummy.com.conf Steps to reproduce: nano /var/cpanel/ApachePHPFPM/system_pool_defaults.yaml Add the entry... pm_max_children: 20 Save the change. Then run... /scripts/php_fpm_config --rebuild Then check for the change in... nano /opt/cpanel/ea-php71/root/etc/php-fpm.d/dummy.com.conf Also check in... WHM > Software > MultiPHP Manager > Pool options Best regards, LBJ 0 -
Solved. With great thanks to cPanelMichael from this forum, and Timothy G from cPanel support, I'd like to summarize in one spot a relatively complete process (along with the "gotchas" to be aware of) when making a global/default change to PHP-FPM configuration values. This works perfectly for us, but if anyone sees an error, please let me know and I'll edit this post for the benefit of others visiting the forum. 1. Individual user configuration files, which take precedence over new global values, are held at... /var/cpanel/userdata/*/*.php-fpm.yaml
If these individual user files contain a conflicting setting, they will take precedence over a new global value. So if you want to set pm_max_children to a new global value for example, you would first need to make sure no conflicting pm_max_children values exist in the individual user files. A completely vanilla user configuration file looks like...--- _is_present: 1
2. To define new default values which you want to apply, the entries need to be added to.../var/cpanel/ApachePHPFPM/system_pool_defaults.yaml
While some directives are accepted without a strict YAML "---" document separator, some critically are not. To avoid confusion and frustration, always begin the system_pool_defaults.yaml with the YAML "---" document separator. For simple PHP-FPM user pool values, the working syntax is along the lines of...pm_max_children: 15 pm_max_requests: 30
For PHP php_admin type values, one of a few working syntaxes is...php_admin_flag_allow_url_fopen: { name: 'php_admin_flag[allow_url_fopen]', value: Off } php_admin_value_error_reporting: { name: 'php_admin_value[error_reporting]', value: E_ALL & ~E_NOTICE & ~E_DEPRECATED & ~E_STRICT }
I prefer the above syntax since it agrees with the official cPanel document at0 -
To update, internal case CPANEL-13411 is open to address an issue where the service check for apache_php_fpm hangs when a domain reaches the max_children PHP-FPM limit.
Internal case CPANEL-13411 still are not fixed?0 -
Hi @LBJ, Following up to thank you for sharing the outcome. The guide you posted should help other users with the same question. @Nirjonadda, there's no update to report on the status of internal case CPANEL-13411 at this time. I'll update this thread as soon as new information is available. Thank you. 0 -
Hi, i've followed the guide but still warning logs are logged, leading to over quota accounts. One problem was that i have some crashes when using "php_admin_value_error_reporting", so after reading a user's file, for example /opt/cpanel/ea-php70/root/etc/php-fpm.d/domain.gr.conf, i've noticed that error_reporting was added as php_value_error_reporting, so i changed to it accordingly. is there something else i should change? my files are : #more /var/cpanel/ApachePHPFPM/system_pool_defaults.yaml pm_max_children : 32 pm_max_requests : 500 php_value_error_reporting: { name: 'php_value[error_reporting]', value: E_ALL & ~E_WARNING & ~E_NOTICE & ~E_DEPRECATED & ~E_STRICT } #more /opt/cpanel/ea-php70/root/etc/php-fpm.d/domain.gr.conf ;;;;;;;;;;;;;;;;;;;;;;;;;;;; ; cPanel FPM Configuration ; ;;;;;;;;;;;;;;;;;;;;;;;;;;;; ; NOTICE This file is generated, please use our WHM User Interface ; to set these value. [some_user] catch_workers_output = yes chdir = /home/some_user group = some_user listen = /opt/cpanel/ea-php70/root/usr/var/run/php-fpm/a3aeeefa43e686e506b95227da7d3ec84b11e212.sock listen.group = nobody listen.mode = 0660 listen.owner = some_user php_admin_flag[allow_url_fopen] = on php_admin_flag[log_errors] = on php_admin_value[disable_functions] = exec,passthru,shell_exec,system php_admin_value[doc_root] = "/home/some_user/public_html" php_admin_value[error_log] = /home/some_user/logs/some_user.php.error.log php_admin_value[short_open_tag] = on php_value[error_reporting] = E_ALL & ~E_WARNING & ~E_NOTICE & ~E_DEPRECATED & ~E_STRICT ping.path = /ping pm = ondemand pm.max_children = 32 pm.max_requests = 500 pm.max_spare_servers = 5 pm.min_spare_servers = 1 pm.process_idle_timeout = 10 pm.start_servers = 0 pm.status_path = /status security.limit_extensions = .phtml .php .php3 .php4 .php5 .php6 .php7 user = some_user best regards, 0 -
Hi, i've followed the guide but still warning logs are logged, leading to over quota accounts.
Hello, Can you verify the specific behavior you are seeking? For instance, do you want to stop the logging of all errors? Thank you.0 -
Hi, i want to log only errors and not warnings, notices, but my logs still getting full of warnings. 0 -
Hi @dzamanakos, Can you open a support ticket so we can take a closer look at the configuration file and see why the changes are not applied? Create Support Ticket - Version 68 Documentation - cPanel Documentation Thank you. 0 -
@cPanelMichael Internal case CPANEL-13411 still are not fixed? 0 -
New @cPanelMichael Internal case CPANEL-13411 still are not fixed?
Hello @Nirjonadda, It looks like the issue in that case was solved via the changes from another case in cPanel & WHM verison 66: Fixed case CPANEL-13521: Update Apache php-fpm service manager module to be less aggressive on checks. Please open a new thread if you continue to experience issues with this service. Thank you.0
Please sign in to leave a comment.
Comments
30 comments