Skip to main content

[EA-9845 ] - WHM turned off PHP-FPM overnight for all domains, brought them ALL down, "File is writeable by group"

Adam Reece | WebBox
Overnight all of our WHM servers (and a few 3rd party ones we manage) decided that all of their websites will suddenly return HTTP 500. Inspecting the Apache error log at "/usr/local/apache/logs/error_log" I found huge volumes of this: SoftException in Application.cpp:267: File "/home/example/public_html/app.php" is writeable by group, referer: https://www.example.com/
I immediately checked the PHP files, which were either 775 or 664 as usual, however noticed that PHP-FPM had been switched off for every domain. The quick answer to this was to turn on PHP-FPM again by: Software > MultiPHP Manager > PHP-FPM > Enable On All Domains Question: Why did PHP-FPM turn itself off overnight for all sites without warning? Interestingly this did not impact two of our WHM servers of which also use CloudLinux. Only the pure CentOS 7 installations (about 10 of them) were impacted.

Comments

28 comments

Date Votes
  • Martin Hinrichsen
    I can confirm.. All sites down this morning cause: All accounts FPM where switched off.
    0
  • infiniteeye
    Same issue today across multiple servers. thanks for posting this thread. cPanel - sorry but this is simply not good enough
    0
  • Matt.R
    Same here, PHP-FPM seems have disabled for all sites (60) on the server at some point during the night (GMT). Also on Centos 7 (cPanel v96.0.9) We do backups during the night with JetBackup, just checking times to see if any correlation.
    0
  • PMD
    Same. Thank you for this thread as I partially got a site working by altering permissions before I found this and was scratching my head for everything else. The backups don't seem correlated. From my looking at various logs and the cpanel updates - the moment apache restarted after the cpanel update is when everything started throwing 500 errors.
    0
  • Matt.R
    [quote] The backups don't seem correlated.
    After checking further I agree.
    0
  • anton_latvia
    I can confirm the same. Even though in my opinion correct permissions are 755 and 644 - still - several servers got this problem that has caused a lot of stress. Turning all to PHP-FPM solved the problem. But I am sure - most of sites on these affected servers had PHP-FPM enabled, but today everything was off. We can not have it like this! There was an update, but update should never change anything in an unappropriate way, so that sites stop working. We are not hosting some dog-and-cat sites - people run businesses, they pay and expect some service - while we pay and expect - that servers will be working. When such things happens, especially without any actions from our side - how does this look like? This has already caused financial loss for customers and us! + even servers with old Centos6 were affected.
    0
  • rs200
    Same for me, unfortunately turning again on PHP-FPM didn't resolve immediately the issue, because i use domain.tld.php-fpm.yaml files inside /var/cpanel/userdata directory for change the root directory of my clients' applications. Fortunately i had few files to modify and the applications are all up again!
    0
  • Adam Reece | WebBox
    We can not have it like this! There was an update, but update should never change anything in an unappropriate way, so that sites stop working. We are not hosting some dog-and-cat sites - people run businesses, they pay and expect some service - while we pay and expect - that servers will be working. When such things happens, especially without any actions from our side - how does this look like? This has already caused financial loss for customers and us!

    I concur. This is the third gigantic screw up on cPanel's part we've experienced this year. We pay our monthly license fees very consistently but don't get any compensation back.
    0
  • Matt.R
    Suspect there will a substantial number adding to this thread as the US wakes up this morning....
    0
  • Qadisha
    Same here, PHP-FPM seems have disabled for all sites (60) on the server at some point during the night (GMT). Also on Centos 7 (cPanel v96.0.9) We do backups during the night with JetBackup, just checking times to see if any correlation.

    Same here and it seems the backup are not affected. As per our findings, everything started around 20.45 CEST. Have you any evidence about timing or process triggering this? The lastlog show something around 0030 CEST, but i'm not sure is related. wp-toolkit Thu Jun 10 00:36:52 +0200 2021 Thanks
    0
  • Reado
    Any word from cPanel yet as to what caused this? One of my servers had PHP-FPM disabled, whereas the other server says this for all domains: "You must install the PHP-FPM package in order to use PHP-FPM." What has cPanel done to EasyApache to cause it to either disable PHP-FPM or remove it entirely?!...
    0
  • Reado
    Any word from cPanel yet as to what caused this? One of my servers had PHP-FPM disabled, whereas the other server says this for all domains: "You must install the PHP-FPM package in order to use PHP-FPM." What has cPanel done to EasyApache to cause it to either disable PHP-FPM or remove it entirely?!...

    OK after further investigation it turns out EA4 listed PHP-FPM has "installed" despite MultiPHP Manager claiming otherwise. Restarting the PHP-FPM service allowed me to enable PHP-FPM on all domains. The server that had the "you must install" message did not show any HTTP 500 messages - all websites were actually running fine. It's a shame that this server was not customer facing and that the server that did displaying HTTP 500 was indeed customer facing! But hey, it's resolved for us now, though I would like an explanation from cPanel as to what caused this in the first place.
    0
  • JoeHabit
    All of our users php fpm was also disabled. It seems to me that the latest commit to the 100-phpfpm_cleanup.pl file could be related to this problem:
    0
  • Stephen Evans
    We've also had the same issue on three server so far this morning and are actively checking others. All CentOS 7. Once was actually pretty unstable with high CPU usage. Apache and PHP-FPM Service restarts were not enough and the server had to be rebooted. Stephen
    0
  • JoeHabit
    Also from the updatelog file: [2021-06-10 03:58:16 +0200] [/usr/local/cpanel/scripts/rpmup] [Thu Jun 10 03:58:10 2021] [100-phpfpm_cleanup] Cleaning up PHP-FPM configs for version 71 since we are removing the package. [2021-06-10 03:58:16 +0200] [/usr/local/cpanel/scripts/rpmup] [Thu Jun 10 03:58:12 2021] [100-phpfpm_cleanup] All domains that were on ea-php71-php-fpm should be back to system defaults. [2021-06-10 03:58:16 +0200] [/usr/local/cpanel/scripts/rpmup] [Thu Jun 10 03:58:12 2021] [100-phpfpm_cleanup] Cleaning up PHP-FPM configs for version 72 since we are removing the package. [2021-06-10 03:58:16 +0200] [/usr/local/cpanel/scripts/rpmup] [Thu Jun 10 03:58:13 2021] [100-phpfpm_cleanup] All domains that were on ea-php72-php-fpm should be back to system defaults. [2021-06-10 03:58:16 +0200] [/usr/local/cpanel/scripts/rpmup] [Thu Jun 10 03:58:13 2021] [100-phpfpm_cleanup] Cleaning up PHP-FPM configs for version 56 since we are removing the package. [2021-06-10 03:58:16 +0200] [/usr/local/cpanel/scripts/rpmup] [Thu Jun 10 03:58:14 2021] [100-phpfpm_cleanup] All domains that were on ea-php56-php-fpm should be back to system defaults. [2021-06-10 03:58:16 +0200] [/usr/local/cpanel/scripts/rpmup] [Thu Jun 10 03:58:14 2021] [100-phpfpm_cleanup] Cleaning up PHP-FPM configs for version 70 since we are removing the package. [2021-06-10 03:58:16 +0200] [/usr/local/cpanel/scripts/rpmup] [Thu Jun 10 03:58:14 2021] [100-phpfpm_cleanup] No domains were on ea-php70-php-fpm. [2021-06-10 03:58:16 +0200] [/usr/local/cpanel/scripts/rpmup] [Thu Jun 10 03:58:15 2021] [100-phpfpm_cleanup] Cleaning up PHP-FPM configs for version 74 since we are removing the package. [2021-06-10 03:58:16 +0200] [/usr/local/cpanel/scripts/rpmup] [Thu Jun 10 03:58:15 2021] [100-phpfpm_cleanup] All domains that were on ea-php74-php-fpm should be back to system defaults. [2021-06-10 03:58:16 +0200] [/usr/local/cpanel/scripts/rpmup] [Thu Jun 10 03:58:15 2021] [100-phpfpm_cleanup] Restarting Apache
    0
  • MikeDVB
    The cPanel QA we know and love. Now with more downtime-inducing bugs as of VC ownership and massive price increases. Maybe CentOS is the minority these days compared to CloudLinux and they forgot to test on base CentOS? Phew. Thank the lord we are paying so much more than we were last year.
    0
  • masterross
    I have a similar issue. All my site were down and the log: [Thu Jun 10 10:20:40.736751 2021] [:error] [pid 19791:tid 47292916741888] [client 66.249.66.19:50671] SoftException in Application.cpp:648: Directory /home is not owned by ceramic1 [Thu Jun 10 10:20:40.736852 2021] [core:error] [pid 19791:tid 47292916741888] [client 66.249.66.19:50671] End of script output before headers: index.php [Thu Jun 10 10:20:40.783966 2021] [:error] [pid 19991:tid 47292800931584] [client 46.148.79.91:36656] SoftException in Application.cpp:648: Directory /home is not owned by mobilneforum [Thu Jun 10 10:20:40.784072 2021] [core:error] [pid 19991:tid 47292800931584] [client 46.148.79.91:36656] End of script output before headers: mobiquo.php [Thu Jun 10 10:20:40.787667 2021] [:error] [pid 19991:tid 47292805134080] [client 46.148.79.91:36656] SoftException in Application.cpp:648: Directory /home is not owned by mobilneforum [Thu Jun 10 10:20:40.787736 2021] [core:error] [pid 19991:tid 47292805134080] [client 46.148.79.91:36656] End of script output before headers: mobiquo.php [Thu Jun 10 10:20:41.337658 2021] [:error] [pid 19991:tid 47292805134080] [client 46.148.79.91:36656] SoftException in Application.cpp:648: Directory /home is not owned by mobilneforum [Thu Jun 10 10:20:41.337741 2021] [core:error] [pid 19991:tid 47292805134080] [client 46.148.79.91:36656] End of script output before headers: mobiquo.php [Thu Jun 10 10:20:41.361356 2021] [:error] [pid 19991:tid 47292805134080] [client 157.55.39.8:63232] SoftException in Application.cpp:648: Directory /home is not owned by mobimart [Thu Jun 10 10:20:41.361428 2021] [core:error] [pid 19991:tid 47292805134080] [client 157.55.39.8:63232] End of script output before headers: showthread.php [Thu Jun 10 10:23:35.497504 2021] [core:error] [pid 19991:tid 47292800931584] [client 185.167.35.105:42928] End of script output before headers: showthread.php
    So I rebuild easy apache with default settings and all was OK except for sites with PHP 5.6 - 7.1. They continue to give error: [Thu Jun 10 16:05:29.009167 2021] [core:error] [pid 26557:tid 47017672058624] [client 157.55.39.77:11328] End of script output before headers: index.php [Thu Jun 10 16:05:31.024846 2021] [:error] [pid 26468:tid 47017751488256] [client 157.55.39.98:56768] SoftException in Application.cpp:648: Directory /home is not owned by linkdecode [Thu Jun 10 16:05:31.024930 2021] [core:error] [pid 26468:tid 47017751488256] [client 157.55.39.98:56768] End of script output before headers: index.php [Thu Jun 10 16:05:31.037466 2021] [:error] [pid 26468:tid 47017751488256] [client 157.55.39.98:56768] SoftException in Application.cpp:648: Directory /home is not owned by linkdecode [Thu Jun 10 16:05:31.037676 2021] [core:error] [pid 26468:tid 47017751488256] [client 157.55.39.98:56768] End of script output before headers: index.php
    Any idea?
    0
  • kennysamuerto
    We have encountered some servers with this problem. This is not the way, cPanel. We can't waste days of work fixing things that break the panel. Each time, the product is more expensive, and offers lower quality.
    0
  • anton_latvia
    Phew. Thank the lord we are paying so much more than we were last year.

    Yes, indeed! We are happy too. But - I wanted to mention CloudLinux / Centos. My personal experience shows that sites on Centos are faster. That resource management, that comes with CloudLinux does add quite a bit of delays. But we love extra fee, though.
    0
  • Matt.R
    [quote] But we love extra fee, though.
    +1 Reassurring to know it's gone towards support and a better product.... :confused:
    0
  • MikeDVB
    The additional funding has certainly gone to ROI and profit for the VC that purchased cPanel. cPanel QA has always left a lot to be desired. This is a fairly large oversight.
    0
  • kennysamuerto
    cPanel has acknowledged the problem, they have opened a case, and there is a workaround.
    0
  • cPJustinD
    Hello everybody! We've filed an internal case on the error, EA-9845, for further review by our development team. There was an issue within one of the packages that were updated with our latest update to EasyApache. We've recently removed some affected packages from mirrors while the issue is being further reviewed to fix the RPM. We've also published an article that covers the case in more detail here:
    0
  • keat63
    I'm not even sure that I had PHP_FPM installed, so it didn't affect me. However, I just got this from my server data centre. PHP-FPM is Disabled after EasyApache Update Why was PHP-FPM disabled for all my domains? As of June 9th 2021, an EasyApache update to ea-apache24-config-1.0-171 has inadvertently disabled PHP-FPM on cPanel/WHM servers. The following components of WHM have been identified to have been affected. ea-apache24-config-runtime-1.0-171.172.2.cpanel.noarch ea-apache24-config-1.0-171.172.2.cpanel.noarch This has also caused custom PHP-FPM configurations to be removed. When will this be resolved? As of yet, no automatic fix has been pushed out by cPanel. However UKFast have successfully tested an interim fix that restores functionality to websites on affected cPanel servers. How to re-enable PHP-FPM for a single domain
    • Log into WHM.
    • Navigate to MultiPHP Manager..
    • In the bottom section, use the tab User Domain Settings, use the search bar to search for your domain.
    • To the far right of your domain, click the toggle icon to enable PHP-FPM.
    How to re-enable PHP-FPM for all domains
    • Log into WHM.
    • Navigate to MultiPHP Manager.
    • In the bottom section, within PHP-FPM select the button Enable On All Domains
    How to re-enable PHP-FPM & restore custom configurations This section requires you to connect to your server over SSH. Please see our guide on
  • Make sure the file is executable by running: chmod +x fix.pl
    • Run the following loop:
    find /var/cpanel/userdata -type f -iname '*fpm.cache' | while read file; do ./fix.pl ${file} > $(echo ${file} | sed 's/cache/yaml/'); done
    • Finally, rebuild all of the PHP-FPM configurations:
    for i in $(cat /etc/userdomains | awk '{print $2}'); do echo "$i"; /scripts/php_fpm_config --rebuild $i; done
    • 0
  • JoeHabit
    I think the correct version numbers in the article should be these: ea-apache24-config-runtime-1.0-171.172.13.cpanel.noarch ea-apache24-config-1.0-171.172.13.cpanel.noarch Also, it would be interesting to know whether the mentioned fix is from UKFast or CP, they looks very similar. If it's from UKFast, then CP should at least credit them in the article.
    0
  • customtacos
    Boy, between this and the recent AutoSSL error, I sure am tired of cPanel updates screwing up my servers. I sure am glad we pay increased licensing fees for such terrible QA.
    0
  • kefah
    Same Isue, any solution to prevent it not to happen again ????!!
    0
  • cPJustinD
    kefah This case has been resolved, and updates have been applied to correct these changes. You should no longer be experiencing issues. If you are still experiencing issues, please open a support ticket so that our analysts can take a closer look. You can open a support ticket using the "Submit a ticket" link in my signature. Thanks!
    0

Please sign in to leave a comment.

Didn't find what you were looking for?

New post