Chkservd thinks PHP-FPM is down

locador

• February 25, 2019 10:16

Hello, I'm going crazy over tailwatchd. It thinks that PHP-FPM is not running, but it is, and it keeps sending notifications constantly. I do have a small modification but I don't believe this should interfere with how the tailwatchd check the services. I'm using a custom compiled "php-fpm" and "php-cgi" that I've replaced within the usual cpanel/easyapache installation dir. The main reason is that some features that my customers requires are not supported by EasyApache. So, my only option is to compile a custom PHP-CGI/FPM and replace the binaries within the original installation dir. And everything was working just fine for years until some days ago, when tailwatchd started to send tons of messages telling that "apache_php_fpm" is down. But it's not, they are running fine and websites are working normally. Here's some useful information: - As you can see, the services are running and websites are functional: root@hexer [/tmp]# ps axf | grep -i fpm | grep -i master 9818 ? Ss 0:01 php-fpm: master process (/usr/local/cpanel/etc/php-fpm.conf) 28856 ? Ss 0:00 php-fpm: master process (/opt/php56/etc/php-fpm.conf) 28861 ? Ss 0:01 php-fpm: master process (/opt/php54/etc/php-fpm.conf)
- Systemctl reports services are up and running: root@hexer [/tmp]# /sbin/service ea-php56-php-fpm status Redirecting to /bin/systemctl status ea-php56-php-fpm.service ? ea-php56-php-fpm.service - The PHP FastCGI Process Manager 5.6 Loaded: loaded (/usr/lib/systemd/system/ea-php56-php-fpm.service; enabled; vendor preset: disabled) Active: active (running) since Mon 2019-02-25 11:38:39 -03; 1h 10min ago Main PID: 28856 (php-fpm) CGroup: /system.slice/ea-php56-php-fpm.service ??28856 php-fpm: master process (/opt/php56/etc/php-fpm.conf) Feb 25 11:38:39 [***HIDEN***] systemd[1]: Started The PHP FastCGI Process Manager 5.6. Feb 25 11:38:39 [***HIDEN***] php-fpm[28856]: Zend Guard Loader requires Zend Engine API version 220100525. Feb 25 11:38:39 [***HIDEN***] php-fpm[28856]: The Zend Engine API version 220131226 which is installed, is newer. Feb 25 11:38:39 [***HIDEN***] php-fpm[28856]: Contact Zend Technologies at http://www.zend.com/ for a later version of Zend Guard Loader. Feb 25 11:38:39 [***HIDEN***] php-fpm[28856]: Zend OPcache requires Zend Engine API version 220100525. Feb 25 11:38:39 [***HIDEN***] php-fpm[28856]: The Zend Engine API version 220131226 which is installed, is newer. Feb 25 11:38:39 [***HIDEN***] php-fpm[28856]: Contact Zend Technologies at http://www.zend.com/ for a later version of Zend OPcache.
root@hexer [/tmp]# /sbin/service ea-php56-php-fpm status Redirecting to /bin/systemctl status ea-php56-php-fpm.service ? ea-php56-php-fpm.service - The PHP FastCGI Process Manager 5.6 Loaded: loaded (/usr/lib/systemd/system/ea-php56-php-fpm.service; enabled; vendor preset: disabled) Active: active (running) since Mon 2019-02-25 11:38:39 -03; 1h 10min ago Main PID: 28856 (php-fpm) CGroup: /system.slice/ea-php56-php-fpm.service ??28856 php-fpm: master process (/opt/php56/etc/php-fpm.conf) Feb 25 11:38:39 [***HIDEN***][1]: Started The PHP FastCGI Process Manager 5.6. Feb 25 11:38:39 [***HIDEN***] php-fpm[28856]: Zend Guard Loader requires Zend Engine API version 220100525. Feb 25 11:38:39 [***HIDEN***] php-fpm[28856]: The Zend Engine API version 220131226 which is installed, is newer. Feb 25 11:38:39 [***HIDEN***] php-fpm[28856]: Contact Zend Technologies at http://www.zend.com/ for a later version of Zend Guard Loader. Feb 25 11:38:39 [***HIDEN***] php-fpm[28856]: Zend OPcache requires Zend Engine API version 220100525. Feb 25 11:38:39 [***HIDEN***] php-fpm[28856]: The Zend Engine API version 220131226 which is installed, is newer. Feb 25 11:38:39 [***HIDEN***] php-fpm[28856]: Contact Zend Technologies at http://www.zend.com/ for a later version of Zend OPcache. root@hexer [/tmp]# /sbin/service ea-php54-php-fpm status Redirecting to /bin/systemctl status ea-php54-php-fpm.service ? ea-php54-php-fpm.service - The PHP FastCGI Process Manager Loaded: loaded (/usr/lib/systemd/system/ea-php54-php-fpm.service; enabled; vendor preset: disabled) Active: active (running) since Mon 2019-02-25 11:38:39 -03; 1h 11min ago Main PID: 28861 (php-fpm) CGroup: /system.slice/ea-php54-php-fpm.service ??25124 php-fpm: pool [***HIDEN***] ??25316 php-fpm: pool [***HIDEN***] ??28861 php-fpm: master process (/opt/php54/etc/php-fpm.conf) Feb 25 11:38:39 [***HIDEN***] systemd[1]: Started The PHP FastCGI Process Manager.
- But restartsrv_apache_php_fpm (invoked by tailwatchd to check the service) thinks it's disabled: root@hexer [/tmp]# /usr/local/cpanel/scripts/restartsrv_apache_php_fpm --check (XID mc2hd4) O servi"o "Apache PHP-FPM 56" est" desativado. The "Apache PHP-FPM 56" service is down. (XID sqp8uk) O servi"o "Apache PHP-FPM 54" est" desativado. The "Apache PHP-FPM 54" service is down.
- And as I don't have a clue about how this compiled script works, I've tried debugging it with strace to try to see what it does: strace -ff -o trace.txt /usr/local/cpanel/scripts/restartsrv_apache_php_fpm --check
- And I can see traces that it reads or stats all the configuration files and also checks the running process. root@hexer [/tmp]# grep fpm trace.txt.27471 execve("/usr/local/cpanel/scripts/restartsrv_apache_php_fpm", ["/usr/local/cpanel/scripts/restar"..., "--check">, [/* 26 vars */]) = 0 stat("/var/cpanel/perl/Cpanel/ServiceManager/Services/Apache_php_fpm.pm", 0x7fff5e54cd18) = -1 ENOENT (No such file or directory) stat("/usr/local/cpanel/Cpanel/ServiceManager/Services/Apache_php_fpm.pm", {st_mode=S_IFREG|0644, st_size=5171, ...}) = 0 open("/usr/local/cpanel/Cpanel/ServiceManager/Services/Apache_php_fpm.pm", O_RDONLY|O_CLOEXEC) = 3 open("/var/run/restartsrv_apache_php_fpm.lock-3bede76ae919-20bddac5f-4440", O_WRONLY|O_CREAT|O_EXCL|O_NONBLOCK|O_CLOEXEC, 0644) = 3 link("/var/run/restartsrv_apache_php_fpm.lock-3bede76ae919-20bddac5f-4440", "/var/run/restartsrv_apache_php_fpm.lock") = 0 lstat("/var/run/restartsrv_apache_php_fpm.lock-3bede76ae919-20bddac5f-4440", {st_mode=S_IFREG|0644, st_size=58, ...}) = 0 unlink("/var/run/restartsrv_apache_php_fpm.lock-3bede76ae919-20bddac5f-4440") = 0 stat("/var/run/restartsrv_apache_php_fpm.lock", {st_mode=S_IFREG|0644, st_size=58, ...}) = 0 open("/var/run/restartsrv_apache_php_fpm", O_WRONLY|O_CREAT|O_APPEND|O_CLOEXEC, 0666) = 4 stat("/opt/cpanel/ea-php55/root/etc/php-fpm.conf", {st_mode=S_IFREG|0600, st_size=561, ...}) = 0 stat("/opt/cpanel/ea-php55/root/usr/sbin/php-fpm", {st_mode=S_IFREG|0755, st_size=4429360, ...}) = 0 stat("/opt/cpanel/ea-php56/root/etc/php-fpm.conf", {st_mode=S_IFREG|0600, st_size=561, ...}) = 0 stat("/opt/cpanel/ea-php56/root/usr/sbin/php-fpm", {st_mode=S_IFREG|0755, st_size=15447136, ...}) = 0 stat("/opt/cpanel/ea-php70/root/etc/php-fpm.conf", {st_mode=S_IFREG|0600, st_size=561, ...}) = 0 stat("/opt/cpanel/ea-php70/root/usr/sbin/php-fpm", {st_mode=S_IFREG|0755, st_size=4244848, ...}) = 0 stat("/opt/cpanel/ea-php71/root/etc/php-fpm.conf", {st_mode=S_IFREG|0600, st_size=561, ...}) = 0 stat("/opt/cpanel/ea-php71/root/usr/sbin/php-fpm", {st_mode=S_IFREG|0755, st_size=4405128, ...}) = 0 stat("/opt/cpanel/ea-php54/root/etc/php-fpm.conf", {st_mode=S_IFREG|0600, st_size=561, ...}) = 0 stat("/opt/cpanel/ea-php54/root/usr/sbin/php-fpm", {st_mode=S_IFREG|0755, st_size=16455640, ...}) = 0 read(6, "php-fpm: master process (/usr/lo"..., 262144) = 76 read(6, "9818 (cpanel_php_fpm) S 1 9818 9"..., 262144) = 351 read(6, "php-fpm: pool cpanelroundcube\0\0\0"..., 262144) = 76 read(6, "25345 (cpanel_php_fpm) S 9818 98"..., 262144) = 345 read(6, "php-fpm: pool cpanelroundcube\0\0\0"..., 262144) = 76 read(6, "26871 (cpanel_php_fpm) S 9818 98"..., 262144) = 344 read(6, "php-fpm: pool cpanelroundcube\0\0\0"..., 262144) = 76 read(6, "26887 (cpanel_php_fpm) S 9818 98"..., 262144) = 344 read(6, "php-fpm: pool cpanelroundcube\0\0\0"..., 262144) = 76 read(6, "26891 (cpanel_php_fpm) S 9818 98"..., 262144) = 344 read(6, "php-fpm: pool cpanelroundcube\0\0\0"..., 262144) = 76 read(6, "26898 (cpanel_php_fpm) S 9818 98"..., 262144) = 344 read(6, "php-fpm: pool cpanelroundcube\0\0\0"..., 262144) = 76 read(6, "26934 (cpanel_php_fpm) S 9818 98"..., 262144) = 344 read(6, "php-fpm: pool cpanelroundcube\0\0\0"..., 262144) = 76 read(6, "27145 (cpanel_php_fpm) S 9818 98"..., 262144) = 344 read(6, "php-fpm: pool [***HIDEN***]\0\0"..., 262144) = 57 read(6, "27385 (php-fpm) S 28861 28861 28"..., 262144) = 340 read(6, "php-fpm: pool [***HIDEN***]"..., 262144) = 57 read(6, "27392 (php-fpm) S 28861 28861 28"..., 262144) = 342 read(6, "php-fpm: pool [***HIDEN***]\0\0\0\0\0"..., 262144) = 57 read(6, "27433 (php-fpm) S 28861 28861 28"..., 262144) = 338 read(6, "php-fpm: pool [***HIDEN***]"..., 262144) = 57 read(6, "27435 (php-fpm) S 28861 28861 28"..., 262144) = 341 read(6, "php-fpm: pool [***HIDEN***]"..., 262144) = 57 read(6, "27439 (php-fpm) S 28861 28861 28"..., 262144) = 340 read(6, "php-fpm: pool [***HIDEN***]"..., 262144) = 57 read(6, "27444 (php-fpm) S 28861 28861 28"..., 262144) = 339 read(6, "php-fpm: master process (/opt/ph"..., 262144) = 57 read(6, "28856 (php-fpm) S 1 28856 28856 "..., 262144) = 341 read(6, "php-fpm: master process (/opt/ph"..., 262144) = 57 read(6, "28861 (php-fpm) S 1 28861 28861 "..., 262144) = 350 openat(AT_FDCWD, "/opt/cpanel/ea-php55/root/etc/php-fpm.d/", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 5 openat(AT_FDCWD, "/opt/cpanel/ea-php56/root/etc/php-fpm.d/", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 5 openat(AT_FDCWD, "/opt/cpanel/ea-php70/root/etc/php-fpm.d/", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 7 openat(AT_FDCWD, "/opt/cpanel/ea-php71/root/etc/php-fpm.d/", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 7 openat(AT_FDCWD, "/opt/cpanel/ea-php54/root/etc/php-fpm.d/", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 7 lstat("/var/run/restartsrv_apache_php_fpm.lock", {st_mode=S_IFREG|0644, st_size=58, ...}) = 0 lstat("/var/run/restartsrv_apache_php_fpm.lock", {st_mode=S_IFREG|0644, st_size=58, ...}) = 0 unlink("/var/run/restartsrv_apache_php_fpm.lock") = 0
- But still, it thinks that the services are disabled. Right now I've have deleted the chkservd files for "apache_php_fpm" to temporarily stop the false-positive notification messages. It would be easier to know exactly how exactly "restartsrv_apache_php_fpm" works... So, how does it really work? What does it check? Etc... What can I do to make sure it sees my php-fpm running? Thanks a lot.

• 

  locador

  • February 25, 2019 15:19

  OOPS! Sorry, I think I've duplicated my message ;) I usually cut and paste back the message before I click on submit to avoid getting an error from the forum and having to write it all again... So, copied twice I think...

  0
• 

  cPanelMichael

  • February 27, 2019 17:11

  Hello @locador, Can you try stopping the master process for the PHP-FPM service that's reported as down, and then starting it directly from the binary? EX: /opt/cpanel/ea-php70/root/usr/sbin/php-fpm
  Starting it from the binary should output more information (and may list error messages). Note that internal case CPANEL-19951 is open to request more intuitive error messages when Chkservd detects that one or more PHP-FPM services fail to start. Thank you.

  0
• 

  locador

  • February 27, 2019 17:24

  Hi Michael, I'm not using php70 on any websites, so this is expected: root@hexer [~]# /opt/cpanel/ea-php70/root/usr/sbin/php-fpm [27-Feb-2019 15:14:12] WARNING: Nothing matches the include pattern '/opt/cpanel/ea-php70/root/etc/php-fpm.d/*.conf' from /opt/cpanel/ea-php70/root/etc/php-fpm.conf at line 8. [27-Feb-2019 15:14:12] ERROR: No pool defined. at least one pool section must be specified in config file [27-Feb-2019 15:14:12] ERROR: failed to post process the configuration [27-Feb-2019 15:14:12] ERROR: FPM initialization failed
  However, 54 and 56 are ok, complete shell log: root@hexer [~]# /opt/cpanel/ea-php54/root/usr/sbin/php-fpm [27-Feb-2019 15:14:47] ERROR: An another FPM instance seems to already listen on /opt/cpanel/ea-php54/root/usr/var/run/php-fpm/e7bcdf970ea206fc5d2c5d09b2e18e869b44430b.sock [27-Feb-2019 15:14:47] ERROR: FPM initialization failed root@hexer [~]# pkill -f php-fpm root@hexer [~]# /opt/cpanel/ea-php54/root/usr/sbin/php-fpm [27-Feb-2019 15:15:06] NOTICE: fpm is running, pid 19519 [27-Feb-2019 15:15:06] NOTICE: ready to handle connections ^C[27-Feb-2019 15:15:10] NOTICE: Terminating ... [27-Feb-2019 15:15:10] NOTICE: exiting, bye-bye! root@hexer [~]# /opt/cpanel/ea-php56/root/usr/sbin/php-fpm Zend Guard Loader requires Zend Engine API version 220100525. The Zend Engine API version 220131226 which is installed, is newer. Contact Zend Technologies at http://www.zend.com/ for a later version of Zend Guard Loader. Zend OPcache requires Zend Engine API version 220100525. The Zend Engine API version 220131226 which is installed, is newer. Contact Zend Technologies at http://www.zend.com/ for a later version of Zend OPcache. [27-Feb-2019 15:15:14] NOTICE: fpm is running, pid 19906 [27-Feb-2019 15:15:14] NOTICE: ready to handle connections [27-Feb-2019 15:15:14] NOTICE: systemd monitor interval set to 10000ms ^C[27-Feb-2019 15:15:17] NOTICE: Terminating ... [27-Feb-2019 15:15:17] NOTICE: exiting, bye-bye!
  After that pkill, manual start for you to see the output and my CRTL-C, I've restarted both using systemctl, and they are running fine, websites working, etc: root@hexer [~]# ps axf | grep -i fpm 18984 pts/0 S+ 0:00 | \_ grep --color=auto -i fpm 9818 ? Ss 0:11 php-fpm: master process (/usr/local/cpanel/etc/php-fpm.conf) 14389 ? S 0:00 \_ php-fpm: pool cpanelroundcube 14827 ? S 0:00 \_ php-fpm: pool cpanelroundcube 28856 ? Ss 0:04 php-fpm: master process (/opt/php56/etc/php-fpm.conf) 28861 ? Ss 0:45 php-fpm: master process (/opt/php54/etc/php-fpm.conf) 18913 ? Sl 0:00 \_ php-fpm: pool [**HIDDEN**]_com_br 18918 ? S 0:00 \_ php-fpm: pool [**HIDDEN**]_com_br 18933 ? S 0:00 \_ php-fpm: pool [**HIDDEN**]_org
  Right now I'm keeping the apache_php_fpm file from chkservd deleted because the error still persists and I've written my own script to monitor this service (by accessing a phpinfo on websites and then restarting if there's any error). Thanks,

  0
• 

  cPanelMichael

  • February 27, 2019 17:43

  Hello @locador, It's possible the custom modifications you've made to PHP-FPM are resulting in this issue. Can you can open a

  0

Please sign in to leave a comment.