chkservd is not adequately restarting apache

Erik Knepfler

• July 20, 2016 12:29

httpd went down for nearly 30 minutes today, need advice: /var/log/chkservd.log ...httpd [Timeout while trying to connect to service: Died][check command:N/A][socket connect:-][socket failure threshold:6/2][fail count:5]Restarting httpd.... [notify:failed service:httpd]]... This message just kept repeating on every chkservd cycle. Note the 6/2 failure threshold /usr/local/apache/logs/error_log [mpm_prefork:notice] [pid 21770] AH00171: Graceful restart requested, doing restart This graceful restart request was repeating in the log and not really accomplishing much it seems. When I noticed, I went into WHM and restarted HTTPD via the Restart Services, which appears to use a stronger restart method. This resulted in many of these in the apache error_log: [Wed Jul 20 09:48:57.695425 2016] [core:warn] [pid 21770] AH00045: child process 20673 still did not exit, sending a SIGTERM After that, everything was fine. This makes me feel that I'd like to reconfigure chkservd to use a stronger restart method, as Graceful doesn't appear to be cutting it in some situations. Thoughts?

• 

  cPanelMichael

  • July 20, 2016 18:15

  Hello, Internal case CPANEL-1443 is open to address concerns that "restartsrv" doesn't clean up or provide a consistent environment before starting services such as Apache. In particular, in notes the difference between graceful restarts with Apache and using a full stop/start command. There's currently no specific time frame on a resolution, however I've added a link to this thread to the case, and will update this thread with any new information as it becomes available. Thank you.

  0
• 

  Erik Knepfler

  • July 20, 2016 18:27

  Perhaps I could simply replace /scripts/restartsrv_httpd with a text file with contents of apachectl -k restart instead of having it be a symlink to restartsrv_base which is binary and I can't edit? I'm afraid to do that since I don't know what else restartsrv_base might be doing.

  0
• 

  cPanelMichael

  • July 26, 2016 16:52

  Perhaps I could simply replace /scripts/restartsrv_httpd with a text file with contents of apachectl -k restart instead of having it be a symlink to restartsrv_base which is binary and I can't edit? I'm afraid to do that since I don't know what else restartsrv_base might be doing.

  
  I don't suggest manually replacing this file with a symbolic link to another location. Here are the commands that Chkservd uses to restart Apache:
  # cat /etc/chkserv.d/httpd service[httpd]=80,GET / HTTP/1.0,HTTP/1..,killall -TERM httpd;sleep 2;killall -9 httpd;/etc/rc.d/init.d/httpd stop;/etc/rc.d/init.d/httpd startssl;/usr/local/apache/bin/apachectl startssl;/etc/rc.d/init.d/httpd start;/usr/local/apache/bin/apachectl start
  You can edit this file, however you may want to first determine what's resulting in Apache failing to respond. Thank you.

  0
• 

  Erik Knepfler

  • July 29, 2016 06:28

  I believe it's an OOM condition that's not easy to figure out. I'm working on that. But when chkservd is stuck in that loop, unable to restart Apache, why does clicking Restart Services > HTTP Server (Apache) work fine to fix it immediately? The two restart scripts must be doing it differently. That link in WHM seems to run something called /scripts/reshttpd (according to the URL) but I cannot find this script to compare them and maybe copy parts of how that works over to /etc/chkdserv.d/httpd - can you point me in the right direction?

  0
• 

  cPanelMichael

  • August 02, 2016 19:20

  Could you verify if you are using EasyApache 3, or EasyApache 4? There's some additional information in the following thread if you plan to use EasyApache 4 (cPanel 58 is now in the "Release" build tier): cPanel version change or /scripts/upcp --force changes /etc/init.d/httd in EA4 Thank you.

  0
• 

  Erik Knepfler

  • August 04, 2016 06:18

  I'm using EasyApache3 because my CentOS hasn't been updated quite yet uname -a: Linux servernamehere 2.6.18-408.el5 #1 SMP Tue Jan 19 09:14:52 EST 2016 x86_64 x86_64 x86_64 GNU/Linux rpm --query centos-release: centos-release-5-11.el5.centos

  0
• 

  cPanelMichael

  • August 07, 2016 21:56

  Would you mind opening a support ticket using the link in my signature so we can take a closer look and reproduce the issue on your system? You can post the ticket number here so we can update this thread with the outcome. Thank you.

  0

Please sign in to leave a comment.