Skip to main content

httpd FAILED and recovered after 20-30min

Answered
Mauro Meroni

Today i had some problems with my whm server.

"httpd" failed twice today, causing 20-30min downtime each time for all my website

Here are the email the logs of one of the two occurances,
on both of them the reaons is:
Service check failed to complete
Unable to connect to port 81 on 127.0.0.1: Connection refused: Died

Does anyone know why it's failing for such a long time? Is there a way to prevent it?

Thanks!

Server  my.whm.addr
Primary IP Address  xxx.yyy.zzz.xxx
Service Name  httpd
Service Status  failed ⛔
Notification  The service “httpd” appears to be down.
Service Check Method  The system failed to connect to this service’s TCP/IP port.
Reason  Service check failed to complete
Unable to connect to port 81 on 127.0.0.1: Connection refused: Died
Number of Restart Attempts  1
Startup Log  Mar 13 13:47:27 my.whm.addr systemd[1]: Starting Apache web server managed by cPanel EasyApache...
Mar 13 13:47:27 my.whm.addr systemd[1]: httpd.service: Can't open PID file /run/apache2/httpd.pid (yet?) after start: Operation not permitted
Mar 13 13:47:27 my.whm.addr systemd[1]: Started Apache web server managed by cPanel EasyApache.
Mar 13 19:25:26 my.whm.addr systemd[1]: httpd.service: Main process exited, code=exited, status=1/FAILURE
Mar 13 19:25:27 my.whm.addr systemd[1]: httpd.service: Failed with result 'exit-code'.
Log Messages  [Thu Mar 13 19:25:27.661866 2025] [log_config:warn] [pid 2301862:tid 2301910] (32)Broken pipe: [client xxx.yyy.zzz.xxx:53030] AH00646: Error writing to |/usr/local/cpanel/bin/splitlogs --dir=/etc/apache2/logs/domlogs --main=my.whm.addr --mainout=/etc/apache2/logs/access_log --sslport=444
[Thu Mar 13 19:25:27.661843 2025] [log_config:warn] [pid 2301862:tid 2301910] (32)Broken pipe: [client xxx.yyy.zzz.xxx:53030] AH00646: Error writing to |/usr/local/cpanel/bin/splitlogs --dir=/etc/apache2/logs/domlogs --main=my.whm.addr --suffix=-bytes_log --sslport=444
httpd: Syntax error on line 3664 of /etc/apache2/conf/httpd.conf: /etc/apache2/conf/httpd.conf:4698: was not closed.\n/etc/apache2/conf/httpd.conf:3664: was not closed.
Memory Information  
Used  26.98 GB
Available  98.2 GB
Installed  125.18 GB
Load Information  0.54 0.67 1.03


Server  my.whm.addr
Primary IP Address  xxx.yyy.zzz.xxx
Service Name  httpd
Service Status  failed ⛔
Notification  The service “httpd” appears to be down.
Service Check Method  The system failed to connect to this service’s TCP/IP port.
Reason  Service check failed to complete
Unable to connect to port 81 on 127.0.0.1: Connection refused: Died
Number of Restart Attempts  2
Startup Log  Mar 13 19:38:55 my.whm.addr systemd[1]: Starting Apache web server managed by cPanel EasyApache...
Mar 13 19:38:55 my.whm.addr restartsrv_httpd[2417134]: httpd: Syntax error on line 3664 of /etc/apache2/conf/httpd.conf: /etc/apache2/conf/httpd.conf:4698: was not closed.\n/etc/apache2/conf/httpd.conf:3664: was not closed.
Mar 13 19:38:55 my.whm.addr systemd[1]: httpd.service: Control process exited, code=exited, status=1/FAILURE
Mar 13 19:38:55 my.whm.addr systemd[1]: httpd.service: Failed with result 'exit-code'.
Mar 13 19:38:55 my.whm.addr systemd[1]: Failed to start Apache web server managed by cPanel EasyApache.
Log Messages  [Thu Mar 13 19:25:27.661866 2025] [log_config:warn] [pid 2301862:tid 2301910] (32)Broken pipe: [client xxx.yyy.zzz.xxx:53030] AH00646: Error writing to |/usr/local/cpanel/bin/splitlogs --dir=/etc/apache2/logs/domlogs --main=my.whm.addr --mainout=/etc/apache2/logs/access_log --sslport=444
[Thu Mar 13 19:25:27.661843 2025] [log_config:warn] [pid 2301862:tid 2301910] (32)Broken pipe: [client xxx.yyy.zzz.xxx:53030] AH00646: Error writing to |/usr/local/cpanel/bin/splitlogs --dir=/etc/apache2/logs/domlogs --main=my.whm.addr --suffix=-bytes_log --sslport=444
httpd: Syntax error on line 3664 of /etc/apache2/conf/httpd.conf: /etc/apache2/conf/httpd.conf:4698: was not closed.\n/etc/apache2/conf/httpd.conf:3664: was not closed.
Memory Information  
Used  26.97 GB
Available  98.22 GB
Installed  125.18 GB
Load Information  0.55 0.58 0.89


Server  my.whm.addr
Primary IP Address  xxx.yyy.zzz.xxx
Service Name  httpd
Service Status  failed ⛔
Notification  The service “httpd” appears to be down.
Service Check Method  The system failed to connect to this service’s TCP/IP port.
Reason  Service check failed to complete
Unable to connect to port 81 on 127.0.0.1: Connection refused: Died
Number of Restart Attempts  3
Startup Log  Mar 13 19:43:56 my.whm.addr systemd[1]: Starting Apache web server managed by cPanel EasyApache...
Mar 13 19:43:57 my.whm.addr restartsrv_httpd[2418963]: httpd: Syntax error on line 3664 of /etc/apache2/conf/httpd.conf: /etc/apache2/conf/httpd.conf:4698: was not closed.\n/etc/apache2/conf/httpd.conf:3664: was not closed.
Mar 13 19:43:57 my.whm.addr systemd[1]: httpd.service: Control process exited, code=exited, status=1/FAILURE
Mar 13 19:43:57 my.whm.addr systemd[1]: httpd.service: Failed with result 'exit-code'.
Mar 13 19:43:57 my.whm.addr systemd[1]: Failed to start Apache web server managed by cPanel EasyApache.
Log Messages  [Thu Mar 13 19:25:27.661866 2025] [log_config:warn] [pid 2301862:tid 2301910] (32)Broken pipe: [client xxx.yyy.zzz.xxx:53030] AH00646: Error writing to |/usr/local/cpanel/bin/splitlogs --dir=/etc/apache2/logs/domlogs --main=my.whm.addr --mainout=/etc/apache2/logs/access_log --sslport=444
[Thu Mar 13 19:25:27.661843 2025] [log_config:warn] [pid 2301862:tid 2301910] (32)Broken pipe: [client xxx.yyy.zzz.xxx:53030] AH00646: Error writing to |/usr/local/cpanel/bin/splitlogs --dir=/etc/apache2/logs/domlogs --main=my.whm.addr --suffix=-bytes_log --sslport=444
httpd: Syntax error on line 3664 of /etc/apache2/conf/httpd.conf: /etc/apache2/conf/httpd.conf:4698: was not closed.\n/etc/apache2/conf/httpd.conf:3664: was not closed.
Memory Information  
Used  27.51 GB
Available  97.67 GB
Installed  125.18 GB
Load Information  0.25 0.43 0.75


Server  my.whm.addr
Primary IP Address  xxx.yyy.zzz.xxx
Service Name  httpd
Service Status  recovered ❇
Notification  The service “httpd” is now operational.
Startup Log  Mar 13 19:48:58 my.whm.addr systemd[1]: Starting Apache web server managed by cPanel EasyApache...
Mar 13 19:48:58 my.whm.addr systemd[1]: httpd.service: Can't open PID file /run/apache2/httpd.pid (yet?) after start: Operation not permitted
Mar 13 19:48:58 my.whm.addr systemd[1]: Started Apache web server managed by cPanel EasyApache.
Log Messages  [Thu Mar 13 19:48:58.770597 2025] [mpm_worker:notice] [pid 2420807:tid 2420807] AH00292: Apache/2.4.62 (cPanel) OpenSSL/3.2.2 mod_bwlimited/1.4 configured -- resuming normal operations
[Thu Mar 13 19:48:58.685812 2025] [security2:notice] [pid 2420804:tid 2420804] ModSecurity for Apache/2.9.7 (http://www.modsecurity.org/) configured.
[Thu Mar 13 19:25:27.661866 2025] [log_config:warn] [pid 2301862:tid 2301910] (32)Broken pipe: [client xxx.yyy.zzz.xxx:53030] AH00646: Error writing to |/usr/local/cpanel/bin/splitlogs --dir=/etc/apache2/logs/domlogs --main=my.whm.addr --mainout=/etc/apache2/logs/access_log --sslport=444
[Thu Mar 13 19:25:27.661843 2025] [log_config:warn] [pid 2301862:tid 2301910] (32)Broken pipe: [client xxx.yyy.zzz.xxx:53030] AH00646: Error writing to |/usr/local/cpanel/bin/splitlogs --dir=/etc/apache2/logs/domlogs --main=my.whm.addr --suffix=-bytes_log --sslport=444
httpd: Syntax error on line 3664 of /etc/apache2/conf/httpd.conf: /etc/apache2/conf/httpd.conf:4698: was not closed.\n/etc/apache2/conf/httpd.conf:3664: was not closed.
Memory Information  
Used  27.07 GB
Available  98.11 GB
Installed  125.18 GB
Load Information  1.64 1.11 0.95

Comments

7 comments

Date Votes
  • William Del Piero cPanel Staff

    Hi,

     

    It appears there is a syntax error in the Apache configuration file based on the log output. Are there any additional errors generated if you try to rebuild the Apache configuration with the following command?

     

    /scripts/rebuildhttpdconf
    0
  • Mauro Meroni

    No:
    /scripts/rebuildhttpdconf
    Built /etc/apache2/conf/httpd.conf OK

    0
  • William Del Piero cPanel Staff

    Hi,

     

    If you do see the issue occur again, usually rebuilding the Apache configuration with the /scripts/rebuildhttpdconf , then restarting Apache ( /scripts/restartsrv_httpd ) will resolve the issue. However, it is unusual to see repeated cases of corruption in the Apache configuration file. To confirm, were any manual edits being made to the Apache configuration on the server? If not, I'd recommend opening a ticket since access to the server would likely be required to determine the cause of the corruption.

    0
  • Mauro Meroni

    Hi, it's a fairly new WHM server, during the initial setup i had the configserver "cPanel Server Services" installed that i know includes "Apache tune and check" so it is possible that they made some edits to the default configs

     

    0
  • William Del Piero cPanel Staff

    Hi,

     

    I'm not familiar with the "Apache tune and check" feature from CSF, but if it makes any manual changes to the Apache configuration it's possible that it caused corruption to the Apache configuration earlier. If you still see the issue happen again with this feature disabled, I'd suggest opening a ticket so this can be reviewed further.

    0
  • stAn

    I believe this issue is caused by writing and reading of httpd.conf file at once by multiple processes within cpanel system. In my case the apache config (httpd.conf) got randomly broken when "lets encrypt certificate" was renewed. (no human action at whm or cpanel at the time of downtime)

    I am experiencing this issue on both ubuntu22+cpanel as well as cloudlinux8+cpanel, but i haven't yet seen this on almalinux 9+cpanel. 

    The issue is present in cpanel for at least last 12 months and still not fixed (i.e. to fix it, the httpd.conf should be written as httpd.conf.tmp and renamed to httpd.conf after it is finished building).

    a root's cron kind of fix could be something like: 

    apachectl configtest || /scripts/rebuildhttpdconf

    Best Regards, Stan

     

    0
  • cPRex Jurassic Moderator

    stAn - do you have a server where you can reproduce this behavior?  If so, we'd be interested in taking a look, although I understand you may not be able to leave a system in the broken state very long for an investigation.

    0

Please sign in to leave a comment.

Didn't find what you were looking for?

New post