Skip to main content

httpd FAILED and recovered after 20-30min

Answered
Mauro Meroni

Today i had some problems with my whm server.

"httpd" failed twice today, causing 20-30min downtime each time for all my website

Here are the email the logs of one of the two occurances,
on both of them the reaons is:
Service check failed to complete
Unable to connect to port 81 on 127.0.0.1: Connection refused: Died

Does anyone know why it's failing for such a long time? Is there a way to prevent it?

Thanks!

Server  my.whm.addr
Primary IP Address  xxx.yyy.zzz.xxx
Service Name  httpd
Service Status  failed ⛔
Notification  The service “httpd” appears to be down.
Service Check Method  The system failed to connect to this service’s TCP/IP port.
Reason  Service check failed to complete
Unable to connect to port 81 on 127.0.0.1: Connection refused: Died
Number of Restart Attempts  1
Startup Log  Mar 13 13:47:27 my.whm.addr systemd[1]: Starting Apache web server managed by cPanel EasyApache...
Mar 13 13:47:27 my.whm.addr systemd[1]: httpd.service: Can't open PID file /run/apache2/httpd.pid (yet?) after start: Operation not permitted
Mar 13 13:47:27 my.whm.addr systemd[1]: Started Apache web server managed by cPanel EasyApache.
Mar 13 19:25:26 my.whm.addr systemd[1]: httpd.service: Main process exited, code=exited, status=1/FAILURE
Mar 13 19:25:27 my.whm.addr systemd[1]: httpd.service: Failed with result 'exit-code'.
Log Messages  [Thu Mar 13 19:25:27.661866 2025] [log_config:warn] [pid 2301862:tid 2301910] (32)Broken pipe: [client xxx.yyy.zzz.xxx:53030] AH00646: Error writing to |/usr/local/cpanel/bin/splitlogs --dir=/etc/apache2/logs/domlogs --main=my.whm.addr --mainout=/etc/apache2/logs/access_log --sslport=444
[Thu Mar 13 19:25:27.661843 2025] [log_config:warn] [pid 2301862:tid 2301910] (32)Broken pipe: [client xxx.yyy.zzz.xxx:53030] AH00646: Error writing to |/usr/local/cpanel/bin/splitlogs --dir=/etc/apache2/logs/domlogs --main=my.whm.addr --suffix=-bytes_log --sslport=444
httpd: Syntax error on line 3664 of /etc/apache2/conf/httpd.conf: /etc/apache2/conf/httpd.conf:4698: was not closed.\n/etc/apache2/conf/httpd.conf:3664: was not closed.
Memory Information  
Used  26.98 GB
Available  98.2 GB
Installed  125.18 GB
Load Information  0.54 0.67 1.03


Server  my.whm.addr
Primary IP Address  xxx.yyy.zzz.xxx
Service Name  httpd
Service Status  failed ⛔
Notification  The service “httpd” appears to be down.
Service Check Method  The system failed to connect to this service’s TCP/IP port.
Reason  Service check failed to complete
Unable to connect to port 81 on 127.0.0.1: Connection refused: Died
Number of Restart Attempts  2
Startup Log  Mar 13 19:38:55 my.whm.addr systemd[1]: Starting Apache web server managed by cPanel EasyApache...
Mar 13 19:38:55 my.whm.addr restartsrv_httpd[2417134]: httpd: Syntax error on line 3664 of /etc/apache2/conf/httpd.conf: /etc/apache2/conf/httpd.conf:4698: was not closed.\n/etc/apache2/conf/httpd.conf:3664: was not closed.
Mar 13 19:38:55 my.whm.addr systemd[1]: httpd.service: Control process exited, code=exited, status=1/FAILURE
Mar 13 19:38:55 my.whm.addr systemd[1]: httpd.service: Failed with result 'exit-code'.
Mar 13 19:38:55 my.whm.addr systemd[1]: Failed to start Apache web server managed by cPanel EasyApache.
Log Messages  [Thu Mar 13 19:25:27.661866 2025] [log_config:warn] [pid 2301862:tid 2301910] (32)Broken pipe: [client xxx.yyy.zzz.xxx:53030] AH00646: Error writing to |/usr/local/cpanel/bin/splitlogs --dir=/etc/apache2/logs/domlogs --main=my.whm.addr --mainout=/etc/apache2/logs/access_log --sslport=444
[Thu Mar 13 19:25:27.661843 2025] [log_config:warn] [pid 2301862:tid 2301910] (32)Broken pipe: [client xxx.yyy.zzz.xxx:53030] AH00646: Error writing to |/usr/local/cpanel/bin/splitlogs --dir=/etc/apache2/logs/domlogs --main=my.whm.addr --suffix=-bytes_log --sslport=444
httpd: Syntax error on line 3664 of /etc/apache2/conf/httpd.conf: /etc/apache2/conf/httpd.conf:4698: was not closed.\n/etc/apache2/conf/httpd.conf:3664: was not closed.
Memory Information  
Used  26.97 GB
Available  98.22 GB
Installed  125.18 GB
Load Information  0.55 0.58 0.89


Server  my.whm.addr
Primary IP Address  xxx.yyy.zzz.xxx
Service Name  httpd
Service Status  failed ⛔
Notification  The service “httpd” appears to be down.
Service Check Method  The system failed to connect to this service’s TCP/IP port.
Reason  Service check failed to complete
Unable to connect to port 81 on 127.0.0.1: Connection refused: Died
Number of Restart Attempts  3
Startup Log  Mar 13 19:43:56 my.whm.addr systemd[1]: Starting Apache web server managed by cPanel EasyApache...
Mar 13 19:43:57 my.whm.addr restartsrv_httpd[2418963]: httpd: Syntax error on line 3664 of /etc/apache2/conf/httpd.conf: /etc/apache2/conf/httpd.conf:4698: was not closed.\n/etc/apache2/conf/httpd.conf:3664: was not closed.
Mar 13 19:43:57 my.whm.addr systemd[1]: httpd.service: Control process exited, code=exited, status=1/FAILURE
Mar 13 19:43:57 my.whm.addr systemd[1]: httpd.service: Failed with result 'exit-code'.
Mar 13 19:43:57 my.whm.addr systemd[1]: Failed to start Apache web server managed by cPanel EasyApache.
Log Messages  [Thu Mar 13 19:25:27.661866 2025] [log_config:warn] [pid 2301862:tid 2301910] (32)Broken pipe: [client xxx.yyy.zzz.xxx:53030] AH00646: Error writing to |/usr/local/cpanel/bin/splitlogs --dir=/etc/apache2/logs/domlogs --main=my.whm.addr --mainout=/etc/apache2/logs/access_log --sslport=444
[Thu Mar 13 19:25:27.661843 2025] [log_config:warn] [pid 2301862:tid 2301910] (32)Broken pipe: [client xxx.yyy.zzz.xxx:53030] AH00646: Error writing to |/usr/local/cpanel/bin/splitlogs --dir=/etc/apache2/logs/domlogs --main=my.whm.addr --suffix=-bytes_log --sslport=444
httpd: Syntax error on line 3664 of /etc/apache2/conf/httpd.conf: /etc/apache2/conf/httpd.conf:4698: was not closed.\n/etc/apache2/conf/httpd.conf:3664: was not closed.
Memory Information  
Used  27.51 GB
Available  97.67 GB
Installed  125.18 GB
Load Information  0.25 0.43 0.75


Server  my.whm.addr
Primary IP Address  xxx.yyy.zzz.xxx
Service Name  httpd
Service Status  recovered ❇
Notification  The service “httpd” is now operational.
Startup Log  Mar 13 19:48:58 my.whm.addr systemd[1]: Starting Apache web server managed by cPanel EasyApache...
Mar 13 19:48:58 my.whm.addr systemd[1]: httpd.service: Can't open PID file /run/apache2/httpd.pid (yet?) after start: Operation not permitted
Mar 13 19:48:58 my.whm.addr systemd[1]: Started Apache web server managed by cPanel EasyApache.
Log Messages  [Thu Mar 13 19:48:58.770597 2025] [mpm_worker:notice] [pid 2420807:tid 2420807] AH00292: Apache/2.4.62 (cPanel) OpenSSL/3.2.2 mod_bwlimited/1.4 configured -- resuming normal operations
[Thu Mar 13 19:48:58.685812 2025] [security2:notice] [pid 2420804:tid 2420804] ModSecurity for Apache/2.9.7 (http://www.modsecurity.org/) configured.
[Thu Mar 13 19:25:27.661866 2025] [log_config:warn] [pid 2301862:tid 2301910] (32)Broken pipe: [client xxx.yyy.zzz.xxx:53030] AH00646: Error writing to |/usr/local/cpanel/bin/splitlogs --dir=/etc/apache2/logs/domlogs --main=my.whm.addr --mainout=/etc/apache2/logs/access_log --sslport=444
[Thu Mar 13 19:25:27.661843 2025] [log_config:warn] [pid 2301862:tid 2301910] (32)Broken pipe: [client xxx.yyy.zzz.xxx:53030] AH00646: Error writing to |/usr/local/cpanel/bin/splitlogs --dir=/etc/apache2/logs/domlogs --main=my.whm.addr --suffix=-bytes_log --sslport=444
httpd: Syntax error on line 3664 of /etc/apache2/conf/httpd.conf: /etc/apache2/conf/httpd.conf:4698: was not closed.\n/etc/apache2/conf/httpd.conf:3664: was not closed.
Memory Information  
Used  27.07 GB
Available  98.11 GB
Installed  125.18 GB
Load Information  1.64 1.11 0.95

Comments

16 comments

Date Votes
  • William Del Piero cPanel Staff

    Hi,

     

    It appears there is a syntax error in the Apache configuration file based on the log output. Are there any additional errors generated if you try to rebuild the Apache configuration with the following command?

     

    /scripts/rebuildhttpdconf
    0
  • Mauro Meroni

    No:
    /scripts/rebuildhttpdconf
    Built /etc/apache2/conf/httpd.conf OK

    0
  • William Del Piero cPanel Staff

    Hi,

     

    If you do see the issue occur again, usually rebuilding the Apache configuration with the /scripts/rebuildhttpdconf , then restarting Apache ( /scripts/restartsrv_httpd ) will resolve the issue. However, it is unusual to see repeated cases of corruption in the Apache configuration file. To confirm, were any manual edits being made to the Apache configuration on the server? If not, I'd recommend opening a ticket since access to the server would likely be required to determine the cause of the corruption.

    0
  • Mauro Meroni

    Hi, it's a fairly new WHM server, during the initial setup i had the configserver "cPanel Server Services" installed that i know includes "Apache tune and check" so it is possible that they made some edits to the default configs

     

    0
  • William Del Piero cPanel Staff

    Hi,

     

    I'm not familiar with the "Apache tune and check" feature from CSF, but if it makes any manual changes to the Apache configuration it's possible that it caused corruption to the Apache configuration earlier. If you still see the issue happen again with this feature disabled, I'd suggest opening a ticket so this can be reviewed further.

    0
  • stAn

    I believe this issue is caused by writing and reading of httpd.conf file at once by multiple processes within cpanel system. In my case the apache config (httpd.conf) got randomly broken when "lets encrypt certificate" was renewed. (no human action at whm or cpanel at the time of downtime)

    I am experiencing this issue on both ubuntu22+cpanel as well as cloudlinux8+cpanel, but i haven't yet seen this on almalinux 9+cpanel. 

    The issue is present in cpanel for at least last 12 months and still not fixed (i.e. to fix it, the httpd.conf should be written as httpd.conf.tmp and renamed to httpd.conf after it is finished building).

    a root's cron kind of fix could be something like: 

    apachectl configtest || /scripts/rebuildhttpdconf

    Best Regards, Stan

     

    0
  • cPRex Jurassic Moderator

    stAn - do you have a server where you can reproduce this behavior?  If so, we'd be interested in taking a look, although I understand you may not be able to leave a system in the broken state very long for an investigation.

    0
  • stAn

    hello,

    this issue is becoming a large issue for us causing lot's of downtime every other day or week (randomly)

    i've traced this issue down to "letsencrypt" autossl and splitlogs - these are 2 services running at the same time that might be restarting/reloading apache while writing it's configuration. 

    i also saw: 


    systemd[1]: httpd.service: State 'stop-sigterm' timed out. Killing

     

    we have cca 160 domains with autossl and it is common for log files to have a few gigabytes (server is fully capable of this - AX162-R at hetzner)

     

     

    i am going to to overide systemd config with : 

    [Manager]
    DefaultTimeoutStopSec=120s

    at /etc/systemd/system.conf.d/stan.conf

     

    to see if it helps it when rebuilding apache config (that might take a few tens of seconds to rebuild and restart). 

     

    BUT the most important "support question" here is - how do i make sure that autossl is not running on 14:22 UTC and/or "split logs" are not running at the same time ? where do i change the timing so that it gets rememebered ? 

    @cPRex - this is a production server of my client, the issue "might" be related to: 

    - ubuntu's systemd timeouts (DefaultTimeoutStopSec ) 

    - as described above (one process writing apache config (maybe autossl) and another one trying to restart apache (maybe splitlogs)

    for the issue to reproduce i guess you'd need them to hit that particular second while writing the config files. (generally i strongly suggest not to write to httpd.conf directly but always write to something like httpd.conf.{unixtime}.tmp and then rename it with single operation, so that the file is never opened for reading and writing by 2 processes ) 

    - it would be really nice if cpanel writes it's logs to /var/log .... (or at least creates symlinks here so that it doesn't take an hour to find them all... )

    thank you for any ideas, 

    best regards, stan

     

     

     

     

     

     

     

    0
  • cPRex Jurassic Moderator

    Neither AutoSSL nor the logging services would cause a "sigkill" to be executed to Apache, so there's likely something else happening.  It would really be best to create a ticket at this point so the system can be examined directly.

    0
  • Mauro Meroni

    It happened again, multiple times, in my server in the past weeks, I will open a ticket then.

    Thanks

    0
  • Mauro Meroni

    Seems like i can't open a ticket because i bought my server and license from OVH...

    0
  • stAn

    same here - all clients run via OVH or Hetzner dedicated servers... but we finally found a solution - we are going to get rid of cpanel everywhere... since it's overpriced software without any kind of support. 

    0
  • cPRex Jurassic Moderator

    Mauro Meroni - you should be contacting your license provider for support and then if they can't fix the issue it would be escalated to us.

    stAn - this an issue with your license provider and not the cPanel support team.  We're always here to help, but we have no control over the quality of support from your provider.

    0
  • stAn

    Can you please advise how to open a ticket when the license comes via hetzner ? hetzner doesn't provide any kind of support for software, they are just hardware renting company and since this is an older client, the cpanel license is still handled with them (for new clients as far as i know they stopped providing cpanel licenses possibly due to this reason). Server is managed by me (as another 3rd party). 

     

    This is as far as i got now in matters of this issue: 

    - open ssl is configured on this server to run on 14:52UTC every day - this is when possibly all downtime happnes

    - when there is a certificate to be renewed and server is under heavy load, the reload (and possible rebuild by autosll) times out (we have 220s systemd timeout, but hundreds of alias domains)

    - since this timeout is reached for httpd reload the httpd is killed and it takes up to 20 minutes for webserver to get up again

     

    i am gong to implement: 

    - make sure that autossl is run purely during business days since it should renew certificates a few days before expiry and thus skipping weekends should be all ok for this

    - after it's run i'll need to run some tests to see if httpd is up and running and serving datas

    - if not, then we'd run rebuild config which fixes this problem together with hard-reset of httpd

     

    now the question is if i modify: /etc/cron.d/cpanel_autossl

    to use my own wrapper script - will this survive cpanel update ? 

     

    thanks, best regards, stan

     

     

     

     

    0
  • ITHKBO
    • Edited

    You should just be able to open your ticket from the Hetzner client portal. If you still have a valid cPanel license from them, they are obligated to serve as the contact point for any questions instead of cPanel directly. They can't simply say they do not provide support for existing licenses.

    My recommendation without recommending a hosting provider directly (which is not allowed here) is that if you are not satisfied with the support channel of your current provider, and you do want to continue using cPanel, which Hetzner will indeed not renew, either buy a direct license or find a provider that offers managed hosting with shared support responsibility by giving access to the support access key of the servers.

    With the support access key, you can independently of your provider create tickets with cPanel directly while still having them as an extra support option in situations where their help is better suited, for example, when it involves information from the hypervisor layer if you rent VPS space, or it is something that happens over multiple servers. You just want to make sure you communicate with them beforehand about who is going to deal with any situation. The last thing you want is duplicate support tickets. If done correctly, shared responsibility can save a lot of time and thus money.

    As for OpenSSL causing downtime for 20 minutes on httpd that is absolutely not normal behavior. Renewal and cert install should restart httpd, but the process should not drop connections, only temporarily interrupt them, and the httpd service should be up in 30 seconds or less if you do not cause a spike that maxes out your CPU. We have about 300 domains on a production server, and we run renewals whenever we find it necessary, even during business hours. We never had a client call because of downtime while doing this. Your hardware AX162-R at hetzner is also higher than ours, so hardware capability should also not be an issue if not oversold too much or at all.

    As for modifying /etc/cron.d/cpanel_autossl it should survive updates.
    Here is the official topic with the procedure; there is no mention of loss after an update.
    https://support.cpanel.net/hc/en-us/articles/6058593251351-How-to-change-the-time-AutoSSL-runs
    Though I can't comment on how it will handle additional scripting, we never had the need for that.

    We use this on our most busy production.
    37      0,3,6,9,12,15,18,21 *       *       *       root    /usr/local/cPanel/bin/autossl_check --all

    0
  • stAn

    Hello, thank you for your feedback, but "i am the hosting" here (i.e. little company taking care of server administration of client's rented hardware) and hetzner is just hardware provider and cpanel license provider, cpanel license directly has another disadvantages (multiple invoices, lack of european payment methods and just another service to be tracked - so it is much more comfortable for clients to order the cpanel as a bundle with another service). but i guess this won't be possible any longer as hetzner does not provide cpanel licenses anylonger since january 25 (https://docs.hetzner.com/robot/dedicated-server/operating-systems/cpanel/). Client runs just 3 production websites on the server (and hundreds of domain aliases). We don't use emails or dns of the cpanel server. 

     

    This is what i did for now to lower the probability of this issue, maybe it helps somebody: 

    \etc\cron.d\cpanel_autossl

    #commented original line, we will see if this gets ovewritten by cpanel update:

    ```

    #52 14  *   *   *   root    /usr/local/cpanel/bin/autossl_check --all
    52  14  *   *   *   root    /root/stan/cpanel_autossl_wrap.sh
    ```
     
    \root\stan\cpanel_autossl_wrap.sh
    ```
    #!/bin/bash
    # Get the day of the week (0=Sunday, 6=Saturday)
    day_of_week=$(date +%w)

    if [ "$day_of_week" -eq 0 ] || [ "$day_of_week" -eq 6 ]; then
      echo "It's the weekend — exiting."
      exit 0
    fi

    echo "It's a weekday continuing script..."

    /usr/local/cpanel/bin/autossl_check --all

    # Run the first script
    /bin/bash /root/stan/check_site.sh
    RETURN_CODE=$?

    # Check if the return code is exactly 1
    if [ "$RETURN_CODE" -eq 1 ]; then
        # Run the second script
        /bin/bash /etc/apache2/rebuild.sh
    fi
    ```
    \root\stan\check_site.sh
    ```
    #!/bin/bash

    KEYWORD="some-keyword-always-on-the-site"
    USER_AGENT="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/137.0.0.0 Safari/537.36 MonitorBot/1.0"
    REBUILD_SCRIPT="/etc/apache2/rebuild.sh"

    # Fetch headers and body
    response=$(curl -s -A "$USER_AGENT" -w "\n%{http_code}" "$URL")

    # Split into content and status
    body=$(echo "$response" | sed '$d')
    status=$(echo "$response" | tail -n1)

    # Debug output (optional)
    echo "HTTP Status: $status"
    echo "Checking for keyword: $KEYWORD"

    # Check conditions
    if [ "$status" -ne 200 ] || ! echo "$body" | grep -q "$KEYWORD"; then
      echo "Page check failed - status=$status or keyword not found. Running rebuild script..."
      #bash "$REBUILD_SCRIPT"
      exit 1
    else
      echo "Page is healthy."
    fi

    exit 0

    ```

    and rebuild.sh
    ```

    /scripts/php_fpm_config --rebuild
    /scripts/rebuildhttpdconf
    /scripts/restartsrv_apache
    /scripts/restartsrv_apache_php_fpm
    ```

     

     

     

     

     

     

    0

Please sign in to leave a comment.

Didn't find what you were looking for?

New post