httpd.service error after cPanel update, but works after restart service
Hi,
Today suddenly our websites stops working because Apache it's not working. After a simple reboot from WHM, everything it's working again.
But this it's not the first time this happen, more or less every month we have this problem someday.
I was looking for the logs today:
[2024-05-21 15:48:29 +0200]: “/usr/local/cpanel/scripts/restartsrv_apache_php_fpm --graceful” called by (1516979 - /usr/local/cpanel/scripts/restartsrv_httpd --no-verbose)
[2024-05-21 15:48:29 +0200]: “/usr/local/cpanel/scripts/restartsrv_apache_php_fpm --graceful” called by (1516979 - /usr/local/cpanel/scripts/restartsrv_httpd --no-verbose)
[2024-05-21 15:48:29 +0200] info [restartsrv_httpd] systemd failed to start the service “httpd” (The “/usr/bin/systemctl restart httpd.service --no-ask-password” command (process 1516978) reported error number 1 when it ended.): Job for httpd.service failed because the control process exited with error code.
See "systemctl status httpd.service" and "journalctl -xe" for details.
This it's after some cPanel seld update, it's rebooting services. Ok, and the journal?
May 21 15:41:45 xxx.com systemd[1]: httpd.service: Main process exited, code=exited, status=1/FAILURE
May 21 15:41:45 xxx.com systemd[1]: httpd.service: Failed with result 'exit-code'.
May 21 15:48:28 xxx.com systemd[1]: Starting Apache web server managed by cPanel EasyApache...
May 21 15:48:29 xxx.com restartsrv_httpd[1516976]: httpd: Syntax error on line 3830 of /etc/apache2/conf/httpd.conf: </VirtualHost>sions> directive missing cl>
May 21 15:48:29 xxx.com systemd[1]: httpd.service: Control process exited, code=exited status=1
May 21 15:48:29 xxx.com systemd[1]: httpd.service: Failed with result 'exit-code'.
May 21 15:48:29 xxx.com systemd[1]: Failed to start Apache web server managed by cPanel EasyApache.
May 21 15:48:29 xxx.com systemd[1]: Starting Apache web server managed by cPanel EasyApache...
May 21 15:48:29 xxx.com restartsrv_httpd[1516995]: httpd: Syntax error on line 3830 of /etc/apache2/conf/httpd.conf: </VirtualHost>sions> directive missing cl>
May 21 15:48:29 xxx.com systemd[1]: httpd.service: Control process exited, code=exited status=1
May 21 15:48:29 xxx.com systemd[1]: httpd.service: Failed with result 'exit-code'.
May 21 15:48:29 xxx.com systemd[1]: Failed to start Apache web server managed by cPanel EasyApache.
May 21 15:48:29 xxx.com systemd[1]: Starting Apache web server managed by cPanel EasyApache...
May 21 15:48:29 xxx.com systemd[1]: httpd.service: Can't open PID file /run/apache2/httpd.pid (yet?) after start: No such file or directory
May 21 15:48:30 xxx.com systemd[1]: Started Apache web server managed by cPanel EasyApache.
As you can see, they complain about conf file, but that file it's touched only by cPanel, and as I said, after a simple reboot one minute later, everything was working again.
So I suspect that maybe after the cPanel update, when cPanel it's still generating the httpd conf file, Apache reboots and find errors on the file, and some seconds later, those errors are gone.
Can be? I would like to open a ticket to report that, but our license it's from OVH so I cannot open a ticket. Unpleasant
Thank you,
-
Hey there! Can you try running these commands to see if Apache starts normally afterward?
cp /etc/apache2/conf/http.conf{,.bak-`date +%Y%m%d`}
/scripts/rebuildhttpdconf
/scripts/restartsrv_httpd0 -
Hi!
Yes, after a minute we detected the issue, we have restarted Apache from web interface without any kind of changes or rebuilds, and works perfectly.
That's why I think it's a bug because Apache was restarted after cPanel update meanwhile httpd.conf was still on the oven, sounds possible?
0 -
Alright, so let's see if we can reproduce it - does it happen again after you run "/scripts/upcp --force" ? If so, are you able to rebuilt the configuration and have it work well after that?
0 -
I don't know how to reproduce because we we were doing nothing. Apache stopped working, we connected to the server and restart Apache, and everything was working again. So we followed the logs to find that, cPanel was updating on that moment and after the update, triggered a restart with error, so we have checked cPanel log and journal:
[2024-05-21 15:41:46 +0200]: “/usr/local/cpanel/scripts/restartsrv_nginx reload” called by (1503021 - /usr/local/cpanel/scripts/ea-nginx)
[2024-05-21 15:48:22 +0200] info [whostmgrd] Successfully verified signature for cpanel (key types: release).
[2024-05-21 15:48:28 +0200] info [Cpanel::HttpUtils::ApRestart] Unable to read PID from /run/apache2/httpd.pid. Force restart activated.
[2024-05-21 15:48:28 +0200]: “/usr/local/cpanel/scripts/restartsrv_httpd --start” called by (1516944 - /usr/local/cpanel/bin/safeapacherestart - safeaprestart - forced restart)
[2024-05-21 15:48:28 +0200]: “/usr/local/cpanel/scripts/restartsrv_apache_php_fpm --graceful” called by (1516948 - /usr/local/cpanel/scripts/restartsrv_httpd --start)
[2024-05-21 15:48:28 +0200]: “/usr/local/cpanel/scripts/restartsrv_apache_php_fpm --graceful” called by (1516948 - /usr/local/cpanel/scripts/restartsrv_httpd --start)
[2024-05-21 15:48:28 +0200]: “/usr/local/cpanel/scripts/restartsrv_apache_php_fpm --graceful” called by (1516960 - /usr/local/cpanel/scripts/restartsrv_httpd --no-verbose)
[2024-05-21 15:48:28 +0200]: “/usr/local/cpanel/scripts/restartsrv_apache_php_fpm --graceful” called by (1516960 - /usr/local/cpanel/scripts/restartsrv_httpd --no-verbose)
[2024-05-21 15:48:29 +0200] info [restartsrv_httpd] systemd failed to start the service “httpd” (The “/usr/bin/systemctl restart httpd.service --no-ask-password” command (process 1516959) reported error number 1 when it ended.): Job for httpd.service failed because the control process exited with error code.
See "systemctl status httpd.service" and "journalctl -xe" for details.
[2024-05-21 15:48:29 +0200]: “/usr/local/cpanel/scripts/restartsrv_apache_php_fpm --graceful” called by (1516979 - /usr/local/cpanel/scripts/restartsrv_httpd --no-verbose)
[2024-05-21 15:48:29 +0200]: “/usr/local/cpanel/scripts/restartsrv_apache_php_fpm --graceful” called by (1516979 - /usr/local/cpanel/scripts/restartsrv_httpd --no-verbose)
[2024-05-21 15:48:29 +0200] info [restartsrv_httpd] systemd failed to start the service “httpd” (The “/usr/bin/systemctl restart httpd.service --no-ask-password” command (process 1516978) reported error number 1 when it ended.): Job for httpd.service failed because the control process exited with error code.
See "systemctl status httpd.service" and "journalctl -xe" for details.May 21 15:41:45 xxx.com systemd[1]: httpd.service: Main process exited, code=exited, status=1/FAILURE
May 21 15:41:45 xxx.com systemd[1]: httpd.service: Failed with result 'exit-code'.
May 21 15:48:28 xxx.com systemd[1]: Starting Apache web server managed by cPanel EasyApache...
May 21 15:48:29 xxx.com restartsrv_httpd[1516976]: httpd: Syntax error on line 3830 of /etc/apache2/conf/httpd.conf: </VirtualHost>sions> directive missing cl>
May 21 15:48:29 xxx.com systemd[1]: httpd.service: Control process exited, code=exited status=1
May 21 15:48:29 xxx.com systemd[1]: httpd.service: Failed with result 'exit-code'.
May 21 15:48:29 xxx.com systemd[1]: Failed to start Apache web server managed by cPanel EasyApache.
May 21 15:48:29 xxx.com systemd[1]: Starting Apache web server managed by cPanel EasyApache...
May 21 15:48:29 xxx.com restartsrv_httpd[1516995]: httpd: Syntax error on line 3830 of /etc/apache2/conf/httpd.conf: </VirtualHost>sions> directive missing cl>
May 21 15:48:29 xxx.com systemd[1]: httpd.service: Control process exited, code=exited status=1
May 21 15:48:29 xxx.com systemd[1]: httpd.service: Failed with result 'exit-code'.
May 21 15:48:29 xxx.com systemd[1]: Failed to start Apache web server managed by cPanel EasyApache.
May 21 15:48:29 xxx.com systemd[1]: Starting Apache web server managed by cPanel EasyApache...
May 21 15:48:29 xxx.com systemd[1]: httpd.service: Can't open PID file /run/apache2/httpd.pid (yet?) after start: No such file or directory
May 21 15:48:30 xxx.com systemd[1]: Started Apache web server managed by cPanel EasyApache.The very next minute, we connected to the server, restarted Apache and everything was working fine again. If now I go to the line of the error (3830), there's no such thing there.
Hence, my suspicion is that perhaps that file was being generated at that very moment, and a few seconds later, it had already changed.
But we weren't doing anything on the server, let alone modifying httpd.conf.
Regards,
0 -
It could just be a one-time event, and if that's the case, we'll likely never get more details.
0 -
Sadly it's not the first time, this happened once a month more or les, but it's the first time we look into it. If this happen again, we will write here again.
Thanks
0 -
And manually running the rebuild/restart/forced update all works fine? I was just hoping we could recreate the issue instead of it happening by itself at an inconvenient time.
0 -
Yes, when we detected the failure, we reboot Apache from WHM and restart without any problem. I suspect that if we had left the 5 minutes, tailwatch would have successfully restarted the service. That is, the failure occurs at that precise moment, but by restarting Apache a few seconds later, the problem no longer exists (and the line in httpd.conf has changed).
0 -
So the Apache conf does change after the restart?
0 -
Not exactly. I can't really know.
My suspicion is that httpd.conf was in some kind of re-generation when Apache was restarted, because the error says:
May 21 15:48:29 xxx.com restartsrv_httpd[1516995]: httpd: Syntax error on line 3830 of /etc/apache2/conf/httpd.conf: </VirtualHost>sions> directive missing cl>
And if I go to line 3830 of httpd.conf right now, or yesterday when the event happened, after 5 minutes, it says
<IfModule userdir_module>
So clearly, when Apache restarted and failed, the file was being generated at that time, or something like that. I checked with ls, and the file is now dated a few hours ago today. That is, it is a live file. And yesterday, within Apache autostart, it was supposed to be generated. And that caused the error. A few seconds later, the error was gone.
0 -
Well that's the thing - I wouldn't expect a *restart* of the service to also perform a *rebuild* of the conf file.
Earlier I posted three commands to make a backup and restart the service - running those works well? If so, it would be best to get a copy of the broken Apache configuration file before making any changes so you can compare the whole file, as I wouldn't expect the line numbers to have the same content after a rebuild has happened.
0 -
Yes, those commands works fine, the httpd.conf is generated well and the service it's restarted without problems.
I would like to catch the httpd.conf in bad state, but it's not possible, because when we connected to the server 30 seconds after the failure, the httpd.conf file was ok in that moment. Only have a bad formation in the very second that apache was restarting. I think.
0 -
I don't have a good explanation for that behavior, especially if we aren't able to catch it.
0 -
Hi,
Today this problem has repeated on a different server, but with the same configuration (CLN 8, mod_lsapi).
[root@servidor3 ~]# tail -f /var/log/apache2/error_log
[ N 2024-07-16 11:03:21.0486 2441875/T19 Ser/Server.h:558 ]: [ServerThr.19] Shutdown finished
[ N 2024-07-16 11:03:21.0486 2441875/T1b Ser/Server.h:558 ]: [ServerThr.20] Shutdown finished
[ N 2024-07-16 11:03:21.0486 2441875/T1h Ser/Server.h:558 ]: [ServerThr.23] Shutdown finished
[ N 2024-07-16 11:03:21.0486 2441875/T1f Ser/Server.h:558 ]: [ServerThr.22] Shutdown finished
[ N 2024-07-16 11:03:21.0486 2441875/T1j Ser/Server.h:558 ]: [ServerThr.24] Shutdown finished
[ N 2024-07-16 11:03:21.0486 2441875/T1l Ser/Server.h:902 ]: [ApiServer] Freed 0 spare client objects
[ N 2024-07-16 11:03:21.0486 2441875/T1l Ser/Server.h:558 ]: [ApiServer] Shutdown finished
[Tue Jul 16 11:03:21.084156 2024] [:notice] [pid 2441863:tid 23001203707072] [host xx.com] mod_lsapi: Selfstarter 2441863 stopped
httpd: Syntax error on line 11216 of /etc/apache2/conf/httpd.conf: /etc/apache2/conf/httpd.conf:32605: <VirtualHost> was not closed.\n/etc/apache2/conf/httpd.conf:11216: <VirtualHost> was not closed.
[ N 2024-07-16 11:03:21.3771 2441875/T1 age/Cor/CoreMain.cpp:1326 ]: Passenger core shutdown finishedWe detected it because Nginx (nginx as proxy reverse with official cpanel tool) gives us bad gateway, but the server didn't detected Apache as service down.
After a simple connection to WHM and restart Apache from the UI, the service it's working again without any issues.
The pattern it's the same:
- Apache auto restarts internally and says a line it's bad on httpd.conf file.
- We restart Apache a few seconds later and the problem it's solved.
Of course, nobody was editing or touching in any way the httpd conf file.
Our main suspicion remains that, with many domains, the auto-generation process of httpd.conf is slow and if just Apache is restarted, this can happen.
Perhaps it would be possible to add a double check, so that if Apache restarts and httpd.conf gives this error, wait a few seconds and restart the service.
What is clear is that it is a very sporadic error, but it has already happened in several servers and without any intervention from our side.
Best regards,
0 -
It would be best to create a ticket about this issue so this can be investigated.
0 -
I know, but I cannot because our licenses are from OVH...
0 -
Sure you can - you just contact OVH.
0 -
We are having same issue on Ubuntu22.04 and Cpanel 120.0.14
httpd: Syntax error on line 13863 of /etc/apache2/conf/httpd.conf: /etc/apache2/conf/httpd.conf:13884: <VirtualHost> was not closed.\n/etc/apache2/conf/httpd.conf:13863: <VirtualHost> was not closed.It happened second day in a row.
Once we run:
/scripts/php_fpm_config --rebuild
/scripts/rebuildhttpdconf
/scripts/restartsrv_apache
/scripts/restartsrv_apache_php_fpmit fixes the problem - but it would be nice if the problem had never happend at first...
Other related logs:
Jul 26 14:22:28 hetzner systemd[1]: httpd.service: Main process exited, code=exited, status=1/FAILURE
Jul 26 14:22:32 hetzner systemd[1]: httpd.service: Failed with result 'exit-code'.
Jul 26 14:22:32 hetzner systemd[1]: httpd.service: Consumed 10h 31min 43.694s CPU time.
Jul 26 14:28:30 hetzner systemd[1]: httpd.service: Can't open PID file /run/apache2/httpd.pid (yet?) after start: Operation not permitted0 -
stAn - was the issue with the Apache rebuild just a one-time event or do you need to continually make that adjustment?
0 -
The issue seems random, there is about 2 percent chance to hit the issue while running manually:
/scripts/rebuildhttpdconf
While doing something vhost related in cpanel/whm.
My assumption is that there is another process at the background which tries to rebuild it as well - something like changing configuration via WHM or Cpanel which runs this command as well and when they are hit twice by 2 processes i believe it ends up with broken config and apache doesn't get restarted after that. (on some kernels i've seen accessing a file opened for writing by another process that requests it for reading, might make the file cut)
BUT for this thread i am reporting this here as it also happened without having anybody logged into WHM or Cpanel, just randomly during some cpanel update or maintaince (session cleaning or similar) - it happened to us twice in last week, error log mentions broken apache config (VirtualHost not closed) + apache gets turned off by broken reload.
This happened to us on this server on July 24 (17:22UTC) and July 26 (14:28UTC).
For now i've implemented a quick-fix to restart apache when it is not running via systemd override (not yet triggered as the error did not repeat since July 26 yet):
/etc/systemd/system/httpd.service.d # cat override.conf
[Service]
Restart=on-failure
RestartSec=30sbut a real fix would mean:
- make sure there cannot be a broken httpd config by cpanel (possibly using tmp files and change inode by using rename/mv when the config building is finished, so there is no other processes reading the tmp while it's being written and inode change (file rename) is faster compared to writing the file itself )
- before doing apache reload, make sure "apachectl configtest" does not return an error code...
I've seen this issue on other servers as well cca 2 years ago (centos + cpanel) - it ended up mixing vhosts - the broken config had shown wrong site (staging subdomain instead of live site) and this was fixed by manual apache config rebuild as well...
continually, yes:
- i've created /etc/apache2/rebuild.sh
with the commands above to make sure all rebuilds of all okay and server uses proper vhosts.
best regards, stan
0 -
That's odd for sure - it would likely be best to create a ticket on this issue so the system can be investigated.
0 -
well, i do not really know how to do the ticket and my account is not associated with clients directly and clients are using cpanel via providers (hetzner or ovh). this issue happened to me on most of cpanel servers in last years on different OSes and i don't think that investigating this on client's production servers would lead to a solution - i'd rather suggest to have it unit tested on cpanel development enviroment per the indices above as it's non-reproducable in most cases. (i.e. hitting a process reading a config file while another process writing it could be just an example of the reason of the issue).
My typical clients got up to 5 cpanel accounts with up to 300 subdomains and domains associated to them, while most of the subdomains or add-on-domains are just aliases and redirects for main domains. We are using purely dedicated servers from Hetzner, Ovh or other companies - all of them running fast nvme's. Service subdomains are usually disabled on cpanel servers that i manage as we don't really use them. Most of the clients do not use cpanel for DNS (we copy the config between g.cloud dns and cpanel).
best regards, stan
0
Please sign in to leave a comment.
Comments
22 comments