stopping whm/cpanel from restarting apache daily
I have WHM/Cpanel on my server. I visit my shell and when I look through the apache logs, I see SIGTERM at around 7am every day and apache restarts.
I checked through anacron and cron configuration and nothing there suggests processing any action in the 7am hour.
However, when I was snooping around in my logs throughout the system, I came across the following file:
/usr/local/cpanel/logs/safeapacherestart_log
and it is updated at the same time apache is being restarted. so something suggests to me that one of the running daemons made by the whm/cpanel developers is causing the apache restart.
Is there any way to disable such a daily apache restart without permanently kiling the normal operation of cpanel/whm?
-
Hey hey! The /usr/local/cpanel/logs/safeapacherestart_log is specific to restarts from the cPanel & WHM service, but I can confirm that it isn't normal for the service to be restarted so frequently. Do you have a user making changes around that time, such as adding or removing domains from the system? If you run "apachectl status" that will show you the uptime of the main process - here is the output from my personal machine:
Current Time: Monday, 11-Nov-2024 22:28:37 EST
Restart Time: Thursday, 31-Oct-2024 05:34:15 EDT0 -
What I did to the server before was arrange things so that temporary files are written to ram drives (and same with pid files) so it puts a lower strain on the server. (reading from ram is faster than reading from a disk). and I did have most logs redirected to the black hole (/dev/null) to not waste disk space but last night I decided to turn them on, and restart whm/cpanel daemons to see what's going on.
According to the apache log, the error happened today at 7:06am. I'll show the cpanel error_log that has entries around that time:
[2024-11-12 05:38:55 -0500] warn [cpsrvd] Duplicate logaccess: at /usr/local/cpanel/Cpanel/Server/Logger.pm line 84. Cpanel::Server::Logger::logaccess(Cpanel::Server::Logger=HASH(0x3b5feb0)) called at /usr/local/cpanel/Cpanel/Server/Connection.pm line 151 Cpanel::Server::Connection::pipehandler(Cpanel::Server::Connection=HASH(0x3b67de8)) called at /usr/local/cpanel/Cpanel/Server/Connection.pm line 142 Cpanel::Server::Connection::check_pipehandler_globals(Cpanel::Server::Connection=HASH(0x3b67de8)) called at /usr/local/cpanel/Cpanel/Server/Connection.pm line 134 Cpanel::Server::Connection::write_buffer(Cpanel::Server::Connection=HASH(0x3b67de8), "HTTP/1.1 401 Access Denied\x{d}\x{a}Set-Cookie: cprelogin=no; HttpOnl"...) called at /usr/local/cpanel/Cpanel/Server.pm line 1035 Cpanel::Server::write_buffer(Cpanel::Server=HASH(0x3bad290), "HTTP/1.1 401 Access Denied\x{d}\x{a}Set-Cookie: cprelogin=no; HttpOnl"...) called at /usr/local/cpanel/Cpanel/Server.pm line 1016 Cpanel::Server::write_content_to_socket(Cpanel::Server=HASH(0x3bad290), 0, "HTTP/1.1 401 Access Denied\x{d}\x{a}Set-Cookie: cprelogin=no; HttpOnl"..., SCALAR(0x3bad7a0)) called at /usr/local/cpanel/Cpanel/Server.pm line 557 Cpanel::Server::badpass(Cpanel::Server=HASH(0x3bad290), __CPANEL_HIDDEN__, 1) called at cpsrvd.pl line 2836 cpanel::cpsrvd::badpass(__CPANEL_HIDDEN__, 1) called at cpsrvd.pl line 5163 cpanel::cpsrvd::handle_auth() called at cpsrvd.pl line 1089 cpanel::cpsrvd::handle_one_connection() called at cpsrvd.pl line 892 cpanel::cpsrvd::script() called at cpsrvd.pl line 380 at /usr/local/cpanel/Cpanel/Server/Logger.pm line 84. Cpanel::Server::Logger::logaccess(Cpanel::Server::Logger=HASH(0x3b5feb0)) called at /usr/local/cpanel/Cpanel/Server/Connection.pm line 151 Cpanel::Server::Connection::pipehandler(Cpanel::Server::Connection=HASH(0x3b67de8)) called at /usr/local/cpanel/Cpanel/Server/Connection.pm line 142 Cpanel::Server::Connection::check_pipehandler_globals(Cpanel::Server::Connection=HASH(0x3b67de8)) called at /usr/local/cpanel/Cpanel/Server/Connection.pm line 134 Cpanel::Server::Connection::write_buffer(Cpanel::Server::Connection=HASH(0x3b67de8), "HTTP/1.1 401 Access Denied\x{d}\x{a}Set-Cookie: cprelogin=no; HttpOnl"...) called at /usr/local/cpanel/Cpanel/Server.pm line 1035 Cpanel::Server::write_buffer(Cpanel::Server=HASH(0x3bad290), "HTTP/1.1 401 Access Denied\x{d}\x{a}Set-Cookie: cprelogin=no; HttpOnl"...) called at /usr/local/cpanel/Cpanel/Server.pm line 1016 Cpanel::Server::write_content_to_socket(Cpanel::Server=HASH(0x3bad290), 0, "HTTP/1.1 401 Access Denied\x{d}\x{a}Set-Cookie: cprelogin=no; HttpOnl"..., SCALAR(0x3bad7a0)) called at /usr/local/cpanel/Cpanel/Server.pm line 557 Cpanel::Server::badpass(Cpanel::Server=HASH(0x3bad290), __CPANEL_HIDDEN__, 1) called at cpsrvd.pl line 2836 cpanel::cpsrvd::badpass(__CPANEL_HIDDEN__, 1) called at cpsrvd.pl line 5163 cpanel::cpsrvd::handle_auth() called at cpsrvd.pl line 1089 cpanel::cpsrvd::handle_one_connection() called at cpsrvd.pl line 892 cpanel::cpsrvd::script() called at cpsrvd.pl line 380 [2024-11-12 07:06:00 -0500] warn [restartsrv_httpd] The 'httpd' service's PID file '/usr/local/apache/logs/httpd.pid' did not appear after 10 seconds. at /usr/local/cpanel/Cpanel/ServiceManager/Base.pm line 1094. Cpanel::ServiceManager::Base::is_up_via_pidfile(Cpanel::ServiceManager::Services::Httpd=HASH(0x216dd48)) called at /usr/local/cpanel/Cpanel/ServiceManager/Base.pm line 760 Cpanel::ServiceManager::Base::start(Cpanel::ServiceManager::Services::Httpd=HASH(0x216dd48)) called at /usr/local/cpanel/Cpanel/ServiceManager/Services/Httpd.pm line 40 Cpanel::ServiceManager::Services::Httpd::start(Cpanel::ServiceManager::Services::Httpd=HASH(0x216dd48)) called at /usr/local/cpanel/Cpanel/ServiceManager/Base.pm line 688 Cpanel::ServiceManager::Base::restart(Cpanel::ServiceManager::Services::Httpd=HASH(0x216dd48), "graceful", undef) called at /usr/local/cpanel/Cpanel/ServiceManager/Base.pm line 528 Cpanel::ServiceManager::Base::run_from_argv(Cpanel::ServiceManager::Services::Httpd=HASH(0x216dd48), "--verbose") called at /usr/local/cpanel/scripts/restartsrv_httpd line 49 main::__ANON__() called at /usr/local/cpanel/3rdparty/perl/514/lib64/perl5/cpanel_lib/Try/Tiny.pm line 80 eval {...} called at /usr/local/cpanel/3rdparty/perl/514/lib64/perl5/cpanel_lib/Try/Tiny.pm line 71 Try::Tiny::try(CODE(0x1848de8), Try::Tiny::Catch=REF(0x1c44be8)) called at /usr/local/cpanel/scripts/restartsrv_httpd line 68 [2024-11-12 07:06:16 -0500] warn [restartsrv_httpd] The 'httpd' service's PID file '/usr/local/apache/logs/httpd.pid' did not appear after 10 seconds. at /usr/local/cpanel/Cpanel/ServiceManager/Base.pm line 1094. Cpanel::ServiceManager::Base::is_up_via_pidfile(Cpanel::ServiceManager::Services::Httpd=HASH(0x216dd48)) called at /usr/local/cpanel/Cpanel/ServiceManager/Base.pm line 760 Cpanel::ServiceManager::Base::start(Cpanel::ServiceManager::Services::Httpd=HASH(0x216dd48)) called at /usr/local/cpanel/Cpanel/ServiceManager/Services/Httpd.pm line 40 Cpanel::ServiceManager::Services::Httpd::start(Cpanel::ServiceManager::Services::Httpd=HASH(0x216dd48)) called at /usr/local/cpanel/Cpanel/ServiceManager/Base.pm line 688 Cpanel::ServiceManager::Base::restart(Cpanel::ServiceManager::Services::Httpd=HASH(0x216dd48), "graceful", undef) called at /usr/local/cpanel/Cpanel/ServiceManager/Base.pm line 528 Cpanel::ServiceManager::Base::run_from_argv(Cpanel::ServiceManager::Services::Httpd=HASH(0x216dd48), "--verbose") called at /usr/local/cpanel/scripts/restartsrv_httpd line 49 main::__ANON__() called at /usr/local/cpanel/3rdparty/perl/514/lib64/perl5/cpanel_lib/Try/Tiny.pm line 80 eval {...} called at /usr/local/cpanel/3rdparty/perl/514/lib64/perl5/cpanel_lib/Try/Tiny.pm line 71 Try::Tiny::try(CODE(0x1848de8), Try::Tiny::Catch=REF(0x1c44be8)) called at /usr/local/cpanel/scripts/restartsrv_httpd line 68 [2024-11-12 07:06:32 -0500] warn [restartsrv_httpd] The 'httpd' service's PID file '/usr/local/apache/logs/httpd.pid' did not appear after 10 seconds. at /usr/local/cpanel/Cpanel/ServiceManager/Base.pm line 1094. Cpanel::ServiceManager::Base::is_up_via_pidfile(Cpanel::ServiceManager::Services::Httpd=HASH(0x216dd48)) called at /usr/local/cpanel/Cpanel/ServiceManager/Base.pm line 760 Cpanel::ServiceManager::Base::start(Cpanel::ServiceManager::Services::Httpd=HASH(0x216dd48)) called at /usr/local/cpanel/Cpanel/ServiceManager/Services/Httpd.pm line 40 Cpanel::ServiceManager::Services::Httpd::start(Cpanel::ServiceManager::Services::Httpd=HASH(0x216dd48)) called at /usr/local/cpanel/Cpanel/ServiceManager/Base.pm line 688 Cpanel::ServiceManager::Base::restart(Cpanel::ServiceManager::Services::Httpd=HASH(0x216dd48), "graceful", undef) called at /usr/local/cpanel/Cpanel/ServiceManager/Base.pm line 528 Cpanel::ServiceManager::Base::run_from_argv(Cpanel::ServiceManager::Services::Httpd=HASH(0x216dd48), "--verbose") called at /usr/local/cpanel/scripts/restartsrv_httpd line 49 main::__ANON__() called at /usr/local/cpanel/3rdparty/perl/514/lib64/perl5/cpanel_lib/Try/Tiny.pm line 80 eval {...} called at /usr/local/cpanel/3rdparty/perl/514/lib64/perl5/cpanel_lib/Try/Tiny.pm line 71 Try::Tiny::try(CODE(0x1848de8), Try::Tiny::Catch=REF(0x1c44be8)) called at /usr/local/cpanel/scripts/restartsrv_httpd line 68So its making me think that it expects the apache PID file to be at
/usr/local/apache/logs/httpd.pid
however, I have the pid on the ram drive accessible by root. If this is what's making the problem happen, then how do I setup whm/cpanel so that it'll look for the pid file where I tell it to?
0 -
Looks like this article might help me.
I followed it to change the httpd config for chkservd and what's interesting is that it send out a get command as HTTP/1.0 to see if the apache server works. What's even crazier is that if chkservd thinks the server doesn't work to its needs, then it will issue a TERM signal and a kill signal then it will issue other commands to stop apache then issue commands to start it again.
so I replaced the whole post commands with a simple /path/to/httpd -k graceful
Why graceful? its less intensive and when I tried the graceful option on my local machine for the first time, apache starts up.
I'm hoping that will solve my morning apache problems at least to some degree.
0 -
That sounds like a good workaround!
0 -
The good news is the enforced hard restart did convert to graceful and subsequently my google rankings somewhat improved but I still see in the logs that something with your chksrvd program is causing the "restart" function (that i changed to graceful) to occur at 7am everyday.
but I noticed that something reverted my /etc/chkserv.d/httpd configuration back to cpanel defaults.
How do I stop that from happening?
0 -
also, why does the configuration begin with:
service[httpd]=80,GET / HTTP/1.0,HTTP/1..,
Apache never returns HTTP/1... as a response or do dots have a special meaning?
0 -
I did some testing on my end with this and we actually don't allow those files to be edited, as they will get overwritten as part of the nightly update. There isn't a safe way to keep that from happening (outside of post-upcp scripts, but that gets messy and usually leads to people forgetting about them, causing future issues.
The better question, is why cPanel wants to restart the server at 7am. I would be watching the server in real-time around that time running "apachectl status" to see what is happening with that service, as chkservd would only be triggered if we can't reach port 80.
As far as the status, we're really just seeing if we can make a connection to port 80, and then the monitoring tool is happy.
0 -
I haven't seen any entry in my apache error_log or access_log that would suggest a restart, but I have seen many entries such as the following:
127.0.0.1 [14/Nov/2024:08:30:02 -0500] GET / HTTP/1.0\n 400 16 - -
I believe your software does that to check the service runs. According to those entries, your software checked the service status once every 10 minutes, and now its doing it once every 2 to 2 1/2 hours.
I noticed (using aureport -x -i and aureport -s -i) that your script safeapacherestart was called at the time the "restart" incident happened everyday. What are all the things in your software that could call safeapacherestart?
0 -
Correct - that localhost entry is the monitoring, which should be happening every 5 minutes by default. The only place that can be changed is in WHM >> Tweak Settings with the "The number of seconds between ChkServd service checks. (Minimum: 60; Maximum: 7,200)" - could you see if that number has been changed from the default of 300?
0 -
Oh ya its set at 7200 which explains the 2 hours. I'm gonna look at all my chkservd settings
0 -
There you go - hopefully that's all that's happening, but you just need to see if Apache really is running low on connections to determine if that is the cause.
0 -
I have looked at the low connection issue in the past. I even set idling timeouts low so people will have a lower chance of performing a slowloris attack. I may need to look into apache RLIMIT settings again incase some approved developer uploads a script that hogs an apache process (as an attempt to wreck the server) then hopefully the server can kill it partway through.
In about an hour or so I'll see what the logs are doing.
Also, why is "HTTP/1.." specified as an expected response in your version of the /etc/chkserv.d/httpd file? I ask because maybe that line might trigger your system to think apache isn't running (because it doesn't return a response beginning with "HTTP/1..") and then it counts how many times it doesn't run and if that number is high enough then it tries to restart apache? I may be overthinking, but normally an apache server returns a response that starts with "HTTP/1.0 " or "HTTP/1.1 ", not "HTTP/1...".
0 -
I think you're giving chkservd too much credit :D
It doesn't keep a count or anything like that - if it senses the service is offline even just one time it attempts a restart. In the case of Apache it's simply that port 80 check so anything that replies with "HTTP/1*" is considered a success - it doesn't have to be an exact match.
0 -
ok im gonna keep looking at the logs but if theres anything else from your services that may trigger an apache restart let me know and ill look at those services as well.
0 -
The only other thing that would do that is the taskqueue process, which can delay service restarts (such as Apache) when you are creating domains. That way, if you're creating a lot of domains in a row, the service doesn't get restarted for each one.
However, this doesn't seem to be what's happening here since this is happening on a regular schedule.
0 -
in WHM I went to server statistics configuration and changed the bandwidth processing frequency from 24 hours to 1 hour then waited a few hours and the apache logs indicate there are more graceful restarts. Makes me wonder if your logging software is deciding to cycle logs on the hour... is there a way I can bypass this and instead have the apache logs rotated when they reach a certain file size?
0 -
That's how things *should* be working on a default system - we have that value threshold set to 300M by default:
https://docs.cpanel.net/whm/service-configuration/log-rotation/
This might be another good thing to have them check in that ticket.
0 -
Just before you said that, I went into the apache log configuration in WHM and unchecked rotating access_log and error_log and I don't see the restart attempts anymore. It's strange how that worked, yet the system configs for anacron, cron, and logrotate didn't help.
0
Please sign in to leave a comment.
Comments
18 comments