Chkservd Service Status Troubleshooting

Symptoms

Sometimes a service will report as down, despite the service being up and connectable through manual checks. 

 

Description 

When a service check fails cPanel monitoring software, checksrvd will alert you when configured through the Contact Manager. This happens when the checksrvd daemon is unable to connect to the local service, such as MySQL, Apache, or Exim.

 

Workaround

The below information contains a few examples of what can be checked to confirm the service is set correct. 

 

How process checks are performed

In /etc/chkserv.d/, there are files for each service. As an example, here is the one for clamd:

 

 

The syntax for these files is as follows:

 

An example of the port/response check can be seen with /etc/chkserv.d/pop:

 

At least for services that look like the above, chkservd runs  /scripts/restartsrv_$SERVICE --check. If that command produces any output, chkservd assumes that the service has failed and restarts it using /scripts/restartsrv_$SERVICE --restart.

 

Permissions

The permissions on the subdirs in /var/cpanel/serviceauth/ need to be 700 (711 on the serviceauth directory itself), with the exception of the /var/cpanel/serviceauth/exim directory, which should be 750.

 

Logging into Services as chkservd

The login information used for the chkservd service authentication test can be found here:

 

 

To construct the username, append the contents of the "send" file to  __cpanel__service__auth__$SERVICE__.

So for the IMAP example used above, assuming the contents are as follows:

 

 

The username to log into IMAP would be:

 

 

Please note the double underscores. With these values, the password would be this:

 

 

Example

Here is an actual example of logging into IMAP:

 

 

Services Up and Reported as Down

 

Unknown HZ Value

A common cause of services reported as down, when they are actually up, is the following error:

 

 

Typically, this is due to a faulty procps. For example:

 

 

To fix this, update procpcs.

 

SSHD failing

Potential cause:

A specific ListenAddress is defined in /etc/ssh/sshd_config. EX:

 

 

This will prevent local connections to SSH:

 

 

Solution:

Disable the ListenAddress line or add an additional ListenAddress line for 127.0.0.1. Then, restart the SSHD service.

 

Exim failing

chkservd thinks Exim is down, though a direct telnet to localhost port shows that it's up and listening.

Potential cause:

A custom ACL in /etc/exim.conf, most likely a banner delay:

 

 

It appears chkservd wants to see the 220 banner almost immediately after connecting to the service. The above custom delay causes the 220 banner to wait about 15 seconds before displaying, but by that time chkservd thinks it's down and does the restart and fires off the warning email.

Solution:

Comment out the delay line in /etc/exim.conf (or use the configuration editor in WHM), restart Exim, and explain to the customer why that particular ACL causes for failures/restarts in chkservd. Note that lowering the delay to 1 second might avoid the failure warnings, but even at 2 seconds it was failing. You can watch for successes/failures by tailing the chkservd log:

 

 

(Hit enter a few times to clear white space, then either wait for the next run or use /scripts/restartsrv_chkservd using screen or a different terminal to force a check.)

Alt. Solution: Check /etc/resolv.conf on server, and ensure the nameservers there are responding quickly enough for Exim.

Exim - Timeout while trying to get data from service: Died at /usr/local/cpanel/Cpanel/TailWatch/ChkServd.pm line 812, <$socket_scc> line 11.

Check the Exim logs:

 

 

If you see this:

 

 

Check to see if the file exists and if it's been renamed:

 

 

Most likely it will show up as __db.ratelimit, go ahead and rename it, then restart Exim, then restart chkservd, and confirm it's working:

 

 

MySQL failing but it's actually up and the configured root MySQL password is correct

See if CloudLinux's db_governor is installed by running:

 

 

If it is installed, the mysql pid file is stored at /var/run/mysqld/mysqld.pid instead of /var/lib/mysql/$HOSTNAME.pid when using db_governor. So, chkservd is simply checking a non-existent pid file to see if mysql is up. A symlink needs to be created to fix this:

 

 

Services listed in Service Manager are not the ones showing in Service Status

Check for unusual entries in  /usr/local/cpanel/Cpanel/TailWatch/ 

 

Timeout while trying to get data from service

Check the clock. One way to check would be to run  top  and then break out and immediately run date. There should be at most a 1-second difference. If there's a drift, it could cause issues with chkservd's tests.

This error can also appear when Exim on port 465 is enabled in both Service Manager and the Exim configuration file. To address this issue, uncheck the box for "Exim on another port" in Service Manager, then verify that the server is still listening on port 465.

 

 

 

Clock issues

As mentioned above, a messed-up clock will cause issues with chkservd. 

 

Networking

For at least cpsrvd, the source IP needs to be localhost. Odd network/route/iptables rules can interfere with this, so that connections to 127.0.0.1 are actually routed via another IP or interface. This prevents authentication from working and should be corrected. This was corrected by reviewing the masqueraded loopback address.

 

chkservd process has become non-responsive (hung)

You may find a customer with a CentOS 7 server who received an email that chkservd has hung for an extended amount of time. If you check their list of running processes, you may see something like this:

 

 

The issue here is that the systemctl status dovecot.service command is using a large amount of CPU. If you strace the process, you may also see something like this:

 

This may indicate that the journals on the server are corrupted. To confirm this, manually run journalctl --verify:

 

If there are any bad journals found, stop journald, rename the old journal directory, then restart journald again:

 

 

Check the logs for restarts:

 

chkservd not checking a service as it's "too soon" since last restart

You can make chkservd check on the current status of a service by moving the service file aside from /var/run/chkservd.services_suspend and then restarting the tailwatchd service. For example, if Exim is not being checked by chkservd:

 

pure-ftpd Error: Home Directory Not Available - aborting

If the /var/cpanel/userhomes/cpanel directory does not exist, or has invalid permissions, chkservd will report FTPD as down when it is actually up, as seen here:

 

 

You can resolve this by creating the directory, and ensuring it has the proper ownership, with the following command:

 

 

cpsrvd giving 401 error

If chkservd is reporting a 401 error from cpsrvd, in many cases this is caused by CSF/iptables routing rules. Here is an example error message that you may see in /var/log/chkservd.log:

 

 

You can verify the routing with the following command:

 

 

chkservd logs - too soon after restart to check

If you want to clear this so the service check occurs on the next chkservd run then you can move the file in /var/run/chkservd.services_suspend/ and restart tailwatchd.