Skip to main content

tailwatchd service failure after update to cPanel & WHM version 88

Michael-Inet
Getting one of these emails every hour after updatenow to '11.88.0.7.' yum update and reboot didn"t help. tl;dr Says tailwatchd is already running so fails to start it. Fair enough, but what process is saying it"s down then that it tries to restart it? Or, more relevant, how do I get them to stop? Service Check Raw Output (XID y4jnv3) The "tailwatchd" service is down. Startup Log Jun 09 02:20:45 srv03 systemd[1]: Starting tailwatchd... Jun 09 02:20:45 srv03 restartsrv_tailwatchd[16039]: tailwatchd is already running (tailwatchd) with PID 804 by root Jun 09 02:20:45 srv03 systemd[1]: tailwatchd.service: control process exited, code=exited status=1 Jun 09 02:20:45 srv03 systemd[1]: Failed to start tailwatchd. Jun 09 02:20:45 srv03 systemd[1]: Unit tailwatchd.service entered failed state. Jun 09 02:20:45 srv03 systemd[1]: tailwatchd.service failed. Log Messages The system could not provide log messages for "tailwatchd" because it failed to read all of the potential log files with the following errors: Error while attempting to open "/var/log/secure": "No such file or directory", Error while attempting to open "/var/log/maillog": "No such file or directory", Error while attempting to open "/var/log/messages": "No such file or directory" Memory Information Used 1,002 MB Available 7.49 GB Installed 8 GB Load Information 0.62 1.06 0.64 Uptime 17 days, 12 hours, 21 minutes, and 1 second
rofl # /usr/local/cpanel/bin/whmapi1 version [whmapi1] This program can only be run by root! CentOS Linux release 7.8.2003 virtuozzo Target version set to '11.88.0.7' ==== Edit === Two updates in one day? Uhm, okay... Still broken, still getting the tailwatchd.service failed every hour... And the update to '11.88.0.9' went FUBAR as well... => Log opened from /usr/local/cpanel/scripts/post_sync_cleanup (2424) at Tue Jun 9 18:12:26 2020 [2020-06-09 18:12:26 -0500] Processing: Restarting queueprocd before taskrun [2020-06-09 18:12:26 -0500] - Processing command `/usr/local/cpanel/scripts/restartsrv_queueprocd` [2020-06-09 18:13:56 -0500] [/usr/local/cpanel/scripts/restartsrv_queueprocd] Waiting for "queueprocd" to start "failed. [2020-06-09 18:13:56 -0500] [/usr/local/cpanel/scripts/restartsrv_queueprocd] [2020-06-09 18:13:56 -0500] [/usr/local/cpanel/scripts/restartsrv_queueprocd] Cpanel::Exception::Services::StartError [2020-06-09 18:13:56 -0500] [/usr/local/cpanel/scripts/restartsrv_queueprocd] Service Status [2020-06-09 18:13:56 -0500] [/usr/local/cpanel/scripts/restartsrv_queueprocd] undefined status from Cpanel::ServiceManager::Services::Queueprocd [2020-06-09 18:13:56 -0500] [/usr/local/cpanel/scripts/restartsrv_queueprocd] [2020-06-09 18:13:56 -0500] [/usr/local/cpanel/scripts/restartsrv_queueprocd] Service Error [2020-06-09 18:13:56 -0500] [/usr/local/cpanel/scripts/restartsrv_queueprocd] (XID dye4e2) The "queueprocd" service failed to start. [2020-06-09 18:13:56 -0500] [/usr/local/cpanel/scripts/restartsrv_queueprocd] [2020-06-09 18:13:56 -0500] [/usr/local/cpanel/scripts/restartsrv_queueprocd] Startup Log [2020-06-09 18:13:56 -0500] [/usr/local/cpanel/scripts/restartsrv_queueprocd] Jun 09 18:12:26 srv03 systemd[1]: Starting cPanel Queue services... [2020-06-09 18:13:56 -0500] [/usr/local/cpanel/scripts/restartsrv_queueprocd] Jun 09 18:12:26 srv03 startqueueprocd[2430]: ==> cPanel TaskQueue Processing Daemon version 1.0 [2020-06-09 18:13:56 -0500] [/usr/local/cpanel/scripts/restartsrv_queueprocd] Jun 09 18:12:26 srv03 systemd[1]: Can't open PID file /var/run/queueprocd.pid (yet?) after start: Too many levels of symbolic links [2020-06-09 18:13:56 -0500] [/usr/local/cpanel/scripts/restartsrv_queueprocd] Jun 09 18:13:56 srv03 systemd[1]: queueprocd.service start operation timed out. Terminating. [2020-06-09 18:13:56 -0500] [/usr/local/cpanel/scripts/restartsrv_queueprocd] Jun 09 18:13:56 srv03 systemd[1]: Failed to start cPanel Queue services. [2020-06-09 18:13:56 -0500] [/usr/local/cpanel/scripts/restartsrv_queueprocd] Jun 09 18:13:56 srv03 systemd[1]: Unit queueprocd.service entered failed state. [2020-06-09 18:13:56 -0500] [/usr/local/cpanel/scripts/restartsrv_queueprocd] Jun 09 18:13:56 srv03 systemd[1]: queueprocd.service failed. [2020-06-09 18:13:56 -0500] [/usr/local/cpanel/scripts/restartsrv_queueprocd] [2020-06-09 18:13:56 -0500] [/usr/local/cpanel/scripts/restartsrv_queueprocd] queueprocd has failed. Contact your system administrator if the service does not automagically recover. => Log closed Tue Jun 9 18:13:56 2020
Time to hire some QA people????

Comments

6 comments

Date Votes
  • Michael-Inet
    WTF?
    Tier Currently Description
    LTS 86.0.21 A well-tested and proven version that cPanel, L.L.C. supports for one full year.
    STABLE 86.0.21 A version that is feature-complete, fully tested, and is in widespread use.
    RELEASE Recommended 88.0.9 A version that is feature-complete and well tested.
    CURRENT 88.0.9 A "release candidate" version that is tested but may not be feature-complete.
    EDGE 88.0.9 A development version for testing only and not for production servers.
    This server is suppose to be on "Stable" but "Release" is selected which is also Edge? Based on the entire thread about "v88 Known Issues," v88 is known to be not ready for production machines, yet it"s allowed to be a "Recommended" tier? Seriously, who"s bright idea is it to have production servers running an experimental release? Is there any other way than completely rebuilding the server to get this back to an actual production level release?
    0
  • jigster
    We have exactly the same issue with tailwatchd errors every hour, on just one server - a cpanel DNSONLY server . Same errors exactly. Any progress on a fix for this?
    0
  • Michael-Inet
    Your Support Ticket ID is: 93497725 Addendum: - Error is happening on a full WHM/cPanel install. - I know of 2 other DNSONLY servers this is not happening on.
    0
  • cPanelLauren
    The program that can't restart *is* tailwatchd - it runs in the background in order to manage other services. I was going to have you check /var/log/messages for the systemd output when it is restarted or run: systemctl start tailwatchd.service
    But it is best this be handled in the ticket since they will have access to the system
    0
  • Michael-Inet
    The program that can't restart *is* tailwatchd

    Thanks for that! This log line is just really confusing "tailwatchd is already running (tailwatchd) with PID 804 by root!" Someone's in the box now looking at it, if he doesn't update here with results, I'll do so when he's finished. Thanks again, Michael
    0
  • Michael-Inet
    Hi All, Not a cPanel problem... For details read the reply below from cPanel support. Including my ticket to my hosting company if you just want to cut&paste: [quote]Hi Ya, Okay, I admit I don"t fully understand the fine level details with this issue, but apparently recent CentOS updates are causing errors with older kernels (symlinks specifically?). Request: Please update the kernel on the node and reboot. I"m copying the email cPanel support sent me below as background information, with links to relevant bug reports, etc. Thanks, Michael Hmm, it expanded links, but should still work... ==== Subject: [cPanel] Re: tailwatchd service failure after update to cPanel & WHM version 88 Date: 06/11/20 14:40:35 From: "Jeremy (cPanel Technical Support)" To: Michael ---------------------------------------------- Jeremy, Jun 11, 2020, 2:40:34 PM CDT Hello again, Thank you for your time and patience. I am sorry for any inconvenience, but at this time I do believe I have some information to share. The issue you are facing is not specific to the recent cPanel updates, but it also related to recent CentOS updates and how CentOS now manages symlinks. If your kernel does not support the new method that systemd uses to open symlinks, you will see errors like these, i.e., "Too many levels of symbolic links": For example, I tried to restart MySQL on your server, and encountered several of this error: Jun 09 02:25:04 srv03 systemd[1]: Stopped MySQL Server. Jun 09 02:25:04 srv03 systemd[1]: Starting MySQL Server... Jun 09 02:25:04 srv03 systemd[1]: Can't open PID file /var/run/mysqld/mysqld.pid (yet?) after start: Too many levels of symbolic links Jun 09 08:52:07 srv03 systemd[1]: Stopped MySQL Server. Jun 10 20:35:10 srv03 systemd[1]: Starting MySQL Server... Jun 10 20:35:10 srv03 systemd[1]: Can't open PID file /var/run/mysqld/mysqld.pid (yet?) after start: Too many levels of symbolic links This problem will be seen on systems with the following configurations: - CentOS 7.7 and OpenVZ/Virtuozzo with kernel prior to 2.6.32-042stab134.7 - CentOS 7.7 with kernel prior to 2.6.39 Your server is currently running the kernel `2.6.32-042stab113.21` **Do not** reboot your server without updating the kernel first. Rebooting the server will likely cause further issues with systemd due to the outdated kernel. I can see your server is running on OpenVZ/Virtuozzo: # cat /var/cpanel/envtype virtuozzo This means that your host will need to update the kernel on the node and reboot to correct this issue. Or, if you have access to the host node you may be able to do this yourself. Please understand this is an operating-system issue with the kernel and systemd, not an issue with cPanel. While the following page is a bug reported against Ubuntu, it is still relevant, and contains a response from OpenVZ developers about the issue: If you have any more questions or issues on this subject, please let us know. I am grateful for your time and patience. Thank you, * * * Jeremy Technical Analyst cPanel, L.L.C. -------------------------------- This email is a service from cPanel.
    0

Please sign in to leave a comment.

Didn't find what you were looking for?

New post