tailwatchd The service “tailwatchd” appears to be down

ally1

• November 29, 2017 10:40

Hello WHM & cPanel support and community, I encountered the tailwatchd problem. I got a lot of emails with the following messages:
---------- Forwarded message ---------- From: cPanel Monitoring Date: Wed, Nov 29, 2017 at 6:37 AM Subject: [***my_server***] FAILED : tailwatchd (10.0.131.33) To: my@email_here The service "tailwatchd" appears to be down. Server ***my_server*** Primary IP Address 10.0.131.33 Service Name tailwatchd Service Status failed Notification The service "tailwatchd" appears to be down. Service Check Raw Output Cpanel::Exception::Timeout/(XID d5nfna) The system failed to lock the file "/var/run/restartsrv_tailwatchd" after 121 seconds. at /usr/local/cpanel/Cpanel/Exception/CORE.pm line 336. Cpanel::Exception::create("Timeout", "The system failed to lock the file \x{e2}\x{80}\x{9c}[_1]\x{e2}\x{80}\x{9d} after [quant,_2"..., ARRAY(0x2928f60)) called at /usr/local/cpanel/Cpanel/Exception.pm line 46 Cpanel::Exception::__ANON__(__CPANEL_HIDDEN__, __CPANEL_HIDDEN__..., ARRAY(0x2928f60)) called at /usr/local/cpanel/Cpanel/SafeFile.pm line 710 Cpanel::SafeFile::_timeout_exception("/var/run/restartsrv_tailwatchd", 121) called at /usr/local/cpanel/Cpanel/SafeFile.pm line 742 Cpanel::SafeFile::_die_if_file_is_flocked_cuz_already_ waited_a_while("/var/run/restartsrv_tailwatchd", 121) called at /usr/local/cpanel/Cpanel/SafeFile.pm line 811 Cpanel::SafeFile::_lock_wait("/var/run/restartsrv_tailwatchd") called at /usr/local/cpanel/Cpanel/SafeFile.pm line 339 Cpanel::SafeFile::_safelock("/var/run/restartsrv_tailwatchd") called at /usr/local/cpanel/Cpanel/SafeFile.pm line 538 Cpanel::SafeFile::_safe_open(undef, ">>", "/var/run/restartsrv_tailwatchd", CODE(0xc0ea38), "safeopen") called at /usr/local/cpanel/Cpanel/SafeFile.pm line 107 Cpanel::SafeFile::safeopen(undef, ">>", "/var/run/restartsrv_tailwatchd") called at /usr/local/cpanel/Cpanel/RestartSrv/Lock.pm line 25 Cpanel::RestartSrv::Lock::new("Cpanel::RestartSrv::Lock", "tailwatchd") called at /usr/local/cpanel/Cpanel/ServiceManager/Base.pm line 183 Cpanel::ServiceManager::Base::lock(Cpanel::ServiceManager:: Services::Tailwatchd=HASH(0x29036e8)) called at /usr/local/cpanel/Cpanel/ServiceManager/Base.pm line 459 Cpanel::ServiceManager::Base::BUILD(Cpanel::ServiceManager:: Services::Tailwatchd=HASH(0x29036e8), HASH(0xc4bd58)) called at /usr/local/cpanel/3rdparty/perl/524/lib64/perl5/cpanel_ lib/x86_64-linux-64int/Cpanel/Class/Object.pm line 49 Cpanel::Class::Object::BUILDALL(Cpanel::ServiceManager::Services::Tailwatchd=HASH(0x29036e8), HASH(0xc4bd58)) called at /usr/local/cpanel/3rdparty/ perl/524/lib64/perl5/cpanel_lib/x86_64-linux-64int/Cpanel/Class/Object.pm line 41 Cpanel::Class::Object::new("Cpanel::ServiceManager::Services::Tailwatchd", "service", "tailwatchd") called at /usr/local/cpanel/Cpanel/ServiceManager.pm line 99 Cpanel::ServiceManager::new("Cpanel::ServiceManager", "service", "tailwatchd") called at bin/restartsrv_base.pl line 62 main::__ANON__() called at /usr/local/cpanel/3rdparty/ perl/524/lib64/perl5/cpanel_lib/Try/Tiny.pm line 99 eval {...} called at /usr/local/cpanel/3rdparty/perl/524/lib64/perl5/cpanel_lib/Try/Tiny.pm line 90 Try::Tiny::try(CODE(0xc7c400), Try::Tiny::Catch=REF(0xbe3310)) called at bin/restartsrv_base.pl line 109 Startup Log No startup log Memory Information Used 3.04 GB Available 12.48 GB Installed 15.52 GB Load Information 1.80 1.54 1.40 Uptime 11 days, 10 hours, 18 minutes, and 20 seconds IOStat Information avg-cpu: %user %nice %system %iowait %steal %idle 26.47 0.67 3.81 31.23 0.05 37.77 Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn xvda 216.64 2090.67 932.57 2064534465 920912136 xvdg 11.34 97.29 103.31 96072106 102014040 xvdh 0.73 18.93 6.78 18696002 6695808 xvdf 2.39 112.79 24.97 111379016 24658896 xvdi 6.53 70.58 526.69 69696360 520108328 Top Processes PID Owner CPU % Memory % Command 85284 msarts 8.53 0.40 php-fpm: pool msarts_org 85056 factnotfiction 4.44 0.32 php-fpm: pool factnotfiction_com 84481 factnotfiction 3.91 0.32 php-fpm: pool factnotfiction_com 83298 factnotfiction 3.59 0.32 php-fpm: pool factnotfiction_com 83214 factnotfiction 3.30 0.32 php-fpm: pool factnotfiction_com Configure Monitor Settings: https://***my_server***:2087/scripts/srvmng#service-chkservd Configure chkservd: https://***my_server***:2087/scripts2/tweaksettings?find= chkservd Disable HTML notifications: https://***my_server***:2087/scripts2/tweaksettings?find= chkservd_plaintext_notify The system generated this notice on 2017-11-29 at 12:37:02 UTC. "Service failures (ChkServd)" notifications are currently configured to have an importance of "High". You can change the importance or disable this type of notification in WHM"s Contact Manager at: https://***my_server*** :2087/scripts2/editcontact?event=chkservd Do not reply to this automated message. [image: cP] Copyright" 2017 cPanel, Inc.
The command /scripts/restartsrv_tailwatchd --status throws the next:
################ Cpanel::Exception::Timeout/(XID 8k6uua) The system failed to lock the file "/var/run/restartsrv_tailwatchd" after 121 seconds. at /usr/local/cpanel/Cpanel/Exception/CORE.pm line 336. Cpanel::Exception::create("Timeout", "The system failed to lock the file \x{e2}\x{80}\x{9c}[_1]\x{e2}\x{80}\x{9d} after [quant,_2"..., ARRAY(0x1696ce8)) called at /usr/local/cpanel/Cpanel/Exception.pm line 46 Cpanel::Exception::__ANON__(__CPANEL_HIDDEN__, __CPANEL_HIDDEN__..., ARRAY(0x1696ce8)) called at /usr/local/cpanel/Cpanel/SafeFile.pm line 710 Cpanel::SafeFile::_timeout_exception("/var/run/restartsrv_tailwatchd", 121) called at /usr/local/cpanel/Cpanel/SafeFile.pm line 742 Cpanel::SafeFile::_die_if_file_is_flocked_cuz_already_waited_a_while("/var/run/restartsrv_tailwatchd", 121) called at /usr/local/cpanel/Cpanel/SafeFile.pm line 811 Cpanel::SafeFile::_lock_wait("/var/run/restartsrv_tailwatchd") called at /usr/local/cpanel/Cpanel/SafeFile.pm line 339 Cpanel::SafeFile::_safelock("/var/run/restartsrv_tailwatchd") called at /usr/local/cpanel/Cpanel/SafeFile.pm line 538 Cpanel::SafeFile::_safe_open(undef, ">>", "/var/run/restartsrv_tailwatchd", CODE(0xc0ea38), "safeopen") called at /usr/local/cpanel/Cpanel/SafeFile.pm line 107 Cpanel::SafeFile::safeopen(undef, ">>", "/var/run/restartsrv_tailwatchd") called at /usr/local/cpanel/Cpanel/RestartSrv/Lock.pm line 25 Cpanel::RestartSrv::Lock::new("Cpanel::RestartSrv::Lock", "tailwatchd") called at /usr/local/cpanel/Cpanel/ServiceManager/Base.pm line 183 Cpanel::ServiceManager::Base::lock(Cpanel::ServiceManager::Services::Tailwatchd=HASH(0x16729a0)) called at /usr/local/cpanel/Cpanel/ServiceManager/Base.pm line 459 Cpanel::ServiceManager::Base::BUILD(Cpanel::ServiceManager::Services::Tailwatchd=HASH(0x16729a0), HASH(0xc4bd58)) called at /usr/local/cpanel/3rdparty/perl/524/lib64/perl5/cpanel_lib/x86_64-linux-64int/Cpanel/Class/Object.pm line 49 Cpanel::Class::Object::BUILDALL(Cpanel::ServiceManager::Services::Tailwatchd=HASH(0x16729a0), HASH(0xc4bd58)) called at /usr/local/cpanel/3rdparty/perl/524/lib64/perl5/cpanel_lib/x86_64-linux-64int/Cpanel/Class/Object.pm line 41 Cpanel::Class::Object::new("Cpanel::ServiceManager::Services::Tailwatchd", "service", "tailwatchd") called at /usr/local/cpanel/Cpanel/ServiceManager.pm line 99 Cpanel::ServiceManager::new("Cpanel::ServiceManager", "service", "tailwatchd") called at bin/restartsrv_base.pl line 62 main::__ANON__() called at /usr/local/cpanel/3rdparty/perl/524/lib64/perl5/cpanel_lib/Try/Tiny.pm line 99 eval {...} called at /usr/local/cpanel/3rdparty/perl/524/lib64/perl5/cpanel_lib/Try/Tiny.pm line 90 Try::Tiny::try(CODE(0xc7d078), Try::Tiny::Catch=REF(0xbe3310)) called at bin/restartsrv_base.pl line 109 ################
The /etc/init.d/cpanel restart command throws the same error. Also I tried to update 'cPanel' server with "/scripts/upcp" command. The cpanel was updated successfully but there still the problem with updating tailwatchd, here is output:
################ *** Running restart_tailwatchd_if_updated *** tailwatchd has newer code than the current running version. Restarting Cpanel::Exception::Timeout/(XID ym38rd) The system failed to lock the file <80><9C>/var/run/restartsrv_tailwatchd<80><9D> after 121 seconds. at /usr/local/cpanel/Cpanel/Exception/CORE.pm line 336. Cpanel::Exception::create("Timeout", "The system failed to lock the file \x{e2}\x{80}\x{9c}[_1]\x{e2}\x{80}\x{9d} after [quant,_2"..., ARRAY(0x1a874c8)) called at /usr/local/cpanel/Cpanel/Exception.pm line 46 Cpanel::Exception::__ANON__(__CPANEL_HIDDEN__, __CPANEL_HIDDEN__..., ARRAY(0x1a874c8)) called at /usr/local/cpanel/Cpanel/SafeFile.pm line 710 Cpanel::SafeFile::_timeout_exception("/var/run/restartsrv_tailwatchd", 121) called at /usr/local/cpanel/Cpanel/SafeFile.pm line 742 Cpanel::SafeFile::_die_if_file_is_flocked_cuz_already_waited_a_while("/var/run/restartsrv_tailwatchd", 121) called at /usr/local/cpanel/Cpanel/SafeFile.pm line 811 Cpanel::SafeFile::_lock_wait("/var/run/restartsrv_tailwatchd") called at /usr/local/cpanel/Cpanel/SafeFile.pm line 339 Cpanel::SafeFile::_safelock("/var/run/restartsrv_tailwatchd") called at /usr/local/cpanel/Cpanel/SafeFile.pm line 538 Cpanel::SafeFile::_safe_open(undef, ">>", "/var/run/restartsrv_tailwatchd", CODE(0xc0ea38), "safeopen") called at /usr/local/cpanel/Cpanel/SafeFile.pm line 107 Cpanel::SafeFile::safeopen(undef, ">>", "/var/run/restartsrv_tailwatchd") called at /usr/local/cpanel/Cpanel/RestartSrv/Lock.pm line 25 Cpanel::RestartSrv::Lock::new("Cpanel::RestartSrv::Lock", "tailwatchd") called at /usr/local/cpanel/Cpanel/ServiceManager/Base.pm line 183 Cpanel::ServiceManager::Base::lock(Cpanel::ServiceManager::Services::Tailwatchd=HASH(0x1a5caa8)) called at /usr/local/cpanel/Cpanel/ServiceManager/Base.pm line 459 Cpanel::ServiceManager::Base::BUILD(Cpanel::ServiceManager::Services::Tailwatchd=HASH(0x1a5caa8), HASH(0xc4bd58)) called at /usr/local/cpanel/3rdparty/perl/524/lib64/perl5/cpanel_lib/x86_64-linux-64int/Cpanel/Class/Object.pm line 49 Cpanel::Class::Object::BUILDALL(Cpanel::ServiceManager::Services::Tailwatchd=HASH(0x1a5caa8), HASH(0xc4bd58)) called at /usr/local/cpanel/3rdparty/perl/524/lib64/perl5/cpanel_lib/x86_64-linux-64int/Cpanel/Class/Object.pm line 41 Cpanel::Class::Object::new("Cpanel::ServiceManager::Services::Tailwatchd", "service", "tailwatchd") called at /usr/local/cpanel/Cpanel/ServiceManager.pm line 99 Cpanel::ServiceManager::new("Cpanel::ServiceManager", "service", "tailwatchd") called at bin/restartsrv_base.pl line 62 main::__ANON__() called at /usr/local/cpanel/3rdparty/perl/524/lib64/perl5/cpanel_lib/Try/Tiny.pm line 99 eval {...} called at /usr/local/cpanel/3rdparty/perl/524/lib64/perl5/cpanel_lib/Try/Tiny.pm line 90 Try::Tiny::try(CODE(0xc7d078), Try::Tiny::Catch=REF(0xbe3310)) called at bin/restartsrv_base.pl line 109 *** Done restart_tailwatchd_if_updated *** ################
I'm sorry for the long explanation. Can someone explain how to fix the abowe issue? Thanks.

• 

  cPanelMichael

  • November 29, 2017 17:55

  Hello, Could you open a support ticket using the link in my signature so we can take a closer look? Thank you.

  0
• 

  ally1

  • November 30, 2017 07:06

  Hello, I opened the new support ticket. Ticket ID is: 9073163 Thanks.

  0
• 

  ally1

  • December 03, 2017 09:48

  Hello, Just for those who encountered the same problem, the issue was with eximstats database that had grown too large (about 13 Gb), and tailwatchd was timing out when attempting to start up. The eximstats_db.sqlite3 files was re-initialized and now tailwatchd is up and working. Thanks cPanel support for the help! Best Regards, Alexander Lymar

  0
• 

  did-vmonroig

  • December 04, 2017 08:29

  Just for those who encountered the same problem, the issue was with eximstats database that had grown too large (about 13 Gb), and tailwatchd was timing out when attempting to start up. The eximstats_db.sqlite3 files was re-initialized and now tailwatchd is up and working.

  
  Hello, Thanks for your report. We had a spam event and same problem started because that database has become huge. Time for cleaning! Regards,

  0

Please sign in to leave a comment.