[Case 161033] MySQL Server Crashing overnight during upcp
We had a very eventful night last night. We saw approximately 10-20% of our servers failing mysql checks last night. Each one was found to have just completed running upcp --cron.
Example log messages below from updatelogs
[20150204.232422] MySQL55-client-5.5.40-1.cp1148
[20150204.232502] MySQL55-test-5.5.40-1.cp1148
[20150204.232520] Giving mysqld 10 seconds to exit nicely
[20150204.232530] MySQL55-server-5.5.40-1.cp1148
[20150204.232534] Starting MySQL using init.d
[20150204.232538] Starting MySQL....[ OK ]
[20150204.232538] Giving mysqld 5 seconds to start
[20150204.232545] Waiting for "mysql" to restart ""The system failed to open the 'mysql' service's PID file '/var/lib/mysql/server1.serverdomain.com.pid': No such file or directory at /usr/local/cpanel/Cpanel/ServiceManager/Base.pm line 595.
[20150204.232551] Looking for 'mysql' as: /usr/bin/mysql
[20150204.232551] Looking for 'mysqlcheck' as: /usr/bin/mysqlcheck
[20150204.232551] The "mysql" service is down.
[20150204.232551] Unable to reset root MySQL password. at /usr/local/cpanel/scripts/securemysql line 151.
[20150204.232552] The "mysql" service is down.-
Hello, Can you please check MySQL server logs and let us know what exact error logs are getting in that file. 0 -
[Case 161033] WHM update knocked my mysql down I was about to go to bed but decided to check my email first, all of a sudden i get screen after screen after screen of several hundred cron notices like this Warning: mysqli_connect(): (HY000/2002): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2) in
so then i went into the account cpanel to see what was up and when i tried to get into the db section it get Error while connecting to MySQL: Can't connect to local MySQL server through socket '/tmp/mysql.sock' (2) Error while connecting to MySQL: Can't connect to local MySQL server through socket '/tmp/mysql.sock' (2) at /usr/local/cpanel/Cpanel/Mysql.pm line 164.
so i went into the root whm and i see the new modsecurity question and answered it, restarted mysql and all was well again. So it appears that update knocked my mysql offline... just wanted to let you know in case you need to do something0 -
Indeed - I've seen a number of systems effected by this problem. I assume cPanel developers are aware of the problem by now, and working on a patch :) 0 -
re: [Case 161033] WHM update knocked my mysql down Yep. I can report this happened on two separate VPSs that I admin today. Looks like the update script just neglected to restart the MySQL process once it finished. Guess automatic package updates are getting turned off for now until CPanel can respond to the issue. 0 -
re: [Case 161033] WHM update knocked my mysql down Waking up this morning, I encountered the same issue. MySQL was down, which impacted all websites on the server. MySQL processes showed no processes working. The changelog for cPanel/WHM (Change Log for 11.48.0.8) reflects a recent update occurred earlier this morning and I believe that MySQL was not restarted. 0 -
Re: [Case 161033] WHM update knocked my mysql down Thanks for the reports, there have also been several tickets opened about this as I understand it. I've updated this thread with a proper case number to reference via the changelog. A fix is on the way soon. 0 -
Re: [Case 161033] WHM update knocked my mysql down ]I was about to go to bed but decided to check my email first, ...
Sleep? Oh come on, who's got time for sleep? :p0 -
MySQL Offline Hi, I have a VPS CENTOS 6.6, WHM 11.48.0 (build 8) The MySQL server went offline for an unknown reason, which of course caused the websites to go down. I logged into WHM, restarted the MySQL server and then everything was fine. I want to know what has caused this to make sure it doesn't happen again. Please help. Thanks! 0 -
Re: [Case 161033] WHM update knocked my mysql down ]Sleep? Oh come on, who's got time for sleep? :p
Now he did say going to bed not to sleep........0 -
Re: [Case 161033] WHM update knocked my mysql down i was going to do another thread but since i guess this has to do with the update ill post it here. what are you guys doing to me, everything was running so perfectly for months and now its all crashing down.... sites are down again and i go to add my ip to the whitelist thinking i got blocked for some reason and i cant even add my ip to the cphulk white list it says this The system failed to add an IP address to cphulkd hosts because the "cphulkd.ip_lists" table may be corrupted and returned the following error: DBD::mysql::db do failed: Table 'cphulkd.ip_lists' doesn't exist.
what am i suppose to do now? i disabled cphulk and reenabled, same thing. so now im doing a total system reboot. Word for the devs and management, i personally dont care about eye candy in WHM, i dont care about fancy js buttons or sliders, i care that it works. So dont make it pretty, make it work... thanks UPDATE: full reboot got the sites back up... so thats ok but i do need to get that table installed or maybe ill wait to see if there is a patch for this for all of us...0 -
Re: [Case 161033] WHM update knocked my mysql down This thread is related: [Case 160953] Upgrades to 11.48 cannot update the cphulkd database schema - cPanel Forums 0 -
Re: [Case 161033] WHM update knocked my mysql down cool fixed.. 0 -
Re: [Case 161033] WHM update knocked my mysql down I was about to go to bed but decided to check my email first,
You can sleep when your dead :) I could not resist0 -
Re: [Case 161033] WHM update knocked my mysql down Hi. Woke up to the same issue of websites not able to connect to mySQL's. A Graceful reboot has sorted it, but will this issue return during the day/weekend or is it fixed? 0 -
Hello :) I've merged these threads. cPanel version 11.48.0.9 has been released to address the MySQL restart issue that occurs during the upcp process: Fixed case 161033: Make MySQL use an init script to start and stop. Thank you. 0
Please sign in to leave a comment.
Comments
17 comments