MySQL Randomly Fails then Recovers

jabelone

• June 18, 2015 06:46

Hi, I'm running a few websites and databases on a VPS I manage. It's running CentOS 6.6 and has been doing it for a while but it's starting to get annoying. :/ MySQL will randomly just "fail" then be recovered. Most sites are WordPress based and I have a mobile app I created that relies on accessing a database so it's quite important. The problem is I have no idea what is causing this to happen. Here is an example email I get from the server every time it happens.
Server: gallifrey.********.com.au Primary IP Address: ***.***.***.*** Service Name: mysql Service Status: failed Notification: The service "mysql" appears to be down. Service Check Method: The system"s command to check or to restart this service failed. Number of Restart Attempts: 1 Startup Log: Starting MySQL...... SUCCESS! Log Messages: 150617 12:09:47 mysqld_safe mysqld from pid file /var/lib/mysql/gallifrey.********.com.au.pid ended 150617 12:09:47 [Note] /usr/sbin/mysqld: Shutdown complete 150617 12:09:46 mysqld_safe mysqld restarted 150617 12:09:46 mysqld_safe Number of processes running now: 0 150617 3:26:20 [Note] /usr/sbin/mysqld: ready for connections. 150617 03:26:15 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql 150617 03:25:04 mysqld_safe mysqld from pid file /var/lib/mysql/gallifrey.********.com.au.pid ended 150617 3:25:04 [Note] /usr/sbin/mysqld: Shutdown complete 150617 03:25:03 mysqld_safe mysqld restarted 150617 03:25:03 mysqld_safe Number of processes running now: 0 150616 12:13:12 [Note] /usr/sbin/mysqld: ready for connections. 150616 12:13:08 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql 150616 12:09:44 mysqld_safe mysqld from pid file /var/lib/mysql/gallifrey.********.com.au.pid ended 150616 12:09:44 [Note] /usr/sbin/mysqld: Shutdown complete 150616 12:09:43 mysqld_safe mysqld restarted 150616 12:09:43 mysqld_safe Number of processes running now: 0 Memory Information: [LIST]

[LIST]

Used: 303 MB

Available: 313 MB

Installed: 616 MB Load Information: 0.20 0.30 0.15 Uptime: 7 days, 19 hours, 33 minutes, and 9 seconds IOStat Information: avg-cpu: %user %nice %system %iowait %steal %idle 2.43 0.44 0.65 3.89 0.03 92.56 Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn vda 11.58 521.90 115.61 351719556 77913264 ChkServd Version: 15.9

Then I get one of these:
Server: gallifrey.********.com.au Primary IP Address: ***.***.***.*** Service Name: mysql Service Status: recovered Notification: The service "mysql" is now operational. Service Check Raw Output: The 'mysql' service passed the check. Startup Log: Starting MySQL...... SUCCESS! Log Messages: 150617 3:26:20 [Note] /usr/sbin/mysqld: ready for connections. 150617 03:26:15 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql 150617 03:25:04 mysqld_safe mysqld from pid file /var/lib/mysql/gallifrey.********.com.au.pid ended 150617 3:25:04 [Note] /usr/sbin/mysqld: Shutdown complete 150617 03:25:03 mysqld_safe mysqld restarted 150617 03:25:03 mysqld_safe Number of processes running now: 0 150616 12:13:12 [Note] /usr/sbin/mysqld: ready for connections. 150616 12:13:08 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql 150616 12:09:44 mysqld_safe mysqld from pid file /var/lib/mysql/gallifrey.********.com.au.pid ended 150616 12:09:44 [Note] /usr/sbin/mysqld: Shutdown complete 150616 12:09:43 mysqld_safe mysqld restarted 150616 12:09:43 mysqld_safe Number of processes running now: 0 150616 3:24:52 [Note] /usr/sbin/mysqld: ready for connections. Memory Information: [LIST]

[LIST]

Used: 308 MB

Available: 308 MB

Installed: 616 MB Load Information: 0.22 0.61 0.40 Uptime: 7 days, 10 hours, 51 minutes, and 16 seconds IOStat Information: avg-cpu: %user %nice %system %iowait %steal %idle 2.45 0.44 0.65 3.95 0.03 92.47 Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn vda 11.68 524.46 117.45 337052860 75479880 ChkServd Version: 15.9

• 

  cPanelMichael

  • July 06, 2015 14:14

  Hello :) Do you notice any additional messages in the MySQL error log (/var/lib/mysql/$hostname.err) at the time of the failures? Thank you.

  0
• 

  jabelone

  • July 13, 2015 12:33

  Hi, sorry for the late reply - for some reason the forum didn't email me when you replied. Here is the last bit of the log from this morning, when it happened again. I have included a lot of the "before it crashed" if that helps. I got a cPanel service monitor email saying it had failed right around 11:22. P.S. Sorry for the terrible formatting in the OP, I don't know what happened and it won't let me edit it. - Removed - No Need For That Much Output in a Thread -

  0
• 

  jabelone

  • July 14, 2015 14:07

  This has been resolved after opening a support ticket with cPanel. As can be seen by these two entries in the log file, the server was running out of memory.
  Jul 14 18:50:15 gallifrey kernel: Out of memory: Kill process 22589 (mysqld) score 159 or sacrifice child Jul 14 18:50:15 gallifrey kernel: Killed process 22589, UID 498, (mysqld) total-vm:1349600kB, anon-rss:161992kB, file-rss:244kB
  

  0
• 

  cPanelMichael

  • July 16, 2015 13:30

  I'm happy to see the issue is now resolved. Thank you for updating us with the outcome.

  0

Please sign in to leave a comment.