Skip to main content

Services (Apache, MySQL, etc.) crashing on server

Ioan Sameli
All our websites went down over the week-end. The server was responding (could login in SSH) but all the services were down. At some point during the week-end I got a client report a database connection error on their site (meaning Apache was running but MySQL crashed at some point - this morning they were all down). A reboot fixed everything, but it's the second time it happens in a few weeks. I checked the Daily Process Log and no process took more than 5% of mem or cpu over the last few days. No website had any traffic peak. The whole setup runs on a decent EC2 instance (m3.medium with 3.75GB of ram), I doubt it's a hardware fault. It's a standard LAMP installation and we haven't added any 3rd party software. [LIST]
  • What could be the reason those service crashes? Are there specific logs I could analyze?
  • Is there a simple way to monitor of MySQL / Apache and get an alert when they crash?
  • Any other advice on how to prevent this to happen, or at least understand what it comes from? Thanks in advance for any help!

    • Comments

    5 comments

    Date Votes
    • cPanelMichael
      Hello :) You can ensure monitoring for your services are enabled via: "WHM Home " Service Configuration " Service Manager" Check the Apache and MySQL error logs and review the time at which those services crashed to see if you can find any useful information: Apache error log - /usr/local/apache/logs/error_log MySQL error log - /var/lib/mysql/$hostname.err Thank you.
      0
    • Ioan Sameli
      Hi Michael, I found something very rotten in the MySQL error log: 150220 6:12:41 [Warning] IP address '122.226.102.45' could not be resolved: Name or service not known 150220 17:57:16 [Warning] IP address '117.21.176.129' could not be resolved: Name or service not known 150221 0:16:29 [Warning] IP address '61.240.144.66' could not be resolved: Temporary failure in name resolution 150221 10:56:32 [Warning] IP address '118.122.252.71' could not be resolved: Name or service not known 150224 13:46:10 [Warning] IP address '61.240.144.64' could not be resolved: Temporary failure in name resolution The IPs are from China, and the time correspond to when the server crashed. Not sure why it made the server crash though. I've realised my MySQL server was open to the world (I'm surprised this is the default cPanel config, it seems really insecure). I limited connections to localhost by adding bind-address = 127.0.0.1 in the my.cnf. I'm not sure how this can crash MySQL though? Would brute force attack crash the MySQL server? Could it also lead to crashing the other services such as Apache and BIND?
      0
    • cPanelMichael
      Note that MySQL is not open by default to the public. The remote IP address must be added as an access host via: "WHM Home " SQL Services " Additional MySQL Access Hosts" Do you see those same entries every time it fails, or was that a one time occurrence? Thank you.
      0
    • Ioan Sameli
      ]Note that MySQL is not open by default to the public.

      Right. Then do you have any idea about how those entries end up in the log? How can the Chinese IP do connection attempts on the SQL server if it doesn't accept external connections? The server crashed twice so far, and twice there were those in the log at approximatively the time it crashed. Not sure if they're the cause, but they seem related. Thanks a lot for your support!
      0
    • cPanelMichael
      The connections are allowed, but authorization will fail. You can disable inbound connections to port 3306 in your firewall if you want to block these connections completely, which is likely a good idea if those IP addresses are attempting to brute force your databases. Thank you.
      0

    Please sign in to leave a comment.

    Didn't find what you were looking for?

    New post