MySQL restarts without InnoDB

robertjw

• August 21, 2015 14:02

Occasionally, once a week or so, my server will be hit with a brute force attack that sucks up all it's memory. When this happens MySQL will crash and restart. The problem is that when it restarts there often won't be enough memory available to restart InnoDB, but MySQL will restart without InnoDB running. Here is an excerpt from the log file: 150821 12:07:59 [Note] InnoDB: Initializing buffer pool, size = 119.0M InnoDB: mmap(127713280 bytes) failed; errno 12 150821 12:07:59 [ERROR] InnoDB: Cannot allocate memory for the buffer pool 150821 12:07:59 [ERROR] Plugin 'InnoDB' init function returned error. 150821 12:07:59 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed. 150821 12:07:59 [ERROR] mysqld: Out of memory (Needed 128909312 bytes) 150821 12:07:59 [Note] Plugin 'FEEDBACK' is disabled. 150821 12:07:59 [Note] Server socket created on IP: '::'. I have one client using a plugin that requires InnoDB, otherwise almost everything else humms along without a problem. So, why doesn't MySQL fail? Can I control this? Can I setup some kind of watchdog that will check if InnoDB is running and either notify me or restart MySQL if it's not?

• 

  24x7server

  • August 22, 2015 06:09

  Hello, Can you please provide use "free -m" command output from your server, And try to increase your server swap space and check if you are getting same issues.

  0
• 

  cPanelMichael

  • August 31, 2015 18:40

  Hello :) I believe the best way to address this issue is to implement firewall rules to prevent the brute force attacks that are taking up your memory. Have you installed a firewall management utility such as CSF to help prevent these types of attacks? Thank you.

  0
• 

  robertjw

  • September 08, 2015 22:33

  Yes, I have CSF installed and ModSecurity enabled with the Comodo WAF plugin. This has almost eliminated the problem, but it still happens about once a week. The server isn't configured with any swap space. free -m total used free shared buffers cached Mem: 3832 3582 250 4 82 2201 -/+ buffers/cache:1298 2533 Swap: 0 0 0 My hosting company set it up this way, and I'm not sure that adding swap is a good idea. If an attack uses up all the swap space too and MySQL crashes, won't I have the same issue?

  0
• 

  cPanelMichael

  • September 09, 2015 18:28

  My hosting company set it up this way, and I'm not sure that adding swap is a good idea. If an attack uses up all the swap space too and MySQL crashes, won't I have the same issue?

  
  You may simply need to add additional physical memory to your system if it's current available memory is not sufficient to handle these types of attacks. Or, you may want to consult with a system administrator to see if these attacks are preventable through other custom firewall rules. Thank you.

  0
• 

  robertjw

  • September 10, 2015 00:51

  My problem isn't really with the attacks. They are a rare occurrence, and I'm not sure how much additional memory would be needed to make sure this never happens. My problem is why MySQL comes up and runs in a failed state? Is there a configuration for MySQL where it would crash if it can't start InnoDB? Is there a way to test for this externally?

  0
• 

  cPanelMichael

  • September 10, 2015 15:49

  Is there a configuration for MySQL where it would crash if it can't start InnoDB?

  
  You should be able to add the following line in your /etc/my.cnf file if you want MySQL to fail to start if InnoDB fails:
  innodb=force
  Thank you.

  0

Please sign in to leave a comment.