MySQL Monitoring failed to detect partial outage
Twice today MySQL crashed, recovered fine the first, but not after the second. No cPanel server monitoring detected the condition, so no automated recovery took place. There are a few log entries that perhaps should be monitored for ... why wouldn't something like 'Fatal error' be detected ? These in combination indicate an issue despite MySQL appearing to be up.
Here are the relevant lines from the second entry indicating an issue
(from /var/lib/mysql/%HOSTNAME%.err )
InnoDB: mmap(136806400 bytes) failed; errno 12
160827 10:38:10 InnoDB: Fatal error: cannot allocate memory for the buffer pool
160827 10:38:10 [ERROR] Plugin 'InnoDB' init function returned error.
160827 10:38:10 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
Here are the 2 full log entries:
160827 9:20:05160827 9:20:05 InnoDB: Unable to allocate memory of size 14909544.
160827 9:20:05 InnoDB: Assertion failure in thread 2751060848 in file mem0mem.c line 361
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to MySQL Bugs.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: MySQL :: MySQL 5.5 Reference Manual :: 14.23.2 Forcing InnoDB Recovery
InnoDB: about forcing recovery.
InnoDB: Unable to allocate memory of size 14909544.
160827 9:20:05 InnoDB: Assertion failure in thread 2755455856 in file mem0mem.c line 361
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to MySQL Bugs.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: MySQL :: MySQL 5.5 Reference Manual :: 14.23.2 Forcing InnoDB Recovery
InnoDB: about forcing recovery.
160827 09:20:05 mysqld_safe Number of processes running now: 0
160827 09:20:05 mysqld_safe mysqld restarted
160827 9:20:05 [Note] /usr/sbin/mysqld (mysqld 5.5.50-cll) starting as process 15494 ...
160827 9:20:05 [Note] Plugin 'FEDERATED' is disabled.
160827 9:20:05 InnoDB: The InnoDB memory heap is disabled
160827 9:20:05 InnoDB: Mutexes and rw_locks use InnoDB's own implementation
160827 9:20:05 InnoDB: Compressed tables use zlib 1.2.3
160827 9:20:05 InnoDB: Using Linux native AIO
160827 9:20:05 InnoDB: Initializing buffer pool, size = 128.0M
160827 9:20:06 InnoDB: Completed initialization of buffer pool
160827 9:20:06 InnoDB: highest supported file format is Barracuda.
InnoDB: Log scan progressed past the checkpoint lsn 21388284244
160827 9:20:06 InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
InnoDB: Doing recovery: scanned up to log sequence number 21388305604
160827 9:20:13 InnoDB: Starting an apply batch of log records to the database...
InnoDB: Progress in percents: 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99
InnoDB: Apply batch completed
160827 9:20:14 InnoDB: Waiting for the background threads to start
160827 9:20:15 InnoDB: 5.5.50 started; log sequence number 21388305604
160827 9:20:15 [Note] Server hostname (bind-address): '0.0.0.0'; port: 3306
160827 9:20:15 [Note] - '0.0.0.0' resolves to '0.0.0.0';
160827 9:20:15 [Note] Server socket created on IP: '0.0.0.0'.
160827 9:20:15 [Note] Event Scheduler: Loaded 0 events
160827 9:20:15 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.5.50-cll' socket: '/var/lib/mysql/mysql.sock' port: 3306 MySQL Community Server (GPL)
=====================================================================
160827 10:38:10160827 10:38:10 InnoDB: Unable to allocate memory of size 14909544.
160827 10:38:10 InnoDB: Assertion failure in thread 2673179504 in file mem0mem.c line 361
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to MySQL Bugs.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: MySQL :: MySQL 5.5 Reference Manual :: 14.23.2 Forcing InnoDB Recovery
InnoDB: about forcing recovery.
InnoDB: Unable to allocate memory of size 14909544.
160827 10:38:10 InnoDB: Assertion failure in thread 2743319408 in file mem0mem.c line 361
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to MySQL Bugs.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: MySQL :: MySQL 5.5 Reference Manual :: 14.23.2 Forcing InnoDB Recovery
InnoDB: about forcing recovery.
160827 10:38:10 mysqld_safe Number of processes running now: 0
160827 10:38:10 mysqld_safe mysqld restarted
160827 10:38:10 [Note] /usr/sbin/mysqld (mysqld 5.5.50-cll) starting as process 27807 ...
160827 10:38:10 [Note] Plugin 'FEDERATED' is disabled.
160827 10:38:10 InnoDB: The InnoDB memory heap is disabled
160827 10:38:10 InnoDB: Mutexes and rw_locks use InnoDB's own implementation
160827 10:38:10 InnoDB: Compressed tables use zlib 1.2.3
160827 10:38:10 InnoDB: Using Linux native AIO
160827 10:38:10 InnoDB: Initializing buffer pool, size = 128.0M
InnoDB: mmap(136806400 bytes) failed; errno 12
160827 10:38:10 InnoDB: Completed initialization of buffer pool
160827 10:38:10 InnoDB: Fatal error: cannot allocate memory for the buffer pool
160827 10:38:10 [ERROR] Plugin 'InnoDB' init function returned error.
160827 10:38:10 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
160827 10:38:10 [Note] Server hostname (bind-address): '0.0.0.0'; port: 3306
160827 10:38:10 [Note] - '0.0.0.0' resolves to '0.0.0.0';
160827 10:38:10 [Note] Server socket created on IP: '0.0.0.0'.
160827 10:38:11 [Note] Event Scheduler: Loaded 0 events
160827 10:38:11 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.5.50-cll' socket: '/var/lib/mysql/mysql.sock' port: 3306 MySQL Community Server (GPL)-
Hello, The log output shows that MySQL started, but with InnoDB corruption. Chkservd currently determines if the MySQL process is running, but it doesn't issue notifications when InnoDB is corrupted. The following thread aims to help users resolve issues with InnoDB: InnoDB Corruption Repair Guide We are currently exploring options to detect and notify users about InnoDB corruption, but there's no specific time frame to offer on that feature at this time. Thank you. 0
Please sign in to leave a comment.
Comments
1 comment