MySQL crash - log files
Hi there,
I'm trying to figure out why our MySQL server went down last night.
I found a lot of errors in our logs.
Here is one of the most important errors:
SQLSTATE[HY000]: General error: 2006 MySQL server has gone away
I did as Bluehost suggested and checked the error file here:
/var/lib/mysql/username.err
and read this thread from the past:
VPS Down What logs file need to be checked
Here are only some of the errors found in the log. I'm hoping you can help tell me what caused the crash..
2017-11-17 02:55:46 14876 [ERROR] InnoDB: Unable to lock ./ibdata1, error: 11
2017-11-17 02:55:46 14876 [Note] InnoDB: Check that you do not already have another mysqld process using the same InnoDB data or log files.
2017-11-17 02:55:47 14876 [ERROR] InnoDB: Unable to lock ./ibdata1, error: 11
2017-11-17 02:55:47 14876 [Note] InnoDB: Check that you do not already have another mysqld process using the same InnoDB data or log files.
2017-11-17 02:55:48 14876 [ERROR] InnoDB: Unable to lock ./ibdata1, error: 11
2017-11-17 02:55:48 14876 [Note] InnoDB: Check that you do not already have another mysqld process using the same InnoDB data or log files.
2017-11-17 02:55:49 14876 [ERROR] InnoDB: Unable to lock ./ibdata1, error: 11
2017-11-17 02:55:49 14876 [Note] InnoDB: Check that you do not already have another mysqld process using the same InnoDB data or log files.
2017-11-17 02:55:50 14876 [ERROR] InnoDB: Unable to lock ./ibdata1, error: 11
2017-11-17 02:55:50 14876 [Note] InnoDB: Check that you do not already have another mysqld process using the same InnoDB data or log files.
2017-11-17 02:55:51 14876 [ERROR] InnoDB: Unable to lock ./ibdata1, error: 11
2017-11-17 02:55:51 14876 [Note] InnoDB: Check that you do not already have another mysqld process using the same InnoDB data or log files.
2017-11-17 02:55:52 14876 [ERROR] InnoDB: Unable to lock ./ibdata1, error: 11
2017-11-17 02:55:52 14876 [Note] InnoDB: Check that you do not already have another mysqld process using the same InnoDB data or log files.
2017-11-17 02:55:53 14876 [ERROR] InnoDB: Unable to lock ./ibdata1, error: 11
2017-11-17 02:55:53 14876 [Note] InnoDB: Check that you do not already have another mysqld process using the same InnoDB data or log files.
2017-11-17 02:55:54 14876 [ERROR] InnoDB: Unable to lock ./ibdata1, error: 11
2017-11-17 02:55:54 14876 [Note] InnoDB: Check that you do not already have another mysqld process using the same InnoDB data or log files.
2017-11-17 02:55:55 14876 [ERROR] InnoDB: Unable to lock ./ibdata1, error: 11
2017-11-17 02:55:55 14876 [Note] InnoDB: Check that you do not already have another mysqld process using the same InnoDB data or log files.
2017-11-17 02:55:56 14876 [ERROR] InnoDB: Unable to lock ./ibdata1, error: 11
2017-11-17 02:55:56 14876 [Note] InnoDB: Check that you do not already have another mysqld process using the same InnoDB data or log files.
2017-11-17 02:55:57 14876 [ERROR] InnoDB: Unable to lock ./ibdata1, error: 11
2017-11-17 02:55:57 14876 [Note] InnoDB: Check that you do not already have another mysqld process using the same InnoDB data or log files.
2017-11-17 02:55:58 14876 [ERROR] InnoDB: Unable to lock ./ibdata1, error: 11
2017-11-17 02:55:58 14876 [Note] InnoDB: Check that you do not already have another mysqld process using the same InnoDB data or log files.
2017-11-17 02:55:59 14876 [ERROR] InnoDB: Unable to lock ./ibdata1, error: 11
2017-11-17 02:55:59 14876 [Note] InnoDB: Check that you do not already have another mysqld process using the same InnoDB data or log files.
2017-11-17 02:56:00 14876 [ERROR] InnoDB: Unable to lock ./ibdata1, error: 11
2017-11-17 02:56:00 14876 [Note] InnoDB: Check that you do not already have another mysqld process using the same InnoDB data or log files.
2017-11-17 02:56:01 14876 [ERROR] InnoDB: Unable to lock ./ibdata1, error: 11
2017-11-17 02:56:01 14876 [Note] InnoDB: Check that you do not already have another mysqld process using the same InnoDB data or log files.
2017-11-17 02:56:02 14876 [ERROR] InnoDB: Unable to lock ./ibdata1, error: 11
2017-11-17 02:56:02 14876 [Note] InnoDB: Check that you do not already have another mysqld process using the same InnoDB data or log files.
2017-11-17 02:56:03 14876 [ERROR] InnoDB: Unable to lock ./ibdata1, error: 11
2017-11-17 02:56:03 14876 [Note] InnoDB: Check that you do not already have another mysqld process using the same InnoDB data or log files.
2017-11-17 02:56:04 14876 [ERROR] InnoDB: Unable to lock ./ibdata1, error: 11
2017-11-17 02:56:04 14876 [Note] InnoDB: Check that you do not already have another mysqld process using the same InnoDB data or log files.
2017-11-17 02:56:05 14876 [ERROR] InnoDB: Unable to lock ./ibdata1, error: 11
2017-11-17 02:56:05 14876 [Note] InnoDB: Check that you do not already have another mysqld process using the same InnoDB data or log files.
2017-11-17 02:56:06 14876 [ERROR] InnoDB: Unable to lock ./ibdata1, error: 11
2017-11-17 02:56:06 14876 [Note] InnoDB: Check that you do not already have another mysqld process using the same InnoDB data or log files.
2017-11-17 02:56:07 14876 [ERROR] InnoDB: Unable to lock ./ibdata1, error: 11
2017-11-17 02:56:07 14876 [Note] InnoDB: Check that you do not already have another mysqld process using the same InnoDB data or log files.
2017-11-17 02:56:08 14876 [ERROR] InnoDB: Unable to lock ./ibdata1, error: 11
2017-11-17 02:56:08 14876 [Note] InnoDB: Check that you do not already have another mysqld process using the same InnoDB data or log files.
2017-11-17 02:56:09 14876 [ERROR] InnoDB: Unable to lock ./ibdata1, error: 11
2017-11-17 02:56:09 14876 [Note] InnoDB: Check that you do not already have another mysqld process using the same InnoDB data or log files.
2017-11-17 02:56:10 14876 [ERROR] InnoDB: Unable to lock ./ibdata1, error: 11
2017-11-17 02:56:10 14876 [Note] InnoDB: Check that you do not already have another mysqld process using the same InnoDB data or log files.
2017-11-17 02:56:11 14876 [ERROR] InnoDB: Unable to lock ./ibdata1, error: 11
2017-11-17 02:56:11 14876 [Note] InnoDB: Check that you do not already have another mysqld process using the same InnoDB data or log files.
2017-11-17 02:56:12 14876 [ERROR] InnoDB: Unable to lock ./ibdata1, error: 11
2017-11-17 02:56:12 14876 [Note] InnoDB: Check that you do not already have another mysqld process using the same InnoDB data or log files.
2017-11-17 02:56:13 14876 [ERROR] InnoDB: Unable to lock ./ibdata1, error: 11
2017-11-17 02:56:13 14876 [Note] InnoDB: Check that you do not already have another mysqld process using the same InnoDB data or log files.
2017-11-17 02:56:14 14876 [ERROR] InnoDB: Unable to lock ./ibdata1, error: 11
2017-11-17 02:56:14 14876 [Note] InnoDB: Check that you do not already have another mysqld process using the same InnoDB data or log files.
2017-11-17 02:56:15 14876 [ERROR] InnoDB: Unable to lock ./ibdata1, error: 11
2017-11-17 02:56:15 14876 [Note] InnoDB: Check that you do not already have another mysqld process using the same InnoDB data or log files.
2017-11-17 02:56:16 14876 [ERROR] InnoDB: Unable to lock ./ibdata1, error: 11
2017-11-17 02:56:16 14876 [Note] InnoDB: Check that you do not already have another mysqld process using the same InnoDB data or log files.
2017-11-17 02:56:17 14876 [ERROR] InnoDB: Unable to lock ./ibdata1, error: 11
2017-11-17 02:56:17 14876 [Note] InnoDB: Check that you do not already have another mysqld process using the same InnoDB data or log files.
2017-11-17 02:56:18 14876 [ERROR] InnoDB: Unable to lock ./ibdata1, error: 11
2017-11-17 02:56:18 14876 [Note] InnoDB: Check that you do not already have another mysqld process using the same InnoDB data or log files.
2017-11-17 02:56:19 14876 [ERROR] InnoDB: Unable to lock ./ibdata1, error: 11
2017-11-17 02:56:19 14876 [Note] InnoDB: Check that you do not already have another mysqld process using the same InnoDB data or log files.
2017-11-17 02:56:20 14876 [ERROR] InnoDB: Unable to lock ./ibdata1, error: 11
2017-11-17 02:56:20 14876 [Note] InnoDB: Check that you do not already have another mysqld process using the same InnoDB data or log files.
2017-11-17 02:56:21 14876 [ERROR] InnoDB: Unable to lock ./ibdata1, error: 11
2017-11-17 02:56:21 14876 [Note] InnoDB: Check that you do not already have another mysqld process using the same InnoDB data or log files.
2017-11-17 02:56:22 14876 [ERROR] InnoDB: Unable to lock ./ibdata1, error: 11
2017-11-17 02:56:22 14876 [Note] InnoDB: Check that you do not already have another mysqld process using the same InnoDB data or log files.
2017-11-17 02:56:23 14876 [ERROR] InnoDB: Unable to lock ./ibdata1, error: 11
2017-11-17 02:56:23 14876 [Note] InnoDB: Check that you do not already have another mysqld process using the same InnoDB data or log files.
2017-11-17 02:56:23 14876 [Note] InnoDB: Unable to open the first data file
2017-11-17 02:56:23 7f62dd70a720 InnoDB: Operating system error number 11 in a file operation.
InnoDB: Error number 11 means 'Resource temporarily unavailable'.
InnoDB: Some operating system error numbers are described at
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/operating-system-error-codes.html
2017-11-17 02:56:23 14876 [ERROR] InnoDB: Can't open './ibdata1'
2017-11-17 02:56:23 14876 [ERROR] InnoDB: Could not open or create the system tablespace. If you tried to add new data files to the system tablespace, and it failed here, you should now edit innodb_data_file_path in my.cnf back to what it was, and remove the new ibdata files InnoDB created in this failed attempt. InnoDB only wrote those files full of zeros, but did not yet use them in any way. But be careful: do not remove old data files which contain your precious data!
2017-11-17 02:56:23 14876 [ERROR] Plugin 'InnoDB' init function returned error.
2017-11-17 02:56:23 14876 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
2017-11-17 02:56:23 14876 [ERROR] Unknown/unsupported storage engine: InnoDB
2017-11-17 02:56:23 14876 [ERROR] Aborting
2017-11-17 02:56:23 14876 [Note] Binlog end
2017-11-17 02:56:23 14876 [Note] Shutting down plugin 'partition'
2017-11-17 02:56:23 14876 [Note] Shutting down plugin 'ARCHIVE'
2017-11-17 02:56:23 14876 [Note] Shutting down plugin 'INNODB_SYS_DATAFILES'
2017-11-17 02:56:23 14876 [Note] Shutting down plugin 'INNODB_SYS_TABLESPACES'
2017-11-17 02:56:23 14876 [Note] Shutting down plugin 'INNODB_SYS_FOREIGN_COLS'
2017-11-17 02:56:23 14876 [Note] Shutting down plugin 'INNODB_SYS_FOREIGN'
2017-11-17 02:56:23 14876 [Note] Shutting down plugin 'INNODB_SYS_FIELDS'
2017-11-17 02:56:23 14876 [Note] Shutting down plugin 'INNODB_SYS_COLUMNS'
2017-11-17 02:56:23 14876 [Note] Shutting down plugin 'INNODB_SYS_INDEXES'
2017-11-17 02:56:23 14876 [Note] Shutting down plugin 'INNODB_SYS_TABLESTATS'
2017-11-17 02:56:23 14876 [Note] Shutting down plugin 'INNODB_SYS_TABLES'
2017-11-17 02:56:23 14876 [Note] Shutting down plugin 'INNODB_FT_INDEX_TABLE'
2017-11-17 02:56:23 14876 [Note] Shutting down plugin 'INNODB_FT_INDEX_CACHE'
2017-11-17 02:56:23 14876 [Note] Shutting down plugin 'INNODB_FT_CONFIG'
2017-11-17 02:56:23 14876 [Note] Shutting down plugin 'INNODB_FT_BEING_DELETED'
2017-11-17 02:56:23 14876 [Note] Shutting down plugin 'INNODB_FT_DELETED'
2017-11-17 02:56:23 14876 [Note] Shutting down plugin 'INNODB_FT_DEFAULT_STOPWORD'
2017-11-17 02:56:23 14876 [Note] Shutting down plugin 'INNODB_METRICS'
2017-11-17 02:56:23 14876 [Note] Shutting down plugin 'INNODB_BUFFER_POOL_STATS'
2017-11-17 02:56:23 14876 [Note] Shutting down plugin 'INNODB_BUFFER_PAGE_LRU'
2017-11-17 02:56:23 14876 [Note] Shutting down plugin 'INNODB_BUFFER_PAGE'
2017-11-17 02:56:23 14876 [Note] Shutting down plugin 'INNODB_CMP_PER_INDEX_RESET'
2017-11-17 02:56:23 14876 [Note] Shutting down plugin 'INNODB_CMP_PER_INDEX'
2017-11-17 02:56:23 14876 [Note] Shutting down plugin 'INNODB_CMPMEM_RESET'
2017-11-17 02:56:23 14876 [Note] Shutting down plugin 'INNODB_CMPMEM'
2017-11-17 02:56:23 14876 [Note] Shutting down plugin 'INNODB_CMP_RESET'
2017-11-17 02:56:23 14876 [Note] Shutting down plugin 'INNODB_CMP'
2017-11-17 02:56:23 14876 [Note] Shutting down plugin 'INNODB_LOCK_WAITS'
2017-11-17 02:56:23 14876 [Note] Shutting down plugin 'INNODB_LOCKS'
2017-11-17 02:56:23 14876 [Note] Shutting down plugin 'INNODB_TRX'
2017-11-17 02:56:23 14876 [Note] Shutting down plugin 'BLACKHOLE'
2017-11-17 02:56:23 14876 [Note] Shutting down plugin 'PERFORMANCE_SCHEMA'
2017-11-17 02:56:23 14876 [Note] Shutting down plugin 'MRG_MYISAM'
2017-11-17 02:56:23 14876 [Note] Shutting down plugin 'CSV'
2017-11-17 02:56:23 14876 [Note] Shutting down plugin 'MEMORY'
2017-11-17 02:56:23 14876 [Note] Shutting down plugin 'MyISAM'
2017-11-17 02:56:23 14876 [Note] Shutting down plugin 'sha256_password'
2017-11-17 02:56:23 14876 [Note] Shutting down plugin 'mysql_old_password'
2017-11-17 02:56:23 14876 [Note] Shutting down plugin 'mysql_native_password'
2017-11-17 02:56:23 14876 [Note] Shutting down plugin 'binlog'
2017-11-17 02:56:23 14876 [Note] /usr/sbin/mysqld: Shutdown complete
2017-11-17 02:56:23 15208 [Note] Plugin 'FEDERATED' is disabled.
2017-11-17 02:56:23 15208 [Note] InnoDB: Using atomics to ref count buffer pool pages
2017-11-17 02:56:23 15208 [Note] InnoDB: The InnoDB memory heap is disabled
2017-11-17 02:56:23 15208 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2017-11-17 02:56:23 15208 [Note] InnoDB: Memory barrier is not used
2017-11-17 02:56:23 15208 [Note] InnoDB: Compressed tables use zlib 1.2.3
2017-11-17 02:56:23 15208 [Note] InnoDB: Using Linux native AIO
2017-11-17 02:56:23 15208 [Note] InnoDB: Using CPU crc32 instructions
2017-11-17 02:56:23 15208 [Note] InnoDB: Initializing buffer pool, size = 128.0M
2017-11-17 02:56:23 15208 [Note] InnoDB: Completed initialization of buffer pool
2017-11-17 02:56:23 15208 [Note] InnoDB: Highest supported file format is Barracuda.
2017-11-17 02:56:23 15208 [Note] InnoDB: Log scan progressed past the checkpoint lsn 23828904506
2017-11-17 02:56:23 15208 [Note] InnoDB: Database was not shutdown normally!
2017-11-17 02:56:23 15208 [Note] InnoDB: Starting crash recovery.
2017-11-17 02:56:23 15208 [Note] InnoDB: Reading tablespace information from the .ibd files...
2017-11-17 02:56:23 15208 [Note] InnoDB: Restoring possible half-written data pages
2017-11-17 02:56:23 15208 [Note] InnoDB: from the doublewrite buffer...
InnoDB: Doing recovery: scanned up to log sequence number 23828927657
2017-11-17 02:56:23 15208 [Note] InnoDB: Starting an apply batch of log records to the database...
InnoDB: Progress in percent: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99
InnoDB: Apply batch completed
2017-11-17 02:56:24 15208 [Note] InnoDB: 128 rollback segment(s) are active.
2017-11-17 02:56:24 15208 [Note] InnoDB: Waiting for purge to start
2017-11-17 02:56:24 15208 [Note] InnoDB: 5.6.35 started; log sequence number 23828927657
2017-11-17 02:56:24 15208 [Note] Server hostname (bind-address): '*'; port: 3306
2017-11-17 02:56:24 15208 [Note] IPv6 is available.
2017-11-17 02:56:24 15208 [Note] - '::' resolves to '::';
2017-11-17 02:56:24 15208 [Note] Server socket created on IP: '::'.
2017-11-17 02:56:24 15208 [Note] Event Scheduler: Loaded 0 events
2017-11-17 02:56:24 15208 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.6.35' socket: '/var/lib/mysql/mysql.sock' port: 3306 MySQL Community Server (GPL)
2017-11-17 02:56:24 15208 [ERROR] /usr/sbin/mysqld: Table './topsect2_wrdp1/wp_options' is marked as crashed and should be repaired
2017-11-17 02:56:24 15208 [Warning] Checking table: './topsect2_wrdp1/wp_options'
2017-11-17 02:57:59 15208 [ERROR] /usr/sbin/mysqld: Table './topsect2_wrdp1/wp_posts' is marked as crashed and should be repaired
2017-11-17 02:57:59 15208 [Warning] Checking table: './topsect2_wrdp1/wp_posts'
2017-11-17 02:58:00 15208 [ERROR] /usr/sbin/mysqld: Table './topsect2_wrdp1/wp_postmeta' is marked as crashed and should be repaired
2017-11-17 02:58:00 15208 [Warning] Checking table: './topsect2_wrdp1/wp_postmeta'
2017-11-17 03:00:39 15208 [ERROR] /usr/sbin/mysqld: Table './topsect2_wrdp1/wp_yoast_seo_meta' is marked as crashed and should be repaired
2017-11-17 03:00:39 15208 [Warning] Checking table: './topsect2_wrdp1/wp_yoast_seo_meta'
2017-11-17 04:20:08 15208 [Warning] IP address '49.4.140.65' could not be resolved: Name or service not known
2017-11-17 08:41:40 15208 [Warning] IP address '121.207.227.164' could not be resolved: Name or service not known
2017-11-17 14:22:49 15208 [ERROR] /usr/sbin/mysqld: Table './topsect2_xc4/xcart_config' is marked as crashed and should be repaired
2017-11-17 14:22:49 15208 [Warning] Checking table: './topsect2_xc4/xcart_config'
2017-11-17 14:22:49 15208 [ERROR] /usr/sbin/mysqld: Table './topsect2_xc4/xcart_form_ids' is marked as crashed and should be repaired
2017-11-17 14:22:49 15208 [Warning] Checking table: './topsect2_xc4/xcart_form_ids'
2017-11-17 14:22:49 15208 [ERROR] /usr/sbin/mysqld: Table './topsect2_xc4/xcart_session_history' is marked as crashed and should be repaired
2017-11-17 14:22:49 15208 [Warning] Checking table: './topsect2_xc4/xcart_session_history'
2017-11-17 14:22:50 15208 [ERROR] /usr/sbin/mysqld: Table './topsect2_xc4/xcart_session_unknown_sid' is marked as crashed and should be repaired
2017-11-17 14:22:50 15208 [Warning] Checking table: './topsect2_xc4/xcart_session_unknown_sid'
2017-11-17 14:30:51 15208 [ERROR] /usr/sbin/mysqld: Table './topsect2_xc4/xcart_sessions_data' is marked as crashed and should be repaired
2017-11-17 14:30:51 15208 [Warning] Checking table: './topsect2_xc4/xcart_sessions_data'
2017-11-17 14:30:52 15208 [ERROR] /usr/sbin/mysqld: Table './topsect2_wrdp1/wp_yumprint_recipe_view' is marked as crashed and should be repaired
2017-11-17 14:30:52 15208 [Warning] Checking table: './topsect2_wrdp1/wp_yumprint_recipe_view'
-
If you suspect the InnoDB has become corrupted, this guide might help InnoDB Corruption Repair Guide 0 -
Hello, The log output suggests potential InnoDB corruption. The link referenced in the previous post is a good place to start. Thank you. 0
Please sign in to leave a comment.
Comments
2 comments