MySQL Upgrade Failed and not Starting

Minutes back I initiated the MySQL 5.6 Upgrade. It got stuck in the middle and not starting again. MySQL is completely down on the server and below is the err log

150625 04:18:35 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
2015-06-25 04:18:36 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2015-06-25 04:18:36 24137 [Note] Plugin 'FEDERATED' is disabled.
2015-06-25 04:18:36 24137 [Note] InnoDB: Using atomics to ref count buffer pool pages
2015-06-25 04:18:36 24137 [Note] InnoDB: The InnoDB memory heap is disabled
2015-06-25 04:18:36 24137 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2015-06-25 04:18:36 24137 [Note] InnoDB: Memory barrier is not used
2015-06-25 04:18:36 24137 [Note] InnoDB: Compressed tables use zlib 1.2.3
2015-06-25 04:18:36 24137 [Note] InnoDB: Using Linux native AIO
2015-06-25 04:18:36 24137 [Note] InnoDB: Using CPU crc32 instructions
2015-06-25 04:18:36 24137 [Note] InnoDB: Initializing buffer pool, size = 128.0M
2015-06-25 04:18:36 24137 [Note] InnoDB: Completed initialization of buffer pool
2015-06-25 04:18:36 24137 [Note] InnoDB: Highest supported file format is Barracuda.
2015-06-25 04:18:36 24137 [Note] InnoDB: The log sequence numbers 3057436588 and 3057436588 in ibdata files do not match the log sequence number 3330449883 in the ib_logfiles!
2015-06-25 04:18:36 24137 [Note] InnoDB: Database was not shutdown normally!
2015-06-25 04:18:36 24137 [Note] InnoDB: Starting crash recovery.
2015-06-25 04:18:36 24137 [Note] InnoDB: Reading tablespace information from the .ibd files...
2015-06-25 04:18:36 24137 [Note] InnoDB: Restoring possible half-written data pages
2015-06-25 04:18:36 24137 [Note] InnoDB: from the doublewrite buffer...
08:18:36 UTC - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

key_buffer_size=8388608
read_buffer_size=131072
max_used_connections=0
max_threads=151
thread_count=0
connection_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 68245 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x40000
/usr/sbin/mysqld(my_print_stacktrace+0x35)[0x8cdd25]
/usr/sbin/mysqld(handle_fatal_signal+0x494)[0x661c54]
/lib64/libpthread.so.0(+0xf710)[0x7fa48959b710]
/usr/sbin/mysqld[0xaa41e4]
/usr/sbin/mysqld[0xaa5dfa]
/usr/sbin/mysqld[0xa236b2]
/usr/sbin/mysqld[0x96c77d]
/usr/sbin/mysqld(_Z24ha_initialize_handlertonP13st_plugin_int+0x48)[0x5a8588]
/usr/sbin/mysqld[0x6e7c51]
/usr/sbin/mysqld(_Z11plugin_initPiPPci+0xbf6)[0x6eb8a6]
/usr/sbin/mysqld[0x59b35d]
/usr/sbin/mysqld(_Z11mysqld_mainiPPc+0x3e5)[0x5a0305]
/lib64/libc.so.6(__libc_start_main+0xfd)[0x7fa488232d5d]
/usr/sbin/mysqld[0x5926a5]
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
150625 04:18:36 mysqld_safe mysqld from pid file /var/lib/mysql/linux001.annaiservers.com.pid ended
150625 04:18:54 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
2015-06-25 04:18:55 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2015-06-25 04:18:55 24296 [Note] Plugin 'FEDERATED' is disabled.
2015-06-25 04:18:55 24296 [Note] InnoDB: Using atomics to ref count buffer pool pages
2015-06-25 04:18:55 24296 [Note] InnoDB: The InnoDB memory heap is disabled
2015-06-25 04:18:55 24296 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2015-06-25 04:18:55 24296 [Note] InnoDB: Memory barrier is not used
2015-06-25 04:18:55 24296 [Note] InnoDB: Compressed tables use zlib 1.2.3
2015-06-25 04:18:55 24296 [Note] InnoDB: Using Linux native AIO
2015-06-25 04:18:55 24296 [Note] InnoDB: Using CPU crc32 instructions
2015-06-25 04:18:55 24296 [Note] InnoDB: Initializing buffer pool, size = 128.0M
2015-06-25 04:18:55 24296 [Note] InnoDB: Completed initialization of buffer pool
2015-06-25 04:18:55 24296 [Note] InnoDB: Highest supported file format is Barracuda.
2015-06-25 04:18:55 24296 [Note] InnoDB: The log sequence numbers 3057436588 and 3057436588 in ibdata files do not match the log sequence number 3330449883 in the ib_logfiles!
2015-06-25 04:18:55 24296 [Note] InnoDB: Database was not shutdown normally!
2015-06-25 04:18:55 24296 [Note] InnoDB: Starting crash recovery.
2015-06-25 04:18:55 24296 [Note] InnoDB: Reading tablespace information from the .ibd files...
2015-06-25 04:18:55 24296 [Note] InnoDB: Restoring possible half-written data pages
2015-06-25 04:18:55 24296 [Note] InnoDB: from the doublewrite buffer...
08:18:55 UTC - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

key_buffer_size=8388608
read_buffer_size=131072
max_used_connections=0
max_threads=151
thread_count=0
connection_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 68245 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x40000
/usr/sbin/mysqld(my_print_stacktrace+0x35)[0x8cdd25]
/usr/sbin/mysqld(handle_fatal_signal+0x494)[0x661c54]
/lib64/libpthread.so.0(+0xf710)[0x7fc0bbba3710]
/usr/sbin/mysqld[0xaa41e4]
/usr/sbin/mysqld[0xaa5dfa]
/usr/sbin/mysqld[0xa236b2]
/usr/sbin/mysqld[0x96c77d]
/usr/sbin/mysqld(_Z24ha_initialize_handlertonP13st_plugin_int+0x48)[0x5a8588]
/usr/sbin/mysqld[0x6e7c51]
/usr/sbin/mysqld(_Z11plugin_initPiPPci+0xbf6)[0x6eb8a6]
/usr/sbin/mysqld[0x59b35d]
/usr/sbin/mysqld(_Z11mysqld_mainiPPc+0x3e5)[0x5a0305]
/lib64/libc.so.6(__libc_start_main+0xfd)[0x7fc0ba83ad5d]
/usr/sbin/mysqld[0x5926a5]
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
150625 04:18:55 mysqld_safe mysqld from pid file /var/lib/mysql/linux001.annaiservers.com.pid ended

Please help me to recover as my client sites were down...

Infopro

June 25, 2015 08:59

Please feel free to open a ticket to cPanel Technical Support directly for faster assistance with this issue.

Logesh K

June 25, 2015 10:34

Raised with Support Request Id 6740711 Thanks

June 25, 2015 11:11

Can anybody help me on this ? I didn't get any Update on my support ticket. My clients sites were already down for hours. Need to fix it at the earliest as possible.

cPanelMichael

July 02, 2015 13:34

Hello :) To update, the user was directed to the following guide in order to repair InnoDB on their system:

MySQL Upgrade Failed and not Starting

Comments

Didn't find what you were looking for?