Mysql cannot start
Hello,
Out of a sudden my mysql server cannot start.
Cloudlinux 8, mysql 8, cpanel 120.0.11
It seems an automated mysql update from version '80037' to '80038' didn't completed.
There is no error in the log just this:
2024-06-30T20:32:29.113796Z 1 [Note] [MY-013018] [InnoDB] Created 128 and tracked 128 new rollback segment(s) in the temporary tablespace. 128 are now active.
2024-06-30T20:32:29.114289Z 1 [Note] [MY-012976] [InnoDB] 8.0.38 started; log sequence number 1121459761877
2024-06-30T20:32:29.115038Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
2024-06-30T20:32:29.122602Z 1 [Note] [MY-013327] [Server] MySQL server upgrading from version '80037' to '80038'.
2024-06-30T20:32:29.303674Z 1 [Note] [MY-012357] [InnoDB] Reading DD tablespace files
2024-06-30T20:32:29Z UTC - mysqld got signal 11 ;
Most likely, you have hit a bug, but this error can also be caused by malfunctioning hardware.
What do you think?
-
I have the same issue!
1 -
Same issue here
1 -
AlmaLinux v8.10.0 STANDARD standardcPanel Version120.0.110
-
to fix, roll back failed update
yum downgrade mysql-community-*4 -
That fixed it, thanks!
1 -
Thanks!
yum downgrade mysql-community-*
fixed it for me!
1 -
Same issue here and same fix here. One hour sweating.
2 -
That fixed it.
Official:https://support.cpanel.net/hc/en-us/articles/24546938662039-Upstream-issue-MySQL-8-0-38-does-not-start-Segmentation-Fault
you might also want to versionlock the mysql-community-* for a while until the issue is resolved to prevent updating it again with the same result:https://support.cpanel.net/hc/en-us/articles/1500013004522-How-to-version-lock-RPM-packages
1 -
Same issue for us - the rollback worked and brought MySQL back online.
0 -
I faced the same issue
to prevent this issue I locked the mysql version by this command on almalinux
dnf install python3-dnf-plugin-versionlock
yum versionlock mysql-community-*2 -
Just confirming we have this issue as well. Not all servers failed the upgrade, some upgraded fine.
Both on AlmaLinux and Ubuntu 20.04.
We also downgraded, which also fixed it for us.
0 -
Fuuuuuu.... We were just doing an upgrade on a legacy server from mySQL 5.7 to mySQL 8.0 to start elevation to AlmaLinux and we ran in to this. I spent hours trying to debug what was going wrong and the inexplicable error with zero pointers and no search results got me to nuke the whole thing and start terminating and then restoring accounts from backups. Damn.
The timing is insane. Was just debating this afternoon: do it now or tomorrow? Nothing was published about this bug yet and we decided to do it today. Aaarghh. Oh well. Murphy's law.
Here's the strange thing by the way: nuking the mySQL datafolder instantly gets mySQL to start again. Just removing all databases isn't enough, but nuking the full contents of that dir gets mySQL back working and happy to restore databases as well. (So don't try this at home unless you have a backup of the datadir and full backups of the cPanel accounts. Its the only reason I even started experimenting with this.)0 -
Same Issue here.
AlmaLinux v8.10.0 STANDARD standardcPanel VersionSolved by rolling back with
yum downgrade mysql-community-*
0 -
Same here.
Solved with
yum downgrade mysql-community-*
0 -
2024-07-01T04:11:58.496989Z 0 [ERROR] [MY-010119] [Server] Aborting
2024-07-01T04:11:58.497325Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.37) MySQL Community Server - GPL.
2024-07-01T04:14:43.120413Z 0 [Warning] [MY-010918] [Server] 'default_authentication_plugin' is deprecated and will be removed in a future release. Please use authentication_policy instead.
2024-07-01T04:14:43.120425Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.37) starting as process 2461
2024-07-01T04:14:43.128669Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2024-07-01T04:14:44.792680Z 1 [Warning] [MY-013541] [InnoDB] Doublewrite page 114 for {space: 22705, page_no:2866} could not be restored. File name unknown for tablespace ID 22705
2024-07-01T04:14:44.792702Z 1 [Warning] [MY-013541] [InnoDB] Doublewrite page 166 for {space: 22909, page_no:133} could not be restored. File name unknown for tablespace ID 22909
2024-07-01T04:14:44.792708Z 1 [Warning] [MY-013541] [InnoDB] Doublewrite page 177 for {space: 22705, page_no:1508} could not be restored. File name unknown for tablespace ID 22705
2024-07-01T04:14:44.792713Z 1 [Warning] [MY-013541] [InnoDB] Doublewrite page 184 for {space: 22860, page_no:21} could not be restored. File name unknown for tablespace ID 22860
2024-07-01T04:14:44.792719Z 1 [Warning] [MY-013541] [InnoDB] Doublewrite page 227 for {space: 22705, page_no:338} could not be restored. File name unknown for tablespace ID 22705
2024-07-01T04:14:44.792728Z 1 [Warning] [MY-013541] [InnoDB] Doublewrite page 235 for {space: 22705, page_no:928} could not be restored. File name unknown for tablespace ID 22705
2024-07-01T04:14:44.792734Z 1 [Warning] [MY-013541] [InnoDB] Doublewrite page 285 for {space: 22860, page_no:5} could not be restored. File name unknown for tablespace ID 22860
2024-07-01T04:14:44.792739Z 1 [Warning] [MY-013541] [InnoDB] Doublewrite page 331 for {space: 22860, page_no:21} could not be restored. File name unknown for tablespace ID 22860
2024-07-01T04:14:44.792749Z 1 [ERROR] [MY-012179] [InnoDB] Could not find any file associated with the tablespace ID: 22705
2024-07-01T04:14:44.792761Z 1 [ERROR] [MY-012179] [InnoDB] Could not find any file associated with the tablespace ID: 22733
2024-07-01T04:14:44.792766Z 1 [ERROR] [MY-012179] [InnoDB] Could not find any file associated with the tablespace ID: 22742
2024-07-01T04:14:44.792772Z 1 [ERROR] [MY-012179] [InnoDB] Could not find any file associated with the tablespace ID: 22744
2024-07-01T04:14:44.792787Z 1 [ERROR] [MY-012179] [InnoDB] Could not find any file associated with the tablespace ID: 22776
2024-07-01T04:14:44.792792Z 1 [ERROR] [MY-012179] [InnoDB] Could not find any file associated with the tablespace ID: 22860
2024-07-01T04:14:44.792797Z 1 [ERROR] [MY-012179] [InnoDB] Could not find any file associated with the tablespace ID: 22909
2024-07-01T04:14:44.792803Z 1 [ERROR] [MY-012964] [InnoDB] Use --innodb-directories to find the tablespace files. If that fails then use --innodb-force-recovery=1 to ignore this and to permanently lose all changes to the missing tablespace(s)
2024-07-01T04:14:44.893047Z 1 [ERROR] [MY-012930] [InnoDB] Plugin initialization aborted with error Generic error.
2024-07-01T04:14:45.293066Z 1 [ERROR] [MY-010334] [Server] Failed to initialize DD Storage Engine
2024-07-01T04:14:45.293188Z 0 [ERROR] [MY-010020] [Server] Data Dictionary initialization failed.
2024-07-01T04:14:45.293205Z 0 [ERROR] [MY-010119] [Server] Aborting
2024-07-01T04:14:45.294063Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.37) MySQL Community Server - GPL.
2024-07-01T04:15:44.058762Z 0 [Warning] [MY-010918] [Server] 'default_authentication_plugin' is deprecated and will be removed in a future release. Please use authentication_policy instead.
2024-07-01T04:15:44.058774Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.37) starting as process 3027
2024-07-01T04:15:44.062210Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2024-07-01T04:15:45.425570Z 1 [Warning] [MY-013541] [InnoDB] Doublewrite page 114 for {space: 22705, page_no:2866} could not be restored. File name unknown for tablespace ID 22705
2024-07-01T04:15:45.425595Z 1 [Warning] [MY-013541] [InnoDB] Doublewrite page 166 for {space: 22909, page_no:133} could not be restored. File name unknown for tablespace ID 22909
2024-07-01T04:15:45.425602Z 1 [Warning] [MY-013541] [InnoDB] Doublewrite page 177 for {space: 22705, page_no:1508} could not be restored. File name unknown for tablespace ID 22705
2024-07-01T04:15:45.425607Z 1 [Warning] [MY-013541] [InnoDB] Doublewrite page 184 for {space: 22860, page_no:21} could not be restored. File name unknown for tablespace ID 22860
2024-07-01T04:15:45.425612Z 1 [Warning] [MY-013541] [InnoDB] Doublewrite page 227 for {space: 22705, page_no:338} could not be restored. File name unknown for tablespace ID 22705
2024-07-01T04:15:45.425617Z 1 [Warning] [MY-013541] [InnoDB] Doublewrite page 235 for {space: 22705, page_no:928} could not be restored. File name unknown for tablespace ID 22705
2024-07-01T04:15:45.425622Z 1 [Warning] [MY-013541] [InnoDB] Doublewrite page 285 for {space: 22860, page_no:5} could not be restored. File name unknown for tablespace ID 22860
2024-07-01T04:15:45.425627Z 1 [Warning] [MY-013541] [InnoDB] Doublewrite page 331 for {space: 22860, page_no:21} could not be restored. File name unknown for tablespace ID 22860
2024-07-01T04:15:45.425634Z 1 [ERROR] [MY-012179] [InnoDB] Could not find any file associated with the tablespace ID: 22705
2024-07-01T04:15:45.425644Z 1 [ERROR] [MY-012179] [InnoDB] Could not find any file associated with the tablespace ID: 22733
2024-07-01T04:15:45.425649Z 1 [ERROR] [MY-012179] [InnoDB] Could not find any file associated with the tablespace ID: 22742
2024-07-01T04:15:45.425654Z 1 [ERROR] [MY-012179] [InnoDB] Could not find any file associated with the tablespace ID: 22744
2024-07-01T04:15:45.425659Z 1 [ERROR] [MY-012179] [InnoDB] Could not find any file associated with the tablespace ID: 22776
2024-07-01T04:15:45.425665Z 1 [ERROR] [MY-012179] [InnoDB] Could not find any file associated with the tablespace ID: 22860
2024-07-01T04:15:45.425669Z 1 [ERROR] [MY-012179] [InnoDB] Could not find any file associated with the tablespace ID: 22909
2024-07-01T04:15:45.425675Z 1 [ERROR] [MY-012964] [InnoDB] Use --innodb-directories to find the tablespace files. If that fails then use --innodb-force-recovery=1 to ignore this and to permanently lose all changes to the missing tablespace(s)
2024-07-01T04:15:45.525835Z 1 [ERROR] [MY-012930] [InnoDB] Plugin initialization aborted with error Generic error.
2024-07-01T04:15:45.925672Z 1 [ERROR] [MY-010334] [Server] Failed to initialize DD Storage Engine
2024-07-01T04:15:45.925815Z 0 [ERROR] [MY-010020] [Server] Data Dictionary initialization failed.
2024-07-01T04:15:45.925838Z 0 [ERROR] [MY-010119] [Server] Aborting
2024-07-01T04:15:45.926176Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.37) MySQL Community Server - GPL.-1 -
thanks its working now after downgrade SQL version
yum downgrade mysql-community-*
1 -
yup, downgrade worked for me as well!
I would have thought these things were tested before releasing in to the wild!
0 -
Hi, same issue for Ubuntu. Can someone provide the exact command for downgrading MySQL for Ubuntu 20.04.5 LTS?
0 -
Hi we had the same issue this morning on Ubuntu 20.04.6 LTS and fixed with
wget https://repo.mysql.com/apt/ubuntu/pool/mysql-8.0/m/mysql-community/mysql-community-server-core_8.0.37-1ubuntu20.04_amd64.deb
dpkg -i mysql-community-server-core_8.0.37-1ubuntu20.04_amd64.debthen marked those packages hold to prevent upgrades
apt-mark hold mysql-client mysql-common mysql-community-client mysql-community-client-core mysql-community-client-plugins mysql-community-server mysql-community-server-core
although maybe there is a better way.
0 -
The downgrade script worked, thank you!
yum downgrade mysql-community-*
1 -
Downgrade did not work for Cloudlinux for one server
0 -
Did
yum downgrade mysql-community-*
I am not able to restart after this getting this error
reached! at /usr/local/cpanel/Cpanel/RestartSrv/Lock.pm line 37.0 -
Had to uninstall mysql server, do a fresh reinstall and add databases from backup to get it running. phpMyAdmin not working after that. Starting a new server to move sites to that one instead but the licensing from cPanel is not working either.
0 -
Update, server just died despite reinstalling. No response from cPanel in either topic and licensing. This is the worst in years.
0 -
For all Ubuntu 20.04 users - SOLVED and TESTED
Just follow steps in the official article published by cPanel support 10 hours ago:
MySQL 8.0.38 was released upstream. After automatic updates, there are multiple reports that MySQL is not starting.
Workaround
- Downgrade to the previous subversion of MySQL 8:
It worked great and all old databases are visible and working.
We were down for 12 hours and 45 mins because of it. Wasted a lot of time trying other methods. I hope this message will save you time.
Good luck!
1 -
Same issue here… same fix. How can this happen?!
0 -
I'm on Cloudlinux 8.10 and had to downgrade this morning also.
Should we still lock MySQL version update at this time? Not sure how to lock the MySQL update.
0 -
To lock the version (and avoid the upgrade) run this
dnf install python3-dnf-plugin-versionlock
yum versionlock mysql-community-*2 -
no response from support on our side either, just a link to a fix that didn't work... gonna try this method now, thank you guys for being such an active group
1 -
I'm wondering 🤔
How a big company like cpanel make this mistake.
How they release an update which not well tested.We received about 50 phone calls at midnight from our clients.
0
Please sign in to leave a comment.
Comments
90 comments