Mysql cannot start
Hello,
Out of a sudden my mysql server cannot start.
Cloudlinux 8, mysql 8, cpanel 120.0.11
It seems an automated mysql update from version '80037' to '80038' didn't completed.
There is no error in the log just this:
2024-06-30T20:32:29.113796Z 1 [Note] [MY-013018] [InnoDB] Created 128 and tracked 128 new rollback segment(s) in the temporary tablespace. 128 are now active.
2024-06-30T20:32:29.114289Z 1 [Note] [MY-012976] [InnoDB] 8.0.38 started; log sequence number 1121459761877
2024-06-30T20:32:29.115038Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
2024-06-30T20:32:29.122602Z 1 [Note] [MY-013327] [Server] MySQL server upgrading from version '80037' to '80038'.
2024-06-30T20:32:29.303674Z 1 [Note] [MY-012357] [InnoDB] Reading DD tablespace files
2024-06-30T20:32:29Z UTC - mysqld got signal 11 ;
Most likely, you have hit a bug, but this error can also be caused by malfunctioning hardware.
What do you think?
-
to fix, roll back failed update
yum downgrade mysql-community-*4 -
Optimizr try to run MySQL password reset from WHM admin panel. It helped in my case.
If MySQL is running correctly and is recognized by WHM/cPanel then you should see MySQL processes.
Try the password reset and report your outcome here for others to save them time.
I wasted approximately 10 hours on various fixes including downgrading, in-place upgrade, purging and clean install... when the fix would take just several minutes if it was properly displayed on cPanel front page. Instead their fix, which I'm referencing above was backdated.
cPanel team - whatever you do at least monitor your own f$#% forum and respond quickly in case of emergency like that. Do you need crisis management training?
Your software package is mature and does excellent job and I am paying for both cPanel and DirectAdmin (among others) and will rate cPanel higher than DA. If you improve your support just a little bit you will win many more customers and they will be happy customers.
Now you have unhappy and frustrated admins looking for help. Where it is?
You should send e-mail immediately to all cPanel subscribers to warn them about the problem and how to fix it, even if it doesn't apply to them. Do it now!4 -
Stupid Cpanel upcp. WHY can't they block this update ?????
3 -
Same issue here and same fix here. One hour sweating.
2 -
I faced the same issue
to prevent this issue I locked the mysql version by this command on almalinux
dnf install python3-dnf-plugin-versionlock
yum versionlock mysql-community-*2 -
To lock the version (and avoid the upgrade) run this
dnf install python3-dnf-plugin-versionlock
yum versionlock mysql-community-*2 -
Mel Tallagsen, Why did you downgrade from 8.0.37 to 8.0.36, isn't 8.0.38 suppose to be the buggy version that crashed MySQL and 8.0.37 works okay (at least in our case), I'm a bit confused and curious now 🙄🤔
2 -
One of our dedicated servers also failed and this solve the issue, thanks for sharing:
Downgrade the problematic update:
yum downgrade mysql-community-*
Now lock the version to avoid a new failures:
yum install yum-versionlock
yum versionlock mysql-community-*
cPanel shuts down direct support for servers that didnt purchased their license directly, like our servers purchased from Bluehost. The Bluehost dedicated support team, does not even qualifies to be called support tem. We are all alone now.
Thanks to the community and CloudLinux support team, that point us in the right direction.2 -
Upgraded to 8.39 a few days ago. All working OK.
2 -
I have the same issue!
1 -
Same issue here
1 -
That fixed it, thanks!
1 -
Thanks!
yum downgrade mysql-community-*
fixed it for me!
1 -
That fixed it.
Official:https://support.cpanel.net/hc/en-us/articles/24546938662039-Upstream-issue-MySQL-8-0-38-does-not-start-Segmentation-Fault
you might also want to versionlock the mysql-community-* for a while until the issue is resolved to prevent updating it again with the same result:https://support.cpanel.net/hc/en-us/articles/1500013004522-How-to-version-lock-RPM-packages
1 -
thanks its working now after downgrade SQL version
yum downgrade mysql-community-*
1 -
The downgrade script worked, thank you!
yum downgrade mysql-community-*
1 -
For all Ubuntu 20.04 users - SOLVED and TESTED
Just follow steps in the official article published by cPanel support 10 hours ago:
MySQL 8.0.38 was released upstream. After automatic updates, there are multiple reports that MySQL is not starting.
Workaround
- Downgrade to the previous subversion of MySQL 8:
It worked great and all old databases are visible and working.
We were down for 12 hours and 45 mins because of it. Wasted a lot of time trying other methods. I hope this message will save you time.
Good luck!
1 -
no response from support on our side either, just a link to a fix that didn't work... gonna try this method now, thank you guys for being such an active group
1 -
Mel Tallagsen
Have you tried the downgrade? This seems to work for most people. If not: what happened instead of mySQL starting OK (and what's the exact error it's getting stuck on)?1 -
@Mel
Check your /etc/my.cnf and see if you have the following:
innodb_force_recovery =If you have - remove that line and restart the mysql server.
1 -
cPanel support team - can you please promote solution to this huge problem?
Quick fix:
1 -
Doing so can maybe will decrease the number of tickets?
Be more proactive guys. We admins have certain responsibilities and uptime is one thing we have to worry about. The latest release was not properly tested you know it know. But you can still turn this fiasco into an example of how bad support can be turned into great support.
An error doesn't become a mistake until you refuse to correct it.
- John F. Kennedy.1 -
I am absolutely amazed by the fast response i got once i set the support ticket for direct response...
I have downgraded the MySQL server for you:
yum downgrade mysql-community-* --disablerepo=base --disablerepo=centos-sclo-rh --disablerepo=centos-sclo-sclo --disablerepo=extras --disablerepo=updates
[TRUNCATED]
Removed:
mysql-community-client.x86_64 0:8.0.37-1.el7 mysql-community-client-plugins.x86_64 0:8.0.37-1.el7
mysql-community-common.x86_64 0:8.0.37-1.el7 mysql-community-devel.x86_64 0:8.0.37-1.el7
mysql-community-icu-data-files.x86_64 0:8.0.37-1.el7 mysql-community-libs.x86_64 0:8.0.37-1.el7
mysql-community-libs-compat.x86_64 0:8.0.37-1.el7 mysql-community-server.x86_64 0:8.0.37-1.el7
Installed:
mysql-community-client.x86_64 0:8.0.36-1.el7 mysql-community-client-plugins.x86_64 0:8.0.36-1.el7
mysql-community-common.x86_64 0:8.0.36-1.el7 mysql-community-devel.x86_64 0:8.0.36-1.el7
mysql-community-icu-data-files.x86_64 0:8.0.36-1.el7 mysql-community-libs.x86_64 0:8.0.36-1.el7
mysql-community-libs-compat.x86_64 0:8.0.36-1.el7 mysql-community-server.x86_64 0:8.0.36-1.el7
Complete!
Please note the MySQL service did not start after the downgrade:/scripts/restartsrv_mysql --status
(XID np6tbn) The “mysql” service is down.
This is because the following option was enabled. I went ahead and disabled it:grep ^#innodb_force /etc/my.cnf
#innodb_force_recovery = 6
MySQL is now working correctly:/scripts/restartsrv_mysql --status
mysqld (/usr/sbin/mysqld) is running as mysql with PID 26192 (systemd+/proc check method).
Please make sure to double-check the article provided before to ensure the version lock is configured to avoid another automatic MySQL upgrade.
Let me know if there is anything else I can do for you.1 -
It's not cPanel fault, obviously, but we miss their support here.
On the other hand, all fundamental packages should be distributed with previous testing.
MySQL 8 was the default option for cPanel for some years, so a little responsibility maybe they have...
1 -
I'm actually seeing that 8.0.39 does *not* fix the segfaults on startup as this isn't covered in the release notes:
https://bugs.mysql.com/bug.php?id=115469
That seems odd to me, but we're still waiting on MySQL to fix that.
Starting in version 122, which is now in Edge, all cPanel installations will default to MariaDB. I still don't have a way to go from MySQL 8 to a MariaDB version, but you could add that feedback to our page at features.cpanel.net and the team will see that.
1 -
This line made me laugh:
like to commend the Oracle team for their swift response to this problem.
LOL
1 -
AlmaLinux v8.10.0 STANDARD standardcPanel Version120.0.110
-
Same issue for us - the rollback worked and brought MySQL back online.
0 -
Just confirming we have this issue as well. Not all servers failed the upgrade, some upgraded fine.
Both on AlmaLinux and Ubuntu 20.04.
We also downgraded, which also fixed it for us.
0 -
Fuuuuuu.... We were just doing an upgrade on a legacy server from mySQL 5.7 to mySQL 8.0 to start elevation to AlmaLinux and we ran in to this. I spent hours trying to debug what was going wrong and the inexplicable error with zero pointers and no search results got me to nuke the whole thing and start terminating and then restoring accounts from backups. Damn.
The timing is insane. Was just debating this afternoon: do it now or tomorrow? Nothing was published about this bug yet and we decided to do it today. Aaarghh. Oh well. Murphy's law.
Here's the strange thing by the way: nuking the mySQL datafolder instantly gets mySQL to start again. Just removing all databases isn't enough, but nuking the full contents of that dir gets mySQL back working and happy to restore databases as well. (So don't try this at home unless you have a backup of the datadir and full backups of the cPanel accounts. Its the only reason I even started experimenting with this.)0
Please sign in to leave a comment.
Comments
90 comments