Server upgraded mysql 8.4 to 9.7
Hello
Yesterday one of my servers updated the mysql84-community-release package. It disabled the mysql8.4 channel and enabled the mysql9.7 channel inside the mysql-community.repo file
2026-04-21T20:02:21-0300 SUBDEBUG Upgrade: mysql84-community-release-el9-4.noarch
2026-04-21T20:02:21-0300 SUBDEBUG Upgraded: mysql84-community-release-el9-3.noarch
And the cpanel nightly update upgraded the mysql version to 9.7:
[2026-04-22 00:13:04 -0300] [/usr/local/cpanel/scripts/update-packages] (1/7): mysql-community-common-9.7.0-1.el9.x86_6 7.1 MB/s | 607 kB 00:00
[2026-04-22 00:13:04 -0300] [/usr/local/cpanel/scripts/update-packages] (2/7): mysql-community-client-9.7.0-1.el9.x86_6 27 MB/s | 3.8 MB 00:00
[2026-04-22 00:13:04 -0300] [/usr/local/cpanel/scripts/update-packages] (3/7): mysql-community-client-plugins-9.7.0-1.e 686 kB/s | 1.5 MB 00:02
[2026-04-22 00:13:04 -0300] [/usr/local/cpanel/scripts/update-packages] (4/7): mysql-community-libs-9.7.0-1.el9.x86_64. 41 MB/s | 1.5 MB 00:00
[2026-04-22 00:13:04 -0300] [/usr/local/cpanel/scripts/update-packages] (5/7): mysql-community-icu-data-files-9.7.0-1.e 1.0 MB/s | 2.3 MB 00:02
[2026-04-22 00:13:04 -0300] [/usr/local/cpanel/scripts/update-packages] (6/7): mysql-community-devel-9.7.0-1.el9.x86_64 2.9 MB/s | 7.8 MB 00:02
[2026-04-22 00:13:04 -0300] [/usr/local/cpanel/scripts/update-packages] (7/7): mysql-community-server-9.7.0-1.el9.x86_6 9.3 MB/s | 56 MB 00:06
[2026-04-22 00:13:04 -0300] [/usr/local/cpanel/scripts/update-packages] Upgrading : mysql-community-common-9.7.0-1.el9.x86_64 1/14
[2026-04-22 00:13:04 -0300] [/usr/local/cpanel/scripts/update-packages] Upgrading : mysql-community-client-plugins-9.7.0-1.el9.x86_64 2/14
[2026-04-22 00:13:04 -0300] [/usr/local/cpanel/scripts/update-packages] Upgrading : mysql-community-libs-9.7.0-1.el9.x86_64 3/14
This broke the user connections because we were using mysql_native_password
I managed to start mysql with --skip-grant-tables to make a full backup, remove mysql 9.7, reinstall mysql 8.4 and restore all databases:
# mysqld --skip-grant-tables &
# mysqldump --all-databases --routines --triggers --events --single-transaction > full_backup.sql
I wonder if this happened to anyone else, very uncommom behavior.
-
Yesterday evening until this morning, for more than 12 continuous hours, I was trying to fix my server with every possible way.
Nothing worked.
I tried resetting passwords, I tried switching to MariaDB, I tried restoring all databases, I tried everything I could think of .... with no result.
For 2 days now I have barely slept (maybe 2 hours total), trying to recover the situation.
I had more than 20+ websites on this server. All of them went down.
In the end, the server became completely unusable and I was forced to set up a new server from scratch and start transferring everything manually.
Even now I am still working on migration, while at the same time dealing with complaints from my clients.
This situation is unacceptable.
All this happened without me making any changes on my side.
Who takes responsibility for this? Who compensates this kind of damage?
Because this is not just technical! this is real business impact.
1 -
Stelios Vlachostergios - were you able to submit a ticket?
0 -
I have not submitted a ticket yet, as I initially assumed the issue was caused by something on my side and I spent significant time trying to resolve it.
During this time, I was contacted immediately by my clients, as their websites were down and required urgent resolution, which added significant pressure to restore services as quickly as possible..
1 -
@stelios this is a hard lesson to learn. I only discovered this issue after I had reverted a backup snapshot. Luckily I had a backup from only a few hours prior. I know how stressful that must have been for you. Cpanel really short have sent out an email alert,
0
Please sign in to leave a comment.
Comments
34 comments