Skip to main content

MySQL8.4 upgraded to MySQL9.7 during nightly updates.

Christopher Barnett
  • Updated

Symptoms

The MySQL service will appear down after the nightly cPanel and OS updates complete.

 

Cause

The cPanel nightly updates trigger system updates, which caused servers using MySQL 8.4 to upgrade to MySQL 9.7. The reason behind it is that it seems the official mysql-community.repo has MySQL 9.7 version is available, and dnf/yum updates will see it as the new version.

This issue is caused by a bug in MySQL repositories that automatically upgrade to the next major version.

MySQL bug report 120315

We've opened an internal case for our development team to investigate this further. For reference, the case number is CPANEL-52811 . Follow this article to receive an email notification when a solution is published in the product.

 

Resolution

The upstream repo for MySQL has been updated, and no servers should experience this issue that haven't already. For any that upgraded and are still having issues, the MySQL version will need to be reverted manually to 8.4. Both scenarios to work around this problem involve MySQL being offline during the upgrade.

Warning: There is a difference between upgrades that failed to init 9.7 and upgrades that successfully init 9.7. The MySQL log shows the following output when the init to 9.7 was successful.

  • First, determine if the upgrade was successful or not:
Successful:

# grep -a 'Server upgrade' /var/log/mysqld.log | tail -2
2026-04-22T18:10:10.405325Z 4 [System] [MY-013381] [Server] Server upgrade from '80409' to '90700' started.
2026-04-22T18:10:12.453060Z 4 [System] [MY-013381] [Server] Server upgrade from '80409' to '90700' completed.

Note the '90700' completed portion.

Failure:

# grep -a 'Server upgrade' /var/log/mysqld.log | tail -2 
2026-04-21T02:35:05.238615Z 4 [System] [MY-013381] [Server] Server upgrade from '80408' to '80409' started. 
2026-04-21T02:37:12.136290Z 4 [System] [MY-013381] [Server] Server upgrade from '80408' to '80409' completed.

Note the '90700' completed portion is missing. 

Note: The MySQL log may be located in a different location. You can check this with the command below. If no location is specified, logs will be located at /var/lib/mysql/HOSTNAME.err, where HOSTNAME is the server's hostname.

# grep -a log-error /etc/my.cnf

Perform the correct steps depending on the above results:

This will cause data loss if the correct set of instructions are not followed exactly. An experienced system administrator with root access is required to perform the tasks below. Review the above to ensure correct instructions are followed.

 
 
Revert back to MySQL 8.4 when the MySQL 9.7 init process is successful

  1. Make a backup of the MySQL datadir

    # cp -Rfp /var/lib/mysql{,.post_9.7_backup}

  2. Disable mysql_native_password=1 , default_authentication_plugin and innodb_log_file_size in the MySQL configuration file, as it was removed in MySQL 9.7.

    # sed -i 's/mysql_native_password/#mysql_native_password/' /etc/my.cnf

    # sed -i 's/authentication_policy=mysql_native_password/#authentication_policy=mysql_native_password/' /etc/my.cnf

    # sed -i 's/innodb_log_file_size/#innodb_log_file_size/' /etc/my.cnf

  3. Set MySQL to start with skip-grant-tables to allow switching the root user off of the removed mysql_native_password plugin.

    # echo skip-grant-tables >> /etc/my.cnf

  4. Restart MySQL 9.7

    # /scripts/restartsrv_mysql

  5. Take note of the root MySQL password.

    # cat /root/.my.cnf

  6. Access the MySQL shell to update the authentication plugin for root.

    # mysql

  7. Inside the MySQL shell, flush the privileges and reset the root user's password. Make sure to replace PASSWORD with the password from Step 5.

    # FLUSH PRIVILEGES;
    # ALTER USER root@localhost IDENTIFIED WITH caching_sha2_password BY 'PASSWORD';

  8. Exit the MySQL shell.

    # QUIT;

  9. Remove the skip-grant-tables setting.

    # sed -i 's/skip-grant-tables//g' /etc/my.cnf

  10. Restart MySQL.

    # /scripts/restartsrv_mysql

  11. Make MySQL dump backups of the databases

    # mysqldump --all-databases --routines --events --triggers --single-transaction --quick --set-gtid-purged=OFF --skip-lock-tables --ignore-table=mysql.user --ignore-table=mysql.db --ignore-table=mysql.tables_priv --ignore-table=mysql.columns_priv --ignore-table=mysql.procs_priv > mysql97_dump.sql

  12. Disable MySQL 9.7 repository

    # dnf config-manager --disable mysql-9.7-lts-community mysql-tools-9.7-lts-community

  13. Enable MySQL 8.4 repository

    # dnf config-manager --enable mysql-8.4-lts-community mysql-tools-8.4-lts-community

  14. Stop MySQL

    # /scripts/restartsrv_mysql --stop

  15. Move the datadir to reinitialize

    # mv /var/lib/mysql /var/lib/mysql97.bak

  16. Remove MySQL 9.7

    # dnf remove mysql-community-server mysql-community-devel mysql-community-client

  17. Clean install MySQL 8.4

    # dnf install mysql-community-server mysql-community-devel mysql-community-client

  18. Initialize MySQL 8.4

    # mysqld --initialize --user=mysql

    # chmod 755 /var/lib/mysql

  19. Set MySQL native passwords back to enabled in my.cnf

    # sed -i '/^\[mysqld\]/a mysql_native_password=1' /etc/my.cnf

  20. Start MySQL 8.4

    # /scripts/restartsrv_mysql --start

  21. Perform a connection check to reset the password

    # /scripts/mysqlconnectioncheck

  22. Import the MySQL 9.7 backups into 8.4

    # mysql < mysql97_dump.sql

  23. Restore the cPanel database grants

    # for each in `awk '{print $2}' /etc/trueuserdomains`; do /usr/local/cpanel/bin/restoregrants --cpuser=$each --db=mysql --all;done

  24. If you are using CloudLinux with CageFS, remount and update CageFS.

    # cagefsctl -M
    # cagefsctl --force-update

  25. To prevent this in the future, the version can be locked in place with the versionlock plugin.

    # dnf install -y python3-dnf-plugin-versionlock
    # dnf versionlock add mysql-community-*

If the 9.7 init process failed, revert to MySQL 8.4 while MySQL 9 is offline.

  1. Make a backup of the MySQL datadir

    # cp -Rfp /var/lib/mysql{,.post_9.7_backup}

  2. Disable the MySQL 9.7 repository

    # yum-config-manager --disable mysql-9.7-lts-community mysql-tools-9.7-lts-community

  3. Enable MySQL 8.4 repository

    # yum-config-manager --enable mysql-8.4-lts-community mysql-tools-8.4-lts-community

  4. Downgrade MySQL back to 8.4

    # dnf downgrade mysql-community* --allowerasing

  5. Verify MySQL 8.4 is now installed

    # rpm -qa | grep '^mysql-community' | sort

  6. To prevent this in the future, the version can be locked in place with the versionlock plugin.

    # dnf install -y python3-dnf-plugin-versionlock
    # dnf versionlock add mysql-community-*

Related articles

Comments

0 comments

Article is closed for comments.