MySQL unable to start after upgrade (5.7->8.0)
Hi Team,
We've just attempted to upgrade from Mysql 5.7 to v8 in readiness for Elevate to Almalinux.
Upon completing the upgrade through the WHM GUI, MySQL is not able to restart. I can see MySQL 8 is installed, but upon restarting, we get the error "Restart failed: timeout (196 seconds) reached! at /usr/local/cpanel/Cpanel/RestartSrv/Lock.pm line 37."
The output from the upgrade is as follows:
Starting process with log file at /var/cpanel/logs/mysql_upgrade.20250202-182636/upgrade_mysql_with_status.log
Obtained version information from system.
(XID 78n9hn) The system failed to connect to the “MySQL” database “mysql” because of an error: CR_CONNECTION_ERROR (Can't connect to local server through socket '/var/lib/mysql/mysql.sock' (2))
Ensuring the “Mysql80” repository is available and working.
Loaded plugins: fastestmirror, universal-hooks
Resolving Dependencies
--> Running transaction check
---> Package mysql80-community-release.noarch 0:el7-11 will be erased
--> Finished Dependency Resolution
Dependencies Resolved
================================================================================
Package Arch Version Repository Size
================================================================================
Removing:
mysql80-community-release noarch el7-11 @mysql80-community 17 k
Transaction Summary
================================================================================
Remove 1 Package
Installed size: 17 k
Downloading packages:
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Erasing : mysql80-community-release-el7-11.noarch 1/1
Verifying : mysql80-community-release-el7-11.noarch 1/1
Removed:
mysql80-community-release.noarch 0:el7-11
Complete!
Loaded plugins: fastestmirror, universal-hooks
Loading mirror speeds from cached hostfile
* EA4-c7: 110.232.143.226
* cpanel-addons-production-feed: 110.232.143.226
* cpanel-plugins: 110.232.143.226
Resolving Dependencies
--> Running transaction check
---> Package mysql80-community-release.noarch 0:el7-7 will be updated
---> Package mysql80-community-release.noarch 0:el7-11 will be an update
--> Finished Dependency Resolution
Dependencies Resolved
================================================================================
Package Arch Version Repository Size
================================================================================
Updating:
mysql80-community-release noarch el7-11 mysql80-community 14 k
Transaction Summary
================================================================================
Upgrade 1 Package
Total download size: 14 k
Downloading packages:
No Presto metadata available for mysql80-community
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Updating : mysql80-community-release-el7-11.noarch 1/2
Cleanup : mysql80-community-release-el7-7.noarch 2/2
Verifying : mysql80-community-release-el7-11.noarch 1/2
Verifying : mysql80-community-release-el7-7.noarch 2/2
Updated:
mysql80-community-release.noarch 0:el7-11
Complete!
checkyum version 22.3 (excludes: bind-chroot)
Ensuring that the package “coreutils” is available.
The package “coreutils” version “8.22-24.el7_9.2” is already installed.
Ensuring that the package “grep” is available.
The package “grep” version “2.20-3.el7” is already installed.
Ensuring that the package “mysql-community-devel” with version matching “8.0” is available.
Ensuring that the package “mysql-community-libs-compat” with version matching “8.0” is available.
Ensuring that the package “mysql-community-server” with version matching “8.0” is available.
Ensuring that the package “perl-DBI” is available.
The package “perl-DBI” version “1.627-4.el7” is already installed.
Ensuring that the package “shadow-utils” is available.
The package “shadow-utils” version “4.6-5.el7” is already installed.
The “Mysql80” repository is available and working.
Installing pre-known Mysql dependencies.
Installed pre-known Mysql dependencies.
Ensuring Mysql packages for version “8.0”.
Verifying that the system is in a state where Mysql packages can be installed.
Verifying that the Mysql packages can be installed by doing a test install.
checkyum version 22.3 (excludes: bind-chroot kernel)
Loaded plugins: fastestmirror, universal-hooks
Loading mirror speeds from cached hostfile
* EA4-c7: 110.232.143.226
* cpanel-addons-production-feed: 110.232.143.226
* cpanel-plugins: 110.232.143.226
Resolving Dependencies
--> Running transaction check
---> Package mysql-community-devel.x86_64 0:8.0.40-1.el7 will be updated
---> Package mysql-community-devel.x86_64 0:8.0.41-1.el7 will be an update
---> Package mysql-community-libs-compat.x86_64 0:8.0.40-1.el7 will be updated
---> Package mysql-community-libs-compat.x86_64 0:8.0.41-1.el7 will be an update
---> Package mysql-community-server.x86_64 0:8.0.40-1.el7 will be updated
---> Package mysql-community-server.x86_64 0:8.0.41-1.el7 will be an update
--> Processing Dependency: mysql-community-common(x86-64) = 8.0.41-1.el7 for package: mysql-community-server-8.0.41-1.el7.x86_64
--> Processing Dependency: mysql-community-icu-data-files = 8.0.41-1.el7 for package: mysql-community-server-8.0.41-1.el7.x86_64
--> Running transaction check
---> Package mysql-community-common.x86_64 0:8.0.40-1.el7 will be updated
---> Package mysql-community-common.x86_64 0:8.0.41-1.el7 will be an update
---> Package mysql-community-icu-data-files.x86_64 0:8.0.40-1.el7 will be updated
---> Package mysql-community-icu-data-files.x86_64 0:8.0.41-1.el7 will be an update
--> Finished Dependency Resolution
Dependencies Resolved
================================================================================
Package Arch Version Repository Size
================================================================================
Updating:
mysql-community-devel x86_64 8.0.41-1.el7 mysql80-community 1.9 M
mysql-community-libs-compat x86_64 8.0.41-1.el7 mysql80-community 669 k
mysql-community-server x86_64 8.0.41-1.el7 mysql80-community 65 M
Updating for dependencies:
mysql-community-common x86_64 8.0.41-1.el7 mysql80-community 666 k
mysql-community-icu-data-files x86_64 8.0.41-1.el7 mysql80-community 2.2 M
Transaction Summary
================================================================================
Upgrade 3 Packages (+2 Dependent packages)
Total download size: 70 M
Exiting on user command
Your transaction was saved, rerun it with:
yum load-transaction /root/tmp/yum_save_tx.2025-02-02.18-29.IWVWns.yumtx
Preinstall check passed.
Running: /usr/local/cpanel/scripts/check_cpanel_pkgs --targets=MySQL41,MySQL50,MySQL51,MySQL55,MySQL56,MySQL57,MySQL80,MariaDB100,MariaDB101,MariaDB102,MariaDB103,MariaDB104,MariaDB105,MariaDB106 --fix --no-broken --no-digest
Hooks system enabled.
Checking for and running RPM::Versions 'pre' hooks for any Packages about to be installed
All required 'pre' hooks have been run
Looking for providers that will cause dependency problems: mysql-server
Uninstalling the following via “rpm” prior to “yum” to avoid dependency problems: mysql-community-server
Removing any of the following incompatible packages: mariadb-client mariadb-devel mariadb-embedded mariadb-embedded-devel mariadb-libs mariadb-libs-compat mariadb-release mariadb-server mariadb-test mysql-client mysql-devel mysql-embedded mysql-embedded-devel mysql-libs mysql-libs-compat mysql-release mysql-server mysql-test mysql55-mysql-bench mysql55-mysql-devel mysql55-mysql-libs mysql55-mysql-server mysql55-mysql-test mysqlclient16 rh-mysql56-mysql-bench rh-mysql56-mysql-common rh-mysql56-mysql-config rh-mysql56-mysql-devel rh-mysql56-mysql-errmsg rh-mysql56-mysql-server rh-mysql56-mysql-test rh-mysql57-mysql-common rh-mysql57-mysql-config rh-mysql57-mysql-devel rh-mysql57-mysql-errmsg rh-mysql57-mysql-server rh-mysql57-mysql-test
Loaded plugins: fastestmirror, universal-hooks
Loading mirror speeds from cached hostfile
* EA4-c7: 110.232.143.226
* cpanel-addons-production-feed: 110.232.143.226
* cpanel-plugins: 110.232.143.226
Resolving Dependencies
--> Running transaction check
---> Package mysql-community-devel.x86_64 0:8.0.40-1.el7 will be updated
---> Package mysql-community-devel.x86_64 0:8.0.41-1.el7 will be an update
---> Package mysql-community-libs-compat.x86_64 0:8.0.40-1.el7 will be updated
---> Package mysql-community-libs-compat.x86_64 0:8.0.41-1.el7 will be an update
---> Package mysql-community-server.x86_64 0:8.0.41-1.el7 will be installed
--> Processing Dependency: mysql-community-common(x86-64) = 8.0.41-1.el7 for package: mysql-community-server-8.0.41-1.el7.x86_64
--> Processing Dependency: mysql-community-icu-data-files = 8.0.41-1.el7 for package: mysql-community-server-8.0.41-1.el7.x86_64
--> Running transaction check
---> Package mysql-community-common.x86_64 0:8.0.40-1.el7 will be updated
---> Package mysql-community-common.x86_64 0:8.0.41-1.el7 will be an update
---> Package mysql-community-icu-data-files.x86_64 0:8.0.40-1.el7 will be updated
---> Package mysql-community-icu-data-files.x86_64 0:8.0.41-1.el7 will be an update
--> Finished Dependency Resolution
Dependencies Resolved
================================================================================
Package Arch Version Repository Size
================================================================================
Installing:
mysql-community-server x86_64 8.0.41-1.el7 mysql80-community 65 M
Updating:
mysql-community-devel x86_64 8.0.41-1.el7 mysql80-community 1.9 M
mysql-community-libs-compat x86_64 8.0.41-1.el7 mysql80-community 669 k
Updating for dependencies:
mysql-community-common x86_64 8.0.41-1.el7 mysql80-community 666 k
mysql-community-icu-data-files x86_64 8.0.41-1.el7 mysql80-community 2.2 M
Transaction Summary
================================================================================
Install 1 Package
Upgrade 2 Packages (+2 Dependent packages)
Total download size: 70 M
Downloading packages:
No Presto metadata available for mysql80-community
--------------------------------------------------------------------------------
Total 25 MB/s | 70 MB 00:02
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Updating : mysql-community-icu-data-files-8.0.41-1.el7.x86_64 1/9
Updating : mysql-community-common-8.0.41-1.el7.x86_64 2/9
Installing : mysql-community-server-8.0.41-1.el7.x86_64 3/9
warning: /etc/my.cnf created as /etc/my.cnf.rpmnew
Updating : mysql-community-libs-compat-8.0.41-1.el7.x86_64 4/9
Updating : mysql-community-devel-8.0.41-1.el7.x86_64 5/9
Cleanup : mysql-community-icu-data-files-8.0.40-1.el7.x86_64 6/9
Cleanup : mysql-community-common-8.0.40-1.el7.x86_64 7/9
Cleanup : mysql-community-devel-8.0.40-1.el7.x86_64 8/9
Cleanup : mysql-community-libs-compat-8.0.40-1.el7.x86_64 9/9
/etc/yum/universal-hooks/pkgs/mysql-community-server/posttrans/100-build_mysql_conf: running in the background.
Verifying : mysql-community-devel-8.0.41-1.el7.x86_64 1/9
Verifying : mysql-community-server-8.0.41-1.el7.x86_64 2/9
Verifying : mysql-community-libs-compat-8.0.41-1.el7.x86_64 3/9
Verifying : mysql-community-common-8.0.41-1.el7.x86_64 4/9
Verifying : mysql-community-icu-data-files-8.0.41-1.el7.x86_64 5/9
Verifying : mysql-community-common-8.0.40-1.el7.x86_64 6/9
Verifying : mysql-community-libs-compat-8.0.40-1.el7.x86_64 7/9
Verifying : mysql-community-icu-data-files-8.0.40-1.el7.x86_64 8/9
Verifying : mysql-community-devel-8.0.40-1.el7.x86_64 9/9
Installed:
mysql-community-server.x86_64 0:8.0.41-1.el7
Updated:
mysql-community-devel.x86_64 0:8.0.41-1.el7
mysql-community-libs-compat.x86_64 0:8.0.41-1.el7
Dependency Updated:
mysql-community-common.x86_64 0:8.0.41-1.el7
mysql-community-icu-data-files.x86_64 0:8.0.41-1.el7
Complete!
Mysql installed from official repository.
Building configuration.
This step may produce some errors or warnings in the log. The errors are usually harmless and are a result of table changes between versions of MySQL or MariaDB.
Running: /usr/local/cpanel/bin/build_mysql_conf
Done building configuration.
Hooks system enabled.
Checking for and running RPM::Versions 'post' hooks for any Packages about to be installed
All required 'post' hooks have been run
Running: /usr/local/cpanel/scripts/check_cpanel_pkgs --targets=MySQL41,MySQL50,MySQL51,MySQL55,MySQL56,MySQL57,MySQL80,MariaDB100,MariaDB101,MariaDB102,MariaDB103,MariaDB104,MariaDB105,MariaDB106 --fix --no-broken --no-digest
The system could not start Mysql 8.0
Restarting mysql service.
Restart failed: timeout (196 seconds) reached! at /usr/local/cpanel/Cpanel/RestartSrv/Lock.pm line 37.
Mysql did not start up after upgrade; there may be a problem in /etc/my.cnf. at /usr/local/cpanel/Whostmgr/Mysql/Upgrade.pm line 855.
The output above says there is an issue with the my.cnf file, however, I've deleted this file, ran the update again so it re-creates the file and it still produces the same error.
If I perform a restart on the service, the following appears in /var/log/messages:
Feb 2 18:54:37 cpanel-host1 mysqld: 2025-02-02T07:54:37Z UTC - mysqld got signal 11 ;
Feb 2 18:54:37 cpanel-host1 mysqld: Most likely, you have hit a bug, but this error can also be caused by malfunctioning hardware.
Feb 2 18:54:37 cpanel-host1 mysqld: BuildID[sha1]=0b2ba3f12ed5126f2249adb2c053fda52da666dc
Feb 2 18:54:37 cpanel-host1 mysqld: Thread pointer: 0x7dcf770
Feb 2 18:54:37 cpanel-host1 mysqld: Attempting backtrace. You can use the following information to find out
Feb 2 18:54:37 cpanel-host1 mysqld: where mysqld died. If you see no messages after this, something went
Feb 2 18:54:37 cpanel-host1 mysqld: terribly wrong...
Feb 2 18:54:37 cpanel-host1 mysqld: stack_bottom = 7fe22a7a4a40 thread_stack 0x100000
Feb 2 18:54:37 cpanel-host1 mysqld: /usr/sbin/mysqld(my_print_stacktrace(unsigned char const*, unsigned long)+0x3d) [0x2132dcd]
Feb 2 18:54:37 cpanel-host1 mysqld: /usr/sbin/mysqld(print_fatal_signal(int)+0x37f) [0xfefd7f]
Feb 2 18:54:37 cpanel-host1 mysqld: /usr/sbin/mysqld(handle_fatal_signal+0xa5) [0xfefe35]
Feb 2 18:54:37 cpanel-host1 mysqld: /lib64/libpthread.so.0(+0xf630) [0x7fe23a18a630]
Feb 2 18:54:37 cpanel-host1 mysqld: /usr/sbin/mysqld(dd::sdi::store(THD*, dd::Table const*)+0x10e) [0x1ec272e]
Feb 2 18:54:37 cpanel-host1 mysqld: /usr/sbin/mysqld(dd::upgrade_57::add_sdi_info(THD*)+0x72c) [0x210128c]
Feb 2 18:54:37 cpanel-host1 mysqld: /usr/sbin/mysqld(dd::upgrade_57::do_pre_checks_and_initialize_dd(THD*)+0x1432) [0x21065d2]
Feb 2 18:54:37 cpanel-host1 mysqld: /usr/sbin/mysqld() [0x10b2fe6]
Feb 2 18:54:37 cpanel-host1 mysqld: /usr/sbin/mysqld() [0x27e4c85]
Feb 2 18:54:37 cpanel-host1 mysqld: /lib64/libpthread.so.0(+0x7ea5) [0x7fe23a182ea5]
Feb 2 18:54:37 cpanel-host1 mysqld: /lib64/libc.so.6(clone+0x6d) [0x7fe23879cb2d]
Feb 2 18:54:37 cpanel-host1 mysqld: Trying to get some variables.
Feb 2 18:54:37 cpanel-host1 mysqld: Some pointers may be invalid and cause the dump to abort.
Feb 2 18:54:37 cpanel-host1 mysqld: Query (0): Connection ID (thread ID): 1
Feb 2 18:54:37 cpanel-host1 mysqld: Status: NOT_KILLED
Feb 2 18:54:37 cpanel-host1 mysqld: The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
Feb 2 18:54:37 cpanel-host1 mysqld: information that should help you find out what is causing the crash.
Feb 2 18:54:37 cpanel-host1 systemd: mysqld.service: main process exited, code=exited, status=2/INVALIDARGUMENT
Feb 2 18:54:37 cpanel-host1 systemd: Unit mysqld.service entered failed state.
Feb 2 18:54:37 cpanel-host1 systemd: mysqld.service failed.
Feb 2 18:54:38 cpanel-host1 mysqld: 2025-02-02T07:54:38.161280Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.41) starting as process 30120
Feb 2 18:54:38 cpanel-host1 mysqld: 2025-02-02T07:54:38.166023Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
Feb 2 18:56:12 cpanel-host1 mysqld: 2025-02-02T07:56:12.039930Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
There is a mysqld process running, but if we attempt to connect we receive : ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)
Any idea what could be causing MySQL to fail and how to resolve?
-
Anything useful in mysql logs either under /var/lib/mysql or at /var/log/mysqld.log?
Andrew N. - cPanel Plesk VMWare Certified Professional
Do you need immediate assistance? 20 minutes response time!* Open a ticket
EmergencySupport - Professional Server Management and One-time Services1
Please sign in to leave a comment.
Comments
1 comment