Skip to main content

Mysql cannot start

Comments

90 comments

  • LvH

    amin alnasra
    cPanel didn't release this, the mySQL team did. 
    I have reported the bug to mySQL (https://bugs.mysql.com/bug.php?id=115469) but they responded that they cannot reproduce the problem. It might help if more people respond and attach logs and/or (gdb-/debug-)traces. 

    0
  • Mel Tallagsen

    im down 60 some sites and clients are outraged... i dont know what im gonna do... fml

    -1
  • Mel Tallagsen

    I ran the command and it seemed to get a couple sites back up and running, but on my Joomla sites Im getting 

    Can't lock file (errno: 165 - Table is read only)

    Im going in and trying to repair them 1 at a time, but Its not working...

    -1
  • LvH

    Mel Tallagsen
    Have you tried the downgrade? This seems to work for most people. If not: what happened instead of mySQL starting OK (and what's the exact error it's getting stuck on)? 

    1
  • Mel Tallagsen

    yes it worked, but didnt work, it downgraded, but seems had no affect on my joomla sites 

    the basic jist is 

    The storage engine for the table doesn't support repair

     

    Can't lock file (errno: 165 - Table is read only)

    0
  • quietFinn

    amin alnasra

    It's not released by cPanel, so you are barking up the wrong tree.

     

    0
  • Mel Tallagsen

    where do we need to bark, i will go there and fill their support portal with my issue...lol

    0
  • KevinLevin

    @Mel

    Check your /etc/my.cnf and see if you have the following:

    innodb_force_recovery = 

    If you have - remove that line and restart the mysql server.

    1
  • Adam Bryner

    Melvin, if you can't figure it out, message me, and I'll try and help you.

    0
  • epesi

    Optimizr try to run MySQL password reset from WHM admin panel. It helped in my case.

    If MySQL is running correctly and is recognized by WHM/cPanel then you should see MySQL processes.

    Try the password reset and report your outcome here for others to save them time.

    I wasted approximately 10 hours on various fixes including downgrading, in-place upgrade, purging and clean install... when the fix would take just several minutes if it was properly displayed on cPanel front page. Instead their fix, which I'm referencing above was backdated.

    cPanel team - whatever you do at least monitor your own f$#% forum and respond quickly in case of emergency like that. Do you need crisis management training?

    Your software package is mature and does excellent job and I am paying for both cPanel and DirectAdmin (among others) and will rate cPanel higher than DA. If you improve your support just a little bit you will win many more customers and they will be happy customers.

    Now you have unhappy and frustrated admins looking for help. Where it is?
    You should send e-mail immediately to all cPanel subscribers to warn them about the problem and how to fix it, even if it doesn't apply to them. Do it now!

    4
  • Mel Tallagsen

    i hit the support from my cpanel, and got them involved, right between Kevens and my last exchange, it was fixed, just like that... so he poked his head into the cpanel and fixed it :) 

    0
  • Mel Tallagsen

    and thank you very much to those who reached out to help, you are greatly appreciated...

    0
  • epesi

    Mel Tallagsen - Calm down and separate issues.

    Do you have MySQL running?
    Check mysql error log for details.

    Can you view the database running Joomla in PhpMyAdmin?
    If not fix MySQL and PhpMyAdmin issues first.

    Can you run phpinfo() from within the directory of Joomla?
    If not fix PHP issues first.

    Enable debugging in PHP and Joomla and inspect logs for more details.

    cPanel forum is not a place to ask for Joomla related problems and this is 100% Joomla only problem according to your posts.

    Remember mc is your friend and logs will tell you everything you need to know.

    Good luck!

    0
  • epesi

    cPanel support team - can you please promote solution to this huge problem?

    Quick fix:

    1
  • epesi

    Doing so can maybe will decrease the number of tickets?

    Be more proactive guys. We admins have certain responsibilities and uptime is one thing we have to worry about. The latest release was not properly tested you know it know. But you can still turn this fiasco into an example of how bad support can be turned into great support.

    An error doesn't become a mistake until you refuse to correct it.
     - John F. Kennedy.

    1
  • Mel Tallagsen

    I am absolutely amazed by the fast response i got once i set the support ticket for direct response... 

     

    I have downgraded the MySQL server for you:

    yum downgrade mysql-community-* --disablerepo=base --disablerepo=centos-sclo-rh --disablerepo=centos-sclo-sclo --disablerepo=extras --disablerepo=updates
    [TRUNCATED]
    Removed:
    mysql-community-client.x86_64 0:8.0.37-1.el7 mysql-community-client-plugins.x86_64 0:8.0.37-1.el7
    mysql-community-common.x86_64 0:8.0.37-1.el7 mysql-community-devel.x86_64 0:8.0.37-1.el7
    mysql-community-icu-data-files.x86_64 0:8.0.37-1.el7 mysql-community-libs.x86_64 0:8.0.37-1.el7
    mysql-community-libs-compat.x86_64 0:8.0.37-1.el7 mysql-community-server.x86_64 0:8.0.37-1.el7

    Installed:
    mysql-community-client.x86_64 0:8.0.36-1.el7 mysql-community-client-plugins.x86_64 0:8.0.36-1.el7
    mysql-community-common.x86_64 0:8.0.36-1.el7 mysql-community-devel.x86_64 0:8.0.36-1.el7
    mysql-community-icu-data-files.x86_64 0:8.0.36-1.el7 mysql-community-libs.x86_64 0:8.0.36-1.el7
    mysql-community-libs-compat.x86_64 0:8.0.36-1.el7 mysql-community-server.x86_64 0:8.0.36-1.el7

    Complete!

     
    Please note the MySQL service did not start after the downgrade:

    /scripts/restartsrv_mysql --status
    (XID np6tbn) The “mysql” service is down.

     
    This is because the following option was enabled. I went ahead and disabled it:

    grep ^#innodb_force /etc/my.cnf
    #innodb_force_recovery = 6

     
    MySQL is now working correctly:

    /scripts/restartsrv_mysql --status
    mysqld (/usr/sbin/mysqld) is running as mysql with PID 26192 (systemd+/proc check method).

     
    Please make sure to double-check the article provided before to ensure the version lock is configured to avoid another automatic MySQL upgrade.
     
    Let me know if there is anything else I can do for you.

    1
  • epesi

    LvH and amin alnasra
    Yes it was MySQL team that released a buggy version that crushed servers, but it was cPanel's responsibility to test it, because we trust them and releases from cPanel directly.

    We were lucky to receive only 2 phone calls and several e-mails, but we are in Europe and most clients are in USA and with -6 hours difference they were asleep and until now have no idea that there was a problem. They will receive a PostMortem notice only, but we were extremely lucky to be doing repairs on Sunday night before busy for businesses Monday.

    cPanel guys - you can save us this frustration just by being more proactive and responsive.

    Do you need help organizing proper CRM/Help Desk operations? I'll be glad to help free of charge to avoid situations like that in the future.

    Lesson number one: please respond at least to all users who reported the problem and fixes and apologize for this f#$*$up, because this is what it is.

    I can't say keep up good job today. Today I have to say: "do a better job guys!"

    But in general, like I stated before, you have a great package and we rely and depend on you.
    We admins.

    0
  • Shyam

    Mel Tallagsen, Why did you downgrade from 8.0.37 to 8.0.36, isn't 8.0.38 suppose to be the buggy version that crashed MySQL and 8.0.37 works okay (at least in our case), I'm a bit confused and curious now 🙄🤔

    2
  • LvH

    epesi Why is it cPanel's responsibility to check upstream packages? cPanel doesn't control the packages. It's not "cPanelOS". An update to mySQL got pushed and your server downloads & install it. If you do not want it to do this automatically, you may wish to suspend automatic updates and manually run them instead. Its the same for kernel updates and such. You can exclude packages from the automatic yum/dnf updates.

    On a sidenote, even if cPanel would test this themselves as well: no guarantee the issue would be seen as it only affects specific circumstances, which are yet to be determined. 

    Either way, I'm not sure why cPanel would be responsible for the upstream packages. *You* are supposed to manage the server, cPanel is not a catch-all solution for full server management. But that's a discussion perhaps better for another topic anyway, maybe here the focus should lie on resolving the problem for people who can't seem to get the downgrade to work. 

    -2
  • Oscar Serrano

    One of our dedicated servers also failed and this solve the issue, thanks for sharing:


    Downgrade the problematic update:
    yum downgrade mysql-community-*

    Now lock the version to avoid a new failures:
    yum install yum-versionlock
    yum versionlock mysql-community-*

    cPanel shuts down direct support for servers that didnt purchased their license directly, like our servers purchased from Bluehost. The Bluehost dedicated support team, does not even qualifies to be called support tem. We are all alone now.

    Thanks to the community and CloudLinux support team, that point us in the right direction.

     

    2
  • Tony

    So what to do now, updating mysql have been flawless for 10 years, now we need to test every version, in false production for hours?

    -1
  • clopezi

    It's not cPanel fault, obviously, but we miss their support here.

    On the other hand, all fundamental packages should be distributed with previous testing.

    MySQL 8 was the default option for cPanel for some years, so a little responsibility maybe they have...

    1
  • Cloudlinux/Cpanel User

    Stupid Cpanel upcp.  WHY can't they block this update ?????

    3
  • Jose Manuel

    Hello,

    We are finding that only some servers are experiencing this issue.

    All our cPanel servers have the same MySQL configuration (my.cnf).

    However, with version 8.0.38-1, some servers are functioning correctly.

    [root@* ~]# yum list installed 'mysql-community-'
    Installed Packages
    mysql-community-client.x86_64 8.0.38-1.el8 @mysql80-community
    mysql-community-client-plugins.x86_64 8.0.38-1.el8 @mysql80-community
    mysql-community-common.x86_64 8.0.38-1.el8 @mysql80-community
    mysql-community-devel.x86_64 8.0.38-1.el8 @mysql80-community
    mysql-community-icu-data-files.x86_64 8.0.38-1.el8 @mysql80-community
    mysql-community-libs.x86_64 8.0.38-1.el8 @mysql80-community
    mysql-community-server.x86_64 8.0.38-1.el8 @mysql80-community
    [root@ ~]# /scripts/restartsrv_mysql
    Waiting for “mysql” to restart ……waiting for “mysql” to initialize ………finished.

    Service Status
    mysqld (/usr/sbin/mysqld) is running as mysql with PID 1756867 (systemd+/proc check method).

    Startup Log
    Jul 03 08:04:35 * systemd[1]: Starting MySQL Server...
    Jul 03 08:04:36 * systemd[1]: Started MySQL Server.

    Log Messages
    2024-07-03T06:04:36.367657Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: '8.0.38' socket: '/var/lib/mysql/mysql.sock' port: 3306 MySQL Community Server - GPL.
    2024-07-03T06:04:34.982555Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.38) MySQL Community Server - GPL.

    And others are not...

    …Unrecognized configuration options may have caused the MySQL startup errors.
    The system has attempted to auto-update your MySQL configuration file for your MySQL version.
    This should resolve any errors that stem from an outdated MySQL configuration file.
    info [restartsrv_mysql] systemd failed to start the service “mysqld” (The “/usr/bin/systemctl restart mysqld.service --no-ask-password” command (process 32888) reported error number 1 when it ended.): Job for mysqld.service failed because the control process exited with error code.
    See "systemctl status mysqld.service" and "journalctl -xe" for details.

    …failed.

    Cpanel::Exception::Services::StartError
    Service Status
    undefined status from Cpanel::ServiceManager::Services::Mysql

    Service Error
    (XID k4usm5) The “mysql” service failed to start.

    Startup Log
    Jul 03 08:07:20 rjausero.grid-dns.net systemd[1]: mysqld.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
    Jul 03 08:07:20 rjausero.grid-dns.net systemd[1]: mysqld.service: Failed with result 'exit-code'.
    Jul 03 08:07:20 rjausero.grid-dns.net systemd[1]: Failed to start MySQL Server.

    mysql has failed. Contact your system administrator if the service does not automatically recover.

    Do we know exactly what is causing this issue in this scenario?

    0
  • ffeingol

    It's been posted before, but here is the related MySQL bug report: https://bugs.mysql.com/bug.php?id=115469  There is nothing that cPanel can do about this until Oracle/MySQL creates a fix.

    0
  • Norbert Peters

    Hi!

    I have the same issue here. MySQL cannot start after the update. I tried to roll back with 

    yum downgrade mysql-community-*

    Unfortunately this does not work at my situation. rolled back to 8.0.37 but still the server cannot start. I had to undo the complete cpanel update by setting back an image backup on the server. 

    how to fix this?! Do i really have to cancel any update of cpanel till this problem is fixed in updates?

    How can i make this work?! 

    0
  • ffeingol

    Norbert Peters you don't have to disable all updates, you just need to hold back the MySQL updates.  It's been posted many times, just just follow this article on how to hold back (RPM lock) the MySQL updates: https://support.cpanel.net/hc/en-us/articles/24546938662039-Upstream-issue-MySQL-8-0-38-does-not-start-Segmentation-Fault

    0
  • Norbert Peters

    ffeingol 

    Thank you for your reply. I found the issue that holded the server from starting back up after downgrading. It works now with all other cpanel updates loaded. I'll lock MySQL from updating for now. 

    I understand that normal cpanel updates should be done as soon as possible, but such a crash suddenly is really bad. Trust is a big thing in financial reporting, and this doesn't help.

    0
  • clopezi

    MySQL says on bug report page that Almalinux it's not supported. And that's it.

    cPanel force us to use by default MySQL 8, so cPanel should do an effort to help here, and provide a migration tool to MariaDB

    0
  • hailfitri fitri

    There is updates on the bug fixed as per MySQL 8.0 official page ( https://dev.mysql.com/doc/relnotes/mysql/8.0/en/news-8-0-38.html ). Mind confirm if we can unlock the MySQL package?

    0

Please sign in to leave a comment.