Skip to main content

Mysql cannot start

KevinLevin

Hello,

Out of a sudden my mysql server cannot start.
Cloudlinux 8, mysql 8, cpanel 120.0.11

It seems an automated mysql update from version '80037' to '80038' didn't completed.

There is no error in the log just this:

2024-06-30T20:32:29.113796Z 1 [Note] [MY-013018] [InnoDB] Created 128 and tracked 128 new rollback segment(s) in the temporary tablespace. 128 are now active.
2024-06-30T20:32:29.114289Z 1 [Note] [MY-012976] [InnoDB] 8.0.38 started; log sequence number 1121459761877
2024-06-30T20:32:29.115038Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
2024-06-30T20:32:29.122602Z 1 [Note] [MY-013327] [Server] MySQL server upgrading from version '80037' to '80038'.
2024-06-30T20:32:29.303674Z 1 [Note] [MY-012357] [InnoDB] Reading DD tablespace files
2024-06-30T20:32:29Z UTC - mysqld got signal 11 ;
Most likely, you have hit a bug, but this error can also be caused by malfunctioning hardware.

 

What do you think?

Comments

90 comments

Date Votes
  • clopezi

    There is updates on the bug fixed as per MySQL 8.0 official page ( https://dev.mysql.com/doc/relnotes/mysql/8.0/en/news-8-0-38.html ). Mind confirm if we can unlock the MySQL package?

    That's the changelog for the buggy version. We should wait for 8.0.39 at least. Mysql didn't says anything new on the bug tracker yet.

    0
  • epesi
    • Edited

    KevinLevin

    2024-06-30T20:32:29Z UTC - mysqld got signal 11 ;
    Most likely, you have hit a bug, but this error can also be caused by malfunctioning hardware.

    This part of error log indicates failed MySQL upgrade.
    (... and it's the same error I was getting).

    Just follow steps outlined in this thread using method appropriate for your OS to downgrade MySQL and hold the upgrade of MySQL for now and you'll be fine.

    This is going on for over a week and there are many recent posts on this support forum with different titles all pointing to the same problem.

    Is cPanel support monitoring this forum?
    It looks like it's not as most updates are posted by end user and cPanel is completely absent in these conversation. I am shocked by the level of ignorance and lack of professionalism.

    Do I have to repeat myself?

    An error doesn't become a mistake until you refuse to correct it.
     - John F. Kennedy.

    0
  • Anoop Kumar

     yum downgrade mysql-community-*
    This system is receiving updates from CloudLinux Network server.
    Last metadata expiration check: 0:13:15 ago on Sat 13 Jul 2024 02:21:26 AM IST.
    Packages for argument mysql-community-* available, but not installed.
    Error: No packages marked for downgrade.

    0
  • Unnamed User

    MySQL  Ver 8.0.39 is now available

    0
  • clopezi

    But still no changelog, and no new information on the bug tracker report :(

    cPanel, we really need a method to drop MySQL 8 and get MariaDB on our servers. You forced us to use MySQL 8 by default and this was a very bad decision, as we are seeing.

    0
  • cPRex Jurassic Moderator

    I'm actually seeing that 8.0.39 does *not* fix the segfaults on startup as this isn't covered in the release notes:

    https://bugs.mysql.com/bug.php?id=115469

    That seems odd to me, but we're still waiting on MySQL to fix that.

    Starting in version 122, which is now in Edge, all cPanel installations will default to MariaDB.  I still don't have a way to go from MySQL 8 to a MariaDB version, but you could add that feedback to our page at features.cpanel.net and the team will see that.

    1
  • BlueSteam

    cPRex, If 8.0.39 doesn't fix the segfaults, then is it safe to assume that we should ignore 8.0.39 and wait for the next release before UNLOCKING the version for upgrade?

    As I still sit the my version locked and waiting for the fix after I had to downgrade because of mysql not starting up.

    0
  • BlueSteam

    @... you're a little late to the party ;-)

    You're trying to suggest a fix to a problem that is already well known and the only fix is to rollback to a previous version which has already been provided by cPanel.  The question now is when do we unlock the version for upgrade.  Read the full thread please before responding.

    0
  • cPRex Jurassic Moderator

    BlueSteam - that's because he's an AI bot and will get deleted as soon as I'm done typing this :D

    0
  • BlueSteam

    ah Understood! LOL  You can remove my comments then too :-P  but if you could answer my question to you then that would be awesome :-D

    0
  • mtindor

    Rex,

    Release notes suggest it is fixed in 8.0.39, does it not?

     

    Changes in MySQL 8.0.39 (2024-07-23, General Availability)

    Bugs Fixed

    • InnoDB: In some cases, following the creation of a very large number of tables (8001 or more), the server could not be restarted successfully. (Bug #36808732)

      References: This issue is a regression of: Bug #33398681.

    • InnoDB: Improved tablespace file scan performance at startup. (Bug #110402, Bug #35200385)

    • Group Replication: Running a CREATE TABLE ... SELECT statement on a source coming from an asynchronous channel to Group Replication led to errors on the replica. (Bug #36784284)

    0
  • cPRex Jurassic Moderator

    No, none of those entries fix the issues we're seeing here.

    BlueSteam - yes, we are still waiting for a better solution from MySQL.  This is also a major part of the reason we'll be switching to MariaDB by default in a future release.

    0
  • BlueSteam

    cPRex Thank you for confirming :-)

    0
  • cPRex Jurassic Moderator

    Sure thing!

    0
  • mtindor

    Say what ? I thought this was all about the case where there are more than 8001 InnoDB tables?   No?

    If not, what's the bug on this one?   The link you had for the bug appears to be private.   It'd be nice to see what's reported on it.   I thought the root cause of all of the server crashes was the 8001+ InnoDB tables bug.

    0
  • cPRex Jurassic Moderator

    They must have changed the status of that link after the fix was released because that was definitely public last week.

    The issue that we were hoping they were going to fix was the segmentation fault problem, which doesn't seem to be the same as that "too many InnoDB tables" problem.

    0
  • LvH

    For the record: the mySQL team closed my bug report (115469) with the notice:
    "Duplicate of 115517" (which, incidentally, has also been closed and locked to the public. But I had read it before and I'm not quite sure if we were actually talking about the same thing. Either way I don't think 115469 can be a dupe of a later submission; but whatever.). 
    Alongside this was a note saying that it's fixed by addressing their "internal Bug#36808732", which is the one also mentioned in the changelog (cited above by cPRex).

    I've submitted a comment to my report 115469 noting that the problem has NOT been solved. Also I think it's a bit weird that they're closing bugs to the public after (allegedly) patching them. But at least I still have access to it myself. Not sure if they'll read the comment though.

    0
  • Ninoslav Stecko

    Almost 45 days ago we have critical mysql situation.

    Any move with this issue ?

    0
  • cPRex Jurassic Moderator

    Oracle made this extra confusing by locking those articles and bug reports, as LvH mentioned, so we have to find other sources.  This is just one of the reasons we are going to be moving to MariaDB as the default on new cPanel installations.

    At this time, everything should be working well according to the release notes here:

    https://www.percona.com/blog/severe-instability-of-mysql-8-0-38-8-4-1-and-9-0-resolved-in-upcoming-releases/?utm_campaign=2024-blog-q3&utm_content=301586985&utm_medium=social&utm_source=twitter&hss_channel=tw-35373186

    I also haven't seen any new reports of the issue for several days.

    0
  • Unnamed User
    • Edited

    This line made me laugh:

     like to commend the Oracle team for their swift response to this problem.

    LOL

    1
  • Adam Bryner

    Rex,

    It's safe to disable versionlock on the MySQL upgrade now on servers?

    Thanks.

    0
  • cPRex Jurassic Moderator

    Yes, I would not expect there to be any issues at this point.

    0
  • Unnamed User

    The link says its fixed in the upcoming release 8.039, at the end of August.

    0
  • cPRex Jurassic Moderator

    Sort of - it says "Percona" plans to have it implemented in their system later this month, but that's unrelated to the actual MySQL release.

    0
  • Ninoslav Stecko

    Has anyone tried disable MySQL versionlock, and upgrade to x.39 ?

    thx.

    0
  • Unnamed User

    Upgraded to 8.39 a few days ago. All working OK.

    2
  • BlueSteam

    Just to be as clear as possible and as safe as possible, once the versionlock is cleared, should we manually force a cPanel update?  or force a specific upgrade on MySQL or should we just wait until the next cPanel update and it will update on it's own?

     

    0
  • Ninoslav Stecko
    • Edited

    Only update MySQL 

     

    In cPanel CP disable monitoring 'MySQL Database Server' in 'Service manager'

    service mysqld status
    service mysqld stop
    yum versionlock delete mysql-community-*
    yum upgrade mysql-community-*
    service mysqld start
    service mysqld status
    mysqladmin version

    In cPanel CP enable monitoring 'MySQL Database Server' in 'Service manager'

    0
  • BlueSteam

    Thanks Ninoslav Stecko
    What I ended up doing was simply running /scripts/upcp from the terminal to upgrade mysql

    In this way I made sure everything was on the correct versions on cPanel everywhere.

    It worked for me and I am now on 8.0.39 and stable.

    0
  • Unnamed User

    We simply removed the version lock and waited for the next upcp run. Worked OK.

    0

Please sign in to leave a comment.

Didn't find what you were looking for?

New post