Skip to main content
cPanel Technical Support has been heavily impacted by hurricane Beryl and our ability to respond to tickets has been hindered as a result. We appreciate your understanding and patience as we address these delays.

Major Issues Lately with MariaDB Upgrades and Updates via WHM

Comments

29 comments

  • cPRex Jurassic Moderator
    Hey there! If you're referring to the issues yesterday, that was a problem with MariaDB breaking their own repositories, so there wasn't anything we could do on our end for that. MariaDB said that was resolved as of 12:30 yesterday, although some people reported odd behavior for another few hours while the mirrors were getting back in sync. If you have ideas about adding repair features, please submit a feature request and I can bring that up during our weekly developer meeting. We're always working on and entertaining new features so I'd be happy to discuss that with them.
    0
  • celiac101
    No, I'm not referring to that. The auto update that ran did alert me to an issue, but when I ran yup update ~6 hours later the update ran fine, found the repositories, and then broke my databases--all of them on different web sites.
    0
  • cPRex Jurassic Moderator
    Any chance you could submit a ticket so we can see that? I'm not saying it's not possible, but a Yum transaction wouldn't touch any data in the database itself, so I'm not sure how the two would be related without us doing some investigation.
    0
  • celiac101
    I did, but fixed this myself in the several hours I was waiting for anyone there to respond...oh...did I mention my issue with how long it is now to get any help from you guys?
    0
  • celiac101
    and yes, yum update, which updated MariaDB, did crash every database on my server to the point where the only way to repair them was a full export import routine, which took a ton of time.
    0
  • cPRex Jurassic Moderator
    How long did you wait for a ticket response? Is your license direct through us so you receive priority support? Could you let me know a ticket number so I could review that? I'm happy to listen to feedback, but I do need some data to act on in order to help. As far as the Yum issues, I suppose anything is possible but I'm just not sure how a Yum transaction would corrupt database data. If that did happen between Yum and MariaDB, at least cPanel tools wouldn't have been involved.
    0
  • celiac101
    #94475125 is the ticket number, and I believe I'm on the misnomer "Priority" support ;) Apparently the paid upgraded "Premium" support is no longer an option, as it now throws an error when I try to do that. I can't tell the exact times on the ticket, but it looks like I submitted it at ~4:30PM Pacific time, and the first response on the ticket was ~4 hours later. It was enough time for a non-expert like me to research how, and to rebuild 5 mysql database, 3 of them being over 5 gigs. I'm only glad I was able successfully dump all the data, so that nothing was lost. But really, I'd love to hear a comment about #2 above. Why the heck, given that essentially all of your customers run mysql databases, doesn't WHM offer more automated recovery options (especially if your software updates/upgrades might be breaking them more often going forward)? BTW, I posted a reply on that ticket 6 hours ago and nobody has replied. I guess it's a good think my sites are not still down and I was able to recover the DBs myself!
    0
  • cPRex Jurassic Moderator
    Thanks for the details. I checked the ticket and confirmed that your license isn't purchased through us, so the ticket is part of the complimentary queue, which has no official timeframe for a reply. You are correct that there are issues with the support bump in the ticket system, and our developers are working to address that. I see that Justin created case CPANEL-41340 with our development team to have them explore the possibility of making some type of automatic database copy during the upgrade process on the server, just in case something goes wrong. Traditionally we try and stay hands-off with data, but we'll let the team explore this and see what they come up with. We also weren't able to access the server in the ticket, so for more details on what may have caused the MySQL problems we'd have to get access to the affected system.
    0
  • celiac101
    Thanks for looking into this...regardless of the license type I'm paying your regularly for my license, which should include reasonable support times, but again, your escalation system has been broken for months now, and I have no issues paying $30 or so in an emergency, but I could not even do that the last two emergencies I had (both caused by WHM). I hope you can fix it soon, but I would fire the person who's been working on it. I could have gotten that fixed for you within a few days...
    0
  • cPRex Jurassic Moderator
    I've got someone looking into this today, so hopefully that is resolved soon. Since the license isn't purchased through cPanel, your first contact for support would normally be the hosting provider where the license was purchased.
    0
  • HenrikSCH
    The is no centos8 folders on the MariaDB servers
    0
  • cPRex Jurassic Moderator
    CentOS 8 is not a supported operating system, so that makes sense to me. You should be using Ubuntu, CentOS 7, AlmaLinux 8, or CloudLinux for a cPanel system at this time.
    0
  • HenrikSCH
    I am running Almalinux 8.6 with latest WHM/cPanel and MariaDB 10.6.9
    0
  • HenrikSCH
    this is the text in my MariaDB106.repo on the server. [MariaDB106] name = MariaDB106 baseurl = gpgcheck=1
    0
  • HenrikSCH
    I updated from MariaDB 10.6.8 to 10.6.9 a few day ago with no problems.
    0
  • HenrikSCH
    Do i need to change the baseurl in the MariaDB106.repo file? And if yes ... what do i need to change it to?
    0
  • ServerHealers
    @HenrikSCH The correct baseurl for AlmaLinux8 would be
    0
  • cPRex Jurassic Moderator
    Thanks @ServerHealers !
    0
  • HenrikSCH
    Working now .. thank you for your help :)
    0
  • HenrikSCH
    Now i have another issue MariaDB stopped working today .. no idea why .. i was not on the server when it happened.
    (XID 2qsvqs) The "mysql" service is down. The subprocess "/usr/local/cpanel/scripts/restartsrv_mysql" reported error number 2 when it ended.
    Aug 19 18:05:54 host.****.com systemd[1]: Starting MariaDB 10.6.9 database server... Aug 19 18:05:54 host.****.com mariadbd[757800]: 2022-08-19 18:05:54 0 [Note] /usr/sbin/mariadbd (server 10.6.9-MariaDB) starting as process 757800 ... Aug 19 18:05:54 host.****.com mariadbd[757800]: 2022-08-19 18:05:54 0 [Warning] Could not increase number of max_open_files to more than 32768 (request: 640863) Aug 19 18:05:54 host.****.com mariadbd[757800]: 2022-08-19 18:05:54 0 [Warning] Changed limits: max_open_files: 32768 max_connections: 800 (was 800) table_cache: 15969 (was 40000) Aug 19 18:05:56 host.****.com systemd[1]: mariadb.service: Main process exited, code=exited, status=1/FAILURE Aug 19 18:05:56 host.****.com systemd[1]: mariadb.service: Failed with result 'exit-code'. Aug 19 18:05:56 host.****.com systemd[1]: Failed to start MariaDB 10.6.9 database server.
    How to fix the i already tried to set /usr/lib/systemd/system/mariadb.service to # Number of files limit. previously [mysqld_safe] open-files-limit LimitNOFILE=infinity and LimitNOFILE=650000 But none of them did it .. how do i fix this?
    0
  • vacancy
    Unable to upgrade from mysql 5.7 to mariadb 10.3 In the error logs, 404 error response comes from mariadb 10.2 repository, the update cannot continue because the updates were made to 10.2 first to update to 10.3. [QUOTE] The system was not able to ensure the availability of the "MariaDB-client" package: (XID se5akg) "/usr/bin/yum" reported error code "1" when it ended: : [Errno 14] HTTPS Error 404 - Not Found
    0
  • vacancy
    I edited the mariadb 10.2 repo file according to the following document, but when I start the update process again, the repo file returns to its original state and the repo files cannot be found again.
    0
  • vacancy
    ...
    0
  • cPRex Jurassic Moderator
    @vacancy - that sounds right, and until the fix is 100% applied to all systems, which will happen this week, people will continue to see errors. If you need to manually perform an update you can run this: echo '142.4.217.28 yum.mariadb.org' >> /etc/hosts then run the update, then remove that hosts file entry.
    0
  • ellaliamop
    After going through 2 days of hell in the last few months, I am sharing my thoughts here: 1) I no longer trust anything to do with WHM's ability to run any type of upgrade or update on MariaDB databases. Within the last few months I've had 2 major crashed database incidents due to:
    • a) a more major upgrade from MariaDB 10.5 to 10.6 using WHM update mysql app;
    • b) a "minor" update to MariaDB 10.6 using the standard yum update command;
    I am not sure what is going on since WHM was purchased by a larger company, but to me it's all bad and after more than 10 years I'm now considering switching to an alternative. I've run WHM DB upgrade and updates for over 10 years without such issues, so having such major problems lately breaks my trust in WHM. 2) Why are there not more built in options to automatically repair DBs that have issues, or in an emergency dump a DB and re-create the DB within the WHM interface? Certainly you could offer far more tools to fix database issues than what you currently offer, which is nearly nothing. How hard would it be to offer an emergency tool that would try to perform a mysql dump and extraction in to a new DB with a "-copy" name added to the DB name. After it ran, if successful, it could display a simple message saying:
    0
  • cPRex Jurassic Moderator
    It's important to note the most recent issue was caused by MariaDB changing their repository structure. I did see the feature request that was submitted about repair options, so we'll see if we can get something like that added in the future.
    0
  • bigadmin
    @HenrikSCH The correct baseurl for AlmaLinux8 would be
    0
  • ServerHealers
    Thank you @ServerHealers the solution provided by you worked like charm .

    Happy to help always! :)
    0

Please sign in to leave a comment.