CPANEL-34745 - MariaDB 10.3.26-1 Breaks PHP < 7.2
Just wanting to post this here, as I've spent the last 2 hours fault finding an issue on my servers this morning.
cPanel servers, all the latest updates. MariaDB automatically updated to 10.3.26-1 over night as part of the cPanel yum update.
As soon as this happened, any XenForo site running on PHP7.2 or below broke!
This also prevents any user accessing MySQL via cPanel: Swapping to 7.3 or 7.4 works. I've had to downgrade MariaDB back to the previous version:
And this fixes the issue. So, I don't know if it's a PHP issue, a MariaDB issue, or something combining everything! I've excluded MariaDB and Galera now from automatic updates on my servers.
[04-Nov-2020 08:24:41 UTC] PHP Fatal error: Uncaught XF\Db\Exception: MySQL query error [1835]: Malformed communication packet in /home/XXX/public_html/community/src/XF/Db/AbstractStatement.php:212
Stack trace:
#0 /home/XXX/public_html/community/src/XF/Db/Mysqli/Statement.php(196): XF\Db\AbstractStatement->getException('MySQL query err...', 1835, 'HY000')
#1 /home/XXX/public_html/community/src/XF/Db/Mysqli/Statement.php(77): XF\Db\Mysqli\Statement->getException('MySQL query err...', 1835, 'HY000')
#2 /home/XXX/public_html/community/src/XF/Db/AbstractAdapter.php(79): XF\Db\Mysqli\Statement->execute()
#3 /home/XXX/public_html/community/src/XF/Db/AbstractAdapter.php(116): XF\Db\AbstractAdapter->query('\n\t\t\tSELECT data...', Array)
#4 /home/XXX/public_html/community/src/XF/DataRegistry.php(137): XF\Db\AbstractAdapter->fetchPairs('\n\t\t\tSELECT data...')
#5 /home/XXX/public_html/community/src/XF/DataRegistry.php(82): XF\DataRegistry->readFromDb(Array, Array)
#6 /home/XXX/public_html/community/src/XF/DataRegistry.php(225): XF\DataRegistry- in /home/XXX/public_html/community/src/XF/Db/AbstractStatement.php on line 212This also prevents any user accessing MySQL via cPanel: Swapping to 7.3 or 7.4 works. I've had to downgrade MariaDB back to the previous version:
# yum history info 264
Loaded plugins: fastestmirror, universal-hooks
Transaction ID : 264
Begin time : Wed Nov 4 05:35:28 2020
Begin rpmdb : 1964:269d3e2a5fc59faed15b162386618da46afe3e28
End time : 05:36:05 2020 (37 seconds)
End rpmdb : 1964:7aac4ca2ee1e5c81b04babc4f774126e1254d10d
User : root
Return-Code : Success
Command Line : --assumeyes --color=never --config /etc/yum.conf update --disablerepo=epel
Transaction performed with:
Installed rpm-4.11.3-43.el7.x86_64 @base
Installed yum-3.4.3-167.el7.centos.noarch @base
Installed yum-metadata-parser-1.1.4-10.el7.x86_64 @anaconda
Installed yum-plugin-fastestmirror-1.1.31-54.el7_8.noarch @updates
Packages Altered:
Updated MariaDB-client-10.3.25-1.el7.centos.x86_64 @MariaDB103
Update 10.3.26-1.el7.centos.x86_64 @MariaDB103
Updated MariaDB-common-10.3.25-1.el7.centos.x86_64 @MariaDB103
Update 10.3.26-1.el7.centos.x86_64 @MariaDB103
Updated MariaDB-compat-10.3.25-1.el7.centos.x86_64 @MariaDB103
Update 10.3.26-1.el7.centos.x86_64 @MariaDB103
Updated MariaDB-devel-10.3.25-1.el7.centos.x86_64 @MariaDB103
Update 10.3.26-1.el7.centos.x86_64 @MariaDB103
Updated MariaDB-server-10.3.25-1.el7.centos.x86_64 @MariaDB103
Update 10.3.26-1.el7.centos.x86_64 @MariaDB103
Updated MariaDB-shared-10.3.25-1.el7.centos.x86_64 @MariaDB103
Update 10.3.26-1.el7.centos.x86_64 @MariaDB103
Updated galera-25.3.30-1.el7.centos.x86_64 @MariaDB103
Update 25.3.31-1.el7.centos.x86_64 @MariaDB103
history infoAnd this fixes the issue. So, I don't know if it's a PHP issue, a MariaDB issue, or something combining everything! I've excluded MariaDB and Galera now from automatic updates on my servers.
-
Just wanted to let others know that we're too experiencing the same error on cPanel's "MySQL databases" page. On older Laravel projects (version 5.8) on PHP 7.2 we observed the exact error message (MySQL query error [1835]: Malformed communication packet). Upgrading those sites from PHP 7.2 to 7.3 solved the problem. Since the problem affects both cPanel (Perl) and PHP, my guess is that the culprit is definitely MariaDB 10.3.26 update. 0 -
Confirmed as an issue on 10.2.x as well. I thought upgrading to 10.3 would fix it, but it didn't. 0 -
Same issue here with servers upgraded to MariaDB 10.2.35 tonight. It seems MariaDB 10.2 and 10.3 tonight upgrades have the same issue with PHP 7.1 and 7.2 al least. 0 -
Mariadb's broken updates are now getting annoying. This is not the first time. 0 -
Same temprorary soluton applied here while waiting on final solution. We have websites that should be run on PHP 7.2. 0 -
Same here, MariaDB to 10.3.26 tonight edit: resolved by downgrading to 10.3.25 0 -
I solved this on my server by upgrading to PHP 7.4 as mentioned. 0 -
Same issue with 10.2.35-MariaDB and PHP 7.0 0 -
Same problem here... Solved downgrade to 10.3.25 0 -
In my case a couple of websites were left without working, but when I changed the version of PHP as recommended here, it was restored. What does not work in all cPanel's of the websites is the access to MySQL" Databases". "The MySQL server is currently offline." "adminbin Cpanel/cpmysql/DBCACHE: exit 11" What I want to mention is that I created a new cPanel for a new client and here if it works well MySQL" Databases. 0 -
Fine. Workaround applied. It's a PITA. It's not cool. But well, it worked. Now, how do we know when a MariaDB release a fix so we can unlock updates? 0 -
When cPanel is aware of these issues, it should send a mailing commenting what the hell is happenning, and provide the workaround. That would save both customers and support staff the hundreds of hours/man to debug, think, investigate, and add to that the time we have to lose apologizing with our end customers. Or if you don't want to spend money sending thousands of emails, add a notification function to WHM so it can show a warning saying "Hey dude! MariaDB did it again, here are the symptoms and the workaround. Thanks for keep paying. Love. cPanel." (Yep, actually I WANT to see such a warning and I assure you other admins will absolutelly love it too.) And please, don't send me to open a Feature Request. Be proactive guys, we are paying a lot of money for you to make the best panel and this feature shouldn't be that complicated. Don't stagnate please. 0 -
As an update to the discussion for our internal case CPANEL-34745. This was determined to be caused by the MariaDB SHOW GRANTS change to include role information printing a specific column ID. In some older systems the column order has been found to contain three columns('password_last_changed,password_lifetime,account_locked') added by MySQL 5.7 to be present before the MariaDB columns('is_role,default_role,max_statement_time') and this results in the incorrect data to be returned. We have updated the article with a method to alter this order and resolve the errors with the cPanel Databases interface reporting The MySQL Server is currently offline. 0 -
Same here, MariaDB to 10.3.26 tonight edit: resolved by downgrading to 10.3.25
do you have a tutorial to downgrade?0 -
After downgrade yesterday and solve problem, today made changes in cPanelBrianK post # yum versionlock clear # /scripts/upcp
And error come back even with /scripts/restartsrv_mysql now MARIADB have last Ver. Lets trymysql -e "ALTER TABLE mysql.user MODIFY COLUMN password_last_changed timestamp AFTER max_statement_time, MODIFY COLUMN password_lifetime smallint AFTER password_last_changed, MODIFY COLUMN account_locked enum('N','Y') AFTER password_lifetime" /scripts/restartsrv_mysql
All issues solved, no error in CPANEL MYSQL DATABASES or CPANEL DB Wizard0 -
What is the estimate of leaving an official and safe solution? 0 -
do you have a tutorial to downgrade?
it was reported in the cpanel article before the last update:yum downgrade MariaDB-server MariaDB-common MariaDB-shared MariaDB-client MariaDB-compat MariaDB-devel0 -
do you have a tutorial to downgrade?
First disable MySQL service monitoring$ whmapi1 configureservice service=mysql enabled=1 monitored=0
Backup MySQL (make sure you have enough disk space)$ mkdir -p ~/mysqlbkp $ service mysql restart --skip-networking --skip-grant-tables $ mysql_upgrade $ mysqldump --all-databases --routines --triggers > ~/mysqlbkp/dbcopy.sql $ service mysql stop $ cp -r /var/lib/mysql/mysql ~/mysqlbkp/ $ service mysql start
Then downgrade:$ yum downgrade MariaDB-server MariaDB-common MariaDB-shared MariaDB-client MariaDB-compat MariaDB-devel
and re-enable MySQL service monitoring:$ whmapi1 configureservice service=mysql enabled=1 monitored=10 -
now MARIADB have last Ver. Lets try
mysql -e "ALTER TABLE mysql.user MODIFY COLUMN password_last_changed timestamp AFTER max_statement_time, MODIFY COLUMN password_lifetime smallint AFTER password_last_changed, MODIFY COLUMN account_locked enum('N','Y') AFTER password_lifetime" /scripts/restartsrv_mysql
All issues solved, no error in CPANEL MYSQL DATABASES or CPANEL DB Wizard
Confirming this fix solves the problem without needing to downgrade.0 -
Confirming this fix solves the problem without needing to downgrade.
This only fixes the MySQL Databases interface issue, not the broken PHP applications that use PHP < 7.2, right?0 -
mysql -e "ALTER TABLE mysql.user MODIFY COLUMN password_last_changed timestamp AFTER max_statement_time, MODIFY COLUMN password_lifetime smallint AFTER password_last_changed, MODIFY COLUMN account_locked enum('N','Y') AFTER password_lifetime" /scripts/restartsrv_mysql
All issues solved, no error in CPANEL MYSQL DATABASES or CPANEL DB Wizard
ERROR 1054 (42S22) at line 1: Unknown column 'password_last_changed' in 'user'0 -
@wintech2003 - correct. CPANEL-34745 is specifically for the interface issue. The PHP update case is something we're monitoring upstream in case UPS-282, with additional details here: After MariaDB update, PHP website has database connection error, 500 error, or white page: MySQL query error: Malformed communication packet 0 -
When cPanel is aware of these issues, it should send a mailing commenting what the hell is happenning, and provide the workaround. That would save both customers and support staff the hundreds of hours/man to debug, think, investigate, and add to that the time we have to lose apologizing with our end customers. Or if you don't want to spend money sending thousands of emails, add a notification function to WHM so it can show a warning saying "Hey dude! MariaDB did it again, here are the symptoms and the workaround. Thanks for keep paying. Love. cPanel." (Yep, actually I WANT to see such a warning and I assure you other admins will absolutelly love it too.) And please, don't send me to open a Feature Request. Be proactive guys, we are paying a lot of money for you to make the best panel and this feature shouldn't be that complicated. Don't stagnate please.
i couldn't agree more, first the price hikes and now they prove why they are worth the extra price tag .... pft. Took me an hour of debugging to finally find this thread here to deal with the issue, which took down hundreds of my client sites.0 -
While I like the idea, I'm not sure that type of notification system would actually work. In order to apply a change to the WHM interface with how things are currently, the server would need to experience an update. So with the MariaDB event here the timeline would look something like this: -issue caused by update Tuesday night/Wednesday morning -problem reported and identified -custom code written to confirm the server is affected by the issue, and added to a display banner in WHM -If the above can be safely done in one day, the next update runs Wednesday night/Thursday morning. This assumes that updates are not disabled to resolve the issue. So at the minimum, you're still looking at 24 hours for any type of update feature or notification with the current system. A new, more active tool might be best, but that would be an entirely new feature in the product. 0 -
Will this have any impact for servers where MariaDB has been downgraded? I'm not prepared to update MariaDB again as there are multiple sites I host that still require < PHP7.3. 0 -
The autofixer is to resolve the issues caused in the cPanel databases interface. It doesn't affect the PHP issues, so you'd still want to stay on the older version if you don't want to upgrade PHP just yet. 0 -
The autofixer is to resolve the issues caused in the cPanel databases interface. It doesn't affect the PHP issues, so you'd still want to stay on the older version if you don't want to upgrade PHP just yet.
I have a last question please. How many time you need approximately for solve this error?. I need know when I will revert the changes ("cPanel" and "yum" updates) and update again MariaDB. Thank you very much. Have a nice day.0
Please sign in to leave a comment.
Comments
89 comments