cPanel daily maintenance running mysqlcheck
Starting on Nov 8, something in the cPanel daily maintenance script is starting a mysqlcheck command for every one of our database tables. Due to the size of our databases, this is causing a 4-6 hour outage of a significant part our operations every day until someone can go in and kill the process.
We are back on the cPanel EOL release of 11.110.0.48 which was just updated on Nov 6.
Obviously I can suppress running the daily maintenance but since it updates site certificates and a lot of other important stuff I don't want to leave it that way.
This is the job that runs:
UID PID PPID C SZ RSS PSR STIME TTY TIME CMD
root 10385 1 0 12673 3388 11 04:24 ? 00:00:00 /usr/bin/mysqlcheck -c -m -A --silent
Please let me know how to fix this.
-
Hey there! The check_mysql script has been present in the nightly update for as long as I can remember, as that is what notifies you if there are problems with the databases on the machine every morning, if you've ever received one of those emails.
This is controlled by the /scripts/check_mysql script that runs during the nightly update, which can be seen in the logs here (I just pulled one from October 2 on my system, but they should all be the same)
[2024-10-02 05:35:16 -0400] Processing: Ensuring an "Active" MySQL profile is set
[2024-10-02 05:35:16 -0400] - Finished in 0.002 seconds
[2024-10-02 05:35:16 -0400] - Processing command `/usr/local/cpanel/scripts/check_mysql`
[2024-10-02 05:35:16 -0400] [/usr/local/cpanel/scripts/check_mysql] “check_mysql” will complete in the background (process ID 2896278).
[2024-10-02 05:35:16 -0400] - Finished command `/usr/local/cpanel/scripts/check_mysql` in 0.108 secondsI don't have any other reports of this check causing issues, even on servers with very large databases, so I'm wondering if something else may be happening here.
If you manually run /scripts/check_mysql on the command line you'll get similar output:
[root@host updatelogs]# /scripts/check_mysql
“check_mysql” will complete in the background (process ID 2235160).If you run that and then watch the MySQL system with the "mysqladmin proc status" command, do you see anything abnormal happening there?
0 -
One huge thing I forgot to mention is that we run with mySQL support disabled in cPanel for a variety of reasons:
This is what is typically in our update log:
[2024-11-13 04:24:46 -0600] Processing: Ensuring an "Active" MySQL profile is set
[2024-11-13 04:24:46 -0600] - Finished in 0.049 seconds
[2024-11-13 04:24:46 -0600] - Processing command `/usr/local/cpanel/scripts/check_mysql`
[2024-11-13 04:24:46 -0600] - Finished command `/usr/local/cpanel/scripts/check_mysql` in 0.121 seconds
There is no indication that the check_mysql process has been started in the background (as I assume there should not be since it is disabled).
Even so, the mysqlcheck command must be running from the maintenance script because when I disabled it entirely in cron the mysqlcheck process did not run. When I just now ran the maintenance script manually, by the time the script completed the mysqlcheck process had been started.Taking your suggestion, I also ran scripts/check_mysql manually. The result is:
MySQL is disabled.
As I would have expected. The mysqlcheck process was not started.
It's in there somewhere and it doesn't seem to check that we have monitoring disabled.0 -
Following up, I did run a quick scan on the scripts directory and came up with this:
./elevate-cpanel: my $mysqlcheck_path = Cpanel::Binaries::path('mysqlcheck');
./elevate-cpanel: my $output = $self->ssystem_capture_output( $mysqlcheck_path, '-c', '-m', '-A', 'That is the exact command we see in our active processes but I have no idea why the elevate-cpanel script would be running, I don't see it in the log. Interesting that the change date on that script is the exact date we started having the problem.
Looking at what the elevate-cpanel script appears to be doing I think we are going to leave our daily update completely disabled until this situation is resolved. If cPanel is trying to update our OS level it would be an unmitigated disaster.0 -
We definitely would never ever try and update someone's OS without telling them - that would be a serious disaster.
That script is just there in case you want to run ELEvate, but that isn't what is calling your mysqlcheck. I believe that the nightly update runs with cron and has different parameters than manually running the script. Let me look into that and I'll get back with you soon!
0 -
Can you send me the output of this command?
grep check_mysql /var/cpanel/updatelogs/last
0 -
[2024-11-14 17:28:13 -0600] - Processing command `/usr/local/cpanel/scripts/check_mysql`
[2024-11-14 17:28:14 -0600] - Finished command `/usr/local/cpanel/scripts/check_mysql` in 0.188 secondsThose times are from when I ran the update manually yesterday. When I ran it I used the same parameters as the cron entry:
/usr/local/cpanel/scripts/upcp --cron
0 -
Ah, you'd need to go back to a different log file to get the one where it wasn't run manually. I just wanted to see if it gave a timeout or anything different when it runs normally.
0 -
The last one that ran by cron is similar:
[2024-11-12 04:24:56 -0600] - Processing command `/usr/local/cpanel/scripts/check_mysql`
[2024-11-12 04:24:56 -0600] - Finished command `/usr/local/cpanel/scripts/check_mysql` in 0.123 secondsThe log for the day before the problem started has similar entries:
[2024-11-07 04:24:54 -0600] - Processing command `/usr/local/cpanel/scripts/check_mysql`
[2024-11-07 04:24:54 -0600] - Finished command `/usr/local/cpanel/scripts/check_mysql` in 0.148 seconds0 -
I ran some process snapshots to determine when mysqlcheck was being called during the daily run:
root 5990 1 5 40794 52080 10 21:43 ? 00:00:00 /usr/local/cpanel/3rdparty/bin/perl /usr/local/cpanel/scripts/check_security_advice_changes --notify -
root 6047 5990 9 42883 55344 9 21:43 ? 00:00:00 /usr/local/cpanel/3rdparty/bin/perl /usr/local/cpanel/scripts/elevate-cpanel --check /var/cpanel/ele
root 6055 6047 0 12671 3384 11 21:43 ? 00:00:00 /usr/bin/mysqlcheck -c -m -A --silentThat is the answer.
0 -
Apparently going to the WHM Security Advisor panel kicks off the same mysqlcheck process. I was hoping to find an option to turn off security advisor warnings for Elevate but I couldn't find anything. I don't need to be told every time it runs about the items that are blocking an upgrade. Actually I would like to be told never again unless I specifically request it.
0 -
So ideally, you'd like an option in WHM, maybe Tweak Settings, to just disable the mysqlcheck completely, right?
0 -
This is definitely caused by the /usr/local/cpanel/scripts/elevate-cpanel running during /scripts/upcp.
This needs to be disabled ASAP as databases with huge table sizes are causing major issues.
For now, we have set the permissions of that script to 000 and set the script to immutable. Please update this thread when a fix is applied.
0 -
So ideally, you'd like an option in WHM, maybe Tweak Settings, to just disable the mysqlcheck completely, right?
At a minimum, there should be an option on the WHM Security Advisor panel to opt-in to running the Elevate script and the default should be off. Even better, the Elevate check should be removed from Security Advisor entirely because it doesn't belong there. Who needs to run that every day? It should be a button somewhere else that I can click on demand to run that script of doom.
0 -
Thanks all - I know what's happening behind the scenes now and I have a plan and I'll update you soon!
0 -
Alright - here's the current plan.
FOR NOW - continue using chmod or chattr on the elevate-cpanel script so it dosn't run.
FOR TOMORROW - I've created case CPANEL-46102 for our developers to either implement a way to disable the elevate-cpanel script completely, or a way to disable the mysqlcheck component. How and what exactly is happening there hasn't been decided but the team is going to talk about it tomorrow and I'll be sure to post once I have more details.
0 -
For now, we have set the permissions of that script to 000 and set the script to immutable.
Thanks for the tip on using lsattr. I had tried setting the execute permission off before but apparently there is a repair process that turns it back on and then runs it.
0 -
One thing I have noticed when using the immutable file strategy is that you will accumulate a version of elevate-cpanel every day because any update cannot replace what you already have in your scripts directory. I had several files with names like elevate-cpanel859970836 which I have deleted.
I suppose I should just manually replace the script I have with the updated version but I'm assuming the behavior would re-appear the next time the script is changed. I didn't see any code changes in the area of where mysqlcheck is called.0 -
Well that's not ideal. BUT, I have confirmed that version 59 of ELevate will run the mysqlcheck in a different mode so it doesn't bog down the system, and only gives output if it actually detects an issue with the databases. I don't have an ETA on when it will be released, but it could be as soon as later today - if not, I'd expect it within the week.
0 -
Version 59 was just released, which contains this fix!
0
Please sign in to leave a comment.
Comments
19 comments