Backups taking longer on CloudLinux and cPanel 66
Hello:
just have actually same experience.... backups are taking almost 18-20 hours..... are incremental, only keeping actual day, at NFS partition.
Before last upgrade to cpanel v64, it was taking maybe 2-3 hours or less.
Any information related???
Thanks.
-
just have actually same experience.... backups are taking almost 18-20 hours..... are incremental, only keeping actual day, at NFS partition. Before last upgrade to cpanel v64, it was taking maybe 2-3 hours or less. Any information related???
There are a couple of cases referenced on the following post: Remote incremental backups - timeouts However, as far as the speed itself, could you open a support ticket using the link in my signature so we can take a closer look? Thank you.0 -
Hello Im having the same issue here. Let me tell my scenario: previous server running CL6 64bits, cpanel 11.66. Incremental backup took 2 or 3 hours to complete, every night. Low CPU load. So we moved all accounts to a new server, better CPU, double RAM, running CloudLinux 7 64 bits, cpanel 11.66. Now incremental backups need from 8 to 12 hours to complete. When the backup start at 01am, load rise from 0.XX to 2.00. Something is wrong, because it's same accounts (in fact less accounts because some users leave the new server), more CPU, more memory, new hard drives. We try replace the backup hard drive, but this dont fix the issue. How the same backup job took 3 hours on a old server and 8 to 12 hours on a more powerfull server? The main change beside the hardware is CL6 to CL7. (fully updated cpanel, kernel) Thank 0 -
Just a update: I use this small script to update the incremental backup: KONTA=`ls -1A /var/cpanel/users/` for x in `echo -n "$KONTA"`;do if [ -d "/home/$x" ]; then /usr/local/cpanel/bin/pkgacct --incremental "$x" /backup/cpbackup/daily backup fi done
Result? backup done in less than 3 hours - like on previous server. If I run incremental backup using the legacy backup OR the new backup system, the result is the same: 12 hours to complete backup. Something is throttling the backup script (legacy and new) of cpanel, I dont know if is CloudLinux 7 related or anything else.0 -
Same problems for me. I had to split accounts into more cpanel servers just to make the backups complete in reasonable time. I believe it may have to do with the new meta files and/or because the changes cpanel made in the way incrementals are made/processed. Pruning of the old backup dir after the backup finishes as well, in my case just the pruning sometimes took 1-2 hours or more). So i personally never saw any backup improvements ever since v60, it only degraded more in terms of performance/impact on the systems. One more thing that I'm not sure if it is related or not, but on systems with MariaDB 10.2 i notice very high memory increase, probably memory leaks. I will probably start testing the method mindnetcombr described, until cpanel will decide to remove that as well. 0 -
One more thing that I'm not sure if it is related or not, but on systems with MariaDB 10.2 i notice very high memory increase, probably memory leaks. I will probably start testing the method mindnetcombr described, until cpanel will decide to remove that as well.
I dont use MariaDB, so I think is not related. I try use legacy backup system (incremental) and same issue. When I use a simple script to pkgacct all accounts (the script on my previous message above), server have no issues. The more strange for me is: this start only when I move from CL6 to CL7. I have servers running CL6 doing incremental backups daily in less than 3 hours (650 accounts average, 1Tb of data average). Sure meta files on new backup system is problem too - and like you said, pruning the old days cause high load on my servers. Based on my experience the new backup system is a total disaster. I hope they keep the script pkgacct - so I can run my own backup script, without any issues.0 -
Hello, Please feel free to open a support ticket if you are facing this issue so we can take a closer look. Post the ticket number here and we will update this thread with the outcome. Thank you. 0 -
Update: now I have 2 servers on same condition, cPanel + CloudLinux 7, incremental backups take 12 hours. If I run the script below, backup complete in 3 hours: KONTA=`ls -1A /var/cpanel/users/` for x in `echo -n "$KONTA"`;do if [ -d "/home/$x" ]; then /usr/local/cpanel/bin/pkgacct --incremental "$x" /backup/cpbackup/daily backup fi done0 -
Hi @mindnetcombr, Could you open a support ticket using the link in my signature so we can take a closer look at one of the affected systems? You can post the ticket number here and we will update this thread with the outcome. Thank you. 0 -
Hi @mindnetcombr, Could you open a support ticket using the link in my signature so we can take a closer look at one of the affected systems? You can post the ticket number here and we will update this thread with the outcome. Thank you.
Hi I have the ticket #8858129 handled by cPanel Shoshana. Your support team is looking on my two servers at same time, doing some research and I think they found a possibly culprit for this, but I'll wait for the investigation to finish before I say anything. You can read on ticket. Thank Michael!0 -
Hello, We are tracking reports of backups taking significantly longer to complete (compared to cPanel version 64) on servers using cPanel version 66 and CloudLinux as part of internal case CPANEL-15825. I'll monitor this case and update this thread with more information as it becomes available. Thank you. 0 -
Hello, I have two servers running CloudLinux 7 and cPanel 11.66, in version 11.64 it took 3 hours, now in version 11.66 each backup takes 12 hours. This has turned into a daily hell, because backups take so long that they eventually reach business hours, and the server gets overloaded. I know that the developers are researching, it was me who opened the ticket that culminated in the opening of the investigation case they are doing, but the days are passing and there is no solution. I have this problem for 15 days. Is there any way to downgrade from 11.66 to 11.64? I believe the 11.64 version dont suffer this problem, If it were possible I might have an immediate solution. 0 -
Hello, Our developers are currently in communication with CloudLinux to further investigate the cause of this issue. I'll continue to monitor the case and update this thread with more information as it becomes available. Is there any way to downgrade from 11.66 to 11.64? I believe the 11.64 version dont suffer this problem, If it were possible I might have an immediate solution.
You can not downgrade to an earlier major version (e.g. 66 to 64) after updating. Thank you.0 -
More info maybe usefull for someone else: CloudLinux 6 dont have this issue. I have servers running CloudLinux 6 and backup run as usual. But all my servers with CloudLinux 7 have the same isse. So if you have a server with CloudLinux 6 dont upgrade to 7 until this issue is fixed. 0 -
my backup took also 11 hours today but i just noticed that it takes every 7 days that long... ( 1 TB of data ) 0 -
Hello cPanelMichael, Any progress on this issue? Thank 0 -
Hi @mindnetcombr, There's no update to report at this time. I'll continue to monitor the case and update this thread once new information is available. Thank you. 0 -
Hello, Backups run perfect using the latest v68 build and activating the "Disable phpMyAdmin information schema searches" which can be found in the Tweak settings. This is much better. Will monitor more servers in the next days. backups start at 00:01 v66 -rw------- 1 root root 882K Oct 12 03:50 1507755722.log -rw------- 1 root root 880K Oct 13 03:26 1507842122.log -rw------- 1 root root 884K Oct 14 04:16 1507928521.log -rw------- 1 root root 882K Oct 15 03:36 1508014922.log -rw------- 1 root root 884K Oct 16 04:01 1508101322.log -rw------- 1 root root 885K Oct 17 04:03 1508187722.log -rw------- 1 root root 889K Oct 18 04:33 1508274122.log -rw------- 1 root root 877K Oct 19 03:38 1508360522.log -rw------- 1 root root 869K Oct 20 04:22 1508446921.log -rw------- 1 root root 869K Oct 21 02:49 1508533321.log v68 -rw------- 1 root root 882K Oct 22 01:42 1508619722.log0 -
Hello, Backups run perfect using the latest v68 build and activating the "Disable phpMyAdmin information schema searches" which can be found in the Tweak settings. This is much better. Will monitor more servers in the next days.
Hello InterServed, Thank you for the information. How are going the tests with cpanel 11.68? Are you using CloudLinux 6 or 7? Here I found this: if I reboot the server (CloudLinux 7) and run backup as soon server boot, the incremental backups run quick. But after 12-24 hours of a boot, the issue begins.0 -
Hello, Backups run perfect using the latest v68 build and activating the "Disable phpMyAdmin information schema searches" which can be found in the Tweak settings. This is much better. Will monitor more servers in the next days.
Hello, InterServed Yesterday I update to 11.68.09 (release) CloudLinux 7 and this not fixed the issue. Unfortunately0 -
Same issue. Still waiting for a fix. It seems nobody cares that, but it's a big issue for backup system. What happened to internal case CPANEL-15825? 0 -
Hello, CPANEL-15825 is still open, however the investigation currently shows the issue relates to a bug with CloudLinux itself as opposed to with the cPanel software. CloudLinux is working to address the issue as part of CloudLinux case number CAG-764. I'll update this thread with more information as it becomes available. Thank you. 0 -
Hello I contacted CloudLinux support, told about this thread and about the case "CAG-764". They sugest me to install Cage-FS beta version, to fix this issue. I installed the beta version of Cage-FS on CloudLinux7, and now backup reduced from 24 hours to 8 hours. It's better, but not fix the issue. I try this on 2 servers running CloudLinux 7, same results. If I reboot my servers and run a custom script to perform incremental backup, this take only 2 or 3 hours. (the script is below) Im giving up on this issue, is much time to fix this - I dont know if is a CloudLinux or cPanel issue - the fact is this affect backup. So I will re-install 2 servers using CloudLinux 6 (no issues). My script, save as like backup.sh, after run you can look the log in /root/logbkp2.log KONTA=`ls -1A /var/cpanel/users/` for x in `echo -n "$KONTA"`;do echo " " >> /root/logbkp2.log echo " " >> /root/logbkp2.log while [ "$(cut -d . -f 1 /proc/loadavg)" -gt "4" ];do echo "..." >> /root/logbkp2.log NOW=$(date +"%d-%m-%Y - %T") echo "HIGH LOAD SLEEPING 20 SECONDS... $NOW" >> /root/logbkp2.log sleep 20 echo "..." >> /root/logbkp2.log done if [ -d "/home/$x" ]; then echo "--------" >> /root/logbkp2.log NOW=$(date +"%d-%m-%Y - %T") echo "ACCOUNT $x STARTED $NOW" >> /root/logbkp2.log echo "--------" >> /root/logbkp2.log /usr/local/cpanel/bin/pkgacct --incremental "$x" /backup/cpbackup/daily backup >> /root/logbkp2.log echo "--------" >> /root/logbkp2.log NOW=$(date +"%d-%m-%Y - %T") echo "ACCOUNT $x ENDED $NOW" >> /root/logbkp2.log echo "--------" >> /root/logbkp2.log fi done0 -
This has been blowing my mind for past few months, and i blamed a new support company that got involved with our two servers. We are trying to backup over 800Gb of data on two servers for a daily backup and it doesnt complete in time due to the size. Should i update this cage-fs to a beta version. comment so i can read for new comments. 0 -
We are trying to backup over 800Gb of data on two servers for a daily backup and it doesnt complete in time due to the size.
The particular issue on this thread relates to a change in the backup time in cPanel version 66. Is that the same issue you are facing? If so, you could try updating to the beta version of CageFS to see if that helps: Beta: CageFS and liblve updated Thank you.0 -
I have now logged this bug with Cloudlinux, as apparently no one else has reported this bug with them.. I will update you all when i find out more from CL Support. 0 -
Last update: on the same hardware I re-install operating system, abandon CL7 and back to CL6. No issues with backup. This bug affect only CL7. Incremental backup, 650 accounts, 800Gb of data, daily update took only 2 hours on CL6. 0 -
Hello, I believe this may solve the slow backup on CloudLinux 7 : Beta: tuned-profiles-cloudlinux updated 0 -
Hello again, One more update about this, i believe even the latest beta CL7 kernel might have an impact on the backup speed, will continue to test more servers for more days. For example, compare the backup logs times (they start generating at 00:01): root@nlam-cp1:[/usr/local/cpanel/logs/cpbackup]: ls -alh total 16M drwx------ 2 root root 4.0K Dec 16 03:23 . drwx--x--x 7 root root 4.0K Dec 16 00:05 .. -rw------- 1 root root 1.4M Dec 6 01:12 1512511502.log -rw------- 1 root root 1.4M Dec 7 01:11 1512597901.log -rw------- 1 root root 1.4M Dec 8 01:57 1512684302.log -rw------- 1 root root 1.5M Dec 9 01:35 1512770701.log -rw------- 1 root root 1.4M Dec 10 01:25 1512857102.log -rw------- 1 root root 1.4M Dec 11 02:00 1512943502.log -rw------- 1 root root 1.4M Dec 12 01:49 1513029901.log -rw------- 1 root root 1.4M Dec 13 01:43 1513116301.log -rw------- 1 root root 1.4M Dec 14 01:12 1513202701.log -rw------- 1 root root 1.4M Dec 15 01:43 1513289101.log -- after latest CL7 beta kernel + tuned-profiles update and a reboot (47min full backups) -rw------- 1 root root 1.4M Dec 16 00:47 1513375501.log0 -
Disabling this did help "phpMyAdmin information schema searches" alot! Also cpuwatch messages were weird. We have a 20 core VM but it kept giving this error: cpuwatch (Mon Dec 18 10:47:26 2017): System load is currently 2.23; waiting for it to go down below 1.75 to continue " cpuwatch (Mon Dec 18 10:47:56 2017): System load is currently 1.35, which is below the threshold of 1.75. Continuing " cpuwatch (Mon Dec 18 10:50:02 2017): System load is currently 2.99; waiting for it to go down below 1.75 to continue " cpuwatch (Mon Dec 18 10:50:47 2017): System load is currently 1.74, which is below the threshold of 1.75. Continuing " How do I adjust it so that it stops cpuwatch from nagging. 0 -
Hello, I believe this may solve the slow backup on CloudLinux 7 : Beta: tuned-profiles-cloudlinux updated
Did you apply this update? If so has it worked..0
Please sign in to leave a comment.
Comments
36 comments