Incremental backup takes longer every day
I think there's a bug in cPanel 60 that causes the incremental backups to take longer every day, until they take more than 10 hours. A server reboot cures this and restarts the cycle.
This doesn't happen to my servers on Cloudlinux 5/Cpanel 58 (which I'm in the process of migrating).
My experience:
We migrated a machine from CL5/cPanel 58 to CL7/cPanel 60. We also upgraded the hardware massively, including a move to full SSD RAIDs.
The first nightly backup took 1 hour, where the previous system was taking 6-7 hours. I was amazed at the difference the new hardware made. Little by little, the incremental backup started to take longer and longer until it was taking more than any of our old servers on non SSD hardware, with the same amount of accounts and data.
After a server reboot, backup time went back to little more than 1 hour. This cycle happened a couple times, each taking about 1-2 weeks.
Now here's the clincher. We migrated this same server to another datacenter, and left the old machine running, with all the accounts suspended, and no traffic in the server. Still, each night, the backups take longer, even with no new data being generated in the server at all!
Sadly, we are experiencing the same problem with the new hardware as well. These machines are very very powerful and their load is minimal.
How can we troubleshoot this?
Here's my backup info:
The Tweak Settings for the backups have been changed to the cPanel defaults. "Extra CPUs for server load" was previously set to 5, and "I/O priority level at which nightly backups are run" to 3. Is it possible that somehow these settings haven't been applied?
cat /var/cpanel/backups/config
---
BACKUPACCTS: 'yes'
BACKUPBWDATA: 'yes'
BACKUPDAYS: 0,1,2,3,4,5,6
BACKUPDIR: /backup
BACKUPENABLE: 'yes'
BACKUPFILES: 'yes'
BACKUPLOGS: 'yes'
BACKUPMOUNT: 'no'
BACKUPSUSPENDEDACCTS: 'yes'
BACKUPTYPE: incremental
BACKUP_DAILY_ENABLE: 'yes'
BACKUP_DAILY_RETENTION: 1
BACKUP_MONTHLY_DATES: 1
BACKUP_MONTHLY_ENABLE: 'no'
BACKUP_MONTHLY_RETENTION: 1
BACKUP_WEEKLY_DAY: 0
BACKUP_WEEKLY_ENABLE: 'no'
BACKUP_WEEKLY_RETENTION: 4
ERRORTHRESHHOLD: 3
GZIPRSYNCOPTS: --rsyncable
KEEPLOCAL: 1
LINKDEST: 0
LOCALZONESONLY: 'no'
MAXIMUM_RESTORE_TIMEOUT: 21600
MAXIMUM_TIMEOUT: 7200
MYSQLBACKUP: accounts
POSTBACKUP: 'no'
PREBACKUP: -1
PSQLBACKUP: 'no'The Tweak Settings for the backups have been changed to the cPanel defaults. "Extra CPUs for server load" was previously set to 5, and "I/O priority level at which nightly backups are run" to 3. Is it possible that somehow these settings haven't been applied?
-
The last backup of the machine that has no new data being generated, most services stopped, and load close to zero took more than 14 hours. It would be great if this could be troubleshooted before I decomission the server. Anyone else having backups that get slower every day? 0 -
Hello! The best way to begin troubleshooting that would be to check what is occurring in the backup logs at /usr/local/cpanel/logs/cpbackup/. You should be able to use the information and the times there to help pinpoint what is taking the most time during the backup. 0 -
Unfortunately, the logs don't solve the mistery. Some accounts, randomly, will simply start taking longer and displaying dots in the logs. It's not always the same account. 0 -
Hello, Feel free to open a support ticket using the link in my signature so we can take a closer look and see what's happening. You can post the ticket number here so we can update this thread with the outcome. Thank you. 0 -
My datacenter has already opened a ticket. It's probably a memory leak related to Cloudlinux 7, and we've had it happen on more than one server already. I don' thave the ticket number though. 0 -
Hi, I noticed this too, on a server not running cloudlinux. I thought it was client uploading more data and didn't care to dig in, but it has increased about 1/6 to 1/8 the time, recently. 0 -
I thought it was client uploading more data and didn't care to dig in, but it has increased about 1/6 to 1/8 the time, recently.
Would you mind opening a support ticket using the link in my signature so we can take a closer look and see what's happening? You can post the ticket number here so we can update this thread with the outcome. Thank you.0
Please sign in to leave a comment.
Comments
7 comments