Backup to Amazon S3 Doesn't Obey Retention Rules
It seems that since the update to cPanel/WHM a few weeks ago [possibly since build 30 of ver 64] (current version I have is cPanel & WHM 64.0 (build 36)) that the backup to S3 has stopped obeying the retention rules like maybe the s3 sync command isn't quite right?
The local folder that I'm using /backups retains the correct number of backups, but the S3 destination does not.
I have "Strictly enforce retention, regardless of backup success." checked, and I have checked the backup log, and see no notices or errors in that.
Thanks.
-
Hello, Could you review the transport logs in the /usr/local/cpanel/logs/cpbackup_transporter/ directory and verify if you see any "prune" activity after the backups are copied to the remote destination? Also, could you verify if you are seeing multiple copies of the same backup archive on the Amazon S3 server? Thank you. 0 -
Last log entry after successful upload of last scheduled domain: [2017-08-02 02:06:08 -0500] info [cpbackup_transporter] cpbackup_transporter - Checking queue for tasks [2017-08-02 02:06:08 -0500] info [cpbackup_transporter] cpbackup_transporter - Exiting - the queue has been emptied; no more work to do [2017-08-02 02:06:08 -0500] info [cpbackup_transporter] cPanel Backup Transporter Queue Daemon is being stopped. No activity or term "prune" found, no mention of deletion sig. to amazon to delete past directory. Checked Amazon S3 per request, the backup files are stored in dated folders, folders past the requested four are not duplicate dates and are not being removed. I can send the cPanel log if needed, as it is not that large since we do not back up all accounts, only active customer domains. 0 -
Hello @geotex, Internal case CPANEL-14906 is open to address an issue where backup pruning does not occur on Amazon S3 destinations when no folder is specified in the destination's configuration. I'll monitor this case and update this thread with more information as it becomes available. As a temporary workaround, you can create a subdirectory in your Amazon S3 instance, and then edit the "Folder" option for the Amazon S3 destination from within "WHM >> Backup Configuration" so that it uses that specific subdirectory. This should allow pruning to occur. Additionally, you may want to manually remove any existing backups that were not pruned from your Amazon S3 destination. Thank you. 0 -
Hi Michael. Thanks for your reply. I thought I was subscribed, but didn't get an email? Anyway... with today's backup, it worked properly, and there is a "Prune" operation in the log. From a previous log, there was a prune, but it didn't actually do it. There were other copy operations AFTER the prune [2017-07-29 05:36:42 -0400] info [cpbackup_transporter] cpbackup_transporter - Processing next task [2017-07-29 05:36:42 -0400] info [cpbackup_transporter] Instantiating Object [2017-07-29 05:36:42 -0400] info [cpbackup_transporter] Starting a "prune" operation on the "xxx Bucket" destination ID "----------------------------". [2017-07-29 05:36:42 -0400] info [cpbackup_transporter] Performing prune operation, retaining 4 items on: xxx Bucket [2017-07-29 05:36:42 -0400] info [cpbackup_transporter] cpbackup_transporter - Checking queue for tasks [2017-07-29 05:36:42 -0400] info [cpbackup_transporter] cpbackup_transporter - Processing next task [2017-07-29 05:36:42 -0400] info [cpbackup_transporter] cpbackup_transporter - Checking queue for tasks [2017-07-29 05:36:42 -0400] info [cpbackup_transporter] cpbackup_transporter - Exiting - the queue has been emptied; no more work to do [2017-07-29 05:36:42 -0400] info [cpbackup_transporter] cPanel Backup Transporter Queue Daemon is being stopped. [2017-07-30 02:04:42 -0400] info [cpbackup_transporter] Initializing log file [2017-07-30 02:04:42 -0400] info [cpbackup_transporter] cPanel Backup Transporter Queue Daemon started. [2017-07-30 02:04:42 -0400] info [cpbackup_transporter] cpbackup_transporter - started [2017-07-30 02:04:42 -0400] info [cpbackup_transporter] cpbackup_transporter - Checking queue for tasks [2017-07-30 02:04:42 -0400] info [cpbackup_transporter] cpbackup_transporter - Processing next task [2017-07-30 02:04:42 -0400] info [cpbackup_transporter] Instantiating Object [2017-07-30 02:04:42 -0400] info [cpbackup_transporter] Starting a "copy" operation on the "xxx Bucket" destination ID "xxxxxxxxxxxxxxxxxxxxxxxx".
However.... today's backup did, and the 'prune' operation was the last item in the log before "Exiting"0 -
Thank you. Set up as recommended, we will see how that goes. I only do full server backups twice a week, so will take a couple of weeks to see if that fixes the issue. Looking forward to seeing your updates post. George 0 -
Hi @HowardE, Do you have anything entered for the "Folder" option in the backup configuration for your Amazon S3 destination in "WHM >> Backup Configuration"? Thank you. 0 -
Hi Michael. I do not have a folder specified. The backup is to the root of this particular bucket. The backup transport appears to work fine, just does not prune. 48031 If making a folder underneath the bucket will make a difference, I can try that? [2017-08-04 05:58:15 -0400] info [cpbackup_transporter] Starting a "prune" operation on the "Howard's Bucket" destination ID "xxxxxxxxxxxxxxxxxxxxxxxxxxxx". [2017-08-04 05:58:15 -0400] info [cpbackup_transporter] Performing prune operation, retaining 4 items on: Howard's Bucket
It says it started to prune. I've enabled logging on that S3 bucket and will report what it says after the next backup. In the meantime, I'm creating a folder called 'backups' and moving the 4 existing backups there, and i'll let the backup run tonight.0 -
If making a folder underneath the bucket will make a difference, I can try that?
Yes, create the folder and then edit the destination in "WHM >> Backup Configuration" to define that folder name. This will act as a temporary workaround to the issue described on my earlier post. Thanks!0 -
It doesn't say if there should be a slash or not. 48035 0 -
So far... adding the folder seems to have worked. I'll confirm after a few more backup cycles. Thanks. 0 -
After a few backups, I can confirm that this is working. 0 -
Enough time has passed to exceed the backup retention, it seems the changes suggested earlier are working, the folder on amazon s3 is deleting now as instructed. 0 -
Hello guys, I'm having the same issue. I need an ETA for when may be expected that Internal case CPANEL-14906 will be finally solved. I manage nearly 25 cPanel servers that make daily backups of hundreds of accounts on to different S3 buckets, on 12 different Amazon accounts, with different combinations of daily/weekly retention rules, all this combined with different S3 lifecycles rules moving old files to Amazon Glacier.. So :) you can bet it's not cool nor easy to reconfigure everything to start using a directory within every bucket, on every Amazon account. I need to know when will this be addressed. Soon, I hope. (More than 1 month have pased, and this bug is costing more money and work to all of us) 0 -
Hi @Kent Brockman, I don't have a specific time frame to offer at this time, but I've noted this forums thread in the internal case and will update this thread with new information as soon as it's available. Thank you. 0 -
ok, I hope you guys can fix this soon. Thanks! 0 -
I think it's cute that Cpanel introduces a bug like this with a major upgrade, then we have wait until the next major upgrade to have it fixed. 0 -
I think it's cute that Cpanel introduces a bug like this with a major upgrade, then we have wait until the next major upgrade to have it fixed.
Hello, I removed the previous posts as they were in reference to separate cases. The particular issue discussed on this thread is part of internal case CPANEL-14906. We may publish a resolution to cPanel 66, but no specific time frame or decision on that is available at this time. I'll update this thread as soon as new information is available. Thank you.0 -
We are suffering from this same issue. I first noticed when our S3 bill shot up to several hundred $'s Having to delete old backups every few days manually in S3 until its resolved, very frustrating. Have tried the 'put them into a sub folder within the bucket' fix and that did not work either. Posting here to track any updates / alternative suggestions. 0 -
Having to delete old backups every few days manually in S3 until its resolved, very frustrating. Have tried the 'put them into a sub folder within the bucket' fix and that did not work either.
Did you also edit the "Folder" option for the Amazon S3 destination from within "WHM >> Backup Configuration" so that it uses the specific subdirectory path you created? If so, feel free to open a support ticket using the link in my signature so we can take a closer look, as it's possible the issue you are facing is unrelated to case CPANEL-14906. Thank you.0 -
We've just been hit by this as well. Our Amazon bill quadrupled because of this! This is not just a minor inconvenience - it is costing your paying customers money! Please sort this out as a matter of urgency! 0 -
Hello I can reproduce this same issue using RSYNC as destination (no Amazon S3). I have my own rsync server, with a folder name, cpanel backup prune old backups. 0 -
Is there any progress in resolving this? It is impacting us as well. 0 -
Hello, To update, internal case CPANEL-14906 was published as part of cPanel version 66.0.26: Fixed case CPANEL-14906: Amazon S3: prune backups when no folder specified. Thank you. 0
Please sign in to leave a comment.
Comments
23 comments