Amazon S3 Transport error
I have backups configured to transfer daily to an Amazon S3 bucket on multiple accounts. This has worked fine in the past, but recently I have been receiving an error message on one very large account, though other accounts are transferring without incident. I thought the problem could be the size of the account, as Amazon has a 5 GB limit for transfers, so I did some maintenance to reduce size for transfer (such as deleting old log files) -- and reduced the size of the backup to 3.3 GB -- but the error persists.
My cpbackup_transporter.log shows Amazon reporting a 400 error on upload, as follows:
Each night 3 attempts are made for the upload for this account -- each time it fails, and then the other accounts on the server are backed up and uploaded to the S3 Bucket without a problem. I have increased timeout settings but that does not appear to be the issue. At this point I am at a loss as to what I can do to resolve this issue.
[2017-05-07 02:13:23 -0700] info [cpbackup_transporter] Uploading /backup/2017-05-07/accounts/account1.tar.gz to /2017-05-07/account1.tar.gz
[2017-05-07 02:13:23 -0700] info [cpbackup_transporter] Attempting to upload /backup/2017-05-07/accounts/account1.tar.gz to /2017-05- 07/account1.tar.gz for destination: Amazon S3
[2017-05-07 02:13:23 -0700] info [cpbackup_transporter] Upload attempt #1 starting for /backup/2017-05-07/accounts/account1.tar.gz to /2017-05-07/account1.tar.gz for destination: Amazon S3
[2017-05-07 02:16:22 -0700] warn [cpbackup_transporter] Upload attempt failed: Amazon::S3: Amazon responded with 400 Bad Request
at /usr/local/cpanel/3rdparty/perl/524/lib64/perl5/cpanel_lib/Amazon/S3/Bucket.pm line 134.
at /usr/local/cpanel/Cpanel/LoggerAdapter.pm line 27.
Cpanel::LoggerAdapter::warn(Cpanel::LoggerAdapter=HASH(0x1938ed8), "Upload attempt failed: Amazon::S3: Amazon responded with 400 "...) called at /usr/local/cpanel/Cpanel/Backup/Queue.pm line 494
Cpanel::Backup::Queue::transport_backup::attempt_to_upload_file(Cpanel::Backup::Queue::transport_backup=HASH(0x1f1fa78), Cpanel:: Transport::Files::AmazonS3=HASH(0x24a5f08), "/backup/2017-05-07/accounts/account1.tar.gz", "/2017-05-07/account1.tar.gz", Cpanel: :LoggerAdapter=HASH(0x1938ed8)) called at /usr/local/cpanel/Cpanel/Backup/Queue.pm line 256
Cpanel::Backup::Queue::transport_backup::process_task(Cpanel::Backup::Queue::transport_backup=HASH(0x1f1fa78), cPanel::TaskQueue: :Task=HASH(0x24134c0), Cpanel::LoggerAdapter=HASH(0x1938ed8)) called at /usr/local/cpanel/3rdparty/perl/524/lib64/perl5/cpanel_lib/cPanel /TaskQueue.pm line 586
eval {...} called at /usr/local/cpanel/3rdparty/perl/524/lib64/perl5/cpanel_lib/cPanel/TaskQueue.pm line 589
cPanel::TaskQueue::__ANON__() called at /usr/local/cpanel/3rdparty/perl/524/lib64/perl5/cpanel_lib/cPanel/StateFile.pm line 238
eval {...} called at /usr/local/cpanel/3rdparty/perl/524/lib64/perl5/cpanel_lib/cPanel/StateFile.pm line 238
cPanel::StateFile::Guard::call_unlocked(cPanel::StateFile::Guard=HASH(0x23e1d90), CODE(0x23e1ca0)) called at /usr/local/cpanel/3r dparty/perl/524/lib64/perl5/cpanel_lib/cPanel/TaskQueue.pm line 594
cPanel::TaskQueue::process_next_task(cPanel::TaskQueue=HASH(0x23e1dd8)) called at /usr/local/cpanel/bin/cpbackup_transporter line 151
eval {...} called at /usr/local/cpanel/bin/cpbackup_transporter line 149
[2017-05-07 02:16:22 -0700] info [cpbackup_transporter] Error encountered
[2017-05-07 02:16:22 -0700] info [cpbackup_transporter] Error is not a reference Amazon::S3: Amazon responded with 400 Bad Request
at /usr/local/cpanel/3rdparty/perl/524/lib64/perl5/cpanel_lib/Amazon/S3/Bucket.pm line 134.Each night 3 attempts are made for the upload for this account -- each time it fails, and then the other accounts on the server are backed up and uploaded to the S3 Bucket without a problem. I have increased timeout settings but that does not appear to be the issue. At this point I am at a loss as to what I can do to resolve this issue.
-
Hi, Check for the Amazon error code.. look for 400 Bad Request means in it.. Error Responses - Amazon Simple Storage Service 0 -
I've done that and it doesn't have helpful information. I wouldn't be posting here if I hadn't first exhausted other resources. 0 -
Hi AM2015, I don't use S3, but I was intrigued by your issue, and thought I would have a look for myself. As you rightly pointed out - the S3 docs unhelpfully report that the string "400 Bad Request" can refer to 51 different issues, without really giving you a clue as to what to look at (thus demonstrating how gratuitous one-liner answers can often have little or no value !) The only thing I picked up on was from the cPanel documentation that warns that the backup folder (if defined) must exist in your S3 bucket prior to attempting the upload. It appears that you are asking the backup system to copy a file from: /backup/2017-05-07/accounts/account1.tar.gz to /2017-05- 07/account1.tar.gz Is it possible this folder does not exist, or has some typo in the path ? If you look at the first line of the log extract you posted, the file path in the bucket is /2017-05-07/ but in the second line, the file path appears as /2017-05- 07/ - see the extra space between the 05- and the 07 ? Now this could, of course, have happened at copy/paste time when you created this post - but it may be worth a second look :) Sorry I can't be more specific, and my apologies in advance if you have already revised all your configuration. I can only suggest to check and re-check you have got all your settings right at the WHM backup end, AND that they coincide with what you have at the S3 end. Hope this is of some help. **edit** I just stumbled across a similar thread at WHM backup results in "Amazon responded with 400" that it may be worth keeping an eye on :) 0 -
I successfully configured WHM to backup to S3 as an "Additional Destination." It worked for two daily backups, then failed for two files in the third backup. The error in WHM's log just says, "Amazon responded with 400." Amazon's page lists 51 possible causes for the 400 error code. Is there a way to find out more specifically what went wrong? I am posting this same question over on Amazon as well. Excerpt from the transporter log: [2017-05-07 14:40:03 -0400] warn [cpbackup_transporter] Upload attempt failed: Amazon::S3: 400 at /usr/local/cpanel/3rdparty/perl/524/lib64/perl5/cpanel_lib/Amazon/S3/Bucket.pm line 134. at /usr/local/cpanel/Cpanel/LoggerAdapter.pm line 27. Cpanel::LoggerAdapter::warn(Cpanel::LoggerAdapter=HASH(0x2512bf0), "Upload attempt failed: Amazon::S3: Amazon responded with 400 "...) called at /usr/local/cpanel/Cpanel/Backup/Queue.pm line 494
What I am running on:/etc/redhat-release:CentOS Linux release 7.3.1611 (Core) /usr/local/cpanel/version:11.62.0.22 /var/cpanel/envtype:virtuozzo CPANEL=stable Server version: Apache/2.4.25 (cPanel) Server built: Apr 7 2017 15:35:17 ea-php-cli Copyright 2016 cPanel, Inc. PHP 5.6.30 (cli) (built: Apr 29 2017 17:39:17) Copyright (c) 1997-2016 The PHP Group Zend Engine v2.6.0, Copyright (c) 1998-2016 Zend Technologies mysql Ver 14.14 Distrib 5.6.35, for Linux (x86_64) using EditLine wrapper0 -
The account file that failed was 2.14 GB. I don't have the tar file for the system, which also failed. Today's backup seems to have worked. The account file that failed before is again 2.14 GB on the VPS. system.tar is 340.2 MB on S3. 0 -
Thank you, but I think the error you noted is just a copy-paste error from my logs. The appropriate directory was created on S3 and all other accounts transferred without a problem. I'd note that I have had this system set up for about 2 years with no problems - it only seems to have started since a recent CPanel upgrade that made some changes to the backup system. I'm currently running v. 64 (build 19). The error started on April 28, so I'm thinking it could be related to this change reported with 64.0.17: Fixed case CPANEL-12742: Update Backup System Feature Showcase with more information.
You are right that the other thread you noted is reporting the same issue.0 -
Hello Everyone, I've merged together a couple of threads related to this topic. Internal case CPANEL-13033 is open to track reports of backups to remote Amazon S3 servers intermittently failing with error messages such as this: info [cpbackup_transporter] Error is not a reference Amazon::S3: Amazon responded with 400 Bad Request
We don't yet have a workaround, however I'll update this thread with more information on the status of this case as it becomes available. Thank you.0 -
Thank you so much. In my case the one account is failing every day - it is also a large account (3.3 gigs) -- whereas several other smaller accounts and system backups are transferring without difficulty. The other accounts are between 60-400 MB. The system_files.tar is also transferring most times (had an error one day only), and is just under 1 GB. Of course I am happy to provide whatever information is needed to help debug in the future. 0 -
I don't know why, but the transfer to Amazon S3 has now been working every day for 3 days. The file that was failing before is 3.1 gigs in the Amazon bucket. My CPanel has now upgraded to v. 64 (build 20) but I can see that the transfers were still failing 1-2 days after upgrade, so I don't think the upgrade is the reason. Maybe the problem was on Amazon's end? (anyway, I'll monitor and post again if I see problems) 0 -
I am experiencing the exact same issue where Amazon S3 backup transfer intermittently fail. Looks like some of large accounts are failing, but not all of large accounts failed. I am currently on cPanel & WHM 64.0 (build 22). Due to this issue, I consistently run out of disk usage for my entire VPS which results in malfunction of all of my clients' website..:( Please update us on the issue when it is resolved. Thanks, 0 -
I still am experiencing the intermittent failures with larger accounts but I have felt it somewhat helpful to edit the etc/cpbackup-exclude.conf to reduce size of backups-- in my case simply by eliminating backup of the user tmp directories, where old log files are stored. I also have eliminating backups of stored backup files -- for example, within Wordpress installations or backups made and stored by the Softaculous add on. Doing that reduced the storage size of the largest account from 3 gigs down to 1 gig. Of course it would be better if I didn't have to do this at all -- but pending resolution of the primary issue, it does at least improve both the problem of AWS transport and impact of local storage of backups. 0 -
I don't know why, but the transfer to Amazon S3 has now been working every day for 3 days. The file that was failing before is 3.1 gigs in the Amazon bucket. My CPanel has now upgraded to v. 64 (build 20) but I can see that the transfers were still failing 1-2 days after upgrade, so I don't think the upgrade is the reason. Maybe the problem was on Amazon's end? (anyway, I'll monitor and post again if I see problems)
The case is still open to track reports of this issue, however thus far it seems related to performance and network issues on the Amazon S3 network. I recommend reaching out to Amazon if you notice any additional failures. Thank you.0 -
Thank you. I'm not surprised -because it has been inconsistent and only on very large files, I suspected the problem might be connectivity to the Amazon bucket. I found some information on the Amazon site that might be helpful: Amazon S3 Multipart Uploads " AWS SDK for PHP documentation This describes multi-part uploads - and although it says that files up to 5 gigs can be uploaded, it also says, "Amazon S3 customers are encouraged to use multipart uploads for objects greater than 100 MB." I don't know whether the current CPanel backup script uses this function or not - but if not, then that could be related to the problem. My problems have only occurred with very large files. I was able to resolve the problem with the large account by excluding nonessential items (like old log files) from the backup, but I have had difficulty with the system backup also failing to transport frequently. One feature request would be to provide more flexibility in the system backup as to how it is transported. I have reduced the size of the system backup for now by disabling the full mysql backup (preserving only account-based backup) -- but that wouldn't be an issue of the full database backup could be stored and transported separately. It would also be nice if the system backup and individual account backups could be placed on different schedules. Anyway, this is just a wish list for now. 0 -
+1 on this and consistently on large files, but at random with some smaller ones. 0 -
+1 on this and consistently on large files, but at random with some smaller ones.
On two VPS's with several dozen accounts each, this began last week and has continued on ONE account (1.4GB). Reviewing the underlying logs, other accounts fail 1-2 times - on the third failure WHM gives up. Looking back, there were NO errors prior to last week. I would note that transfers from NYC to S3 (Northern VA) all work fine, while transfers from Salt Lake City have problems. Sounds like an S3 issue - but it also sounds like the multipart transfer would help. What are the plans - if any - to address (on cPanel end)? Thanks, - Lou0 -
One more datapoint: Salt Lake City --> VA Fails Salt Lake City --> N. California Success NYC --> VA Success 0 -
Mine quit erroring out on 6/9, with no changes done on this end, so I figured a WHM update took care of it... 0 -
Any update? I moved my S3 backups to a different S3 center. Problem stopped for a week and now restarted! Also smaller accounts are failing too. 0 -
I should add we are on 64.0.33 BTW, on the Amazon S3 forums this has been raised with no help or solutions offered... 0 -
This describes multi-part uploads - and although it says that files up to 5 gigs can be uploaded, it also says, "Amazon S3 customers are encouraged to use multipart uploads for objects greater than 100 MB." I don't know whether the current CPanel backup script uses this function or not - but if not, then that could be related to the problem.
Sounds like an S3 issue - but it also sounds like the multipart transfer would help. What are the plans - if any - to address (on cPanel end)?
Multi-part uploads are utilized if the file size is greater than 20MB. I don't have any new information to report on the status of this case at this time, but I'll continue to monitor it and update this thread with more information as it becomes available. Thank you.0 -
Hello, My client faced the issue, the transport process stopped since 14th May 2017. Is there any update on this? 0 -
Hello, A change that will retry upload attempts when the "400" error is returned by Amazon is under testing as part of CPANEL-13033. There's currently no specific time frame to offer on when this change will be implemented, but I'll continue to monitor the case and update this thread with more information as it becomes available. Thank you. 0 -
Hello, To update, the inclusion of CPANEL-13033 is planned for cPanel version 68 to go along with the following case: Fixed case CPANEL-14617: Support for AWS V4 signatures in Amazon transport. Thank you. 0 -
Hello, To update, the inclusion of CPANEL-13033 is planned for cPanel version 68 to go along with the following case: Fixed case CPANEL-14617: Support for AWS V4 signatures in Amazon transport. Thank you.
Has this fix been released yet? I've been having issues sending backups to S3 for the last 2-3 weeks. Running v. 66.0.23 (latest stable). It has worked for years but all of a sudden it errors out with Transport Errors every time.0
Please sign in to leave a comment.
Comments
25 comments