Skip to main content

[CPANEL-20584] Amazon S3 backup transport failure

David Colter
Hello, Below is an excerpt of an email that's been received 3 days in a row now. Preview of transport errors log: Unable to send "/backup/2018-07-26/backup_incomplete" to destination "WHM Backups"
A note to include with this "suddenly occurring" message is that one week ago, I reconfigured WHM Backup Configuration to 'stop' keeping a copy of the backup on my server. It appears that system files and all accounts are being backed up, though. The S3 backups previous to a week ago did have a backup_incomplete item (?? folder or file) along with accounts and system in that dated backup folder.

Comments

12 comments

Date Votes
  • David Colter
    Below is a tail output excerpt. There were 3 unsuccessful attempts to upload /2018-07-26/backup_incomplete, for reason: 'attempted to validate a file that does not exist'. This occurs after successfully uploading the other user account files and one system file. From log file /usr/local/cpanel/logs/cpbackup_transporter.log: ....... [2018-07-26 02:09:31 -0400] info [cpbackup_transporter] Uploading backup metadata /backup/2018-07-26/accounts/.master.meta to /2018-07-26/accounts/.master.meta (from 2018-07-26/accounts/.master.meta) [2018-07-26 02:09:31 -0400] info [cpbackup_transporter] Attempting to upload /backup/2018-07-26/accounts/.master.meta to /2018-07-26/accounts/.master.meta for destination: WHM Backups [2018-07-26 02:09:31 -0400] info [cpbackup_transporter] Upload attempt #1 starting for /backup/2018-07-26/accounts/.master.meta to /2018-07-26/accounts/.master.meta for destination: WHM Backups [2018-07-26 02:09:31 -0400] info [cpbackup_transporter] Successful transfer of /backup/2018-07-26/accounts/.master.meta to /2018-07-26/accounts/.master.meta for destination WHM Backups [2018-07-26 02:09:31 -0400] info [cpbackup_transporter] The backup has been successfully uploaded at least once, now we will delete the local copy since keeplocal (0) is disabled. [2018-07-26 02:09:31 -0400] info [cpbackup_transporter] There are no more files or directories in the backup path and keeplocal (0) is disabled, so now we will delete /backup/2018-07-26 entirely. [2018-07-26 02:09:31 -0400] info [cpbackup_transporter] cpbackup_transporter - Checking queue for tasks [2018-07-26 02:09:31 -0400] info [cpbackup_transporter] cpbackup_transporter - Processing next task [2018-07-26 02:09:31 -0400] info [cpbackup_transporter] Instantiating Object [2018-07-26 02:09:31 -0400] info [cpbackup_transporter] Starting a "copy" operation on the "WHM Backups" destination ID "c1lOWveFr53CVCO9kzutQA0g". [2018-07-26 02:09:31 -0400] info [cpbackup_transporter] Validating destination path /2018-07-26/ [2018-07-26 02:09:32 -0400] info [cpbackup_transporter] Path exists [2018-07-26 02:09:32 -0400] info [cpbackup_transporter] Uploading account backup /backup/2018-07-26/backup_incomplete to /2018-07-26/backup_incomplete (from 2018-07-26/backup_incomplete) [2018-07-26 02:09:32 -0400] info [cpbackup_transporter] Attempting to upload /backup/2018-07-26/backup_incomplete to /2018-07-26/backup_incomplete for destination: WHM Backups [2018-07-26 02:09:32 -0400] info [cpbackup_transporter] Upload attempt #1 starting for /backup/2018-07-26/backup_incomplete to /2018-07-26/backup_incomplete for destination: WHM Backups [2018-07-26 02:09:32 -0400] warn [cpbackup_transporter] Upload attempt failed: "Cpanel::Transport::Files::put" attempted to validate a file that does not exist: /backup/2018-07-26/backup_incomplete at /usr/local/cpanel/Cpanel/LoggerAdapter.pm line 27. Cpanel::LoggerAdapter::warn(Cpanel::LoggerAdapter=HASH(0x23188c0), "Upload attempt failed: \x{e2}\x{80}\x{9c}Cpanel::Transport::Files::put\x{e2}\x{80}\x{9d} at"...) called at /usr/local/cpanel/Cpanel/Backup/Queue.pm line 609 Cpanel::Backup::Queue::transport_backup::attempt_to_upload_file(Cpanel::Backup::Queue::transport_backup=HASH(0x2183190), Cpanel::Transport::Files::AmazonS3=HASH(0x2e4b938), "/backup/2018-07-26/backup_incomplete", "/2018-07-26/backup_incomplete", Cpanel::LoggerAdapter=HASH(0x23188c0)) called at /usr/local/cpanel/Cpanel/Backup/Queue.pm line 301 Cpanel::Backup::Queue::transport_backup::process_task(Cpanel::Backup::Queue::transport_backup=HASH(0x2183190), cPanel::TaskQueue::Task=HASH(0x45062d8), Cpanel::LoggerAdapter=HASH(0x23188c0)) called at /usr/local/cpanel/3rdparty/perl/526/lib64/perl5/cpanel_lib/cPanel/TaskQueue.pm line 624 eval {...} called at /usr/local/cpanel/3rdparty/perl/526/lib64/perl5/cpanel_lib/cPanel/TaskQueue.pm line 627 cPanel::TaskQueue::__ANON__() called at /usr/local/cpanel/3rdparty/perl/526/lib64/perl5/cpanel_lib/cPanel/StateFile.pm line 223 eval {...} called at /usr/local/cpanel/3rdparty/perl/526/lib64/perl5/cpanel_lib/cPanel/StateFile.pm line 223 cPanel::StateFile::Guard::call_unlocked(cPanel::StateFile::Guard=HASH(0x3748778), CODE(0x2db4e28)) called at /usr/local/cpanel/3rdparty/perl/526/lib64/perl5/cpanel_lib/cPanel/TaskQueue.pm line 632 cPanel::TaskQueue::process_next_task(cPanel::TaskQueue=HASH(0x2d70810)) called at /usr/local/cpanel/bin/cpbackup_transporter line 167 eval {...} called at /usr/local/cpanel/bin/cpbackup_transporter line 165 [2018-07-26 02:09:32 -0400] info [cpbackup_transporter] warn [cpbackup_transporter] Upload attempt failed: "Cpanel::Transport::Files::put" attempted to validate a file that does not exist: /backup/2018-07-26/backup_incomplete [2018-07-26 02:09:32 -0400] info [cpbackup_transporter] Upload attempt #2 starting for /backup/2018-07-26/backup_incomplete to /2018-07-26/backup_incomplete for destination: WHM Backups [2018-07-26 02:09:32 -0400] warn [cpbackup_transporter] Upload attempt failed: "Cpanel::Transport::Files::put" attempted to validate a file that does not exist: /backup/2018-07-26/backup_incomplete at /usr/local/cpanel/Cpanel/LoggerAdapter.pm line 27. Cpanel::LoggerAdapter::warn(Cpanel::LoggerAdapter=HASH(0x23188c0), "Upload attempt failed: \x{e2}\x{80}\x{9c}Cpanel::Transport::Files::put\x{e2}\x{80}\x{9d} at"...) called at /usr/local/cpanel/Cpanel/Backup/Queue.pm line 609 Cpanel::Backup::Queue::transport_backup::attempt_to_upload_file(Cpanel::Backup::Queue::transport_backup=HASH(0x2183190), Cpanel::Transport::Files::AmazonS3=HASH(0x2e4b938), "/backup/2018-07-26/backup_incomplete", "/2018-07-26/backup_incomplete", Cpanel::LoggerAdapter=HASH(0x23188c0)) called at /usr/local/cpanel/Cpanel/Backup/Queue.pm line 301 Cpanel::Backup::Queue::transport_backup::process_task(Cpanel::Backup::Queue::transport_backup=HASH(0x2183190), cPanel::TaskQueue::Task=HASH(0x45062d8), Cpanel::LoggerAdapter=HASH(0x23188c0)) called at /usr/local/cpanel/3rdparty/perl/526/lib64/perl5/cpanel_lib/cPanel/TaskQueue.pm line 624 eval {...} called at /usr/local/cpanel/3rdparty/perl/526/lib64/perl5/cpanel_lib/cPanel/TaskQueue.pm line 627 cPanel::TaskQueue::__ANON__() called at /usr/local/cpanel/3rdparty/perl/526/lib64/perl5/cpanel_lib/cPanel/StateFile.pm line 223 eval {...} called at /usr/local/cpanel/3rdparty/perl/526/lib64/perl5/cpanel_lib/cPanel/StateFile.pm line 223 cPanel::StateFile::Guard::call_unlocked(cPanel::StateFile::Guard=HASH(0x3748778), CODE(0x2db4e28)) called at /usr/local/cpanel/3rdparty/perl/526/lib64/perl5/cpanel_lib/cPanel/TaskQueue.pm line 632 cPanel::TaskQueue::process_next_task(cPanel::TaskQueue=HASH(0x2d70810)) called at /usr/local/cpanel/bin/cpbackup_transporter line 167 eval {...} called at /usr/local/cpanel/bin/cpbackup_transporter line 165 [2018-07-26 02:09:32 -0400] info [cpbackup_transporter] warn [cpbackup_transporter] Upload attempt failed: "Cpanel::Transport::Files::put" attempted to validate a file that does not exist: /backup/2018-07-26/backup_incomplete [2018-07-26 02:09:32 -0400] info [cpbackup_transporter] Upload attempt #3 starting for /backup/2018-07-26/backup_incomplete to /2018-07-26/backup_incomplete for destination: WHM Backups [2018-07-26 02:09:32 -0400] warn [cpbackup_transporter] Upload attempt failed: "Cpanel::Transport::Files::put" attempted to validate a file that does not exist: /backup/2018-07-26/backup_incomplete at /usr/local/cpanel/Cpanel/LoggerAdapter.pm line 27. Cpanel::LoggerAdapter::warn(Cpanel::LoggerAdapter=HASH(0x23188c0), "Upload attempt failed: \x{e2}\x{80}\x{9c}Cpanel::Transport::Files::put\x{e2}\x{80}\x{9d} at"...) called at /usr/local/cpanel/Cpanel/Backup/Queue.pm line 609 Cpanel::Backup::Queue::transport_backup::attempt_to_upload_file(Cpanel::Backup::Queue::transport_backup=HASH(0x2183190), Cpanel::Transport::Files::AmazonS3=HASH(0x2e4b938), "/backup/2018-07-26/backup_incomplete", "/2018-07-26/backup_incomplete", Cpanel::LoggerAdapter=HASH(0x23188c0)) called at /usr/local/cpanel/Cpanel/Backup/Queue.pm line 301 Cpanel::Backup::Queue::transport_backup::process_task(Cpanel::Backup::Queue::transport_backup=HASH(0x2183190), cPanel::TaskQueue::Task=HASH(0x45062d8), Cpanel::LoggerAdapter=HASH(0x23188c0)) called at /usr/local/cpanel/3rdparty/perl/526/lib64/perl5/cpanel_lib/cPanel/TaskQueue.pm line 624 eval {...} called at /usr/local/cpanel/3rdparty/perl/526/lib64/perl5/cpanel_lib/cPanel/TaskQueue.pm line 627 cPanel::TaskQueue::__ANON__() called at /usr/local/cpanel/3rdparty/perl/526/lib64/perl5/cpanel_lib/cPanel/StateFile.pm line 223 eval {...} called at /usr/local/cpanel/3rdparty/perl/526/lib64/perl5/cpanel_lib/cPanel/StateFile.pm line 223 cPanel::StateFile::Guard::call_unlocked(cPanel::StateFile::Guard=HASH(0x3748778), CODE(0x2db4e28)) called at /usr/local/cpanel/3rdparty/perl/526/lib64/perl5/cpanel_lib/cPanel/TaskQueue.pm line 632 cPanel::TaskQueue::process_next_task(cPanel::TaskQueue=HASH(0x2d70810)) called at /usr/local/cpanel/bin/cpbackup_transporter line 167 eval {...} called at /usr/local/cpanel/bin/cpbackup_transporter line 165 [2018-07-26 02:09:32 -0400] info [cpbackup_transporter] warn [cpbackup_transporter] Upload attempt failed: "Cpanel::Transport::Files::put" attempted to validate a file that does not exist: /backup/2018-07-26/backup_incomplete [2018-07-26 02:09:32 -0400] info [cpbackup_transporter] cpbackup_transporter - Checking queue for tasks [2018-07-26 02:09:32 -0400] info [cpbackup_transporter] cpbackup_transporter - Processing next task [2018-07-26 02:09:32 -0400] info [cpbackup_transporter] cpbackup_transporter - Checking queue for tasks [2018-07-26 02:09:32 -0400] info [cpbackup_transporter] cpbackup_transporter - Waiting up to 299s for new tasks .......
    0
  • cPanelLauren
    Hi @David Colter Looking at the file name: /backup/2018-07-26/backup_incomplete It seems that there may have been an issue with the backup process, to begin with. Based on: info [cpbackup_transporter] warn [cpbackup_transporter] Upload attempt failed: "Cpanel::Transport::Files::put" attempted to validate a file that does not exist: /backup/2018-07-26/backup_incomplete
    It seems that the file may not actually exist though it's in the backup metadata. Can you confirm that this is the case? You can run the following and reply with the output: ls -lah /backup/2018-07-26/backup_incomplete
    Thanks!
    0
  • David Colter
    Remembering that I stopped backing up locally a few days ago, there is no folder /backup/2018-07-26. In fact, there was no 'dated' backup folder (locally) for each of the days an error notice was sent. # ls -lah /backup/2018-07-26/backup_incomplete /bin/ls: cannot access /backup/2018-07-26/backup_incomplete: No such file or directory
    I re-instated local backups yesterday, and did not receive any warnings during the 2am backup run today. I would like to stop backing up locally and only have the S3 backups, without receiving a daily warning that something is wrong. Can you please explain why the metadata would be containing the file? Do the backups first get saved locally, then get transported and finally, the local is deleted? Or perhaps, suggest a solution? Thank you.
    0
  • cPanelLauren
    Remembering that I stopped backing up locally a few days ago, there is no folder /backup/2018-07-26. In fact, there was no 'dated' backup folder (locally) for each of the days an error notice was sent

    That makes a bit more sense as to why you received that error in that case.
    Can you please explain why the metadata would be containing the file?

    The only circumstance in which I would assume the metadata would contain data on the backup when it was later cancelled is if the backup process was killed or abruptly ended.
    Do the backups first get saved locally, then get transported and finally, the local is deleted?

    This is exactly how this happens - the local backup is not deleted until the transport is successful. Any configuration of the automated backup system will have this same behavior as well.
    I re-instated local backups yesterday, and did not receive any warnings during the 2am backup run today.

    Is the error in regard to the incomplete backup only occurring when you're keeping backups locally or do you mean you stopped the transport?
    0
  • David Colter
    Is the error in regard to the incomplete backup only occurring when you're keeping backups locally or do you mean you stopped the transport?

    I have not ever 'stopped' the transport. The error only occurs when a local copy is NOT being kept.
    0
  • cPanelLauren
    Hi @David Colter Thank you for clarifying. That is not the expected behavior, you should be able to transport to the remote server without keeping a local copy. I'm not sure what would cause that specific behavior in this case. Can you please open a ticket using the link in my signature? Once open please reply with the Ticket ID here so that we can update this thread with the resolution once the ticket is resolved. Thanks!
    0
  • David Colter
    Your Support Request ID is: 9997463
    0
  • David Colter
    That ticket had a response from S. Young, with [QUOTE]I was able to find an internal case for this issue with the case ID CPANEL-20584 which is being worked on by the dev team. Once the update has been tested and pushed out it will be on part of our changelog located at: 72 Change Log - Change Logs - cPanel Documentation
    0
  • cPanelLauren
    Hi @David Colter I have located the case as well and it looks as though it is specifically related to the backup_incomplete touch file you were seeing and Amazon S3. It will be fixed in v74 of cPanel/WHM then backported to 70 and 72. I'll update here as soon as I have confirmation that the issue is resolved Thanks!
    0
  • David Colter
    In the case this may be relevant to the subject issue - every time /usr/local/cpanel/bin/backup runs, an email is sent alerting the following: Use of uninitialized value in pattern match (m//) at /usr/local/cpanel/Cpanel/Filesys.pm line 74. This is consistently reported, since the Backup Manager was set to not save a local copy of the backup and appears to occur very near the beginning of the script. Additionally, the server is also reporting, ~ 8 minutes later, "unable to send ... /backup_incomplete ... " to the S3 destination.
    0
  • Ishware
    I'm also getting the message from the above every day: Use of uninitialized value in pattern match (m//) at /usr/local/cpanel/Cpanel/Filesys.pm line 74. Posting here so I can get any updated replies on that :)
    0
  • cPanelLauren
    Hello everyone, I would like to reiterate that this issue is fixed in a future version, we're just awaiting the release of the fix. If you're still experiencing issues with this once the build of v74 with the resolution included is released please let us know. We'll announce here as soon as the fix is published. Thanks!
    0

Please sign in to leave a comment.

Didn't find what you were looking for?

New post