CPANEL-36050 - Backblaze B2: Backup transport errors
We have occasional problems when uploading to Backblaze B2, with backups failing to upload.
There are errors in cpbackup_transporter.log like the following example (actual account name has been replaced with "accountx"):
The backup software then immediately tries twice more:
After failing for a third time, it gives up and immediately moves onto the next account:
This carries on with a number of accounts failing to upload until there is a successful upload attempt 3 seconds later:
I'm assuming the "error: CODE :500" is the error code received from Backblaze. If so, then according to the Backblaze blog post below [1], an error code of 500 is to be expected. If the application receives an error 500 code, it should immediately try again after getting a new url. However, if it gets another 500 code, it should wait a few seconds before trying again. It would appear that the WHM backup software is not doing this. This has resulted in the failure to upload a number of accounts. [1]
[2021-02-23 05:03:19 +0000] info [cpbackup_transporter] Uploading account backup /backup_cpanel/2021-02-23/accounts/account1.tar.gz to backup_cpanel/2021-02-23/accounts/account1.tar.gz (from 2021-02-23/accounts/account1.tar.gz)
[2021-02-23 05:03:19 +0000] info [cpbackup_transporter] Attempting to upload /backup_cpanel/2021-02-23/accounts/account1.tar.gz to backup_cpanel/2021-02-23/accounts/account1.tar.gz for destination: Backblaze B2
[2021-02-23 05:03:19 +0000] info [cpbackup_transporter] Upload attempt #1 starting for /backup_cpanel/2021-02-23/accounts/account1.tar.gz to backup_cpanel/2021-02-23/accounts/account1.tar.gz for destination: Backblaze B2
[2021-02-23 05:03:19 +0000] warn [cpbackup_transporter] Upload attempt failed: Cpanel::Exception::HTTP::Network/(XID 4rn8t3) The system failed to send an HTTP "GET" request to "https://api.backblazeb2.com/b2api/v1/b2_authorize_account" because of an error: CODE :500: MSG :500 Can't connect to api.backblazeb2.com:443 (Name or service not known): Can't connect to api.backblazeb2.com:443 (Name or service not known)
Name or service not known at /usr/local/cpanel/3rdparty/perl/530/lib/perl5/cpanel_lib/LWP/Protocol/http.pm line 50.
at /usr/local/cpanel/Cpanel/Transport/Files/Backblaze.pm line 122.
Cpanel::Transport::Files::Backblaze::_error(Cpanel::Transport::Files::Backblaze=HASH(0x37befe8), "HTTP::Network", HASH(0x31660e8)) called at /usr/local/cpanel/Cpanel/Transport/Files/Backblaze.pm line 272
[Lots more perl stuff snipped]The backup software then immediately tries twice more:
[2021-02-23 05:03:19 +0000] info [cpbackup_transporter] Upload attempt #2 starting for /backup_cpanel/2021-02-23/accounts/account1.tar.gz to backup_cpanel/2021-02-23/accounts/account1.tar.gz for destination: Backblaze B2
[2021-02-23 05:03:19 +0000] warn [cpbackup_transporter] Upload attempt failed: Cpanel::Exception::HTTP::Network/(XID 9ed6ug) The system failed to send an HTTP "GET" request to "https://api.backblazeb2.com/b2api/v1/b2_authorize_account" because of an error: CODE :500: MSG :500 Can't connect to api.backblazeb2.com:443 (Name or service not known): Can't connect to api.backblazeb2.com:443 (Name or service not known)
...
[2021-02-23 05:03:19 +0000] info [cpbackup_transporter] Upload attempt #3 starting for /backup_cpanel/2021-02-23/accounts/account1.tar.gz to backup_cpanel/2021-02-23/accounts/account1.tar.gz for destination: Backblaze B2
[2021-02-23 05:03:19 +0000] warn [cpbackup_transporter] Upload attempt failed: Cpanel::Exception::HTTP::Network/(XID 24zw5x) The system failed to send an HTTP "GET" request to "https://api.backblazeb2.com/b2api/v1/b2_authorize_account" because of an error: CODE :500: MSG :500 Can't connect to api.backblazeb2.com:443 (Name or service not known): Can't connect to api.backblazeb2.com:443 (Name or service not known)After failing for a third time, it gives up and immediately moves onto the next account:
[2021-02-23 05:03:19 +0000] info [cpbackup_transporter] Upload attempt #1 starting for /backup_cpanel/2021-02-23/accounts/account2.tar.gz to backup_cpanel/2021-02-23/accounts/account2.tar.gz for destination: Backblaze B2
[2021-02-23 05:03:19 +0000] warn [cpbackup_transporter] Upload attempt failed: Cpanel::Exception::HTTP::Network/(XID amkjxg) The system failed to send an HTTP "GET" request to "https://api.backblazeb2.com/b2api/v1/b2_authorize_account" because of an error: CODE :500: MSG :500 Can't connect to api.backblazeb2.com:443 (Name or service not known): Can't connect to api.backblazeb2.com:443 (Name or service not known)This carries on with a number of accounts failing to upload until there is a successful upload attempt 3 seconds later:
[2021-02-23 05:03:22 +0000] info [cpbackup_transporter] Upload attempt #1 starting for /backup_cpanel/2021-02-23/accounts/account3.tar.gz to backup_cpanel/2021-02-23/accounts/account3.tar.gz for destination: Backblaze B2
[2021-02-23 05:03:25 +0000] info [cpbackup_transporter] Successful transfer of /backup_cpanel/2021-02-23/accounts/account3.tar.gz to backup_cpanel/2021-02-23/accounts/account3.tar.gz for destination Backblaze B2I'm assuming the "error: CODE :500" is the error code received from Backblaze. If so, then according to the Backblaze blog post below [1], an error code of 500 is to be expected. If the application receives an error 500 code, it should immediately try again after getting a new url. However, if it gets another 500 code, it should wait a few seconds before trying again. It would appear that the WHM backup software is not doing this. This has resulted in the failure to upload a number of accounts. [1]
-
Hey there! Thanks for bringing this to my attention. I've created case CPANEL-36050 to have our developer explore the possibility of introducing a delay in the Backblaze upload settings when an error is encountered. I'm following along with that case and I'll make sure to post an update here as soon as I hear something. 0 -
Thanks Rex. 0 -
Sure thing! 0 -
Update - we have a fix for this that will be included in version 96 when this gets released :D 0 -
Great, thanks. 0 -
Sorry, but still seeing this error in version v.96.0.9. Should be pausing for a few seconds after failure #2, but that's not happening. First few backups upload with no problem, and then B2 get's busy and the script does not pause. Here's an excerpt from my logs/transport_errors.txt [2021-06-06 04:50:53 -0700] info [cpbackup_transporter] cpbackup_transporter - Processing next task [2021-06-06 04:50:53 -0700] info [cpbackup_transporter] Instantiating Object [2021-06-06 04:50:53 -0700] info [cpbackup_transporter] Starting a "copy" operation on the "Backblaze" destination ID "[[hidden]]". [2021-06-06 04:50:53 -0700] info [cpbackup_transporter] Validating destination path weekly/2021-06-06/accounts/ [2021-06-06 04:50:53 -0700] info [cpbackup_transporter] Making Path weekly/2021-06-06/accounts/ [2021-06-06 04:50:53 -0700] info [cpbackup_transporter] Path creation successful [2021-06-06 04:50:53 -0700] info [cpbackup_transporter] Uploading account backup /backup/2021-06-06/accounts/[[hidden]].tar.gz to weekly/2021-06-06/accounts/[[hidden]].tar.gz (from weekly/2021-06-06/accounts/[[hidden]].tar.gz) [2021-06-06 04:50:53 -0700] info [cpbackup_transporter] Attempting to upload /backup/2021-06-06/accounts/[[hidden]].tar.gz to weekly/2021-06-06/accounts/[[hidden]].tar.gz for destination: Backblaze [2021-06-06 04:50:53 -0700] info [cpbackup_transporter] Upload attempt #1 starting for /backup/2021-06-06/accounts/[[hidden]].tar.gz to weekly/2021-06-06/accounts/[[hidden]].tar.gz for destination: Backblaze [2021-06-06 04:50:53 -0700] warn [cpbackup_transporter] Upload attempt failed: Cpanel::Exception::HTTP::Network/(XID npjcr9) The system failed to send an HTTP "GET" request to " because of an error: CODE :500: MSG :500 Can't connect to api.backblazeb2.com:443 (Name or service not known): Can't connect to api.backblazeb2.com:443 (Name or service not known) Name or service not known at /usr/local/cpanel/3rdparty/perl/532/lib/perl5/cpanel_lib/LWP/Protocol/http.pm line 50. at /usr/local/cpanel/Cpanel/Transport/Files/Backblaze.pm line 122. [[DUMP]] [2021-06-06 04:50:53 -0700] info [cpbackup_transporter] Upload attempt #2 starting for /backup/2021-06-06/accounts/[[hidden]].tar.gz to weekly/2021-06-06/accounts/[[hidden]].tar.gz for destination: Backblaze [2021-06-06 04:50:53 -0700] warn [cpbackup_transporter] Upload attempt failed: Cpanel::Exception::HTTP::Network/(XID peybg9) The system failed to send an HTTP "GET" request to " because of an error: CODE :500: MSG :500 Can't connect to api.backblazeb2.com:443 (Name or service not known): Can't connect to api.backblazeb2.com:443 (Name or service not known) Name or service not known at /usr/local/cpanel/3rdparty/perl/532/lib/perl5/cpanel_lib/LWP/Protocol/http.pm line 50. at /usr/local/cpanel/Cpanel/Transport/Files/Backblaze.pm line 122. [[DUMP]] [2021-06-06 04:50:53 -0700] info [cpbackup_transporter] Upload attempt #3 starting for /backup/2021-06-06/accounts/[[hidden]].tar.gz to weekly/2021-06-06/accounts/[[hidden]].tar.gz for destination: Backblaze [2021-06-06 04:50:53 -0700] warn [cpbackup_transporter] Upload attempt failed: Cpanel::Exception::HTTP::Network/(XID cquwju) The system failed to send an HTTP "GET" request to " because of an error: CODE :500: MSG :500 Can't connect to api.backblazeb2.com:443 (Name or service not known): Can't connect to api.backblazeb2.com:443 (Name or service not known) Name or service not known at /usr/local/cpanel/3rdparty/perl/532/lib/perl5/cpanel_lib/LWP/Protocol/http.pm line 50. at /usr/local/cpanel/Cpanel/Transport/Files/Backblaze.pm line 122. [[DUMP]] [2021-06-06 04:50:53 -0700] info [cpbackup_transporter] cpbackup_transporter - Processing next task [2021-06-06 04:50:53 -0700] info [cpbackup_transporter] Instantiating Object [2021-06-06 04:50:53 -0700] info [cpbackup_transporter] Starting a "copy" operation on the "Backblaze" destination ID "[[hidden]]". [2021-06-06 04:50:53 -0700] info [cpbackup_transporter] Validating destination path 2021-06-06/accounts/ [2021-06-06 04:50:53 -0700] info [cpbackup_transporter] Making Path 2021-06-06/accounts/ [2021-06-06 04:50:53 -0700] info [cpbackup_transporter] Path creation successful [2021-06-06 04:50:53 -0700] info [cpbackup_transporter] Uploading account backup /backup/2021-06-06/accounts/[[hidden]].tar.gz to 2021-06-06/accounts/[[hidden]].tar.gz (from 2021-06-06/accounts/[[hidden]].tar.gz) [2021-06-06 04:50:53 -0700] info [cpbackup_transporter] Attempting to upload /backup/2021-06-06/accounts/[[hidden]].tar.gz to 2021-06-06/accounts/[[hidden]].tar.gz for destination: Backblaze [2021-06-06 04:50:53 -0700] info [cpbackup_transporter] Upload attempt #1 starting for /backup/2021-06-06/accounts/[[hidden]].tar.gz to 2021-06-06/accounts/[[hidden]].tar.gz for destination: Backblaze [2021-06-06 04:50:53 -0700] warn [cpbackup_transporter] Upload attempt failed: Cpanel::Exception::HTTP::Network/(XID xnpbj9) The system failed to send an HTTP "GET" request to " because of an error: CODE :500: MSG :500 Can't connect to api.backblazeb2.com:443 (Name or service not known): Can't connect to api.backblazeb2.com:443 (Name or service not known) Name or service not known at /usr/local/cpanel/3rdparty/perl/532/lib/perl5/cpanel_lib/LWP/Protocol/http.pm line 50. at /usr/local/cpanel/Cpanel/Transport/Files/Backblaze.pm line 122. [[DUMP]] [2021-06-06 04:50:53 -0700] info [cpbackup_transporter] Upload attempt #2 starting for /backup/2021-06-06/accounts/[[hidden]].tar.gz to 2021-06-06/accounts/[[hidden]].tar.gz for destination: Backblaze [2021-06-06 04:50:53 -0700] warn [cpbackup_transporter] Upload attempt failed: Cpanel::Exception::HTTP::Network/(XID 7pygtp) The system failed to send an HTTP "GET" request to " because of an error: CODE :500: MSG :500 Can't connect to api.backblazeb2.com:443 (Name or service not known): Can't connect to api.backblazeb2.com:443 (Name or service not known) Name or service not known at /usr/local/cpanel/3rdparty/perl/532/lib/perl5/cpanel_lib/LWP/Protocol/http.pm line 50. at /usr/local/cpanel/Cpanel/Transport/Files/Backblaze.pm line 122. [[DUMP]] [2021-06-06 04:50:53 -0700] info [cpbackup_transporter] Upload attempt #3 starting for /backup/2021-06-06/accounts/[[hidden]].tar.gz to 2021-06-06/accounts/[[hidden]].tar.gz for destination: Backblaze [2021-06-06 04:50:53 -0700] warn [cpbackup_transporter] Upload attempt failed: Cpanel::Exception::HTTP::Network/(XID w73j6d) The system failed to send an HTTP "GET" request to " because of an error: CODE :500: MSG :500 Can't connect to api.backblazeb2.com:443 (Name or service not known): Can't connect to api.backblazeb2.com:443 (Name or service not known) Name or service not known at /usr/local/cpanel/3rdparty/perl/532/lib/perl5/cpanel_lib/LWP/Protocol/http.pm line 50. at /usr/local/cpanel/Cpanel/Transport/Files/Backblaze.pm line 122. [[DUMP]] [2021-06-06 04:50:53 -0700] info [cpbackup_transporter] cpbackup_transporter - Processing next task [2021-06-06 04:50:53 -0700] info [cpbackup_transporter] Instantiating Object [2021-06-06 04:50:53 -0700] info [cpbackup_transporter] Starting a "copy" operation on the "Backblaze" destination ID "[[hidden]]". [2021-06-06 04:50:53 -0700] info [cpbackup_transporter] Validating destination path weekly/2021-06-06/accounts/ [2021-06-06 04:50:53 -0700] info [cpbackup_transporter] Making Path weekly/2021-06-06/accounts/ [2021-06-06 04:50:53 -0700] info [cpbackup_transporter] Path creation successful [2021-06-06 04:50:53 -0700] info [cpbackup_transporter] Uploading account backup /backup/2021-06-06/accounts/[[hidden]].tar.gz to weekly/2021-06-06/accounts/[[hidden]].tar.gz (from weekly/2021-06-06/accounts/[[hidden]].tar.gz) [2021-06-06 04:50:53 -0700] info [cpbackup_transporter] Attempting to upload /backup/2021-06-06/accounts/[[hidden]].tar.gz to weekly/2021-06-06/accounts/[[hidden]].tar.gz for destination: Backblaze [2021-06-06 04:50:53 -0700] info [cpbackup_transporter] Upload attempt #1 starting for /backup/2021-06-06/accounts/[[hidden]].tar.gz to weekly/2021-06-06/accounts/[[hidden]].tar.gz for destination: Backblaze [2021-06-06 04:50:53 -0700] warn [cpbackup_transporter] Upload attempt failed: Cpanel::Exception::HTTP::Network/(XID r9aur6) The system failed to send an HTTP "GET" request to " because of an error: CODE :500: MSG :500 Can't connect to api.backblazeb2.com:443 (Name or service not known): Can't connect to api.backblazeb2.com:443 (Name or service not known) Name or service not known at /usr/local/cpanel/3rdparty/perl/532/lib/perl5/cpanel_lib/LWP/Protocol/http.pm line 50. at /usr/local/cpanel/Cpanel/Transport/Files/Backblaze.pm line 122. [[DUMP]] [2021-06-06 04:50:53 -0700] info [cpbackup_transporter] Upload attempt #2 starting for /backup/2021-06-06/accounts/[[hidden]].tar.gz to weekly/2021-06-06/accounts/[[hidden]].tar.gz for destination: Backblaze [2021-06-06 04:50:53 -0700] warn [cpbackup_transporter] Upload attempt failed: Cpanel::Exception::HTTP::Network/(XID pmqg37) The system failed to send an HTTP "GET" request to " because of an error: CODE :500: MSG :500 Can't connect to api.backblazeb2.com:443 (Name or service not known): Can't connect to api.backblazeb2.com:443 (Name or service not known) Name or service not known at /usr/local/cpanel/3rdparty/perl/532/lib/perl5/cpanel_lib/LWP/Protocol/http.pm line 50. at /usr/local/cpanel/Cpanel/Transport/Files/Backblaze.pm line 122. [[DUMP]] [2021-06-06 04:50:53 -0700] info [cpbackup_transporter] Upload attempt #3 starting for /backup/2021-06-06/accounts/[[hidden]].tar.gz to weekly/2021-06-06/accounts/[[hidden]].tar.gz for destination: Backblaze [2021-06-06 04:50:53 -0700] warn [cpbackup_transporter] Upload attempt failed: Cpanel::Exception::HTTP::Network/(XID nxv2q7) The system failed to send an HTTP "GET" request to " because of an error: CODE :500: MSG :500 Can't connect to api.backblazeb2.com:443 (Name or service not known): Can't connect to api.backblazeb2.com:443 (Name or service not known) Name or service not known at /usr/local/cpanel/3rdparty/perl/532/lib/perl5/cpanel_lib/LWP/Protocol/http.pm line 50. at /usr/local/cpanel/Cpanel/Transport/Files/Backblaze.pm line 122. [[DUMP]] [2021-06-06 04:50:53 -0700] info [cpbackup_transporter] cpbackup_transporter - Processing next task [2021-06-06 04:50:53 -0700] info [cpbackup_transporter] Instantiating Object [2021-06-06 04:50:53 -0700] info [cpbackup_transporter] Starting a "copy_backup_metadata" operation on the "Backblaze" destination ID "[[hidden]]". [2021-06-06 04:50:53 -0700] info [cpbackup_transporter] Uploading backup metadata file /backup/2021-06-06/accounts/.master.meta to Backblaze [2021-06-06 04:50:53 -0700] info [cpbackup_transporter] Validating destination path 2021-06-06/accounts/ [2021-06-06 04:50:53 -0700] info [cpbackup_transporter] Making Path 2021-06-06/accounts/ [2021-06-06 04:50:53 -0700] info [cpbackup_transporter] Path creation successful [2021-06-06 04:50:53 -0700] info [cpbackup_transporter] Uploading backup metadata /backup/2021-06-06/accounts/.master.meta to 2021-06-06/accounts/.master.meta (from 2021-06-06/accounts/.master.meta) [2021-06-06 04:50:53 -0700] info [cpbackup_transporter] Attempting to upload /backup/2021-06-06/accounts/.master.meta to 2021-06-06/accounts/.master.meta for destination: Backblaze [2021-06-06 04:50:53 -0700] info [cpbackup_transporter] Upload attempt #1 starting for /backup/2021-06-06/accounts/.master.meta to 2021-06-06/accounts/.master.meta for destination: Backblaze [2021-06-06 04:50:53 -0700] warn [cpbackup_transporter] Upload attempt failed: Cpanel::Exception::HTTP::Network/(XID zxfwpf) The system failed to send an HTTP "GET" request to " because of an error: CODE :500: MSG :500 Can't connect to api.backblazeb2.com:443 (Name or service not known): Can't connect to api.backblazeb2.com:443 (Name or service not known) Name or service not known at /usr/local/cpanel/3rdparty/perl/532/lib/perl5/cpanel_lib/LWP/Protocol/http.pm line 50. at /usr/local/cpanel/Cpanel/Transport/Files/Backblaze.pm line 122. [[DUMP]] [2021-06-06 04:50:53 -0700] info [cpbackup_transporter] Upload attempt #2 starting for /backup/2021-06-06/accounts/.master.meta to 2021-06-06/accounts/.master.meta for destination: Backblaze [2021-06-06 04:50:53 -0700] warn [cpbackup_transporter] Upload attempt failed: Cpanel::Exception::HTTP::Network/(XID qudqvq) The system failed to send an HTTP "GET" request to " because of an error: CODE :500: MSG :500 Can't connect to api.backblazeb2.com:443 (Name or service not known): Can't connect to api.backblazeb2.com:443 (Name or service not known) Name or service not known at /usr/local/cpanel/3rdparty/perl/532/lib/perl5/cpanel_lib/LWP/Protocol/http.pm line 50. at /usr/local/cpanel/Cpanel/Transport/Files/Backblaze.pm line 122. [[DUMP]] [2021-06-06 04:50:53 -0700] info [cpbackup_transporter] Upload attempt #3 starting for /backup/2021-06-06/accounts/.master.meta to 2021-06-06/accounts/.master.meta for destination: Backblaze [2021-06-06 04:50:53 -0700] warn [cpbackup_transporter] Upload attempt failed: Cpanel::Exception::HTTP::Network/(XID g2882n) The system failed to send an HTTP "GET" request to " because of an error: CODE :500: MSG :500 Can't connect to api.backblazeb2.com:443 (Name or service not known): Can't connect to api.backblazeb2.com:443 (Name or service not known) Name or service not known at /usr/local/cpanel/3rdparty/perl/532/lib/perl5/cpanel_lib/LWP/Protocol/http.pm line 50. at /usr/local/cpanel/Cpanel/Transport/Files/Backblaze.pm line 122. [[DUMP]] [2021-06-06 04:50:53 -0700] info [cpbackup_transporter] cpbackup_transporter - Processing next task [2021-06-06 04:50:53 -0700] info [cpbackup_transporter] Instantiating Object [2021-06-06 04:50:53 -0700] info [cpbackup_transporter] Starting a "prune" operation on the "Backblaze" destination ID "[[hidden]]". 0 -
OP here. We also continued to see this. A cPanel Analyst has recently been working on our system and think they have identified a fix. There is a support article you can follow, but I'm not sure if I can link to this. May be cPRex can add more detail. 0 -
Hello phil99. Would you mind providing the ticket ID so that we can review the resolution information that was provided in the ticket? 0 -
Hi, ticket #94327364. 0 -
I am also seeing issues with Backblaze offsite backups since Version 96.0.9 The problems I am seeing are: 1. When transferring files, it is no longer creating a new folder on Backblaze It now always uses the same folder 2021-05-29/ and overwrites the backup from the day before. It should create a new folder for each day - 2021-06-09 etc. 2021-06-09 is being saved to 2021-05-29/ [2021-06-09 02:03:10 +0100] info [cpbackup_transporter] Initializing log file [2021-06-09 02:03:10 +0100] info [cpbackup_transporter] cpbackup_transporter - parent starting [2021-06-09 02:03:10 +0100] info [cpbackup_transporter] cpbackup_transporter - child starting [2021-06-09 02:03:10 +0100] info [cpbackup_transporter] cPanel Backup Transporter Queue Daemon started. [2021-06-09 02:03:10 +0100] info [cpbackup_transporter] cpbackup_transporter - started [2021-06-09 02:03:10 +0100] info [cpbackup_transporter] cpbackup_transporter - Checking queue for tasks [2021-06-09 02:03:10 +0100] info [cpbackup_transporter] cpbackup_transporter - Processing next task [2021-06-09 02:03:10 +0100] info [cpbackup_transporter] Instantiating Object [2021-06-09 02:03:10 +0100] info [cpbackup_transporter] Starting a "copy" operation on the "server3-v2" destination ID "6a8vXi_rRC0gbsB9MJJ6xRe5". [2021-06-09 02:03:10 +0100] info [cpbackup_transporter] Validating destination path 2021-05-29/accounts/ [2021-06-09 02:03:11 +0100] info [cpbackup_transporter] Path exists 2021-06-06 is being saved to 2021-05-29/ [2021-06-06 02:03:11 +0100] info [cpbackup_transporter] Base path for destination is server3 [2021-06-06 02:03:11 +0100] info [cpbackup_transporter] Validating destination path server3/2021-05-28/accounts/ [2021-06-06 02:03:13 +0100] info [cpbackup_transporter] Path exists All new backup are saved to 2021-05-29/ 2. It is also skipping some backups and the error is info [cpbackup_transporter] Deep recursion on subroutine "Cpanel::Transport::Files::Backblaze::_do_request" at /usr/local/cpanel/Cpanel/Transport/Files/Backblaze.pm line 269. 3. I am also seeing this error 2021-06-09 04:19:26 +0100] info [cpbackup_transporter] cpbackup_transporter - Processing next task [2021-06-09 04:19:26 +0100] info [cpbackup_transporter] Instantiating Object [2021-06-09 04:19:28 +0100] info [cpbackup_transporter] Starting a "copy" operation on the "backup06" destination ID "RXeICSEibtmSh81YlG0oFuEP". [2021-06-09 04:19:28 +0100] info [cpbackup_transporter] Base path for destination is server3 [2021-06-09 04:19:28 +0100] info [cpbackup_transporter] Validating destination path server3/2021-05-29/accounts/ [2021-06-09 04:19:29 +0100] info [cpbackup_transporter] Path exists [2021-06-09 04:19:29 +0100] info [cpbackup_transporter] Uploading account backup /backup/2021-05-29/accounts/[hidden].tar.gz to server3/2021-05-29/accounts/[hidden].tar.gz (from 2021-05-29/accounts/[hidden].tar.gz) [2021-06-09 04:19:29 +0100] info [cpbackup_transporter] Attempting to upload /backup/2021-05-29/accounts/[hidden].tar.gz to server3/2021-05-29/accounts/[hidden].tar.gz for destination: backup06 [2021-06-09 04:19:29 +0100] info [cpbackup_transporter] Upload attempt #1 starting for /backup/2021-05-29/accounts/[hidden].tar.gz to server3/2021-05-29/accounts/[hidden].tar.gz for destination: backup06 [2021-06-09 04:20:46 +0100] warn [cpbackup_transporter] Upload attempt failed: rsync failed: command exited with code 255: Unknown rsync error rsync: [sender] write error: Broken pipe (32) rsync error: unexplained error (code 255) at io.c(820) [sender=3.1.2] at /usr/local/cpanel/Cpanel/LoggerAdapter.pm line 27. Cpanel::LoggerAdapter::warn(Cpanel::LoggerAdapter=HASH(0x13af588), "Upload attempt failed: rsync failed: command exited with code"...) called at /usr/local/cpanel/Cpanel/Backup/Queue.pm line 633 Cpanel::Backup::Queue::transport_backup::attempt_to_upload_file(Cpanel::Backup::Queue::transport_backup=HASH(0x11916e8), Cpanel::Transport::Files::Rsync=HASH(0x242eb20), "/backup/2021-05-29/accounts/[hidden].tar.gz", "server3/2021-05-29/accounts/[hidden].tar.gz", Cpanel::LoggerAdapter=HASH(0x13af588)) called at /usr/local/cpanel/Cpanel/Backup/Queue.pm line 333 Cpanel::Backup::Queue::transport_backup::process_task(Cpanel::Backup::Queue::transport_backup=HASH(0x11916e8), Cpanel::TaskQueue::Task=HASH(0x390fac8), Cpanel::LoggerAdapter=HASH(0x13af588)) called at /usr/local/cpanel/Cpanel/TaskQueue.pm line 635 eval {...} called at /usr/local/cpanel/Cpanel/TaskQueue.pm line 638 Cpanel::TaskQueue::__ANON__() called at /usr/local/cpanel/Cpanel/StateFile.pm line 241 eval {...} called at /usr/local/cpanel/Cpanel/StateFile.pm line 241 Cpanel::StateFile::Guard::call_unlocked(Cpanel::StateFile::Guard=HASH(0x3bb6188), CODE(0x2404bc8)) called at /usr/local/cpanel/Cpanel/TaskQueue.pm line 643 Cpanel::TaskQueue::process_next_task(Cpanel::TaskQueue=HASH(0x2190298)) called at /usr/local/cpanel/bin/cpbackup_transporter line 168 eval {...} called at /usr/local/cpanel/bin/cpbackup_transporter line 166 [2021-06-09 04:20:48 +0100] info [cpbackup_transporter] Upload attempt #2 starting for /backup/2021-05-29/accounts/[hidden].tar.gz to server3/2021-05-29/accounts/[hidden].tar.gz for destination: backup06 [2021-06-09 04:21:29 +0100] warn [cpbackup_transporter] Upload attempt failed: rsync failed: command exited with code 255: Unknown rsync error rsync: [sender] write error: Broken pipe (32) rsync error: unexplained error (code 255) at io.c(820) [sender=3.1.2] at /usr/local/cpanel/Cpanel/LoggerAdapter.pm line 27. Cpanel::LoggerAdapter::warn(Cpanel::LoggerAdapter=HASH(0x13af588), "Upload attempt failed: rsync failed: command exited with code"...) called at /usr/local/cpanel/Cpanel/Backup/Queue.pm line 633 Cpanel::Backup::Queue::transport_backup::attempt_to_upload_file(Cpanel::Backup::Queue::transport_backup=HASH(0x11916e8), Cpanel::Transport::Files::Rsync=HASH(0x242eb20), "/backup/2021-05-29/accounts/[hidden].tar.gz", "server3/2021-05-29/accounts/[hidden].tar.gz", Cpanel::LoggerAdapter=HASH(0x13af588)) called at /usr/local/cpanel/Cpanel/Backup/Queue.pm line 333 Cpanel::Backup::Queue::transport_backup::process_task(Cpanel::Backup::Queue::transport_backup=HASH(0x11916e8), Cpanel::TaskQueue::Task=HASH(0x390fac8), Cpanel::LoggerAdapter=HASH(0x13af588)) called at /usr/local/cpanel/Cpanel/TaskQueue.pm line 635 eval {...} called at /usr/local/cpanel/Cpanel/TaskQueue.pm line 638 Cpanel::TaskQueue::__ANON__() called at /usr/local/cpanel/Cpanel/StateFile.pm line 241 eval {...} called at /usr/local/cpanel/Cpanel/StateFile.pm line 241 Cpanel::StateFile::Guard::call_unlocked(Cpanel::StateFile::Guard=HASH(0x3bb6188), CODE(0x2404bc8)) called at /usr/local/cpanel/Cpanel/TaskQueue.pm line 643 Cpanel::TaskQueue::process_next_task(Cpanel::TaskQueue=HASH(0x2190298)) called at /usr/local/cpanel/bin/cpbackup_transporter line 168 eval {...} called at /usr/local/cpanel/bin/cpbackup_transporter line 166 2021-06-06 02:09:28 +0100] info [cpbackup_transporter] cpbackup_transporter - Processing next task [2021-06-06 02:09:28 +0100] info [cpbackup_transporter] Instantiating Object [2021-06-06 02:09:29 +0100] info [cpbackup_transporter] Starting a "copy" operation on the "backup06" destination ID "RXeICSEibtmSh81YlG0oFuEP". [2021-06-06 02:09:29 +0100] info [cpbackup_transporter] Base path for destination is server3 [2021-06-06 02:09:29 +0100] info [cpbackup_transporter] Validating destination path server3/2021-05-28/accounts/ [2021-06-06 02:09:30 +0100] info [cpbackup_transporter] Path exists [2021-06-06 02:09:30 +0100] info [cpbackup_transporter] Uploading account backup /backup/2021-05-28/accounts/[hidden].tar.gz to server3/2021-05-28/accounts/[hidden].tar.gz (from 2021-05-28/accounts/[hidden].tar.gz) [2021-06-06 02:09:30 +0100] info [cpbackup_transporter] Attempting to upload /backup/2021-05-28/accounts/[hidden].tar.gz to server3/2021-05-28/accounts/[hidden].tar.gz for destination: backup06 [2021-06-06 02:09:30 +0100] info [cpbackup_transporter] Upload attempt #1 starting for /backup/2021-05-28/accounts/[hidden].tar.gz to server3/2021-05-28/accounts/[hidden].tar.gz for destination: backup06 [2021-06-06 02:09:30 +0100] warn [cpbackup_transporter] Upload attempt failed: rsync failed: command exited with code 23: partial transfer due to error rsync: change_dir "/backup/2021-05-28/accounts" failed: No such file or directory (2) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1179) [sender=3.1.2] at /usr/local/cpanel/Cpanel/LoggerAdapter.pm line 27. Cpanel::LoggerAdapter::warn(Cpanel::LoggerAdapter=HASH(0x20e5888), "Upload attempt failed: rsync failed: command exited with code"...) called at /usr/local/cpanel/Cpanel/Backup/Queue.pm line 633 Cpanel::Backup::Queue::transport_backup::attempt_to_upload_file(Cpanel::Backup::Queue::transport_backup=HASH(0x1ec76e8), Cpanel::Transport::Files::Rsync=HASH(0x3ea51e8), "/backup/2021-05-28/accounts/[hidden].tar.gz", "server3/2021-05-28/accounts/[hidden].tar.gz", Cpanel::LoggerAdapter=HASH(0x20e5888)) called at /usr/local/cpanel/Cpanel/Backup/Queue.pm line 333 Cpanel::Backup::Queue::transport_backup::process_task(Cpanel::Backup::Queue::transport_backup=HASH(0x1ec76e8), Cpanel::TaskQueue::Task=HASH(0x3e7bc90), Cpanel::LoggerAdapter=HASH(0x20e5888)) called at /usr/local/cpanel/Cpanel/TaskQueue.pm line 635 eval {...} called at /usr/local/cpanel/Cpanel/TaskQueue.pm line 638 Cpanel::TaskQueue::__ANON__() called at /usr/local/cpanel/Cpanel/StateFile.pm line 241 eval {...} called at /usr/local/cpanel/Cpanel/StateFile.pm line 241 Cpanel::StateFile::Guard::call_unlocked(Cpanel::StateFile::Guard=HASH(0x2f97350), CODE(0x325a1a8)) called at /usr/local/cpanel/Cpanel/TaskQueue.pm line 643 Cpanel::TaskQueue::process_next_task(Cpanel::TaskQueue=HASH(0x2ec65b8)) called at /usr/local/cpanel/bin/cpbackup_transporter line 168 eval {...} called at /usr/local/cpanel/bin/cpbackup_transporter line 166 [2021-06-06 02:09:32 +0100] info [cpbackup_transporter] Upload attempt #2 starting for /backup/2021-05-28/accounts/[hidden].tar.gz to server3/2021-05-28/accounts/[hidden].tar.gz for destination: backup06 4. and this error: SSL connect attempt failed because of handshake problems [2021-05-26 04:02:08 +0100] info [cpbackup_transporter] Attempting to upload /backup/2021-05-26/accounts/[hidden].tar.gz to 2021-05-26/accounts/[hidden].tar.gz for destination: Backblaze Backups [2021-05-26 04:02:08 +0100] info [cpbackup_transporter] Upload attempt #1 starting for /backup/2021-05-26/accounts/[hidden].tar.gz to 2021-05-26/accounts/[hidden].tar.gz for destination: Backblaze Backups [2021-05-26 04:50:53 +0100] info [cpbackup_transporter] Deep recursion on subroutine "Cpanel::Transport::Files::Backblaze::_do_request" at /usr/local/cpanel/Cpanel/Transport/Files/Backblaze.pm line 297. [2021-05-26 04:50:53 +0100] info [cpbackup_transporter] Deep recursion on subroutine "Cpanel::Transport::Files::Backblaze::_do_request" at /usr/local/cpanel/Cpanel/Transport/Files/Backblaze.pm line 269. [2021-05-30 00:31:57 +0100] warn [cpbackup_transporter] Upload attempt failed: Cpanel::Exception::HTTP::Network/(XID bfj2ry) The system failed to send an HTTP "GET" request to " because of an error: CODE :500: MSG :500 Can't connect to api.backblazeb2.com:443 (SSL connect attempt failed because of handshake problems): Can't connect to api.backblazeb2.com:443 (SSL connect attempt failed because of handshake problems) SSL connect attempt failed because of handshake problems at /usr/local/cpanel/3rdparty/perl/532/lib/perl5/cpanel_lib/LWP/Protocol/http.pm line 50. at /usr/local/cpanel/Cpanel/Transport/Files/Backblaze.pm line 122. Cpanel::Transport::Files::Backblaze::_error(Cpanel::Transport::Files::Backblaze=HASH(0x1c40e90), "HTTP::Network", HASH(0x2e6d8518)) called at /usr/local/cpanel/Cpanel/Transport/Files/Backblaze.pm line 273 Cpanel::Transport::Files::Backblaze::_do_request(Cpanel::Transport::Files::Backblaze=HASH(0x1c40e90), __CPANEL_HIDDEN__, __CPANEL_HIDDEN__, __CPANEL_HIDDEN__, __CPANEL_HIDDEN__, __CPANEL_HIDDEN__, __CPANEL_HIDDEN__, __CPANEL_HIDDEN__, ...) called at /usr/local/cpanel/Cpanel/Transport/Files/Backblaze.pm line 297 Cpanel::Transport::Files::Backblaze::_get_authorize_token(Cpanel::Transport::Files::Backblaze=HASH(0x1c40e90), 1) called at /usr/local/cpanel/Cpanel/Transport/Files/Backblaze.pm line 268 Cpanel::Transport::Files::Backblaze::_do_request(Cpanel::Transport::Files::Backblaze=HASH(0x1c40e90), "content_type", "application/octet-stream", "api_ver", "v1", "custom_url", "[[HIDDEN]]"..., "payload", ...) called at /usr/local/cpanel/Cpanel/Transport/Files/Backblaze.pm line 269 Cpanel::Transport::Files::Backblaze::_do_request(Cpanel::Transport::Files::Backblaze=HASH(0x1c40e90), "extra_headers", HASH(0x37e3770), "payload", CODE(0x3844800), "lives_on_error", 1, "request_type", ...) called at /usr/local/cpanel/Cpanel/Transport/Files/Backblaze.pm line 269 Cpanel::Transport::Files::Backblaze::_do_request(Cpanel::Transport::Files::Backblaze=HASH(0x1c40e90), __CPANEL_HIDDEN__, __CPANEL_HIDDEN__..., __CPANEL_HIDDEN__, __CPANEL_HIDDEN__, __CPANEL_HIDDEN__, __CPANEL_HIDDEN__, __CPANEL_HIDDEN__, ...) called at /usr/local/cpanel/Cpanel/Transport/Files/Backblaze.pm line 269 Cpanel::Transport::Files::Backblaze::_do_request(Cpanel::Transport::Files::Backblaze=HASH(0x1c40e90), "request_type", "POST", "lives_on_error", 1, "payload", CODE(0x3844800), "extra_headers", ...) called at /usr/local/cpanel/Cpanel/Transport/Files/Backblaze.pm line 269 0 -
Update - the fix is scheduled to be included in version 96.0.11. 0 -
Hi All our servers have all been upgraded to 96.0.11 The path creation issues seems to be resolved as the path is being created. The backups are not working due to the following error [2021-06-16 23:23:47 +0100] info [cpbackup_transporter] Upload attempt #1 starting for /backup/2021-06-16/accounts/[hidden].tar.gz to 2021-06-16/accounts/[hidden].tar.gz for destination: Backblaze Backups [2021-06-17 00:06:04 +0100] info [cpbackup_transporter] Deep recursion on subroutine "Cpanel::Transport::Files::Backblaze::_do_request" at /usr/local/cpanel/Cpanel/Transport/Files/Backblaze.pm line 297. [2021-06-17 00:06:04 +0100] info [cpbackup_transporter] Deep recursion on subroutine "Cpanel::Transport::Files::Backblaze::_do_request" at /usr/local/cpanel/Cpanel/Transport/Files/Backblaze.pm line 269. The servers are all crashing as a result and I have had to disable the offsite backups to Backblaze Can you review and provide a solution? Thanks 0 -
@dmart1n - could you submit a ticket to our team so we can check this on one of your affected machines? 0 -
I'm in the same boat as dmart1n with exactly the same errors affecting at least two of my servers, I have a ticket open #94335602 0 -
Our team has opened case CPANEL-37482 to track the issue on new tickets as this seems to be happening again after it was recently fixed. You'll be getting a reply to the ticket saying something similar most likely, but we are aware of the issue. 0 -
@MindServer - we aren't expecting the issue to be resolved on version 94. The case mentioned above is still being investigated at this time. 0 -
Can you let me know what you mean when you say "block the server" ? Are you unable to access the cPanel machine at all? If so, the best place to look would be the error logs at /usr/local/cpanel/logs/error_log and the backup log at /usr/local/cpanel/logs/cpbackup/ for the most recent backup run. This doesn't sound like it is related to the Backblaze issue, so it might be best if this was in a separate thread. 0 -
You've posted many times and I've always thought your English is just fine! This is the type of issue we might just need a ticket to examine. If possible we could run the backup at a specific time and watch the server to see how it behaves and if the problem happens again, as those logs do not show anything that would be a problem, except the backup just stopping. It's possible there could be useful data in /var/log/messages on the system, as that file could report problems with the server itself. 0 -
The memory issues in the second paste are related to the CloudLinux LVE tools killing off a process using too much RAM, so that isn't an overall issue with the server. I don't see anything in there that would indicate an obvious issue. It might be best to have someone watch the machine while the backups run to see if there are any issues that can be seen during that time. 0
Please sign in to leave a comment.
Comments
20 comments