[CPANEL-27679] Backblaze B2 - large file not reassembled
Posting this as a new thread, as it appears to be different than [CPANEL-25247], where not all parts were uploaded.
We have been testing backups to Backblaze B2. Last night for the first time, we uploaded a file larger than 5GB (the tar.gz size was about 11GB).
According to the transport log, the file was successfully uploaded to Backblaze. However, when looking at the file on Backblaze, it has been split into 3 parts. There are two files of 5GB plus one of about 1GB. There is also a related manifest file:
account.tar.gz._b2_manifest 93bytes
account.tar.gz._b2_part_1 5GB
account.tar.gz._b2_part_2 5GB
account.tar.gz._b2_part_3 1.2GB
The Backblaze B2 API imposes a limit of 5GB on the size of the file that can be uploaded. Larger files have to be split into parts for upload. However, once all parts are uploaded they need to be transformed into a single file with a separate API call. It would appear that all parts were successfully uploaded, but the reassembly didn't take place.
There were no errors in the transport log to indicate this has not been done, it reports a successful transfer.
-
We let this account backup run again last night. This time other errors occurred as well as the parts not being reassembled. The first attempt to upload the account.tar.gz failed with: [2019-05-10 02:48:16 +0100] info [cpbackup_transporter] Upload attempt #1 starting for /backup_cpanel/2019-05-10/accounts/account.tar.gz to backup_cpanel/2019-05-10/accounts/account.tar.gz for destination: Backblaze B2 [2019-05-10 05:09:06 +0100] warn [cpbackup_transporter] Upload attempt failed: Cpanel::Exception::HTTP::Network/(XID 3yu366) The system failed to send an HTTP "POST" request to "https://api002.backblazeb2.com/b2api/v1/b2_finish_large_file" because of an error: CODE :400: MSG :400 Bad Request: { "code": "bad_request", "message": "Part number 951 has not been uploaded", "status": 400 }
There are then various other messages in the transport log from perl modules. A second upload attempt is then started and appears to have succeeded, though the parts were not reassembled. Also, this meant that on Backblaze: - there were two copies of the manifest - 2 copies of part 1 (each 5GB) - 2 copies of part 2 (one 5GB, on an unfinsihed file of 0 bytes) - 1 copy of part 3 (1.2GB) Hope that is helpful!0 -
Hello @phil99, Can you verify if you have reproduced this behavior after updating to cPanel & WHM version 78.0.23? Thank you. 0 -
Hi Michael Yes the upgrade to 78.0.23 occurred just after 23:00 on 8th May, so both of these runs (9th and 10th May) were on that version . Thanks. 0 -
Hello @phil99, While it may seem as though the backup/transport succeeded, it's possible there was an error during the backup process that wasn't fully recorded in the logs. In particular, case CPANEL-26914 is open to address an issue where large file transports fail when the authorization token expires due to the filepart counter incrementing inappropriately. Can you open a 0 -
Hi Michael Support request ID: 12263991 Thanks. 0 -
Hi The conclusion of the support ticket was also that the error message in the second run was similar to that seen in CPANEL-26914, and we would have to wait for the fix to that to be released to see if it fixes the error we saw. There was no mention of the the fact that the file was not reassembled. I wonder if the developers have been consulted as to whether this is the same issue? Thanks. 0 -
Hello @phil99, Thank you again for opening the support ticket. Your forum threads and tickets have helped us identify multiple issues with the BackBlaze B2 backup destination. Edit: Jul-16-2019: CPANEL-27679 will address this issue through the addition of a separate upload pathway for files larger than 5GB. Thank you. 0 -
Hi Michael That's great. Looking forward to the fix being published so we can test again. Regards, Phil 0 -
Hi Last night, for the first time since upgrading to WHM 80.0.11, we again tested backing up the same large account as reported in the original post. Exactly the same thing has happened. The file has been split into 3 parts (two of 5GB and one of 1.8GB), and whilst all parts appear to have been successfully uploaded, the parts have not been reassembled into a single file on Backblaze. Again there are no errors in the log file, a successful transfer has been reported. 0 -
Exactly the same thing has happened. The file has been split into 3 parts (two of 5GB and one of 1.8GB), and whilst all parts appear to have been successfully uploaded, the parts have not been reassembled into a single file on Backblaze. Again there are no errors in the log file, a successful transfer has been reported.
Hello @phil99, Can you report this as a separate defect using our0 -
Hi Michael Request ID is: 12453139 Some further information (which I have expanded on in the ticket with some log entries) is that the transport of the split file seems to take an excessively long time. An 11.8GB file took over 2 and a half hours to transport. Another account which was a 4.8GB file (and so presumably wasn't split) took just under 11 minutes in the same run. Let us know if you need any more information. Regards, Phil 0 -
Hi Phil, Thanks, I've added a note to the ticket explaining the issue. I'll monitor the ticket and report the outcome here once it's closed. Thank you. 0 -
Hello, To update, case CPANEL-27679 was opened to address an issue where split backup files are not reassembled on the Backblaze destination. I'll update this thread with more information on the status of this case as it becomes available. Thank you. 0 -
Hello, Is there any progress on this? We would like to use B2 as we can save a lot of money in storage costs by using B2 vs other cloud platforms. Thanks 0 -
Is there any progress on this? We would like to use B2 as we can save a lot of money in storage costs by using B2 vs other cloud platforms.
We don't have a firm time frame to share on the publication of a fix at this time, however I've shared your feedback to the internal case and will update this thread as soon as new information is available. Thank you for your patience.0 -
Just to add that we have been using the backup to Backblaze successfully for our smaller accounts. The tar.gz file must be less than 5GB (5 billion bytes). Once they get larger than this, they need to be split, which is when the problems occur. 0 -
Backups are such an essential part of our job as sysadmins. I don't understand how something so essential can be overlooked. Apparently my backups have not been working for weeks, and I'm just now seeing this. There was no error message to let me know. IT was lucky that I went to restore a backup in a non emergency situation. Had there been a panicked client in need of help, this would have been a massive headache. 0 -
Hello @nhadsall, Can you confirm if you are experiencing issues with the BackBlaze B2 backup destination for all accounts, or is the issue isolated to accounts over 5GB (the issue noted in this thread)? The priority of this case was increased based on the feedback from this thread, but I don't have a specific time frame to offer on a solution at this time. I'll provide more updates as they become available. Thank you. 0 -
Hello @nhadsall, Can you confirm if you are experiencing issues with the BackBlaze B2 backup destination for all accounts, or is the issue isolated to accounts over 5GB (the issue noted in this thread)? The priority of this case was increased based on the feedback from this thread, but I don't have a specific time frame to offer on a solution at this time. I'll provide more updates as they become available. Thank you.
Hello, Yes this is for all accounts over 5GB. All smaller accounts work fine. About 80% of our clients are over 5gb. This is a pretty big important issue since it does impact mission critical operations. Thanks0 -
Hello Everyone, I'm continuing to monitor this case and will report back here as soon as more information on the status of a fix is available. In the meantime, keep in mind that backups over 5G are still stored successfully on the remote destination. The issue here is that the larger backups are not automatically re-assembled on the BackBlaze destination, making it more difficult to manually move and restore the archives using traditional file transport methods. If you need to restore an account, and the backup is split into chunks on the BackBlaze destination, then you can execute the following command to download the archive from BackBlaze to your cPanel & WHM server in it's full state: /scripts/cpbackup_transport_file --transport $ID --download backup/2019-07-23/accounts/username.tar.gz --download-to /backup/username.tar.gz
Notes: 1. Replace "$ID" with the ID of your BackBlaze B2 backup destination. You can find the ID by executing the following command:whmapi1 backup_destination_list
2. Replace "backup/2019-07-23/accounts/username.tar.gz" with the path to the backup you want to use on the BackBlaze destination. Note that while you won't see "username.tar.gz" as a file on BackBlaze because the archive is split, you can still use this filename in the command to represent the username of the cPanel account and the script will detect which files to download. 3. Replace "/backup/username.tar.gz" with the full path to filename on the cPanel & WHM server where you will store the backup archive. Let me know if this helps. Thank you.0 -
Hi are there any updates or ETA on getting this fixed? Thanks! 0 -
Hello @nhadsall, There's no update to report at this time, but the workaround noted in my previous post to this thread allows for for functionality in the meantime. Thank you. 0 -
This also has affected us. If this isn't fixed in the next release it would be great if the method to download the split files as mentioned here in the forums was documented somewhere and perhaps a notice on the backups page about b2's current major caveat 0 -
Hello @thirstygrunt, There is no confirmation of a planned date or version for case CPANEL-27679 at this time. I'll continue to monitor the status of this case and report back here as soon as new information is available. A documentation case is not open at this time due to the potential of a change stemming from this case (regarding the requirement to download the split files). Thank you. 0 -
Hello, Is there any priority for fixing this? We are still using B2 and Amazon so doubling our out going backup bandwidth is getting pretty expensive. I was told to leave both on as backup destinations because we have B2 tied into some customer web dashboards we made. Thanks 0 -
Is there any priority for fixing this? We are still using B2 and Amazon so doubling our out going backup bandwidth is getting pretty expensive. I was told to leave both on as backup destinations because we have B2 tied into some customer web dashboards we made.
Hello :) To update, we're tentatively (provided no issues arise during quality testing) planning to publish CPANEL-27679 in an upcoming build of cPanel & WHM version 84. This case will modify the BackBlaze B2 driver to utilize BackBlaze's large_file_upload API so that files up to 10TB are automatically re-assembled on the BackBlaze destination. I'll update this thread with more information as it becomes available. Thank you.0 -
Hello :) To update, we're tentatively (provided no issues arise during quality testing) planning to publish CPANEL-27679 in an upcoming build of cPanel & WHM version 84. This case will modify the BackBlaze B2 driver to utilize the large_file_upload API so that files up to 10TB are automatically re-assembled on the BackBlaze destination. I'll update this thread with more information as it becomes available. Thank you.
Very happy to hear that! Thanks0 -
Hello Everyone, The fix was released to cPanel & WHM 83.9999.157 today (this is a development build for version 84): Fixed case CPANEL-27679: Modify BackBlaze B2 driver to utilize large_file_upload API up to the 10TB file size limitation and stop splitting files after 5GB. You can read about our Release Tiers on the document below: Thank you. 0 -
Hello Everyone, The fix was released to cPanel & WHM 83.9999.157 today (this is a development build for version 84): Fixed case CPANEL-27679: Modify BackBlaze B2 driver to utilize large_file_upload API up to the 10TB file size limitation and stop splitting files after 5GB. You can read about our Release Tiers on the document below: Thank you.
Very happy to hear progress! THANK YOU for keeping all of us updated.0
Please sign in to leave a comment.
Comments
29 comments