Solution for large files being sent to Dropbox?
I have set up a backup system to my Dropbox -- external to cPanel and the local server environment. I have used the script described here: Custom Backup Transport: Dropbox | cPanel Blog as well as various Forum posts to set up the Dropbox backup.
The script and the WHM backup interface appear to work correctly for most accounts, but there are persistent accounts that fail on _most_ but not all backups.
I believe the reason may be something like the time taken to transport the data to the external Dropbox fails due to the size of the transfers (each failed account is well over 1Gb+ due to email accounts).
Backups are compressed - backups are maximised timeout (50,000 seconds) - smaller backups are transported ok.
Example result error message is:
Unable to send "/backup/2019-01-22/accounts/accountname.tar.gz" to destination "Dropbox"
So;
It seems to be a timeout issue for the 50,000 seconds being needed to transport 1Gb+ (so on a minimum sized 1Gb backup file the backup needs to be transported at 1Mbps as a MINIMUM, so for a 3-4Gb file (of which we have a few accounts); thy need to be transported consistently at 3-4Mbps which from one server to another across the open net seems to be unlikely to transport :-(
Can cPanel provide any assistance to making this work? IS this likely to be a Dropbox issue or a Panel issue?
-
To clarify further: 1) Can we increase the 50k seconds hardlimit timeout value and see if this helps move larger files across to the backup destination? 2) If not, can the 50k timeout limit be set on a per-file basis rather than a per-transaction basis, if not already? (I am unclear on this) 3) Can the transport mechanism be able to report why the backup failed, for instance; if the timeout was reached? Thanks! 0 -
Yes there is a support ticket now open. I will update here as news comes through. 0 -
Hello @martin MHC, Thanks for opening the ticket. I'll monitor it and update this thread with more information as it becomes available. Note that for reference, the corresponding report on the GitHub repo for this script is available at: TECH-577: large size files are not being uploaded " Issue #6 " CpanelInc/backup-transport-dropbox Thank you. 0 -
Hello @cPanelMichael - sure, as referenced just a couple of posts above - are you able to give any answers to points a,b,c -if these are possible currently and if so how to achieve them, or if not currently possible any chance they'd be candidates for future versions? Searching the online feature requests system I can't see anything for extending the 50k time limit or asking for more verbose error reporting in email (if possible), should I put these as future feature requests, or is there somethig I'm unaware of as to these topics? Cheers 0 -
Hi @martin MHC This should now be corrected in the 'development' branch. Please update the Github issues if you have any problems. Thanks! 0 -
Hello @quanin, ) Can we increase the 50k seconds hardlimit timeout value and see if this helps move larger files across to the backup destination? 2) If not, can the 50k timeout limit be set on a per-file basis rather than a per-transaction basis, if not already? (I am unclear on this) 3) Can the transport mechanism be able to report why the backup failed, for instance; if the timeout was reached?
It looks like the issue was addressed, but I did want to answer your earlier questions: 1. A timeout value between 300 and 50,000 seconds is required and entering a value higher than 50,000 seconds is not supported. 2. The timeout is enforced on each upload attempt (e.g. each attempt to upload account.tar.gz to the remote destination). This is not a configurable option. 3. Transport failures/errors are logged to the /usr/local/cpanel/logs/cpbackup_transporter.log file. Is there a specific event/error that isn't currently logged?Also, please tell me you're advocating for this to be added to CPanel by default. Please? I find Dropbox a lot less finicky than Google Drive, which I only signed up for because CPanel backups.
I don't have anything new to report on the status of native support for DropBox integrated into the product as a backup destination, but I do see a fair amount of votes on the official feature request page: Add remote backup destination Dropbox I suggest adding voting and adding feedback directly to the feature request page if you haven't already done so. I've voted for it myself, and will push others to vote for it whenever I see questions/feedback about the use of DropBox as a backup destination. This is the best approach to getting native support added to the product. Thank you.0 -
Hello @cPanelMichael Just as a final note - William has put a lot of effort into this issue for me/us (and others) and the development branch on the GitHub is now the main branch/master. CpanelInc/backup-transport-dropbox This script now successfully and correctly resolves my issues raised in this thread and causes complete file transfers to my Dropbox destination. Many thanks to all for this :-) 0
Please sign in to leave a comment.
Comments
8 comments