S3FS for backup?
Hey guys,
Managed to get s3fs to work with my CloudLinux instance and it seems to be working well.
I've mounted it on my fstab as per the documentation
I configured the backup on WHM to that mountpoint. The backup works fine for the system files, but it will backup exactly one Account then fail. In the backup log, the last few lines are:
I've also set the backup configuration to uncompressed because I thought it was because the system couldn't run tar on the bucket, but it's still tarballing the files? Anyone have success with s3fs and backup? Is there a recommended setting?
s3fs#bucket /backup-s3 fuse _netdev,allow_other 0 0I configured the backup on WHM to that mountpoint. The backup works fine for the system files, but it will backup exactly one Account then fail. In the backup log, the last few lines are:
info [backup] Successfully backed up account "" to "/backup-s3/2018-09-30/accounts"
[2018-09-30 11:09:16 +1000] info [backup] Adding metadata information for to backup at /backup-s3/2018-09-30
[2018-09-30 11:09:17 +1000] info [backup] Queuing transport of file: /backup-s3/2018-09-30/backup_incomplete
[2018-09-30 11:09:17 +1000] info [backup] no_transport = 0 .. and queueid = TQ:TaskQueue:838
[2018-09-30 11:09:17 +1000] info [backup] leaving queue_backup_transport_item
[2018-09-30 11:09:17 +1000] warn [backup] Pruning of backup files skipped due to errors. at /usr/local/cpanel/bin/backup line 392.
bin::backup::run("bin::backup", "--force") called at /usr/local/cpanel/bin/backup line 120
[2018-09-30 11:09:21 +1000] info [backup] Scheduling backup metadata rebuild
[2018-09-30 11:09:21 +1000] info [backup] Queuing transport reporter
[2018-09-30 11:09:21 +1000] info [backup] no_transport = 0 .. and queueid = TQ:TaskQueue:839
[2018-09-30 11:09:21 +1000] info [backup] leaving queue_backup_transport_item
[2018-09-30 11:09:21 +1000] info [backup] Completed at Sun Sep 30 11:09:21 2018
[2018-09-30 11:09:21 +1000] info [backup] Final state is Backup::PartialFailure (0)
[2018-09-30 11:09:21 +1000] info [backup] Sent Backup::PartialFailure notification.I've also set the backup configuration to uncompressed because I thought it was because the system couldn't run tar on the bucket, but it's still tarballing the files? Anyone have success with s3fs and backup? Is there a recommended setting?
-
I spoke with cPanel support and ended up concluding that it's not possible to use S3FS without hardlink support. Instead I created a simple backup script which will transfer backups to our S3 bucket. To avoid taking up a ton of space it transfers the backups one at a time and removing the file one at a time to S3. There is a condition there that will allow accounts less than 1GB to just wait until the very end of the script to bulk upload to S3. This is to help speed up the total time of the script (saves a few minutes). You can turn this off by changing the number to 0 (so even very small backups will transfer one at a time). Feel free to use it WITHOUT WARRANTY. Contributions to this code is appreciated! #!/bin/bash #Please install AWS's CLI tools using pip. # ------------ EDIT THIS --------------- # Change TMPDIR to where you want to put the .tar temporarily, this needs to be on your filesystem with Root Permissions. TMPDIR="/backupTmp" #This is your Final Directory - Use S3FS to attach an AWS Drive BUCKETNAME="s3bucketname" #What is the filesize you want to skip transffering on # - this is so you can send multiple small files in batches. FILESIZELIMIT=1000000000 #1000Mb # ----------- STOP EDITING -------------- echo "" echo "--- Starting individual Backup script --- " DATE=`date +%Y-%m-%d` cd "$TMPDIR" for f in /var/cpanel/users/* ; do user=$(basename $f) #If we have a username in the arguments you can use ./backup.sh to force only one backup. if [ -n "$1" ] && [ "$1" != "$user" ] then echo "- Skipping $user - " continue fi echo "" echo "" echo "------------ Backing up $user ----------------- " #start backup /usr/local/cpanel/scripts/pkgacct --backup --nocompress --split $user $TMPDIR FILESIZE=$(wc -c < "$user.tar") if [ $FILESIZE -le $FILESIZELIMIT ] #If the compressed file is less than 500MB.. Skip sending it to S3 so we can batch it. then echo "--- Skipping small TAR for batch backup ---- " continue else echo "-------- Moving Large file to S3 -------- " ~/.local/bin/aws s3 mv "$user.tar" "s3://$BUCKETNAME/$DATE/accounts/$user.tar" fi done echo "-------- Batch Moving to S3 -------- " ~/.local/bin/aws s3 mv "$TMPDIR" "s3://$BUCKETNAME/$DATE/accounts/" --exclude '*' --include '*.tar' --recursive echo " ------------------ COMPLETE ------------------"0 -
Hi @massafiri Thank you for following up on the issue and providing a workaround. I strongly urge you to file a feature request if you'd like to see this functionality as something available in the product. Please click the link in my signature to open it! Thanks! 0
Please sign in to leave a comment.
Comments
2 comments