Skip to main content

Remote incremental backups issue

uk01
Hi there, Quick question for you guys New incremental backup started this week with v66, however we realised after the first night that we needed to switch change remote backups to rsync for them to work with incremental, so we did that yesterday. backup server 1 (mounted) tuesday night - backup folder size 56GB last night - backup folder size 28GB backup server 2 (remote) tuesday night - backup folder size 0GB last night - backup folder size 28GB So all is working fine, my only question is that if it's rsync'd last nights backup folder over to the remote server, what happens to the hard links to the other 28GB of files which are not in the tuesday night folder on the remote server? If we retain 5 days incremental I'm guessing this sorts itself out after 5 days when tuesdays folder is ditched by the first backup server. Am I right? Cheers EDIT I didn't realise after 14 hours the backups were still running, they normally only take 4 hours, so I expected the second night incremental to be much less. Its now backup server 1 (mounted) tuesday night - backup folder size 56GB last night - backup folder size 56GB <<< the same size?! backup server 2 (remote) tuesday night - backup folder size 0GB last night - backup folder size 37GB (probably still rsync'ing) I'll see what happens tonight, I'm not sure why it's 56GB again, maybe it restarted the incremental full copy when I added the remote.

Comments

12 comments

Date Votes
  • cPanelMichael
    I'll see what happens tonight, I'm not sure why it's 56GB again, maybe it restarted the incremental full copy when I added the remote.

    Hello, The disk space will appear to be the same, but it's not actually duplicating the space. It's simply the nature of how hard links work. This is described in more detail on the following thread: How do the backups work exactly? Let us know if this helps. Thanks!
    0
  • uk01
    ahh, right, even when i run du -msh /foldername, I'll take a read of the thread. Could you also clarify this bit if you don't mind, many thanks "So all is working fine, my only question is that if it's rsync'd last nights backup folder over to the remote server, what happens to the hard links to the other 28GB of files which are not in the tuesday night folder on the remote server? If we retain 5 days incremental I'm guessing this sorts itself out after 5 days when tuesdays folder is ditched by the first backup server. Am I right?"
    0
  • uk01
    ps strange how it's taking all day to run, I'll see what happens tonight but it's filling the backup server up. Hopefully sort itself out
    0
  • cPanelMichael
    Could you also clarify this bit if you don't mind, many thanks "So all is working fine, my only question is that if it's rsync'd last nights backup folder over to the remote server, what happens to the hard links to the other 28GB of files which are not in the tuesday night folder on the remote server? If we retain 5 days incremental I'm guessing this sorts itself out after 5 days when tuesdays folder is ditched by the first backup server. Am I right?"

    The remote incremental backup process (with rsync) is designed to check if the files associated with the account backup on the cPanel server also exist on the remote backup destination. If the files do not exist on the remote backup destination (or if the files have changed), then it copies the actual files. Otherwise, it makes use of hard links when the files already exist on the remote backup destination. The following document is available if you'd like to read more about how account backup information is stored:
    0
  • uk01
    Very well thought out! I love cpanel, it gets better and better.
    0
  • uk01
    Unfortunately, as predicted yesterday (18 hours to backup instead of 4), this is not working properly. Last night it started backing up the whole thing again, our backup drives are 99% full and we've used nearly 1TB bandwidth in 24hrs. This puts our monthly predicted bandwidth 5Tb over our limit so I'm now going to have to stop backups until the issue is resolved. Back to compressed.
    0
  • InterServed
    Hi, Same boat here, incremental backup before took about 2 hours to finish, with v66 it took even 8 hours. Everything started the get - - up ever since they implemented date for the folders (had a few talks about this with them and they never wanted to support the previous incremental method). In the past i simply backup /localbackup/incremental and rsync this offsite (worked awesome). Since the date implementation in incremental backups (da heck with it) all is working 10 times slower. For me all they did was to ruin the product instead of improving it. I will pay 10 times more attention for cPanel changes from now on. So many years of cPanel and yet not a decent backup system (shame). For large servers, all the crap about meta and hard-links (way to much overhead i assume) -> BIG BIG BIG NO THANK YOU ! Wish they would at least still support the previous incremental method (optional) or if they had conducted any kind of feedback about the implications of this changes.
    0
  • cPanelMichael
    Hello, Internal case CPANEL-15493 is open to address an issue where the removal of the local backup directory fails after a successful transport to a remote destination when incremental backups are enabled in cPanel version 66. I'll monitor this case and update this thread with more information as it becomes available. As far as the performance of remote incremental backups, could you open a support ticket using the link in my signature so we can take a closer look? You can post the ticket number here so we can update this thread with the outcome. Thank you.
    0
  • graham_w
    Not sure if it's related, but I am using the rsync additional destination in WHM Backup for uncompressed backups and it seems to open two rsync processes and they both try and rsync the same file. I can also see on the remote server that its trying to save 2 copies of the same file. Once I kill one of the processes, it seems to complete the rsync as expected. Have opened ticket 8829847 to allow you guys to take a look.
    0
  • uk01
    Hello, Internal case CPANEL-15493 is open to address an issue where the removal of the local backup directory fails after a successful transport to a remote destination when incremental backups are enabled in cPanel version 66. I'll monitor this case and update this thread with more information as it becomes available. As far as the performance of remote incremental backups, could you open a support ticket using the link in my signature so we can take a closer look? You can post the ticket number here so we can update this thread with the outcome. Thank you.

    sorry, only just looked back at this thread. We didn't open a ticket, we went back to compressed backups for now as every night cpanel was backing up the full accounts from 5 servers and using all our bandwidth, then storing the full backups multiple times and using all backup space. I know the folders "look" like they are using space, which is how hard links work but we KNOW they were as the backup space is gone and all our bandwidth was used. Incremental should use much less! I'm wondering that incremental does not play well with an SSHFS mounted backup space, because effectively cpanel is saving to a folder called /backup, which has nothing in it, it's a remote sshfs folder, so it saved everything every night because it looks in the /backup folder and see anything, so saves everything again, instead of being able to know its a remote backup server it's linked to. We'll come back to this when I have more time, but for now I can confirm incremental does not work with mounted backup drives.
    0
  • uk01
    ps we have mounted drives via sshfs because a/ we don't want the backups thrashing our top spec ssds and b/ we want to make use of the space on each server, not have it 50% empty for backups. Our hourly backups backup directly to the backup server through an agent.
    0
  • uk01
    I've now found out this issue relates to hardlinking. Sshfs has supported hardlinking for some time now but cpanel doesn't allow it. See the discussion on the feature request here Provide SSHFS support for Backups Until this is implemented sshfs will not work with incremental.
    0

Please sign in to leave a comment.

Didn't find what you were looking for?

New post