Skip to main content

Incremental Backups taking up too much space

adam987123
Hello I am a bit confused about Incremental backups. I am using them on my server which is 100GB. The backup is about 14GB in size. I have incremental backups set up daily with retention for 30 days. I find that after 9 days I am now running out of space (backups are using 50GB of space) - How is this possible? My only thoughts are that the incremental backups create a full backup each week? Is this correct? If so can I disable this? Or if not what could be happening? Any help or advice will be appreciated. Many Thanks Adam

Comments

18 comments

Date Votes
  • 24x7server
    Hi, If you carefully check what you configured you will see what is the cause of this: You have set daily backup with retention for 30 days, so every day backup will be retained. Now if you have a 14GB of backup, then each day it will do backup and store it separately as you configured for 30 days. It will keep 30 days backup, so 14GB x 30 days should be the disk space it will occupy. You may got confused youself with the backup. Retention will retain all the days backup and not overwrite any backup. it will overwrite only after 30 days..
    0
  • adam987123
    Hi, If you carefully check what you configured you will see what is the cause of this: You have set daily backup with retention for 30 days, so every day backup will be retained. Now if you have a 14GB of backup, then each day it will do backup and store it separately as you configured for 30 days. It will keep 30 days backup, so 14GB x 30 days should be the disk space it will occupy. You may got confused youself with the backup. Retention will retain all the days backup and not overwrite any backup. it will overwrite only after 30 days..

    Hi I get this but it is incremental backups. It is only suppose to save the files that have changed each day. This is the first time i have used incremental backups so so i am not sure if this is correct. I don't think the documentation is very good for this and i am a little confused with how it works?
    0
  • cPanelMichael
    Hello, The disk space will appear to be the same, but it's not actually duplicating the space. It's simply the nature of how hard links work. This is described in more detail on the following thread: How do the backups work exactly? Let us know if this helps. Thanks!
    0
  • adam987123
    Hello, The disk space will appear to be the same, but it's not actually duplicating the space. It's simply the nature of how hard links work. This is described in more detail on the following thread: How do the backups work exactly? Let us know if this helps. Thanks!

    Thak you for the reply. So are you saying that when I go to Show Current Disk Usage on my control panel this is not the correct figure? I am also getting DISKWARN email from Cpanel should this be ignored? Does this also mean if I use the new feature in Cpanel 66 where I can save incremental backups to a remote location (in my case AWS S3) this will use the full disk space or will this just save the changed files? When I delete all backups the total disk usage is 23GB after 9 days the 100GB disk is almost full.
    0
  • uk01
    Same issue here, incremental is using all our backup space and doing full backup every day... Used 1tb bandwidth in 2 days!
    0
  • 24x7server
    Hi I get this but it is incremental backups. It is only suppose to save the files that have changed each day. This is the first time i have used incremental backups so so i am not sure if this is correct. I don't think the documentation is very good for this and i am a little confused with how it works?

    You did not seem to be having the concept of the increatment and retention of backup understood completely. Suppose you have a daily backup configured with 1 retention, so on Monday a backup is performed of 14 GB, on Tuesday agai backup will start, but since you have 1 retention, it will stat overwriting the previous backup and add the files those ar new and only increase it to suppose 1 GB, so making it 15 GB and this goes for a week and at the end of the week you only have 1 daily backup configured (Incremental), so the data all over the week go on syncing into the same backup making it 14+7 21 GB. Now, what you have done is you create more retention, take for example you made 7 retentions of daily backup, so a backup will done on Monday of 14 GB, on tuesday it will take fresh backup and so on and at the end of the week, you will have 7 retention configured and that will have monday to sunday backup 14x7 = 98 GB of data. and when the next week comes on Monday backup will again start, but since you only had 7 days backup retention and 7-day backup is complete, it will overwrite the first backup i.e, of Monday, then next day, it will change the tuesday backup, so at the end of each week you will have 7 backups separte... And now back to your scenario, you have configured 30 days retention, now you can calculate what will happen.. 14x30 = 420 GB.. I hope your concept about retention is clear now..
    0
  • cPanelMichael
    Hello, Internal case CPANEL-15493 is open to address an issue where the removal of the local backup directory fails after a successful transport to a remote destination when incremental backups are enabled in cPanel version 66. I'll monitor this case and update this thread with more information as it becomes available. Thank you.
    0
  • tui
    Hello, Internal case CPANEL-15493 is open to address an issue where the removal of the local backup directory fails after a successful transport to a remote destination when incremental backups are enabled in cPanel version 66. I'll monitor this case and update this thread with more information as it becomes available. Thank you.

    Hello, any update on this case?
    0
  • cPanelMichael
    Hello, any update on this case?

    A resolution is in testing, but there's currently no time frame available for the resolution's publication. I'll continue to monitor this case and update this thread with new information as it becomes available. Thank you.
    0
  • tui
    A resolution is in testing, but there's currently no time frame available for the resolution's publication. I'll continue to monitor this case and update this thread with new information as it becomes available. Thank you.

    I see that cPanel V66 is now in Stable tier... I do not think this version should have been put on this tier with this big flaw ... all people expecting this release to use incremental remote backups will fill their local and backup hard drives in a few days and all their servers will crash... this is not responsible for your part
    0
  • brt
    The problem I am experiencing isn't that backups are using too much space, but my remote backup directories (daily, etc.) aren't being completely removed. They're mostly empty, but almost all have 1-3 accounts remaining in them -- usually large accounts with a ton of files. I presume it's timing out?
    0
  • cPanelMichael
    The problem I am experiencing isn't that backups are using too much space, but my remote backup directories (daily, etc.) aren't being completely removed. They're mostly empty, but almost all have 1-3 accounts remaining in them -- usually large accounts with a ton of files.

    There are separate cases open referenced on the following thread that might relate to this specific issue: Remote incremental backups - timeouts Thank you.
    0
  • tui
    Any update on this?
    0
  • cPanelMichael
    Any update on this?

    There's no new information to remote on internal case CPANEL-15493 at this time. I'll continue to monitor the status of this case and update this thread with more information as it becomes available. Thank you. Update: The resolution associated with internal case CPANEL-15493 is scheduled for inclusion with cPanel version 68.
    0
  • cPanelMichael
    Hello, To update, internal case CPANEL-15493 is included with cPanel version 66.0.26: Fixed case CPANEL-15493: Make sure incremental dirs are removed when asked. Thank you.
    0
  • tui
    Excellent, let me test it!
    0
  • tui
    Hello again, Now that issues are resolved and a temporary workaround is available for CPANEL-15398 i enabled the incremental remote backups for my servers, and it seems that everything is working as expected, i just have a few questions: When i go to my backup server, i could see that backups are there and space is incremental as you can see in follow lines of the backup folder of one server, all other servers backups folders has the same structure and same pattern of folder size (first backup folder is bigger than then following days folders):
    root@backupsrv:/home/serverX# du -csh * 6.5G 2017-10-24 305M 2017-10-25 530M 2017-10-26 510M 2017-10-27 8G total
    The first folder is the first full backup (its needed for incremental, obviously) as you can see is the largest folder and is the same size as the sum of the sizes of the accounts on the server, so the size is correct and its full, then the folders of following days are smaller because only files with changes are backed up; i have 6 days of retention, it's two days before this is accomplished, so my question, What will happen on day 7 with the largest folder? this has to be removed, but what about all the original data that has not been modified on other days? Will the data of day one be copied to the folder of day two and so on? Or how does this will work? Thanks in advice
    0
  • cPanelMichael
    Hi @tui, The original files are moved to the existing backup directories once their associated backup directory is pruned. You can read more about how this works at: How do the backups work exactly? Thank you.
    0

Please sign in to leave a comment.

Didn't find what you were looking for?

New post