Skip to main content

Legacy restore, why copy twice?

mindnetcombr
I know legacy backup/restore is dead, but today I need restore a cpanel server with legacy restore, 1Tb of data. All data stored on a backup hdd, without compression, without .tar or .tar.gz - my backup is incremental, file by file, folder by folder. This is how legacy restore work: 1) copy the account backup to a temporary folder inside the backup disk; 2) copy the data copied to temporary folder, to the final destination (/home); Repeat this 600 times, with 1Tb of data. A big waste of time and resources, why legacy copy to a temporary location to after this restore the account? Why not restore direct from /backup/cpbackup/daily/account_login/ ???? This temporary copy double the time necessary to restore, and stress the backup disk with a lot of writes. I dont know how the new restore system work, I hope not in the same way. This dont make sense, just lock backups while do restore.

Comments

18 comments

Date Votes
  • 24x7server
    Hi, The simplest way for you would be that you just create an archive from those user backup folder and restore it directly.. # tar -zcvf user.tar.gz user This will create a tar and then you can restore it directly..
    0
  • mindnetcombr
    Hi, The simplest way for you would be that you just create an archive from those user backup folder and restore it directly.. # tar -zcvf user.tar.gz user This will create a tar and then you can restore it directly..

    Yes but I will lose more time, to compact the backup to .tar.gz, and when use /restorepkg /backup/cpbackup/daily/user.tar.gz this script will create temporary files again on /backup/cpabackup/daily/cpanelpkgrestore.TMP.work.SOMETHING/ Compact to .tar.gz to restore will not avoid cpanel to use the temporary folder. This issue is similar to this one: restorepkg can't restore from a read-only file system cpanel legacy restore ALWAYS copy the backup to a temporary folder before restore. This step is not necessary since in a incremental backup all files are uncompressed.
    0
  • cPanelMichael
    Hello, You can browse to "WHM >> Backup Configuration", configure the "Incremental" backup type, configure the local backup directory to the partition where the incremental backups are stored, and then move the existing incremental backups to the "/$configured-backup-dir/date/accounts" location. Once you do this, the backups should appear in WHM's
    0
  • mindnetcombr
    Hello, You can browse to "WHM >> Backup Configuration", configure the "Incremental" backup type, configure the local backup directory to the partition where the incremental backups are stored, and then move the existing incremental backups to the "/$configured-backup-dir/date/accounts" location. Once you do this, the backups should appear in WHM's
    0
  • 24x7server
    Hi, Restoration will create a temporary session for restoration. What exact is the issue you are facing with temporary sessions/copy?
    0
  • mindnetcombr
    Hi, Restoration will create a temporary session for restoration. What exact is the issue you are facing with temporary sessions/copy?

    the issue is the time. for every account restored I need wait cpanel create a temporary copy. So one account with 80Gb, 200.000 inodes, this take almost 1 hour. I have 600 accounts, fresh hardware, and more than 30 hours of restore. If dont have the temporary copy, this will take half time. This is the issue. thank
    0
  • 24x7server
    Hi, I would suggest you do as suggested by the Michael for one account and see what occurs and how much time it takes for it to restore..
    0
  • mindnetcombr
    Yes I will! thank everyone!
    0
  • cPanelMichael
    Michael, in this case the "new" backup system will restore without create a temporary copy? Restore will go direct from date/accounts to user account?

    Temporarily files are utilized, however the restore feature runs a check before starting the restoration process to ensure enough disk space is available. EX:
    Target "/backup/2017-08-17/accounts" on host "123.hostname.tld" has 11.16 GB free and requires at least 67.4 MB free, which includes space for temporary files.
    Thank you.
    0
  • mindnetcombr
    Hello, Today I run a test: using the new backup system, I restored one account with 14Gb. Unfortunately the backup system first copie the entire backup (14Gb) to a temporary directory /backup/date/accounts/cpanelpkgrestore.TMP.work.RuFrBXh5 Well... I have the incremental backup, so all files on my backup are ready to be restored (no .tar or .tar.gz). The restore system copy all data to a temporary folder, and only after this restore the account - dont make any sense with a incremental backup setup. This "step" (copy to temporary folder) only add time to restore and unnecessarily generates writing on the backup disk, which may be undesirable, because when restoring a backup is a moment that can be critical and the last thing we want is to stress the backup disk. I understand when the backup have the format .tar or .tar.gz is necessary copy to a temporary folder to extract the files, but with "incremental" option this temporary copy is useless. Maybe someone from cpanel reads this and agrees with my argument, if any, thank you bring the matter to the developers. There could be an automatic scan if the backup is incremental so do not use temporary folder. To finish: 1Tb restore of 600 accounts, in a new hardware (Xeon 1270 V6 + 32Gb RAM + Adaptec RAID1 7.2K) took 30 hours. Half this spent with cpanel copying backup to temporary folders unnecessarily. Thank
    0
  • cPanelMichael
    Hello, The following feature would likely help to some extent if you did not need to restore the entire account: cPanel user Granular backup/restore However, I encourage you to open a separate feature request if you'd like to see new functionality that doesn't make use of a temporary directory for restoration purposes: Submit A Feature Request Thank you.
    0
  • mindnetcombr
    Hello, The following feature would likely help to some extent if you did not need to restore the entire account: cPanel user Granular backup/restore However, I encourage you to open a separate feature request if you'd like to see new functionality that doesn't make use of a temporary directory for restoration purposes: Submit A Feature Request Thank you.

    Thank you. I will do this, and I will pray that other users will need to do incremental backup restore faster, so maybe - with many votes - cpanel.net decides to make this small change.
    0
  • Nguger Tarhely
    We are struggling with the same problem. We have a file level backup from users accounts on a remote server. After we mount the remote server's disk, and try to restore the backup with /scripts/restorepkg /mnt/something/$USERNAME, the script make a copy from the given user's backup, which make a very big - and unnecessary - overhead.
    Thank you. I will do this, and I will pray that other users will need to do incremental backup restore faster, so maybe - with many votes - cpanel.net decides to make this small change.

    Do you have any update about this topic? If you submitted a feature request, can you share the request's link, please?
    0
  • cPanelMichael
    Hi @Nguger Tarhely, I believe this is the feature request you are looking for: Restore backup from FTP storage In the meantime, how large are the individual accounts you are attempting to restore? You may want to consider restoring the accounts without the home directory, and then manually moving the home directory in place after the account itself is restored. Thank you.
    0
  • Nguger Tarhely
    Our problem is a little bit different. We have uncompressed backup on remote server. We attached the backup server's file system with NFS (we tried with ISCSI also, but the problem is the same). We can restore the user from backup with /scripts/restorepkg /mnt/something/randomusername, where "randomusername" folder contains the uncompressed backup: [SPOILER="find randomusername -maxdepth 1"> randomusername randomusername/cron randomusername/bandwidth randomusername/autossl.json randomusername/mma randomusername/logs randomusername/psql randomusername/cp randomusername/ssl randomusername/va randomusername/sslcerts randomusername/has_sslstorage randomusername/suspendinfo randomusername/mm randomusername/sslkeys randomusername/bandwidth_db.json randomusername/resellerfeatures randomusername/resellerconfig randomusername/psql_grants.sql randomusername/shadow randomusername/shell randomusername/homedir randomusername/version randomusername/ips randomusername/bandwidth_db randomusername/digestshadow randomusername/meta randomusername/sds2 randomusername/vf randomusername/mms randomusername/ssldomain randomusername/userconfig randomusername/locale randomusername/mysql-timestamps randomusername/addons randomusername/resellerpackages randomusername/domainkeys randomusername/apache_tls randomusername/quota randomusername/bandwidth_db_data.json randomusername/counters randomusername/httpfiles randomusername/mysql.sql randomusername/proftpdpasswd randomusername/dnszones randomusername/sds randomusername/homedir_paths randomusername/vad randomusername/userdata randomusername/suspended randomusername/psql_users.sql randomusername/mysql randomusername/pds When we run the /scripts/restorepkg /mnt/something/randomusername the restorepkg make a full copy from /mnt/something/randomusername to /mnt/something/cpanelpkgrestoreXXXXX, see below: [SPOILER=/scripts/restorepkg /mnt/something/randomusername output] [ 17059][RESTORE:1 ][A:randomusername ]: Target "/mnt/something/cpanelpkgrestore.TMP.work.86_1V5fk/unsafe_to_read_archive" on host "server.example.com" has 806.22 GB free and requires at least 491.08 MB free, which includes space for temporary files. [ 17059][RESTORE:1 ][A:randomusername ]: Target "/mnt/something/cpanelpkgrestore.TMP.work.86_1V5fk/unsafe_to_read_archive" on host "server.example.com" has 60,503,178 inodes free and requires at least 1,472 inodes free, which includes space for temporary files. During the restorepkg, I ran du -sh /mnt/something/cpanelpkgrestore.TMP.work.86_1V5fk which returned with nearly the same size as mentoined on the restorepkg's output (494 Mb). After the copy, the /scripts/restorepkg restore the user, but this copy is clearly unneccessary and make big overhead, because the data copied twice. Why is it important to copy the uncompressed backup to another folder? Can the restorepkg skip this step? Also, I checked the /scripts/restorepkg help, but it does not have any option to skip home directory: [SPOILER=/scripts/restorepkg --help output] /usr/local/cpanel/bin/restorepkg [--unrestricted] [--restricted] [--force] [--newuser ] [--allow_reseller] [--skipaccount] [--ip=(y|n|custom IP)] -- [cpuser|/path/to/cpuser-file|/path/to/extracted-cpuser-file|/path/to/incremental_backup] --allow_reseller Restore reseller privileges if the archive contains them. NOTE: This option is only available in unrestricted mode. --force Force the package to restore and overwrite all account settings and databases. If the account already exists on the system this implies --skipaccount. --ip Assign a dedicated IP address. You may specify a particular IP address, or "y" to have one randomly assigned. --newuser Specify the username to use for the restored account instead of the one contained within the archive. Databases and DB users will retain their names unless a conflict is detected. This option cannot be used with --skipaccount. --skipaccount Restore the package into the existing account with the same username. This option cannot be used in conjunction with --newuser. --restricted Use the Restricted Restore feature to restore the package. --unrestricted Restore the package in unrestricted mode. This flag is on by default and overridden by the --restricted flag. If we can skip the home directory, it would be nice. If you need more details, please let me know.
    0
  • cPanelMichael
    Hello, The temporary directory is for staging purposes during the restoration process. It's how the integrity of the backup archive used for the restore is validated. You could open a separate feature request if you have a specific request to change this behavior. As far as restoring the account without the home directory in the situation you described, you'd need to move the account's public_html directory from the directory you are restoring it from to another location (outside of the account). Thank you.
    0
  • plague
    4 years later and the restore process is still wasting time and resources duplicating incremental backups. Yesterday I needed to restore a 220GB account on a incremental backup. It took SIX hours just to duplicate HALF the backup on the "cpanelpkgrestore.TMP.work.xxx" folder! As a workaround, you can move the huge folders to somewhere else outside the user backup, restore the account and then rsync the folder to the restored account manually. I hope you find this thread before wasting a whole day waiting on a backup restoration.
    0
  • cPanelAnthony
    4 years later and the restore process is still wasting time and resources duplicating incremental backups. Yesterday I needed to restore a 220GB account on a incremental backup. It took SIX hours just to duplicate HALF the backup on the "cpanelpkgrestore.TMP.work.xxx" folder! As a workaround, you can move the huge folders to somewhere else outside the user backup, restore the account and then rsync the folder to the restored account manually. I hope you find this thread before wasting a whole day waiting on a backup restoration.

    As Michael suggested previously, you could open a feature request using the link in my signature if you are interested in seeing an option for restoring backups without temporary space for an integrity check. If you do so, please update this thread.
    0

Please sign in to leave a comment.

Didn't find what you were looking for?

New post