Skip to main content

Backup restoration causing high load averages?

RicardoFC
Hi, My load averages are always something like 3.02 4.23 3.89, however once I restore a site it goes to 30.04 29.93 25.78, as site is large server load average is quite high during a lot of time, is there a way to limit backup restoration to use this much CPU?

Comments

19 comments

Date Votes
  • Suede Y
    Hey there Ricardo, You may be able to do it using CPU watch through SSH like this: /usr/local/cpanel/bin/cpuwatch 4 /scripts/restorepkg username This should have the CPU watch keep an eye on the process and pause it when the load goes over 4 and if it hits that load it pauses the process until the load is under that specific number again. For your server you may want to do a higher number but that should be a good way for that.
    0
  • RicardoFC
    Hey there Ricardo, You may be able to do it using CPU watch through SSH like this: /usr/local/cpanel/bin/cpuwatch 4 /scripts/restorepkg username This should have the CPU watch keep an eye on the process and pause it when the load goes over 4 and if it hits that load it pauses the process until the load is under that specific number again. For your server you may want to do a higher number but that should be a good way for that.

    Tried, still using that amount of CPU, if the server starts unzipping files it stops the transference process but it keeps unzipping, once it reaches below 4 it justs says the restoration is done :(
    0
  • cPanelMichael
    Hello, Could you open a support ticket using the link in my signature so we can take a closer look and verify if it's mostly related to your server's hardware, or if there's anything in the restoration process that's leading to the increased load? Thank you.
    0
  • Suede Y
    Tried, still using that amount of CPU, if the server starts unzipping files it stops the transference process but it keeps unzipping, once it reaches below 4 it justs says the restoration is done :(

    I"m sorry to hear that. I did see that Michael recommended to make a ticket. Can you update the thread when you can to see what happened? I"m pretty curious on the outcome of the ticket.
    0
  • cPanelMichael
    Hello, Feel free to post the support ticket number here should you decide to open a ticket and we can update this thread with the outcome. Thanks!
    0
  • Bidi
    Hello, I got the same problem, is there a way to do the limit as default ? When i restore account it dose high load, even when i restore it from cpmove file (WHM), even when some customer restore hes website when i was using jetbackup (we uninstaled because the server was getting crazy) high load 30 40 % and usualy the load is 2, 3 Thank you
    0
  • cPanelMichael
    Hello @Bidi, I encourage you to open a feature request if you'd like to see an option that allows you to configure a default cpuwatch value when using the Restorepkg script: Submit A Feature Request Note that generally a backup restoration shouldn't lead to an excessive server load. You may want to look into upgrading your server hardware or checking to ensure there are no hardware issues with your disk. Thank you.
    0
  • rarod
    I opened a ticket to JetBackup about this restore issue. They answer: Hello, This is not an issue. You are requesting new feature, and we will consider it. Thank you,
    So no luck.
    0
  • JetAppsClark
    Hello rarod, Can you please update us if you were able to make some progress with this issue? At a glance it looks like an IO issue, usually caused when backup folder and data folder are on the same mount, or when the server is over crowded with huge IOwait. If you're still encountering these issues, please let us know via our Helpdesk so our support techs can assist you!
    0
  • rarod
    Can you please update us if you were able to make some progress with this issue?

    I can add some details about this: I have high IO during restores, in theory JetBackup has options to use CloudLinux LVE for limit IO. But this is not working in restores because some restore processess run with root user instead of regular user.
    At a glance it looks like an IO issue, usually caused when backup folder and data folder are on the same mount, or when the server is over crowded with huge IOwait.

    Backup destination is a remote server.
    f you're still encountering these issues, please let us know via our Helpdesk so our support techs can assist you!

    I already have opened ticket: #433207. This was closed with: "This is not an issue. You are requesting new feature, and we will consider it."
    0
  • elialum
    Rarod, Can you tell if it's a MySQL issue, ? Thanks, Eli.
    0
  • rarod
    Rarod, Can you tell if it's a MySQL issue, ? Thanks, Eli.

    I do not think so, my MySQL data directory is in SSD disks (so it is difficult to have IO problems). The problem is with user files, they are stored in HDDs. Can you take a look at JetBackup ticket #433207? There you can see that some processess are not using LVE limits.
    0
  • Alejandro Castro
    I do not think so, my MySQL data directory is in SSD disks (so it is difficult to have IO problems). The problem is with user files, they are stored in HDDs. Can you take a look at JetBackup ticket #433207? There you can see that some processess are not using LVE limits.

    I agree with this. I've faced similar problems when restoring accounts using Jetbackup. If the account have a large number of files the process can get a spike in cpu load and high io wait. Adding ionice, renice or lve don't seem to work at all
    0
  • elialum
    Hi,
    I do not think so, my MySQL data directory is in SSD disks (so it is difficult to have IO problems). The problem is with user files, they are stored in HDDs. Can you take a look at JetBackup ticket #433207? There you can see that some processess are not using LVE limits.

    I asked because you mentioned you opened a ticket with Oracle, so it made sense that it's a MySQL issue. Does your IOwait usually high ? Can you post some averages ? As for the ticket you have with us (#433207), can you re-open it and grant me access to the server with permission to restore an account and test it ? If so, please provide as much information as possible (test account name), and a link to this thread. Thanks :) Eli.
    0
  • rarod
    I asked because you mentioned you opened a ticket with Oracle, so it made sense that it's a MySQL issue.

    This was not me.
    Does your IOwait usually high ? Can you post some averages ?

    Here you can see clearly the restore: i.imgur.com/El8nJVa.png
    As for the ticket you have with us (#433207), can you re-open it and grant me access to the server with permission to restore an account and test it ? If so, please provide as much information as possible (test account name), and a link to this thread.

    I will do it right now Thank you.
    0
  • rarod
    JetBackup tested the restore issue in one of my servers. The answer: according the the logs I've collected the major reason for the high IO was "gtar" process in which cPanel "restorepkg" is using when restoring the account. As you are using JetBackup 3.1, we believe that upgrading to the latest 3.2.15 version could help as we changed the entire concept and shouldn't use gtar while restoring. Furthermore, with the latest JetBackup 3.2 there is much more debug info which will help us understand better what is going on behind the scenes.
    I will wait for 3.2 and test if the issue is fixed.
    0
  • rarod
    The issue is not fixed. Now we are in Jetbackup 4.0.6 and I had to disable restores in some some of my non-ssd servers to avoid the issue.
    0
  • elialum
    Rarod, Thank you for bringing this issue back :) From the information we gathered from your server, seems like the server load is caused indeed from cPanel's internal /script/restorepkg script. I believe that this is a great example for "it's not a bug, it's a feature" case. I'll try to explain - cPanel's internal /script/restorepkg system should accept "path/to/incremental_backup" as per their help -
    root@cpanel-dev-cpanel home2]# /scripts/restorepkg --help /usr/local/cpanel/bin/restorepkg [--unrestricted] [--restricted] [--force] [--newuser ] [--allow_reseller] [--skipaccount] [--ip=(y|n|custom IP)] [--disable=[,]] -- [cpuser|/path/to/cpuser-file|/path/to/extracted-cpuser-file|/path/to/incremental_backup]
    Regardless of JetBackup, let's make a test with a small account - Create an incremental backup - /scripts/pkgacct --incremental frooty /home2/ Restore the incremental backup - /scripts/restorepkg --force /home2/cpmove-frooty Although we used "incremental" backup to restore the backup, seems like cPanel will create an archive, and then will extract it again -
    8683 frooty 20 0 121708 1244 960 S 14.0 0.0 0:00.42 /usr/bin/gtar --extract --no-same-owner --preserve-permissions --file - --no-wildcards-m+ 8682 root 20 0 121860 1068 884 D 11.0 0.0 0:00.33 /usr/bin/gtar --create --file - --no-wildcards-match-slash --anchored --sparse --blockin+
    This is also documented in the restore log -
    [ 6581][RESTORE:1 ][A:frooty ]: ArchiveManager [ 6581][RESTORE:1 ][A:frooty ]: Preparing archive for restoration " [ 6581][RESTORE:1 ][A:frooty ]: This archive"s payload appears to be in the archive"s "cpmove-frooty" directory. [ 6581][RESTORE:1 ][A:frooty ]: ArchiveManager [ 6581][RESTORE:1 ][A:frooty ]: The system successfully prepared the archive for restoration.
    When doing this with 18G account, you should expect high load :) Going back to JetBackup, we are working on a workaround for this where you should be able to restore files directly to the homedir (reverse rsync), allowing much faster restores and using less CPU.
    0
  • rarod
    Hello, Thanks, I am glad that you are working on a workaround for the feature. Any ETA?
    0

Please sign in to leave a comment.

Didn't find what you were looking for?

New post