Skip to main content

Remote incremental backups - timeouts

sp3ctre69
I have implemented remote incremental backups and by and large they seem to work well. However, I have had timeout issues during the pruning phase. Interestingly, when I look on the remote server it does look like it has done it. My concern is not so much around the timeouts, but rather the effect of them. In my old backup method if there was a timeout issue that wasn't corrected the backup would be incomplete, but the next day I would have a full backup in place. Now I have the remote system in place do I still get the same effect. i.e. if the folder with yesterdays date is incomplete on the remote server will there be a gap in my backup our will the gaps be filled during the next days cycle? Hopefully I am being clear, if not let me know and I'll try to elaborate.

Comments

51 comments

Date Votes
  • uk01
    data well in excess of 1/2 TB, a timeout of 30 doesn't always allow for a prune LBJ

    Ours is under 500gb, several GB per server per night. Good point on the SATA, we wouldn't use SSD for backup!
    0
  • cPanelMichael
    Hello, Internal case CPANEL-17253 addresses an issue where if the backup transporter hits the destination timeout (e.g. MAXIMUM_TIMEOUT value in /var/cpanel/backups/config) on an upload attempt, then subsequent attempts will ignore the timeout entirely, causing remaining transports to be delayed or possibly skipped. This affects multiple transporter types (including rsync). The resolution is included with cPanel version 70: Fixed case CPANEL-17253: Ensure timeout is set for each upload attempt. Thank you.
    0
  • LBJ
    Hello, Internal case CPANEL-17253 addresses an issue where if the backup transporter hits the destination timeout (e.g. MAXIMUM_TIMEOUT value in /var/cpanel/backups/config) on an upload attempt, then subsequent attempts will ignore the timeout entirely, causing remaining transports to be delayed or possibly skipped. This affects multiple transporter types (including rsync). The resolution is included with cPanel version 70: Fixed case CPANEL-17253: Ensure timeout is set for each upload attempt. Thank you.

    Yes, but that doesn't apply to the issue being discussed here. The problem being encountered by the above posters is where the upload has completed and the prune on the remote location is also completely successful, but cPanel's rsync.pm bales out too early on the prune process (spawned by the backup) and generates a spurious error email. Best regards, LBJ
    0
  • cPanelMichael
    The problem being encountered by the above posters is where the upload has completed and the prune on the remote location is also completely successful, but cPanel's rsync.pm bales out too early on the prune process (spawned by the backup) and generates a spurious error email.

    Could you open a support ticket so we can take a closer look at an affected system? You can post the ticket number here and I'll update this thread with the outcome. Thank you.
    0
  • LBJ
    Could you open a support ticket so we can take a closer look at an affected system? You can post the ticket number here and I'll update this thread with the outcome. Thank you.

    Thank you for the offer, but we've patched rsync.pm on our servers to give it a more reasonable timeout on the prune process and it now works as expected. Any of the above posters should be able to help you out with a sample system though. Best regards, LBJ
    0
  • brt
    I'm still getting a daily backup transport error email, saying Unable to prune transport -- but the backup completes and the destination(s) DO prune successfully; the oldest backup does delete as expected. I believe I have the timeout set at the max, although deleting hundreds of thousands of files does take a long time. I'm sick of this email. Please advise.
    0
  • cPanelMichael
    I'm still getting a daily backup transport error email, saying Unable to prune transport -- but the backup completes and the destination(s) DO prune successfully; the oldest backup does delete as expected.

    I encourage you to open a support ticket so we can take a closer look at the affected system to see what's happening. Let us know the ticket number and we'll update this thread with the outcome. Thank you.
    0
  • LBJ
    I'm still getting a daily backup transport error email, saying Unable to prune transport -- but the backup completes and the destination(s) DO prune successfully; the oldest backup does delete as expected. I believe I have the timeout set at the max, although deleting hundreds of thousands of files does take a long time. I'm sick of this email. Please advise.

    G'day brt, Can you please confirm that the error email is raised 60 minutes after the primary backup completes? You'll find the primary backup completion time in the last line of the current backup log at... /usr/local/cpanel/logs/cpbackup/ We've found on all our boxes that if you use the standard processing, anything spawned at the end of the primary backup process times out 60 minutes after the backup process spawns it. This includes the pruning process and anything run via the post backup hook. The MAXIMUM_TIMEOUT being set high won't fix this. Best regards, LBJ
    0
  • brt
    G'day brt, Can you please confirm that the error email is raised 60 minutes after the primary backup completes? You'll find the primary backup completion time in the last line of the current backup log at... /usr/local/cpanel/logs/cpbackup/ We've found on all our boxes that if you use the standard processing, anything spawned at the end of the primary backup process times out 60 minutes after the backup process spawns it. This includes the pruning process and anything run via the post backup hook. The MAXIMUM_TIMEOUT being set high won't fix this. Best regards, LBJ

    - Removed -
    0
  • LBJ
    - Removed -

    G'day brt, Is the error raised 195 minutes after spawning every day on your servers? Best regards, LBJ
    0
  • brt
    Why was my screenshot removed? Today my backup process completed email came at 5:03 and the backup transport errors arrived at 6:43.
    0
  • Infopro
    New Why was my screenshot removed?

    Please attach screenshots to your posts, do not link to them. Guide To Opening An Effective Forums Thread
    0
  • sp3ctre69
    Thank you for the offer, but we've patched rsync.pm on our servers to give it a more reasonable timeout on the prune process and it now works as expected. Any of the above posters should be able to help you out with a sample system though. Best regards, LBJ

    Interesting, I have had the same issue come back (sites have grown, so maybe the cause. I have changed the timeout in Rsync.pm to 300 (from 30). Will see if that helps.
    0
  • sp3ctre69
    Interesting, I have had the same issue come back (sites have grown, so maybe the cause. I have changed the timeout in Rsync.pm to 300 (from 30). Will see if that helps.

    Oh well, didn"t work
    0
  • cPanelMichael
    Interesting, I have had the same issue come back (sites have grown, so maybe the cause. I have changed the timeout in Rsync.pm to 300 (from 30). Will see if that helps.

    Hello, Can you verify the step-by-step details of the specific issue you are facing? There are a couple of different issues reported on this thread, so I want to be sure I understand exactly what you are reporting. Thank you.
    0
  • brt
    I am STILL getting *daily* backup transport emails reporting "Unable to prune transport "xxxxxxx" "ssh slave failed: timed out" These backup destinations are not high performance, nor should they need to be. Also, pruning does seem to finish, as I don't have files building up over time. Can someone please help make this stop?
    0
  • cPanelMichael
    I am STILL getting *daily* backup transport emails reporting "Unable to prune transport "xxxxxxx" "ssh slave failed: timed out" These backup destinations are not high performance, nor should they need to be. Also, pruning does seem to finish, as I don't have files building up over time. Can someone please help make this stop?

    Hello, Can you open a support ticket using the link in my signature so we can take a closer look? Thank you.
    0
  • brt
    Ticket created
    0
  • cPanelMichael
    Ticket created

    Hi @brt, Can you post the ticket number here or send it via a private message? Thank you.
    0
  • CanSpace
    Thank you for the offer, but we've patched rsync.pm on our servers to give it a more reasonable timeout on the prune process and it now works as expected. Any of the above posters should be able to help you out with a sample system though. Best regards, LBJ

    How did you patch this? Edit: Nevermind - I assumed this file would be encoded or something. Does seem like a fairly straightforward patch. I'm surprised this hasn't been mentioned more, and I'm surprised that cPanel (seemingly without purpose) limits us to a max 300 second timeout.
    0
  • CanSpace
    Sadly patching did not resolve the issue. The directories *are* pruned successfully, as per other posters here, but we are still getting these emails nightly. Any other suggestions?
    0

Please sign in to leave a comment.

Didn't find what you were looking for?

New post