Skip to main content

CPANEL-43516 - The system could not prune backups

Comments

22 comments

  • GOT
    Strange, could we see the logs? Not the entire thing, just that one part.
    0
  • Luis Mota
    I think I found the problem "The system could not prune the "[_1]" directory due to an error." Now I need to find "[_1]"" directory -.-' edit:
    0
  • Luis Mota
    I can't find this dir. What is strange is that the error is displayed after the .master.meta sync [2019-05-20 02:40:30 +0100] info [cpbackup_transporter] Upload attempt #1 starting for /backup/2019-05-20/accounts/.master.meta to backup_s3/2019-05-20/accounts/.master.meta for destination: Nameofmybackupdestination [2019-05-20 02:40:30 +0100] info [cpbackup_transporter] Successful incremental transfer of /backup/2019-05-20/accounts/.master.meta to backup_s3/2019-05-20/accounts/.master.meta for destination Nameofmybackupdestination [2019-05-20 02:40:30 +0100] info [cpbackup_transporter] cpbackup_transporter - Checking queue for tasks [2019-05-20 02:40:30 +0100] info [cpbackup_transporter] cpbackup_transporter - Waiting up to 299s for new tasks [2019-05-20 02:40:31 +0100] info [cpbackup_transporter] cpbackup_transporter - Checking queue for tasks [2019-05-20 02:40:31 +0100] info [cpbackup_transporter] cpbackup_transporter - Waiting up to 298s for new tasks [2019-05-20 02:40:32 +0100] info [cpbackup_transporter] cpbackup_transporter - Checking queue for tasks [2019-05-20 02:40:32 +0100] info [cpbackup_transporter] cpbackup_transporter - Waiting up to 297s for new tasks ( ... ) [2019-05-20 02:45:30 +0100] info [cpbackup_transporter] cpbackup_transporter - Waiting up to 0s for new tasks [2019-05-20 02:45:31 +0100] info [cpbackup_transporter] cpbackup_transporter - Checking queue for tasks [2019-05-20 02:45:31 +0100] info [cpbackup_transporter] cpbackup_transporter - Exiting - the queue has been emptied; no more work to do after waiting for 300s [2019-05-20 02:45:31 +0100] info [cpbackup_transporter] cPanel Backup Transporter Queue Daemon is being stopped. [2019-05-20 02:47:46 +0100] info [cpbackup_transporter] Initializing log file [2019-05-20 02:47:46 +0100] info [cpbackup_transporter] cpbackup_transporter - parent starting [2019-05-20 02:47:46 +0100] info [cpbackup_transporter] cpbackup_transporter - child starting [2019-05-20 02:47:46 +0100] info [cpbackup_transporter] cPanel Backup Transporter Queue Daemon started. [2019-05-20 02:47:46 +0100] info [cpbackup_transporter] cpbackup_transporter - started [2019-05-20 02:47:46 +0100] info [cpbackup_transporter] cpbackup_transporter - Checking queue for tasks [2019-05-20 02:47:46 +0100] info [cpbackup_transporter] cpbackup_transporter - Processing next task [2019-05-20 02:47:47 +0100] info [cpbackup_transporter] Instantiating Object [2019-05-20 02:47:47 +0100] info [cpbackup_transporter] Starting a "prune" operation on the "Nameofmybackupdestination" destination ID "usi2ESkroDPNCGkcqDUkoTvT". [2019-05-20 02:47:47 +0100] info [cpbackup_transporter] Base path for destination is backup_s3/ [2019-05-20 02:47:47 +0100] info [cpbackup_transporter] Performing prune operation, retaining 1 items on: Metodo Rsync - para incremental [2019-05-20 02:47:48 +0100] info [cpbackup_transporter] Pruning backup directory: backup_s3/2019-05-19, from Nameofmybackupdestination [2019-05-20 02:54:46 +0100] info [cpbackup_transporter] ERROR: Pruning backup_s3/2019-05-19 from Nameofmybackupdestination: ssh slave failed: timed out [2019-05-20 02:54:46 +0100] info [cpbackup_transporter] The system could not prune the "[_1]"" directory due to an error. [2019-05-20 02:54:46 +0100] info [cpbackup_transporter] Read the go.cpanel.net/directorypruning documentation for solutions to successfully prune the directory. [2019-05-20 02:54:46 +0100] info [cpbackup_transporter] cpbackup_transporter - Checking queue for tasks
    0
  • cPanelMichael
    Hello @Luis Mota, Can you open a
    0
  • Luis Mota
    Good Morning Michael, here it goes 12356745
    0
  • cPanelMichael
    Hello, To update, internal case CPANEL-25619 is open to address an issue where false backup pruning failures are reported when the rm command times out on the remote backup destination. The pruning itself should continue to succeed despite the errors reported in /usr/local/cpanel/logs/cpbackup_transporter.log. I'll monitor the status of this case and update this thread for more information as it becomes available. Thank you.
    0
  • Luis Mota
    I'll apreciate that. Thanks :)
    0
  • cheke
    I Have the same issue, any update of this? thanks
    0
  • cPanelMichael
    I Have the same issue, any update of this?

    Hello :) There's no update to report at this time. I'll continue to monitor the case and update this thread with more information as it becomes available. Note the pruning itself should continue to succeed despite the errors reported in /usr/local/cpanel/logs/cpbackup_transporter.log. Thank you.
    0
  • vf-hostmaster
    I believe I am also experiencing this issue. We get an alert about backup transport errors The log states [2023-06-21 02:32:36 -0500] info [cpbackup_transporter] ERROR: Pruning ******* from *******: ssh slave failed: timed out [2023-06-21 02:32:36 -0500] info [cpbackup_transporter] The system could not prune the "*******" directory due to an error. But the folder it was trying to prune looks like it was removed as desired.
    0
  • FrancXPT
    Aug 2023 Problem still exist without anything to fix it, any update of this? Thanks
    0
  • cPRex Jurassic Moderator
    @FrancXPT - can you post the specific error you're seeing from the log?
    0
  • FrancXPT
    @FrancXPT - can you post the specific error you're seeing from the log?

    All Backups are marked complete, but the Transport Always has an error: Unable to prune transport "XXX.XXXX.com" Error pruning "2023-08-18" from "XXX.XXXX.com": ssh slave failed: timed out The system could not prune the "2023-08-18" directory due to an error. Read the go.cpanel.net/directorypruning documentation for solutions to successfully prune the directory. On inspection, Remote Backups seems fine. I'm using incremental backups and rsync for remote server.
    0
  • cPRex Jurassic Moderator
    Thanks for the additional details. That is definitely the same error. Let me see if the team wants to pick up this case, I know it's a few years old at this point. I'll reply with more details once I have them!
    0
  • cPRex Jurassic Moderator
    I've let the backup team know about the issue, but there isn't any type of ETA for getting this resolved. Since there's only been a few reports of this issue over the last several years, it isn't something critical, and since you know things are working well it's safe to ignore that error.
    0
  • FrancXPT
    I've let the backup team know about the issue, but there isn't any type of ETA for getting this resolved. Since there's only been a few reports of this issue over the last several years, it isn't something critical, and since you know things are working well it's safe to ignore that error.

    Thanks, I hope it gets fixed soon, I'm suggestion to let us specify a bigger timeout than 300sec for big servers this problem is happening on multiple servers when choosing incremental backups. Thanks again for your response and have a good day.
    0
  • nlaruelle
    Thanks, I hope it gets fixed soon, I'm suggestion to let us specify a bigger timeout than 300sec for big servers this problem is happening on multiple servers when choosing incremental backups. Thanks again for your response and have a good day.

    The same report and suggestion apply here. We are pruning over 100GB of data on a SATA destination, but for each of our servers we keep encountering the notorious "Backup transport errors : Unable to prune transport" Error pruning " from " ssh slave failed: timed out" error. This occurs even though we have confirmed that the directory has been well physically removed from the disk after manual verification. The timeout of 300 or 30 seconds appears to be too short for a SATA drive with a large amount of data. This issue is frustrating because, with everyday "Backup transport error," we can't determine quickly when an actual backup failure has occurred for real. If we lose focus on this particular alert: "Backup transport errors," it could take days to the administrator to troubleshoot a serious issue with backup transportation. Do you have an update about this concern ( @cPRex @cPanelMichael ? ), plenty of reports can be found here in the forums since 2018. Thanks in advance!
    0
  • cPRex Jurassic Moderator
    I do see that the case has been closed on our side as "won't fix" as apparently this would take quite a bit of redesigning of the backup tool to resolve. Since things do actually work and there is no data loss, the developers have just opted to not fix this one due to the limited number of reports of the issue. I know that's not an ideal answer, but that's where things are right now.
    0
  • vf-hostmaster
    I do see that the case has been closed on our side as "won't fix" as apparently this would take quite a bit of redesigning of the backup tool to resolve. Since things do actually work and there is no data loss, the developers have just opted to not fix this one due to the limited number of reports of the issue. I know that's not an ideal answer, but that's where things are right now.

    WE have been waiting for this to be addressed... The fact that you are not addressing real issues simply because a lot of noise hasn't been made has us extremely alarmed. Does this mean we need to hire someone to complain on this forum every week?
    0
  • cPRex Jurassic Moderator
    @vf-hostmaster - here is what I've done on my end to see if we can get some progress on this. I've went ahead and created a new case, CPANEL-43516, so our developers are aware people are still interested in this issue. I also added details from both this thread and other Forums posts, as well as data I found while searching through our ticket system's history. I'm hoping this additional information will let the developers properly gauge the number of users that are affected. I'll also update the title of this Forums thread to address the new case number. I'm hoping I'll hear something positive soon as I brought this up with our team today.
    0
  • Matyas Matyas

    Please  allow setting longer transport timeout. The maximum of 300  seconds is not enough to prune a a large directory on SATA drive backup.

    We are getting transport errors even if the transport is correct and the directory gets pruned.

    This is a false alarm and gets us into "The Boy Who Cried Wolf" mode and ignore real transport errors.

    The only workaround is to edit the perl scripts and increase the timeout but those get overwritten at first update.

    Please fix it by allowing a longer transport timeout! Thank you!

    1
  • cPRex Jurassic Moderator

    Matyas Matyas - I've added your concerns to the case!

    0

Please sign in to leave a comment.