Error Pruning - ssh slave failed: timed out
Hi Michael,
I am still getting this most nights. The problem is when we log tickets about this we don't seem to get the same level of interest/investigation than we sometimes do on the forum. My last ticket about this issue was 9366189. Feel free to take a look but they said it was my end. The problem is I was using an Ubuntu server for SFTP and was told I should use CentOS, so I installed a fresh CentOS box and still have the issue. Like others have said, I am not using SSD's for the backup server, but there is a level of write-cache.
I am getting tired of logging tickets for this for first line support to shut it down saying it is my issue. I think there are sufficient people reporting this for it to be considered a real issue now.
BTW, it happened last night on build 70.0.31. I was hoping your backup changes might have fixed it.
-
Hi @sp3ctre69, Let's start from the beginning to see if we can determine what exactly is causing the backup failures. First, could you run the commands referenced below on the affected cPanel server and let us know the output? cat /usr/local/cpanel/version cat /var/cpanel/backups/config cat /var/cpanel/backups/$$$.backup_destination
Replace "/$$$.backup_destination" with the actual file name associated with the remote backup destination on your system, and ensure to exclude the hostname, ID, and password from the output. Additionally, can you provide us with the error messages that appear in /usr/local/cpanel/logs/cpbackup_transporter.log from the most recent backup transport? Thank you.0 -
Thanks.. here you go: cat /usr/local/cpanel/version 11.70.0.31
cat /var/cpanel/backups/config --- BACKUPACCTS: 'yes' BACKUPBWDATA: 'yes' BACKUPDAYS: 0,1,2,3,4,5,6 BACKUPDIR: /backup BACKUPENABLE: 'yes' BACKUPFILES: 'yes' BACKUPLOGS: 'no' BACKUPMOUNT: 'no' BACKUPSUSPENDEDACCTS: 'no' BACKUPTYPE: incremental BACKUP_DAILY_ENABLE: 'yes' BACKUP_DAILY_RETENTION: 5 BACKUP_MONTHLY_DATES: 1,15 BACKUP_MONTHLY_ENABLE: 'yes' BACKUP_MONTHLY_RETENTION: 1 BACKUP_WEEKLY_DAY: 0 BACKUP_WEEKLY_ENABLE: 'no' BACKUP_WEEKLY_RETENTION: 4 CHECK_MIN_FREE_SPACE: 1 ERRORTHRESHHOLD: 3 FORCE_PRUNE_DAILY: 0 FORCE_PRUNE_MONTHLY: 0 FORCE_PRUNE_WEEKLY: 0 GZIPRSYNCOPTS: --rsyncable KEEPLOCAL: 1 LINKDEST: 0 LOCALZONESONLY: 'no' MAXIMUM_RESTORE_TIMEOUT: 21600 MAXIMUM_TIMEOUT: 14400 MIN_FREE_SPACE: 5 MIN_FREE_SPACE_UNIT: percent MYSQLBACKUP: accounts POSTBACKUP: 'no' PREBACKUP: -1 PSQLBACKUP: 'no'
cat SFTPrsync_UID_**ID**.backup_destination --- authtype: password disabled: 0 host: ***myhostname*** id: **ID** name: SFTPrsync password: ***mypassword*** path: /home/****/backup/***servername***/ port: 22 timeout: 300 type: Rsync upload_system_backup: 1 username: **myusername**0 -
... and the log file... l08NIt4V[2018-05-03 02:36:39 +0100] info [cpbackup_transporter] Pruning backup directory: home/**path**/2018-04-28, from SFTPrsync [2018-05-03 02:42:44 +0100] warn [cpbackup_transporter] Error pruning home/**path**/2018-04-28 from SFTPrsync: ssh slave failed: timed out at /usr/local/cpanel/Cpanel/LoggerAdapter.pm line 27. Cpanel::LoggerAdapter::warn(Cpanel::LoggerAdapter=HASH(0x259af70), "Error pruning home/**path**/2018-04-28 from SF"...) called at /usr/local/cpanel/Cpanel/Backup/Queue.pm line 724 Cpanel::Backup::Queue::transport_backup::attempt_to_prune_destination(Cpanel::Backup::Queue::transport_backup=HASH(0x2573190), Cpanel::Transport::Files::Rsync=HASH(0x37226b8), 5, undef, Cpanel::LoggerAdapter=HASH(0x259af70)) called at /usr/local/cpanel/Cpanel/Backup/Queue.pm line 233 Cpanel::Backup::Queue::transport_backup::process_task(Cpanel::Backup::Queue::transport_backup=HASH(0x2573190), cPanel::TaskQueue::Task=HASH(0x37226e8), Cpanel::LoggerAdapter=HASH(0x259af70)) called at /usr/local/cpanel/3rdparty/perl/526/lib64/perl5/cpanel_lib/cPanel/TaskQueue.pm line 624 eval {...} called at /usr/local/cpanel/3rdparty/perl/526/lib64/perl5/cpanel_lib/cPanel/TaskQueue.pm line 627 cPanel::TaskQueue::__ANON__() called at /usr/local/cpanel/3rdparty/perl/526/lib64/perl5/cpanel_lib/cPanel/StateFile.pm line 223 eval {...} called at /usr/local/cpanel/3rdparty/perl/526/lib64/perl5/cpanel_lib/cPanel/StateFile.pm line 223 cPanel::StateFile::Guard::call_unlocked(cPanel::StateFile::Guard=HASH(0x2f01af8), CODE(0x33b9af8)) called at /usr/local/cpanel/3rdparty/perl/526/lib64/perl5/cpanel_lib/cPanel/TaskQueue.pm line 632 cPanel::TaskQueue::process_next_task(cPanel::TaskQueue=HASH(0x2591d20)) called at /usr/local/cpanel/bin/cpbackup_transporter line 163 eval {...} called at /usr/local/cpanel/bin/cpbackup_transporter line 161 [2018-05-03 02:42:44 +0100] info [cpbackup_transporter] warn [cpbackup_transporter] Error pruning home/**path**/2018-04-28 from SFTPrsync: ssh slave failed: timed out0 -
Hello @sp3ctre69, I suspect this is happening because a file transported during the backup process isn't writable by the user configured to transport the files to your remote rsync destination. We do have a couple of open internal cases related to this issue, but first let's make sure that's what is actually causing the issue on your system. Can you confirm if the username configured as part of your Rsync backup destination is a non-root user? If so, do you have root access to the rsync destination, or is your access limited to a non-privileged user? Thank you. 0 -
Hello @sp3ctre69, I suspect this is happening because a file transported during the backup process isn't writable by the user configured to transport the files to your remote rsync destination. We do have a couple of open internal cases related to this issue, but first let's make sure that's what is actually causing the issue on your system. Can you confirm if the username configured as part of your Rsync backup destination is a non-root user? If so, do you have root access to the rsync destination, or is your access limited to a non-privileged user? Thank you.
Hi, Yes, the user is non-root, but I do have full root access to the server.0 -
Yes, the user is non-root, but I do have full root access to the server.
Are you able to temporarily modify the Rsync backup destination so that it uses root instead of a non-root user? If the issue no longer occurs after switching to the root user, then that would confirm the issue is related to one of the cases referenced in my previous response. Thank you.0 -
weird, I can logon as root but when I use root in the backup settings it won't validate. 0 -
weird, I can logon as root but when I use root in the backup settings it won't validate.
Hi @sp3ctre69, Have you tried running a rsync command from the command line of the cPanel server to the backup destination to verify it works? Does it validate if you setup key authentication for root on the remote backup destination and then set that as the authentication type for the Rsync backup destination? Thank you.0 -
Hi @sp3ctre69, Have you tried running a rsync command from the command line of the cPanel server to the backup destination to verify it works? Does it validate if you setup key authentication for root on the remote backup destination and then set that as the authentication type for the Rsync backup destination? Thank you.
Not tried key based yet, but it looks like it does connect ok, it just fails to write the test file. If I change the destination location to /tmp/ it validates fine. Not sure why root would not be able to write the file though. Error is "Could not upload test file: child exited with code 1" If I rsync over ssh from the server to the backup location specified in this setup I can transfer files fine as root.0 -
Not tried key based yet, but it looks like it does connect ok, it just fails to write the test file. If I change the destination location to /tmp/ it validates fine. Not sure why root would not be able to write the file though. Error is "Could not upload test file: child exited with code 1"
Hello @sp3ctre69, If you create a new Rsync destination with these settings instead of editing the existing destination, does it make a difference? Thank you.0 -
Hello @sp3ctre69, If you create a new Rsync destination with these settings instead of editing the existing destination, does it make a difference? Thank you.
No, same error when I create a new destination0 -
Hello @sp3ctre69, The backup directory on remote systems is always created relative to the home directory of the logged in user on the remote system. Thus, you'll need to use the following value for the backup directory when configuring the Rsync destination since the user was changed to root: /../home/backups/servername/
I used this value on a test system and it worked well. Let me know if this helps. Thank you.0 -
Hi @sp3ctre69, Have you tried running a rsync command from the command line of the cPanel server to the backup destination to verify it works? Does it validate if you setup key authentication for root on the remote backup destination and then set that as the authentication type for the Rsync backup destination? Thank you.
Thanks, looks like it validated ok now. Will see how it runs tonight.0 -
Thanks, looks like it validated ok now. Will see how it runs tonight.
Hello @sp3ctre69, Let us know how it goes. Thanks!0 -
Hello @sp3ctre69, Let us know how it goes. Thanks!
Hi, It worked ok last night, so far so good, although it didn"t used to fail every time so will see how it goes in the next day or two.0 -
Hello, For reference, the internal case numbers associated with this issue are CPANEL-18049 and CPANEL-17941. CPANEL-17941 is open to improve the logging so that this issue is better detected during troubleshooting. CPANEL-18049 is open to determine if this is preventable by checking for this scenario during the destination validation process or by modifying permission values. I'll monitor both cases and update this thread with more information on the status of each case as it becomes available. Thank you. Update: Internal case CPANEL-17941 is planned for inclusion with cPanel & WHM version 76. 0 -
Seems to have worked every night since changing over to root. Cheers, Jim 0 -
Hi Jim, I'm glad to see it's working well thus far. I'll keep an eye on those cases and update this thread once a solution is published. Thank you. 0 -
Looks like I'm getting quite a few other errors in the transporter now: rsync failed: command exited with code 12: error in rsync protocol data stream0 -
rsync failed: command exited with code 12: error in rsync protocol data stream
Hello @sp3ctre69, I've seen this particular error message in cases where the remote destination is out of available disk space or inodes. Can you check to see if that's the case on your remote destination? Note we have an internal case open (HB-2414) to improve the transport logging to better reflect the actual reason for error messages like the one you noted. Thank you.0 -
Ah, rookie error, that was exactly it.... working nicely again now. 0 -
Ah, rookie error, that was exactly it.... working nicely again now.
I'm glad to see it's working well again. Thanks!0 -
Thanks for setting me straight, have a good weekend! :) 0 -
We get this every night on every server, no inode issues, plenty of space just cpanel started doing it a few months ago. Backups generally seem fine though 0 -
We get this every night on every server, no inode issues, plenty of space just cpanel started doing it a few months ago. Backups generally seem fine though
Hello @uk01, Can you verify which specific error message you see every night? Also, could you run the commands referenced below on the affected cPanel server and let us know the output?cat /usr/local/cpanel/version cat /var/cpanel/backups/config cat /var/cpanel/backups/$$$.backup_destination
Replace "/$$$.backup_destination" with the actual file name associated with the remote backup destination on your system, and ensure to exclude the hostname, ID, and password from the output. Additionally, can you provide us with the error messages that appear in /usr/local/cpanel/logs/cpbackup_transporter.log from the most recent backup transport? Thank you.0 -
Hi the message is below, we"re on the latest v68, pending upgrade to v70 I"ll check the commands you sent. Transport errors encountered. The system encountered errors during transport of the backup files. Below is a preview of the attached log file. Preview of transport errors log: Unable to prune transport "time4backups" Error pruning "home/ssd0-incremental/2018-05-04" from "time4backups": ssh slave failed: timed out This notice is the result of a request from "cpbackup". 0 -
I"ll check the commands you sent.
Thanks, I look forward to your response.0
Please sign in to leave a comment.
Comments
27 comments