Backup transport errors. Transfer aborted. No space left on device?

We received a notice "Backup transport errors" with the log saying "452 Transfer aborted. No space left on device" but the destination still has plenty of space left (35% free space left). The account being backed up is just less than 1GB but the destination has 160GB free. The same backup destination is used by a different cpanel server and that backs up accounts with no issue. Any ideas what could cause this? Thanks. This happens to several accounts of the server, here's snippet of one of the errors: [QUOTE]Unable to send "/backup/weekly/2018-07-14/accounts/xxxxx.tar.gz" to destination "xxxxx"
From/usr/local/cpanel/logs/cpbackup_transporter.log:

[2018-07-14 15:18:32 +0800] info [cpbackup_transporter] Instantiating Object
[2018-07-14 15:18:33 +0800] info [cpbackup_transporter] Net::FTP=GLOB(0x28a6580) <<<  220 ProFTPD 1.3.3a Server (xxxxx) [178.32.222.126]
[2018-07-14 15:18:33 +0800] info [cpbackup_transporter] Net::FTP=GLOB(0x28a6580) >>>  USER xxxxx
[2018-07-14 15:18:33 +0800] info [cpbackup_transporter] Net::FTP=GLOB(0x28a6580) <<<  331 Password required for xxxxx
[2018-07-14 15:18:33 +0800] info [cpbackup_transporter] Net::FTP=GLOB(0x28a6580) >>>  PASS ....
[2018-07-14 15:18:33 +0800] info [cpbackup_transporter] Net::FTP=GLOB(0x28a6580) <<<  230 User xxxxx logged in
[2018-07-14 15:18:33 +0800] info [cpbackup_transporter] Net::FTP=GLOB(0x28a6580) >>>  TYPE I
[2018-07-14 15:18:33 +0800] info [cpbackup_transporter] Net::FTP=GLOB(0x28a6580) <<<  200 Type set to I
[2018-07-14 15:18:33 +0800] info [cpbackup_transporter] Starting a "copy" operation on the "xxxxx" destination ID "xxxxx".
[2018-07-14 15:18:33 +0800] info [cpbackup_transporter] Base path for destination is /
[2018-07-14 15:18:33 +0800] info [cpbackup_transporter] Validating destination path /weekly/2018-07-14/accounts/
[2018-07-14 15:18:33 +0800] info [cpbackup_transporter] Net::FTP=GLOB(0x28a6580) >>>  PASV
[2018-07-14 15:18:33 +0800] info [cpbackup_transporter] Net::FTP=GLOB(0x28a6580) <<<  227 Entering Passive Mode (178,32,222,126,120,76).
[2018-07-14 15:18:33 +0800] info [cpbackup_transporter] Net::FTP=GLOB(0x28a6580) >>>  LIST /weekly/2018-07-14/accounts/
[2018-07-14 15:18:34 +0800] info [cpbackup_transporter] Net::FTP=GLOB(0x28a6580) <<<  150 Opening BINARY mode data connection for file list
[2018-07-14 15:18:34 +0800] info [cpbackup_transporter] Net::FTP=GLOB(0x28a6580) <<<  226 Transfer complete
[2018-07-14 15:18:34 +0800] info [cpbackup_transporter] Path exists
[2018-07-14 15:18:34 +0800] info [cpbackup_transporter] Uploading account backup /backup/weekly/2018-07-14/accounts/xxxxx.tar.gz to /weekly/2018-07-14/accounts/xxxxx.tar.gz (from weekly/2018-07-14/accounts/xxxxx.tar.$
[2018-07-14 15:18:34 +0800] info [cpbackup_transporter] Attempting to upload /backup/weekly/2018-07-14/accounts/xxxxx.tar.gz to /weekly/2018-07-14/accounts/xxxxx.tar.gz for destination:  xxxxx
[2018-07-14 15:18:34 +0800] info [cpbackup_transporter] Upload attempt #1 starting for /backup/weekly/2018-07-14/accounts/xxxxx.tar.gz to /weekly/2018-07-14/accounts/xxxxx.tar.gz for destination:  xxxxx
[2018-07-14 15:18:34 +0800] info [cpbackup_transporter] Net::FTP=GLOB(0x28a6580) >>>  PASV
[2018-07-14 15:18:35 +0800] info [cpbackup_transporter] Net::FTP=GLOB(0x28a6580) <<<  227 Entering Passive Mode (178,32,222,126,124,72).
[2018-07-14 15:18:35 +0800] info [cpbackup_transporter] Net::FTP=GLOB(0x28a6580) >>>  STOR /weekly/2018-07-14/accounts/xxxxx.tar.gz
[2018-07-14 15:18:35 +0800] info [cpbackup_transporter] Net::FTP=GLOB(0x28a6580) <<<  150 Opening BINARY mode data connection for /weekly/2018-07-14/accounts/xxxxx.tar.gz
[2018-07-14 15:18:35 +0800] info [cpbackup_transporter] Net::FTP=GLOB(0x28a6580) >>>  "ABOR
[2018-07-14 15:18:35 +0800] info [cpbackup_transporter] Net::FTP=GLOB(0x28a6580) <<<  452 Transfer aborted. No space left on device
[2018-07-14 15:18:35 +0800] info [cpbackup_transporter] Net::FTP=GLOB(0x28a6580) <<<  226 Abort successful
[2018-07-14 15:18:35 +0800] info [cpbackup_transporter] put( /backup/weekly/2018-07-14/accounts/xxxxx.tar.gz, /weekly/2018-07-14/accounts/xxxxx.tar.gz ) FAILED
[2018-07-14 15:18:35 +0800] info [cpbackup_transporter] Abort successful

[2018-07-14 15:18:35 +0800] warn [cpbackup_transporter] Upload attempt failed: Abort successful
 at /usr/local/cpanel/Cpanel/LoggerAdapter.pm line 27.
        Cpanel::LoggerAdapter::warn(Cpanel::LoggerAdapter=HASH(0x1d74078), "Upload attempt failed: Abort successful\x{a}") called at /usr/local/cpanel/Cpanel/Backup/Queue.pm line 609
        Cpanel::Backup::Queue::transport_backup::attempt_to_upload_file(Cpanel::Backup::Queue::transport_backup=HASH(0x1bdd190), Cpanel::Transport::Files::FTP=HASH(0x1bfa9b0), "/backup/weekly/2018-07-14/accounts/xxxxx.tar.$
        Cpanel::Backup::Queue::transport_backup::process_task(Cpanel::Backup::Queue::transport_backup=HASH(0x1bdd190), cPanel::TaskQueue::Task=HASH(0x30845d8), Cpanel::LoggerAdapter=HASH(0x1d74078)) called at /usr/local/cpane$
        eval {...} called at /usr/local/cpanel/3rdparty/perl/526/lib64/perl5/cpanel_lib/cPanel/TaskQueue.pm line 627
        cPanel::TaskQueue::__ANON__() called at /usr/local/cpanel/3rdparty/perl/526/lib64/perl5/cpanel_lib/cPanel/StateFile.pm line 223
        eval {...} called at /usr/local/cpanel/3rdparty/perl/526/lib64/perl5/cpanel_lib/cPanel/StateFile.pm line 223
        cPanel::StateFile::Guard::call_unlocked(cPanel::StateFile::Guard=HASH(0x1bfa3e0), CODE(0x2899880)) called at /usr/local/cpanel/3rdparty/perl/526/lib64/perl5/cpanel_lib/cPanel/TaskQueue.pm line 632
        cPanel::TaskQueue::process_next_task(cPanel::TaskQueue=HASH(0x1bfa2d8)) called at /usr/local/cpanel/bin/cpbackup_transporter line 167
        eval {...} called at /usr/local/cpanel/bin/cpbackup_transporter line 165
[2018-07-14 15:18:35 +0800] info [cpbackup_transporter] warn [cpbackup_transporter] Upload attempt failed: Abort successful

[2018-07-14 15:18:36 +0800] info [cpbackup_transporter] Net::FTP=GLOB(0x28a5fe0) <<<  220 ProFTPD 1.3.3a Server (xxxxx) [178.32.222.126]
[2018-07-14 15:18:36 +0800] info [cpbackup_transporter] Net::FTP=GLOB(0x28a5fe0) >>>  USER xxxxx
[2018-07-14 15:18:36 +0800] info [cpbackup_transporter] Net::FTP=GLOB(0x28a5fe0) <<<  331 Password required for xxxxx
[2018-07-14 15:18:36 +0800] info [cpbackup_transporter] Net::FTP=GLOB(0x28a5fe0) >>>  PASS ....
[2018-07-14 15:18:36 +0800] info [cpbackup_transporter] Net::FTP=GLOB(0x28a5fe0) <<<  230 User xxxxx logged in
[2018-07-14 15:18:36 +0800] info [cpbackup_transporter] Net::FTP=GLOB(0x28a5fe0) >>>  TYPE I
[2018-07-14 15:18:36 +0800] info [cpbackup_transporter] Net::FTP=GLOB(0x28a5fe0) <<<  200 Type set to I
[2018-07-14 15:18:36 +0800] info [cpbackup_transporter] Upload attempt #2 starting for /backup/weekly/2018-07-14/accounts/xxxxx.tar.gz to /weekly/2018-07-14/accounts/xxxxx.tar.gz for destination:  xxxxx
[2018-07-14 15:18:36 +0800] info [cpbackup_transporter] Net::FTP=GLOB(0x28a5fe0) >>>  PASV
[2018-07-14 15:18:36 +0800] info [cpbackup_transporter] Net::FTP=GLOB(0x28a5fe0) <<<  227 Entering Passive Mode (178,32,222,126,124,18).
[2018-07-14 15:18:36 +0800] info [cpbackup_transporter] Net::FTP=GLOB(0x28a5fe0) >>>  STOR /weekly/2018-07-14/accounts/xxxxx.tar.gz
[2018-07-14 15:18:37 +0800] info [cpbackup_transporter] Net::FTP=GLOB(0x28a5fe0) <<<  150 Opening BINARY mode data connection for /weekly/2018-07-14/accounts/xxxxx.tar.gz
[2018-07-14 15:18:37 +0800] info [cpbackup_transporter] Net::FTP=GLOB(0x28a5fe0) >>>  "ABOR
[2018-07-14 15:18:37 +0800] info [cpbackup_transporter] Net::FTP=GLOB(0x28a5fe0) <<<  452 Transfer aborted. No space left on device
[2018-07-14 15:18:37 +0800] info [cpbackup_transporter] Net::FTP=GLOB(0x28a5fe0) <<<  226 Abort successful
[2018-07-14 15:18:37 +0800] info [cpbackup_transporter] put( /backup/weekly/2018-07-14/accounts/xxxxx.tar.gz, /weekly/2018-07-14/accounts/xxxxx.tar.gz ) FAILED
[2018-07-14 15:18:37 +0800] info [cpbackup_transporter] Abort successful

[2018-07-14 15:18:37 +0800] warn [cpbackup_transporter] Upload attempt failed: Abort successful
 at /usr/local/cpanel/Cpanel/LoggerAdapter.pm line 27.
        Cpanel::LoggerAdapter::warn(Cpanel::LoggerAdapter=HASH(0x1d74078), "Upload attempt failed: Abort successful\x{a}") called at /usr/local/cpanel/Cpanel/Backup/Queue.pm line 609
        Cpanel::Backup::Queue::transport_backup::attempt_to_upload_file(Cpanel::Backup::Queue::transport_backup=HASH(0x1bdd190), Cpanel::Transport::Files::FTP=HASH(0x1bfa9b0), "/backup/weekly/2018-07-14/accounts/xxxxx.tar.$
        Cpanel::Backup::Queue::transport_backup::process_task(Cpanel::Backup::Queue::transport_backup=HASH(0x1bdd190), cPanel::TaskQueue::Task=HASH(0x30845d8), Cpanel::LoggerAdapter=HASH(0x1d74078)) called at /usr/local/cpane$
        eval {...} called at /usr/local/cpanel/3rdparty/perl/526/lib64/perl5/cpanel_lib/cPanel/TaskQueue.pm line 627
        cPanel::TaskQueue::__ANON__() called at /usr/local/cpanel/3rdparty/perl/526/lib64/perl5/cpanel_lib/cPanel/StateFile.pm line 223
        eval {...} called at /usr/local/cpanel/3rdparty/perl/526/lib64/perl5/cpanel_lib/cPanel/StateFile.pm line 223
        cPanel::StateFile::Guard::call_unlocked(cPanel::StateFile::Guard=HASH(0x1bfa3e0), CODE(0x2899880)) called at /usr/local/cpanel/3rdparty/perl/526/lib64/perl5/cpanel_lib/cPanel/TaskQueue.pm line 632
        cPanel::TaskQueue::process_next_task(cPanel::TaskQueue=HASH(0x1bfa2d8)) called at /usr/local/cpanel/bin/cpbackup_transporter line 167
        eval {...} called at /usr/local/cpanel/bin/cpbackup_transporter line 165
[2018-07-14 15:18:39 +0800] info [cpbackup_transporter] warn [cpbackup_transporter] Upload attempt failed: Abort successful

24x7server

July 14, 2018 15:51

Hi, No space left on device clearly means that the disk space is the issue. #df -h #df -hi Can you provide the output of the above 2 commands here..

kabatak

July 15, 2018 09:22

Hello, here's the output:

root@xxxxx:~# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda1             461G  278G  160G  64% /
tmpfs                 991M     0  991M   0% /lib/init/rw
udev                   10M  148K  9.9M   2% /dev
tmpfs                 991M     0  991M   0% /dev/shm
root@xxxxx:~# df -hi
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/sda1                29M     75K     29M    1% /
tmpfs                   248K       5    248K    1% /lib/init/rw
udev                    248K    2.2K    246K    1% /dev
tmpfs                   248K       1    248K    1% /dev/shm

cPanelLauren

July 16, 2018 16:42

Hi @kabatak Is there any quotas in place on the destination server? If you have root access you can run sun something like: repquota -a
Also, is this occurring for all accounts or just one? Thanks!

July 16, 2018 17:41

I got "-bash: repquota: command not found" error running that command. This is Debian btw. The "No space left on device" error appears to be happening on many accounts but not all of it.

July 16, 2018 17:50

Hi @kabatak The error does clearly sound like it's a space issue though I can see that it seems you have enough space. To look into this further could you please open a ticket using the link in my signature? Once open please reply with the ticket ID so that we can follow up here with the outcome of the ticket. Thanks!

July 16, 2018 18:16

@cPanelLauren. I created a ticket 9924671. Thanks!

July 16, 2018 18:26

Thanks @kabatak, I found your ticket and I'm now following it, I also noted it with the output of df you provided earlier. I'll update here when there's new information to report. Thanks!

July 17, 2018 13:07

Hi @kabatak We looked into this issue and it does indeed appear that the backup server was running out of space. The issue is that the backups that are transported are almost the same size as the available space in the destination server - The scheduled backups generated are 150GB and 158GB:

[05:55:38 server root@9924671 ~]cPs# du -shx /backup/weekly/2018-07-* 158G    /backup/weekly/2018-07-07 150G    /backup/weekly/2018-07-14

Because the pruning operation takes place after the successful transfer the server was indeed running out of space. Please let us know if you have any questions or concerns in regard to this! Thanks! [COLOR=rgb(0, 0, 0)]

Backup transport errors. Transfer aborted. No space left on device?

Comments

Didn't find what you were looking for?