WHM 11.52 Transfer Tool
I didn't open a ticket for this because I don't necessarily think anything was functioning improperly. But I'd like to point out how flawed this Transfer Tool stuff is under certain circumstances.
a. Using New Backup scheme on all servers (not legacy)
b. All servers running WHM 11.52
c. All backup drives are single drives
I log into New Server, select Transfer Tool, and connect to Old Server
I told it I do NOT want to transfer bandwidth data
Under "Advanced"
- Compressed Transfers is selected by default
- Use Incremental Backups Speed-up is selected by default
- Use custom account packaging modules from "/var/cpanel/lib/Whostmgr/Pkgacct" is selected
So, I find an account I want to transfer (a 7 GB account, 95% mail)
I start the transfer process
I then go looking around to see what is going on, and from what I can see here is what happens:
a. a copy of either the live account or a copy of an incremental backup is put in:
/backup/incremental/cpmove/cpmove-accountname-split/cpmove-accountname
b. a bunch of tar.gz files are created and put in:
/backup/incremental/cpmove/cpmove-accountname-split
Ok, sounds good so far.
The "live account" is 7.1 GB and the incremental backup is 7.6 GB in size
Now, here is the kicker, the .tar.gz files start to get created. tar(archive).gz(compression). Would you not presume that emails would compress down significantly? Perhaps even 50% or more? I would think so.
I bailed out of the transfer after I saw that it has created cpmove-accountname.tar.gz.part00036. These are 256 MB archive parts.
36 x 256 MB = 9 GB. And it was still going.
1. how was the tar.gz'd files for the account over 9 GB in size (and counting), when the original live uncompressed account was at maximum 7.6 GB?
Something is flawed here. There is no way it should be taking an account (especially one that is predominantly email) and then [supposedly] compressing it into parts, the sum of which is significantly greater than the original live uncompressed account.
2. Can you imagine what this type of activity does when ALL of the back/forth copying and compressing is being done on the single backup drive? You can guess that iowait is pretty high for sure. The building of the compressed data should not be taking place on the same drive that the data-to-be-compressed is on, if at all possible.
Seems to me that everything that was done to "speed up" the process of doing a transfer was not only totally undone but worsened by (a) all activity taking place on the same partition/drive and (b) resultant compressed data being significantly large in size than the original uncompressed data.
Next time I'll flip some switches in the Advanced settings and see if it doesn't play better. A server to server live transfer of the data (over a stream) would have been much much faster than having the server first make a copy of either the live site or the incremental backup and then compressing that [no space saving advantage in my case] _on_the_same_partition.
It took 40 minutes for it to generate the 36 256-MB tar.gz parts and it was still going. It hadn't even gotten to the transfer stage.
Again, something aint right here!
Mike
-
- Use custom account packaging modules from "/var/cpanel/lib/Whostmgr/Pkgacct" is selected
Hello :) Could you elaborate on which custom modules you are utilizing? Thank you.0 -
No I cannot. I am not specifically using any custom modules. That box just happens to be checked by default under Advanced. If there is a custom module actually being used, I wouldn't know what it is. There are no modules in /var/cpanel/perl/Whostmgr/Transfers/Systems . In fact, that directory structure does not even exist. Mike 0 -
Yet another example: - customer has no more than about 750 MB tied to their account - express transfer created 1.31 GB of .tar.gz files and transferred them up to the server So, 2x larger than the site itself (and, supposely compressed) root@blah [/mnt/largebackup/incremental/cpmove/cpmove-customer-split]# ls -alt total 1312628 drwx------ 2 root root 4096 Feb 8 13:07 ./ -rw------- 1 root root 64105264 Feb 8 13:07 cpmove-customer.tar.gz.part00006 -rw------- 1 root root 255997700 Feb 8 13:07 cpmove-customer.tar.gz.part00005 -rw------- 1 root root 256008132 Feb 8 13:06 cpmove-customer.tar.gz.part00004 -rw------- 1 root root 255995941 Feb 8 13:06 cpmove-customer.tar.gz.part00003 -rw------- 1 root root 256004442 Feb 8 13:05 cpmove-customer.tar.gz.part00002 -rw------- 1 root root 255999515 Feb 8 13:04 cpmove-customer.tar.gz.part00001 drwx------ 3 root root 4096 Feb 8 13:03 ../ root@blah [/mnt/largebackup/incremental/cpmove/cpmove-customer-split]# quota customer Disk quotas for user customer (uid 601): Filesystem blocks quota limit grace files quota limit grace /dev/sda7 8 1024000 1024000 3 0 0 /dev/sdb1 748204 1024000 1024000 14282 0 0 /dev/sda5 64 1024000 1024000 22 0 0 root@blah [/mnt/largebackup/incremental/cpmove/cpmove-customer-split]# du -s /home/customer/ 748196 /home/customer/ 0 -
I believe the following feature requests would address your concerns: Immediate Transfer of Remote Backup Archives Prevent WHM Back-Up from Crashing Servers / Using All Disk Space Thank you. 0 -
I believe the following feature requests would address your concerns: Immediate Transfer of Remote Backup Archives Prevent WHM Back-Up from Crashing Servers / Using All Disk Space Thank you.
From what I can see, not at all. My issue wasn't that it left archives on the server after transferring somewhere. My issue is that it is creating multi-part archives that total 2x the space of the original account. They are supposed to be tar.gz (compressed) to begin with. So if an account is being tar.gz'd for transfer and the account is 2 GB in size, then (a) the total parts should at the very least be less than the total size of the live account -- and realistically should be much less since compression is being used. What I'm seeing is that during the preparing of the account for transfer, it creates a multipart tar.gz that [compressed] is twice the size of the original account. Ridiculous. I have to turn off all of the options that are defaulted to On in order for it to do the transfer without spending an insane amount of time archiving/compressing first. I have to turn off using compression and I have to turn off using backups to "speed up" things. It's a combination of the defaulted options using the backup drive for all of the disk activity (thus lots of read/read during the packaging and archiving on the usually unRAIDed backup drive as well as it creating multipart tar.gz files that seem to always equal twice the size of the original live account. Call it what you want, that isn't proper behavior. But I can't spend any more time on it. I just posted to make others aware of what is going on, if they happen to be doing account transfers and think that it's taking an insane amount of time to do a single transfer. Mike0 -
From what I can see, not at all. My issue wasn't that it left archives on the server after transferring somewhere. My issue is that it is creating multi-part archives that total 2x the space of the original account.
Is "rsync" installed on both the source and destination servers? If so, the transfer should automatically use "rsync" to copy over files for the homedir of the account. Thank you.0 -
Is "rsync" installed on both the source and destination servers? If so, the transfer should automatically use "rsync" to copy over files for the homedir of the account. Thank you.
- (group 1) Server 1 = CentOS 5 and WHM 11.52 LTS - (group 2) Servers 2/3/4/5 = CloudLinux 6 and latest Release version of '54' - all servers run exact same backup configuration If I log in to WHM on a group 2 server and initiate an Express Transfer from Server 1, it behaves just as I would expect and is quite speedy. If I log into WHM on any of the group 2 servers and attempt to initiate an Express Transfer from one of the other group 2 servers, that's where the insanity begins. The Transfer tool seems to default with some options checked, such as (a) Compressed Transfers, (b) Use Incremental Backups Speedup and (c) Use custom account packaging modules from "/var/cpanel/lib/Whostmgr/Pkgacct". If I disable all three of these items, then it behaves as I want it to / would expect it to when connecting initiating a transfer from one group 2 server to another group 2 server. Anyway, I'm working around it by disabling those defaulted items. But if I fail to do that, it really becomes irritating very quickly as the disk thrashing on the backup drive starts and the tar.gz parts generated during the transfer seem neverending. (Incidentally, I don't have the time to sit and see just how many tar.gz parts it would make during any transfer -- once it got as far as making enough .tar.gz parts to equal 2x the size of the live account, I gave up. mike0 -
Hi Mike, Would you please open a ticket and post the # so we can look into this? Thanks 0 -
Hi Mike, Would you please open a ticket and post the # so we can look into this? Thanks
Hi Nick. I'll certainly consider it. However, I don't have any accounts scheduled to be migrated to reproduce the problem and right now do not feel like using somebody's account as a guinea pig. When I have an account on the schedule to migrate during the coming week (and I will have some on the schedule) I'll let you guys migrate it and see for yourself what I'm describing. On the other hand, if you guys think you can see what I'm talking about without actually doing an Express Transfer on one of my live accounts, I'll open a ticket earlier. I can't remember if this only occurs during an Express Transfer or if it happens during a regular Transfer. I've been focused on Express Transfers as of late. Mike0 -
Hi Nick. I'll certainly consider it. However, I don't have any accounts scheduled to be migrated to reproduce the problem and right now do not feel like using somebody's account as a guinea pig. When I have an account on the schedule to migrate during the coming week (and I will have some on the schedule) I'll let you guys migrate it and see for yourself what I'm describing.
Ensure you reference this thread when opening the support ticket, and post the ticket number here. This will allow us to follow the progress of the support ticket and update this thread with the outcome. Thank you.0 -
FYI. Just an update on this thread. I may have figured out what was going on -- and if so, it was an oversight on my part. I firewall the WHM ports on the boxes I manage. When I was doing transfers from my older machines to my newest machine, that is when I would see problem that I originally described. It wasn't until this week, when I was doing a transfer and noticed the fleeting red notice come up during transfer, that the new server was unable to access the old server via the WHM API. So, instead it had to resort to using SSH to transfer things to the new server. Like I said, that red notice was fleeting, and I apparently never noticed it before. I suspect my problem all along (no streaming transfers) was related to this. As soon as I allowed access to the WHM port on the server(s) I was transferring from, to the servers I was transferring to, like magic things started working like I expected them to. Transfers were streaming. There was no archiving occurring beforehand. I didn't have to go into Advanced settings and change anything. So I'm pretty sure this was all my fault because I wasn't paying attention, and at this time I don't think I have an issue with the transfer functions. mike 0
Please sign in to leave a comment.
Comments
11 comments