Backups hanging
I've just realised that the automated back I've set up in WHM (11.48.4 build 4, CENTOS 6.6 x86_64) hasn't been running for the last month (yikes).
This appears to be caused by a backup process that had hung and therefore (presumably) no other backup process would run.
I've just manually tried running the backup script (/usr/local/cpanel/bin/backup) and it runs, and then hangs while creating the archive for an account:
[2015-05-30 18:38:39 +0100] Copying cpuser file.......[2015-05-30 18:38:39 +0100] Done
[2015-05-30 18:38:39 +0100] Copying crontab file.......[2015-05-30 18:38:39 +0100] Done
[2015-05-30 18:38:39 +0100] Copying quota info.......[2015-05-30 18:38:39 +0100] Done
[2015-05-30 18:38:39 +0100] Storing Subdomains....
[2015-05-30 18:38:39 +0100] Done
[2015-05-30 18:38:39 +0100] Storing Parked Domains....
[2015-05-30 18:38:39 +0100] Done
[2015-05-30 18:38:39 +0100] Storing Addon Domains....
[2015-05-30 18:38:39 +0100] Done
[2015-05-30 18:38:39 +0100] Storing SSL domain......[2015-05-30 18:38:39 +0100] Done
[2015-05-30 18:38:39 +0100] Copying password.......[2015-05-30 18:38:39 +0100] Done
[2015-05-30 18:38:39 +0100] Copying shell.......[2015-05-30 18:38:39 +0100] Done
[2015-05-30 18:38:39 +0100] Creating Archive ....................................................................................
................................................................................
A few more lines of dots and then nothing. strace on the backup process unfortunately shows very little of use:
Process 15233 attached - interrupt to quit
wait4(16698,
So I guess the most important question is, how can I complete a backup?!
Secondary question - should I not have received a warning that backups were not running due to an existing backup process?
Allan
-
A little update on this - I've disabled the backups for the account that the hang appears to occur on and the back now completes for all other accounts. But of course I have one account that it now not being backed up due to this issue. Does anyone know what might be causing this and how to resolve it? Allan 0 -
Hello, How large is the account that fails to backup? Have you tried manually generating a backup for this account via "/scripts/pkgacct"? Thank you. 0 -
Thanks for your reply. The account causing the problem is 767MB is size. Running `/scripts/pkgacct` for that account also hangs while "Creating Archive". 0 -
Do you have any process resource monitors installed that could be preventing the backup from completing? How's your resource usage during the backup? Are you reaching any memory limits, or VPS resource limits (if it's a VPS)? Thank you. 0 -
From `top` at the point when the hang occurs: top - 14:08:14 up 382 days, 4:35, 3 users, load average: 1.74, 1.15, 0.66 Tasks: 379 total, 2 running, 377 sleeping, 0 stopped, 0 zombie Cpu(s): 5.2%us, 2.3%sy, 0.0%ni, 87.4%id, 5.0%wa, 0.0%hi, 0.1%si, 0.0%st Mem: 32836812k total, 27873704k used, 4963108k free, 711472k buffers Swap: 16490488k total, 685112k used, 15805376k free, 21474548k cached During the "Creating Archive" step up until that point gzip is using 100% CPU. Then it hangs and gzip disappears from the top list. The backup process is still running: # ps aux | grep pkg root 4248 2.7 0.1 265344 56704 pts/1 S+ 14:05 0:07 pkgacct - myAccount - av: 3 myAccount 4572 1.4 0.1 265344 54288 pts/1 S+ 14:05 0:03 pkgacct - myAccount - av: 3 root 5145 0.0 0.0 103244 892 pts/2 S+ 14:09 0:00 grep pkg gzip also appears to still be running # ps aux | grep gzip root 4571 39.2 0.0 4300 804 pts/1 S+ 14:05 2:03 /usr/bin/gzip -6 root 5250 0.0 0.0 103244 892 pts/2 S+ 14:11 0:00 grep gzip And there is loads of disk space available (no disk has above 50% usage). Allan 0 -
Feel free to open a support ticket using the link in my signature so we can take a closer look. You can post the ticket number here so we can update this thread with the outcome. Thank you. 0 -
Thank you - ticket #6615903. 0 -
For anyone else who comes across this, the issue was a FIFO file that had been created by `screen`. The gzip process would just hang on that. A fix is apparently in place and coming in a future release according to the ticket. Until then, removing the FIFO or adding it to the backup ignore files will do the job. Allan 0 -
Hello, Internal case number 185949 is open to address an issue where pkgacct hangs on pipe files. There's currently no time frame available for a resolution, but you can monitor our Thank you. 0 -
Ah - thanks you for the clarification. I misunderstood the reply in the ticket in that case. Out of interest, this issue caused my backs to not run for about a month - should there be anything that monitors that and would notify me (other than me realising that the backups were not running - which I clean didn't!). Allan 0 -
Out of interest, this issue caused my backs to not run for about a month - should there be anything that monitors that and would notify me (other than me realising that the backups were not running - which I clean didn't!).
You should already receive a notification if the backup fails. It looks like the backup process actually hung in this case due to the internal case that's open, and thus no notification was sent. Thank you.0 -
This is obviously a frightening bug as it means a FIFO created in one account can stop backups across an *entire server*. I'm glad a fix is going through the process. The FIFO we had one one of our servers was called tmp/spamhack.fifo so another helpful trick might be to run this once to exclude all files with .fifo on the end. Limited, but it helps a little. echo "*.fifo" >> /etc/cpbackup-exclude.conf0
Please sign in to leave a comment.
Comments
13 comments