Issue with cpbackup causing continual high loads

simonautomatic

• September 09, 2016 06:52

I'm seeing a new issue in WHM58.0 build 28 where load averages rise to between 7-10 or higher (I have 4 Cores) and stay consistently over 5 forever, once cpbackup has started. I tested this by rebooting the server early one morning (07:00). Load averages returned to around 0.6 - 1, which is normal. Then according analysis from sar, at 01:10 the load averages had immediately risen to 4-5, and then carried on rising to 7-8 where they mostly plateaued. Checking crontab revealed that the only task which commences at 01:00 is cpbackup. Sure enough I found a few cpbackup tasks still running at around 11:00 / 12:00. Normally backups should have completed by around 08:30 at the latest. I killed the cpbackup processes, but this still doesn't return load average to anything normal. I'm at a loss. This looks like it might be IO related, as processor usage doesn't look dramatic at all. But what on earth is hanging on to so many IO processes, even after cpbackup has been killed off? And why is it suddenly having such a struggle to complete? I'd actually reduced the number of accounts and filesize of stuff that needs to be backed up. ------------- CENTOS 6.8 x86_64 standard 8GB Ram / 4 cores

• 

  Infopro

  • September 09, 2016 12:34

  Is the system sending you an email about backups completing with log attached? You might want to review that log for clues as to whats holding things up.

  0
• 

  simonautomatic

  • September 09, 2016 13:20

  Normally it would, but now because the load is so high, the backups never complete and thus no report is emailed. I guess I can check the log file, it must be on disk somewhere, right?

  0
• 

  simonautomatic

  • September 09, 2016 13:36

  So looking at /usr/local/cpanel/logs/cpbackup/.... doesn't really give any clues. I see the backups running, and gradually pausing more and more, waiting for the server load to reduce below 3.5 Also from time to time I see: Load watching suspended due to SIGUSR1

  0
• 

  Infopro

  • September 09, 2016 13:58

  Going by your first post, is it normal for your backups to take 7 hours? How big are your largest accounts?

  0
• 

  simonautomatic

  • September 09, 2016 15:13

  Hi, Yes it's normal as there are 50-60 accounts, and the largest is 30GB. However what has been happening for the past few days (since this problem with high loads began) is that the backup never completes. BY the morning, loads are way up (7 or 8, ie: double the number of cores) so the backup sits there doing nothing as the load never goes down. Please note that it is *not* other activity on the server that is causing this high load. I can see that quite clearly by a) the fact that processor usage is not abnormally high and b) after a reboot, load returned to its usual averages (<1) until cpbackup started up again.

  0
• 

  Infopro

  • September 09, 2016 15:36

  If it was me, I might exclude (from within Backup Settings) that largest account from being backed up temporarily, to see if the backup completes successfully without it. You can force a manual backup with this command:
  /usr/local/cpanel/bin/backup --force
  

  0
• 

  simonautomatic

  • September 09, 2016 15:43

  I could try that, but I think you're misunderstanding the issue here. The issue is whay on earth am I seeing such huge load averages, continually, after cpbackup has started running. They never go down, the only solution I found is to reboot the server.

  0
• 

  Infopro

  • September 09, 2016 16:03

  Understood. Would you still see high load without that 30GB account in the mix though? Might be worth trying to find out.

  0

Please sign in to leave a comment.