Server “Freezes” after Backup Routine
Every time system run backup routine server “freezes”
Backup is setup for once a week (Sunday 2 AM – Compressed) and since a few weeks I’m getting such behavior.
Backup files are created correctly and there are no error messages at the files created at /usr/local/cpanel/logs/cpbackup/
Last backup was finished in 17 minutes – so process started at 2:00 AM and Finished at 2:17AM
Checking at /var/log/messages file, I can’t see anything unusual except around 2 hours later Backup was finished (4:14AM) when I see a line full of NUL|NUL|NUL|
After this “nul line” nothing else has been registered until system reboot.
I've checked around different logs but nothing unusual. /var/log/apache2/error.log shows typical ModSecurity warnings but nothing around this period of time (02:00 – 4:14 AM). This log also stops recording anything just like message file and only starts recording after reboot.
This issue only happens after Backup routine as described.
By the way, I’m on VPS and when this happens, I can’t access Cpanel and have to reboot system via Host panel reboot button.
Any ideas of what and where should I look for?
-
Hey there! Do you not have a console level access through your hosting provider where you can see the terminal even if direct SSH access isn't working?
I wouldn't expect the backup process to take the server offline how you are describing, but it's possible you are experiencing high load.Has your hosting provider offered any details?
0 -
Nope, only way to reboot server on such situation is accessing hosting provider panel that gives me only a Reboot Button, nothing else. At least it works.... lol...
By coincidence this happens only on Sundays when backup is scheduled to run and I could bet it was about that but checking at all logs I did today, I could see server stuck about 2 ours later backup finished without any issues.
I have no other cron on the same timeframe as well as any WordPress/Wocommerce cron.
And this intrigues me the most.
I have contacted hosting provider and they opened a ticket to investigate that, as they have logs for the entire server as I can see only my VPS logs, though preliminary they found nothing and suspecting about a server load spike or some WAF rules (I use OWASP rules). As I mentioned there's nothing I could see to suspect about that.
That's why asking here as you guys always have good tips for further investigations.
0 -
I usually do have good tips, but it looks like you've tried them all :D
/var/log/messages is always a good spot for server-wide issues, but if there's no error messages there *and* nothing odd shows up in the backup logs, cPanel itself seems to think things are working well.
Is it possible there's something helpful in the sar logs? https://support.cpanel.net/hc/en-us/articles/360047748654-How-to-diagnose-high-loads-with-sar-command
That would track the load on the server over time and give you data to review after an issue has happened.
0 -
I’ve checked on Sar logs and server loads were less than 1.00 during the backup routine as well as the moment server got stuck (2 hours after backup routine).
As a last resort, despite none evidence at all, at least from the logs of my VPS stack (no answer from hosting provider so far), I was suspecting about that my IPtables to block IPs was too big and somehow could be the origin of this issue.
I had an absurdly basic firewall that’s part of hosting vps package that deals with iptables instead if ipset.
I was reluctant about CSF but why not, huh? It gave me quite some time to get it done but I should have done it before, if you know what I mean.
Let’s see how it goes.
0
Please sign in to leave a comment.
Comments
4 comments