Out of space error
Guys.
I was called out on Saturday to investigate web site errors, and even though i winged it and got the server back up an running, I've no real idea what caused it.
The initial call was for a MYSQL error on one of the web sites, which I assumed was a crashed table.
When I tried to log in to Cpanel for this account I got a message
"
The filesystem mounted at / on this server is running out of disk space."
Running df -h indicated that the server was out of space. 974gb used out of 975gb
running du -sk *| sort -n narrowed a large chunk used in the backup folder, so I removed all backups and df -h then told me that i was using 18gb of 975gb
so now I can get back in to Cpanel, and sort out the corrupted table.
Looking in my email logs, i have absolutely hundreds and hundreds of these messages
Jul 18 04:39:32 host lfd[12700]: *User Processing* PID:11898 Kill:0 User:user Time:1845 EXE:/usr/bin/php CMD:/usr/bin/php /home/user/public_html/index.php
so I don't have a single clue as to what was the cause.
Could anyone help shed any light ?
-
Hello, You have installed CSF firewall on your server and due to that you are getting these all alert from LFD. CSF send you emails about processes that are running too long. You can add that process in csf.pignore (/etc/csf/csf.pignore) file, if you want to ignore that process alert 0 -
Sounds like you were backing up to the same HDD as your accounts are on. (bad idea) ...indicated that the server was out of space. 974gb used out of 975gb
All kinds of things can break and or generate errors when you run out of disk space. Now that you've cleared the backups and given the system room, have the emails stopped?Looking in my email logs, i have absolutely hundreds and hundreds of these messages
That's not the full email is it? How long was it running, that's included in the email too.0 -
The backup is configured to an FTP account on another server. And since clearing the space, the errors have indeed stopped. I had 800 of those messages, seems it started in the early hours and continued all night. I checked the space usage this morning and it was up to 25gb, so i guess i'll have to keep my eye on this now for a few days/weeks and see if this starts to grow again. 0 -
Hello :) Keep in mind that even though the backup destination is a remote server, the backup archive is temporarily generated locally before transferred to that remote server via FTP. Thank you. 0
Please sign in to leave a comment.
Comments
4 comments