Skip to main content

Monitor server going to power state 4?

rudtek
I have a VPS server running on Bluehost. The server doesn't have a lot on it, but keeps going down in a power state 4. Bluehost won't provide suggestions on how to monitor to see what is causing the problem. I've tried using top and watching, but is there any way to have a logging to see what is going on when the server stops? I'm not usually on my computer when it happens and so can't find what's going on or how to fix it. Maybe like a performance logger? i'm not even sure what I should be looking for!

Comments

12 comments

Date Votes
  • Jcats
    Sounds like a terrific web host! When you say power state are you referring to it going into sleep?
    0
  • rudtek
    they said powerstate 4 meant the server CPU is overwhelmed and so just locks up and my only option is to have them reset. I want to be able what processes / accounts are causing this and they don't know how to do that, because according to them, all the proccess usage data is reset when the server is reset.
    0
  • Jcats
    cPanel should send out emails as long as you have the email set in WHM > Basic WebHost Manager Setup should include ps.txt which will be most useful shows you processes. did you check dmesg or /var/log/messages for OOM messages?
    0
  • rudtek
    hey Jcats. Thanks for your help. My emails are set up correctly, and I do get system notices. The problem is when the system gets overloaded it just stops, so no emails go out at all. That's why i'm wondering if there is something like a process logger that would show who is using what cpu at the time of system lockup
    0
  • Jcats
    Just do a quick Google, example: How to Get a Email when System Load Average is High ? (Crontab) then use that on a 1 minute cronjob If the server locks up with no emails then you may need to monitor in faster intervals in which I would use a 'while' loop with 'sleep' so you can have the script execute as quickly as you want, every second or even every tenth of a second for example. I can help with that if necessary. Did you happen to check for the OOM messages by any chance?
    0
  • rudtek
    It's weird, I tried accessing that file and the server stopped. I'm resetting again right now.
    0
  • Jcats
    Which file and how did you try and access it? The more details the better :)
    0
  • rudtek
    sorry, I was trying to access /var/logs/messages. Got back up and downloaded the file. Looks like there may be some answers here. maybe my server is being attacked? there are 33000 lines of these errors. Jun 18 10:41:07 server pam_pwdfile[3655]: couldn't open password file /etc/techproxy.shadow Jun 18 10:41:07 server PAM-hulk[3655]: Brute force detection active: 580 LOGIN DENIED -- EXCESSIVE FAILURES -- IP TEMP BANNED Jun 18 10:41:09 server pam_pwdfile[3655]: couldn't open password file /etc/techproxy.shadow Jun 18 10:41:09 server PAM-hulk[3655]: Brute force detection active: 580 LOGIN DENIED -- EXCESSIVE FAILURES -- IP TEMP BANNED Jun 18 10:41:10 server pam_pwdfile[3665]: couldn't open password file /etc/techproxy.shadow Jun 18 10:41:10 server PAM-hulk[3665]: Brute force detection active: 580 LOGIN DENIED -- EXCESSIVE FAILURES -- IP TEMP BANNED Jun 18 10:41:12 server pam_pwdfile[3655]: couldn't open password file /etc/techproxy.shadow Jun 18 10:41:12 server PAM-hulk[3655]: Brute force detection active: 580 LOGIN DENIED -- EXCESSIVE FAILURES -- IP TEMP BANNED Jun 18 10:41:13 server pam_pwdfile[3665]: couldn't open password file /etc/techproxy.shadow Jun 18 10:41:13 server PAM-hulk[3665]: Brute force detection active: 580 LOGIN DENIED -- EXCESSIVE FAILURES -- IP TEMP BANNED Jun 18 10:41:15 server pam_pwdfile[3665]: couldn't open password file /etc/techproxy.shadow Jun 18 10:41:15 server PAM-hulk[3665]: Brute force detection active: 580 LOGIN DENIED -- EXCESSIVE FAILURES -- IP TEMP BANNED Jun 18 10:41:16 server pam_pwdfile[3690]: couldn't open password file /etc/techproxy.shadow Jun 18 10:41:16 server PAM-hulk[3690]: Brute force detection active: 580 LOGIN DENIED -- EXCESSIVE FAILURES -- IP TEMP BANNED Jun 18 10:41:19 server pam_pwdfile[3690]: couldn't open password file /etc/techproxy.shadow Jun 18 10:41:19 server PAM-hulk[3690]: Brute force detection active: 580 LOGIN DENIED -- EXCESSIVE FAILURES -- IP TEMP BANNED Jun 18 10:41:19 server pam_pwdfile[3714]: couldn't open password file /etc/techproxy.shadow Jun 18 10:41:19 server PAM-hulk[3714]: Brute force detection active: 580 LOGIN DENIED -- EXCESSIVE FAILURES -- IP TEMP BANNED Jun 18 10:41:22 server pam_pwdfile[3714]: couldn't open password file /etc/techproxy.shadow Jun 18 10:41:22 server PAM-hulk[3714]: Brute force detection active: 580 LOGIN DENIED -- EXCESSIVE FAILURES -- IP TEMP BANNED Jun 18 10:41:23 server pam_pwdfile[3690]: couldn't open password file /etc/techproxy.shadow Jun 18 10:41:23 server PAM-hulk[3690]: Brute force detection active: 580 LOGIN DENIED -- EXCESSIVE FAILURES -- IP TEMP BANNED Jun 18 10:41:24 server pam_pwdfile[3714]: couldn't open password file /etc/techproxy.shadow Jun 18 10:41:24 server PAM-hulk[3714]: Brute force detection active: 580 LOGIN DENIED -- EXCESSIVE FAILURES -- IP TEMP BANNED looking in my /etc directory, there is no file there named techproxy.shadow
    0
  • Jcats
    I would check: /usr/local/cpanel/logs/cphulkd.log
    to see what is being hit. I would also go into: WHM > cPHulk Brute Force Protection and check the box: "Block IP addresses at the firewall level if they trigger brute force protection" this way as long as the attacks you are getting are not from hundreds/thousands of different IP's your server shouldn't get completely consumed since the firewall will stop the excessive resource usage that is occurring now.
    0
  • rudtek
    okay. I'll try that. i would love help on that script too. It's a bit over my head.
    0
  • Jcats
    Just run these via SSH: yum -y install bc
    cat > "/root/checkload.sh" < 3 | bc -l) if [ "\$load" -ne 0 ]; then echo "Your Server Load Alert Needs Attention! " | mail -s "System Load Alert \$load" mail@helloacm.com fi EOF
    chmod +x /root/checkload.sh
    Replace the 'replace@me.com' with your actual email address before you paste the next line: sed -i 's/mail@helloacm.com/replace@me.com/g' /root/checkload.sh
    crontab -l | { cat; echo "* * * * * /root/checkload.sh"; } | crontab -
    This will send you an email anytime your server load hits a 1 minute load average of 3 or higher. I'm not sure of your average server load so if its higher than 3 then set 3 higher in the /root/checkload.sh script on this line load=$(echo $(cat /proc/loadavg | awk '{print $1}') \> 3 | bc -l)
    otherwise, the above commands will create the script as well as the cronjob so no need to do anything else.
    0
  • cPanelLauren
    Hi @rudtek I agree with the advice provided by @Jcats. You might also want to look at why this is giving the error it is: Jun 18 10:41:15 server pam_pwdfile[3665]: couldn't open password file /etc/techproxy.shadow
    The file /etc/techproxy.shadow is something added by your hosting provider specifically but it shouldn't be outputting that error all the time. Once the IP's are added to the firewalls block list you may find that the issue stops.
    0

Please sign in to leave a comment.

Didn't find what you were looking for?

New post