Server Low on Memory (Process Terminated) - High Load Events
Somewhat frequently, my server experiences a very high load and subsequently issues these email notifications about terminating processes. In most of these cases, the high load is seemingly due to a denial-of-service attack, as I can see the requests that are being made and their contents are invalid or specifically indicate that they are probing the server for some known vulnerability.
I have a firewall and other modalities in place to detect and block IPs issuing such requests, but the server does still experience a significant load due to the volume of requests coming in, even though it's ultimately blocking them and no breach ever occurs.
My suggestion in this case is relevant to the email notification that is received when these events occur: "Out of memory: A process was terminated because the system is low on memory." The email contains various pieces of information, including the current server load (very helpful), and iostat output and Top Processes.'
However, what I would like to see is a list of the top 10 IP addresses with the most connections to the server over http ports. When I observe one of these attacks in real time, I typically log into the server (assumiing it's still responsive enough) and issue a netstat command like this:
netstat -tn | awk '$4 ~ /:(80|443)$/ {print $5}' | cut -d: -f1 | sort | uniq -c | sort -nr | head -n 10
Then I take the most active IPs and run them through a WHOIS to give me the geolocation and owner info. Sometimes it looks like Google doing it indexing of the hosted sites (although that rarely brings down the server), but often it's certain hosting providers that I've learned over the years tend to host clients that utilize their services for malicious scripts and bots that roam the web looking for servers to attack.
When I catch the attack in progress, I have a script I can run that will execute (assuming the server isn't completely unresponsive):
http://localhost/whm-server-status
and parse the output to get the connecting IPs and their request content strings, filtering out the requests from 127.0.0.1. This helps me determine which external requests are likely malicious so I can block them
But since I most typically catch the issue after the fact, having at least the netstat output in the notification would be helpful in order to be able to block those IPs (or even their CIDR block) for some time to thwart additional attacks in the short term. Having output like the whm-server-status script from the time of the event would be outstanding.
Alternatively, even just having this event information summarized and logged on the server in a text file would be helpful, perhaps with a notation about it's name/location in the email notification.
-
Hey there! Since this would be new functionality it would need to be part of our Feature Request system. Would you like me to get that submitted for you so I can bring it up for review?
0 -
Yes, thanks!
0 -
Done! If everything follows the typical schedule we'll talk about this on September 18th. I've linked this thread so I'll be sure to post an update once I know more.
0 -
Awesome! Thanks so much!
0
Please sign in to leave a comment.
Comments
4 comments