Server Low on Memory (Process Terminated) - High Load Events
Somewhat frequently, my server experiences a very high load and subsequently issues these email notifications about terminating processes. In most of these cases, the high load is seemingly due to a denial-of-service attack, as I can see the requests that are being made and their contents are invalid or specifically indicate that they are probing the server for some known vulnerability.
I have a firewall and other modalities in place to detect and block IPs issuing such requests, but the server does still experience a significant load due to the volume of requests coming in, even though it's ultimately blocking them and no breach ever occurs.
My suggestion in this case is relevant to the email notification that is received when these events occur: "Out of memory: A process was terminated because the system is low on memory." The email contains various pieces of information, including the current server load (very helpful), and iostat output and Top Processes.'
However, what I would like to see is a list of the top 10 IP addresses with the most connections to the server over http ports. When I observe one of these attacks in real time, I typically log into the server (assumiing it's still responsive enough) and issue a netstat command like this:
netstat -tn | awk '$4 ~ /:(80|443)$/ {print $5}' | cut -d: -f1 | sort | uniq -c | sort -nr | head -n 10
Then I take the most active IPs and run them through a WHOIS to give me the geolocation and owner info. Sometimes it looks like Google doing it indexing of the hosted sites (although that rarely brings down the server), but often it's certain hosting providers that I've learned over the years tend to host clients that utilize their services for malicious scripts and bots that roam the web looking for servers to attack.
When I catch the attack in progress, I have a script I can run that will execute (assuming the server isn't completely unresponsive):
http://localhost/whm-server-status
and parse the output to get the connecting IPs and their request content strings, filtering out the requests from 127.0.0.1. This helps me determine which external requests are likely malicious so I can block them
But since I most typically catch the issue after the fact, having at least the netstat output in the notification would be helpful in order to be able to block those IPs (or even their CIDR block) for some time to thwart additional attacks in the short term. Having output like the whm-server-status script from the time of the event would be outstanding.
Alternatively, even just having this event information summarized and logged on the server in a text file would be helpful, perhaps with a notation about it's name/location in the email notification.
-
Hey there! Since this would be new functionality it would need to be part of our Feature Request system. Would you like me to get that submitted for you so I can bring it up for review?
0 -
Yes, thanks!
0 -
Done! If everything follows the typical schedule we'll talk about this on September 18th. I've linked this thread so I'll be sure to post an update once I know more.
0 -
Awesome! Thanks so much!
0 -
This turned into a very interesting discussion on our side. Apparently we legally *can't* provide the IP addresses in the email as they are potential personally identifiable information, and sending those over email isn't something we can do.
BUT...I've created case CPANEL-57328 to see if we could provide a link in that email to a page on the server with more information. The team will discuss this so I can't say for sure if it will happen just yet, but I've linked this thread to the case so if I hear an update on my end I'll be sure to post!
0 -
Thanks! Ultimately, we can get the IPs by searching the access log for that time. I was just looking to cut out that step since that part is pretty time consuming. But I totally understand about the legal implications. Even if those active IPs and (if possible) their respective request URLs can be logged server-side and we can get to it, that would be very helpful. The request content really helps determine if the activity looks suspicious. I am constantly adding mod_security rules to automatically block certain URL contents that are known to be malicious or vulnerability hunting.
As an added bonus perhaps in the future, it would be great if the UI for that server-side info page showed the WHOIS country/location information for each IP, and had an option to selectively add the listed IPs to the CSF deny file for some configurable amount of time and then restart the firewall. This would help tremendously when we are looking at an attack in real-time, but would also help block suspicious IPs for some future time so they can't immediately try again when the server recovers.
1
Please sign in to leave a comment.
Comments
6 comments