Server under attack email "no such person" denial of service from Iran, Uzbekistan, Kyrgyzstan, Azerbaijan, Kazakhstan
A client's primary cPanel mail server with 1000s of users is under attack.
Unfortunately the staff at Hi-Velocity doesn't understand the problem properly and / or they are unable to help after three days. Our service is down for three days.
The problem is quite simple to witness:
Every second around 30 to 60 "no such person" deliveries to our email server. The IP addresses as seen by WHOIS have a clear pattern from the countries mentioned in the subject of this post but there are more, e.g. Russia. It's clear the attackers are targeting our existing customer base and using a dictionary names attack for any user that might not even exist. If we only had 1 to 5 attempt per second, this would be okay, but it's a very powerful server and it's processing as fast as it can. The thing is Exim has a 'smtp_accept_max' value of 200 and this is quickly overwhelmed within about 1 minute. After 1 minute, the server starts responding with "Too many connections" We've upped the 200 limit to 1 000 but that gets overwhelmed within 2 minutes. We've upped the limit to 10 000 which then maxes out on it's own at 4 000 and the load of the server shoots up to 600. We've begged Hi-Velocity to get in touch with WHM/cPanel direct but got more nonsensical answers. The generic answers we're getting is "block logins" and "implement DoS mitigation for your web server". I tried logging a direct support ticket but it seems the 2FA for cPanel is going to an email address of our system that is under attack. We have CSF. We've tried adding CC_DENY but with both the Maxmind and other databases it's evident that these dodgy countries don't have entries: Below you'll see an example. For a test, we added CA, Canada. The countries that are bad actors simply don't have entries. Also just tailing the log file again for "no such person" and then doing WHOIS shows the bad actors are still sending. ``` csf: IPSET creating set cc_ir DROP all opt -- in * out * 0.0.0.0/0 -> 0.0.0.0/0 match-set cc_ir src csf: IPSET creating set cc_kz DROP all opt -- in * out * 0.0.0.0/0 -> 0.0.0.0/0 match-set cc_kz src csf: IPSET creating set cc_az DROP all opt -- in * out * 0.0.0.0/0 -> 0.0.0.0/0 match-set cc_az src csf: IPSET creating set cc_kg DROP all opt -- in * out * 0.0.0.0/0 -> 0.0.0.0/0 match-set cc_kg src csf: IPSET creating set cc_uz DROP all opt -- in * out * 0.0.0.0/0 -> 0.0.0.0/0 match-set cc_uz src csf: IPSET creating set cc_ca DROP all opt -- in * out * 0.0.0.0/0 -> 0.0.0.0/0 match-set cc_ca src csf: IPSET loading set cc_ca with 5695 entries ``` I would like professional help from cPanel themselves PLEASE.
tail -f /var/log/exim_mainlog | grep -i "no such person"Every second around 30 to 60 "no such person" deliveries to our email server. The IP addresses as seen by WHOIS have a clear pattern from the countries mentioned in the subject of this post but there are more, e.g. Russia. It's clear the attackers are targeting our existing customer base and using a dictionary names attack for any user that might not even exist. If we only had 1 to 5 attempt per second, this would be okay, but it's a very powerful server and it's processing as fast as it can. The thing is Exim has a 'smtp_accept_max' value of 200 and this is quickly overwhelmed within about 1 minute. After 1 minute, the server starts responding with "Too many connections" We've upped the 200 limit to 1 000 but that gets overwhelmed within 2 minutes. We've upped the limit to 10 000 which then maxes out on it's own at 4 000 and the load of the server shoots up to 600. We've begged Hi-Velocity to get in touch with WHM/cPanel direct but got more nonsensical answers. The generic answers we're getting is "block logins" and "implement DoS mitigation for your web server". I tried logging a direct support ticket but it seems the 2FA for cPanel is going to an email address of our system that is under attack. We have CSF. We've tried adding CC_DENY but with both the Maxmind and other databases it's evident that these dodgy countries don't have entries: Below you'll see an example. For a test, we added CA, Canada. The countries that are bad actors simply don't have entries. Also just tailing the log file again for "no such person" and then doing WHOIS shows the bad actors are still sending. ``` csf: IPSET creating set cc_ir DROP all opt -- in * out * 0.0.0.0/0 -> 0.0.0.0/0 match-set cc_ir src csf: IPSET creating set cc_kz DROP all opt -- in * out * 0.0.0.0/0 -> 0.0.0.0/0 match-set cc_kz src csf: IPSET creating set cc_az DROP all opt -- in * out * 0.0.0.0/0 -> 0.0.0.0/0 match-set cc_az src csf: IPSET creating set cc_kg DROP all opt -- in * out * 0.0.0.0/0 -> 0.0.0.0/0 match-set cc_kg src csf: IPSET creating set cc_uz DROP all opt -- in * out * 0.0.0.0/0 -> 0.0.0.0/0 match-set cc_uz src csf: IPSET creating set cc_ca DROP all opt -- in * out * 0.0.0.0/0 -> 0.0.0.0/0 match-set cc_ca src csf: IPSET loading set cc_ca with 5695 entries ``` I would like professional help from cPanel themselves PLEASE.
-
As a temporary measure to get things stable, I'd just shut off email and then block the most common IPs you can find in the mail log. If there's no mailserver for them to connect to, they'll likely stop trying to attempt as that is just a waste of resources on their end. Then, after some IPs have been blocked in the firewall, you can turn the mail service back on. 0 -
Here is the outcome of this issue and advice if someone else has to ever go through this pain. Once you have been become distributed target of a "No such person" attack, blocking individual IPs is going to be impossible. Here is one reason why it's impossible: WHM/cPanel doesn't have a firewall for blocking IPs that are trying to send email to your server. WHM/cPanel has a firewall called cPHulk, but cPHulk only helps for logging in attacks. That means, users who are trying to log in. In this particular case, with his particular method, it's a very specific SMTP port 25 sending attack that I'll term for now "No such person at this address" attack. We got this en-masse, from mostly x-communist countries. It's clear most of the these zombie computers were using the same playbook. Whilst trying to solve this problem, at Hi-Velocity and at WHM I worked with about 12 engineers, 1st level, 2nd level, and so called 3rd level. It's incredible to find how many of the engineers simply didn't get that it's a SMTP sending attack, not a logging in attack. It's also incredulous how many engineers recommended HTTP/S blocking. How many times I had to explain, by way of tail, that the problem is quite simple people targeting 100s of users that do not exist on this server, every split second. So your next big problem, since WHM doesn't give you the tools to ward off this attack, is to install another firewall, such as CSF. Next next problem here is you're pretty much on your own. Most ticket engineers have a number of tickets to attend to and can't sit around too long for a complex problem to be solved. At least what I found, one of this distributed nature. Also, WHM ticket staff will entertain these queries but sound every one of them off with "CSF is 3rd party" and there's only so much that they can do. Hats off the WHM though, as I could log into my own cPanel reseller account and solve another cPanel reseller's problem. Once you have an engineer that actually has the skill, they will recommend one or two things and then conveniently disappear into the system to go and attend to their other tickets. It's incredible how many times I had to go back to people and tell them thanks for the suggestion but it didn't work. How many had the honesty to come back to THE SAME PROBLEM and admit it's still not working, without giving up. I guess is this is why this took almost 4 days. So hear me out - with some of these complex issues you're completely on your own. In the end the what it seems it the most interesting behavior and solution is from CSF. When you add CC_DENY to CSF, it doesn't exactly work the first time. At I explain to a number of engineers, `csr -r` show empty country IPSETs: csf: IPSET creating set cc_in DROP all opt -- in * out * 0.0.0.0/0 -> 0.0.0.0/0 match-set cc_in src csf: IPSET creating set cc_bd DROP all opt -- in * out * 0.0.0.0/0 -> 0.0.0.0/0 match-set cc_bd src csf: IPSET creating set cc_ir DROP all opt -- in * out * 0.0.0.0/0 -> 0.0.0.0/0 match-set cc_ir src csf: IPSET creating set cc_kz DROP all opt -- in * out * 0.0.0.0/0 -> 0.0.0.0/0 match-set cc_kz src
However, come back the next day and now you IPSETs populated like so:csf: IPSET creating set cc_in DROP all opt -- in * out * 0.0.0.0/0 -> 0.0.0.0/0 match-set cc_in src csf: IPSET loading set cc_in with 6444 entries csf: IPSET creating set cc_bd DROP all opt -- in * out * 0.0.0.0/0 -> 0.0.0.0/0 match-set cc_bd src csf: IPSET loading set cc_bd with 1825 entries csf: IPSET creating set cc_ir DROP all opt -- in * out * 0.0.0.0/0 -> 0.0.0.0/0 match-set cc_ir src csf: IPSET loading set cc_ir with 1560 entries csf: IPSET creating set cc_kz DROP all opt -- in * out * 0.0.0.0/0 -> 0.0.0.0/0 match-set cc_kz src csf: IPSET loading set cc_kz with 473 entries
Summary: - It's a waste of time to block individual IPs, /24s, and /16s. - You have to use a 3rd party firewall that does actual IP blocking. Don't bank on cPHulk. - If it's CSF, you have to wait "some time" for the rules to be updated with actual networks0 -
It's important to note that cPanel has never managed a firewall service. That's deeper into the network management than we like to get, as we'd hate to end up being the one to accidentally block something or someone. CSF is a great tool, but they are completely unrelated to our company so we can't help with issues related to them. For large-scale attacks, such as the one you experienced or a large DDoS of Apache, external tools are going to be required - unfortunately that's just how it is. An external firewall managed at your host would keep such traffic from ever getting to your server at all. 0
Please sign in to leave a comment.
Comments
4 comments