mod_evasive issues
I was facing a DDOS attack and installed mod_evasive. Ran into a few issues, some related to configuration (so I'll indicate how I've solved them, for others that face the same issue), some I think are bugs in mod_evasive.
ISSUE 1 (bug?):
Setting DOSBlockingPeriod in /etc/apache2/conf.d/300-mod_evasive.conf doesn't have any effect (yes, I did run "service httpd reload"). I wanted to set it to 300 (seconds), but it still uses the default setting 10 (seconds). It's not defined anywhere else (grepped all files in the /etc/apache2 tree).
ISSUE 2 (configuration):
If you leave DOSBlockingHttpResponse set to 403 and you have ErrorDocument set for 403 responses which require a 403.shtml file in public_html... then you get a 404 error if that file is missing. Changing DOSBlockingHttpResponse to 429 solves that, because (at least in my config) it doesn't have an ErrorDocument attached to it, so it just works. Plus it's a much clearer error message to distinguish it from actual permission problems that might also result in a 403.
ISSUE 3 (configuration):
If setting DOSEmailNotify doesn't result in mails getting sent, in WHM go to "Tweak settings", search for "nobody" and set the "Prevent "nobody" from sending mail" to "off".
ISSUE 4 (bug or feature?):
I only seem to get email notifications when triggering DOSSiteCount, but not when triggering DOSPageCount.
edit: found the cause, it's not related to the type of notification: 0 * * * * /usr/bin/find /var/log/apache2/mod_evasive -ctime +1 -type f | /usr/bin/xargs -r rm
ISSUE 5 (bug?): The notifications that I'm getting had an empty subject and what was supposed to be in headers (subject / to) ended up in the body of the email. A very dirty way to fix that is:
Originally, it calls the /bin/mail program with the recipient email addres as argument (%s), but if you leave out the argument and instead supply the option "-t" it parses the supplied headers for the recipient address. Solved the issue for me, although it'll probably be undone if there's ever an update of the mod_evasive package installed. edit: see also nobody@my.server.tld), you can change the name in /etc/passwd. 5th parameter on the line that starts with "nobody" is the name used. Changed that from "Nobody" to "Nobody - my.server.tld" so I could distinguish the mails better in my mailbox. ISSUE 7 (wishlist): If issue 4 gets fixed, it would be really nice if the notification mail didn't just show the IP addres, but also the URL that was requested too many times. Or even include it in both DOSSite and DOSPage notifications, but in that case indicate which type of notification Site/Page it is. Otherwise one has to search all access logs to figure out which website was attacked (I was hoping for some kind of log entry somewhere in /var/log, but couldn't find it). ISSUE 8: Wordpress sites (configuration): Probably happens with other websites as well, but the default DOSSiteCount 100 setting immediately got me blocked when visiting one of the hosted wordpress website, because one page consisted of way more than 100 elements which got loaded within the 2 seconds set in DOSSiteInterval and thus triggerd mod_evasive. I've bumped that setting to 500. The DDOS I faced had a few hundred IPnrs each requesting the same URL, but at a relatively slow rate (once per 2-5 seconds), so the default setting DosPageCount 4 and DosPageInterval 2 (which requires over 2 requests per seconds to start blocking) didn't have any effect. So I raised that to blocking when exceeding 3 requests for the same URL per 20 seconds. However, if you click through the Wordpress backend, on most pages it loads the exact same JS (and other) files, so clicking 4 times in the backend resulted in blocking. Raised that to 15 identical requests per 60 seconds, which may still require extra tuning. I'm hoping for a mod_evasive package update for the issues marked bug/wishlist (if there's another way to fix it, that's fine too). --Ruud
ISSUE 5 (bug?): The notifications that I'm getting had an empty subject and what was supposed to be in headers (subject / to) ended up in the body of the email. A very dirty way to fix that is:
sed -i 's/mail %s/mail -t/' /usr/lib64/apache2/modules/mod_evasive24.soOriginally, it calls the /bin/mail program with the recipient email addres as argument (%s), but if you leave out the argument and instead supply the option "-t" it parses the supplied headers for the recipient address. Solved the issue for me, although it'll probably be undone if there's ever an update of the mod_evasive package installed. edit: see also nobody@my.server.tld), you can change the name in /etc/passwd. 5th parameter on the line that starts with "nobody" is the name used. Changed that from "Nobody" to "Nobody - my.server.tld" so I could distinguish the mails better in my mailbox. ISSUE 7 (wishlist): If issue 4 gets fixed, it would be really nice if the notification mail didn't just show the IP addres, but also the URL that was requested too many times. Or even include it in both DOSSite and DOSPage notifications, but in that case indicate which type of notification Site/Page it is. Otherwise one has to search all access logs to figure out which website was attacked (I was hoping for some kind of log entry somewhere in /var/log, but couldn't find it). ISSUE 8: Wordpress sites (configuration): Probably happens with other websites as well, but the default DOSSiteCount 100 setting immediately got me blocked when visiting one of the hosted wordpress website, because one page consisted of way more than 100 elements which got loaded within the 2 seconds set in DOSSiteInterval and thus triggerd mod_evasive. I've bumped that setting to 500. The DDOS I faced had a few hundred IPnrs each requesting the same URL, but at a relatively slow rate (once per 2-5 seconds), so the default setting DosPageCount 4 and DosPageInterval 2 (which requires over 2 requests per seconds to start blocking) didn't have any effect. So I raised that to blocking when exceeding 3 requests for the same URL per 20 seconds. However, if you click through the Wordpress backend, on most pages it loads the exact same JS (and other) files, so clicking 4 times in the backend resulted in blocking. Raised that to 15 identical requests per 60 seconds, which may still require extra tuning. I'm hoping for a mod_evasive package update for the issues marked bug/wishlist (if there's another way to fix it, that's fine too). --Ruud
-
@manager23 - thanks for taking the time to write this up. I'll do some additional testing on my end tomorrow and send you some more details then. 0 -
@manager23 - we've had some things come up so I'm not sure if I'll get to this today. I will definitely test this and report back! 0 -
@cPRex: About DOSBlockingPeriod: I don't think it explains #1. I'm testing this by first triggering a block via a quick series of requests. Then wait 15 seconds before doing another request. If I have set the DOSBlockingPeriod to 30, I would expect to still be blocked, but the block is removed after 10 seconds of inactivity, regardless of what I specify for DOSBlockingPeriod. But I'll report it to the upstream maintainer. Should I report back here if upstream code is updated or is that checked automatically somehow? edit (wanted to edit topic start, but probably too late): ISSUE 9: mod_evasive vs mod_remoteip If you have both installed are using (for example) Cloudflare and want to whitelist the original visitor IPnr, you'll find that while mod_remoteip can retrieve the original visitor IPnr and it gets stored in access logs just fine, mod_evasive unfortunately ignores what mod_remoteip does and still uses the Cloudflare IPnrs. If you then try to whitelist the netblocks on Ubuntu Manpage: sipcalc - IP subnet calculator) to do it like this, because you end up with a list of 6000+ netblocks, so it's not something you want to do manually: curl https://www.cloudflare.com/ips-v4 | \ xargs -n1 sipcalc -s /24 | \ grep Network | \ cut -d- -f2 | \ cut -d' ' -f2 | \ sed -e 's/\.0$/.*/' -e 's/^/DOSWhitelist /' \ > /etc/apache2/mod_evasive.cloudflare.ipv4.conf
And then add this to the mod_evasive config file followed by a reload of the apache config (service httpd reload):Include /etc/apache2/mod_evasive.cloudflare.ipv4.conf
You'd have to do this periodically, in case Cloudflare adds more netblocks. edit2 (adding to issue 9): In Cloudflare DNS, you also have to make sure the IPnr of the cPanel server is only listed via IPv4 (A) and not IPv6 (AAAA), otherwise you'd have to whitelist IPv6 netblocks as well. Didn't find documentation on how to whitelist IPv6 addresses. Looking at the code, it seems to be targeted only at IPv4 addresses... and seems I misread the documentations. For IPv4, instead of 162.158.* (which doesn't give an error, but doesn't work either) it seems that 162.158.*.* would work, so you can use wildcards like 1.*.*.*, 1.2.*.* and 1.2.3.* to match class A, B and C netblocks respectively. You'd still have to convert from the CIDR notation cloudflare uses to a series of class B and C networks (which results in a smaller set than just generating class C netblocks). Leaving that as an exercise to the reader ;)0 -
Feel free to post back here if you do hear from the upstream crew about any changes! 0 -
I just noticed the issue about the nobody@ email issues with my own mod_evasive installs but i found that the fix suggested: [QUOTE] ISSUE 5 (bug?): The notifications that I'm getting had an empty subject and what was supposed to be in headers (subject / to) ended up in the body of the email. A very dirty way to fix that is: sed -i 's/mail %s/mail -t/' /usr/lib64/apache2/modules/mod_evasive24.so
Originally, it calls the /bin/mail program with the recipient email addres as argument (%s), but if you leave out the argument and instead supply the option "-t" it parses the supplied headers for the recipient address. Solved the issue for me, although it'll probably be undone if there's ever an update of the mod_evasive package installed. edit: see also0
Please sign in to leave a comment.
Comments
7 comments