Unknown robot (identified by 'bot*')
A lot of my customer sites are seeing HUGE upticks in traffic due to unidentified bots. As in, 90% of traffic coming from unidentified bots. awstats reports them as
There has always been quite a bit of this, but lately it seems far worse. I'm starting to think it might make sense to set up server-wide rules to whitelist the major/important bots and drop the rest. Has anyone tried this? Have you ended up accidentally over-blocking good bots? I don't want to get into an endless cat-and-mouse game - just looking for a good formula that will let as much good bot traffic through as possible while dumping the rest. If you've done, did you do it at the apache level or firewall level? Any good tools or formulae to share? Thanks.
Unknown robot (identified by 'bot*')There has always been quite a bit of this, but lately it seems far worse. I'm starting to think it might make sense to set up server-wide rules to whitelist the major/important bots and drop the rest. Has anyone tried this? Have you ended up accidentally over-blocking good bots? I don't want to get into an endless cat-and-mouse game - just looking for a good formula that will let as much good bot traffic through as possible while dumping the rest. If you've done, did you do it at the apache level or firewall level? Any good tools or formulae to share? Thanks.
-
Hello :) You may find the following thread helpful: Bot Control With Mod_Security Thank you. 0 -
cPanel's new ModSecurity module landed a few days ago and I've been giving it a workout. I added and activated a single custom rule, which culls bad bot activity by cross referencing a bots RBL from sbl-xbl.spamhaus.org. In the past eight hours it's blocked ~50,000 requests by bad bots to all of my customer sites. Absolutely incredible. The hits are not just on comment forms etc., but on images, homepages, RSS feeds, PDFs, everything. Will let it run for a couple of days then use awstats to see how this affects bandwidth usage by the heaviest-hit customers. Here's the rule I'm using (all on one line of course): SecRule REMOTE_ADDR "@rbl sbl-xbl.spamhaus.org" "phase:1,id:'981138',t:none,pass,nolog,auditlog,msg:'RBL Match for SPAM Source',tag:'AUTOMATION/MALICIOUS',severity:'2',setvar:'tx.msg=%{rule.msg}',setvar:tx.automation_score=+%{tx.warning_anomaly_score},setvar:tx.anomaly_score=+%{tx.warning_anomaly_score},setvar:tx.%{rule.id}-AUTOMATION/MALICIOUS-%{matched_var_name}=%{matched_var},setvar:ip.spammer=1,expirevar:ip.spammer=86400,setvar:ip.previous_rbl_check=1,expirevar:ip.previous_rbl_check=86400,skipAfter:END_RBL_CHECK" 0 -
I had a problem with bots but not like 90% I am checking your script shacker23 Thanks! 0 -
I didn't write the rule - it's directly out of the spiderlabs modsec rules manual on github. But yeah - I'm seeing overall traffic cut down on most large customer sites by 50% overnight. No false positives to report (yet). Very promising. 0 -
OK, I misspoke on the traffic reduction - looks like awstats hadn't completed its run when I wrote earlier. I am seeing the massive numbers of log entries in ModSecurity, but I am NOT seeing a huge reduction in traffic after all via awstats. My understanding is that ModSecurity hits should be appearing in apache's error_log, but I'm not seeing them there. It feels like mod security is logging, but not actually blocking. Is there some integration option or documentation I'm not seeing somewhere? Thanks. 0 -
Do you have CSF (Config Server Firewall) installed? It will block your mod_security hits. 0 -
Oh! Surely there must be a way to configure ConfigServer to work *with* ModSecurity rather than against it? Any idea how? 0 -
OK, so cpanel's modsecurity logs to /usr/local/apache/logs/modsec_audit.log . In CSF configuration, I set MODSEC_LOG to that path and restarted CSF. So then I tail -f /var/log/lfd.log . I see CSF's normal block entries showing up in lfd.log, but no sign of the tons of hits that are showing up in the ModSecurity log watcher in the gui. Have I got things wired up wrong, or am I looking in the wrong place? 0 -
Hi, I was wondering if it is feasible to write a rule that will allow all known bots except a list of known bots that I can update in a file; and disallow all unknown bots except a list of allowed unknown bots that I can update in a 2nd file When I mean known and unknown bots, I refer to what I see in awstats, it seems awstats can identify most mainstream bots... Is such a rule possible? Advisable? 0
Please sign in to leave a comment.
Comments
9 comments