Block Unwanted Bots Agent and Proxys

Hy guys, so after 4h of google-ing and testing i quit :) the think is i made a file .htaccess in /home/ into my server and i added the way it is into it. But i think is not working, i used User Agent Switch to make the tests but still nothink. The think is i whant to block this sort of bad agents and proxy traffic to ower websites, i think nobody whants bad traffic right ? Like proxys ones, witch there is a software it grabs 100k proxys and after start sending trafic to a website and makes masive traffic and a masive usage on the server. So here is it, did i made sompting wrong ? :D


RewriteEngine On

RewriteCond %{HTTP:VIA}                 !^$ [OR]
RewriteCond %{HTTP:FORWARDED}           !^$ [OR]
RewriteCond %{HTTP:USERAGENT_VIA}       !^$ [OR]
RewriteCond %{HTTP:X_FORWARDED_FOR}     !^$ [OR]
RewriteCond %{HTTP:PROXY_CONNECTION}    !^$ [OR]
RewriteCond %{HTTP:XPROXY_CONNECTION}   !^$ [OR]
RewriteCond %{HTTP:HTTP_PC_REMOTE_ADDR} !^$ [OR]
RewriteCond %{HTTP:HTTP_CLIENT_IP}      !^$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^$        [OR]
RewriteCond %{HTTP_REFERER} amazonaws\.com [OR]
RewriteCond %{HTTP_USER_AGENT} ^AISearchBot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^woriobot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^heritrix [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^NetSeer [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Nutch [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Baiduspider [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^aipbot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Anarchie [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^ASPSeek [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^attach [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^autoemailspider [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^BlackWidow [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Bot\ mailto:craftbot@yahoo.com [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^ChinaClaw [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Custo [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^DISCo [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Download\ Demon [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^eCatch [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^EirGrabber [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^EmailSiphon [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^EmailWolf [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Express\ WebPictures [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^ExtractorPro [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^EyeNetIE [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^FlashGet [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^GetRight [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^GetWeb! [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Go!Zilla [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^MJ12bot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Go-Ahead-Got-It [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^GrabNet [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Grafula [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^HMView [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^HTTrack [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^iblog [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Image\ Stripper [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Image\ Sucker [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Indy\ Library [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^InterGET [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Internet\ Ninja [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^JetCar [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^JOC\ Web\ Spider [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^larbin [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^LeechFTP [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Linkwalker [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Mass\ Downloader [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^MIDown\ tool [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Mister\ PiX [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^nameprotect [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Navroad [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^NearSite [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^NetAnts [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^NetSpider [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Net\ Vampire [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^NetZIP [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Octopus [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Offline\ Explorer [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Offline\ Navigator [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^PageGrabber [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Papa\ Foto [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^pavuk [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^pcBrowser [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^RealDownload [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^ReGet [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^searchestate [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^SiteSnagger [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^SmartDownload [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^SuperBot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^SuperHTTP [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Surfbot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^tAkeOut [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Teleport\ Pro [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^TurnitinBot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^VoidEYE [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Web\ Image\ Collector [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Web\ Sucker [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^WebAuto [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^WebCopier [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^WebFetch [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^WebGo\ IS [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^WebLeacher [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^WebReaper [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^WebSauger [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Website\ eXtractor [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Website\ Quester [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^WebStripper [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^WebWhacker [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^WebZIP [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Wget [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Widow [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^WWWOFFLE [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Xaldon\ WebSpider [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Xenu [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Zeus [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^curl/ [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^HTMLParser [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Jakarta\ Commons [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Java [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^libcurl [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^LWP::Simple [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^lwp-request [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Microsoft\ Data\ Access [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Microsoft\ URL\ Control [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^MS\ Web\ Services\ Client\ Protocol [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^PECL::HTTP [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^POE-Component-Client-HTTP [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^PycURL [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Snoopy [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^VB\ Project [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^WWW::Mechanize [NC,OR]
RewriteCond %{HTTP_USER_AGENT} RPT-HTTPClient [NC,OR]

RewriteCond %{HTTP_USER_AGENT} ^.*(<|>|'|%0A|%0D|%27|%3C|%3E|%00).* [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^.*(HTTrack|Wordpress|wp|emailwolf|clshttp|archiver|loader|email|nikto|miner|python).* [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^.*(winhttp|libwww\-perl|emailwolf|curl|wget|harvest|scan|grab|extract).* [NC]
RewriteRule ^(.*)$ - [F,L]

cPanelMichael

February 09, 2016 17:53

Hello :) Have you considered using Mod_Security instead of Mod_Rewrite to block user agents? Thank you.

quizknows

February 09, 2016 19:52

This would be my recommendation. There are a couple good ways to do this with ModSecurity. One would be individual rules like:


SecRule REQUEST_HEADERS:User-Agent "^NetSeer" "deny,id:12345"

Or, you could make a list of user agents in a file like /usr/local/apache/conf/my_custom_list.conf and then make a modsec rule like:


SecRule REQUEST_HEADERS:User-Agent "@pmFromFile /usr/local/apache/conf/my_custom_list.conf" "deny,id:12346"

This method is detailed in the modsecurity manual here Reference Manual " SpiderLabs/ModSecurity Wiki " GitHub

Bidi

February 10, 2016 05:46

i will try to make mod_sec way :) i hope not braking to many thinks, but there is another one, witch drives me crazy is this (on the pic) and i dont know what agent to write to block them, if no agent or sompting like this. They ar some sort of proxys, or mirc bots i think. [Removed]

February 10, 2016 15:41

Please ensure you attach images directly to the post, instead of linking to a third-party image hosting website. This is listed at: Guide To Opening An Effective Forums Thread Thank you.

February 12, 2016 06:43

I fix it, i made all the thinks in mod_sec rules, even for proxys, bad agents, fake hits ...etc thank you, works like a charm :D

February 12, 2016 16:48

I am happy to see you were able to address the issue. Thank you for updating us with the outcome.

February 17, 2016 14:43

Whell i`m back :( cuz of this rulls witch brakes my websites and i dont understand whant i`m doing bad, dose anyone got any ideas ?

 # Block empty User-Agents.
SecRule REQUEST_HEADERS:User-Agent "@eq 0" \
"id:'13009',phase:2,t:none,deny,status:406,log,msg:'Fake Agent - Detectat'"

and

SecRule REQUEST_HEADERS:User-Agent "^$" \
"id:'13006',phase:2,t:none,deny,status:406,log,msg:'Fake Agent - Detectat'"

February 17, 2016 19:18

Try this instead of those two:


# Block empty User-Agents.
SecRule &REQUEST_HEADERS:User-Agent "@eq 0" \
"id:'13009',phase:2,t:none,deny,status:406,log,msg:'Fake Agent - Detectat'"

If you're using an operator like @eq sometimes you need to add the & before the matched var.

February 17, 2016 19:25

Hy i forget to mention :( i had like theath before but the problem is when i had theath rule enabled and on WHM i go to Apache Status i got this error. Do i have to whitelist sompting ? Apache server status for ************ Failed to receive status information from Apache. And i dont understand why. Thank you for your reply

February 18, 2016 01:23

That is the correct syntax (with the &) however I don't think WHM provides a User Agent when it queries server status. The log looks like this:


127.0.0.1 - - [17/Feb/2016:21:20:01 -0500] "GET /whm-server-status HTTP/1.0" 200 6153

Try this:


# Block empty User-Agents.
SecRule &REQUEST_HEADERS:User-Agent "@eq 0" \
"id:'13009',phase:2,t:none,deny,chain,status:406,log,msg:'Fake Agent - Detectat'"
SecRule REMOTE_ADDR "!@ipMatch 127.0.0.1"

This will allow 127.0.0.1 to query the server without a user agent specified but other IPs will not be allowed to. That should fix WHM server status for you.

February 18, 2016 15:58

I had added lest see how it works :D thank you

Block Unwanted Bots Agent and Proxys

Comments

Didn't find what you were looking for?