Globally stop bad bots in Apache
Usually use a tip on htaccess for stop badbots for some sites.
But in last times, too many sites has same problem: too many bad boots with high traffic.
What is best method for put on httpd.conf or for global sites ?
# Block Bad Bots & Scrapers
SetEnvIfNoCase User-Agent "MJ12bot" bad_bot
SetEnvIfNoCase User-Agent "pingdom" bad_bot
SetEnvIfNoCase User-Agent "Aboundex" bad_bot
SetEnvIfNoCase User-Agent "80legs" bad_bot
SetEnvIfNoCase User-Agent "360Spider" bad_bot
SetEnvIfNoCase User-Agent "^Java" bad_bot
SetEnvIfNoCase User-Agent "^Cogentbot" bad_bot
SetEnvIfNoCase User-Agent "^Alexibot" bad_bot
SetEnvIfNoCase User-Agent "^asterias" bad_bot
SetEnvIfNoCase User-Agent "^attach" bad_bot
SetEnvIfNoCase User-Agent "^BackDoorBot" bad_bot
SetEnvIfNoCase User-Agent "^BackWeb" bad_bot
SetEnvIfNoCase User-Agent "Bandit" bad_bot
SetEnvIfNoCase User-Agent "^BatchFTP" bad_bot
SetEnvIfNoCase User-Agent "^Bigfoot" bad_bot
SetEnvIfNoCase User-Agent "^Black.Hole" bad_bot
SetEnvIfNoCase User-Agent "^BlackWidow" bad_bot
SetEnvIfNoCase User-Agent "^BlowFish" bad_bot
SetEnvIfNoCase User-Agent "^BotALot" bad_bot
SetEnvIfNoCase User-Agent "Buddy" bad_bot
SetEnvIfNoCase User-Agent "^BuiltBotTough" bad_bot
SetEnvIfNoCase User-Agent "^Bullseye" bad_bot
SetEnvIfNoCase User-Agent "^BunnySlippers" bad_bot
SetEnvIfNoCase User-Agent "^Cegbfeieh" bad_bot
SetEnvIfNoCase User-Agent "^CheeseBot" bad_bot
SetEnvIfNoCase User-Agent "^CherryPicker" bad_bot
SetEnvIfNoCase User-Agent "^ChinaClaw" bad_bot
SetEnvIfNoCase User-Agent "Collector" bad_bot
SetEnvIfNoCase User-Agent "Copier" bad_bot
SetEnvIfNoCase User-Agent "^CopyRightCheck" bad_bot
SetEnvIfNoCase User-Agent "^cosmos" bad_bot
SetEnvIfNoCase User-Agent "^Crescent" bad_bot
SetEnvIfNoCase User-Agent "^Custo" bad_bot
SetEnvIfNoCase User-Agent "^AIBOT" bad_bot
SetEnvIfNoCase User-Agent "^DISCo" bad_bot
SetEnvIfNoCase User-Agent "^DIIbot" bad_bot
SetEnvIfNoCase User-Agent "^DittoSpyder" bad_bot
SetEnvIfNoCase User-Agent "^Download\ Demon" bad_bot
SetEnvIfNoCase User-Agent "^Download\ Devil" bad_bot
SetEnvIfNoCase User-Agent "^Download\ Wonder" bad_bot
SetEnvIfNoCase User-Agent "^dragonfly" bad_bot
SetEnvIfNoCase User-Agent "^Drip" bad_bot
SetEnvIfNoCase User-Agent "^eCatch" bad_bot
SetEnvIfNoCase User-Agent "^EasyDL" bad_bot
SetEnvIfNoCase User-Agent "^ebingbong" bad_bot
SetEnvIfNoCase User-Agent "^EirGrabber" bad_bot
SetEnvIfNoCase User-Agent "^EmailCollector" bad_bot
SetEnvIfNoCase User-Agent "^EmailSiphon" bad_bot
SetEnvIfNoCase User-Agent "^EmailWolf" bad_bot
SetEnvIfNoCase User-Agent "^EroCrawler" bad_bot
SetEnvIfNoCase User-Agent "^Exabot" bad_bot
SetEnvIfNoCase User-Agent "^Express\ WebPictures" bad_bot
SetEnvIfNoCase User-Agent "Extractor" bad_bot
SetEnvIfNoCase User-Agent "^EyeNetIE" bad_bot
SetEnvIfNoCase User-Agent "^Foobot" bad_bot
SetEnvIfNoCase User-Agent "^flunky" bad_bot
SetEnvIfNoCase User-Agent "^FrontPage" bad_bot
SetEnvIfNoCase User-Agent "^Go-Ahead-Got-It" bad_bot
SetEnvIfNoCase User-Agent "^gotit" bad_bot
SetEnvIfNoCase User-Agent "^GrabNet" bad_bot
SetEnvIfNoCase User-Agent "^Grafula" bad_bot
SetEnvIfNoCase User-Agent "^Harvest" bad_bot
SetEnvIfNoCase User-Agent "^hloader" bad_bot
SetEnvIfNoCase User-Agent "^HMView" bad_bot
SetEnvIfNoCase User-Agent "^HTTrack" bad_bot
SetEnvIfNoCase User-Agent "^humanlinks" bad_bot
SetEnvIfNoCase User-Agent "^IlseBot" bad_bot
SetEnvIfNoCase User-Agent "^Image\ Stripper" bad_bot
SetEnvIfNoCase User-Agent "^Image\ Sucker" bad_bot
SetEnvIfNoCase User-Agent "Indy\ Library" bad_bot
SetEnvIfNoCase User-Agent "^InfoNaviRobot" bad_bot
SetEnvIfNoCase User-Agent "^InfoTekies" bad_bot
SetEnvIfNoCase User-Agent "^Intelliseek" bad_bot
SetEnvIfNoCase User-Agent "^InterGET" bad_bot
SetEnvIfNoCase User-Agent "^Internet\ Ninja" bad_bot
SetEnvIfNoCase User-Agent "^Iria" bad_bot
SetEnvIfNoCase User-Agent "^Jakarta" bad_bot
SetEnvIfNoCase User-Agent "^JennyBot" bad_bot
SetEnvIfNoCase User-Agent "^JetCar" bad_bot
SetEnvIfNoCase User-Agent "^JOC" bad_bot
SetEnvIfNoCase User-Agent "^JustView" bad_bot
SetEnvIfNoCase User-Agent "^Jyxobot" bad_bot
SetEnvIfNoCase User-Agent "^Kenjin.Spider" bad_bot
SetEnvIfNoCase User-Agent "^Keyword.Density" bad_bot
SetEnvIfNoCase User-Agent "^larbin" bad_bot
SetEnvIfNoCase User-Agent "^LexiBot" bad_bot
SetEnvIfNoCase User-Agent "^lftp" bad_bot
SetEnvIfNoCase User-Agent "^libWeb/clsHTTP" bad_bot
SetEnvIfNoCase User-Agent "^likse" bad_bot
SetEnvIfNoCase User-Agent "^LinkextractorPro" bad_bot
SetEnvIfNoCase User-Agent "^LinkScan/8.1a.Unix" bad_bot
SetEnvIfNoCase User-Agent "^LNSpiderguy" bad_bot
SetEnvIfNoCase User-Agent "^LinkWalker" bad_bot
SetEnvIfNoCase User-Agent "^lwp-trivial" bad_bot
SetEnvIfNoCase User-Agent "^LWP::Simple" bad_bot
SetEnvIfNoCase User-Agent "^Magnet" bad_bot
SetEnvIfNoCase User-Agent "^Mag-Net" bad_bot
SetEnvIfNoCase User-Agent "^MarkWatch" bad_bot
SetEnvIfNoCase User-Agent "^Mass\ Downloader" bad_bot
SetEnvIfNoCase User-Agent "^Mata.Hari" bad_bot
SetEnvIfNoCase User-Agent "^Memo" bad_bot
SetEnvIfNoCase User-Agent "^Microsoft.URL" bad_bot
SetEnvIfNoCase User-Agent "^Microsoft\ URL\ Control" bad_bot
SetEnvIfNoCase User-Agent "^MIDown\ tool" bad_bot
SetEnvIfNoCase User-Agent "^MIIxpc" bad_bot
SetEnvIfNoCase User-Agent "^Mirror" bad_bot
SetEnvIfNoCase User-Agent "^Missigua\ Locator" bad_bot
SetEnvIfNoCase User-Agent "^Mister\ PiX" bad_bot
SetEnvIfNoCase User-Agent "^moget" bad_bot
SetEnvIfNoCase User-Agent "^Mozilla/3.Mozilla/2.01" bad_bot
SetEnvIfNoCase User-Agent "^Mozilla.*NEWT" bad_bot
SetEnvIfNoCase User-Agent "^NAMEPROTECT" bad_bot
SetEnvIfNoCase User-Agent "^Navroad" bad_bot
SetEnvIfNoCase User-Agent "^NearSite" bad_bot
SetEnvIfNoCase User-Agent "^NetAnts" bad_bot
SetEnvIfNoCase User-Agent "^Netcraft" bad_bot
SetEnvIfNoCase User-Agent "^NetMechanic" bad_bot
SetEnvIfNoCase User-Agent "^NetSpider" bad_bot
SetEnvIfNoCase User-Agent "^Net\ Vampire" bad_bot
SetEnvIfNoCase User-Agent "^NetZIP" bad_bot
SetEnvIfNoCase User-Agent "^NextGenSearchBot" bad_bot
SetEnvIfNoCase User-Agent "^NG" bad_bot
SetEnvIfNoCase User-Agent "^NICErsPRO" bad_bot
SetEnvIfNoCase User-Agent "^niki-bot" bad_bot
SetEnvIfNoCase User-Agent "^NimbleCrawler" bad_bot
SetEnvIfNoCase User-Agent "^Ninja" bad_bot
SetEnvIfNoCase User-Agent "^NPbot" bad_bot
SetEnvIfNoCase User-Agent "^Octopus" bad_bot
SetEnvIfNoCase User-Agent "^Offline\ Explorer" bad_bot
SetEnvIfNoCase User-Agent "^Offline\ Navigator" bad_bot
SetEnvIfNoCase User-Agent "^Openfind" bad_bot
SetEnvIfNoCase User-Agent "^OutfoxBot" bad_bot
SetEnvIfNoCase User-Agent "^PageGrabber" bad_bot
SetEnvIfNoCase User-Agent "^Papa\ Foto" bad_bot
SetEnvIfNoCase User-Agent "^pavuk" bad_bot
SetEnvIfNoCase User-Agent "^pcBrowser" bad_bot
SetEnvIfNoCase User-Agent "^PHP\ version\ tracker" bad_bot
SetEnvIfNoCase User-Agent "^Pockey" bad_bot
SetEnvIfNoCase User-Agent "^ProPowerBot/2.14" bad_bot
SetEnvIfNoCase User-Agent "^ProWebWalker" bad_bot
SetEnvIfNoCase User-Agent "^psbot" bad_bot
SetEnvIfNoCase User-Agent "^Pump" bad_bot
SetEnvIfNoCase User-Agent "^QueryN.Metasearch" bad_bot
SetEnvIfNoCase User-Agent "^RealDownload" bad_bot
SetEnvIfNoCase User-Agent "Reaper" bad_bot
SetEnvIfNoCase User-Agent "Recorder" bad_bot
SetEnvIfNoCase User-Agent "^ReGet" bad_bot
SetEnvIfNoCase User-Agent "^RepoMonkey" bad_bot
SetEnvIfNoCase User-Agent "^RMA" bad_bot
SetEnvIfNoCase User-Agent "Siphon" bad_bot
SetEnvIfNoCase User-Agent "^SiteSnagger" bad_bot
SetEnvIfNoCase User-Agent "^SlySearch" bad_bot
SetEnvIfNoCase User-Agent "^SmartDownload" bad_bot
SetEnvIfNoCase User-Agent "^Snake" bad_bot
SetEnvIfNoCase User-Agent "^Snapbot" bad_bot
SetEnvIfNoCase User-Agent "^Snoopy" bad_bot
SetEnvIfNoCase User-Agent "^sogou" bad_bot
SetEnvIfNoCase User-Agent "^SpaceBison" bad_bot
SetEnvIfNoCase User-Agent "^SpankBot" bad_bot
SetEnvIfNoCase User-Agent "^spanner" bad_bot
SetEnvIfNoCase User-Agent "^Sqworm" bad_bot
SetEnvIfNoCase User-Agent "Stripper" bad_bot
SetEnvIfNoCase User-Agent "Sucker" bad_bot
SetEnvIfNoCase User-Agent "^SuperBot" bad_bot
SetEnvIfNoCase User-Agent "^SuperHTTP" bad_bot
SetEnvIfNoCase User-Agent "^Surfbot" bad_bot
SetEnvIfNoCase User-Agent "^suzuran" bad_bot
SetEnvIfNoCase User-Agent "^Szukacz/1.4" bad_bot
SetEnvIfNoCase User-Agent "^tAkeOut" bad_bot
SetEnvIfNoCase User-Agent "^Teleport" bad_bot
SetEnvIfNoCase User-Agent "^Telesoft" bad_bot
SetEnvIfNoCase User-Agent "^TurnitinBot/1.5" bad_bot
SetEnvIfNoCase User-Agent "^The.Intraformant" bad_bot
SetEnvIfNoCase User-Agent "^TheNomad" bad_bot
SetEnvIfNoCase User-Agent "^TightTwatBot" bad_bot
SetEnvIfNoCase User-Agent "^Titan" bad_bot
SetEnvIfNoCase User-Agent "^True_Robot" bad_bot
SetEnvIfNoCase User-Agent "^turingos" bad_bot
SetEnvIfNoCase User-Agent "^TurnitinBot" bad_bot
SetEnvIfNoCase User-Agent "^URLy.Warning" bad_bot
SetEnvIfNoCase User-Agent "^Vacuum" bad_bot
SetEnvIfNoCase User-Agent "^VCI" bad_bot
SetEnvIfNoCase User-Agent "^VoidEYE" bad_bot
SetEnvIfNoCase User-Agent "^Web\ Image\ Collector" bad_bot
SetEnvIfNoCase User-Agent "^Web\ Sucker" bad_bot
SetEnvIfNoCase User-Agent "^WebAuto" bad_bot
SetEnvIfNoCase User-Agent "^WebBandit" bad_bot
SetEnvIfNoCase User-Agent "^Webclipping.com" bad_bot
SetEnvIfNoCase User-Agent "^WebCopier" bad_bot
SetEnvIfNoCase User-Agent "^WebEMailExtrac.*" bad_bot
SetEnvIfNoCase User-Agent "^WebEnhancer" bad_bot
SetEnvIfNoCase User-Agent "^WebFetch" bad_bot
SetEnvIfNoCase User-Agent "^WebGo\ IS" bad_bot
SetEnvIfNoCase User-Agent "^Web.Image.Collector" bad_bot
SetEnvIfNoCase User-Agent "^WebLeacher" bad_bot
SetEnvIfNoCase User-Agent "^WebmasterWorldForumBot" bad_bot
SetEnvIfNoCase User-Agent "^WebReaper" bad_bot
SetEnvIfNoCase User-Agent "^WebSauger" bad_bot
SetEnvIfNoCase User-Agent "^Website\ eXtractor" bad_bot
SetEnvIfNoCase User-Agent "^Website\ Quester" bad_bot
SetEnvIfNoCase User-Agent "^Webster" bad_bot
SetEnvIfNoCase User-Agent "^WebStripper" bad_bot
SetEnvIfNoCase User-Agent "^WebWhacker" bad_bot
SetEnvIfNoCase User-Agent "^WebZIP" bad_bot
SetEnvIfNoCase User-Agent "Whacker" bad_bot
SetEnvIfNoCase User-Agent "^Widow" bad_bot
SetEnvIfNoCase User-Agent "^WISENutbot" bad_bot
SetEnvIfNoCase User-Agent "^WWWOFFLE" bad_bot
SetEnvIfNoCase User-Agent "^WWW-Collector-E" bad_bot
SetEnvIfNoCase User-Agent "^Xaldon" bad_bot
SetEnvIfNoCase User-Agent "^Xenu" bad_bot
SetEnvIfNoCase User-Agent "^Zeus" bad_bot
SetEnvIfNoCase User-Agent "ZmEu" bad_bot
SetEnvIfNoCase User-Agent "^Zyborg" bad_bot
# Vulnerability Scanners
SetEnvIfNoCase User-Agent "Acunetix" bad_bot
SetEnvIfNoCase User-Agent "FHscan" bad_bot
# Aggressive Chinese Search Engine
SetEnvIfNoCase User-Agent "Baiduspider" bad_bot
# Aggressive Russian Search Engine
SetEnvIfNoCase User-Agent "Yandex" bad_bot
Order Allow,Deny
Allow from all
# Cyveillance
deny from 38.100.19.8/29
deny from 38.100.21.0/24
deny from 38.100.41.64/26
deny from 38.105.71.0/25
deny from 38.105.83.0/27
deny from 38.112.21.140/30
deny from 38.118.42.32/29
deny from 65.213.208.128/27
deny from 65.222.176.96/27
deny from 65.222.185.72/29
Deny from env=bad_bot
-
Hi. According manual tamainut001/44_tamainut.conf # zip -r tamainut001.zip tamainut001 # md5sum tamainut001.zip 02e20c3e46431cff58b84137d801d4f0 tamainut001.zip
put file on my server example.com/tamainut001.zip put file badbots.txt on Cpanel server with content60Spider 80legs AIBOT Aboundex Acunetix AhrefsBot Alexibot Anonymizer Attributor ...
Edit meta_tamainut001.yaml--- 1.0.0: MD5: 7ce5b898da4e038cb70b38fe96a9e427 distribution: tamainut-0 url: https://example.com/tamainut001.zip attributes: description: 'Anti Bad Bots' name: 'Anti Bad Bots' vendor_url: https://example.com/ report_url: https://example.com/intranet/submitticket.php
Go to my WHM / ModSecuroty Vendors / Add Vendor Vendor Configuration URL
If use rule, for example putting an other vendor, such configserver, rule work (not block bad Apache restart, and log every accesos from bad bot) Some help please..0 -
You can post "similar" solution for ModSecurity ? @Tearabite 0 -
You can post "similar" solution for ModSecurity ? @Tearabite
Our rule is virtually the same - just different filenames and slightly different list of bots and we have set it up as a custom rule instead of a vendor configuration, so basically the same thing. We also block by known paths/vulnerability probes.0 -
Error:API failure: The vendor metadata does not contain an entry for your version of ModSecurity, "2.9.2". The only version of ModSecurity this rule set supports is "1.0.0".
Hello, You used the following entry in your metadata file:1.0.0:
This needs to be the version of Mod_Security installed on the server. EX:2.9.2:
Thank you.0 -
Well, but not work now by MD5 API failure: The downloaded vendor archive does not match the expected digest (MD5 or SHA512).
On meta_tamainut.yaml same MD5 summd5sum tamainut001.zip 9a1896a0191******a515f3de39982550 tamainut001.zip0 -
Wel.. After try with SHA512 hash, work fine. Tahnks. 0 -
Wel.. After try with SHA512 hash, work fine. Tahnks.
Please Do post here the tutorial. How to block bad bot?0 -
Does this correct the location of the place "Post VirtualHost Include" adding to block some bad bots and hosts? # Start Block Bad Bots and Scrapers SetEnvIfNoCase User-Agent "MJ12bot" bad_bot SetEnvIfNoCase User-Agent "pingdom" bad_bot SetEnvIfNoCase User-Agent "Aboundex" bad_bot SetEnvIfNoCase User-Agent "80legs" bad_bot SetEnvIfNoCase User-Agent "360Spider" bad_bot SetEnvIfNoCase User-Agent "^Java" bad_bot SetEnvIfNoCase User-Agent "^Cogentbot" bad_bot SetEnvIfNoCase User-Agent "^Alexibot" bad_bot SetEnvIfNoCase User-Agent "^asterias" bad_bot SetEnvIfNoCase User-Agent "^attach" bad_bot SetEnvIfNoCase User-Agent "^BackDoorBot" bad_bot SetEnvIfNoCase User-Agent "^BackWeb" bad_bot SetEnvIfNoCase User-Agent "Bandit" bad_bot SetEnvIfNoCase User-Agent "^BatchFTP" bad_bot SetEnvIfNoCase User-Agent "^Bigfoot" bad_bot SetEnvIfNoCase User-Agent "^Black.Hole" bad_bot SetEnvIfNoCase User-Agent "^BlackWidow" bad_bot SetEnvIfNoCase User-Agent "^BlowFish" bad_bot SetEnvIfNoCase User-Agent "^BotALot" bad_bot SetEnvIfNoCase User-Agent "Buddy" bad_bot SetEnvIfNoCase User-Agent "^BuiltBotTough" bad_bot SetEnvIfNoCase User-Agent "^Bullseye" bad_bot SetEnvIfNoCase User-Agent "^BunnySlippers" bad_bot SetEnvIfNoCase User-Agent "^Cegbfeieh" bad_bot SetEnvIfNoCase User-Agent "^CheeseBot" bad_bot SetEnvIfNoCase User-Agent "^CherryPicker" bad_bot SetEnvIfNoCase User-Agent "^ChinaClaw" bad_bot SetEnvIfNoCase User-Agent "Collector" bad_bot SetEnvIfNoCase User-Agent "Copier" bad_bot SetEnvIfNoCase User-Agent "^CopyRightCheck" bad_bot SetEnvIfNoCase User-Agent "^cosmos" bad_bot SetEnvIfNoCase User-Agent "^Crescent" bad_bot SetEnvIfNoCase User-Agent "^Custo" bad_bot SetEnvIfNoCase User-Agent "^AIBOT" bad_bot SetEnvIfNoCase User-Agent "^DISCo" bad_bot SetEnvIfNoCase User-Agent "^DIIbot" bad_bot SetEnvIfNoCase User-Agent "^DittoSpyder" bad_bot SetEnvIfNoCase User-Agent "^Download\ Demon" bad_bot SetEnvIfNoCase User-Agent "^Download\ Devil" bad_bot SetEnvIfNoCase User-Agent "^Download\ Wonder" bad_bot SetEnvIfNoCase User-Agent "^dragonfly" bad_bot SetEnvIfNoCase User-Agent "^Drip" bad_bot SetEnvIfNoCase User-Agent "^eCatch" bad_bot SetEnvIfNoCase User-Agent "^EasyDL" bad_bot SetEnvIfNoCase User-Agent "^ebingbong" bad_bot SetEnvIfNoCase User-Agent "^EirGrabber" bad_bot SetEnvIfNoCase User-Agent "^EmailCollector" bad_bot SetEnvIfNoCase User-Agent "^EmailSiphon" bad_bot SetEnvIfNoCase User-Agent "^EmailWolf" bad_bot SetEnvIfNoCase User-Agent "^EroCrawler" bad_bot SetEnvIfNoCase User-Agent "^Exabot" bad_bot SetEnvIfNoCase User-Agent "^Express\ WebPictures" bad_bot SetEnvIfNoCase User-Agent "Extractor" bad_bot SetEnvIfNoCase User-Agent "^EyeNetIE" bad_bot SetEnvIfNoCase User-Agent "^Foobot" bad_bot SetEnvIfNoCase User-Agent "^flunky" bad_bot SetEnvIfNoCase User-Agent "^FrontPage" bad_bot SetEnvIfNoCase User-Agent "^Go-Ahead-Got-It" bad_bot SetEnvIfNoCase User-Agent "^gotit" bad_bot SetEnvIfNoCase User-Agent "^GrabNet" bad_bot SetEnvIfNoCase User-Agent "^Grafula" bad_bot SetEnvIfNoCase User-Agent "^Harvest" bad_bot SetEnvIfNoCase User-Agent "^hloader" bad_bot SetEnvIfNoCase User-Agent "^HMView" bad_bot SetEnvIfNoCase User-Agent "^HTTrack" bad_bot SetEnvIfNoCase User-Agent "^humanlinks" bad_bot SetEnvIfNoCase User-Agent "^IlseBot" bad_bot SetEnvIfNoCase User-Agent "^Image\ Stripper" bad_bot SetEnvIfNoCase User-Agent "^Image\ Sucker" bad_bot SetEnvIfNoCase User-Agent "Indy\ Library" bad_bot SetEnvIfNoCase User-Agent "^InfoNaviRobot" bad_bot SetEnvIfNoCase User-Agent "^InfoTekies" bad_bot SetEnvIfNoCase User-Agent "^Intelliseek" bad_bot SetEnvIfNoCase User-Agent "^InterGET" bad_bot SetEnvIfNoCase User-Agent "^Internet\ Ninja" bad_bot SetEnvIfNoCase User-Agent "^Iria" bad_bot SetEnvIfNoCase User-Agent "^Jakarta" bad_bot SetEnvIfNoCase User-Agent "^JennyBot" bad_bot SetEnvIfNoCase User-Agent "^JetCar" bad_bot SetEnvIfNoCase User-Agent "^JOC" bad_bot SetEnvIfNoCase User-Agent "^JustView" bad_bot SetEnvIfNoCase User-Agent "^Jyxobot" bad_bot SetEnvIfNoCase User-Agent "^Kenjin.Spider" bad_bot SetEnvIfNoCase User-Agent "^Keyword.Density" bad_bot SetEnvIfNoCase User-Agent "^larbin" bad_bot SetEnvIfNoCase User-Agent "^LexiBot" bad_bot SetEnvIfNoCase User-Agent "^lftp" bad_bot SetEnvIfNoCase User-Agent "^libWeb/clsHTTP" bad_bot SetEnvIfNoCase User-Agent "^likse" bad_bot SetEnvIfNoCase User-Agent "^LinkextractorPro" bad_bot SetEnvIfNoCase User-Agent "^LinkScan/8.1a.Unix" bad_bot SetEnvIfNoCase User-Agent "^LNSpiderguy" bad_bot SetEnvIfNoCase User-Agent "^LinkWalker" bad_bot SetEnvIfNoCase User-Agent "^lwp-trivial" bad_bot SetEnvIfNoCase User-Agent "^LWP::Simple" bad_bot SetEnvIfNoCase User-Agent "^Magnet" bad_bot SetEnvIfNoCase User-Agent "^Mag-Net" bad_bot SetEnvIfNoCase User-Agent "^MarkWatch" bad_bot SetEnvIfNoCase User-Agent "^Mass\ Downloader" bad_bot SetEnvIfNoCase User-Agent "^Mata.Hari" bad_bot SetEnvIfNoCase User-Agent "^Memo" bad_bot SetEnvIfNoCase User-Agent "^Microsoft.URL" bad_bot SetEnvIfNoCase User-Agent "^Microsoft\ URL\ Control" bad_bot SetEnvIfNoCase User-Agent "^MIDown\ tool" bad_bot SetEnvIfNoCase User-Agent "^MIIxpc" bad_bot SetEnvIfNoCase User-Agent "^Mirror" bad_bot SetEnvIfNoCase User-Agent "^Missigua\ Locator" bad_bot SetEnvIfNoCase User-Agent "^Mister\ PiX" bad_bot SetEnvIfNoCase User-Agent "^moget" bad_bot SetEnvIfNoCase User-Agent "^Mozilla/3.Mozilla/2.01" bad_bot SetEnvIfNoCase User-Agent "^Mozilla.*NEWT" bad_bot SetEnvIfNoCase User-Agent "^NAMEPROTECT" bad_bot SetEnvIfNoCase User-Agent "^Navroad" bad_bot SetEnvIfNoCase User-Agent "^NearSite" bad_bot SetEnvIfNoCase User-Agent "^NetAnts" bad_bot SetEnvIfNoCase User-Agent "^Netcraft" bad_bot SetEnvIfNoCase User-Agent "^NetMechanic" bad_bot SetEnvIfNoCase User-Agent "^NetSpider" bad_bot SetEnvIfNoCase User-Agent "^Net\ Vampire" bad_bot SetEnvIfNoCase User-Agent "^NetZIP" bad_bot SetEnvIfNoCase User-Agent "^NextGenSearchBot" bad_bot SetEnvIfNoCase User-Agent "^NG" bad_bot SetEnvIfNoCase User-Agent "^NICErsPRO" bad_bot SetEnvIfNoCase User-Agent "^niki-bot" bad_bot SetEnvIfNoCase User-Agent "^NimbleCrawler" bad_bot SetEnvIfNoCase User-Agent "^Ninja" bad_bot SetEnvIfNoCase User-Agent "^NPbot" bad_bot SetEnvIfNoCase User-Agent "^Octopus" bad_bot SetEnvIfNoCase User-Agent "^Offline\ Explorer" bad_bot SetEnvIfNoCase User-Agent "^Offline\ Navigator" bad_bot SetEnvIfNoCase User-Agent "^Openfind" bad_bot SetEnvIfNoCase User-Agent "^OutfoxBot" bad_bot SetEnvIfNoCase User-Agent "^PageGrabber" bad_bot SetEnvIfNoCase User-Agent "^Papa\ Foto" bad_bot SetEnvIfNoCase User-Agent "^pavuk" bad_bot SetEnvIfNoCase User-Agent "^pcBrowser" bad_bot SetEnvIfNoCase User-Agent "^PHP\ version\ tracker" bad_bot SetEnvIfNoCase User-Agent "^Pockey" bad_bot SetEnvIfNoCase User-Agent "^ProPowerBot/2.14" bad_bot SetEnvIfNoCase User-Agent "^ProWebWalker" bad_bot SetEnvIfNoCase User-Agent "^psbot" bad_bot SetEnvIfNoCase User-Agent "^Pump" bad_bot SetEnvIfNoCase User-Agent "^QueryN.Metasearch" bad_bot SetEnvIfNoCase User-Agent "^RealDownload" bad_bot SetEnvIfNoCase User-Agent "Reaper" bad_bot SetEnvIfNoCase User-Agent "Recorder" bad_bot SetEnvIfNoCase User-Agent "^ReGet" bad_bot SetEnvIfNoCase User-Agent "^RepoMonkey" bad_bot SetEnvIfNoCase User-Agent "^RMA" bad_bot SetEnvIfNoCase User-Agent "Siphon" bad_bot SetEnvIfNoCase User-Agent "^SiteSnagger" bad_bot SetEnvIfNoCase User-Agent "^SlySearch" bad_bot SetEnvIfNoCase User-Agent "^SmartDownload" bad_bot SetEnvIfNoCase User-Agent "^Snake" bad_bot SetEnvIfNoCase User-Agent "^Snapbot" bad_bot SetEnvIfNoCase User-Agent "^Snoopy" bad_bot SetEnvIfNoCase User-Agent "^sogou" bad_bot SetEnvIfNoCase User-Agent "^SpaceBison" bad_bot SetEnvIfNoCase User-Agent "^SpankBot" bad_bot SetEnvIfNoCase User-Agent "^spanner" bad_bot SetEnvIfNoCase User-Agent "^Sqworm" bad_bot SetEnvIfNoCase User-Agent "Stripper" bad_bot SetEnvIfNoCase User-Agent "Sucker" bad_bot SetEnvIfNoCase User-Agent "^SuperBot" bad_bot SetEnvIfNoCase User-Agent "^SuperHTTP" bad_bot SetEnvIfNoCase User-Agent "^Surfbot" bad_bot SetEnvIfNoCase User-Agent "^suzuran" bad_bot SetEnvIfNoCase User-Agent "^Szukacz/1.4" bad_bot SetEnvIfNoCase User-Agent "^tAkeOut" bad_bot SetEnvIfNoCase User-Agent "^Teleport" bad_bot SetEnvIfNoCase User-Agent "^Telesoft" bad_bot SetEnvIfNoCase User-Agent "^TurnitinBot/1.5" bad_bot SetEnvIfNoCase User-Agent "^The.Intraformant" bad_bot SetEnvIfNoCase User-Agent "^TheNomad" bad_bot SetEnvIfNoCase User-Agent "^TightTwatBot" bad_bot SetEnvIfNoCase User-Agent "^Titan" bad_bot SetEnvIfNoCase User-Agent "^True_Robot" bad_bot SetEnvIfNoCase User-Agent "^turingos" bad_bot SetEnvIfNoCase User-Agent "^TurnitinBot" bad_bot SetEnvIfNoCase User-Agent "^URLy.Warning" bad_bot SetEnvIfNoCase User-Agent "^Vacuum" bad_bot SetEnvIfNoCase User-Agent "^VCI" bad_bot SetEnvIfNoCase User-Agent "^VoidEYE" bad_bot SetEnvIfNoCase User-Agent "^Web\ Image\ Collector" bad_bot SetEnvIfNoCase User-Agent "^Web\ Sucker" bad_bot SetEnvIfNoCase User-Agent "^WebAuto" bad_bot SetEnvIfNoCase User-Agent "^WebBandit" bad_bot SetEnvIfNoCase User-Agent "^Webclipping.com" bad_bot SetEnvIfNoCase User-Agent "^WebCopier" bad_bot SetEnvIfNoCase User-Agent "^WebEMailExtrac.*" bad_bot SetEnvIfNoCase User-Agent "^WebEnhancer" bad_bot SetEnvIfNoCase User-Agent "^WebFetch" bad_bot SetEnvIfNoCase User-Agent "^WebGo\ IS" bad_bot SetEnvIfNoCase User-Agent "^Web.Image.Collector" bad_bot SetEnvIfNoCase User-Agent "^WebLeacher" bad_bot SetEnvIfNoCase User-Agent "^WebmasterWorldForumBot" bad_bot SetEnvIfNoCase User-Agent "^WebReaper" bad_bot SetEnvIfNoCase User-Agent "^WebSauger" bad_bot SetEnvIfNoCase User-Agent "^Website\ eXtractor" bad_bot SetEnvIfNoCase User-Agent "^Website\ Quester" bad_bot SetEnvIfNoCase User-Agent "^Webster" bad_bot SetEnvIfNoCase User-Agent "^WebStripper" bad_bot SetEnvIfNoCase User-Agent "^WebWhacker" bad_bot SetEnvIfNoCase User-Agent "^WebZIP" bad_bot SetEnvIfNoCase User-Agent "Whacker" bad_bot SetEnvIfNoCase User-Agent "^Widow" bad_bot SetEnvIfNoCase User-Agent "^WISENutbot" bad_bot SetEnvIfNoCase User-Agent "^WWWOFFLE" bad_bot SetEnvIfNoCase User-Agent "^WWW-Collector-E" bad_bot SetEnvIfNoCase User-Agent "^Xaldon" bad_bot SetEnvIfNoCase User-Agent "^Xenu" bad_bot SetEnvIfNoCase User-Agent "^Zeus" bad_bot SetEnvIfNoCase User-Agent "ZmEu" bad_bot SetEnvIfNoCase User-Agent "^Zyborg" bad_bot # Vulnerability Scanners SetEnvIfNoCase User-Agent "Acunetix" bad_bot SetEnvIfNoCase User-Agent "FHscan" bad_bot # Aggressive Chinese Search Engine SetEnvIfNoCase User-Agent "Baiduspider" bad_bot # Aggressive Russian Search Engine SetEnvIfNoCase User-Agent "Yandex" bad_bot # End Block Bad Bots and Scrapers0 -
Hello @Nirjonadda, The original poster setup their own custom ModSecurity vendor to block those bots: How to Create a ModSecurity Vendor - cPanel Knowledge Base - cPanel Documentation Thank you. 0 -
You can just add a custom ModSecurity rule, regardless of what vendor you're using, and don't have to create a vendor. The steps are: - Create a folder in /etc called "my_user-agent_blacklist".
- Create a text file in the /etc/my_user-agent_blacklist/ folder called "userdata_bl_agents" that contains a string representing each bot you want to block, one on each line. You don't need to use the entire name of the bot, just a representative string that will be specific to that user agent; e.g.:
archive.org_bot AhrefsBot Baiduspider BLEXBot Contacts-Crawler CrawlDaddy deadlinkchecker Domain Re-Animator ...
- In WHM, go Security Center -> ModSecurity Tools -> Rules List.
- Click Add Rule.
- Paste in the following for the rule:
SecRule REQUEST_HEADERS:User-Agent "@pmFromFile /etc/my_user-agent_blacklist/userdata_bl_agents" \ "id:1,msg:'Blacklisted by userdata_bl_agents|%{tx.0}|%{tx.mode}',phase:2,severity:WARNING,capture,logdata:'%{TX.0}',drop,t:'none'"
Make sure "Enable Rule" and "Deploy and Restart Apache" are checked. Click "Save". Your ModSecurity Tools Hits List will start listing all the bots it blocks. If you want to add a bot, just add it to the text file, and ModSecurity will do the rest.0 -
To block bad bots and IP's we use BotBanish It does a very good job at it even with the new rules that apply for Apache 2.4 that don't apply for Apache 2.2. 0 -
To block bad bots and IP's we use BotBanish It does a very good job at it even with the new rules that apply for Apache 2.4 that don't apply for Apache 2.2.
How to use BotBanish for Xenforo forum? Does have any add-on for Xenforo?0 -
I would suggest using the website version since there is no version for Xenforo... 0
Please sign in to leave a comment.
Comments
16 comments