Apache Include to block unwanted bots
We are trying to block bad bots on the Apache side. The reason we have not used ModSecurity is because we believe it takes extra load to block traffic than if it was included in Apache file.
The below script works in Apache Config -> Pre VirtualHost Include
However, the issue that it only works for non-SSL sites. If the bot tries to access the page using httpS , the below rule does not get triggered. We observed it, by testing with curl.
Any thoughts on this?
RewriteEngine On
# Exempt local IP address 127.0.0.1 from the following rules
RewriteCond %{REMOTE_ADDR} ^127\.0\.0\.1$
RewriteRule .* - [L]
# Block specific unwanted User-Agents
RewriteCond %{HTTP_USER_AGENT} (gumgum-bot|postmanruntime|ag_dm_spider|scrapy|chimebot|crawl|amazonbot|amazon|scanner|yandex|^-$|^$|CCBot|ChatGPT-User|GPTBot|Google-Extended|Applebot-Extended|anthropic-ai|ClaudeBot|Omgilibot|Omgili|FacebookBot|Diffbot|Bytespider|ImagesiftBot|PerplexityBot|Go-http-client|cohere-ai) [NC]
# Rule to block these requests
RewriteRule .* - [F,L]
-
Hey there! I don't have a good explanation for why this wouldn't handle https traffic as well, as there is nothing in there that is sorting the type of traffic. If you have a system where this is happening on it might be best to create a ticket so the system can be examined, although we don't provide support for custom code like this.
0
Please sign in to leave a comment.
Comments
1 comment