How to disallow some crawler globally in httpd conf?

postcd

• May 27, 2014 04:43

Hello, if i want to disable some web crawler globally on all existing and future accounts, how to do it persistently? example this can be added to .htaccess but i need persistent global file of httpd ? I found Home " Service Configuration " Apache Configuration " Include Editor there are 3 include files, not sure if i can use this and add mentioned thing? persistently? Also what if i want to add some robots.txt rule globally? like: [QUOTE]user-agent: AhrefsBot Crawl-Delay: 3

• 

  cPanelMichael

  • May 27, 2014 13:57

  Hello :) You will need to follow the instructions from this document: Changes Contained Within a VirtualHost Directive As for your second question, it's answered on this external website: [url=http://stackoverflow.com/questions/227101/can-i-block-search-crawlers-for-every-site-on-an-apache-web-server]Can I block search crawlers for every site on an Apache web server? - Stack Overflow Thanks.

  0

Please sign in to leave a comment.