Whitelisting the Search Engine Spiders

Avalon

• May 21, 2015 04:30

Today I noticed that mod_security blocked valid Google spiders from accessing some of my servers and while it was relatively easy to unblock them. Then I proceed to add the "big name" search engines to the CSF ignore list but I know this list is incomplete:
.googlebot.com .crawl.yahoo.net .search.msn.com
Does anyone have a listing of what domains are used for other popular search engines like DuckDuckGo, Yandex, Baidu, etc. I haven't had false positives with any of the other engines yet but better to be prepared. I didn't think it would be so hard to find a listing of crawler domains to be honest.

• 

  cPanelMichael

  • May 21, 2015 14:41

  Hello, You may find this third-party URL helpful:

  0
• 

  Avalon

  • May 22, 2015 01:16

  Hello, You may find this third-party URL helpful:

  0

Please sign in to leave a comment.