How can I check for web crawlers accessing my domain?

Introduction

Web crawlers often have the word 'bot' in their user agents title. And when they access your domain, just as any other entity might, an entry is left in the domlogs. You may parse the domlogs to check if this has occurred for your domain.

Procedure

You can parse your domlogs using a command similar to the following command, where you replace yourdomain.tld with your actual domain:

awk -F '"'  '$6 ~ /[bB]ot/ {split($0,x," "); print x[1]}' /var/log/apache2/domlogs/cptest.tld{,-ssl_log} | sort | uniq -c | sort -nr | head