How to find which account is crawled by Bots?

Logesh K

• July 07, 2019 21:33

My server was on High load and high outgoing bandwidth. Upon checking with the iftop tool I found lot of IP's with hostname of the Bots - semrush, ahreafs, baidu were crawling the websites on my server. But I am unable to find out which account/domain was crawled by these bots. Can someone help me in this? Thanks

• 

  cPanelLauren

  • July 08, 2019 21:09

  Hi @Logesh K The domain access logs should give you this data, they can be found at /etc/apache2/domlogs/ for each domain.

  0
• 

  Logesh K

  • July 17, 2019 17:33

  Hi @Logesh K The domain access logs should give you this data, they can be found at /etc/apache2/domlogs/ for each domain.

  
  Thanks for the reply. But I don't find it easy to check the numerous files manually to find out the domain. Is there any way to find realtime data on this? Because once my server freezed due to ahrefs and semrush bots, while I was watching the bandwidth data using iftop.

  0
• 

  Henry Carter

  • July 31, 2019 07:52

  Hello

  0

Please sign in to leave a comment.