Hammered by Googlebot , getting flooded by invalid URLs on one domain

hrodrigues

• June 29, 2020 05:18

Hi all, I have one particular customer who is getting flooded by Googlebot, with requests to inexistent URLs. I have tried saying Google not to index my using meta tags, Disallow: on robots.txt, and on Google search console I requested clearing cache and temporary removal of the entire domain. None worked. I still see hundreds of requests 24x7 on my apache status, as attached picture shows. some of them enjoying keepalive timer. Blocking Googlebot ips on firewall work, but is NOT an option as this is a shared server, it would harm other server users by preventing their sites from getting crawled. How can I improve that situation, perhaps a modsecurity rule denying Googlebot /24 subnet or it"s User Agent to that particular domain? If yes, What would the rule look like? Please help

• 

  cPanelLauren

  • June 29, 2020 21:41

  Well you wouldn't want google to NOT index the site that would be bad for the site as a whole. I'd suggest looking at the suggestions from google for other users that have experienced issues with this like Change Googlebot crawl rate - Search Console Help

  0
• 

  hrodrigues

  • July 13, 2020 02:15

  Thanks a lot! Lowering the crawl rate at Google search console, reduced server load by half. You may close the thread

  0

Please sign in to leave a comment.