Introduction
You may experience slow load times or increased resource usage for a domain when Google is crawling it. If a site is being crawled by Google, "Googlebot" will appear in the user agent for requests in the domlogs for the site. They will look something like the following.
198.51.100.2 - - [22/Aug/2021:17:02:34 +0800] "GET / HTTP/2" 200 15807 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.119 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
Procedure
The following documentation covers changing the crawl rate settings at Google.
Comments
0 comments
Article is closed for comments.