Apache Bug (Build 11.38.x)? (get /robots.txt)
We have a problem on 2 servers since ~july/august 2013, they have every day a lot of processes like this here:
945950 1/43/7079 G 1.80 68687 0 2.7 0.50 175.53 66.249.66.104 HOSTNAME:4 GET /robots.txt HTTP/1.1
872984 1/9/36597 G 0.19 11952 0 2.7 0.19 1167.38 66.249.66.36 HOSTNAME:4 GET /robots.txt HTTP/1.1
This results in heavy load (this processes take 100% cpu usage -> as nobody).
Is there a solution?
Add this IP's to the host access access control have no effect.
Only this "get /robots.txt processes need so much cpu.
-
Hello :) Do you have a third-party firewall application such as CSF installed on your system? If not, it may be a good idea to install a firewall so you can block IP addresses from accessing your server. You may also want to install Mod_Security to help with certain access attempts. Thank you. 0 -
The 66.249.66.* address space belongs to Google's web crawlers. Blocking them will have a negative effect on your (or your customer's) website ranking, obviously. You can add a meta tag to the website if you really want google to stop visiting. [QUOTE]
You can also register for Google's Webmaster Tools, and change how fast (and how often) the googlebot crawls your website.0
Please sign in to leave a comment.
Comments
2 comments