OWASP Blocking Google

keat63

• March 18, 2015 06:55

It seems every few days, i'm seeing Goolge being blocked by one OWASP rule after the other. Some of the errors look quite alarming. Of course blocking good bots is not good for SEO, so rather than opening all these rules, is there a way to configure this to allow the good bots.

• 

  cPanelMichael

  • March 18, 2015 14:37

  Hello, I suggest using "WHM Home " Security Center " Report ModSecurity Hit" to report the rule if it's not working as intended. Or, you may want to disable the rule completely. I'll leave this thread open for other users to add feedback regarding adding a custom rule to allow specific bots. Thank you.

  0
• 

  Marius

  • March 19, 2015 08:25

  cPanelMichael: from cpanel documentation:

  0
• 

  cPanelMichael

  • March 19, 2015 12:55

  Note: This option does not appear if the vendor does not accept reports So OWASP does not accept to report rule(latest WHM installed).

  
  Could you elaborate on how you came to the conclusion that OWASP is not accepting reports? There was a problem in the past, but it's now addressed:

  0
• 

  Marius

  • March 19, 2015 19:30

  cPanelMichael you have right, now working to submit.(I've submitted...) The big problem is that CSF + Mod Security still ban Google bot even if I add in /etc/csf/csf.rignore the line: .googlebot.com Mod security ID who ban GOOGLE BOT: mod_security (id:981138) mod_security (id:981140) Till problem is solved by OWASP know someone a Mod Security rule to accept GOOGLE BOT?

  0
• 

  cPanelMichael

  • March 19, 2015 19:51

  You can disable the rule completely to prevent it from happening. Thank you.

  0
• 

  quizknows

  • March 19, 2015 20:12

  Depends what rules are being tripped. While disabling them will work, it might not be the best idea. That said, anything google is tripping is probably an erroneous or bad rule and relatively safe to whitelist. Anyway, whitelisting the googlebot user agent is a bad idea; lots of attack scripts use that hoping people have it whitelisted. What you would want to do is find a list of IPs that google bot is known to use, and a rule could be made from that to whitelist them. However, it is better to report or disable any rules google is tripping (AFTER making sure it's a legit google IP and not some bot stealing their user agent, since that's extremely common).

  0
• 

  vicos

  • March 29, 2015 20:07

  I just installed OWASP and have the same issue with Rule 981138 blocking Googlebots. I did some checking and it is an error in the way the rule is written. 981138 checks the IP address against the Project HoneyPot database (a DNS blacklist). So, I contacted them to ask if they knew if they Google listed. They said yes, but they clearly identify it as a search crawler: Any time the last octet of the response is a 0, the IP is indicated as a search engine crawler. You can find more information here:

  0
• 

  cPanelMichael

  • April 06, 2015 19:01

  Will reporting this here get this fixed or do I need to go somewhere else? Checking the DNS blacklist is very valuable, so having to disable it because of this glitch is a shame.

  
  You will need to use the "WHM Home " Security Center " Report ModSecurity Hit" option to report the rule if it's not working as intended. Thank you.

  0

Please sign in to leave a comment.