Question on cause of intermittent 403 errors
We are seeing some problems with Googlebot indexing, due to a CPanel server returning 403 responses from time to time.
For three Googlebot requests in a 24 hour period for a given URL, it might respond successfully to the first two - then with a 403 status code the third time (don't ask why Google is requesting the same page three times in a day, when thousands of other pages exist - we think this may be because of an issue a couple of months ago, when a site manager accidentally blocked Googlebot from all pages on the site, and Google probably suspects it will find 403s and is therefore going looking for them).
We've checked that CPanel does not have the firewall plugin.
We suspect there may be a pattern here related to server load - there is an import script that runs overnight for around an hour - however, to the best of our knowledge, Apache on CPanel would not return a 403 response when a server is under heavy load.
In case it's relevant the site uses Prestashop - without any additional firewall module (we suspected this as a cause, because we're aware firewall add-ons to websites can cause 403 errors).
A manual request of the page in browser always works - and checking the page's status within Google search console always works and says there are no problems.
The site uses Cloudflare so that was the prime suspect initially, but we've confirmed bot fight mode is off, and there is a top priority rule which turns off checks for known bots (that includes Googlebot) and then skips all security checks, as recommended in some online threads. We've also confirmed the 403 lines are logged on the server itself, which would not be the case if Cloudflare was blocking the traffic (or some other firewall sitting in front of the server was blocking the traffic).
Any ideas, or details of similar experiences / resolutions, would be welcome.
-
Hey there! When you say the "403 lines are logged on the server itself" are you seeing them directly in the Apache log (/etc/apache2/logs/error_log)? If so, do you have ModSecurity enabled and that is possible what is causing this? I'm obviously guessing, but we've gotta start somewhere.
0 -
Please find below a couple of examples from the log (I believe these were from the per-account log files, not root level):-
66.249.68.69 - - [31/May/2026:19:01:29 +0000] "GET /3401-DK-22606 HTTP/1.1" 200 24310 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/148.0.7778.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"66.249.68.64 - - [01/Jun/2026:01:01:29 +0000] "GET /3401-DK-22606 HTTP/1.1" 403 - "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/148.0.7778.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
Checking the mod_security log from that time period when the 403 response happens, we have 2x 301 redirects around this time, but for other URLs:
--8ca84d19-A--
[01/Jun/2026:01:00:49.928881 +0000] ahzZwRWwkMSSrTK5qH3ZnwAAAAM 216.244.66.234 10364 [...] 80
--8ca84d19-B--
GET /353-C13T04424010-CRC HTTP/1.1
[snip]
--8ca84d19-F--
HTTP/1.1 301 Moved Permanently
--c2ede127-A--
[01/Jun/2026:01:01:57.451894 +0000] ahzaBRv56-RISpDlWm7FOwAAAAc 216.244.66.234 12883 [...] 80
--c2ede127-B--
GET /16555-[snip]6534 HTTP/1.1
[snip]
--c2ede127-F--
HTTP/1.1 301 Moved Permanently
Since these are simple GET requests we didn't think Mod_security would be involved. I would agree if it was a GET request with querystring parameters, a POST request, or cookies - then it very likely could be responsible.
0 -
My only other idea would be if mod_evasive is present on the system, leading to restrictions from high-traffic connections, like a crawler bot:
https://support.cpanel.net/hc/en-us/articles/360052575294-About-mod-evasive-related-403-errors
If not, it might be best to create a ticket so this can be examined directly.
0
Please sign in to leave a comment.
Comments
3 comments