modsec denies Amazon crawler access to JPG images with 403
Hi all, this is a similar post to https://support.cpanel.net/hc/en-us/community/posts/19163689316375-crawler-looks-malicious-but-attacker-says-not where a request for /robots.txt was denied, but there is no actual solution posted there and I'm still stuck.
I'm getting this in my apache error_log:
[Wed Apr 02 07:00:54.892667 2025] [security2:error] [pid 2508495:tid 2508533] [client 54.240.197.18:6730] [client 54.240.197.18] ModSecurity: Warning. Match of "rx ^(?:(?:\\\\*|[^
\\"(),\\\\/:;<=>?![\\\\x5c\\\\]{}]+)\\\\/(?:\\\\*|[^\\"(),\\\\/:;<=>?![\\\\x5c\\\\]{}]+))(?:\\\\s*+;\\\\s*+(?:(?:charset\\\\s*+=\\\\s*+(?:\\"?(?:iso-8859-15?|windows-1252|utf-8)\\
\\b\\"?))|(?:(?:c(?:h(?:a(?:r(?:s(?:e[^t\\"(),\\\\/:;<=>?![\\\\x5c\\\\]{}]|[^e\\"(),/:;<=>?![\\\\x5c ..." against "REQUEST_HEADERS:Accept" required. [file "/etc/apache2/conf.d/mod
sec_vendor_configs/OWASP3/rules/REQUEST-920-PROTOCOL-ENFORCEMENT.conf"] [line "1161"] [id "920600"] [msg "Illegal Accept header: charset parameter"] [data "text/html, image/gif, i
mage/jpeg, *; q=.2, */*; q=.2"] [severity "CRITICAL"] [ver "OWASP_CRS/3.3.7"] [tag "application-multi"] [tag "language-multi"] [tag "platform-multi"] [tag "attack-protocol"] [tag
"paranoia-level/1"] [tag "OWASP_CRS"] [hostname "www.premierrange.co.uk"] [uri "/images/variations/splashback_variations/digital/Blue-Bronze/Blue-Bronze-Without.jpg"] [unique_id "
Z-zSliJMHBcpUXqiDtqqWgAAAEs"]
The image being requested does exist, and I can request it from my browser or wget command line just fine. It look like Amazon's crawler is triggering an OWASP rule checking the Accept header and complaining about the charset.
I have modsec_audit.log logging the requests and I think this is a relevant request from Amazon:
--10417f44-A--
[02/Apr/2025:07:00:54.893238 +0100] Z-zSliJMHBcpUXqiDtqqWgAAAEs 54.240.197.18 6730 185.4.176.211 443
--10417f44-B--
GET /images/variations/splashback_variations/digital/Blue-Bronze/Blue-Bronze-Without.jpg HTTP/1.1
User-Agent: Java/1.8.0_442
Host: www.premierrange.co.uk
Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2
Connection: keep-alive
--10417f44-F--
HTTP/1.1 404 Not Found
Upgrade: h2,h2c
Connection: Upgrade
--10417f44-H--
Message: Warning. Match of "rx ^(?:(?:\\*|[^\"(),\\/:;<=>?![\\x5c\\]{}]+)\\/(?:\\*|[^\"(),\\/:;<=>?![\\x5c\\]{}]+))(?:\\s*+;\\s*+(?:(?:charset\\s*+=\\s*+(?:\"?(?:iso-8859-15?|windows-1252|utf-8)\\b\"?))|(?:(?:c(?:h(?:a(?:r(?:s(?:e[^t\"(),\\/:;<=>?![\\x5c\\]{}]|[^e\"(),/:;<=>?![\\x5c ..." against "REQUEST_HEADERS:Accept" required. [file "/etc/apache2/conf.d/modsec_vendor_configs/OWASP3/rules/REQUEST-920-PROTOCOL-ENFORCEMENT.conf"] [line "1161"] [id "920600"] [msg "Illegal Accept header: charset parameter"] [data "text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2"] [severity "CRITICAL"] [ver "OWASP_CRS/3.3.7"] [tag "application-multi"] [tag "language-multi"] [tag "platform-multi"] [tag "attack-protocol"] [tag "paranoia-level/1"] [tag "OWASP_CRS"]
Message: Access denied with code 403 (phase 2). Operator GE matched 5 at TX:anomaly_score. [file "/etc/apache2/conf.d/modsec_vendor_configs/OWASP3/rules/REQUEST-949-BLOCKING-EVALUATION.conf"] [line "94"] [id "949110"] [msg "Inbound Anomaly Score Exceeded (Total Score: 5)"] [severity "CRITICAL"] [ver "OWASP_CRS/3.3.7"] [tag "application-multi"] [tag "language-multi"] [tag "platform-multi"] [tag "attack-generic"]
Message: Warning. Operator GE matched 5 at TX:inbound_anomaly_score. [file "/etc/apache2/conf.d/modsec_vendor_configs/OWASP3/rules/RESPONSE-980-CORRELATION.conf"] [line "92"] [id "980130"] [msg "Inbound Anomaly Score Exceeded (Total Inbound Score: 5 - SQLI=0,XSS=0,RFI=0,LFI=0,RCE=0,PHPI=0,HTTP=0,SESS=0): individual paranoia level scores: 5, 0, 0, 0"] [ver "OWASP_CRS/3.3.7"] [tag "event-correlation"]
Apache-Error: [file "apache2_util.c"] [line 277] [level 3] [client 54.240.197.18] ModSecurity: Warning. Match of "rx ^(?:(?:\\\\\\\\*|[^\\\\"(),\\\\\\\\/:;<=>?![\\\\\\\\x5c\\\\\\\\]{}]+)\\\\\\\\/(?:\\\\\\\\*|[^\\\\"(),\\\\\\\\/:;<=>?![\\\\\\\\x5c\\\\\\\\]{}]+))(?:\\\\\\\\s*+;\\\\\\\\s*+(?:(?:charset\\\\\\\\s*+=\\\\\\\\s*+(?:\\\\"?(?:iso-8859-15?|windows-1252|utf-8)\\\\\\\\b\\\\"?))|(?:(?:c(?:h(?:a(?:r(?:s(?:e[^t\\\\"(),\\\\\\\\/:;<=>?![\\\\\\\\x5c\\\\\\\\]{}]|[^e\\\\"(),/:;<=>?![\\\\\\\\x5c ..." against "REQUEST_HEADERS:Accept" required. [file "/etc/apache2/conf.d/modsec_vendor_configs/OWASP3/rules/REQUEST-920-PROTOCOL-ENFORCEMENT.conf"] [line "1161"] [id "920600"] [msg "Illegal Accept header: charset parameter"] [data "text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2"] [severity "CRITICAL"] [ver "OWASP_CRS/3.3.7"] [tag "application-multi"] [tag "language-multi"] [tag "platform-multi"] [tag "attack-protocol"] [tag "paranoia-level/1"] [tag "OWASP_CRS"] [hostname "www.premierrange.co.uk"] [uri "/images/variations/splashback_variations/digital/Blue-Bronze/Blue-Bronze-Without.jpg"] [unique_id "Z-zSliJMHBcpUXqiDtqqWgAAAEs"]
Apache-Error: [file "apache2_util.c"] [line 277] [level 3] [client 54.240.197.18] ModSecurity: Access denied with code 403 (phase 2). Operator GE matched 5 at TX:anomaly_score. [file "/etc/apache2/conf.d/modsec_vendor_configs/OWASP3/rules/REQUEST-949-BLOCKING-EVALUATION.conf"] [line "94"] [id "949110"] [msg "Inbound Anomaly Score Exceeded (Total Score: 5)"] [severity "CRITICAL"] [ver "OWASP_CRS/3.3.7"] [tag "application-multi"] [tag "language-multi"] [tag "platform-multi"] [tag "attack-generic"] [hostname "www.premierrange.co.uk"] [uri "/images/variations/splashback_variations/digital/Blue-Bronze/Blue-Bronze-Without.jpg"] [unique_id "Z-zSliJMHBcpUXqiDtqqWgAAAEs"]
Apache-Error: [file "core.c"] [line 4935] [level 6] AH00128: File does not exist: /home/premier/public_html/403.shtml
Apache-Error: [file "apache2_util.c"] [line 277] [level 3] [client 54.240.197.18] ModSecurity: Warning. Operator GE matched 5 at TX:inbound_anomaly_score. [file "/etc/apache2/conf.d/modsec_vendor_configs/OWASP3/rules/RESPONSE-980-CORRELATION.conf"] [line "92"] [id "980130"] [msg "Inbound Anomaly Score Exceeded (Total Inbound Score: 5 - SQLI=0,XSS=0,RFI=0,LFI=0,RCE=0,PHPI=0,HTTP=0,SESS=0): individual paranoia level scores: 5, 0, 0, 0"] [ver "OWASP_CRS/3.3.7"] [tag "event-correlation"] [hostname "www.premierrange.co.uk"] [uri "/403.shtml"] [unique_id "Z-zSliJMHBcpUXqiDtqqWgAAAEs"]
Action: Intercepted (phase 2)
Apache-Handler: default-handler
Stopwatch: 1743573654892512 753 (- - -)
Stopwatch2: 1743573654892512 753; combined=447, p1=195, p2=211, p3=0, p4=0, p5=41, sr=19, sw=0, l=0, gc=0
Producer: ModSecurity for Apache/2.9.7 (http://www.modsecurity.org/); OWASP_CRS/3.3.7.
Server: Apache
Engine-Mode: "ENABLED"
--10417f44-Z--
So the Accept header that OWASP is complaining about contains "text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2" which seems fine to me.
How can I change or relax my config (in apache or modsec rules) to allow Amazon access? I'm not very familiar with the OWASP configuration. I presume simply editing the files on disk will result in them being overwritten next time the owning package is updated.
Thank you!
-
I found that you can disable that rule by its id 920600 globally with a single line in the apache conf:
SecRuleRemoveById 920600
I have small config files in /etc/apache2/conf.d/userdata/ssl/2_4/<username>/<domainname> and can use the following three commands to rebuild the apache config and restart the server:
# /scripts/rebuildhttpdconf
# /scripts/restartsrv_apache
# /scripts/restartsrv_apache_php_fpmIt would be nice to use a more precise rule to only exclude the Amazon crawler, but it doesn't seem to use a sensible user agent or other string I can identify it by ("User-Agent: Java/1.8.0_442", really Amazon?).
Open to any other suggestions, my inventory import is now succeeding 100% as Amazon can now request my images OK.
0 -
Hey there! From the cPanel side of things, whitelisting the rule is the best option I would have available. Some users have went deeper into the regex of the rule and customized that:
but that isn't something we'd support on our side. You're welcome to try it if you want to go that far, but I can't say if it will cause other issues or not.
1
Please sign in to leave a comment.
Comments
2 comments