modsec errors
Hello,
I have these errors in my modsec log file. Are they normal or should I worry?
I think what worries me is how they all say "[:error]" I don't think that's correct. It makes me think that perhaps modsecurity is configured incorrectly or something. If I look at /var/log/apache2/modsec_audit.log and go to the very end, I see stuff like:
If I go up to --6083872e-A-- and look through, I see a spider.html tried getting /robots.txt, then received a 404 (I don't have a robots.txt yet). So is the Rule processing failed normal as well? There's a whole bunch of them.
[Tue Aug 09 12:23:46.751981 2016] [:error] [pid 25520:tid 140358038025984] [client 180.76.15.12] ModSecurity: Warning. Match of "pm AppleWebKit Android" against "REQUEST_HEADERS:User-Agent" required. [file "/etc/apache2/conf.d/modsec_vendor_configs/OWASP/rules/REQUEST-20-PROTOCOL-ENFORCEMENT.conf"> [line "299"> [id "960015"> [rev "3"> [msg "Request Missing an Accept Header"> [severity "NOTICE"> [ver "OWASP_CRS/3.0.0"> [maturity "9"> [accuracy "8"> [tag "Host: example.com"> [tag "application-multi"> [tag "language-multi"> [tag "platform-multi"> [tag "attack-protocol"> [tag "OWASP_CRS/PROTOCOL_VIOLATION/MISSING_HEADER_ACCEPT"> [tag "WASCTC/WASC-21"> [tag "OWASP_TOP_10/A7"> [tag "PCI/6.5.10"> [hostname "example.com"> [uri "/"> [unique_id "V6oDkijphM-boQz4tY@byQAAAI0">
[Tue Aug 09 12:23:21.440134 2016] [:error] [pid 25483:tid 140358038025984] [client 180.76.15.141] ModSecurity: Warning. Match of "pm AppleWebKit Android" against "REQUEST_HEADERS:User-Agent" required. [file "/etc/apache2/conf.d/modsec_vendor_configs/OWASP/rules/REQUEST-20-PROTOCOL-ENFORCEMENT.conf"> [line "299"> [id "960015"> [rev "3"> [msg "Request Missing an Accept Header"> [severity "NOTICE"> [ver "OWASP_CRS/3.0.0"> [maturity "9"> [accuracy "8"> [tag "Host: example.com"> [tag "application-multi"> [tag "language-multi"> [tag "platform-multi"> [tag "attack-protocol"> [tag "OWASP_CRS/PROTOCOL_VIOLATION/MISSING_HEADER_ACCEPT"> [tag "WASCTC/WASC-21"> [tag "OWASP_TOP_10/A7"> [tag "PCI/6.5.10"> [hostname "example.com"> [uri "/"> [unique_id "V6oDeU-dXvZ3MKBwtByNrAAAAA0">
[Tue Aug 09 11:29:14.271924 2016] [:error] [pid 25520:tid 140358079985408] [client 180.76.15.17] ModSecurity: Warning. Match of "pm AppleWebKit Android" against "REQUEST_HEADERS:User-Agent" required. [file "/etc/apache2/conf.d/modsec_vendor_configs/OWASP/rules/REQUEST-20-PROTOCOL-ENFORCEMENT.conf"> [line "299"> [id "960015"> [rev "3"> [msg "Request Missing an Accept Header"> [severity "NOTICE"> [ver "OWASP_CRS/3.0.0"> [maturity "9"> [accuracy "8"> [tag "Host: example.com"> [tag "application-multi"> [tag "language-multi"> [tag "platform-multi"> [tag "attack-protocol"> [tag "OWASP_CRS/PROTOCOL_VIOLATION/MISSING_HEADER_ACCEPT"> [tag "WASCTC/WASC-21"> [tag "OWASP_TOP_10/A7"> [tag "PCI/6.5.10"> [hostname "example.com"> [uri "/"> [unique_id "V6n2yijphM-boQz4tY@brAAAAIk">
[Tue Aug 09 11:28:36.761229 2016] [:error] [pid 25516:tid 140358079985408] [client 180.76.15.34] ModSecurity: Warning. Match of "pm AppleWebKit Android" against "REQUEST_HEADERS:User-Agent" required. [file "/etc/apache2/conf.d/modsec_vendor_configs/OWASP/rules/REQUEST-20-PROTOCOL-ENFORCEMENT.conf"> [line "299"> [id "960015"> [rev "3"> [msg "Request Missing an Accept Header"> [severity "NOTICE"> [ver "OWASP_CRS/3.0.0"> [maturity "9"> [accuracy "8"> [tag "Host: example.com"> [tag "application-multi"> [tag "language-multi"> [tag "platform-multi"> [tag "attack-protocol"> [tag "OWASP_CRS/PROTOCOL_VIOLATION/MISSING_HEADER_ACCEPT"> [tag "WASCTC/WASC-21"> [tag "OWASP_TOP_10/A7"> [tag "PCI/6.5.10"> [hostname "example.com"> [uri "/"> [unique_id "V6n2pAhxWZGr@fprNLDS6AAAAEk">
I think what worries me is how they all say "[:error]" I don't think that's correct. It makes me think that perhaps modsecurity is configured incorrectly or something. If I look at /var/log/apache2/modsec_audit.log and go to the very end, I see stuff like:
--6083872e-H--
Message: Rule processing failed.
Apache-Handler: default-handler
...
If I go up to --6083872e-A-- and look through, I see a spider.html tried getting /robots.txt, then received a 404 (I don't have a robots.txt yet). So is the Rule processing failed normal as well? There's a whole bunch of them.
-
I think what worries me is how they all say "[:error]" I don't think that's correct. It makes me think that perhaps modsecurity is configured incorrectly or something.
Hello, You can verify if the request was actually blocked by reviewing the entry in the following log file:/usr/local/apache/logs/modsec_audit.log
Thank you.0 -
Hi Spork Schivago, I was getting the same entries in the logs, and non of the rules seemed to be being processed. Now I have to admit I did not spend the amount of time that I perhaps should have done in diagnosing why I was getting these errors - what I did find was switching OFF the rules/REQUEST-10-IP-REPUTATION.conf in Home " Security Center " ModSecurity" Vendors " Select Vendor Rule Sets seemed to fix everything, and there were no more errors and everything worked as I expected. 0 -
Hello, You can verify if the request was actually blocked by reviewing the entry in the following log file:
/usr/local/apache/logs/modsec_audit.log
Thank you.
Okay. /usr/local/apache/logs is a symbolic link to /etc/apache2/logs which is a symbolic link to /var/log/apache2, so the modsec_audit.log in the /var/log/apache2 directory is the correct modsec_audit.log that I've been looking at. This is what I see for those specific entries I listed in post #1. There's a ton of them and that one IP address has connected multiple times over multiple days. This worries me a bit also. I don't see why csf / lfd isn't blocking the IP address. I would think modsec would at least permanently block the address so it couldn't continue to try and attack my server in the future. Obviously, I'm not understanding something here. Blocking the hacking attempts is good, but the attacking IP address should be banned permanently so they cannot continue to attack. I thought this was what modsecurity was for.--abde3a2d-A-- [09/Aug/2016:12:23:46 --0400] V6oDkijphM-boQz4tY@byQAAAI0 180.76.15.12 49267 104.238.117.105 80 --abde3a2d-B-- GET / HTTP/1.1 Host: example.com Accept-Language: zh-cn Connection: close User-Agent: Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html) --abde3a2d-F-- HTTP/1.1 301 Moved Permanently Location: https://example.com/ Content-Length: 227 Connection: close Content-Type: text/html; charset=iso-8859-1 --abde3a2d-E-- --abde3a2d-H-- Message: Rule processing failed. Message: Warning. Match of "pm AppleWebKit Android" against "REQUEST_HEADERS:User-Agent" required. [file "/etc/apache2/conf.d/modsec_vendor_configs/OWASP/rules/REQUEST-20-PROTOCOL-ENFORCEMENT.conf"> [line "299"> [id"960015"> [rev "3"> [msg "Request Missing an Accept Header"> [severity "NOTICE"> [ver "OWASP_CRS/3.0.0"> [maturity "9"> [accuracy "8"> [tag "Host: example.com"> [tag "application-multi"> [tag "language-multi"> [tag "platform-multi"> [tag "attack-protocol"> [tag "OWASP_CRS/PROTOCOL_VIOLATION/MISSING_HEADER_ACCEPT"> [tag "WASCTC/WASC-21"> [tag "OWASP_TOP_10/A7"> [tag "PCI/6.5.10"> Stopwatch: 1470759826750832 1928 (- - -) Stopwatch2: 1470759826750832 1928; combined=1317, p1=207, p2=746, p3=0, p4=89, p5=203, sr=69, sw=72, l=0, gc=0 Response-Body-Transformed: Dechunked Producer: ModSecurity for Apache/2.9.0 (http://www.modsecurity.org/); OWASP_CRS/3.0.0. Server: Apache Engine-Mode: "ENABLED" --abde3a2d-Z--
That's one of the ones from the first post. It's a pain copying and pasting the text because of the very long lines, so I only did one of the entries but they're all similar. How can I tell from that one entry there if the block actually succeeded or not? To me, the whole Rule processing failed says it did not succeed. If you'd like, I can copy and paste the very long OWASP rule ID #: 960015. Just doing a quick search through the modsec_audit.log file, I searched for Rule processing failed and see a whole bunch. But from the very first rule processing failed search in the file, I'm thinking the rule that fails is not the rules that it's reporting on. For instance, I see the message of Rules processing failed, then I see a bunch of rules after that which seems to of succeeded. Just to make sure I'm saying this properly, I think with the one I put in the code tags, id # "960015" succeeded but something before that failed. There's nothing the modsec_debug.log file. I wonder if this could be with the local loopback ip address rule I added a while back with the help of some cPanel users. This is the rule I added:# ... # REQUEST-01-COMMON-EXCEPTIONS.conf # ... # # Exception for cPanel scripts (whm-server-status, etc) # SecRule REMOTE_ADDR "^127.0.0.1" phase:1,nolog,allow,id:'981022',ctl:ruleEngine=off
I don't think this is the problem though. id # 960015 gets processed before rule id # 981022, right? If there was something wrong with my rule (it worked before), then I should see the message about the failure after the rule id # 960015 is processed, not before, right? Thanks.0 -
Hi Spork Schivago, I was getting the same entries in the logs, and non of the rules seemed to be being processed. Now I have to admit I did not spend the amount of time that I perhaps should have done in diagnosing why I was getting these errors - what I did find was switching OFF the rules/REQUEST-10-IP-REPUTATION.conf in Home " Security Center " ModSecurity" Vendors " Select Vendor Rule Sets seemed to fix everything, and there were no more errors and everything worked as I expected.
Thank you. I guess I'll have to go through one by one, turning off rules to try and figure out which file contains the offending rule. Does anyone have any suggestions on how I could try to get modsec to trigger? For example, let's say I disable the REQUEST-01-COMMON-EXCEPTIONS.conf files...how could I get modsecurity2 to trip and start processing rules, to see if the message pops up?0 -
I have to go to a class here so I won't be back until around 8PM or so. But when I was searching (I didn't go through all of the entries because the file is very big), I noticed every single one that I looked at that had the rule processing failed show REQUEST-20-PROTOCOL-ENFORCEMENT.conf rules processed. Perhaps the bad rule is in REQUEST-20-PROTOCOL-ENFORCEMENT.conf The files in /etc/apache2/conf.d/modsec_vendor_configs/OWASP/rules/ show they were last accessed, modified and changed on 2016-08-09 @ 01:06:56 This is around the time upcp gets ran on my system. Was there an update to the OWASP rules? Did something recently change? Wish I had an old copy of the directory so I could compare the difference between the files. Thanks. 0 -
I enabled debug level 4 for modsec temporarily and I see some issues. For the rule processing failed one, right before the message, I see in the modsec_debug.log file: Reciper: Invoking rule 7f9da819d868; [file "/etc/apache2/conf.d/modsec_vendor_configs/OWASP/rules/REQUEST-10-IP-REPUTATION.conf"> [line "60"> [id "981138">. Transformation completed in 0 usec. Executing operator "rbl" with param "dnsbl.httpbl.org" against TX:real_ip. RBL httpBl called but no key defined: set SecHttpBlKey Operator completed in 25 usec. Operator error: RBL httpBl called but no key defined: set SecHttpBlKey Rule returned -1. Rule processing failed.
It shows the person (or webcrawler) was visiting cpanel.example.com. I'd like to think the cpanel, whm, webmail, etc subdomains have a valid robots.txt so search engines don't return results for that stuff. If not, I'd like to setup a robots.txt for that. Any idea how I go about setting SecHttpBlKey and what I'd set it to? I believe this has something to do with Project HoneyPot, correct? Thanks!0 -
Okay, I got my Project HoneyPot HTTP:BL key and I put it in WHM's ModSec Project HoneyPot Http:BL API Key's section, to set the SecHttpBlKey. Now, I just gotta figure out how to download and setup the Google Safe Browsing Database and the Geolocation Database SecGeoLookupDb. Then I think I'll be all set. Thanks. 0 -
For the Geolocation Database, does cPanel support the GeoLite2 Databases or should I be using the legacy databases? 0 -
Hi Spork Schivago, I was getting the same entries in the logs, and non of the rules seemed to be being processed. Now I have to admit I did not spend the amount of time that I perhaps should have done in diagnosing why I was getting these errors - what I did find was switching OFF the rules/REQUEST-10-IP-REPUTATION.conf in Home " Security Center " ModSecurity" Vendors " Select Vendor Rule Sets seemed to fix everything, and there were no more errors and everything worked as I expected.
Hello, I think perhaps the error message I was getting was because I did not have a Project HoneyPot API key configured under the Modsecurity Configuration. I would have thought that without the key, modsecurity2 wouldn't be trying to use the API. Once I obtained a free key and plopped it in, I was good to go. Perhaps you should consider getting one as well? They're free and the HTTP:Bl stuff is much like the normal Project HoneyPot stuff, but instead of being used for spam, it's used for bad people. For example, let's say my IP address, 192.168.2.2, is attacking lots of sites. If I understand everything correctly, my IP address will get added to the HTTP:Bl database. When I try connecting to someone's site who has modsecurity2 configured and running, along with an HTTP:Bl API key, I'll automatically be blocked, instead of being allowed to the their site.0 -
Got the key and configured it and switched that rule-set back in - will update tomorrow how we are getting on. Thank you very much. 0 -
Overnight logs and modsec rewrites and blocks indicate that it is now working perfectly with all the OSWAP vendor rules enabled. Thank you very much for the tip about the Project Honey Pot Http:BL API Key - perhaps the docs could be updated to mention that the rules/REQUEST-10-IP-REPUTATION.conf needs this key configured to run? 0 -
Overnight logs and modsec rewrites and blocks indicate that it is now working perfectly with all the OSWAP vendor rules enabled. Thank you very much for the tip about the Project Honey Pot Http:BL API Key - perhaps the docs could be updated to mention that the rules/REQUEST-10-IP-REPUTATION.conf needs this key configured to run?
You're very welcome. I suspect perhaps it's a bug? I would think without the API key configured properly, modsec wouldn't try using rules based on it. There's a new version of modsecurity2 coming out, modsecurity 3.0. When that comes, hopefully cPanel implements it. Currently, I get lots of messages in my log files about some mutex failing. This should be fixed in modsecurity 3.0. Also, the way they're writing it, there shouldn't be anymore conflicts between modsec and modRUID2. I have problems with my system now. Chkrootkit found something new, something it never found, 2 hidden processes. I believe it's a false warning, it discovered the LKM rootkit....but there's some real weird stuff going on now so I'm going to open a new thread about it..0
Please sign in to leave a comment.
Comments
12 comments