ModSecurity: Rule processing failed.

CanSpace

February 09, 2015 17:57

Yet another ModSecurity thread... So as per our other recent post, we've recently switched from our own customized version of the AtomiCorp rules to the OWASP rules provided by cPanel. On all of our servers, /usr/local/apache/logs/error_log is filled with these:

[Mon Feb 09 18:51:54 2015] [error] [client xxxx] ModSecurity: Rule processing failed. [hostname "xxxx"> [uri "xxxx"> [unique_id "VNlIGsBjLFwAACtCD9YAAAAb">
[Mon Feb 09 18:51:55 2015] [error] [client xxxx] ModSecurity: Geo Lookup: Failed to lock proc mutex: Identifier removed [hostname "xxxx"> [uri "/"> [unique_id "VNlIG8BjLFwAACp6PnUAAAAW">
[Mon Feb 09 18:51:55 2015] [error] [client xxxx] ModSecurity: Geo Lookup: Failed to lock proc mutex: Identifier removed [hostname "xxxx"> [uri "/"> [unique_id "VNlIG8BjLFwAACp6PnUAAAAW">
[Mon Feb 09 18:51:56 2015] [error] [client xxxx] ModSecurity: Rule processing failed. [hostname "xxxx"> [uri "xxxx"> [unique_id "VNlIHMBjLFwAACtGEKAAAAAJ">
[Mon Feb 09 18:52:00 2015] [error] [client xxxx] ModSecurity: Rule processing failed. [hostname "xxxx"> [uri "xxxx"> [unique_id "VNlIIMBjLFwAACtKEY4AAAAL">
[Mon Feb 09 18:52:01 2015] [error] [client xxxx] ModSecurity: Rule processing failed. [hostname "xxxx"> [uri "xxxx"> [unique_id "VNlIIcBjLFwAACviGXEAAAAG">
[Mon Feb 09 18:52:01 2015] [error] [client xxxx] ModSecurity: Rule processing failed. [hostname "xxxx"> [uri "xxxx"> [unique_id "VNlIIcBjLFwAACviGXIAAAAG">

This particular server is a brand new server with a fresh copy of cPanel, and actually never had the AtomiCorp rules in place to begin with - we just installed the OWASP as a "ModSecurity Vendor". If I click the ModSecurity Tools link I do see a ton of hits being processed correctly... so I'm not sure what's going on here. I have a ticket open with cPanel - they cleared our custom whitelist but the errors persist, so they've escalated the ticket. This is happening on all our servers - so I can't imaging we're the only ones having this issue...

Comments

49 comments

quizknows

March 02, 2015 15:22
A buddy of mine says he drastically reduced (and possibly eliminated) these messages by disabling the rule that does the GeoIP lookup. Seems to be consistent with the post above.
0
cPanelMichael

March 03, 2015 16:38
]Just for clarification, when you say that there is an internal case and give us the number, is there a way that we can monitor the progress or ideally get some work around or additional diagnostic. Still have all modsecurity turned off as to all the problems this has caused, but do not think that is a interim reasonable solution.

The internal case is not viewable to the public. It's provided as a reference so you can monitor our change log and determine when a resolution or change has been implemented. There's currently no update to this case, but I will update this thread with more information should it become available. Thank you.
0
keat63

March 03, 2015 17:48
It does appear that disabling rule 900050 stops this error. ModSecurity: Geo Lookup: Failed to lock proc mutex: Permission denied. I do have two others though, which i assumed were related, but now maybe not. ModSecurity: Audit log: Failed to create subdirectories: /usr/local/apache/logs/modsec_audit/user/20150303/20150303-1848 (Read-only file system) and ModSecurity: collection_store: Failed to access DBM file /var/cpanel/secdatadir/ip
0
Solokron

April 08, 2015 00:13
Two years later and we are still seeing this issue with modsecurity and ruid2? What a damn headache! Why has this not been corrected by anyone?
0
sonicthoughts

April 08, 2015 00:36
Two years later and we are still seeing this issue with modsecurity and ruid2? What a damn headache! Why has this not been corrected by anyone?

I know cPanel has competing priorities, but after wasting hours on this (GeoIP rule didn't do it for me), placing a support ticket that said it was out of scope to config modsecurity and watching this thread, the tradgedy is that people give up and leave modsec disabled. Including modsec seems to be a priority for cPanel, hope making it work (with common configs) is also a priority ....
0
cPanelMichael

April 10, 2015 17:35
Hello, Much of the issue is related to limitations with Mod_Security itself with regard to shared storage and handlers like Mod_ruid2 and MPM-ITK. It's not something that cPanel can really solve, but that being said, the case is still open in order to find the best possible way to avoid these types of issues for customers. Thank you.
0
Maarten de Boer

April 10, 2015 20:52
Disabling rule 900050 does the trick, if you have a recent version of WHM you can do so by searching for 900050 in the ModSecurity Tools page search bar. If you can't find it or don't have a recent version, add this directive to your apache configuration: SecRuleRemoveById 900050
0
freedomizer

April 21, 2015 09:17
if there is no way to disable the rules that make mod_security access /var/cpanel/secdatadir/ip can we at least supress the logging of the error: ModSecurity: collection_store: Failed to access DBM file "/var/cpanel/secdatadir/ip": Permission denied this error is slowly taking out weeks or months of SSD life and is indeed logging excessively causing lots of IO overhead
0
quizknows

April 21, 2015 19:25
/var/cpanel/secdatadir/ip is used for storing collections data. You would have to find the rule that uses the "initcol" function for "ip," and comment out the rule entirely so the collection is not made to begin with. This will obviously break any rules that need the collection data, but most webapp defence rules (aside from brute force) don't use collections much. If you are using the WHM vendor management for the automated rule updates, you would want to disable updates for the file that initiates the collections if you do this.
0
freedomizer

April 23, 2015 07:52
Hello thank you for the idea I looked under /usr/local/apache/conf/modsec_vendor_configs/OWASP/modsecurity_crs_10_setup.conf and seems like the following rules use IP collections: 900018, 900020, 900021 After disabling these rules, the "Failed to access DBM file "/var/cpanel/secdatadir/ip" errors stopped. I would rather disable a few rules and have at least partial security than disabling mod_security globally. note: mod_ruid2 is enabled.
# # -- [[ Global and IP Collections ]] ----------------------------------------------------- # # Create both Global and IP collections for rules to use # There are some CRS rules that assume that these two collections # have already been initiated. # SecRule REQUEST_HEADERS:User-Agent "^(.*)$" "id:'900018', phase:1, t:none,t:sha1,t:hexEncode, setvar:tx.ua_hash=%{matched_var}, nolog, pass" SecRule REQUEST_HEADERS:x-forwarded-for "^\b(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})\b" "id:'900019', phase:1, t:none, capture, setvar:tx.real_ip=%{tx.1}, nolog, pass" SecRule &TX:REAL_IP "!@eq 0" "id:'900020', phase:1, t:none, initcol:global=global, initcol:ip=%{tx.real_ip}_%{tx.ua_hash}, nolog, pass" SecRule &TX:REAL_IP "@eq 0" "id:'900021', phase:1, t:none, initcol:global=global, initcol:ip=%{remote_addr}_%{tx.ua_hash}, setvar:tx.real_ip=%{remote_addr}, nolog, pass" # # Set the SecCollectionTimeout directive to a lower setting (default # is 1 hour). Reducing this setting increases performance by cleaning # out old/stale entries. # Ref: https://github.com/SpiderLabs/ModSecurity/wiki/Reference-Manual#wiki-SecCollectionTimeout # Using 600 (10 mins). SecCollectionTimeout 600
0
sonicthoughts

April 27, 2015 18:07
FYI - related CPANEL issues viz. modsec on github:
0
bertelschmitt

December 07, 2015 08:19
I have successfully turned off all my "rule processing failed" errs by: WHM > Security Center > Modsecurity Vendors Edit: OWASP ModSecurity Core Rule Set Turn off: rules/REQUEST-10-IP-REPUTATION.conf Keep all others All quiet in the logs! Nothing lost for me anyway. In my book, shutting out whole countries is stupid and racist anyway.
0
SecondSight

August 03, 2016 19:13
Hello : I'm having this error messages problem on my server (Centos with WHM/CPanel). If I turn off rules/REQUEST-10-IP-REPUTATION.conf, I no longer have these error messages, but I would like to find the rules which cause the problems. Disabling 900018 and 900020 is useless. Disabling 900021 makes all error messages disappear except this one : [client 66.249.93.95] ModSecurity: Rule processing failed. 66.249.93.95 is linked to google-proxy-66-249-93-95.google.com. Do you know which rule I should disable to remove this "[client 66.249.93.95] ModSecurity: Rule processing failed" error ? Or, how should I proceed to find it easily ? Thank you. :)
0
Solokron

August 03, 2016 21:01
If it is a part of the above chain of rules then this would make sense. With configserver and modsec monitoring enabled in it you do not have to worry about having the the ip reputation ruleset missing as much. You can also jump into your csf/csf.blocklists file and add some supplemental blocklists such as the following:
# Name: FireHol # FireHOL IP Lists | IP Blacklists | IP Reputation Feeds FIREHOL|86400|0|https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/firehol_level1.netset # Name: ALTTOR # Category: anonymizers # Maintainer: Tor Network Status # Maintainer URL: TorStatus - Tor Network Status # Information: TOR Exit Nodes List ALTTOR|86400|0|http://torstatus.blutmagie.de/ip_list_exit.php/Tor_ip_list_EXIT.csv # Name: BAMBENEK_C2 # Category: malware # Maintainer: Bambenek Consulting # Maintainer URL: OSINT Feeds from Bambenek Consulting # Information: Master feed of known, active and non-sinkholed C&Cs IP addresses BAMBENEK_C2|86400|0|http://osint.bambenekconsulting.com/feeds/c2-ipmasterlist.txt # Name: BDEALL # Category: attacks # Maintainer: Blocklist.de # Maintainer URL: www.blocklist.de -- Fail2Ban-Reporting Service (we sent Reports from Attacks on Postfix, SSH, Apache-Attacks, Spambots, irc-Bots, Reg-Bots, DDos and more) from Fail2Ban via X-ARF. # Information: Blocklist.de attacking IP addresses (all) BDEALL|86400|0|http://lists.blocklist.de/lists/all.txt # Name: BDS_ATIF # Category: reputation # Maintainer: Binary Defense Systems # Maintainer URL: Binary Defense Systems | Defend. Protect. Secure. # Information: Artillery Threat Intelligence Feed and Banlist Feed BDS_ATIF|172800|0|https://www.binarydefense.com/banlist.txt # Name: BFB # Category: attacks # Maintainer: Daniel Gerzo # Maintainer URL: Daniel Gerzo " BruteForceBlocker # Information: BruteForceBlocker IP List BFB|86400|0|http://danger.rulez.sk/projects/bruteforceblocker/blist.php # Name: BI_ANY_2_30D # Category: attacks # Maintainer: BadIPs.com # Maintainer URL: https://www.badips.com/ # Information: Bad IPs in category any with score above 2 and age less than 30d BI_ANY_2_30D|172800|0|https://www.badips.com/get/list/any/2?age=30d # Name: BI_BRUTEFORCE_2_30D # Category: attacks # Maintainer: BadIPs.com # Maintainer URL: https://www.badips.com/ # Information: Bad IPs in category bruteforce with score above 2 and age less than 30d BI_BRUTEFORCE_2_30D|172800|0|https://www.badips.com/get/list/bruteforce/2?age=30d # Name: BI_PROXY_2_30D # Category: attacks # Maintainer: BadIPs.com # Maintainer URL: https://www.badips.com/ # Information: Bad IPs in category proxy with score above 2 and age less than 30d BI_PROXY_2_30D|172800|0|https://www.badips.com/get/list/proxy/2?age=30d # Name: BI_SSH_2_30D # Category: attacks # Maintainer: BadIPs.com # Maintainer URL: https://www.badips.com/ # Information: Bad IPs in category ssh with score above 2 and age less than 30d BI_SSH_2_30D|172800|0|https://www.badips.com/get/list/ssh/2?age=30d # Name: BLOCKLIST_DE # Category: attacks # Maintainer: Blocklist.de # Maintainer URL: https://www.blocklist.de/ # Information: IPs that have been detected by fail2ban in the last 48 hours BLOCKLIST_DE|86400|0|http://lists.blocklist.de/lists/all.txt # Name: BLOCKLIST_DE_APACHE # Category: attacks # Maintainer: Blocklist.de # Maintainer URL: https://www.blocklist.de/ # Information: All IP addresses which have been reported within the last 48 hours as having run attacks on the service Apache, # Apache-DDOS, RFI-Attacks. BLOCKLIST_DE_APACHE|86400|0|http://lists.blocklist.de/lists/apache.txt # Name: BLOCKLIST_DE_BRUTEFORCE # Category: attacks # Maintainer: Blocklist.de # Maintainer URL: https://www.blocklist.de/ # Information: All IPs which attacks Joomlas, Wordpress and other Web-Logins with Brute-Force Logins. BLOCKLIST_DE_BRUTEFORCE|86400|0|http://lists.blocklist.de/lists/bruteforcelogin.txt # Name: BLOCKLIST_DE_STRONGIPS # Category: attacks # Maintainer: Blocklist.de # Maintainer URL: https://www.blocklist.de/ # Information: All IPs which are older then 2 month and have more then 5.000 attacks. BLOCKLIST_DE_STRONGIPS|86400|0|http://lists.blocklist.de/lists/strongips.txt # Name: ET_COMPROMISED # Category: attacks # Maintainer: Emerging Threats # Maintainer URL: http://www.emergingthreats.net/ # Information: Compromised hosts ET_COMPROMISED|86400|0|http://rules.emergingthreats.net/blockrules/compromised-ips.txt # Name: ET_DSHIELD # Category: attacks # Maintainer: Emerging Threats # Maintainer URL: http://www.emergingthreats.net/ # Information: Dshield blocklist ET_DSHIELD|86400|0|http://rules.emergingthreats.net/fwrules/emerging-PIX-DSHIELD.rules # Name: ET_TOR # Category: anonymizers # Maintainer: Emerging Threats # Maintainer URL: http://www.emergingthreats.net/ # Information: Of TOR network IPs ET_TOR|86400|0|http://rules.emergingthreats.net/blockrules/emerging-tor.rules # Name: FEODO # Category: malware # Maintainer: Abuse.ch # Maintainer URL: https://feodotracker.abuse.ch/ # Information: Trojan includes IPs which are being used by Feodo (also known as Cridex or Bugat) which commits ebanking fraud FEODO|86400|0|https://feodotracker.abuse.ch/blocklist/?download=ipblocklist # Name: GREENSNOW # Category: attacks # Maintainer: GreenSnow.co # Maintainer URL: https://greensnow.co/ # Information: Is a team harvesting a large number of IPs from different computers located around the world. GreenSnow is comparable # with SpamHaus.org for attacks of any kind except for spam. Their list is updated automatically and you can withdraw at # any time your IP address if it has been listed. Attacks / bruteforce that are monitored are: Scan Port, FTP, POP3, # mod_security, IMAP, SMTP, SSH, cPanel, etc. GREENSNOW|86400|0|http://blocklist.greensnow.co/greensnow.txt # Name: MALC0DE # Category: malware # Maintainer: malc0de.com # Maintainer URL: http://malc0de.com/ # Information: Malicious IPs of the last 30 days MALC0DE|172800|0|http://malc0de.com/bl/IP_Blacklist.txt

Hello : I'm having this error messages problem on my server (Centos with WHM/CPanel). If I turn off rules/REQUEST-10-IP-REPUTATION.conf, I no longer have these error messages, but I would like to find the rules which cause the problems. Disabling 900018 and 900020 is useless. Disabling 900021 makes all error messages disappear except this one : [client 66.249.93.95] ModSecurity: Rule processing failed. 66.249.93.95 is linked to google-proxy-66-249-93-95.google.com. Do you know which rule I should disable to remove this "[client 66.249.93.95] ModSecurity: Rule processing failed" error ? Or, how should I proceed to find it easily ? Thank you. :)
0
hrace009

August 28, 2016 22:34
Hi, I use disable rules bellow on my previous control panel (non cPanel), anyone can give me dirrection where to put it at cPanel?
## Rules for the CWP ## SecRuleRemoveById 960017 SecRuleRemoveById 960015 SecRuleRemoveById 960009 ######################################## ## Removed Rules for Joomla, WordPress and Drupal CMSs ## ######################################## ## Joomla ## SecRuleRemoveById 960024 SecRuleRemoveById 950120 SecRuleRemoveById 981173 SecRuleRemoveById 950901 SecRuleRemoveById 981257 SecRuleRemoveById 981245 SecRuleRemoveById 973338 SecRuleRemoveById 973300 SecRuleRemoveById 973304 SecRuleRemoveById 973333 SecRuleRemoveById 973333 ## Drupal ## SecRuleRemoveById 981231 ## Removed rules for the webftp_simple ## SecRuleRemoveById 950922 SecRuleRemoveById 981000 SecRuleRemoveById 950109 ## phpMyAdmin ## SecRuleRemoveById 981205 SecRuleRemoveById 970901 SecRuleRemoveById 960904 SecRuleRemoveById 960915 SecRuleRemoveById 981318 SecRuleRemoveById 981320 SecRuleRemoveById 981240 ## Wordpress ## SecRuleRemoveById 981242 SecRuleRemoveById 981246 SecRuleRemoveById 981243 SecRuleRemoveById 959073 SecRuleRemoveById 958030 SecRuleRemoveById 970009 SecRuleRemoveById 973306 SecRuleRemoveById 973344 SecRuleRemoveById 981172 SecRuleRemoveById 981317 SecRuleRemoveById 958291 SecRuleRemoveById 970903 SecRuleRemoveById 950010 SecRuleRemoveById 981319 SecRuleRemoveById 950007 SecRuleRemoveById 950001 SecRuleRemoveById 959070 SecRuleRemoveById 981249 SecRuleRemoveById 973336 ## NEW WORDPRESS ## SecRuleRemoveById 970003 SecRuleRemoveById 981240 SecRuleRemoveById 981256 SecRuleRemoveById 973301 SecRuleRemoveById 981241 SecRuleRemoveById 973347 SecRuleRemoveById 981248 SecRuleRemoveById 970015 SecRuleRemoveById 973335 SecRuleRemoveById 973334 SecRuleRemoveById 973332 SecRuleRemoveById 981318 SecRuleRemoveById 981244 SecRuleRemoveById 960035 SecRuleRemoveById 960008 SecRuleRemoveById 960915 SecRuleRemoveById 950907 SecRuleRemoveById 950000 SecRuleRemoveById 981001 SecRuleRemoveById 950103 SecRuleRemoveById 960006 SecRuleRemoveById 958057 SecRuleRemoveById 959072 SecRuleRemoveById 981277 SecRuleRemoveById 910006 SecRuleRemoveById 950000 SecRuleRemoveById 950005 SecRuleRemoveById 950006 SecRuleRemoveById 950117 SecRuleRemoveById 950907 SecRuleRemoveById 958039 SecRuleRemoveById 958051 SecRuleRemoveById 959006 SecRuleRemoveById 960008 SecRuleRemoveById 960010 SecRuleRemoveById 960011 SecRuleRemoveById 960012 SecRuleRemoveById 960035 SecRuleRemoveById 960335 SecRuleRemoveById 960904 SecRuleRemoveById 960915 SecRuleRemoveById 970003 SecRuleRemoveById 970015 SecRuleRemoveById 973301 SecRuleRemoveById 973302 SecRuleRemoveById 973316 SecRuleRemoveById 973330 SecRuleRemoveById 973331 SecRuleRemoveById 973332 SecRuleRemoveById 973334 SecRuleRemoveById 973335 SecRuleRemoveById 973347 SecRuleRemoveById 981172 SecRuleRemoveById 981240 SecRuleRemoveById 981241 SecRuleRemoveById 981244 SecRuleRemoveById 981248 SecRuleRemoveById 981255 SecRuleRemoveById 981256 SecRuleRemoveById 981260 SecRuleRemoveById 981317 SecRuleRemoveById 981318 SecRuleRemoveById phpids-17 SecRuleRemoveById phpids-20 SecRuleRemoveById phpids-21 SecRuleRemoveById phpids-30 SecRuleRemoveById phpids-61 SecRuleRemoveById phpids-17 SecRuleRemoveById phpids-20 SecRuleRemoveById phpids-21 SecRuleRemoveById phpids-30 SecRuleRemoveById phpids-61 SecRuleRemoveById 958056 SecRuleRemoveById 950108 SecRuleRemoveById 958056 SecRuleRemoveById 958057 SecRuleRemoveById 959071 SecRuleRemoveById 959072 SecRuleRemoveById 960010 SecRuleRemoveById 960020 ## NEW WORDPRESS ADDITIONAL ## SecRuleRemoveById 959151 SecRuleRemoveById 981250 SecRuleRemoveById 973337 SecRuleRemoveById 950911 SecRuleRemoveById 958049 SecRuleRemoveById 973314 SecRuleRemoveById 973348 SecRuleRemoveById 973321 SecRuleRemoveById 981247 ##XENFORO SecRuleRemoveById 973308 SecRuleRemoveById 958011 SecRuleRemoveById 973329 ##WHMCS SecRuleRemoveById 973322
0
cPanelMichael

August 29, 2016 21:53
Hi, I use disable rules bellow on my previous control panel (non cPanel), anyone can give me dirrection where to put it at cPanel?

Hello, Let us know if this document helps if you are using EasyApache 4: Apache Module: ModSecurity - EasyApache 4 - cPanel Documentation Thank you.
0
hrace009

August 31, 2016 02:09
Hello, Let us know if this document helps if you are using EasyApache 4:
0
joako

February 26, 2017 02:45
Is there a fix for this yet? I see in the log: ModSecurity: collection_store: Failed to access DBM file "/var/cpanel/secdatadir/ip": Permission denied And that particular file does not exist: # ls /var/cpanel/secdatadir/ip /bin/ls: cannot access /var/cpanel/secdatadir/ip: No such file or directory I'm using the default mod_security rules that ship with cpanel.
0
cPanelMichael

February 27, 2017 15:41
ModSecurity: collection_store: Failed to access DBM file "/var/cpanel/secdatadir/ip": Permission denied

This is discussed on the following thread: Easyapache 4 + Modsecurity + Mod_ruid2 errors Thank you.
0

Please sign in to leave a comment.

Comments

Didn't find what you were looking for?