nginx caching pages with 403 errors sent by Imunify360
Hi,
I am having a problem with nginx caching 403 errors sent by Imunify360.
Sometimes Imunify360 blocks a rogue visitor (bot, crawler, attacker etc) and reply with a 403 error, that's OK and expected... however, sometimes (1 in 1M) nginx caches the reply sent to the rogue visitor. I noticed the problem because my monitoring system randomly sent me alerts about websites returning "403 Forbidden", but the sites were UP and didn't show any 403 error.
This is an example log from mod_security:
Apache-Error: [file "apache2_util.c"> [line 275] [level 3] [client 200.82.211.53] ModSecurity: Access denied with code 403 (phase 2). Matched phrase "wp-config.php" at MATCHED_VAR. [file "/etc/apache2/conf.d/modsec_vendor_configs/imunify360-full-apache/013_i360_1_generic.conf"> [line "20"> [id "77140166"> [msg "IM360 WAF: Blocking directory traversal attempt||MVN:MATCHED_VAR||MV:/wp-content/wp-config.php||T:APACHE||"> [severity "CRITICAL"> [tag "service_gen"> [hostname "MYWEBSITE.COM"> [uri "/wp-content/plugins/site-editor/editor/extensions/pagebuilder/includes/ajax_shortcode_pattern.php"> [unique_id "ZKl8ID4Rr5zWAuthgBqyagAAABA">
Message: Access denied with code 403 (phase 2). [file "/etc/apache2/conf.d/modsec_vendor_configs/imunify360-full-apache/003_i360_2_bruteforce.conf"> [line "202"> [id "33303"> [msg "IM360 WAF: WordPress Bruteforce RBL block||Name:MYWEBSITE.COM||T:APACHE||MV:16-50.34.124.229.250"> [severity "CRITICAL"> [tag "wp_core">
Immunify360 is right blocking the attack, but the next visitors get 403 Errors inside the website's code, only visible in the console. Example:
The work-around is to flush the nginx's cache Nginx Manager > Clear Cache for all users. but it is annoying, time consuming and very random.
If i do nothing, the problem fixes itself after 1 hour. It happens in all my cPanel servers with imunify360+nginx.
Notes:
- I replaced Nginx manager with Engintron, no difference
- I am not using mod_evasive
- I already contacted Imunify360 support, no solution. Anyway, Imunify360 is right blocking bad users as expected.
How can I make nginx stop caching pages with 403 errors?
Many thanks
-
Hey there! If you replaced Nginx with Engintron and are still seeing the issue, that would seem to rule out cPanel's Nginx implementation. I also reached out to our team and confirmed that we don't cache anything related to 403 pages by default, so this would be unexpected behavior. If you can come up with a way to reproduce the problem it might be worth submitting a ticket to our team so we can take a look at the system directly, because from everything I can find this should not be happening. 0 -
Hey there! If you replaced Nginx with Engintron and are still seeing the issue, that would seem to rule out cPanel's Nginx implementation. I also reached out to our team and confirmed that we don't cache anything related to 403 pages by default, so this would be unexpected behavior. If you can come up with a way to reproduce the problem it might be worth submitting a ticket to our team so we can take a look at the system directly, because from everything I can find this should not be happening.
Hi cPREX I don't see why it would rule out the cPanel's Nginx implementation, it just confirms it is related with Nginx. if it you don't cache anything related to 403 pages by default, why does flushing the cache fix the issue? Many thanks0 -
Engintron uses a completely different system than us, so if both implementations have the issue, it seems that it would be something else. If you can provide steps to reproduce the issue, we'd love to take a look! 0 -
Engintron uses a completely different system than us, so if both implementations have the issue, it seems that it would be something else. If you can provide steps to reproduce the issue, we'd love to take a look!
Engintron uses a completely different system than us, so if both implementations have the issue, it seems that it would be something else. If you can provide steps to reproduce the issue, we'd love to take a look!
A different implementation of the same software: nginx. I wonder if you ever tested Nginx manager on servers running Imunify360 before release Nginx Manager, because the issue requires Nginx Manager to occur. I understand engintron developers do not test compatibility with Imunify360 because it is a free plugin, but cPanel is commercial service. Sharing an issue with a free plugin is not excuse to don't test/investigate compatibility problems, specially when you also recommend Imunify products and include its free version by default. Maybe you have not received more complains because the problem occurs very infrequently, it is very hard to catch without an automated monitoring system and fixes itself. Unfortunately i don't have any way to reproduce the issue, Immunify360 is blocking attacks all the time but only a few of the 403 replies get cached. Many thanks0 -
As Imunify is supported, that is something that was tested, but I don't have any other details to provide. If it is very rare that the 403 is getting cached, why wouldn't it be happening every time? Unfortunately, without more details I can't really provide much assistance with this one. If you do find a way to make this happen, please open a ticket with our team. 0 -
Hello cPRex, Note, I'm not the user who started this thread "Anthill-Inc", but we work together. Thanks for your feedback about this issue and your willingness to help. I thought it would be helpful to provide a non-technical perspective about this problem (I'm the non-technical one here). As the person who opened this thread stated, this is a very annoying issue, but for a long time (more than a year) it wasn't a critical issue. We have 2 different monitoring systems (NIXStats and HextrixTools) and they always both report the 403 errors when they happen. Because this went on for so long and we got sick of the false alarms from the monitoring systems, we configure a rule in the monitoring system to ignore the 403 error. Whenever we setup a new site we need to configure this rule in the monitors. However, recently Google Search started picking up the 403 and sending notices to our clients that their sites couldn't be crawled because of the 4xx errors. This has escalated the problem because what was previously "annoying" is now a customer service problem for us. Clients don't understand that the site isn't down, and they also need to go through the steps to fix the page indexing issues. alt="Google_Search_403.jpg">83565 I hope that provides some additional perspective why we trying to get this resolved. Thanks again. 0 -
Oh I get the "why" but from our side it's just like taking a car to a mechanic - we have to be able to replicate the problem or else there isn't going to be anything for us to fix. With the issue being that infrequent it will be very difficult to track down. You're welcome to submit a ticket to our team to see if we can find anything while directly examining the server, as that is the best option to get you more details. 0 -
Hello, Because this happens entirely randomly the only way I think we can "duplicate" the problem is to open a support ticket and add you to our monitoring system. You'll be notified when the problem happens, but the cache expires in 60 minutes so it's unlikely that you'll be able catch it in the act, but you'll have a specific timestamp to investigate. Feasible? We're happy to give you access to troubleshoot, let me know if there's a different method to catch this when it's happening. Thanks. 0 -
That may or may not be something we'd catch. But...we might be able to find other evidence of it happening, too. 0 -
As luck would have it, it's happening right now to a site. I got the notice at 11:31AM EDT - should I submit a ticket? 0 -
Yes please, and post the number here :D 0 -
Ticket 95094804 - however it took me longer than I expected to create the ticket and I have not added the cPanel support IPs to the trusted list yet, so we might be out of time. 0 -
I was going to say, I'd give it some priority, but that extra 15 minutes likely didn't help our cause. 0 -
Agreed. We'll get our cPanel servers setup for remote access by cPanel support... when the problem happens again we'll have everything in order so that we can quickly submit a support ticket and you'll have all the access you need. If I see you online when we submit the ticket I'll post the ID here. Thanks. 0 -
Sounds good! I'm usually around during first-shift hours on weekdays, so hopefully I'll see it quickly. 0
Please sign in to leave a comment.
Comments
15 comments