Very high CPU loads from brute force attempts - CSF/LFD
Over the last few days, we've experienced nearly 100% CPU load on a daily basis. Throughout the day there are hundreds of
and
processes running, all of which are to block IP addresses from brute force attempts to our website's admin page. There are also hundreds of
and
processes which are related to the login/database attempts. What can we do to bring down the CPU load? The number of tasks running has never been this high. It's now anywhere from 200 - 600 tasks. CSF and cPhulk are configured to block these attacks, but there are far too many that it's overloading the server and regular tasks (e.g. sending/receiving email from server) are nearly impossible. LF_MODSEC Log
top (there are hundreds of these processes/commands)
csfand
lfdprocesses running, all of which are to block IP addresses from brute force attempts to our website's admin page. There are also hundreds of
mysqldand
httpdprocesses which are related to the login/database attempts. What can we do to bring down the CPU load? The number of tasks running has never been this high. It's now anywhere from 200 - 600 tasks. CSF and cPhulk are configured to block these attacks, but there are far too many that it's overloading the server and regular tasks (e.g. sending/receiving email from server) are nearly impossible. LF_MODSEC Log
[Tue Apr 26 07:25:21.273168 2022] [:error] [pid 11162] [client 173.249.19.246:59052] [client 173.249.19.246] ModSecurity: Access denied with code 406 (phase 1).
Pattern match "Mozilla/5.0 \\\\(X11; Ubuntu; Linux x86_64; rv:62\\\\.0\\\\) Gecko\\\\/20100101 Firefox\\\\/62\\\\.0" at REQUEST_HEADERS:User-Agent.
[file "/etc/apache2/conf.d/modsec/modsec2.user.conf"> [line "1"> [id "91996789"> [msg "BAD UA BLOCK"> [hostname "mydomain.com">
[uri "/admin/"> [unique_id "YmfWoRM2Fjtq6jMg5hEDNgAAAAw">
top (there are hundreds of these processes/commands)
Tasks: 329 total, 2 running, 323 sleeping, 1 stopped, 3 zombie
%Cpu(s): 70.8 us, 18.4 sy, 0.0 ni, 0.0 id, 9.5 wa, 0.0 hi, 1.3 si, 0.0 st
KiB Mem : 3880140 total, 545588 free, 1364884 used, 1969668 buff/cache
KiB Swap: 4194300 total, 3958600 free, 235700 used. 2191960 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
30930 root 20 0 179440 30608 2824 S 14.6 0.8 0:00.43 csf
30926 root 20 0 179280 30420 2820 S 14.2 0.8 0:00.42 csf
30931 root 20 0 179284 30412 2820 S 14.2 0.8 0:00.42 csf
30932 root 20 0 179280 30484 2820 S 14.2 0.8 0:00.42 csf
30925 root 20 0 179132 30388 2820 S 13.9 0.8 0:00.41 csf
30928 root 20 0 179132 30408 2820 S 13.6 0.8 0:00.40 csf
30929 root 20 0 179136 30420 2820 S 13.6 0.8 0:00.40 csf
30934 root 20 0 179136 30392 2820 R 13.6 0.8 0:00.40 csf
30935 root 20 0 179136 30420 2820 R 13.6 0.8 0:00.40 csf
30927 root 20 0 179000 30052 2820 R 13.2 0.8 0:00.39 csf
30936 root 20 0 178212 29304 2816 R 12.9 0.8 0:00.38 csf
30937 root 20 0 178340 29528 2820 R 12.9 0.8 0:00.38 csf
30933 root 20 0 173964 27292 2800 R 11.9 0.7 0:00.35 csf
22275 root 20 0 188452 35236 896 S 0.0 0.9 0:00.01 lfd - (child) blocking 148.202.167.75
23194 root 20 0 188452 35236 896 S 0.0 0.9 0:00.01 lfd - (child) blocking 38.135.34.49
23196 root 20 0 188452 35236 896 S 0.0 0.9 0:00.02 lfd - (child) blocking 138.97.220.166
23198 root 20 0 188452 35236 896 S 0.0 0.9 0:00.02 lfd - (child) blocking 212.47.227.85
23356 root 20 0 188452 35236 896 S 0.0 0.9 0:00.01 lfd - (child) blocking 163.172.53.199
23473 root 20 0 188452 35236 896 S 0.0 0.9 0:00.01 lfd - (child) blocking 189.254.45.110
23930 root 20 0 188452 35236 896 S 0.0 0.9 0:00.01 lfd - (child) blocking 91.238.161.177
24066 root 20 0 188452 35236 896 S 0.0 0.9 0:00.02 lfd - (child) blocking 62.173.139.188
24181 root 20 0 188452 35236 896 S 0.0 0.9 0:00.02 lfd - (child) blocking 178.32.202.97
24182 root 20 0 188452 35236 896 S 0.0 0.9 0:00.02 lfd - (child) blocking 128.199.241.20
24185 root 20 0 188452 35236 896 S 0.0 0.9 0:00.02 lfd - (child) blocking 178.128.151.87
24190 root 20 0 188452 35236 896 S 0.0 0.9 0:00.01 lfd - (child) blocking 79.175.127.171
24340 root 20 0 188452 35236 896 S 0.0 0.9 0:00.01 lfd - (child) blocking 45.80.153.73
24342 root 20 0 188452 35236 896 S 0.0 0.9 0:00.01 lfd - (child) blocking 12.12.141.226
24344 root 20 0 188452 35236 896 S 0.0 0.9 0:00.02 lfd - (child) blocking 212.7.211.113
24345 root 20 0 188452 35236 896 S 0.0 0.9 0:00.02 lfd - (child) blocking 217.115.118.126
24348 root 20 0 188452 35236 896 S 0.0 0.9 0:00.01 lfd - (child) blocking 79.172.201.113
24351 root 20 0 188452 35236 896 S 0.0 0.9 0:00.01 lfd - (child) blocking 76.245.195.148
24434 root 20 0 188452 35236 896 S 0.0 0.9 0:00.01 lfd - (child) blocking 51.178.185.66
24437 root 20 0 188452 35236 896 S 0.0 0.9 0:00.01 lfd - (child) blocking 141.94.32.98
24438 root 20 0 188452 35236 896 S 0.0 0.9 0:00.01 lfd - (child) blocking 178.128.55.40
24535 root 20 0 188452 35236 896 S 0.0 0.9 0:00.01 lfd - (child) blocking 138.97.220.166
24699 root 20 0 188452 35236 896 S 0.0 0.9 0:00.02 lfd - (child) blocking 202.29.148.67
24875 root 20 0 188452 35236 896 S 0.0 0.9 0:00.02 lfd - (child) blocking 159.203.28.59
24877 root 20 0 188452 35236 896 S 0.0 0.9 0:00.01 lfd - (child) blocking 31.173.68.7
24878 root 20 0 188452 35236 896 S 0.0 0.9 0:00.02 lfd - (child) blocking 188.40.33.77
24879 root 20 0 188452 35236 896 S 0.0 0.9 0:00.01 lfd - (child) blocking 203.23.49.192
24880 root 20 0 188452 35236 896 S 0.0 0.9 0:00.01 lfd - (child) blocking 162.214.104.98
25106 root 20 0 188452 35236 896 S 0.0 0.9 0:00.01 lfd - (child) blocking 43.229.77.90
25107 root 20 0 188452 35236 896 S 0.0 0.9 0:00.01 lfd - (child) blocking 38.135.34.49
25109 root 20 0 188452 35236 896 S 0.0 0.9 0:00.01 lfd - (child) blocking 178.62.213.36
25110 root 20 0 188452 35236 896 S 0.0 0.9 0:00.01 lfd - (child) blocking 103.144.82.1
25111 root 20 0 188452 35236 896 S 0.0 0.9 0:00.01 lfd - (child) blocking 207.180.213.165
25446 root 20 0 188452 35236 896 S 0.0 0.9 0:00.01 lfd - (child) blocking 45.79.68.53
25447 root 20 0 188452 35236 896 S 0.0 0.9 0:00.01 lfd - (child) blocking 47.88.23.114
25448 root 20 0 188452 35236 896 S 0.0 0.9 0:00.01 lfd - (child) blocking 139.59.68.9
25449 root 20 0 188452 35236 896 S 0.0 0.9 0:00.01 lfd - (child) blocking 46.101.188.174
25450 root 20 0 188452 35236 896 S 0.0 0.9 0:00.01 lfd - (child) blocking 207.180.236.152
25451 root 20 0 188452 35236 896 S 0.0 0.9 0:00.02 lfd - (child) blocking 203.210.87.64
25837 root 20 0 188452 35236 896 S 0.0 0.9 0:00.02 lfd - (child) blocking 76.103.114.159
25935 root 20 0 188452 35236 896 S 0.7 0.9 0:00.02 lfd - (child) blocking 180.242.130.79
25939 root 20 0 188452 35236 896 S 0.3 0.9 0:00.01 lfd - (child) blocking 103.41.204.29
25940 root 20 0 188452 35236 896 S 0.3 0.9 0:00.01 lfd - (child) blocking 138.201.142.73
25941 root 20 0 188452 35236 896 S 0.3 0.9 0:00.01 lfd - (child) blocking 182.70.248.147
25944 root 20 0 188452 35236 896 S 0.3 0.9 0:00.01 lfd - (child) blocking 178.128.155.255
25946 root 20 0 188452 35236 896 S 0.3 0.9 0:00.01 lfd - (child) blocking 51.79.248.189
25948 root 20 0 188452 35236 896 S 0.3 0.9 0:00.01 lfd - (child) blocking 92.205.25.196
21206 root 20 0 188452 35232 892 S 0.0 0.9 0:00.01 lfd - (child) blocking 178.128.150.247
23352 root 20 0 188452 35232 892 S 0.0 0.9 0:00.02 lfd - (child) blocking 213.187.11.93
23688 root 20 0 188452 35232 892 S 0.0 0.9 0:00.01 lfd - (child) blocking 185.21.217.56
23694 root 20 0 188452 35232 892 S 0.0 0.9 0:00.01 lfd - (child) blocking 51.77.214.27
23923 root 20 0 188452 35232 892 S 0.0 0.9 0:00.02 lfd - (child) blocking 185.148.3.93
24064 root 20 0 188452 35232 892 S 0.0 0.9 0:00.02 lfd - (child) blocking 51.15.181.37
24697 root 20 0 188452 35232 892 S 0.0 0.9 0:00.01 lfd - (child) blocking 145.131.25.246
24874 root 20 0 188452 35232 892 S 0.0 0.9 0:00.01 lfd - (child) blocking 83.137.145.154
25103 root 20 0 188452 35232 892 S 0.0 0.9 0:00.01 lfd - (child) blocking 12.12.141.226
25258 root 20 0 188452 35232 892 S 0.0 0.9 0:00.02 lfd - (child) blocking 219.153.110.7
25259 root 20 0 188452 35232 892 S 0.0 0.9 0:00.01 lfd - (child) blocking 68.183.175.58
25261 root 20 0 188452 35232 892 S 0.0 0.9 0:00.02 lfd - (child) blocking 185.149.103.55
25445 root 20 0 188452 35232 892 S 0.0 0.9 0:00.02 lfd - (child) blocking 196.41.123.124
25452 root 20 0 188452 35232 892 S 0.0 0.9 0:00.01 lfd - (child) blocking 51.77.214.27
25639 root 20 0 188452 35232 892 S 0.0 0.9 0:00.02 lfd - (child) blocking 185.116.215.125
25727 root 20 0 188452 35232 892 S 0.0 0.9 0:00.01 lfd - (child) blocking 190.105.205.100
25728 root 20 0 188452 35232 892 S 0.0 0.9 0:00.01 lfd - (child) blocking 51.178.136.52
1351 nobody 20 0 547088 22652 3624 S 0.3 0.6 0:00.13 httpd
3111 nobody 20 0 546820 21616 2900 S 0.3 0.6 0:00.09 httpd
3157 nobody 20 0 546820 21616 2900 S 0.3 0.6 0:00.09 httpd
-
From the output you shared, it seems an attack from multiple IPs, you should reach out to your data center or server provider, as they have specialized equipment to put in place to help mitigate the attack until it slows down or dies off. Other options: - Using CDN or external firewall services such as Cloudflare can help buffer traffic to the server. OR - Consult a 0 -
I would also recommend an external tool. If you try and handle this locally, you'll still need to use your server's CPU processing power to handle the traffic. An external solution would completely remove your CPU from the equation, and your hosting provider likely has an external firewall or DoS prevention tools available. 0 -
We've had the VPS configured with CSF and cPhulk for several years now with no issues until now. Are there any Mod_Security rules or anything like that you would recommend looking at? 0 -
mod_security wouldn't handle that type of traffic, as that watches uploads through the web and not brute force attacks. 0 -
mod_security wouldn't handle that type of traffic, as that watches uploads through the web and not brute force attacks.
Look at the logs below. It shows Mod_Security rules are working to block the following...Message: Access denied with code 406 (phase 1). Pattern match "Mozilla/5.0 \\(X11; Ubuntu; Linux x86_64; rv:62\\.0\\) Gecko\\/20100101 Firefox\\/62\\.0" at REQUEST_HEADERS:User-Agent. [file "/etc/apache2/conf.d/modsec/modsec2.user.conf"> [line "1"> [id "91996789"> [msg "BAD UA BLOCK"> Apache-Error: [file "apache2_util.c"> [line 271] [level 3] [client 80.253.246.193] ModSecurity: Access denied with code 406 (phase 1). Pattern match "Mozilla/5.0 \\\\\\\\(X11; Ubuntu; Linux x86_64; rv:62\\\\\\\\.0\\\\\\\\) Gecko\\\\\\\\/20100101 Firefox\\\\\\\\/62\\\\\\\\.0" at REQUEST_HEADERS:User-Agent. [file "/etc/apache2/conf.d/modsec/modsec2.user.conf"> [line "1"> [id "91996789"> [msg "BAD UA BLOCK"> [hostname "mydomain.com"> [uri "/admin/"> [unique_id "YmhEsUkj-UhCPIMszGtGEQAAAA8"> Action: Intercepted (phase 1) Stopwatch: 1651000497800531 351 (- - -) Stopwatch2: 1651000497800531 351; combined=47, p1=20, p2=0, p3=0, p4=0, p5=27, sr=0, sw=0, l=0, gc=0 Producer: ModSecurity for Apache/2.9.3 (http://www.modsecurity.org/); OWASP_CRS/3.0.2. Server: Apache Engine-Mode: "ENABLED"0 -
For that particular request, it seems they are trying to log in to domain.com/admin and their activity is triggering mod_security. However, if this is happening as frequently as you report, it would still be best to look to external firewall solutions since your server would have to process anything related to mod_security. Ideally, if the traffic can be stopped before they are even able to reach the server, that would be the ideal solution. 0 -
For that particular request, it seems they are trying to log in to domain.com/admin and their activity is triggering mod_security. However, if this is happening as frequently as you report, it would still be best to look to external firewall solutions since your server would have to process anything related to mod_security. Ideally, if the traffic can be stopped before they are even able to reach the server, that would be the ideal solution.
OK. Can you think of anything else we should look at before going with an external option? We have not had these type of attacks before -- that's what's alarming and confusing.0 -
I personally don't have additional ideas, but that's mostly how everything works - I don't *need* to enable a slow query log, until I notice database slowness. I don't *need* to increase my RAM until my users grow and it starts running low. You didn't *need* to explore DoS protection options, until today. If you'd like to submit a ticket to our team we could at least check the system for known security problems and ensure cPanel itself is working well. 0 -
I personally don't have additional ideas, but that's mostly how everything works - I don't *need* to enable a slow query log, until I notice database slowness. I don't *need* to increase my RAM until my users grow and it starts running low. You didn't *need* to explore DoS protection options, until today. If you'd like to submit a ticket to our team we could at least check the system for known security problems and ensure cPanel itself is working well.
OK. We've had CSF and cPhulk since the beginning. Just odd how it only started presenting an issue years later.0 -
the overload could be because csf should execute the blocking and write the ips inside the csf.deny file. The attack seem to be string chains inside the User Agent header, and then returning a 406 error. Perhaps you could mitigate the situation moving the blocking into the .htacces of the atacked website, and then disabling the modsecurity rule. In that way the csf could be free of all that work The ideal thing would be studying the logs, to see what those headers contains. And then build the precise rule inside the .htaccess with related strings. Something like this although with your strings: RewriteCond %{HTTP_USER_AGENT} (libwww-perl|wget|python|nikto|curl|scan|java|winhttp|clshttp|loader) [NC,OR] RewriteCond %{HTTP_USER_AGENT} (<|>|'|%0A|%0D|%27|%3C|%3E|%00) [NC,OR] ... and so on you can find many examples of HTTP_USER_AGENT for .htaccess everywhere In that way you could disable the modsecurity rule, to convert that overload in simple common requests. And csf would be free of all that work. In case of different attacked websites you could include these rules inside the Apache configuration and later disable that modsecurity rule 0 -
the overload could be because csf should execute the blocking and write the ips inside the csf.deny file. The attack seem to be string chains inside the User Agent header, and then returning a 406 error. Perhaps you could mitigate the situation moving the blocking into the .htacces of the atacked website, and then disabling the modsecurity rule. In that way the csf could be free of all that work The ideal thing would be studying the logs, to see what those headers contains. And then build the precise rule inside the .htaccess with related strings. Something like this although with your strings: RewriteCond %{HTTP_USER_AGENT} (libwww-perl|wget|python|nikto|curl|scan|java|winhttp|clshttp|loader) [NC,OR] RewriteCond %{HTTP_USER_AGENT} (<|>|'|%0A|%0D|%27|%3C|%3E|%00) [NC,OR] ... and so on you can find many examples of HTTP_USER_AGENT for .htaccess everywhere In that way you could disable the modsecurity rule, to convert that overload in simple common requests. And csf would be free of all that work. In case of different attacked websites you could include these rules inside the Apache configuration and later disable that modsecurity rule
have you heard of this? Looks like it has all the htaccess rewrite rules that would be necessary:0 -
The .htaccess rules seem to be working pretty well. Thanks for the suggestion. Quick question... Apache logs are showing an error for this rewrite rule. Any ideas? RewriteCond: cannot compile regular expression
RewriteCond %{HTTP_USER_AGENT} (acapbot|acoonbot|asterias|attackbot|backdorbot|becomebot|binlar|blackwidow|blekkobot|blexbot|blowfish|bullseye|bunnys|butterfly|careerbot|casper|checkpriv|cheesebot|cherrypick|chinaclaw|choppy|clshttp|cmsworld|copernic|copyrightcheck|cosmos|crescent|cy_cho|datacha|demon|diavol|discobot|dittospyder|dotbot|dotnetdotcom|dumbot|emailcollector|emailsiphon|emailwolf|extract|eyenetie|feedfinder|flaming|flashget|flicky|foobot|g00g1e|getright|gigabot|go-ahead-got|gozilla|grabnet|grafula|harvest|heritrix|httrack|icarus6j|jetbot|jetcar|jikespider|kmccrew|leechftp|libweb|linkextractor|linkscan|linkwalker|loader|masscan|miner|mechanize|morfeus|moveoverbot|netmechanic|netspider|nicerspro|nikto|ninja|nutch|octopus|pagegrabber|petalbot|planetwork|postrank|proximic|purebot|pycurl|python|queryn|queryseeker|radian6|radiation|realdownload|scooter|seekerspider|semalt|siclab|sindice|sistrix|sitebot|siteexplorer|sitesnagger|skygrid|smartdownload|snoopy|sosospider|spankbot|spbot|sqlmap|stackrambler|stripper|sucker|surftbot|sux0r|suzukacz|suzuran|takeout|teleport|telesoft|true_robots|turingos|turnit|vampire|vikspider|voideye|webleacher|webreaper|webstripper|webvac|webviewer|webwhacker|winhttp|wwwoffle|woxbot|xaldon|xxxyy|yamanalab|yioopbot|youda|zeus|zmeu|zune|zyborg) [NC] RewriteRule .* - [F,L]0 -
Following up on this... We noticed immediate changes when we enabled the DoS rules in ModSecurity (which are disabled by default). /etc/apache2/conf.d/modsec_vendor_configs/OWASP3/crs-setup.conf (below shows the rule enabled without the '#') # # Optional DoS protection against clients making requests too quickly. # # When a client is making more than 100 requests (excluding static files) within # 60 seconds, this is considered a 'burst'. After two bursts, the client is # blocked for 600 seconds. # # Requests to static files are not counted towards DoS; they are listed in the # 'tx.static_extensions' setting, which you can change in this file (see # section "HTTP Policy Settings"). # # For a detailed description, see rule file REQUEST-912-DOS-PROTECTION.conf. # # Uncomment this rule to use this feature: # SecAction \ "id:900700,\ phase:1,\ nolog,\ pass,\ t:none,\ setvar:'tx.dos_burst_time_slice=60',\ setvar:'tx.dos_counter_threshold=100',\ setvar:'tx.dos_block_timeout=600'"0 -
I would try somehting like this depending of the attack strings: RewriteCond %{HTTP_USER_AGENT} ^.*(string1|string2|string3.....).*$ [NC] RewriteRule ^ 406 [L,R] letters between [..] like [NC] are conditional flags when there is more than one rule, and also for other purposes. Errors can appears because this reason. Here one manual: 0
Please sign in to leave a comment.
Comments
14 comments