High CPU on random times
Hi all,
I know this is a common topic, but I couldn't find a case similar to mine. I get random CPU overloads (400+ 1min load on a 20-core server). During this overload I run a
because I suspect DDoS attack due to the randomness of the problem. Strangely the most connections come from the server's own ip address (over 700 at the time of the overload). Also I get capped on the MySQL connections limit (402) while on normal operation the connection doesn't exceed 10-25. Restarting PHP-FPM restores normal operation (1.4 to 2 one-minute load and ~49 own ip connections). 1 min interval SAR output:
I've optimized MySQL, PHP-FPM, Apache. I have configures all CSF DDoS options, CPHulk, ModSecurity, mod-reqtimeout on EA4, all sites are running through CloudFlare... I really don't know what else I should check. It's a standalone dedicated server with 256MB of RAM and 20CPUs. Any help is much appreciated, Kostas
netstat -anp |grep 'tcp\|udp' | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -nbecause I suspect DDoS attack due to the randomness of the problem. Strangely the most connections come from the server's own ip address (over 700 at the time of the overload). Also I get capped on the MySQL connections limit (402) while on normal operation the connection doesn't exceed 10-25. Restarting PHP-FPM restores normal operation (1.4 to 2 one-minute load and ~49 own ip connections). 1 min interval SAR output:
10:09:01 PM 2 1235 3.76 3.70 3.65 0
10:10:01 PM 12 1238 3.35 3.63 3.62 0
10:11:01 PM 9 1236 2.11 3.27 3.50 0
10:12:01 PM 6 1236 1.58 2.91 3.36 0
10:13:01 PM 5 1235 2.30 2.88 3.32 0
10:14:01 PM 2 1232 2.66 2.87 3.29 0
10:15:01 PM 5 1233 2.70 2.86 3.26 0
10:16:01 PM 3 1231 4.64 3.29 3.38 0
10:17:01 PM 18 1231 7.70 4.21 3.68 0
10:18:02 PM 82 1759 66.16 18.39 8.40 0
10:18:02 PM runq-sz plist-sz ldavg-1 ldavg-5 ldavg-15 blocked
10:19:01 PM 140 2256 264.44 82.54 30.98 0
10:20:01 PM 281 2266 314.34 129.90 50.49 0
10:21:01 PM 201 2269 369.08 178.98 72.33 0
10:22:01 PM 310 2272 369.74 213.31 90.79 0
10:23:02 PM 144 2275 372.83 241.40 108.04 0
10:24:01 PM 14 1470 184.35 214.74 107.45 0
10:25:01 PM 4 1480 72.76 177.11 101.23 0
10:26:01 PM 10 1461 33.53 146.92 95.60 0
10:27:01 PM 2 1452 17.10 121.53 90.07 0
10:28:01 PM 6 1436 10.01 100.48 84.81 0
10:29:01 PM 6 1431 7.78 83.50 79.95 0
10:30:01 PM 10 1422 5.84 69.17 75.25 0
10:31:01 PM 6 1421 6.70 57.80 70.95 0
10:32:01 PM 6 1423 5.24 48.01 66.75 0
10:33:01 PM 6 1405 3.45 39.71 62.72 0
10:34:01 PM 1 1407 3.58 33.16 59.03 0
I've optimized MySQL, PHP-FPM, Apache. I have configures all CSF DDoS options, CPHulk, ModSecurity, mod-reqtimeout on EA4, all sites are running through CloudFlare... I really don't know what else I should check. It's a standalone dedicated server with 256MB of RAM and 20CPUs. Any help is much appreciated, Kostas
-
From my investigation I noticed that during the load spikes I get errors on the apache log with numerous lines regarding a Facebook (?) ipv6 range. Here's a sample that matches the time of the server load: [Sat Nov 04 05:42:11.522931 2023] [proxy_fcgi:error] [pid 555473:tid 23403623724800] (70007)The timeout specified has expired: [client 2a03:2880:22ff:1::face:b00c:0] AH01075: Error dispatching request to : (polling) [Sat Nov 04 05:42:11.582929 2023] [proxy_fcgi:error] [pid 555473:tid 23403667851008] (70007)The timeout specified has expired: [client 2a03:2880:22ff:1::face:b00c:0] AH01075: Error dispatching request to : (polling) [Sat Nov 04 05:42:11.702922 2023] [proxy_fcgi:error] [pid 555569:tid 23403634231040] (70007)The timeout specified has expired: [client 2a03:2880:3ff:5::face:b00c:0] AH01075: Error dispatching request to : (polling) [Sat Nov 04 05:42:11.790925 2023] [proxy_fcgi:error] [pid 555387:tid 23403617421056] (70007)The timeout specified has expired: [client 2a03:2880:22ff:78::face:b00c:0] AH01075: Error dispatching request to : (polling) [Sat Nov 04 05:42:11.796471 2023] [proxy_fcgi:error] [pid 555473:tid 23403630028544] (70007)The timeout specified has expired: [client 2a03:2880:20ff:1::face:b00c:0] AH01075: Error dispatching request to : (polling) [Sat Nov 04 05:42:11.798940 2023] [proxy_fcgi:error] [pid 555473:tid 23403642636032] (70007)The timeout specified has expired: [client 2a03:2880:10ff:b::face:b00c:0] AH01075: Error dispatching request to : (polling) [Sat Nov 04 05:42:11.866920 2023] [proxy_fcgi:error] [pid 537817:tid 23403634231040] (70007)The timeout specified has expired: [client 2a03:2880:21ff:f::face:b00c:0] AH01075: Error dispatching request to : (polling) [Sat Nov 04 05:42:11.878923 2023] [proxy_fcgi:error] [pid 537817:tid 23403627927296] (70007)The timeout specified has expired: [client 2a03:2880:25ff:2::face:b00c:0] AH01075: Error dispatching request to : (polling) [Sat Nov 04 05:42:12.006926 2023] [proxy_fcgi:error] [pid 555387:tid 23403625826048] (70007)The timeout specified has expired: [client 2a03:2880:25ff:76::face:b00c:0] AH01075: Error dispatching request to : (polling) [Sat Nov 04 05:42:12.010931 2023] [proxy_fcgi:error] [pid 555473:tid 23403661547264] (70007)The timeout specified has expired: [client 2a03:2880:25ff:4::face:b00c:0] AH01075: Error dispatching request to : (polling) [Sat Nov 04 05:42:12.011990 2023] [proxy_fcgi:error] [pid 555473:tid 23403644737280] (70007)The timeout specified has expired: [client 2a03:2880:22ff:5::face:b00c:0] AH01075: Error dispatching request to : (polling) [Sat Nov 04 05:42:12.070934 2023] [proxy_fcgi:error] [pid 555473:tid 23403625826048] (70007)The timeout specified has expired: [client 2a03:2880:25ff:77::face:b00c:0] AH01075: Error dispatching request to : (polling) [Sat Nov 04 05:42:12.150926 2023] [proxy_fcgi:error] [pid 555473:tid 23403617421056] (70007)The timeout specified has expired: [client 2a03:2880:22ff:9::face:b00c:0] AH01075: Error dispatching request to : (polling) [Sat Nov 04 05:42:12.226929 2023] [proxy_fcgi:error] [pid 555569:tid 23403648939776] (70007)The timeout specified has expired: [client 2a03:2880:20ff:a::face:b00c:0] AH01075: Error dispatching request to : (polling) [Sat Nov 04 05:42:12.486932 2023] [proxy_fcgi:error] [pid 555569:tid 23403640534784] (70007)The timeout specified has expired: [client 2a03:2880:25ff:4::face:b00c:0] AH01075: Error dispatching request to : (polling) [Sat Nov 04 05:42:12.490948 2023] [proxy_fcgi:error] [pid 555676:tid 23403667851008] (70007)The timeout specified has expired: [client 2a03:2880:3ff:1::face:b00c:0] AH01075: Error dispatching request to : (polling) [Sat Nov 04 05:42:12.630928 2023] [proxy_fcgi:error] [pid 537435:tid 23403621623552] (70007)The timeout specified has expired: [client 2a03:2880:20ff:d::face:b00c:0] AH01075: Error dispatching request to : (polling) [Sat Nov 04 05:42:12.789034 2023] [proxy_fcgi:error] [pid 533495:tid 23403665749760] (70007)The timeout specified has expired: [client 2a03:2880:10ff:5::face:b00c:0] AH01075: Error dispatching request to : (polling) [Sat Nov 04 05:42:12.790925 2023] [proxy_fcgi:error] [pid 491091:tid 23403621623552] (70007)The timeout specified has expired: [client 44.230.252.91:0] AH01075: Error dispatching request to : (polling)
I've already set limits per connection on CSF, would anyone recommend to blacklist those ipv6 or would it harm my client's actual Facebook traffic?0 -
Hey there! I think the most useful thing in this case would be to run "apachectl status" or "apachectl full status" while the issue is happening. That would confirm if there is a DoS of some sort, and you would be able to see the connections to the server in real-time. 0 -
Hi @cPRex Thank you for the suggestion. Already ran a status during the load. In normal conditions the current requests doesn't exceed 7-10 and I don't have that many "Sending Reply" status. Here are the results: [QUOTE] Apache Server Status for localhost (via 127.0.0.1) Server Version: Apache/2.4.58 (cPanel) OpenSSL/1.1.1f mod_bwlimited/1.4 mod_fcgid/2.3.9 Phusion_Passenger/6.0.18 Server MPM: event Server Built: Oct 19 2023 00:00:00 ---------------------------------------------------------------------- Current Time: Tuesday, 07-Nov-2023 12:10:50 EET Restart Time: Thursday, 02-Nov-2023 01:24:53 EET Parent Server Config. Generation: 9 Parent Server MPM Generation: 8 Server uptime: 5 days 10 hours 45 minutes 56 seconds Server load: 298.77 132.08 53.74 Total accesses: 2675495 - Total Traffic: 205.3 GB - Total Duration: 2383517127 CPU Usage: u739.89 s224.15 cu8593.47 cs2525.51 - 2.57% CPU load 5.68 requests/sec - 457.2 kB/second - 80.4 kB/request - 890.87 ms/request 181 requests currently being processed, 0 workers gracefully restarting, 194 idle workers +-----------------------------------------------------------------------------------+ | | | |Connections |Threads |Async connections | | |Slot|PID |Stopping|---------------+-------------+-----------------------+-------| | | | |total|accepting|busy|graceful|idle|writing|keep-alive|closing| |----+-------+--------+-----+---------+----+--------+----+-------+----------+-------| |0 |1241426|no |25 |yes |24 |0 |1 |0 |0 |0 | |----+-------+--------+-----+---------+----+--------+----+-------+----------+-------| |1 |1241498|no |20 |yes |22 |0 |3 |0 |0 |0 | |----+-------+--------+-----+---------+----+--------+----+-------+----------+-------| |2 |1307602|no |8 |yes |8 |0 |17 |0 |0 |0 | |----+-------+--------+-----+---------+----+--------+----+-------+----------+-------| |3 |1241680|no |5 |yes |4 |0 |21 |0 |0 |0 | |----+-------+--------+-----+---------+----+--------+----+-------+----------+-------| |4 |1307702|no |8 |yes |5 |0 |20 |0 |0 |0 | |----+-------+--------+-----+---------+----+--------+----+-------+----------+-------| |5 |1307789|no |0 |yes |5 |0 |20 |0 |0 |0 | |----+-------+--------+-----+---------+----+--------+----+-------+----------+-------| |6 |1307867|no |8 |yes |10 |0 |15 |0 |0 |0 | |----+-------+--------+-----+---------+----+--------+----+-------+----------+-------| |7 |1195632|no |19 |yes |19 |0 |6 |0 |0 |0 | |----+-------+--------+-----+---------+----+--------+----+-------+----------+-------| |8 |1241995|no |17 |yes |24 |0 |1 |0 |0 |0 | |----+-------+--------+-----+---------+----+--------+----+-------+----------+-------| |9 |1307868|no |8 |yes |6 |0 |19 |0 |0 |0 | |----+-------+--------+-----+---------+----+--------+----+-------+----------+-------| |14 |1196239|no |9 |yes |9 |0 |16 |0 |0 |0 | |----+-------+--------+-----+---------+----+--------+----+-------+----------+-------| |21 |1196247|no |17 |yes |18 |0 |7 |0 |0 |0 | |----+-------+--------+-----+---------+----+--------+----+-------+----------+-------| |23 |1195592|no |10 |yes |10 |0 |15 |0 |0 |0 | |----+-------+--------+-----+---------+----+--------+----+-------+----------+-------| |28 |1196252|no |13 |yes |11 |0 |14 |0 |0 |0 | |----+-------+--------+-----+---------+----+--------+----+-------+----------+-------| |37 |1196261|no |6 |yes |6 |0 |19 |0 |0 |0 | |----+-------+--------+-----+---------+----+--------+----+-------+----------+-------| |Sum |15 |0 |173 | |181 |0 |194 |0 |0 |0 | +-----------------------------------------------------------------------------------+ WWWWWWWWWWWWWWWW_WWWWWWWWWWWWWWWWWWWW_W_W_WWWWWWWWW_W_WW________ ___W__WW_W_______W__WW___W______________W_______WW_W____W_____WW _________WW________W___W__W___W_____WW_WWW_W_W_WWWW__WWWWWW__WWW WWWWWW__WWWWWWWW_WWWWWWWWWWWWWWWW__WW__W_W_________WW_____...... ................................................................ ..............................___W_W_W________WWWWW___W......... ................................................................ ................................................................ .............WWWWW__WWW_WW_WWW__WW_WWW.........................W _WW__W__WW__W__W__W_W___........................................ ............................................................WWWW _WW____W___W__W__W__W........................................... ................................................................ ................................................................ ............................._W_W__W__WW____W_________.......... ................................................................ ................................................................ ................................................................ ................................................................ ................................................................ ................................................................ ................................................................ ................................................................ ................................................................ ................................................................ ................................................................ ............................................................. Scoreboard Key: "_" Waiting for Connection, "S" Starting up, "R" Reading Request, "W" Sending Reply, "K" Keepalive (read), "D" DNS Lookup, "C" Closing connection, "L" Logging, "G" Gracefully finishing, "I" Idle cleanup of worker, "." Open slot with no current process Error writing to stdout: Broken pipe. 0 -
That all looks like normal output to me. Since the issue only happens for a few minutes, it's going to be tricky to diagnose no matter what. Running "fullstatus" would show what pages are being connected to at that exact time. I'm just wondering if there is a certain page that is causing an issue because you seem to be flooded with the timeout errors, and details on how to fix that can be found here: 0
Please sign in to leave a comment.
Comments
4 comments