Apache MPM Event - Scoreboard not full
Currently on Apache 2.4.57 and I've noticed a pattern that will leads to Apache needing to be restarted. When examining the error logs, there will be a large spike in mod security blocked requests from multiple IPs followed by a lot of scoreboard errors over the next couple of hours [QUOTE][mpm_event:error] [pid 23084:tid 47651189847104] AH03490: scoreboard is full, not at MaxRequestWorkers.Increase ServerLimit.
with [QUOTE][http2:warn] [pid 24661:tid 47651596809984] [client 52.167.144.172:51392] h2_stream(24661-1193-1,CLEANUP): started=1, scheduled=1, ready=0, out_buffer=0
interspersed. Finally there will be multiple [QUOTE][core:warn] [pid 23084:tid 47651189847104] AH00045: child process 27439 still did not exit, sending a SIGTERM
. Current mpm event settings:
It's my understanding that MaxRequestWorkers does not "count" gracefully shutting down threads but it still counts towards ThreadsPerChild. I've tried increasing the server limit to 30, but we still experience the same issue. First question, is it better to increase the server limit to deal with the issue or should I adjust AsyncRequestWorkerFactor? Second question is about cPanel's support documentation related to the issue. Tags " apache/httpd, 2.4.7 was released on Nov 19, 2013 and 2.4.54 was released on Jun 8, 2022.
with [QUOTE][http2:warn] [pid 24661:tid 47651596809984] [client 52.167.144.172:51392] h2_stream(24661-1193-1,CLEANUP): started=1, scheduled=1, ready=0, out_buffer=0
interspersed. Finally there will be multiple [QUOTE][core:warn] [pid 23084:tid 47651189847104] AH00045: child process 27439 still did not exit, sending a SIGTERM
. Current mpm event settings:
AsyncRequestWorkerFactor 2
ServerLimit 20
MaxRequestWorkers 2000
StartServers 20
MaxConnectionsPerChild 10000
ThreadsPerChild 100
# MaxRequestWorkers/4
MinSpareThreads 500
# MinSpareThreads * 3
MaxSpareThreads 1500
ThreadLimit 128
It's my understanding that MaxRequestWorkers does not "count" gracefully shutting down threads but it still counts towards ThreadsPerChild. I've tried increasing the server limit to 30, but we still experience the same issue. First question, is it better to increase the server limit to deal with the issue or should I adjust AsyncRequestWorkerFactor? Second question is about cPanel's support documentation related to the issue. Tags " apache/httpd, 2.4.7 was released on Nov 19, 2013 and 2.4.54 was released on Jun 8, 2022.
-
Hey there! I believe that support article may be inaccurate as Apache has listed that bug as being resolved since 2017:
Not seeing anything out of the ordinary, but I would imagine I would need to catch the server during the issue and not after cPanel has forced a restart. Just to confirm, I have this block in our pre-virtual-host include. Is that the correct place to add mpm event configuration and is the if block correct?AsyncRequestWorkerFactor 2 ServerLimit 20 MaxRequestWorkers 2000 StartServers 20 MaxConnectionsPerChild 10000 ThreadsPerChild 100 # MaxRequestWorkers/4 MinSpareThreads 500 # MinSpareThreads * 3 MaxSpareThreads 1500 ThreadLimit 128
0 -
Yes, you would need to check the scoreboard during the issue to see what is happening. You wouldn't be able to change the mpm through an include, and I wouldn't expect those settings to be applied since they are called later in the configuration than where the include is read. You can check the current mpm by checking "apachectl status" as that will show a "Server MPM:" line in the output. If it isn't set to Event already and that is what you want, you would need to change this through the WHM >> EasyApache 4 interface. If it is already Event, there is no reason to have that include as you would adjust those values through WHM >> Apache Configuration. 0 -
Yes, you would need to check the scoreboard during the issue to see what is happening. You wouldn't be able to change the mpm through an include, and I wouldn't expect those settings to be applied since they are called later in the configuration than where the include is read. You can check the current mpm by checking "apachectl status" as that will show a "Server MPM:" line in the output. If it isn't set to Event already and that is what you want, you would need to change this through the WHM >> EasyApache 4 interface. If it is already Event, there is no reason to have that include as you would adjust those values through WHM >> Apache Configuration.
Sorry, I should have been clearer. MPM event is set through Easy Apache. I have that config in the pre virtual host file since WHM doesn't have every value I would like to set; specifically ThreadsPerChild, MinSpareThreads, MaxSpareThreads, and ThreadLimit. I went ahead and set the other directives there so all the mpm configuration would be in one spot. Is there a better way to set these directives than what we're using? Checking "apachectl status" confirms MPM is Event. [QUOTE]Server MPM: event
Couple of follow-up questions. Is there a good way to confirm the final values Apache is using? If I catch the server during one of these issues, is there anything specific I'm looking for with "apachectl fullstatus"? I'm guessing the scoreboard full of G or I?0 -
Thanks for that clarification. Yes, if you want to add values that aren't available in the interface, that is the correct include to use. Since you have the options in the file, it would be using those configurations. The closest option available that I know of is this: apachectl -t -D DUMP_RUN_CFG
which will give you the following output:[root@host /]# apachectl -t -D DUMP_RUN_CFG ServerRoot: "/etc/apache2" Main DocumentRoot: "/etc/apache2/htdocs" Main ErrorLog: "/etc/apache2/logs/error_log" Mutex rewrite-map: dir="/run/apache2" mechanism=fcntl Mutex ssl-stapling-refresh: using_defaults Mutex fcgid-proctbl: using_defaults Mutex ssl-stapling: using_defaults Mutex proxy: using_defaults Mutex ssl-cache: dir="/run/apache2" mechanism=fcntl Mutex default: dir="/var/run/apache2/" mechanism=default Mutex fcgid-pipe: using_defaults PidFile: "/run/apache2/httpd.pid" Define: DUMP_RUN_CFG Define: MODSEC_2.5 Define: MODSEC_2.9 User: name="nobody" id=65534 Group: name="nobody" id=65534
"apachectl -t -D DUMP_MODULES" will show the modules loaded on the server as well. There's some good details about this option here that you might want to read as well: Apache Web Server Dump Current Running Configuration Settings - CodingShower As far as the output, yes, you'd want to see what letters show up in the scoreboard, or if there are any free idle workers.0 -
Thank you cPRex, using some of the suggested commands I was able to realize what was happening. I had the wrong statement in the IfModule block so it was using default values for Min/Max Spare Threads and Threads per child. Essentially I thought the server was limited to 2000 connections when it was actually being limited to 500. In addition it was constantly shutting down and spinning up new threads because of the low Min/Max Spare Thread directives (75 and 250). So I think it was getting into a state where all the threads were either in use or shutting down but the scorecard wasn't full. Eventually it would become unable to respond in a timely manner to requests and WHM would restart the service. I realized this when checking "apachectl fullstatus" since the scorecard looked a little small for the number of processes that should be running. For anyone who stumbles upon this I was using this statement for the block: ...
I should have used:...0 -
Nice catch! I'm glad you were able to find that! 0
Please sign in to leave a comment.
Comments
7 comments