Skip to main content

AH00288: scoreboard is full, not at MaxRequestWorkers – overlapping worker generations after graceful restarts

Zoltan Egri | 1b.hu

We have been investigating the same Apache failure mode on two separate production cPanel servers running Apache 2.4 with worker MPM.

The primary error was:

 
AH00288: scoreboard is full, not at MaxRequestWorkers
 

The most interesting part is "not at MaxRequestWorkers".

The servers were not actually at the MaxRequestWorkers limit, so simply increasing the request-worker limit did not appear to address the condition being reported.

During the investigation we reconstructed Apache process generations around graceful restarts. We were able to confirm that child processes from an older Apache generation were still alive after a later graceful restart had already created a new generation.

Typical log sequence:

 
AH00297: SIGUSR1 received. Doing graceful restart
AH00292: Apache/2.4.x configured -- resuming normal operations
 

In one observed incident, worker processes created around 07:44 were still present after another graceful restart at approximately 07:50.

This meant that the previous Apache generation, the newly started generation and several older workers attempting to finish existing connections were present simultaneously.

AH00288 appeared while these generations overlapped.

In the same problematic time windows we also saw:

 
AH00646: Error writing to |/usr/local/cpanel/bin/splitlogs: Broken pipe
 

cPanel documents a graceful-restart-related splitlogs race under CPANEL-53462 and references Apache Bug 61926.

I want to be careful about causality here.

We are not claiming that CPANEL-53462 or the splitlogs Broken pipe condition causes AH00288.

What we can demonstrate from the production logs is that both conditions occurred around graceful restart activity and that multiple Apache worker generations remained alive simultaneously.

We also checked the more obvious resource-exhaustion explanations.

We did not find corresponding evidence of:

  • OOM or out-of-memory events
  • killed processes
  • hung tasks
  • nf_conntrack table exhaustion
  • segmentation faults
  • CPU saturation sufficient to explain the outage

Another confusing symptom was that Apache itself remained active:

 
systemctl is-active httpd
active
 

while real HTTPS requests could already time out.

A process-level or TCP-level health check could therefore succeed while actual HTTP/HTTPS service was effectively unavailable.

The affected worker-MPM configuration used:

 
MaxConnectionsPerChild 10000
 

Based on the observed behavior and cPanel's AH00288 documentation, we changed this to:

 
MaxConnectionsPerChild 50000
 

on both affected systems.

We deliberately did not increase MaxRequestWorkers because AH00288 explicitly reported that the scoreboard was full while the server was not at that limit.

The change was applied through the cPanel configuration layer rather than by directly editing the generated:

 
/etc/apache2/conf/httpd.conf
 

Afterwards we performed a clean Apache stop/start so that old worker generations from the incident would not remain alive during the validation period.

We consider MaxConnectionsPerChild 50000 a mitigation, not a proven universal fix.

The real validation is whether the servers continue to survive future automatic graceful restart cycles without returning to AH00288.

Imunify360-related activity was one source of graceful restarts visible in our logs.

However, we are not claiming that Imunify360 causes AH00288. It is simply one observed source of graceful restart activity in these environments.

We published the complete production investigation here:

AH00288 on cPanel: why the Apache scoreboard fills up after graceful restarts
https://1b.hu/en/blog/ah00288-cpanel-apache-scoreboard-full-graceful-restart

Related earlier thread from the same hosting environment:

CPANEL-53195 – External requests trigger HTTP 500 via 404 safelock failure
https://support.cpanel.net/hc/en-us/community/posts/43150129994007-CPANEL-53195-External-requests-trigger-HTTP-500-via-cPanel-404-safelock-failure

I would be interested in cPanel's view on three points:

  1. Has cPanel observed AH00288 in worker MPM environments where old Apache generations remain in graceful state across repeated graceful restarts?
  2. Is there any known relationship between the conditions tracked under CPANEL-53462 / Apache Bug 61926 and scoreboard pressure beyond the splitlogs Broken pipe symptom itself?
  3. Is increasing MaxConnectionsPerChild still the recommended mitigation when AH00288 occurs while Apache explicitly reports that it is not at MaxRequestWorkers?

I can provide additional anonymized process timelines and Apache log sequences if useful.

Comments

3 comments

Date Votes
  • cPRex Jurassic Moderator

    Hey there!  First of all, this is incredibly well documented - thanks for being so clear and providing this level of detail.

    This sounds like known Apache behavior, unrelated to any cPanel tools.  I'll answer your three questions first and then send my findings:

    1 - Yes, this is expected behavior with the worker MPM
    2 - No
    3 - Partially.  MaxConnectionsPerChild helps, but GracefulShutdownTimeout and ServerLimit address the scoreboard directly and may also need some tweaking.

    Additional details below - I had to dig for some of this, but hopefully it's helpful.

    Your findings are all correct, and you were right not to touch MaxRequestWorkers. The scoreboard is sized by ServerLimit × ThreadLimit, not by MaxRequestWorkers. MaxRequestWorkers caps workers that are actively serving requests; the scoreboard counts every slot, including children draining in graceful state that are serving nobody. So AH00288 is describing exactly what your timelines show.

    MaxConnectionsPerChildset to 10000 on a busy worker-MPM box means constant child recycling, and every recycling child holds its slot until its threads finish. Add frequent graceful restarts on top and the generations stack up. Raising it to 50000 will help, but you may also want to adjust GracefulShutdownTimeout as it defaults to 0, meaning Apache waits forever for those old children. Set it to 30s or 60s and the pileup is now limited no matter what's stalling them. Bumping ServerLimit also gives you some scoreboard headroom.

    Graceful children generally only linger if their threads are stuck on something, such as long keepalives, slow clients, etc. That's the actual root cause behind the overlap.

    If you want us to take a deeper look at your particular situation you're always welcome to create a ticket!

     

    0
  • Zoltan Egri | 1b.hu

    Thanks for the detailed explanation — it was very helpful.

    I did some additional investigation on the server based on your comments, and I can now provide a bit more runtime data.

    The server is using the worker MPM with the following explicit configuration:

    StartServers 5
ServerLimit 256
MaxRequestWorkers 150
MaxConnectionsPerChild 50000

    There is currently no explicit ThreadsPerChild, ThreadLimit, or GracefulShutdownTimeout directive in the Apache configuration.

    The runtime process layout also appears consistent with the default ThreadsPerChild 25: normal worker children show 27 LWPs, while MaxRequestWorkers is 150.

    One particularly interesting finding is the frequency of graceful restarts. There were many SIGUSR1 graceful restarts during a single day. I was also able to confirm that at least several of these were initiated through cPanel's restartsrv_httpd.

    For example:

    Sep 4 08:05:32 Started /usr/local/cpanel/scripts/restartsrv_httpd
Sep 4 09:32:08 Started /usr/local/cpanel/scripts/restartsrv_httpd
Sep 4 09:40:22 Started /usr/local/cpanel/scripts/restartsrv_httpd
Sep 4 09:45:21 Started /usr/local/cpanel/scripts/restartsrv_httpd
Sep 4 11:25:34 Started /usr/local/cpanel/scripts/restartsrv_httpd

    The 11:25 event is especially interesting because the Apache timeline is:

    11:25:34  cPanel restartsrv_httpd started
11:25:37  SIGUSR1 received. Doing graceful restart
11:29:42  AH00288: scoreboard is full, not at MaxRequestWorkers
11:30:17  AH00288
11:30:18  AH00288

    Another example:

    11:34:06  SIGUSR1 received. Doing graceful restart
11:34:26  AH00288: scoreboard is full, not at MaxRequestWorkers

    So there does seem to be a strong temporal relationship between graceful restarts and the AH00288 condition, although not every graceful restart produces AH00288.

    I also want to clarify one point from my earlier process observations. One of the single-threaded processes I initially suspected might be a lingering worker turned out to be the mod_lsapi Selfstarter:

    mod_lsapi: Selfstarter 22894 started

    so I am no longer treating that process as evidence of a stuck worker generation.

    There is also a separate load issue on this server: at other times Apache logs both:

    AH00287: server is within MinSpareThreads of MaxRequestWorkers
AH00286: server reached MaxRequestWorkers

    Therefore I don't want to conflate the two conditions. The server sometimes genuinely reaches MaxRequestWorkers, but the AH00288 events appear separately and are frequently associated with graceful restart activity.

    One thing I would particularly appreciate clarification on is GracefulShutdownTimeout.

    The Apache documentation describes this directive in the context of a graceful-stop signal. In this case the events we are investigating are normal graceful restarts (SIGUSR1 / apachectl -k graceful).

    Does GracefulShutdownTimeout also limit old-generation children during a SIGUSR1 graceful restart with the worker MPM, or does it only apply to graceful shutdown (graceful-stop)?

    I would like to confirm that distinction before changing it to 30 or 60 seconds.

    Also, since ServerLimit is already 256 while MaxRequestWorkers is only 150, I am hesitant to increase ServerLimit further without first understanding why the old generations are consuming the scoreboard.

    At this point, the frequent graceful restarts themselves look like an important part of the problem, so I am also investigating what is causing so many restartsrv_httpd executions.

    Thanks again for taking the time to look into this.

    0
  • cPRex Jurassic Moderator

    GracefulShutdownTimeout only applies to graceful-stop, so that was a good catch.  That does shift where to look.  Since nothing times out those children, the only way to bound how long they hold slots is to bound how long their requests can run: Timeout, KeepAliveTimeout, and on this box the mod_lsapi and PHP-side limits, given mod_lsapi is in the request path.

    Your numbers are the more interesting part, though. ThreadsPerChild defaults to 25, so MaxRequestWorkers 150 needs 6 children per generation, and ServerLimit 256 gives you 256 process slots. Filling that takes around 42 generations' worth of children, considerably more than ordinary restart overlap accounts for.

    So the first thing I'd want to know: how many httpd processes are actually alive, and how old are they?

    ps -eo pid,lstart,nlwp,cmd | grep -c '[h]ttpd'
ps -eo pid,lstart,nlwp,cmd | grep '[h]ttpd' | sort -k2

    If that number is near 250, you've confirmed the slots are accumulating across restarts.

    I would also say that you're right not to raise ServerLimit.  Raising it at this point would only obscure the cause.

    0

Please sign in to leave a comment.

Didn't find what you were looking for?

New post