Too many open files error updating EA3

Hi, We try to recompile apache through EA3 and in the logs /usr/local/cpanel/logs/easy/apache/build.1545162648 it is shown error:

APR Version: 1.5.2
checking for chosen layout... apr
checking for gcc... gcc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables...
checking whether we are cross compiling... configure: error: in `/home/cpeasyapache/src/httpd-2.4/srclib/apr':
configure: error: cannot run C compiled programs.
If you meant to cross compile, use `--host'.
See `config.log' for more details
configure failed for srclib/apr
!! './configure --disable-v4-mapped --enable-access-compat=static --enable-actions=static --enable-alias=static --enable-asis=static --enable-auth_basic=static --enable-authn_core=s   tatic --enable-authn_file=static --enable-authz_core=static --enable-authz_groupfile=static --enable-authz_host=static --enable-authz_user=static --enable-autoindex=static --enable-   cache=static --enable-cache-disk=static --enable-cgi=static --enable-dav=static --enable-dav-fs=static --enable-dbd=static --enable-deflate=static --enable-dir=static --enable-env=s   tatic --enable-expires=static --enable-filter=static --enable-headers=static --enable-include=static --enable-log_config=static --enable-logio=static --enable-mime=static --enable-m   odules=none --enable-negotiation=static --enable-proxy=static --enable-proxy-connect=static --enable-proxy-http=static --enable-rewrite=static --enable-setenvif=static --enable-slot   mem_shm=static --enable-socache_dbm=static --enable-socache_shmcb=static --enable-speling=static --enable-ssl=static --enable-status=static --enable-suexec=static --enable-unique-id   =static --enable-unixd=static --enable-userdir=static --enable-version=static --prefix=/usr/local/apache --with-crypto --with-included-apr --with-mpm=event --with-pcre=/opt/pcre --w   ith-ssl=/usr --with-suexec-caller=nobody --with-suexec-docroot=/ --with-suexec-gidmin=100 --with-suexec-logfile=/usr/local/apache/logs/suexec_log --with-suexec-uidmin=100 --with-sue   xec-userdir=public_html' failed with exit code '256' !!
!! Restoring original working apache !!
!! Executing '/usr/local/cpanel/scripts/initsslhttpd' !!
!! Restarting 'httpd' ... !!
!! Failed to restart Apache. !!
!! Apache restart failed. Unable to load pid from pid file and no httpd process found in process list.

If apache restart reported success but it failed soon after, it may be caused by oddities with mod_ssl.

You should run /usr/local/cpanel/scripts/ssl_crt_status as part of your troubleshooting process. Pass it --help for more details.

Also be sure to examine apache's various log files.
Apache Restart Output:
Apache could not be started due to an error: The "/usr/local/cpanel/scripts/restartsrv_httpd" command (process 7981) reported error number 255 when it ended. Waiting for "httpd" to    start """failed.

Cpanel::Exception::Services::StartError
Service Status

Service Error
        (XID wd36kw) The "httpd" service failed to start.

Log Messages
        [Tue Dec 18 19:55:30.751188 2018] [log_config:error] [pid 8047:tid 140636001519552] (24)Too many open files: AH00649: could not open transfer log file /usr/local/apache/doml   ogs/styldomain.tld-bytes_log.
        [Tue Dec 18 19:54:54.610151 2018] [log_config:error] [pid 7996:tid 139694653900736] (24)Too many open files: AH00649: could not open transfer log file /usr/local/apache/doml   ogs/styldomain.tld-bytes_log.
        [Tue Dec 18 19:54:43.330933 2018] [log_config:error] [pid 7988:tid 139649734272960] (24)Too many open files: AH00649: could not open transfer log file /usr/local/apache/doml   ogs/styldomain.tld-bytes_log.
        [Tue Dec 18 19:49:00.467282 2018] [log_config:error] [pid 4169:tid 139976113272768] (24)Too many open files: AH00649: could not open transfer log file /usr/local/apache/doml   ogs/styldomain.tld-bytes_log.
        [Tue Dec 18 19:42:20.359566 2018] [log_config:error] [pid 3860:tid 140094067562432] (24)Too many open files: AH00649: could not open transfer log file /usr/local/apache/doml   ogs/styldomain.tld-bytes_log.
        [Tue Dec 18 19:34:25.887231 2018] [log_config:error] [pid 3523:tid 140539395446720] (24)Too many open files: AH00649: could not open transfer log file /usr/local/apache/doml   ogs/styldomain.tld-bytes_log.
        [Tue Dec 18 19:30:19.314142 2018] [log_config:error] [pid 2314:tid 140096112945088] (24)Too many open files: AH00649: could not open transfer log file /usr/local/apache/doml   ogs/styldomain.tld-bytes_log.
        [Tue Dec 18 19:25:48.066770 2018] [log_config:error] [pid 6356:tid 139718856509376] (24)Too many open files: AH00649: could not open transfer log file /usr/local/apache/doml   ogs/styldomain.tld-bytes_log.
        [Tue Dec 18 19:25:07.790871 2018] [log_config:error] [pid 6315:tid 139692460992448] (24)Too many open files: AH00649: could not open transfer log file /usr/local/apache/doml   ogs/styldomain.tld-bytes_log.
        [Tue Dec 18 18:59:47.867904 2018] [log_config:error] [pid 5644:tid 139911821748160] (24)Too many open files: AH00649: could not open transfer log file /usr/local/apache/doml   ogs/styldomain.tld-bytes_log.
        [Tue Dec 18 18:53:03.579395 2018] [log_config:error] [pid 5421:tid 140481138522048] (24)Too many open files: AH00649: could not open transfer log file /usr/local/apache/doml   ogs/styldomain.tld-bytes_log.
        [Tue Dec 18 18:52:41.317741 2018] [log_config:error] [pid 5382:tid 139717595887552] (24)Too many open files: AH00649: could not open transfer log file /usr/local/apache/doml   ogs/styldomain.tld-bytes_log.
        [Tue Dec 18 17:56:45.554470 2018] [log_config:error] [pid 3646:tid 140099434481600] (24)Too many open files: AH00649: could not open transfer log file /usr/local/apache/doml   ogs/styldomain.tld-bytes_log.
        [Tue Dec 18 17:55:56.666375 2018] [log_config:error] [pid 3598:tid 139978897274816] (24)Too many open files: AH00649: could not open transfer log file /usr/local/apache/doml   ogs/styldomain.tld-bytes_log.
        [Tue Dec 18 17:48:48.893335 2018] [log_config:error] [pid 3400:tid 139713762670528] (24)Too many open files: AH00649: could not open transfer log file /usr/local/apache/doml   ogs/styldomain.tld-bytes_log.
        [Tue Dec 18 17:46:40.624624 2018] [log_config:error] [pid 3292:tid 139655740245952] (24)Too many open files: AH00649: could not open transfer log file /usr/local/apache/doml   ogs/styldomain.tld-bytes_log.
        [Tue Dec 18 10:16:07.903678 2018] [:error] [pid 11051:tid 140122347525888] [client 185.216.32.134:35830] File does not exist: /usr/local/apache/htdocs/xmlrpc.php
        [Tue Dec 18 09:10:44.146975 2018] [core:notice] [pid 11012:tid 140122548135872] AH00094: Command line: '/usr/local/apache/bin/httpd -D SSL'
        [Tue Dec 18 09:10:44.146835 2018] [mpm_event:notice] [pid 11012:tid 140122548135872] AH00489: Apache/2.4.34 (Unix) OpenSSL/1.0.1e-fips mod_bwlimited/1.4 configured -- resumi   ng normal operations
        [Tue Dec 18 09:10:44.112749 2018] [core:warn] [pid 11012:tid 140122548135872] AH00098: pid file /usr/local/apache/logs/httpd.pid overwritten -- Unclean shutdown of previous    Apache run?
        [Tue Dec 18 09:10:38.510102 2018] [:notice] [pid 10983:tid 140122548135872] ModSecurity for Apache/2.9.2 (http://www.modsecurity.org/) configured.
        [Tue Dec 18 09:10:38.509702 2018] [suexec:notice] [pid 10983:tid 140122548135872] AH01232: suEXEC mechanism enabled (wrapper: /usr/local/apache/bin/suexec)
        [Tue Dec 18 08:56:08.137579 2018] [:error] [pid 10150:tid 140429236360960] [client 212.237.1.209:43398] File does not exist: /usr/local/apache/htdocs/setup.php

httpd has failed. Contact your system administrator if the service does not automagically recover.
 warn [restartsrv_httpd] The system encountered an error while starting the "httpd" service with the command "/usr/local/apache/bin/apachectl start": The subprocess reported error n   umber 1 when it ended.
warn [restartsrv_httpd] The system encountered an error while starting the "httpd" service with the command "/usr/local/apache/bin/apachectl start": The subprocess reported error nu   mber 1 when it ended.
warn [restartsrv_httpd] The system encountered an error while starting the "httpd" service with the command "/usr/local/apache/bin/apachectl start": The subprocess reported error nu   mber 1 when it ended.

Log:
[Tue Dec 18 19:42:20.359566 2018] [log_config:error] [pid 3860:tid 140094067562432] (24)Too many open files: AH00649: could not open transfer log file /usr/local/apache/domlogs/styl   domain.tld-bytes_log.
AH00015: Unable to open logs
[Tue Dec 18 19:49:00.467282 2018] [log_config:error] [pid 4169:tid 139976113272768] (24)Too many open files: AH00649: could not open transfer log file /usr/local/apache/domlogs/styl   domain.tld-bytes_log.
AH00015: Unable to open logs
[Tue Dec 18 19:54:43.330933 2018] [log_config:error] [pid 7988:tid 139649734272960] (24)Too many open files: AH00649: could not open transfer log file /usr/local/apache/domlogs/styl   domain.tld-bytes_log.
AH00015: Unable to open logs
[Tue Dec 18 19:54:54.610151 2018] [log_config:error] [pid 7996:tid 139694653900736] (24)Too many open files: AH00649: could not open transfer log file /usr/local/apache/domlogs/styl   domain.tld-bytes_log.
AH00015: Unable to open logs
[Tue Dec 18 19:55:30.751188 2018] [log_config:error] [pid 8047:tid 140636001519552] (24)Too many open files: AH00649: could not open transfer log file /usr/local/apache/domlogs/styl   domain.tld-bytes_log.
AH00015: Unable to open logs !!
Building global cache for cpanel...Done
!! Executing '/usr/local/cpanel/scripts/initfpsuexec' !!
!! Executing '/usr/local/cpanel/scripts/initsslhttpd' !!
!! Executing '/usr/local/cpanel/scripts/fixmailman' !!
!! '/usr/local/cpanel/scripts/fixmailman' failed with exit code '256' !!
!! Executing '/usr/local/cpanel/scripts/update_apachectl' !!
!! Executing '/usr/local/cpanel/scripts/enablefileprotect' !!
Compiling report...
Sending report. As some of the attached files may be quite large, this could take a few minutes. Thanks for your patience.
!! If you  want to create a support ticket with cPanel regarding this please reference 'BuildAP Report Id': '24023313' !!
Report processed.
!! Verbose logfile is at '/usr/local/cpanel/logs/easy/apache/build.1545162648' !!

The same is shown in the apache log and If we remove the subdomain that shows in the error, it happens to give error in another subdomain. tail -f /usr/local/apache/logs/error_log

[Tue Dec 18 20:14:35.374166 2018] [log_config:error] [pid 8794:tid 140711960737728] (24)Too many open files: AH00649: could not open transfer log file /usr/local/apache/domlogs/example.com-bytes_log.
AH00015: Unable to open logs
[Tue Dec 18 20:20:20.693260 2018] [log_config:error] [pid 8962:tid 140365733238720] (24)Too many open files: AH00649: could not open transfer log file /usr/local/apache/domlogs/example.net-bytes_log.
AH00015: Unable to open logs
[Tue Dec 18 20:20:28.787921 2018] [log_config:error] [pid 8990:tid 140635434727360] (24)Too many open files: AH00649: could not open transfer log file /usr/local/apache/domlogs/example.net-bytes_log.
AH00015: Unable to open logs
[Tue Dec 18 20:21:18.660956 2018] [log_config:error] [pid 9027:tid 140366961801152] (24)Too many open files: AH00649: could not open transfer log file /usr/local/apache/domlogs/example.net-bytes_log.
AH00015: Unable to open logs

I checked that the client has more than 4,000 subdomains created in the control panel:

root@server.domain.tld [~]# grep "DNS" /var/cpanel/users/f56 | wc -l
4304
root@server.domain.tld [~]#

root@server.domain.tld [~]# egrep soft /etc/security/limits.conf
#        - "soft" for enforcing the soft limits
#*               soft    core            0
#@faculty        soft    nproc           20
*       soft nofile 20480
root@server.domain.tld [~]#
root@server.domain.tld [~]# egrep hard /etc/security/limits.conf
#        - "hard" for enforcing hard limits
#*               hard    rss             10000
#@student        hard    nproc           20
#@faculty        hard    nproc           50
#ftp             hard    nproc           0
*       hard nofile 20480
root@server.domain.tld [~]#

I tried to change the values, but it did not work.

root@server.domain.tld [~]# ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 15749
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 10240
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 10240
cpu time               (seconds, -t) unlimited
max user processes              (-u) 15749
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited
root@server.domain.tld [~]#

root@server.domain.tld [~]# ulimit -Hn

10240
root@server.domain.tld [~]# ulimit -Sn
10240
root@server.domain.tld [~]#

root@server.domain.tld [~]#  grep '' /usr/local/cpanel/version
11.68.0.39
root@server.domain.tld [~]#
root@server.domain.tld [~]# named -version
BIND 9.8.2rc1-RedHat-9.8.2-0.68.rc1.el6_10.1
root@server.domain.tld [~]#
root@server.domain.tld [~]# cat /etc/[A-Za-z]*[_-][rv]e[lr]*
CentOS release 6.10 (Final)
CentOS release 6.10 (Final)
CentOS release 6.10 (Final)
cpe:/o:centos:linux:6:GA
root@server.domain.tld [~]#

Can you help us?

cPanelLauren

December 19, 2018 16:17

Hi @Luana Premoli Can you try enabling Piped Logging for apache at WHM>>Service Configuration>>Apache Configuration and then running E3 once more? Furthermore I do need to point out that EA3 is EOL and new versions of cPanel aren't shipped with it, is there a reason you haven't switched over to EA4?

Luana Premoli

December 19, 2018 18:20

We upgraded to EA4 and fixed the error by making the following changes:

root@server.domain.tld [public_html]# egrep nofile /etc/security/limits.conf
#        - nofile - max number of open file descriptors
*       soft nofile 60480
*       hard nofile 60480
root@server.domain.tld [public_html]#

root@server.domain.tld [public_html]# egrep FD /etc/sysconfig/httpd
# cPanel Specific Variables to allow users to override FD limits
MIN_FD_LIMIT=10000
MAX_FD_LIMIT=60000
root@server.domain.tld [public_html]#

Thanks

December 19, 2018 19:30

Hi @Luana Premoli I'm really glad to hear that you were able to resolve the issue AND get moved over to EA4!!! That's great news! Thanks for updating us with the steps that fixed it for you as well.

Too many open files error updating EA3

Comments

Didn't find what you were looking for?