ERROR: failed to prepare the stderr pipe: Too many open files (24)

APatchworkBoy

• September 30, 2019 06:15

Probably one for @cPanelMichael - I'm finding PHP-FPM stalling, and on checking /opt/cpanel/ea-php71/root/usr/var/log/php-fpm/error.log I'm seeing "ERROR: failed to prepare the stderr pipe: Too many open files (24)" Using the {user} as which php-fpm is specifically running, I'm seeing the following: # ulimit -aSH core file size (blocks, -c) 0 data seg size (kbytes, -d) 200000 scheduling priority (-e) unlimited file size (blocks, -f) unlimited pending signals (-i) unlimited max locked memory (kbytes, -l) unlimited max memory size (kbytes, -m) 200000 open files (-n) 5000 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) unlimited real-time priority (-r) unlimited stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 384 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited
Within /etc/security/limits.conf I've got... {user} hard nofile 500000 {user} soft nofile 500000
And within /etc/security/limits.d/90-nproc.conf I've got... {user} soft nproc 384 {user} soft nofile 5000
And running lsof | wc -l for the user in question returns: 870
In /opt/cpanel/ea-php71/root/etc/php-fpm.d/{user}.conf I've got... [{user}] catch_workers_output = yes chdir = /home/{user} group = "{user}" listen = /opt/cpanel/ea-php71/root/usr/var/run/php-fpm/1733bd16bede7c482577faa383082e8365142cd2.sock listen.group = "nobody" listen.mode = 0660 listen.owner = "{user}" php_admin_flag[allow_url_fopen] = on php_admin_flag[log_errors] = on php_admin_value[disable_functions] = exec,passthru,shell_exec,system php_admin_value[doc_root] = "/home/{user}/public_html/" php_admin_value[error_log] = /home/{user}/logs/{user}.php.error.log php_admin_value[max_input_vars] = 4000 php_admin_value[short_open_tag] = on php_value[error_reporting] = E_ALL & ~E_NOTICE ;php_value[error_reporting] = E_ALL ping.path = /ping pm = ondemand pm.max_children = 200 pm.max_requests = 300 pm.max_spare_servers = 5 pm.min_spare_servers = 1 pm.process_idle_timeout = 20 pm.start_servers = 10 pm.status_path = /status security.limit_extensions = .phtml .php .php3 .php4 .php5 .php6 .php7 user = "{user}"
Any ideas/suggestions on how I can diagnose this further? I'm not seeing any mention of hitting max_children in logs or anything else to indicate issues, just this Too Many Files (24) error... previous posts I've looked at for same error have referred to poss fixes for CentOS7, but not found any referring to CentOS6... CentOS6.10 / WHMv82.0.16 Apache2.4.41 / PHP7.1 / PHP-FPM / MPM: Worker

• 

  cPanelLauren

  • October 04, 2019 18:37

  I've actually seen this previously - what is the output of the following? cat /proc/`pgrep fpm -u 0`/limits | grep 'Max open files'
  I did also notice that you've modified the 90-nproc.conf file as this is the default: cat /etc/security/limits.d/90-nproc.conf # Default limit for number of user's processes to prevent # accidental fork bombs. # See rhbz #432903 for reasoning. * soft nproc 1024 root soft nproc unlimited
  In a lot of cases this has been really low and the services were inheriting the system values. I've seen this resolved in cases for cPanel PHP-FPM on CentOS 6 and Apache PHP-FPM on CentOS 7 but not for Apache PHP-FPM on CentOS 6. On CentOS 7 the resolution was to add some systemd configuration drop-in files. For CentOS 6 with cPanel PHP-FPM the resolution was to create a new .conf include in /var/cpanel/php-fpm.d/ # cat /var/cpanel/php-fpm.d/rlimit.conf [global] rlimit_files = 4096
  This wouldn't be where you'd add an include for Apache PHP-FPM though, this would need to be added to /opt/cpanel/ea-php71/root/etc/php-fpm.d/
  as follows: [root@centos6 php-fpm.d]# cat /opt/cpanel/ea-php71/root/etc/php-fpm.d/rlimit.conf [global] rlimit_files = 4096
  since that's where ALL of the conf files for ea-php71 reside and the master conf file includes from here: [root@centos6 etc]# grep include /opt/cpanel/ea-php71/root/etc/php-fpm.conf include=/opt/cpanel/ea-php71/root/etc/php-fpm.d/*.conf
  Then restart the service with the following: /scripts/restartsrv_apache_php_fpm --stop /scripts/restartsrv_apache_php_fpm --start
  Ultimately though, I can't help but wonder if this is not occurring due to some configuration changes that have been made on the system, you might try the proposed resolution and let me know if that works, I'm not 100% certain on it though it does appear that it should work.

  0
• 

  APatchworkBoy

  • October 04, 2019 21:23

  Thanks @cPanelLauren - shall give that a whirl on Monday and let you know. It"s all being caused by our net connection being maxed out and a custom web app handling it badly... problem only lasts a few minutes - I just need to get the server to survive those few minutes until I get chance to add some suitable timeout/retry-middleware to thenetworg"s oauth2-client. In meantime the app just keeps trying, as do the users, and more and more pools fire up until our connection clears. Bad coding on my part generally!

  0
• 

  cPanelLauren

  • October 04, 2019 21:24

  Ahh, well at the end of the day, at least you know the underlying cause. I'll await your update Monday!

  0

Please sign in to leave a comment.