Skip to main content

AH01095: prefetch request body failed to 0.0.0.0:0 (httpd-UDS) from (my server ip)

nootkan
AlmaLinux v8.8.0 STANDARD standard cPanel Version 114.0.8 Apache Version 2.4.57 MySQL Version 8.0.34 Architecture x86_64 Kernel Version 4.18.0-477.27.1.el8_8.x86_64 Recently migrated to a new server from old one running Centos 7 (last supported version) had "Way To Web" install their csf/mailscanner package for me as I have done everytime I migrated to a new server in the past. Seeing a few server issues I am trying to clean up and the following ones seem to be eluding me. I seem to be seeing this (AH01095: prefetch request body failed to 0.0.0.0:0 (httpd-UDS) from (my server ip)) a lot inside my apache2 error logs and was wondering what it means? I discovered it after troubleshooting why a webpage ( [uri "/needleSearch/assets/vendor/bootstrap-5.2.2-dist/css/bootstrap.min.css"> [unique_id "ZTgDuy-qu_T7p9VrV-cwMwAAAUs">, referer: Looking at my logs again I see the modsecurity errors disappeared. I watched my apache2 error log live with "tail -f /etc/apache2/logs/error_log" and saw multiple issues with the AH01095: prefetch request body failed to 0.0.0.0:0 (httpd-UDS) from (my server ip) error after multiple "ctrl F5" of the webpage in question. Are the modsecurity errors and my changing the permissions for the secdatadir folder/files related to the AH01095: prefetch request body failed errors somehow? Just a note my server logs now seem to be filling up with these errors: AH01095: prefetch request body failed to 0.0.0.0:0 (httpd-UDS) from (my server ip) Not sure what to do now and need some guidance from the experts please.

Comments

18 comments

Date Votes
  • cPRex Jurassic Moderator
    • Edited

    Hey there! I believe this article will help you get that sorted out:

    https://support.cpanel.net/hc/en-us/articles/360052471154-How-to-fix-Failed-to-access-DBM-file

    0
  • nootkan

    Hey there! I believe this article will help you get that sorted out:

    Hi, cPRex, thanks for the reply and the link.

    Checked that out, and mod_ruid2 was disabled and mod_suex was enabled. So that is not the issue.

    Changed the permissions back to the original root:nobody to see if that stops the influx of AH01095: prefetch request body failed to 0.0.0.0:0 (httpd-UDS) from (my server ip) messages and reduces the load on my server to what it was before the change.

    Hopefully I can find out what is going on.
     
    0
  • cPRex Jurassic Moderator
    If that didn't take care of the issue, could you submit a ticket to our team so we can do some additional troubleshooting?
    0
  • nootkan
    cPRex, okay will do. After changing permissions back and restarting apache, the [quote]AH01095: prefetch request body failed to 0.0.0.0:0 (httpd-UDS) from (my server ip)
    messages seemed to slow down and only appear after the modsecurity messages one of which is associated with a google ip and the others with mine: [Tue Oct 24 13:49:44.514317 2023] [mpm_worker:notice] [pid 3528336:tid 22853442694336] AH00295: caught SIGTERM, shutting down [Tue Oct 24 13:49:45.050826 2023] [security2:notice] [pid 3543122:tid 23131270712512] ModSecurity for Apache/2.9.7 (http://www.modsecurity.org/) configured. [Tue Oct 24 13:49:45.050853 2023] [security2:notice] [pid 3543122:tid 23131270712512] ModSecurity: APR compiled version="1.7.4"; loaded version="1.7.4" [Tue Oct 24 13:49:45.050856 2023] [security2:notice] [pid 3543122:tid 23131270712512] ModSecurity: PCRE compiled version="8.42 "; loaded version="8.42 2018-03-20" [Tue Oct 24 13:49:45.050858 2023] [security2:notice] [pid 3543122:tid 23131270712512] ModSecurity: LUA compiled version="Lua 5.3" [Tue Oct 24 13:49:45.050860 2023] [security2:notice] [pid 3543122:tid 23131270712512] ModSecurity: YAJL compiled version="2.1.0" [Tue Oct 24 13:49:45.050862 2023] [security2:notice] [pid 3543122:tid 23131270712512] ModSecurity: LIBXML compiled version="2.11.5" [Tue Oct 24 13:49:45.050864 2023] [security2:notice] [pid 3543122:tid 23131270712512] ModSecurity: Status engine is currently disabled, enable it by set SecStatusEngine to On. [Tue Oct 24 13:49:45.051154 2023] [suexec:notice] [pid 3543122:tid 23131270712512] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec) [Tue Oct 24 13:49:45.097343 2023] [mpm_worker:notice] [pid 3543125:tid 23131270712512] AH00292: Apache/2.4.57 (cPanel) OpenSSL/1.1.1k mod_bwlimited/1.4 configured -- resuming normal operations [Tue Oct 24 13:49:45.097362 2023] [core:notice] [pid 3543125:tid 23131270712512] AH00094: Command line: '/usr/sbin/httpd' [Tue Oct 24 13:50:38.146125 2023] [security2:error] [pid 3543130:tid 23131035764480] [client 114.119.132.187:57185] [client 114.119.132.187] ModSecurity: collections_remove_stale: Failed to access DBM file "/var/cpanel/secdatadir/nobody-ip": Permission denied [hostname "my website"> [uri "/index.php"> [unique_id "ZTguHlNFVT-aRg-KFv0UWQAAAAQ">, referer: my website/search/tags,childcraft%2Bencyclopedia%2Bcollection/sOrder,i_price/iOrderType,desc/category,105 [Tue Oct 24 13:50:43.706579 2023] [proxy:error] [pid 3543133:tid 23131035764480] (70014)End of file found: [client my server ip:33906] AH01095: prefetch request body failed to 0.0.0.0:0 (httpd-UDS) from my server ip () [Tue Oct 24 13:50:52.730692 2023] [proxy:error] [pid 3543134:tid 23131033663232] (70014)End of file found: [client my server ip:47648] AH01095: prefetch request body failed to 0.0.0.0:0 (httpd-UDS) from my server ip () [Tue Oct 24 13:50:59.928181 2023] [proxy:error] [pid 3543134:tid 23131031561984] (70014)End of file found: [client my server ip:47656] AH01095: prefetch request body failed to 0.0.0.0:0 (httpd-UDS) from my server ip () [Tue Oct 24 13:51:28.123035 2023] [security2:error] [pid 3543133:tid 23131027359488] [client 66.249.64.161:61052] [client 66.249.64.161] ModSecurity: collections_remove_stale: Failed to access DBM file "/var/cpanel/secdatadir/nobody-global": Permission denied [hostname "my website"> [uri "/wp-content/plugins/nextgen-gallery/products/photocrati_nextgen/modules/ajax/static/ajax.min.js"> [unique_id "ZTguUMPEJgAU5nf-WyQ_BQAAAMg">, referer: my website/frames/ [Tue Oct 24 13:51:28.123071 2023] [security2:error] [pid 3543133:tid 23131027359488] [client 66.249.64.161:61052] [client 66.249.64.161] ModSecurity: collections_remove_stale: Failed to access DBM file "/var/cpanel/secdatadir/nobody-ip": Permission denied [hostname "my website"> [uri "/wp-content/plugins/nextgen-gallery/products/photocrati_nextgen/modules/ajax/static/ajax.min.js"> [unique_id "ZTguUMPEJgAU5nf-WyQ_BQAAAMg">, referer: my website/frames/ [Tue Oct 24 13:51:35.640452 2023] [proxy:error] [pid 3543132:tid 23131023156992] (70014)End of file found: [client my server ip:58878] AH01095: prefetch request body failed to 0.0.0.0:0 (httpd-UDS) from my server ip ()
    0
  • cPRex Jurassic Moderator
    Let me know that ticket number once you have it so I can follow along.
    0
  • nootkan
    [quote]Let me know that ticket number once you have it so I can follow along.
    cPRex, here is the support ticket id: #95143242 as requested.
    0
  • cPRex Jurassic Moderator
    Thanks for that - I'm following along with that ticket now.
    0
  • nootkan
    Just thought I'd let you know that you were right cPRex about the [quote] /var/cpanel/secdatadir/nobody-global": Permission denied
    fix for the modsecurity issues. Turns out the two issues were unrelated and support discovered I had PHP-FPM enabled for all domains and once I disabled them my [quote] (AH01095: prefetch request body failed to 0.0.0.0:0 (httpd-UDS) from (my server ip))
    issues were resolved also. Thanks for all your help.
    0
  • cPRex Jurassic Moderator
    Nice - I'm glad we could help!
    0
  • Metro2

    Sure would love to see the actual helpful content of this thread before it was destroyed by zendesk , as I'm dealing with something very similar right now and it's killing my business.

    0
  • cPRex Jurassic Moderator

    Metro2 - updated the links!

    0
  • Metro2

    Thank you!

    Apparently my issue is different. Occasionally this is happening when users are trying to edit their WP sites:

    [proxy:error] [pid 1720442:tid 1720501] (103)Software caused connection abort: [remote 123.456.123.456:51153] AH01095: prefetch request body failed to /opt/cpanel/ea-php83/root/usr/var/run/php-fpm/7084330874ef36e06ba2b0eea6274145ce3c4b12.sock:0 (example.com) from 123.456.123.456 ()

    I have modsecurity disabled (because the cPanel OWASP rules were breaking a lot of functions, and I could not find exclusions to work with) and I don't have mod_ruid2 enabled. CloudLinux 8 and cPanel 120.0.14 , and /var/cpanel/secdatadir/ contents are this:

    drwxrwx--T   2 root   nobody  4096 Jul 26 10:00 ./
    drwx--x--x 127 root   root   12288 Jul 26 11:12 ../
    -rw-r-----   1 nobody nobody     0 Jul 26 10:00 nobody-global.dir
    -rw-r-----   1 nobody nobody     0 Jul 26 10:00 nobody-global.pag
    -rw-r-----   1 nobody nobody     0 Jul 26 10:00 nobody-ip.dir
    -rw-r-----   1 nobody nobody     0 Jul 26 10:00 nobody-ip.pag

    So while my issue seems related to this topic, not quite the same. :/

     

    0
  • cPRex Jurassic Moderator

    Around the same time in the log, do you see any entries showing "timeout" errors too?  If so, editing the ProxyTimeout setting as described in https://support.cpanel.net/hc/en-us/articles/1500007032562-How-to-modify-system-PHP-FPM-settings-in-WHM could be helpful.

    Other than that, I don't have too many ideas on this one.  If you can reproduce the issue while editing a certain WordPress site, it's the perfect candidate for a ticket.

    0
  • Metro2
    • Edited

    Thank you. I'm not really seeing many abnormal timeouts, but here's a small excerpt from a test session I did with a user:


    [Fri Jul 26 02:08:47.751937 2024] [proxy_fcgi:error] [pid 1720442:tid 1721917] [remote   123.456.123.456:51092] AH01067: Failed to read FastCGI header
    [Fri Jul 26 02:08:47.751961 2024] [proxy_fcgi:error] [pid 1720442:tid 1721917] (104)Connection reset by peer: [remote   123.456.123.456:51092] AH01075: Error dispatching request to :
    [Fri Jul 26 02:08:48.897258 2024] [proxy_fcgi:error] [pid 1720442:tid 1720486] [remote   123.456.123.456:51092] AH01067: Failed to read FastCGI header
    [Fri Jul 26 02:08:48.897283 2024] [proxy_fcgi:error] [pid 1720442:tid 1720486] (104)Connection reset by peer: [remote   123.456.123.456:51092] AH01075: Error dispatching request to :
    [Fri Jul 26 02:08:49.689334 2024] [proxy_fcgi:error] [pid 1720442:tid 1720503] [remote   123.456.123.456:51092] AH01067: Failed to read FastCGI header
    [Fri Jul 26 02:08:49.689369 2024] [proxy_fcgi:error] [pid 1720442:tid 1720503] (104)Connection reset by peer: [remote   123.456.123.456:51092] AH01075: Error dispatching request to :
    [Fri Jul 26 02:08:50.561965 2024] [proxy_fcgi:error] [pid 1720442:tid 1720499] [remote   123.456.123.456:51092] AH01067: Failed to read FastCGI header
    [Fri Jul 26 02:08:50.566690 2024] [proxy_fcgi:error] [pid 1720442:tid 1720499] (104)Connection reset by peer: [remote   123.456.123.456:51092] AH01075: Error dispatching request to :
    [Fri Jul 26 02:08:51.853219 2024] [proxy_fcgi:error] [pid 1720442:tid 1720520] [remote   123.456.123.456:51092] AH01067: Failed to read FastCGI header
    [Fri Jul 26 02:10:53.352749 2024] [proxy_fcgi:error] [pid 1720442:tid 1720504] [remote   123.456.123.456:51144] AH01067: Failed to read FastCGI header
    [Fri Jul 26 02:10:53.352786 2024] [proxy_fcgi:error] [pid 1720442:tid 1720504] (104)Connection reset by peer: [remote   123.456.123.456:51144] AH01075: Error dispatching request to :
    [Fri Jul 26 02:11:02.680077 2024] [proxy_fcgi:error] [pid 1720442:tid 1720513] [remote   123.456.123.456:51144] AH01067: Failed to read FastCGI header
    [Fri Jul 26 02:11:02.680115 2024] [proxy_fcgi:error] [pid 1720442:tid 1720513] (104)Connection reset by peer: [remote   123.456.123.456:51144] AH01075: Error dispatching request to :


    [Fri Jul 26 02:11:34.441421 2024] [proxy:error] [pid 1720442:tid 1720501] (103)Software caused connection abort: [remote   123.456.123.456:51153] AH01095: prefetch request body failed to /opt/cpanel/ea-php83/root/usr/var/run/php-fpm/7084330874ef36e06ba2b0eea6274145ce3c4b12.sock:0 (justfixthismess.com) from   123.456.123.456 ()


    [Fri Jul 26 02:11:41.129018 2024] [proxy_fcgi:error] [pid 1720442:tid 1720514] [remote   123.456.123.456:51153] AH01067: Failed to read FastCGI header
    [Fri Jul 26 02:11:41.129053 2024] [proxy_fcgi:error] [pid 1720442:tid 1720514] (104)Connection reset by peer: [remote   123.456.123.456:51153] AH01075: Error dispatching request to :
    [Fri Jul 26 02:13:27.268981 2024] [proxy_fcgi:error] [pid 1720456:tid 1720551] [remote   123.456.123.456:51187] AH01067: Failed to read FastCGI header
    [Fri Jul 26 02:13:27.269029 2024] [proxy_fcgi:error] [pid 1720456:tid 1720551] (104)Connection reset by peer: [remote   123.456.123.456:51187] AH01075: Error dispatching request to :
    [Fri Jul 26 02:14:22.580261 2024] [proxy_fcgi:error] [pid 1771239:tid 1771386] [remote   123.456.123.456:51233] AH01067: Failed to read FastCGI header
    [Fri Jul 26 02:14:22.580302 2024] [proxy_fcgi:error] [pid 1771239:tid 1771386] (104)Connection reset by peer: [remote   123.456.123.456:51233] AH01075: Error dispatching request to :
    [Fri Jul 26 02:14:24.612724 2024] [proxy_fcgi:error] [pid 1771239:tid 1771383] [remote   123.456.123.456:51233] AH01067: Failed to read FastCGI header
    [Fri Jul 26 02:14:24.612763 2024] [proxy_fcgi:error] [pid 1771239:tid 1771383] (104)Connection reset by peer: [remote   123.456.123.456:51233] AH01075: Error dispatching request to :
    [Fri Jul 26 02:14:37.566794 2024] [proxy_fcgi:error] [pid 1720442:tid 1720499] [remote   123.456.123.456:51242] AH01067: Failed to read FastCGI header
    [Fri Jul 26 02:14:37.566833 2024] [proxy_fcgi:error] [pid 1720442:tid 1720499] (104)Connection reset by peer: [remote   123.456.123.456:51242] AH01075: Error dispatching request to :

     

    And it just goes on and on like that. And they get "503" error.

    This is happening on many customer WP sites not just one and I noticed that when it does, their account starts hitting max physical memory limit of 1GB that I have set in CloudLinux LVE manager. Ironically it was not happening on my old CL 6 / cPanel 110 / MySQL 5.7 / PHP 7.4 servers, but now that I've migrated everyone to my brand new more powerful servers running MySQL 8 and ea-PHP 8.3 at a better data center, this trouble has started. Extremely stressful.

    Edit - server has Intel Xeon Silver 4214 CPU @ 2.20GHz (24 cores) and 192GB DDR4 RAM and NVMe SSD - plenty of power.

    0
  • cPRex Jurassic Moderator

    As a test, can you try disabling PHP-FPM and turning it back on? 

    0
  • Metro2

    I'm trying to find a user who can do a test session with me and to do exactly that! I don't take any chances on editing their websites without them. I'm also setting up a test WP site shortly.

    0
  • Metro2

    I disabled PHP-FPM just on the account that I showed the Apache error log excerpt from and had the customer test. As you suspected, the errors I posted above stopped. The site is very slow now and hitting CPU LVE throttles during edit, but no 503's.

    0
  • cPRex Jurassic Moderator

    That at least tells us something is up with FPM, although I can't say for sure what.  This would be an ideal one for a ticket since it's easy to reproduce.

    0

Please sign in to leave a comment.

Didn't find what you were looking for?

New post