Apache Proxy, cpanel services and vulnerability scanners

bloatedstoat

• September 05, 2026 16:33
• Edited

Hello,

We're seeing a ton of this nonsense from Google Ips, in short targeted vulnerability scanning and all proxy stuff:

grep "09/05/2026:14:47" /usr/local/cpanel/logs/access_log
_GOOGLE_IP_REDACTED_ proxy - [09/05/2026:14:47:20 -0000] "GET / HTTP/1.1" 200 0 "-" "Mozilla/5.0 (Linux; Android 10; K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/147.0.0.0 Mobile Safari/537.36" "-" "X-Forwarded-For: _GOOGLE_IP_REDACTED_" 443
_GOOGLE_IP_REDACTED_ proxy - [09/05/2026:14:47:21 -0000] "GET /400.shtml HTTP/1.1" 404 0 "-" "Mozilla/5.0 (Linux; Android 10; K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/147.0.0.0 Mobile Safari/537.36" "-" "-" 443
_GOOGLE_IP_REDACTED_ proxy - [09/05/2026:14:47:21 -0000] "GET /400.shtml HTTP/1.1" 500 0 "-" "Mozilla/5.0 (Linux; Android 10; K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/147.0.0.0 Mobile Safari/537.36" "-" "-" 443
_GOOGLE_IP_REDACTED_ proxy - [09/05/2026:14:47:21 -0000] "GET /asset-manifest.json HTTP/1.1" 404 0 "-" "Mozilla/5.0 (Linux; Android 10; K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/147.0.0.0 Mobile Safari/537.36" "-" "X-Forwarded-For: _GOOGLE_IP_REDACTED_" 443
_GOOGLE_IP_REDACTED_ proxy - [09/05/2026:14:47:21 -0000] "GET /asset-manifest.json HTTP/1.1" 500 0 "-" "Mozilla/5.0 (Linux; Android 10; K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/147.0.0.0 Mobile Safari/537.36" "-" "X-Forwarded-For: _GOOGLE_IP_REDACTED_" 443
_GOOGLE_IP_REDACTED_ proxy - [09/05/2026:14:47:21 -0000] "GET /api/fs/read?path=/app/.env&allowOutsideWorkspace=true HTTP/1.1" 404 0 "-" "Mozilla/5.0 (Linux; Android 10; K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/147.0.0.0 Mobile Safari/537.36" "-" "-" 443
_GOOGLE_IP_REDACTED_ proxy - [09/05/2026:14:47:21 -0000] "GET /api/fs/read?path=/app/.env&allowOutsideWorkspace=true HTTP/1.1" 500 0 "-" "Mozilla/5.0 (Linux; Android 10; K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/147.0.0.0 Mobile Safari/537.36" "-" "-" 443
_GOOGLE_IP_REDACTED_ proxy - [09/05/2026:14:47:22 -0000] "GET /assets/manifest.json HTTP/1.1" 404 0 "-" "Mozilla/5.0 (Linux; Android 10; K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/147.0.0.0 Mobile Safari/537.36" "-" "X-Forwarded-For: _GOOGLE_IP_REDACTED_" 443
_GOOGLE_IP_REDACTED_ proxy - [09/05/2026:14:47:22 -0000] "GET /assets/manifest.json HTTP/1.1" 500 0 "-" "Mozilla/5.0 (Linux; Android 10; K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/147.0.0.0 Mobile Safari/537.36" "-" "X-Forwarded-For: _GOOGLE_IP_REDACTED_" 443

Excerpt from the cpanel log:
/usr/local/cpanel/logs/error_log:

[2026-09-06 00:47:21 +1000] warn [whostmgrd] Duplicate logaccess (GET /400.shtml HTTP/1.1):  at /usr/local/cpanel/Cpanel/Server/Logger.pm line 91, <GEN12> line 2.
    Cpanel::Server::Logger::logaccess(Cpanel::Server::Logger=HASH(0x44f6560)) called at /usr/local/cpanel/Cpanel/Server.pm line 1102
    Cpanel::Server::body_internal_error(Cpanel::Server=HASH(xxxxxxxxx), 500, "Error ID 18cff0713d22b") called at /usr/local/cpanel/Cpanel/Server.pm line 1025
    Cpanel::Server::internal_error_with_http_status(Cpanel::Server=HASH(xxxxxxxxx), 500, "Error ID 18cff0713d22b") called at /usr/local/cpanel/Cpanel/Server.pm line 1008
    Cpanel::Server::internal_error(Cpanel::Server=HASH(xxxxxxxxx), "Error ID 18cff0713d22b") called at cpsrvd.pl line 1050
    cpanel::cpsrvd::__ANON__(__CPANEL_HIDDEN__...) called at /usr/local/cpanel/Cpanel/Server.pm line 2651
    Cpanel::Server::_block_login_if_brute(Cpanel::Server=HASH(xxxxxxxxx), "user", undef, "status", -1, "skip_hulk", 0) called at /usr/local/cpanel/Cpanel/Server.pm line 2611
    Cpanel::Server::check_hulk_preauth_ratelimiting(Cpanel::Server=HASH(xxxxxxxxx), __CPANEL_HIDDEN__, 0, __CPANEL_HIDDEN__, undef, __CPANEL_HIDDEN__, 1787388023, __CPANEL_HIDDEN__, ...) called at cpsrvd.pl line 7193
    cpanel::cpsrvd::docheckpass_whostmgrd(__CPANEL_HIDDEN__, __CPANEL_HIDDEN__, __CPANEL_HIDDEN__, undef, __CPANEL_HIDDEN__, undef, __CPANEL_HIDDEN__, undef, ...) called at cpsrvd.pl line 1850
    cpanel::cpsrvd::handle_one_connection(4) called at cpsrvd.pl line 1214
    cpanel::cpsrvd::script() called at cpsrvd.pl line 465
cpsrvd 18cff0713d22b: Missing required arguments! at /usr/local/cpanel/Cpanel/Server.pm line 2651, <GEN13> line 2.

[2026-09-06 00:47:21 +1000] warn [whostmgrd] Duplicate logaccess (GET /asset-manifest.json HTTP/1.1):  at /usr/local/cpanel/Cpanel/Server/Logger.pm line 91, <GEN13> line 2.
    Cpanel::Server::Logger::logaccess(Cpanel::Server::Logger=HASH(xxxxxxxxx)) called at /usr/local/cpanel/Cpanel/Server.pm line 1102
    Cpanel::Server::body_internal_error(Cpanel::Server=HASH(xxxxxxxxx), 500, "Error ID 18cff0713d22b") called at /usr/local/cpanel/Cpanel/Server.pm line 1025
    Cpanel::Server::internal_error_with_http_status(Cpanel::Server=HASH(xxxxxxxxx), 500, "Error ID 18cff0713d22b") called at /usr/local/cpanel/Cpanel/Server.pm line 1008
    Cpanel::Server::internal_error(Cpanel::Server=HASH(xxxxxxxxx), "Error ID 18cff0713d22b") called at cpsrvd.pl line 1050
    cpanel::cpsrvd::__ANON__(__CPANEL_HIDDEN__...) called at /usr/local/cpanel/Cpanel/Server.pm line 2651
    Cpanel::Server::_block_login_if_brute(Cpanel::Server=HASH(xxxxxxxxx), "user", undef, "status", -1, "skip_hulk", 0) called at /usr/local/cpanel/Cpanel/Server.pm line 2611
    Cpanel::Server::check_hulk_preauth_ratelimiting(Cpanel::Server=HASH(xxxxxxxxx), __CPANEL_HIDDEN__, undef, __CPANEL_HIDDEN__, __CPANEL_HIDDEN__..., __CPANEL_HIDDEN__, __CPANEL_HIDDEN__, __CPANEL_HIDDEN__, ...) called at cpsrvd.pl line 7193
    cpanel::cpsrvd::docheckpass_whostmgrd(__CPANEL_HIDDEN__, __CPANEL_HIDDEN__, __CPANEL_HIDDEN__, undef, __CPANEL_HIDDEN__, undef, __CPANEL_HIDDEN__, undef, ...) called at cpsrvd.pl line 1850
    cpanel::cpsrvd::handle_one_connection(4) called at cpsrvd.pl line 1214
    cpanel::cpsrvd::script() called at cpsrvd.pl line 465
cpsrvd 18cff0713d22b: Missing required arguments! at /usr/local/cpanel/Cpanel/Server.pm line 2651, <GEN14> line 2.

We have whm and cpanel ports closed in our firewall, it appears that the requests are actually travelling via Apache.

The <Proxymatch> line in our conf file shows:

<Proxymatch ^https?://127\.0\.0\.1:(2082|2083|2077|2078|2079|2080|2086|2087|2095|2096)/>

My understanding here is that Apache has a rule matching proxy requests destined for 127.0.0.1 on cPanel/WHM-related ports.

My cPanel/WHM server's /usr/local/cpanel/logs/access_log contains entries where external clients appear as proxy (shown above), and Apache has proxy_module, proxy_http_module, and proxy_wstunnel_module loaded. Is this normal for a standard cPanel installation?

This seems to be what's happening:

Scanner requests arbitrary URL
Apache receives it on :443
cPanel proxy rules match
Request gets sent toward a cPanel service
cpsrvd doesn't get the arguments it expects
"Missing required arguments!"
cpsrvd generates HTTP 500

500 Error ID 18cff0713d22b
Missing required arguments!

For an arbitrary external HTTPS request that produces both a 404 and a subsequent cPanel 500 with cpsrvd: Missing required arguments, what is the exact request-processing path?

Does Apache's Proxymatch rule forward the request to a local cPanel service, and if so, why does the request produce both 404 and 500 responses? And crucially, can an unauthenticated Internet client send a request to my public HTTPS endpoint that causes Apache/cPanel to proxy the request to any of the localhost ports listed in these Proxymatch directives?

The particularly important thing for us is to establish whether the external requests are being passed through cPanel's legitimate proxy mechanism and whether they are reaching WHM/cPanel itself or merely producing errors.

I'm keen to try and shut this down, it really is getting on my nerves now.

Is anyone else getting this too?

Thank you.

• 

  Marcin M

  • September 09, 2026 11:12

  You can use fail2ban to exclude all ip's reported in those logs for x period time i cut the traffic with this by 2/3 

  0
• 

  cPRex Jurassic Moderator

  • September 10, 2026 16:39

  ProsySubdomains would forward the request to the cPanel service, so that would be a good explanation for what you're seeing: https://docs.cpanel.net/knowledge-base/general-systems-administration/service-and-proxy-subdomains/

   

  0
• 

  Zoltan Egri | 1b.hu

  • October 09, 2026 15:54

  Hi,

  I agree with cPRex that Proxy Subdomains explains why these requests reach the cPanel service.

  However, I think it is worth distinguishing the request routing from the actual cause of the HTTP 500 responses.

  We investigated a potentially related issue on a production server (CPANEL-53195), where external requests for a non-existent resource triggered cPanel's internal 404 handling mechanism.

  Instead of a normal 404 response, we observed:

  safelock: Failed to create a lockfile '/var/cpanel/cache/404/webmaild.lock' Permission denied

  This resulted in HTTP 500 responses. cPanel Technical Support reproduced the behavior and escalated our findings to the development team.

  I'm not suggesting that this is necessarily the same issue, especially if your logs show different exceptions.

  It may be worth checking /usr/local/cpanel/logs/error_log for safelock failures, FileCreateError exceptions, or permission errors at the same timestamps as the affected requests in access_log.

  Fail2ban can certainly reduce unwanted traffic, but if a request exposes an internal cPanel error-handling problem, blocking IP addresses only mitigates the symptoms.

  We documented our investigation here:

  https://support.cpanel.net/hc/en-us/community/posts/43150129994007-CPANEL-53195-External-requests-trigger-HTTP-500-via-cPanel-404-safelock-failure

  I'd be interested to know whether you see any similar errors.

  Zoltan Egri | 1b.hu

  1
• 

  bloatedstoat

  • October 11, 2026 14:44

  Hello Zoltan,

  Yes identical behaviour. Here are some excerpts from our error_log, both cpaneld and webmaild including /unprotected/json-minified.js.map

  /usr/local/cpanel/logs/error_log:
      Cpanel::Server::_process_login_template_via_socket(Cpanel::Server=HASH(0x42ee9f0), "/usr/local/cpanel/var/cpsrvd_template_worker.sock", "en", ARRAY(0x21eecb8), "document", "./unprotected/json-minified.js.map", "page_to_show", "fourohfour", ...) called at /usr/local/cpanel/Cpanel/Server.pm line 1854
      Cpanel::Server::process_login_template(Cpanel::Server=HASH(0x42ee9f0), "cpanel", "SERVER_NAME", "cpanel.domain_name", "SERVER_PORT", 443, "document", "./unprotected/json-minified.js.map", ...) called at /usr/local/cpanel/Cpanel/Server.pm line 3141
      Cpanel::Server::process_and_send_login_template(Cpanel::Server=HASH(0x42ee9f0), "fourohfour", "HTTP/1.1 404 Not Found\x{d}\x{a}Content-type: text/html; charset=\"utf"..., HASH(0x4358ac8)) called at cpsrvd.pl line 3925
      cpanel::cpsrvd::_generate_and_cache_404("/var/cpanel/cache/404/cpaneld") called at cpsrvd.pl line 3877
      cpanel::cpsrvd::_send_cached_404_or_generate() called at cpsrvd.pl line 3836
      cpanel::cpsrvd::dofourohfour("document", "./unprotected/json-minified.js.map", "user", "cpanellogin") called at cpsrvd.pl line 5321
      cpanel::cpsrvd::handle_unprotected_docs() called at cpsrvd.pl line 1426
      cpanel::cpsrvd::handle_one_connection(3) called at cpsrvd.pl line 1214
      cpanel::cpsrvd::script() called at cpsrvd.pl line 465
  [2026-09-04 23:07:41 +1000] warn [cpaneld] safelock: Failed to create a lockfile '/var/cpanel/cache/404/cpaneld.lock-de94d15753ab-60fdfff7-6f204a' in the directory '/var/cpanel/cache/404' that isn't writable: Permission denied
  
  [2026-09-10 10:29:10 +1000] warn [webmaild] safelock: Failed to create a lockfile '/var/cpanel/cache/404/webmaild.lock-10776360b25b3-1e66220f-7309c6' in the directory '/var/cpanel/cache/404' that isn't writable: Permission denied
  [2026-09-10 10:29:10 +1000] warn [webmaild] safelock: Failed to create a lockfile '/var/cpanel/cache/404/webmaild.lock-2545f6435d159-1e66220f-7309c6' in the directory '/var/cpanel/cache/404' that isn't writable: Permission denied
  [2026-09-10 10:29:10 +1000] warn [webmaild] safelock: waited for lock (/var/cpanel/cache/404/webmaild.lock) 90 times
  cpsrvd 890b2852c728: Cpanel::Exception::IO::FileCreateError/(XID 3d33d3) The system failed to create the file “/var/cpanel/cache/404/webmaild.lock” (as [asis,EUID]: 985, [asis,EGID]: 979 979) because of the following error: Permission denied
   at /usr/local/cpanel/Cpanel/SafeFile.pm line 429, <GEN432> line 4.
      Cpanel::SafeFile::_safelock("/var/cpanel/cache/404/webmaild") called at /usr/local/cpanel/Cpanel/SafeFile.pm line 619
      Cpanel::SafeFile::_safe_open(undef, "<", "/var/cpanel/cache/404/webmaild", CODE(0x211d230), "safeopen") called at /usr/local/cpanel/Cpanel/SafeFile.pm line 175
      Cpanel::SafeFile::safeopen(undef, "<", "/var/cpanel/cache/404/webmaild") called at cpsrvd.pl line 3865
      cpanel::cpsrvd::_send_cached_404_or_generate() called at cpsrvd.pl line 3836
      cpanel::cpsrvd::dofourohfour("document", "./unprotected/json-minified.js.map", "user", "cpanellogin") called at cpsrvd.pl line 5321
      cpanel::cpsrvd::handle_unprotected_docs() called at cpsrvd.pl line 1426
      cpanel::cpsrvd::handle_one_connection(11) called at cpsrvd.pl line 1214
      cpanel::cpsrvd::script() called at cpsrvd.pl line 465
      Cpanel::Server::_process_login_template_via_socket(Cpanel::Server=HASH(0x3036f50), "/usr/local/cpanel/var/cpsrvd_template_worker.sock", "en", ARRAY(0x21eec18), "_is_trial", 0, "fourohfour", 1, ...) called at /usr/local/cpanel/Cpanel/Server.pm line 1854
      Cpanel::Server::process_login_template(Cpanel::Server=HASH(0x3036f50), "cpanel", "host", "cpanel.domain_name.com", "fourohfour", 1, "_is_trial", 0, ...) called at /usr/local/cpanel/Cpanel/Server.pm line 3141
      Cpanel::Server::process_and_send_login_template(Cpanel::Server=HASH(0x3036f50), "fourohfour", "HTTP/1.1 404 Not Found\x{d}\x{a}Content-type: text/html; charset=\"utf"..., HASH(0x30a0248)) called at cpsrvd.pl line 3925
      cpanel::cpsrvd::_generate_and_cache_404("/var/cpanel/cache/404/cpaneld") called at cpsrvd.pl line 3877
      cpanel::cpsrvd::_send_cached_404_or_generate() called at cpsrvd.pl line 3836
      cpanel::cpsrvd::dofourohfour("document", "./unprotected/json-minified.js.map", "user", "cpanellogin") called at cpsrvd.pl line 5321
      cpanel::cpsrvd::handle_unprotected_docs() called at cpsrvd.pl line 1426
      cpanel::cpsrvd::handle_one_connection(3) called at cpsrvd.pl line 1214
      cpanel::cpsrvd::script() called at cpsrvd.pl line 465
  [2026-09-10 10:29:10 +1000] warn [cpaneld] safelock: Failed to create a lockfile '/var/cpanel/cache/404/cpaneld.lock-268a0089c3d06-1e66220f-918e46' in the directory '/var/cpanel/cache/404' that isn't writable: Permission denied
  [2026-09-10 10:29:10 +1000] warn [cpaneld] safelock: Failed to create a lockfile '/var/cpanel/cache/404/cpaneld.lock-10e7383cadb63-1e66220f-918e46' in the directory '/var/cpanel/cache/404' that isn't writable: Permission denied
  [2026-09-10 10:29:10 +1000] warn [cpaneld] safelock: Failed to create a lockfile '/var/cpanel/cache/404/cpaneld.lock-12f5d35190cb7-1e66220f-918e46' in the directory '/var/cpanel/cache/404' that isn't writable: Permission denied
  
  
  [2026-09-04 23:08:31 +1000] warn [cpaneld] safelock: Failed to create a lockfile '/var/cpanel/cache/404/cpaneld.lock-70adaa135922-43c93177-527472' in the directory '/var/cpanel/cache/404' that isn't writable: Permission denied
  [2026-09-04 23:08:31 +1000] warn [cpaneld] safelock: Failed to create a lockfile '/var/cpanel/cache/404/cpaneld.lock-83861e1b56f8-43c93177-527472' in the directory '/var/cpanel/cache/404' that isn't writable: Permission denied
  [2026-09-04 23:08:31 +1000] warn [cpaneld] safelock: waited for lock (/var/cpanel/cache/404/cpaneld.lock) 90 times
  cpsrvd 3654c4589dc53: Cpanel::Exception::IO::FileCreateError/(XID vpqfnz) The system failed to create the file “/var/cpanel/cache/404/cpaneld.lock” (as [asis,EUID]: 985, [asis,EGID]: 979 979) because of the following error: Permission denied
   at /usr/local/cpanel/Cpanel/SafeFile.pm line 429, <GEN111> line 4.
      Cpanel::SafeFile::_safelock("/var/cpanel/cache/404/cpaneld") called at /usr/local/cpanel/Cpanel/SafeFile.pm line 619
      Cpanel::SafeFile::_safe_open(undef, ">", "/var/cpanel/cache/404/cpaneld", CODE(0x2122d30), "safeopen") called at /usr/local/cpanel/Cpanel/SafeFile.pm line 175
      Cpanel::SafeFile::safeopen(undef, ">", "/var/cpanel/cache/404/cpaneld") called at cpsrvd.pl line 3934
      cpanel::cpsrvd::_generate_and_cache_404("/var/cpanel/cache/404/cpaneld") called at cpsrvd.pl line 3877
      cpanel::cpsrvd::_send_cached_404_or_generate() called at cpsrvd.pl line 3836
      cpanel::cpsrvd::dofourohfour("document", "./unprotected/json-minified.js.map", "user", "cpanellogin") called at cpsrvd.pl line 5321
      cpanel::cpsrvd::handle_unprotected_docs() called at cpsrvd.pl line 1426
      cpanel::cpsrvd::handle_one_connection(3) called at cpsrvd.pl line 1214
      cpanel::cpsrvd::script() called at cpsrvd.pl line 465

  0
• 

  Zoltan Egri | 1b.hu

  • October 11, 2026 16:02

  Hi bloatedstoat,

  Thank you for confirming and sharing the logs.

  The cpaneld and webmaild errors appear to point to the same 404 cache locking issue, particularly the permission failures under /var/cpanel/cache/404.

  I checked another server running cPanel 110.0 (build 150). That server does not have the /var/cpanel/cache/404 directory, and its cPanel error log contains no matching permission-denied or file-creation errors. Therefore, I cannot reproduce the issue on that version.

  Could you check the ownership and permissions of /var/cpanel/cache and /var/cpanel/cache/404, and identify which system user corresponds to UID 985 on your server?

  It would also be helpful to know your exact cPanel version.

  This may help determine whether the failure is caused by incorrect filesystem permissions or by the privilege context used when generating the cached 404 response.

  Best regards,
  Zoltan Egri | 1b.hu

  0
• 

  bloatedstoat

  • October 15, 2026 00:19

  Sure no problem.

  /var/cpanel/cache (755)
  drwxr-xr-x   2 root root

  /var/cpanel/cache/404 (644)
  -rw-r--r-- 1 root root 5749 Oct 15 09:15 cpaneld
  -rw-r--r-- 1 root root 5750 Oct 15 10:50 webmaild
  -rw-r--r-- 1 root root 5752 Oct 15 10:44 whostmgrd

   

   

   

  0

Please sign in to leave a comment.