Skip to main content

cPHulkd Failing Repeatedly

paulapatrice
Set up a VPS and updated it to easyApache4 by importing a profile from a previously-configured, smooth-running server. Also imported the Tweak Settings config. New VPS ran cPanel & WHM ver 64, build 33. cPanel updates are set to manual. After running the VPS without errors for approx 2 weeks I started getting 1-2 email notifications per day alerting that cPhulkd had failed, then recovered. That increased to 3-4 failures per day. A similar issue is mentioned here but it applies to an earlier version of cPanel. My other VPS is running CP 64 build 24 without any errors, so the fix to that issue seems to have worked. There were two errors in the logs that appeared to occur at the same time when cPhulkd failed: warn [cphulkd] 25313 Failed to fork(): Cannot allocate memory send: Connection refused at /usr/local/cpanel/Cpanel/Linux/Netlink.pm line 125, <$client_socket> line 6. ...caught at /usr/local/cpanel/Cpanel/Ident.pm line 49, <$client_socket> line 6. Since that seemed to indicate a memory issue I thought it could be due to malicious activity attempts being blocked, so in WHM I tried disabling cPhulk's option to check usernames, and only left the IP option enabled. cPhulkd notifications stopped for ~ 20 hours, then started again at double the failure rate. I tried changing the following settings to see if they'd help manage or free up memory: - Service Manager > enabled / monitor PHP-FPM service for cPanel Daemons - Tweak Settings > System > Changed "Max cPanel process memory" from "768" to "1024" > Changed "ChkServd TCP check failure threshold" from "3" to "5" - Tweak Settings > Changed "Log rotation size threshold" from "300" to "200" None of that helped, so I updated to the latest stable version of cPanel & WHM, v64 build 36. That immediately made the failures worse. It's now failing every hour, sometimes twice in an hour. My inbox is full of alerts from the past 36 hours. I'm fairly new to VPS management and not adept with SSH command line, though I can use the terminal when needed and directed. Any help in troubleshooting would be appreciated. I'm out of ideas. Attached snapshots: cPhulkd failure notice, recovered notice, top processes

Comments

4 comments

Date Votes
  • cPanelMichael
    Hello, Could you open a support ticket using the link in my signature so we can take a closer look at the affected system? Thank you.
    0
  • paulapatrice
    Apparently I'm unable to grant access via the support wizard nor customer portal: Error:WHM Authorization failed with the following error: API failure: (XID f5pf7u) The temporary wheel user "get" operation failed with a "sudo" error: The system cannot create a temporary user because this server does not permit wheel users to use "sudo".. You may skip this server or correct the problem and try again. I submitted the ticket with Priority anyway, ID 8779571
    0
  • cPanelMichael
    Hello, To update, it looks like this issue stemmed from a lack of available memory on the system. Thank you.
    0
  • paulapatrice
    FOLLOW UP: SOLVED The error messages were: The service "cphulkd" appears to be down. The system"s command to check or to restart this service failed. The subprocess "/usr/local/cpanel/scripts/restartsrv_cphulkd" reported error number 9 when it ended. The solution: Appears to have been caused by a failure of the host server to allocate memory to the VPS, perhaps due to an overloaded host node (as per cPanel support tech's troubleshooting). As a result cPanel services were failing when coming out of dormant state to run. Issue seems to have fixed itself, presumably by the host fixing its server. Since the VPS dashboard nor WHM showed this memory issue and the server appeared to have 95% free RAM, I am posting the cPanel Support Analyst's method of troubleshooting below for others' reference if they have the same problem. Tech Support Notes: I noticed you were on an OpenVZ server. Open VZ servers have a file at /proc/user_beancounters that keeps track of when a container hits faults. I noticed your server has been hitting memory related faults : [01:34:09 [servername] root@#### ~]cPs# grep -v '\ 0$' /proc/user_beancounters Version: 2.5 uid resource held maxheld barrier limit failcnt privvmpages 270575 277331 1572864 1572864 1840 Notice the failcnt column. This means your server is failing to allocate memory 1840 times. This isn't a reflection of the RAM on your server being allocated but rather a failure to allocate ram from the host server. This leads me to believe that your host node you are on is overloaded and that is why you are getting failure to allocate memory errors. Linux is trying to get memory but there isn't any available on the host node to get which can cause issues like this. In regards to how to check the memory usage on a Linux server you can use the free command to do so . (free(1): amount of free/used memory in system - Linux man page )
    0

Please sign in to leave a comment.

Didn't find what you were looking for?

New post