cPanel upgrade hanging on dcpumon

Hello, everyone! I've been experiencing an issue with one of our servers since May 17th: the process dcpumon keeps hanging indefinitely during cPanel update and eventually is killed. The alert e-mails I'm getting every day ever since contain the following text:

Update log preview:
...
...
[2018-06-10 03:51:59 -0300] E [/usr/local/cpanel/bin/dcpumon] The "/usr/local/cpanel/bin/dcpumon" command (process 74271) ended prematurely because it received the "KILL" (9) signal.
...
[2018-06-10 03:52:07 -0300] [/usr/local/cpanel/scripts/migrate_local_ini_to_php_ini] No local.ini.
[2018-06-10 03:52:07 -0300] [/usr/local/cpanel/scripts/migrate_local_ini_to_php_ini] " done!
[2018-06-10 03:52:07 -0300] - Finished command `/usr/local/cpanel/scripts/migrate_local_ini_to_php_ini --run --verbose` in 1.332 seconds
[2018-06-10 03:52:07 -0300] 92% complete
[2018-06-10 03:52:08 -0300] 93% complete
[2018-06-10 03:52:08 -0300] - Finished in 0.352 seconds
[2018-06-10 03:52:08 -0300] Processing: Ensuring an "Active" MySQL profile is set
[2018-06-10 03:52:08 -0300] - Processing command `/usr/local/cpanel/scripts/check_mysql`
[2018-06-10 03:52:08 -0300] [/usr/local/cpanel/scripts/check_mysql] "check_mysql" will complete in the background (process ID 75644).
[2018-06-10 03:52:08 -0300] - Finished command `/usr/local/cpanel/scripts/check_mysql` in 0.497 seconds
[2018-06-10 03:52:08 -0300] 94% complete
[2018-06-10 03:52:08 -0300] Processing: Checking CloudLinux installation
[2018-06-10 03:52:08 -0300] - Processing command `/usr/local/cpanel/bin/cloudlinux_update`
[2018-06-10 03:52:11 -0300] - Finished command `/usr/local/cpanel/bin/cloudlinux_update` in 3.004 seconds
[2018-06-10 03:52:11 -0300] Processing: Updating plugins data cache
[2018-06-10 03:52:11 -0300] - Processing command `/usr/local/cpanel/bin/refresh_plugin_cache`
[2018-06-10 03:52:12 -0300] - Finished command `/usr/local/cpanel/bin/refresh_plugin_cache` in 0.640 seconds
[2018-06-10 03:52:12 -0300] 95% complete
=> Log closed Sun Jun 10 03:52:12 2018
----------------------------------------------------------------------------------------------------
=> Log opened from cPanel Update (upcp) - Slave (72346) at Sun Jun 10 03:52:12 2018

Here is what I see in the upgrade log:


[2018-06-10 01:53:57 -0300]    - Finished in 0.002 seconds
[2018-06-10 01:53:57 -0300]    - Finished in 0.000 seconds
[2018-06-10 01:53:57 -0300]    - Finished in 0.000 seconds
[2018-06-10 01:53:57 -0300]    - Finished in 0.000 seconds
[2018-06-10 01:53:57 -0300]    - Finished in 0.782 seconds
[2018-06-10 01:53:58 -0300]    - Finished in 0.003 seconds
[2018-06-10 01:53:58 -0300]   Processing: Cleaning SpamAssassin DBM files
[2018-06-10 01:53:58 -0300]    - Processing command `/usr/local/cpanel/scripts/spamassassin_dbm_cleaner`
[2018-06-10 01:53:58 -0300]      [/usr/local/cpanel/scripts/spamassassin_dbm_cleaner] /usr/local/cpanel/scripts/spamassassin_dbm_cleaner: version 1.0
[2018-06-10 01:53:59 -0300]      [/usr/local/cpanel/scripts/spamassassin_dbm_cleaner] Checking SpamAssassin dbm databases....Done
[2018-06-10 01:53:59 -0300]      [/usr/local/cpanel/scripts/spamassassin_dbm_cleaner] Checked 16 files for 14 user(s), 16 ok, 0 broken
[2018-06-10 01:53:59 -0300]    - Finished command `/usr/local/cpanel/scripts/spamassassin_dbm_cleaner` in 1.093 seconds
[2018-06-10 01:53:59 -0300]    - Finished in 0.542 seconds
=> Log closed Sun Jun 10 01:53:59 2018
[2018-06-10 01:53:59 -0300]    - Finished in 0.192 seconds
[2018-06-10 01:53:59 -0300]   Processing: Checking for new security advice
[2018-06-10 01:53:59 -0300]    - Processing command `/usr/local/cpanel/scripts/check_security_advice_changes --notify`
[2018-06-10 01:54:36 -0300]      [/usr/local/cpanel/scripts/check_security_advice_changes] There are no changes to the Security Advisor state that require notification.
[2018-06-10 01:54:36 -0300]    - Finished command `/usr/local/cpanel/scripts/check_security_advice_changes --notify` in 37.078 seconds
[2018-06-10 01:54:36 -0300]   Processing: Running former postinstall scripts
[2018-06-10 01:54:36 -0300]    - Processing command `/usr/local/cpanel/bin/dcpumon --killproc`
[2018-06-10 01:54:37 -0300]      [/usr/local/cpanel/bin/dcpumon] Loading Symbol Table... /usr/local/cpanel/etc/sym/BitchX.sym  ..Done
[2018-06-10 01:54:37 -0300]      [/usr/local/cpanel/bin/dcpumon] Loading Symbol Table... /usr/local/cpanel/etc/sym/bnc.sym  ..Done
[2018-06-10 01:54:37 -0300]      [/usr/local/cpanel/bin/dcpumon] Loading Symbol Table... /usr/local/cpanel/etc/sym/eggdrop.sym  ..Done
[2018-06-10 01:54:37 -0300]      [/usr/local/cpanel/bin/dcpumon] Loading Symbol Table... /usr/local/cpanel/etc/sym/generic-sniffers.sym  ..Done
[2018-06-10 01:54:37 -0300]      [/usr/local/cpanel/bin/dcpumon] Loading Symbol Table... /usr/local/cpanel/etc/sym/guardservices.sym  ..Done
[2018-06-10 01:54:37 -0300]      [/usr/local/cpanel/bin/dcpumon] Loading Symbol Table... /usr/local/cpanel/etc/sym/ircd.sym  ..Done
[2018-06-10 01:54:37 -0300]      [/usr/local/cpanel/bin/dcpumon] Loading Symbol Table... /usr/local/cpanel/etc/sym/psyBNC.sym  ..Done
[2018-06-10 01:54:37 -0300]      [/usr/local/cpanel/bin/dcpumon] Loading Symbol Table... /usr/local/cpanel/etc/sym/ptlink.sym  ..Done
[2018-06-10 01:54:37 -0300]      [/usr/local/cpanel/bin/dcpumon] Loading Symbol Table... /usr/local/cpanel/etc/sym/services.sym  ..Done
[2018-06-10 01:54:37 -0300]      [/usr/local/cpanel/bin/dcpumon] Kill Proc Enabled
[2018-06-10 01:54:37 -0300]      [/usr/local/cpanel/bin/dcpumon] Added Contact cpanel@[our_domain]

It won't get any further from there. When attempting to run the command directly, dcpumon doesn't get stuck or fails at all:


[root@hostname ~]$ /usr/local/cpanel/bin/dcpumon
Added Contact cpanel@[our_domain]
[root@hostname ~]$ echo $?
0
[root@hostname ~]$

I tried temporarily disabling the server's firewall and ran the upgrade again, both forced and otherwise, to no avail. The box runs on CloudLinux 6 and cPanel version is currently v70.0.48 (which comes to show that the upgrade is indeed working despite the aforementioned error). Nothing relevant comes up on Google. Is anyone else experiencing something similar? Could it be some sort of cosmetic error that can be ignored? Thanks!

cPanelLauren

June 11, 2018 15:29

Hi @Bretas I haven't seen this specific issue come across in any recent cases or tickets. If you run /scripts/dcpumon-wrapper
does it also complete successfully? For some reason when the update completes it' looks like it's not getting the output it's expecting untill the process is killed. It does complete successfully though.

Bretas

June 12, 2018 02:40

Thank you for your response, Lauren! Yes, that command finishes successfully as well. It behaves much the same way as "/usr/local/cpanel/bin/dcpumon":


[root@hostname ~]$ /scripts/dcpumon-wrapper
Added Contact cpanel@[our_domain]
[root@hostname ~]$ echo $?
0
[root@hostname ~]$

Thanks!

June 12, 2018 08:06

PS: This forum is not allowing me to edit the post above to include the following text ("Your content can not be submitted. This is likely because your content is spam-like"...). So, sorry for posting twice in a row: I guess I didn't include in my original post the first line (the red one) of the error message we are being emailed. It mentions that the exit code is 256, which is why I keep checking the exit from these commands, I'm trying to reproduce the error cPanel is getting during the update so we can narrow down the root cause. Here it is:


     Maintenance ended; however, it did not exit cleanly (256). The following events were logged: "bin/dcpumon". Review the update logs to determine why the update failed.

June 12, 2018 12:43

Hi @Bretas Thank you for getting back to me with that output. Both commands essentially do the same thing which is why I was curious. At this point, I'd like to see if you could please open a ticket using the link in my signature. Once open please post the ticket ID here so I can update this thread with the outcome. Thanks!

June 13, 2018 04:28

Sorry I'm late! OK, I just opened the ticket #9727897. Much appreciated!

June 13, 2018 12:50

Hi @Bretas No worries! I'm currently watching the progress of the ticket and I'll update this thread with the outcome. Thanks!

jsilvestre

June 14, 2018 21:50

Hello, It's a know issue and have an internal case CPANEL-21205 (since today). As workaround, please disable Background Process Killer. Best Regards,

June 15, 2018 12:03

That's correct, actually as a result of @Bretas ticket we found that the background process killer was consuming a large amount of memory resulting in the process hanging. We have only had 2 reports thus far of this occurring but are hoping to add some more examples to the case. If you are experiencing this issue and have not opened a ticket as of yet please reply to this thread so I can note it in the internal case. If you're unsure you're experiencing this issue and would like for us to take a closer look, please use the link in my signature to open a ticket with us and let us know the ticket ID here. Thanks!

morrow95

June 18, 2018 00:22

I contacted support yesterday for this same issue and support told me there was an open case already for it. CPANEL-21205 If it helps at all my support ticket is/was number 9747109.

June 18, 2018 12:48

Hi @morrow95 Thanks, I just double checked the case and found that your ticket was added to it as an example. Thank you for letting us know! I also see that the case has been updated to indicate that they are testing a fix for it currently. I'll update here again when the case is marked as completed. Thanks!

cPanel upgrade hanging on dcpumon

Comments

Didn't find what you were looking for?