[CPANEL-24832] Multiple service failure reports for cpsrvd, IMAP, and Exim
So this used to be a very occasional problem I thought could be attributed to memory issues, but while I've solved the memory problem and it appeared stable for a stretch, we're back to this problem again, but now more often. At least daily, sometimes once every couple days, I'll see alerts that multiple services have failed. CPSRVD is a usual suspect, as is the Bind nameserver. clamd, IMAP, and Exim sometimes follow shortly thereafter. The weird part is, all of the services are still running... it's just that occasionally, they like to throw mostly unhelpful errors.
What I believe is happening is something all of the services that fail commonly use, possibly for authentication, is failing and it's only after CPanel goes through the usual restart routines and stumbles on the thing that's failing that the issue actually self-corrects, until the next time. But until CPanel hits on the service that's failing, I'll see things like this.
dovecot: auth: Error: auth worker: Aborted PASSV request for some@email.address: Shutting down
dovecot: auth: Error: net_connect_unix(anvil-auth-penalty) failed: Permission denied
Or, from another affected service:
2019-02-19 21:38:07.244 [31472] 1gwHlt-0008BS-4Z == some@email.address R=virtual_user T=dovecot_virtual_delivery_no_batch defer (-44): LMTP error after RCPT TO:: 451 4.3.0 Temporary internal error
On the occasion I try to access a site on this server when these issues are on-going, I'll get a 503 error instead of what I'm trying to access. As said, this is all very temporary--usually resolved in about 5 minutes from when CPanel notices, but I'd be curious to know what causes it and if there's any way it can be avoided. The only thing my own digging has been able to uncover is that all of the affected services seem to rely on one common mechanism, and it's that mechanism rather than the individual services that has fallen sideways.
-
Same issue here, the workaround is to disable cpsrvd in the Dormant services doesn't work 0 -
We believe we have tracked the problem down to a race condition with systemd. The problem was less likely to occur on an older version of cPanel because cpsrvd startup was not as fast as it is on newer versions. If our initial analysis is correct the issue will affect any systemd service that has Type=forking and KillMode=process. This means it can also cause seemingly random restart failure in cpgreylistd, cphulkd, dnsadmin, queueprocd, tailwatchd, etc. Can everyone who has reported the issue also confirm that it is unique to CentOS 7/CloudLinux 7? Additionally, the problem may be solved in systemd version 232 or newer. We are working on a workaround for the systemd version (219) that ships with CentOS 7/CloudLinux 7. 0 -
Yes to CentOS 7/CloudLinux 7 just occurred again for me. I've been able to stabilise things bit by restarting cpsrvd every morning at 5am. I now get this problem once a month. 0 -
Interesting, I have this issue on one virtual server (Virtuozzo) with CentOS7 too. Always thought is was a issue with Virtuzzo or the node setup. Made a parallel fresh installation on the same node and had no issues there. Same host, same CentOS7 template, but newer cPanel-Installation. Now i stumble upon this bugreport. 0 -
Same issue here, the workaround is to disable cpsrvd in the Dormant services doesn't work
Did you try to disable all service in Dormants services? How often do you see issues? I can see no pattern. Somestimes the issue is a few times a day. Then no issues for days.0 -
I have the same problem on my Centos 7.6 VPS (systemd 219) and I have found numerous posts about it being buggy for more than cPanel. My issue is happening (almost) daily late at night (ranging from 11pm to 6am) for about 20 minutes which wasn't a huge deal for me or my clients, so I just waited for some cPanel update to fix it. Apparently after 5 months this is still an issue and I just stumbled upon this thread now. I am trying the Dormant services workaround for the moment but have also found this permament fix by facebook sysadmins for the more dauntless: - Removed - I can't really try this fix on a production server with dozens of clients so if anyone goes through with it please post an update. P.S. I cannot post a ticket because the domain of my server has changed and my cpanel account doesn't work any more. 0 -
Hello @rallisf1, This is fixed in cPanel & WHM version 80: Fixed case CPANEL-24832: Workaround systemd race condition which can cause it to kill cpsrvd. Version 80 is currently only published to the EDGE release tier as a development version, however there is an active request to backport the case to cPanel & WHM version 78. I'll continue to monitor the case and update this thread with more information as it becomes available. Thank you. 0 -
Hello @The Old Man, Can you see if disabling cpsrvd in the Dormant services section under the Software tab in WHM >> Tweak Settings addresses the issue? This should solve the problem until case CPANEL-24832 is published. Thank you.
Thank you for this! Our server is running the latest version of CPanel/WHM and this problem has been driving me crazy for the last several months. Issue is: Failed: cpsrvd "ip address" The Service cpsrvd appears to be down I can confirm it has done this at least once a day (sending emails and texts) for the last 4 months. Mine does have a time pattern and it always happens around 12 midnight. So I did what your post asks by disabling cpsrvd in the dormant services section and it has finally stopped. I honestly had to look at my phone twice to see if this was for real. haha. Thanks!0 -
Hello Everyone, I've marked this thread as SOLVED with cPanel & WHM version 80 now in the RELEASE build tier. Let us know if any additional issues are noticed after upgrading to version 80. Thank you. 0 -
Hello, For anyone using cPanel & WHM version 78 (LTS), CPANEL-24832 was published to 78.0.32 today: 78 Change Log - Change Logs - cPanel Documentation Thank you. 0 -
I can confirm this is not fixed in v80.0.20 0 -
I can confirm this is not fixed in v80.0.20
Hello @janipewter, Can you provide a full description of the symptoms you notice on the server running cPanel & WHM version 80.0.20? Thank you.0 -
Exactly the same behaviour as before. IMAP, Exim and Cpsrvd all fail and then recover a short while later. It seems to be completely random whether it happens or not, but when it does, it's usually between midnight and 0200. It never happens more than once per day. 0 -
Exactly the same behaviour as before. IMAP, Exim and Cpsrvd all fail and then recover a short while later. It seems to be completely random whether it happens or not, but when it does, it's usually between midnight and 0200. It never happens more than once per day.
Hello @janipewter, We'll need to take a closer look at your system to see why this is happening. Could you submit a support ticket and post the ticket number here once it's opened? I'll follow the ticket and report the outcome here once it's closed. Thank you.0 -
Support ticket created. 12902305 Thanks 0 -
Support ticket created. 12902305
To update, it looks like the service failure reported in the ticket was a one-time occurrence and not a new occurrence of the issue reported on this thread. Thank you.0 -
Been 5 years, but am having this issue with a current cPanel / WHM version on Alma Linux OS. Disabled the dormant service in Tweak Settings just now as a temporary resolution, but would like to know if anyone solved this one? Thanks.
0 -
Dan Root - it would seem unlikely that this was the same problem as this original report, so it might be best to open a new thread with more specific details about your situation.
0 -
cPRex - If you see a new thread for this can you please link it here also so we will see it?
I still have a (minor) issue with p0f going down and then being restarted by cPanel. Typically it's only on the DNS servers, and is right after/during upcp running. It may or may not be related...
Also, I have noticed with the new forum that all links to the old form are broken. This has been a constant irritation. Any chance of a fix? I bet not, and that's sad and poor planning. Ya know?
-Pete
0 -
PeteS - any links that go to something newer than 10 years old should still work. Anything before that time is no longer accessible.
I can't say I'm familiar with that happening, but if it's something that is recurring or if you have a way to reproduce you can always make a ticket with our team.
0 -
cPRex - Fair enough o the 10 yr cutoff. :) I have no idea of the age of a link, but it's possible that they have all been that stale...
On the p0f service issue, I haven't tried manually running upcp to see if it's repeatable, and it's intermittent. It's not been a cause of any trouble so I've back-burnered it hope for a "magical fix" on a future update. ;) I am saving the errors though for an eventual eval and probably a ticket.
-Pete
1
Please sign in to leave a comment.
Comments
51 comments