[CPANEL-33077] Letsencrypt transition to ISRGs Root (Important!!!!!)
-
We also update ca-certificate to latest version, ca-certificates-2021.2.50-72.el7_9.noarch. AutoSSL showing this log for any domains: Certificate expiry: 12/29/21, 4:26 PM UTC (89.96 days from now) ERROR Defect: OPENSSL_VERIFY: The certificate chain failed OpenSSL"s verification (0:10:CERT_HAS_EXPIRED). ERROR Defect: OPENSSL_VERIFY: The certificate chain failed OpenSSL"s verification (1:10:CERT_HAS_EXPIRED). ERROR Defect: OPENSSL_VERIFY: The certificate chain failed OpenSSL"s verification (2:10:CERT_HAS_EXPIRED). ERROR Defect: OPENSSL_VERIFY: The certificate chain failed OpenSSL"s verification (3:10:CERT_HAS_EXPIRED). 0 -
Wonder if "rpm -e --nodeps ca-certificates-2021.2.50-72.el7_9.noarch" and reinstalling would help, but it has so many dependencies that I would rather wait for an official solution :-) 0 -
On CloudLinux 6 ELS I was given a link (by CloudLinux) for updated OpenSSL packages. I'm not sure that I'm privy to share the link because I don't know if I'm a "guinea pig" I can say that after installing the update and restarting all services ( those showing need for restarting via /usr/bin/needs-restarting), at least some things work now 1. AutoSSL seems to run and proper process new certs (except that Lets Encrypt throttled my attempt to "Run AutoSSL for all users") so not all were updated. 2. www.customerdomain.com/webmail now work without cert warnings 3. some previously expired website certs are now working BUT, I have not been able to confirm that Exim/Dovecot are working (when people connect to mail.theirdomain.com over SSL) Mike 0 -
On CloudLinux 6 ELS I was given a link (by CloudLinux) for updated OpenSSL packages. I'm not sure that I'm privy to share the link because I don't know if I'm a "guinea pig" I can say that after installing the update and restarting all services ( those showing need for restarting via /usr/bin/needs-restarting), at least some things work now 1. AutoSSL seems to run and proper process new certs (except that Lets Encrypt throttled my attempt to "Run AutoSSL for all users") so not all were updated. 2. www.customerdomain.com/webmail now work without cert warnings 3. some previously expired website certs are now working BUT, I have not been able to confirm that Exim/Dovecot are working (when people connect to mail.theirdomain.com over SSL) Mike
I can confirm that Dovecot / Exim are not using the certificates, even though they are valid . Im not sure how to force those services to start using the certs again. Because of that, I can't confirm that EVERYTHING is fixed. Anyone know how to force the email service (dovecot / exim) to use the customer-domain certs that are valid? Mike0 -
Just tried switching to cPanel powered by Sectigo and receive this error when running AutoSSL: The "cPanel (powered by Sectigo)" provider cannot currently accept incoming requests. The system will try again later.
I'm guessing everyone else is trying the same switch and cPanel can't handle it?0 -
Assuming all SSL certs are renewed / valid : It would be great if somebody at cPanel would tell us how to rebuild all of the sni information so that Dovecot, Exim, and FTP make use of them. 0 -
This is what happen when cPanel try to focus on make money with price increases and ugly themes instead of important things, this thread was created in jun 2020 and cPanel just let it pass, now that this exploded they are over time trying to fix it or figure how to workaround this.. meanwhile thousands of users and servers are facing this and dealing with this and waiting for a cPanel fix or telling clients that they need to reconfigure their mails clients, thousands of clients dont even know how to put a mail account on their devices and they depend of support teams or tech guy... imagine that suddenly nobody in your company can access to their mail account and you have employees in other countries that depend of mails... how can you deal with 100 or 1000 devices to reconfigure them in a insecure way temporally because cpanel is not working and is playing with drawings, sand and useless updates since the last 2 years? now multiply this by all companies affected by this and in pandemic with home office... but cpanel still playing with themes and wants to raise prices for that 2 hours of work of ugly themes and bugs 0 -
This is what happen when cPanel try to focus on make money with price increases and ugly themes instead of important things, this thread was created in jun 2020 and cPanel just let it pass, now that this exploded they are over time trying to fix it or figure how to workaround this.. meanwhile thousands of users and servers are facing this and dealing with this and waiting for a cPanel fix or telling clients that they need to reconfigure their mails clients, thousands of clients dont even know how to put a mail account on their devices and they depend of support teams or tech guy... imagine that suddenly nobody in your company can access to their mail account and you have employees in other countries that depend of mails... how can you deal with 100 devices to reconfigure them? now multiply this by all companies affected by this and in pandemic with home office... but cpanel wants to raise prices and that is more important for them...
I agree with you, that's total disregard for customers and a lack of planning.0 -
Any solution? I got 23 calls on my phone in 30 minutes from costumers... 0 -
I don't see a reference in this thread to https://twitter.com/search?q=%23letsencrypt As far as I can see this only affects Dovecot, not Exim - we're getting loads of tickets emailed by affected customers ... but know they won't see our email replies! 0 -
This is what happen when cPanel try to focus on make money....
Fully agree. It's possible to justify the price rises if the product 'Just works' but events like this show a huge hole in the way cPanel is being run. In the 4 hours since we lodged a priority ticket when this kicked off we've only received one reply from support. And that was over 2 hours ago asking us to check the rpm version of the CA certificates. Communication is important. We need to know what's going on so we can relay it on to at least attempt to reassure our frustrated clients. Surely someone at cPanel can send out some ticket replies, keep us in the loop etc. Even if it's just a 'We're looking into this'. Having to come to the forum to work out what's happening doesn't exactly match the premium price tag.0 -
To be fair - I went to raise a ticket and saw the annoucement at the top of the page ... which led me here to discover more. 0 -
I don't see a reference in this thread to As far as I can see this only affects Dovecot, not Exim - we're getting loads of tickets emailed by affected customers ... but know they won't see our email replies!
For us at least switching to cPanel Sectigo is not working as their server is not issuing SSL certificates:The "cPanel (powered by Sectigo)" provider cannot currently accept incoming requests. The system will try again later.
Anyone else seeing this?0 -
I don't see a reference in this thread to As far as I can see this only affects Dovecot, not Exim - we're getting loads of tickets emailed by affected customers ... but know they won't see our email replies!
Interesting writeup by cPanel. I haven't used cPanel for SSLs (other than primary hostname). I've always used Letsencrypt. If we were to feel we absolutely had to try the cPanel option, is that free? Is there a charge? Do they rate limit? So many issues combined. You've got the underlying issue that caused the whole mess, whatever that is. And then you have the issue of LetsEncrypt throttling mass cert renewals (such as when runs WHM --> AutoSSL --> Run AutoSSL for All Users. Ok on Dovecot issues but not Exim. I haven't verified that myself. I know that Dovecot is definitely an issue. But I think Dovecot's issue was outlined by you or someone else earlier in this thread or another one -- the place where all the information is held for the dovecot SNI information isn't getting populated with the information from the new certs that were in fact renewed without error. Mike0 -
You can't switch to cPanel for SSL :rolleyes: The "cPanel (powered by Sectigo)" provider cannot currently accept incoming requests. The system will try again later. 0 -
As several of you have seen, the cPanel Sectigo provider is currently overwhelmed due to this issue. However, these errors will be intermittent as it is based on capcity. Switching to the cPanel SSL provider is free, and you can see the limits here: We are currently sending the following message to users that are submitting tickets to our team as it is the most effective workaround at this time: Thank you for your patience. We are currently investigating this issue and are tracking it internally as UPS-403. We will be publishing more information here: https://support.cpanel.net/hc/en-us/articles/4409770365335 This is related to the recent expiration of the DST Root CA X3 Cert from Let's Encrypt. We believe this to be causing issues with the SNI configuration. We are currently working with our developers on a more permanent solution that would correct the certificates already installed on the server. Once this is complete the page above will be updated. However, if absolutely required you can bypass these errors by switching to using the cPanel Store as the AutoSSL certificate provider and issuing new certificates. Running this command below will set cPanel as the AutoSSL provided and then run a check for all of the domains on the server: whmapi1 set_autossl_provider provider='cPanel' ; /usr/local/cpanel/bin/autossl_check -all If you have any questions, or if there is anything else we can assist you with, please let us know. We would be glad to help!0 -
@cPRex I followed your instructions for switching to cPanel/Sectigo and it went through everything successfully (or appears to have). The issue is, Dovecot is not making use of any of those certs like it should be. So POP3/IMAP-over-SSL using the customer's mail.customerdomain.com is not working. Mike 0 -
It's also important to note that the overall issue is only effecting Let's Encrypt, which isn't something that is provided by cPanel, and has a separate terms of service as outlined here:
Yes... the root problem is Let's Encrypt, but this is nothing new as at least you (cPanel) where awarded about this change since more than one year, you had more than one year to investigate, update and push a update in time, not today that was the end day, instead of that you where focused on making ugly themes and increasing prices for that themes, in this thread there are so many questions, post and updates about this since past year and you ignore them... so the problem is caused by you and your lack of focusing in important things, if you where able to pick this in time nobody using cpanel would have this problem, why im not having this problem with my servers that dont use cpanel? thats because we prepared our servers for this in advance, so dont wash your hands saying that is Let's Encrypt problem, face up and accept that it is a problem for not doing your work and anticipating this at the time, you had more than a year to do what you are doing now0 -
@mtindor - I would expect that a certificate that was reinstalled would work normally with all services, as that would trigger the service to rebuild and restart. Just as a test, could you try manually restarting Dovecot to see if that gets things working? If not, feel free to submit a ticket to our team so we can take a look. 0 -
I've done some further testing ... Exim is affected as well as Dovecot ... sorry for my mis-information previously. @cPRex is a patch likely in the next few hours or do we need to switch to Sectigo and loose the wildcards some clients rely on? 0 -
We are working on a more permanent patch right now, and it is in review. It will fix existing certificates, but we're actually seeing that Let's Encrypt still issuing new certificates that are having issues. So even once our fix is applied, we can't guarantee everything will work properly as some of it is still out of our control. Our best recommendation at this time would be to switch to cPanel/Sectigo if it is absolutely critical, or wait for our patch to be released soon. I expect "soon" to mean "some point this evening" although the situation is still developing and it's hard for me to provide an accurate timeframe. 0 -
@mtindor - I would expect that a certificate that was reinstalled would work normally with all services, as that would trigger the service to rebuild and restart. Just as a test, could you try manually restarting Dovecot to see if that gets things working? If not, feel free to submit a ticket to our team so we can take a look.
Before, when I was using Lets Encrypt. It did go through and renew the expired SSLs. But none of them deposited appropriate information in /var/cpanel/ssl/domain_tls I switched to cPanel/Sectigo. It appeared to only attempt to renew certs that weren't already valid on the system. So all the other ones that were renewed by Lets Encrypt that didn't cause information to be deposited into /var/cpanel/ssl/domain_tls stayed hte same. I'm on cPanel/Sectigo now. But the only way I could figure out to get a new Sectigo SSL provisioned and resultant info put in /var/cpanel/ssl/domain_tls was to log into the user's cPanel account, UNInstall the SSL certificate for the site. Go to WHM --> AutoSSL --> and run AutoSSL for that user. The new cert would eventually be provisioned and information put in /var/cpanel/ssl/domain_tls. And I'm presuming that for those domains that I diid this, email service is probably restored. Yes, I just did this and verified that email service for that domain is restored. So must be go into every cPanel account and (a) manually UNinstall every SSL certificate, (b) go and renew those certs in AutoSSL one at time for everything to work? Mike0 -
That's part of the reason we're encouraging people to wait for an official fix - we're getting some reports of AutoSSL not properly replacing certificates, maybe due to ratelimiting, or maybe due to other issues. 0 -
That's part of the reason we're encouraging people to wait for an official fix - we're getting some reports of AutoSSL not properly replacing certificates, maybe due to ratelimiting, or maybe due to other issues.
So if the automated process (WHM -- AutoSSL -- Run AutoSSL for all users) doesn't complete because of throttling by LetsEncrypt (or even Sectigo), then I'm guessing technically the "followthrough" doesn't occur where it sets all of the certificates up for email usage? Hmm. And if there are truly legitimate SSL certs on the server (supposedly I have a ton for a ton sites that do not show any issues in a web browser) for which there is no data in /var/cpanell/ssl/domain_tls . And apparently that's why there is no SSL for mail.thatdomain.com for email use. Ok I guess I shall wait. In the meantime, for thoroughly pissed off customers, I'm UNinstalling their SSL via their cPanel interface and then having AutoSSL reprovision it (which seems to do the trick). Mike0 -
Ok I guess I shall wait. In the meantime, for thoroughly pissed off customers, I'm UNinstalling their SSL via their cPanel interface and then having AutoSSL reprovision it (which seems to do the trick).
That "may" backfire on you also, as you may run into domain or IP level rate limiting.0 -
That "may" backfire on you also, as you may run into domain or IP level rate limiting.
And it does / did backfire. A full run on all users definitely gets rate-limited with LetsEncrypt, didn't get throttled with Sectigo (but only had a couple invalid SSL to be reprovisioned). For single-user attempts, you are right -- Lets Encrypt is rate-limiting even those. I'm sure it is based upon traffic from the server IP. For Sectigo it's hit-or-miss. Works 1 or 2 times out of every three that I've tried so far. For the ones it didn't work for, it claims the server will re-attempt the provisioning at a later time. Still, with this method, I was able to get 5 or 6 customers (the most vocal, who called in) taken care of. I'm going to try to take care of the remaining squeaky wheels in this fashion. Then wait for a 'fix' for the rest. Of course, as long as Lets Encrypt and Sectigo are swamped / rate limiting, even when a "fix" comes out to correct all this, Lets Encrypt and Sectigo will be swamped and rate-limiting the hell out of us and that will end up being the hopefully final (and painful) issue to deal with.0 -
Was there something cPanel could have done before to avoid this? Not sure how other control panels, even Plesk, managed to sort this changes with Let's Encrypt. 0 -
How about just run yum update -y ca-certs ? What do you think? 0 -
If it fails to renew, I don't understand why it's not being added to the "Pending Queue". It appears that some domains just flat out fail and are never tried again. Clearly the Pending Queue exists, but doesn't function properly. So glad to have the new f***ing themes though. Glad that's where our ever-increasing money is going. 0 -
Ok I guess I shall wait. In the meantime, for thoroughly pissed off customers, I'm UNinstalling their SSL via their cPanel interface and then having AutoSSL reprovision it (which seems to do the trick).
Are you having them reprovisioned with LE or Sectigo in autoSSL?0
Please sign in to leave a comment.
Comments
203 comments