Skip to main content

[CPANEL-33077] Letsencrypt transition to ISRG’s Root (Important!!!!!)

ciao70
Hi, In reference to this notice

Comments

198 comments

Date Votes
  • DHarry
    I've ran the autofixer and I think I'm confused as to what it's supposed to "fix". Our issue is that our certificate chain looks like this: i:/C=US/O=Let's Encrypt/CN=R3 1 s:/C=US/O=Let's Encrypt/CN=R3 i:/C=US/O=Internet Security Research Group/CN=ISRG Root X1 2 s:/C=US/O=Internet Security Research Group/CN=ISRG Root X1 i:/O=Digital Signature Trust Co./CN=DST Root CA X3
    We're also getting the verification error, "unable to get local issuer certificate". Was the autofixer supposed to resolve this issue? Or, is there something else I need to attempt? Or, is this the exact issue still being researched? Tomorrow is going to be hell for me if I cannot provide any updates.
    0
  • ciao70
    Hello, This morning our site certificate was updated and the cross signature with DST Root is gone. Now all devices older than Android 7.1.1 get error Did it happen to anyone else too? I am really confused with all these problems
    0
  • ciao70
    Yes, I have the same problem, old Android versions cannot load the websites with SSL. Have a nice day.

    We fixed it by manually re-entering the third DST Root chain and disabled autossl for the time being I had to download the cross signed ca cert and manually add it to combined cert used by apache
    0
  • net@work
    We fixed it by manually re-entering the third DST Root chain and disabled autossl for the time being I had to download the cross signed ca cert and manually add it to combined cert used by apache

    Hello, As I have the same issue with older devices than Android 7.1.1 get error is possible to describe how we can manage to solve this? As most of us will have the exact same problem. If we reinstall the certificate after cPanel plugin updated (2 days before - cpanel-letsencrypt-v2-1.02-1.2.1) will correct this problem or we must do something else? The LetsEncrypt says:
    0
  • monarobase
    The extending is done by including the expired certificate because older android don"t verify if a root certificate is expired, but that expired certificate causes issues with other devices, some security solutions block it and cPanel detected the certs as faulty. We had random issues for different customers running up to date software when that root chain was included. For old android devices I believe that setting e-mail to "TLS accept all certificates" and installing Firefox should resolve the problem for most users. These users will have issues with lots of websites as lots of people will decide to not include the expired root certificate due to compatibility issues with other devices.
    0
  • net@work
    Hello.
    The extending is done by including the expired certificate because older android don"t verify if a root certificate is expired, but that expired certificate causes issues with other devices, some security solutions block it and cPanel detected the certs as faulty.

    Thank you for that kind of view on this particular problem.
    These users will have issues with lots of websites as lots of people will decide to not include the expired root certificate due to compatibility issues with other devices.

    That's correct I believe. You know if we change to autossl cpanel powered by sectigo old devices like android before 7.1.1 will be ok as before or we have problems also? Because
    0
  • ciao70
    The extending is done by including the expired certificate because older android don"t verify if a root certificate is expired, but that expired certificate causes issues with other devices, some security solutions block it and cPanel detected the certs as faulty. We had random issues for different customers running up to date software when that root chain was included. For old android devices I believe that setting e-mail to "TLS accept all certificates" and installing Firefox should resolve the problem for most users. These users will have issues with lots of websites as lots of people will decide to not include the expired root certificate due to compatibility issues with other devices.

    Hello, What would be the devices that have problems? Cpanel doesn't seem to me to have communicated that it would remove DST Root. As I wrote previously, we re-entered DST Root and disabled Autossl. I expect some press release from Cpanel to explain the situation Thanks
    0
  • ciao70
    [QUOTE="net@work, post: 2878501, member: 813191"> Hello, As I have the same issue with older devices than Android 7.1.1 get error is possible to describe how we can manage to solve this? As most of us will have the exact same problem. If we reinstall the certificate after cPanel plugin updated (2 days before - cpanel-letsencrypt-v2-1.02-1.2.1) will correct this problem or we must do something else? The LetsEncrypt says:
    0
  • jorbox
    Hello, Honestly, I don't know at all, we put the DST Root certificate back manually. Let's see what Cpanel says about it.

    can you tell us how ?
    0
  • net@work
    Hello, Honestly, I don't know at all, we put the DST Root certificate back manually. Let's see what Cpanel says about it.

    First of all thank you for your answer. Can you tell us how you put DST Root certificate manually and if you face any error with dovecot/exim? Is from this list?
    0
  • jorbox
    I think chrome & outlook should include there own list like Firefox do, one of my client same very angry because he uses an old android phone I told him to install firefox and the website worked, I told him also that chrome will release an update for this soon,,
    0
  • ciao70
    [QUOTE="net@work, post: 2878553, member: 813191"> First of all thank you for your answer. Can you tell us how you put DST Root certificate manually and if you face any error with dovecot/exim?
    DST Root only for apache What I wanted to know from Cpanel is why on Apache we ended up with Certificates provided 2 instead of 3? DST Root CA X3 was gone ?
    Certificates provided 3 (4032 bytes)
    Chain issues None
    #2
    Subject R3
    Valid until Mon, 15 Sep 2025 16:00:00 UTC (expires in 3 years and 11 months)
    Key RSA 2048 bits (e 65537)
    Issuer ISRG Root X1
    Signature algorithm SHA256withRSA
    #3
    Subject ISRG Root X1
    Valid until Mon, 30 Sep 2024 18:14:03 UTC (expires in 2 years and 11 months)
    Key RSA 4096 bits (e 65537)
    Issuer DST Root CA X3
    Signature algorithm SHA256withRSA
    Now all devices even those older than Android 7.1.1 are able to connect again Sorry for my English
    0
  • cPRex Jurassic Moderator
    I'm not sure about the cert2 vs 3 issue being mentioned. I didn't see a lot of updates over the weekend, but at this time we're still looking into the root cause of why some users encountered permission problems in the /var/cpanel/ssl/domain_tls/ directory. Once I hear more about that, I'll be sure to post. As far as an "announcement" I believe the current plan is to update or create a support article once this is all said and done.
    0
  • dandadude
    In case this might help anyone: - I have ran the permission fix 2 days ago but problems have arisen with some domains since then (regarding the common name again of course) - I ran the permission fix now again and all is well (probably they had new certs generated since then, didn't check) - I have put the permission fix in my cron.hourly, just in case (until we have a real solution from cPanel)
    0
  • cPRex Jurassic Moderator
    Here's an article specifically for the permissions issue:
    0
  • ciao70
    Here's an article specifically for the permissions issue:
    0
  • cPRex Jurassic Moderator
    @ciao70 - that's correct. Articles that are linked to cases allow the end-user to "follow" the case for updates, which requires a login.
    0
  • cPRex Jurassic Moderator
    Update - we *potentially* have a cause and a fix for the odd permissions in /var/cpanel/ssl/domain_tls. I can't say more just yet (well, really I don't want to accidentally lie or spread false hope if it turns out to not be true...) but I'm hoping we'll have something official in place soon.
    0
  • dolphyn
    Here's what I think @ciao70 is suggesting, as a solution to the Android problem, and I think it has worked for me to allow websites to be viewed. (This doesn't address any email or SNI issues such as ERR_CERT_COMMON_NAME_INVALID.) Simply append the cross-signed ISRG Root X1 certificate (available https://letsencrypt.org/certs/isrg-root-x1-cross-signed.pem # Append to each "combined" file (that doesn't already include the cross-signed certificate) for file in /var/cpanel/ssl/apache_tls/*/combined; do if [ ! -z $(grep "CCBEigAwIBAgIQQAF3ITfU6UK47naqPGQK" $file) ]; then echo "FOUND in $file" else echo "" >> $file cat ~/isrg-root-x1-cross-signed.pem >> $file fi done # Graceful Restart Apache apachectl -k graceful
    0
  • Duplika
    Is there any news regarding the SSL and Android issue? Many clients upset, pity cPanel didn't plan ahead on this.
    0
  • cPRex Jurassic Moderator
    Here is what I know at this point: -CPANEL-38838 has been created with a possible fix for the permissions problems we've seen with the /var/cpanel/ssl/domain_tls directory -once that fix is official, we'll backport it to v98 and v94. -we have an additional case as well that will help with the OpenSSL logic to verify what chains are trusted and what is not -there is also a Let's Encrypt plugin update that will happen at some point soon Once all of that happens, we do expect that to help with the Android issues some users have been seeing and to resolve the issues in general. I'll be sure to post updates as I get them.
    0
  • sparek-3
    After looking into this, this is what I've been able to find. Someone feel free to correct me if I'm wrong any where in this. There are two certificates issued by DST Root CA X3 and I don't know if cPanel is aware of this or really what they have done in regards to this. There is (was) a DST Root CA X3 certificate: [font="courier new"]Issuer: DST Root CA X3 Server Name: R3 Expires: September 29, 2021 and there's also a: [font="courier new"]Issuer: DST Root CA X3 Server Name: ISRG Root X1 Expires: September 30, 2024 The one that expires on September 30, 2024 is the same one that @dolphyn (post [url=https://forums.cpanel.net/threads/cpanel-33077-letsencrypt-transition-to-isrgs-root-important.673981/post-2879289]#175) is referencing from [plain]https://letsencrypt.org/certs/isrg-root-x1-cross-signed.pem[/plain] I don't think cPanel realizes that these are two different certificates. They've just decided that all DST Root CA X3 certificates are bad. If you don't want to read any further, that's my best guess as to what happened with cPanel's "fix". There's also a third certificate: [font="courier new"]Issuer: ISRG Root X1 Server Name: R3 Expires: September 15, 2025 This is the primary, and proper CA Bundle that should be used with Let's Encrypt certificates for modern systems. This is detailed at:
    0
  • cPRex Jurassic Moderator
    I'm going to be posting a detailed summary soon
    0
  • cPRex Jurassic Moderator
    Here's a link to the summary of the Let's Encrypt events:
    0
  • sparek-3
    I guess the part I'm confused about is -Domain TLS, which includes the email services on the machine, failed, as that service does not allow the installation of invalid certificates. This caused users to receive an SSL error when connecting to the mail server as they would have been presented with the hostname SSL instead of the domain SSL. What certificate in the chain was considered invalid? And what determines validity? When issuing a Let's Encrypt certificate, the certificate for the domain is valid - yes? ISRG Root X1 (expires September 15, 2025) is valid - yes? DST Root CA X3 (expires September 30, 2024) is valid - yes? So why isn't that being installed?
    0
  • cPRex Jurassic Moderator
    The problem with that is the two-fold issue - the OpenSSL logic, and DomainTLS being much more strict about certificates than Apache. Let's Encrypt is still sending the older certificate at this time, IdenTrust DST Root R3, which fails the verification, so Domain TLS rejects that as being valid and not allowing the install to go through. This is what our first autofixer aimed to resolve.
    0
  • sparek-3
    No. I don't think that's true. I think that's where cPanel's confusing the two. And that's why I think it's faulty cPanel logic that's blocking this. But I'm man enough to admit that I may be wrong. I would just like to have it pointed out where I am wrong. The DST Root CA X3 (expired September 29, 2021) certificate: -----BEGIN CERTIFICATE----- MIIEZTCCA02gAwIBAgIQQAF1BIMUpMghjISpDBbN3zANBgkqhkiG9w0BAQsFADA/ MSQwIgYDVQQKExtEaWdpdGFsIFNpZ25hdHVyZSBUcnVzdCBDby4xFzAVBgNVBAMT DkRTVCBSb290IENBIFgzMB4XDTIwMTAwNzE5MjE0MFoXDTIxMDkyOTE5MjE0MFow MjELMAkGA1UEBhMCVVMxFjAUBgNVBAoTDUxldCdzIEVuY3J5cHQxCzAJBgNVBAMT AlIzMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAuwIVKMz2oJTTDxLs jVWSw/iC8ZmmekKIp10mqrUrucVMsa+Oa/l1yKPXD0eUFFU1V4yeqKI5GfWCPEKp Tm71O8Mu243AsFzzWTjn7c9p8FoLG77AlCQlh/o3cbMT5xys4Zvv2+Q7RVJFlqnB U840yFLuta7tj95gcOKlVKu2bQ6XpUA0ayvTvGbrZjR8+muLj1cpmfgwF126cm/7 gcWt0oZYPRfH5wm78Sv3htzB2nFd1EbjzK0lwYi8YGd1ZrPxGPeiXOZT/zqItkel /xMY6pgJdz+dU/nPAeX1pnAXFK9jpP+Zs5Od3FOnBv5IhR2haa4ldbsTzFID9e1R oYvbFQIDAQABo4IBaDCCAWQwEgYDVR0TAQH/BAgwBgEB/wIBADAOBgNVHQ8BAf8E BAMCAYYwSwYIKwYBBQUHAQEEPzA9MDsGCCsGAQUFBzAChi9odHRwOi8vYXBwcy5p ZGVudHJ1c3QuY29tL3Jvb3RzL2RzdHJvb3RjYXgzLnA3YzAfBgNVHSMEGDAWgBTE p7Gkeyxx+tvhS5B1/8QVYIWJEDBUBgNVHSAETTBLMAgGBmeBDAECATA/BgsrBgEE AYLfEwEBATAwMC4GCCsGAQUFBwIBFiJodHRwOi8vY3BzLnJvb3QteDEubGV0c2Vu Y3J5cHQub3JnMDwGA1UdHwQ1MDMwMaAvoC2GK2h0dHA6Ly9jcmwuaWRlbnRydXN0 LmNvbS9EU1RST09UQ0FYM0NSTC5jcmwwHQYDVR0OBBYEFBQusxe3WFbLrlAJQOYf r52LFMLGMB0GA1UdJQQWMBQGCCsGAQUFBwMBBggrBgEFBQcDAjANBgkqhkiG9w0B AQsFAAOCAQEA2UzgyfWEiDcx27sT4rP8i2tiEmxYt0l+PAK3qB8oYevO4C5z70kH ejWEHx2taPDY/laBL21/WKZuNTYQHHPD5b1tXgHXbnL7KqC401dk5VvCadTQsvd8 S8MXjohyc9z9/G2948kLjmE6Flh9dDYrVYA9x2O+hEPGOaEOa1eePynBgPayvUfL qjBstzLhWVQLGAkXXmNs+5ZnPBxzDJOLxhF2JIbeQAcH5H0tZrUlo5ZYyOqA7s9p O5b85o3AM/OJ+CktFBQtfvBhcJVd9wvlwPsk+uyOy2HI7mNxKKgsBTt375teA2Tw UdHkhVNcsAKX1H7GNNLOEADksd86wuoXvg== -----END CERTIFICATE----- The DST Root CA X3 (expires September 30, 2024) certificate: -----BEGIN CERTIFICATE----- MIIFYDCCBEigAwIBAgIQQAF3ITfU6UK47naqPGQKtzANBgkqhkiG9w0BAQsFADA/ MSQwIgYDVQQKExtEaWdpdGFsIFNpZ25hdHVyZSBUcnVzdCBDby4xFzAVBgNVBAMT DkRTVCBSb290IENBIFgzMB4XDTIxMDEyMDE5MTQwM1oXDTI0MDkzMDE4MTQwM1ow TzELMAkGA1UEBhMCVVMxKTAnBgNVBAoTIEludGVybmV0IFNlY3VyaXR5IFJlc2Vh cmNoIEdyb3VwMRUwEwYDVQQDEwxJU1JHIFJvb3QgWDEwggIiMA0GCSqGSIb3DQEB AQUAA4ICDwAwggIKAoICAQCt6CRz9BQ385ueK1coHIe+3LffOJCMbjzmV6B493XC ov71am72AE8o295ohmxEk7axY/0UEmu/H9LqMZshftEzPLpI9d1537O4/xLxIZpL wYqGcWlKZmZsj348cL+tKSIG8+TA5oCu4kuPt5l+lAOf00eXfJlII1PoOK5PCm+D LtFJV4yAdLbaL9A4jXsDcCEbdfIwPPqPrt3aY6vrFk/CjhFLfs8L6P+1dy70sntK 4EwSJQxwjQMpoOFTJOwT2e4ZvxCzSow/iaNhUd6shweU9GNx7C7ib1uYgeGJXDR5 bHbvO5BieebbpJovJsXQEOEO3tkQjhb7t/eo98flAgeYjzYIlefiN5YNNnWe+w5y sR2bvAP5SQXYgd0FtCrWQemsAXaVCg/Y39W9Eh81LygXbNKYwagJZHduRze6zqxZ Xmidf3LWicUGQSk+WT7dJvUkyRGnWqNMQB9GoZm1pzpRboY7nn1ypxIFeFntPlF4 FQsDj43QLwWyPntKHEtzBRL8xurgUBN8Q5N0s8p0544fAQjQMNRbcTa0B7rBMDBc SLeCO5imfWCKoqMpgsy6vYMEG6KDA0Gh1gXxG8K28Kh8hjtGqEgqiNx2mna/H2ql PRmP6zjzZN7IKw0KKP/32+IVQtQi0Cdd4Xn+GOdwiK1O5tmLOsbdJ1Fu/7xk9TND TwIDAQABo4IBRjCCAUIwDwYDVR0TAQH/BAUwAwEB/zAOBgNVHQ8BAf8EBAMCAQYw SwYIKwYBBQUHAQEEPzA9MDsGCCsGAQUFBzAChi9odHRwOi8vYXBwcy5pZGVudHJ1 c3QuY29tL3Jvb3RzL2RzdHJvb3RjYXgzLnA3YzAfBgNVHSMEGDAWgBTEp7Gkeyxx +tvhS5B1/8QVYIWJEDBUBgNVHSAETTBLMAgGBmeBDAECATA/BgsrBgEEAYLfEwEB ATAwMC4GCCsGAQUFBwIBFiJodHRwOi8vY3BzLnJvb3QteDEubGV0c2VuY3J5cHQu b3JnMDwGA1UdHwQ1MDMwMaAvoC2GK2h0dHA6Ly9jcmwuaWRlbnRydXN0LmNvbS9E U1RST09UQ0FYM0NSTC5jcmwwHQYDVR0OBBYEFHm0WeZ7tuXkAXOACIjIGlj26Ztu MA0GCSqGSIb3DQEBCwUAA4IBAQAKcwBslm7/DlLQrt2M51oGrS+o44+/yQoDFVDC 5WxCu2+b9LRPwkSICHXM6webFGJueN7sJ7o5XPWioW5WlHAQU7G75K/QosMrAdSW 9MUgNTP52GE24HGNtLi1qoJFlcDyqSMo59ahy2cI2qBDLKobkx/J3vWraV0T9VuG WCLKTVXkcGdtwlfFRjlBz4pYg1htmf5X6DYO8A4jqv2Il9DjXA6USbW1FzXSLr9O he8Y4IWS6wY7bCkjCWDcRQJMEhg76fsO3txE+FiYruq9RUWhiF1myv4Q6W+CyBFC Dfvp7OOGAN6dEOM4+qR9sdjoSYKEBpsr6GtPAQw4dy753ec5 -----END CERTIFICATE----- Those are two different certificates. Let's Encrypt is sending DST Root CA X3 (expires September 30, 2024) If DST Root CA X3 (expires September 30, 2024) is included in the CA bundle then Domain TLS certificates don't get installed. But why is DST Root CA X3 (expires September 30, 2024) invalid? The only thing I can surmise is that cPanel is seeing DST Root CA X3 and assuming it's invalid. It's not actually checking it. Thus faulty cPanel logic.
    0
  • cPanelFelipe
    LE is not sending DST Root CA X3, or any other root. They"re sending an intermediate that points back to the expired root. You can see this if you grab CPAN"s Net::ACME2 and try out the example scripts to grab a certificate. (You"ll have to tweak them to point to LE"s production servers rather than the staging ones.)
    0
  • cPanelFelipe
    The DST-issued intermediate that LE sends has the same name (i.e., Subject) as LE"s new root: "ISRG Root X1". Clients that have the actual root cert can thus ignore that extra intermediate. The problem is that OpenSSL versions prior to 1.1.0"like the 1.0.2 that CentOS 7 uses"do not, by default, ignore the extra intermediate when doing a standalone verification (such as what AutoSSL does). We"ll be shipping a fix soon to supported cP versions that configures our standalone OpenSSL verifications always to ignore extra intermediates. This will allow us to install that extra intermediate again, which will restore compatibility with old Android devices that may not have the new LE root (and don"t care about the expiration of DST Root CA X3).
    0
  • eva2000
    The problem with that is the two-fold issue - the OpenSSL logic, and DomainTLS being much more strict about certificates than Apache. Let's Encrypt is still sending the older certificate at this time, IdenTrust DST Root R3, which fails the verification, so Domain TLS rejects that as being valid and not allowing the install to go through. This is what our first autofixer aimed to resolve.

    You should be able to configure the shorter chain
    0

Please sign in to leave a comment.

Didn't find what you were looking for?

New post