Skip to main content

[CPANEL-33077] Letsencrypt transition to ISRG’s Root (Important!!!!!)

ciao70
Hi, In reference to this notice

Comments

203 comments

Date Votes
  • cPRex Jurassic Moderator
    The problem with that is the two-fold issue - the OpenSSL logic, and DomainTLS being much more strict about certificates than Apache. Let's Encrypt is still sending the older certificate at this time, IdenTrust DST Root R3, which fails the verification, so Domain TLS rejects that as being valid and not allowing the install to go through. This is what our first autofixer aimed to resolve.
    0
  • sparek-3
    No. I don't think that's true. I think that's where cPanel's confusing the two. And that's why I think it's faulty cPanel logic that's blocking this. But I'm man enough to admit that I may be wrong. I would just like to have it pointed out where I am wrong. The DST Root CA X3 (expired September 29, 2021) certificate: -----BEGIN CERTIFICATE----- MIIEZTCCA02gAwIBAgIQQAF1BIMUpMghjISpDBbN3zANBgkqhkiG9w0BAQsFADA/ MSQwIgYDVQQKExtEaWdpdGFsIFNpZ25hdHVyZSBUcnVzdCBDby4xFzAVBgNVBAMT DkRTVCBSb290IENBIFgzMB4XDTIwMTAwNzE5MjE0MFoXDTIxMDkyOTE5MjE0MFow MjELMAkGA1UEBhMCVVMxFjAUBgNVBAoTDUxldCdzIEVuY3J5cHQxCzAJBgNVBAMT AlIzMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAuwIVKMz2oJTTDxLs jVWSw/iC8ZmmekKIp10mqrUrucVMsa+Oa/l1yKPXD0eUFFU1V4yeqKI5GfWCPEKp Tm71O8Mu243AsFzzWTjn7c9p8FoLG77AlCQlh/o3cbMT5xys4Zvv2+Q7RVJFlqnB U840yFLuta7tj95gcOKlVKu2bQ6XpUA0ayvTvGbrZjR8+muLj1cpmfgwF126cm/7 gcWt0oZYPRfH5wm78Sv3htzB2nFd1EbjzK0lwYi8YGd1ZrPxGPeiXOZT/zqItkel /xMY6pgJdz+dU/nPAeX1pnAXFK9jpP+Zs5Od3FOnBv5IhR2haa4ldbsTzFID9e1R oYvbFQIDAQABo4IBaDCCAWQwEgYDVR0TAQH/BAgwBgEB/wIBADAOBgNVHQ8BAf8E BAMCAYYwSwYIKwYBBQUHAQEEPzA9MDsGCCsGAQUFBzAChi9odHRwOi8vYXBwcy5p ZGVudHJ1c3QuY29tL3Jvb3RzL2RzdHJvb3RjYXgzLnA3YzAfBgNVHSMEGDAWgBTE p7Gkeyxx+tvhS5B1/8QVYIWJEDBUBgNVHSAETTBLMAgGBmeBDAECATA/BgsrBgEE AYLfEwEBATAwMC4GCCsGAQUFBwIBFiJodHRwOi8vY3BzLnJvb3QteDEubGV0c2Vu Y3J5cHQub3JnMDwGA1UdHwQ1MDMwMaAvoC2GK2h0dHA6Ly9jcmwuaWRlbnRydXN0 LmNvbS9EU1RST09UQ0FYM0NSTC5jcmwwHQYDVR0OBBYEFBQusxe3WFbLrlAJQOYf r52LFMLGMB0GA1UdJQQWMBQGCCsGAQUFBwMBBggrBgEFBQcDAjANBgkqhkiG9w0B AQsFAAOCAQEA2UzgyfWEiDcx27sT4rP8i2tiEmxYt0l+PAK3qB8oYevO4C5z70kH ejWEHx2taPDY/laBL21/WKZuNTYQHHPD5b1tXgHXbnL7KqC401dk5VvCadTQsvd8 S8MXjohyc9z9/G2948kLjmE6Flh9dDYrVYA9x2O+hEPGOaEOa1eePynBgPayvUfL qjBstzLhWVQLGAkXXmNs+5ZnPBxzDJOLxhF2JIbeQAcH5H0tZrUlo5ZYyOqA7s9p O5b85o3AM/OJ+CktFBQtfvBhcJVd9wvlwPsk+uyOy2HI7mNxKKgsBTt375teA2Tw UdHkhVNcsAKX1H7GNNLOEADksd86wuoXvg== -----END CERTIFICATE----- The DST Root CA X3 (expires September 30, 2024) certificate: -----BEGIN CERTIFICATE----- MIIFYDCCBEigAwIBAgIQQAF3ITfU6UK47naqPGQKtzANBgkqhkiG9w0BAQsFADA/ MSQwIgYDVQQKExtEaWdpdGFsIFNpZ25hdHVyZSBUcnVzdCBDby4xFzAVBgNVBAMT DkRTVCBSb290IENBIFgzMB4XDTIxMDEyMDE5MTQwM1oXDTI0MDkzMDE4MTQwM1ow TzELMAkGA1UEBhMCVVMxKTAnBgNVBAoTIEludGVybmV0IFNlY3VyaXR5IFJlc2Vh cmNoIEdyb3VwMRUwEwYDVQQDEwxJU1JHIFJvb3QgWDEwggIiMA0GCSqGSIb3DQEB AQUAA4ICDwAwggIKAoICAQCt6CRz9BQ385ueK1coHIe+3LffOJCMbjzmV6B493XC ov71am72AE8o295ohmxEk7axY/0UEmu/H9LqMZshftEzPLpI9d1537O4/xLxIZpL wYqGcWlKZmZsj348cL+tKSIG8+TA5oCu4kuPt5l+lAOf00eXfJlII1PoOK5PCm+D LtFJV4yAdLbaL9A4jXsDcCEbdfIwPPqPrt3aY6vrFk/CjhFLfs8L6P+1dy70sntK 4EwSJQxwjQMpoOFTJOwT2e4ZvxCzSow/iaNhUd6shweU9GNx7C7ib1uYgeGJXDR5 bHbvO5BieebbpJovJsXQEOEO3tkQjhb7t/eo98flAgeYjzYIlefiN5YNNnWe+w5y sR2bvAP5SQXYgd0FtCrWQemsAXaVCg/Y39W9Eh81LygXbNKYwagJZHduRze6zqxZ Xmidf3LWicUGQSk+WT7dJvUkyRGnWqNMQB9GoZm1pzpRboY7nn1ypxIFeFntPlF4 FQsDj43QLwWyPntKHEtzBRL8xurgUBN8Q5N0s8p0544fAQjQMNRbcTa0B7rBMDBc SLeCO5imfWCKoqMpgsy6vYMEG6KDA0Gh1gXxG8K28Kh8hjtGqEgqiNx2mna/H2ql PRmP6zjzZN7IKw0KKP/32+IVQtQi0Cdd4Xn+GOdwiK1O5tmLOsbdJ1Fu/7xk9TND TwIDAQABo4IBRjCCAUIwDwYDVR0TAQH/BAUwAwEB/zAOBgNVHQ8BAf8EBAMCAQYw SwYIKwYBBQUHAQEEPzA9MDsGCCsGAQUFBzAChi9odHRwOi8vYXBwcy5pZGVudHJ1 c3QuY29tL3Jvb3RzL2RzdHJvb3RjYXgzLnA3YzAfBgNVHSMEGDAWgBTEp7Gkeyxx +tvhS5B1/8QVYIWJEDBUBgNVHSAETTBLMAgGBmeBDAECATA/BgsrBgEEAYLfEwEB ATAwMC4GCCsGAQUFBwIBFiJodHRwOi8vY3BzLnJvb3QteDEubGV0c2VuY3J5cHQu b3JnMDwGA1UdHwQ1MDMwMaAvoC2GK2h0dHA6Ly9jcmwuaWRlbnRydXN0LmNvbS9E U1RST09UQ0FYM0NSTC5jcmwwHQYDVR0OBBYEFHm0WeZ7tuXkAXOACIjIGlj26Ztu MA0GCSqGSIb3DQEBCwUAA4IBAQAKcwBslm7/DlLQrt2M51oGrS+o44+/yQoDFVDC 5WxCu2+b9LRPwkSICHXM6webFGJueN7sJ7o5XPWioW5WlHAQU7G75K/QosMrAdSW 9MUgNTP52GE24HGNtLi1qoJFlcDyqSMo59ahy2cI2qBDLKobkx/J3vWraV0T9VuG WCLKTVXkcGdtwlfFRjlBz4pYg1htmf5X6DYO8A4jqv2Il9DjXA6USbW1FzXSLr9O he8Y4IWS6wY7bCkjCWDcRQJMEhg76fsO3txE+FiYruq9RUWhiF1myv4Q6W+CyBFC Dfvp7OOGAN6dEOM4+qR9sdjoSYKEBpsr6GtPAQw4dy753ec5 -----END CERTIFICATE----- Those are two different certificates. Let's Encrypt is sending DST Root CA X3 (expires September 30, 2024) If DST Root CA X3 (expires September 30, 2024) is included in the CA bundle then Domain TLS certificates don't get installed. But why is DST Root CA X3 (expires September 30, 2024) invalid? The only thing I can surmise is that cPanel is seeing DST Root CA X3 and assuming it's invalid. It's not actually checking it. Thus faulty cPanel logic.
    0
  • cPanelFelipe
    LE is not sending DST Root CA X3, or any other root. They"re sending an intermediate that points back to the expired root. You can see this if you grab CPAN"s Net::ACME2 and try out the example scripts to grab a certificate. (You"ll have to tweak them to point to LE"s production servers rather than the staging ones.)
    0
  • cPanelFelipe
    The DST-issued intermediate that LE sends has the same name (i.e., Subject) as LE"s new root: "ISRG Root X1". Clients that have the actual root cert can thus ignore that extra intermediate. The problem is that OpenSSL versions prior to 1.1.0"like the 1.0.2 that CentOS 7 uses"do not, by default, ignore the extra intermediate when doing a standalone verification (such as what AutoSSL does). We"ll be shipping a fix soon to supported cP versions that configures our standalone OpenSSL verifications always to ignore extra intermediates. This will allow us to install that extra intermediate again, which will restore compatibility with old Android devices that may not have the new LE root (and don"t care about the expiration of DST Root CA X3).
    0
  • eva2000
    The problem with that is the two-fold issue - the OpenSSL logic, and DomainTLS being much more strict about certificates than Apache. Let's Encrypt is still sending the older certificate at this time, IdenTrust DST Root R3, which fails the verification, so Domain TLS rejects that as being valid and not allowing the install to go through. This is what our first autofixer aimed to resolve.

    You should be able to configure the shorter chain
    0
  • eva2000
    We"ll be shipping a fix soon to supported cP versions that configures our standalone OpenSSL verifications always to ignore extra intermediates

    I guess that means you are rebuilding the Cpanel openssl 1.0.2 package with
    That flag does not ignore extra intermediates buts changes how it walks up the chain to verify so essentially will ignore the expired DST ROOT CA X3 cert Or just using openssl verify with -trusted first flag which does the same thing
    0
  • cPanelFelipe
    I guess that means you are rebuilding the Cpanel openssl 1.0.2 package with not need to have trusted-first mode configured manually; only standalone verification seems to need it. You can see this if you `openssl s_client` to helloworld.letsencrypt.org.
    0
  • eva2000
    OpenSSL 1.0.2 does not need to have trusted-first mode configured manually

    Yup same from my tests as long as expired root is removed from the trust store - ca-certificates yum updated
    0
  • net@work
    We"ll be shipping a fix soon to supported cP versions that configures our standalone OpenSSL verifications always to ignore extra intermediates. This will allow us to install that extra intermediate again, which will restore compatibility with old Android devices that may not have the new LE root (and don"t care about the expiration of DST Root CA X3).

    Hello, Can you please be more specific for "We"ll be shipping a fix soon". You have any timeline when this problem will be fixed and all Android issues been correct? Thank you.
    0
  • cPRex Jurassic Moderator
    @net@work - I don't have any timelines at this point. I'll be sure to update this thread when I get them.
    0
  • jorbox
    @net@work - I don't have any timelines at this point. I'll be sure to update this thread when I get them.

    Can you please check it, days are passing and we lose a lot of customers
    0
  • cPanelAnthony
    @jorbox I just checked in and am also not seeing a timeline as Rex reported. I apologize for any inconvenience and thank you for your patience.
    0
  • cPRex Jurassic Moderator
    At this point I know we weren't able to get to it in the next v98 release coming out. However, we're hoping it will get to the one after that. I'll be sure to post once I get more details.
    0
  • jandafields
    Can you please check it, days are passing and we lose a lot of customers

    Why don't you simply switch the AutoSSL provider to cPanel instead of LetsEncrypt? That's about a 5 minute job and your customers should all be happy again.
    0
  • monarobase
    Switching to cPanel causes issues for customers who have created a CAA record to only allow let"s Encrypt and will also cause problems for customers who require wildcard certificates.
    0
  • jorbox
    Why don't you simply switch the AutoSSL provider to cPanel instead of LetsEncrypt? That's about a 5 minute job and your customers should all be happy again.

    Thank you but Sub domains, wildcard are not enabled with cPanel being an AutoSSL provider, even mail.domain.com, a lot of customers will have problems with their websites, its less than a 5 minute job but it will make hours of problems, my customers wont be happy they may kill me instead.
    0
  • spikey221
    I was wondering if there is any way to resolve this for those still on Centos 6? I've recently purchased a business that has about 130 Wordpress sites on a Centos 6 server running cPanel 86.0.40. I've run the steps here RHEL/CentOS 6 OpenSSL client compatibility after DST Root CA X3 expiration to get a newer version of openssl and updated certificates installed but I suspect this isn't the problem here from reading the post-mortem. I full understand that I need to get these sites off this server but that takes time so in the meantime is there a way we can manually patch autossl so that it doesn't spam Let's Encrypt for new certificates? as this is also causing issues when trying to migrate sites to a new server when we're unable to issue a certificate on that server due to rate limiting! :)
    0
  • ciao70
    At this point I know we weren't able to get to it in the next v98 release coming out. However, we're hoping it will get to the one after that. I'll be sure to post once I get more details.

    v100 release is out You have news? Thanks
    0
  • cPanelAnthony
    Hello! I will check on this with our team when they are in tomorrow and update this thread.
    0
  • spikey221
    I was wondering if there is any way to resolve this for those still on Centos 6? I've recently purchased a business that has about 130 Wordpress sites on a Centos 6 server running cPanel 86.0.40. I've run the steps here [CODE=bash] yum install wget yum install krb5-devel zlib-devel lksctp-tools-devel util-linux make gcc rpm-build curl -o openssl-1.0.2k-21.el7_9.src.rpm https://vault.centos.org/7.9.2009/updates/Source/SPackages/openssl-1.0.2k-21.el7_9.src.rpm rpm -i openssl-1.0.2k-21.el7_9.src.rpm cd ~/rpmbuild/SOURCES/ sed -i 's/secure_getenv(/getenv(/g' *patch cd ../SPECS/ sed -i 's/%patch68 -p1 -b .secure-getenv/#%patch68 -p1 -b .secure-getenv/g' openssl.spec rpmbuild -bb openssl.spec cd ../RPMS/x86_64 rpm -U openssl-libs-1.0.2k-21.el6.x86_64.rpm openssl-1.0.2k-21.el6.x86_64.rpm openssl-devel-1.0.2k-21.el6.x86_64.rpm
    Do this to rebuild the certificate store: [CODE=bash] yum install asciidoc java-1.6.0-openjdk mkdir -p /dl/ca-cert; cd /dl/ca-cert; curl -o ca-certificates-2020.2.41-65.1.el6_10.src.rpm https://vault.centos.org/6.10/updates/Source/SPackages/ca-certificates-2020.2.41-65.1.el6_10.src.rpm rpm -i ca-certificates-2020.2.41-65.1.el6_10.src.rpm curl -o ca-certificates-2021.2.50-72.el7_9.src.rpm https://vault.centos.org/7.9.2009/updates/Source/SPackages/ca-certificates-2021.2.50-72.el7_9.src.rpm rpm2cpio ca-certificates-2021.2.50-72.el7_9.src.rpm | cpio -idmv cp certdata.txt ~/rpmbuild/SOURCES/ sed -i 's/Version: 2020.2.41/Version: 2021.2.50/g' ~/rpmbuild/SPECS/ca-certificates.spec cd ~/rpmbuild/SPECS rpmbuild -bb ca-certificates.spec cd /root/rpmbuild/RPMS/noarch/ rpm -U ca-certificates-2021.2.50-65.1.el6.noarch.rpm
    That's verbatim from the linked page except that I added the installation of the openssl-devel package as that is required by cPanel. I turned off AutoSSL for a couple of days to allow the rate limits to expire, then turned it on and tested one site. It renewed successfully and now no longer complains about the broken trust chain. I must add that I did run the fix script mentioned in previous posts but I didn't think that would have worked because my version of WHM is too old. Perhaps this also contributed to things working again.
    0
  • cPanelAnthony
    As an update! While I can't make any guarantees, it looks like the fix was proposed for version 102.
    0
  • sparek-3
    By the time the "fix" gets rolled out. Everyone affected will have either upgraded to a supported OS or moved to another system where this works.
    0
  • eva2000
    For CentOS 6 and cPanel I found the cleanest way is just to spin up a hourly CentOS 7 VPS server with the likes of Upcloud or Hetzner and install cPanel trial license and then do a WHM data transfer from CentOS 6 cPanel to CentOS 7 cPanel. Then I rebuild the CentOS 6 server with CentOS 7 and then migrated data back. Only took a few hours so hourly VPS cost was low.
    0

Please sign in to leave a comment.

Didn't find what you were looking for?

New post