Let's Encrypt validation problems after DST root expiration

[Walter Hop]

Hello all,

We run a Subversion service over HTTPS, and we use Let's Encrypt to provide HTTPS certificates.

Traditionally this worked perfectly, but since October 1st, some of our customers using TortoiseSVN are now getting a HTTPS validation error such as:

"Certificate validation failed

Error validating server certificate for https://svn3.sliksvn.com:443:

Unknown certificate issuer.

Fingerprint: 52:2C:0B:1D:1A:CD:B7:E4:F9:B8:52:7C:C3:53:37:CB:01:D2:70:7C

Distiguished name: R3, Let's Encrypt, US

Do you want to proceed?"

Example screenshot: https://aws1.discourse-cdn.com/letsencrypt/original/3X/4/c/4c86de3dafa73b97b19696857e8d253b740aeef4.png

The timing is not a coincidence. There was a change in Let's Encrypt's validation as of 1 October: https://letsencrypt.org/docs/dst-root-ca-x3-expiration-september-2021/

In short, Let's Encrypt certificates were trusted by operating systems based on two root CAs, "ISRG Root" (Let's Encrypt's new root CA) and "DST Root CA" (which already existed and is more widely trusted on old devices).

As of yesterday, "DST Root CA" has expired, leaving only the "ISRG Root" as a trust provider. All major operating systems including Windows ship the ISRG Root in their trust store since years, so the DST expiry has been a non-event for most use cases.

However, we have a small number of TortoiseSVN users who are reporting that validation of Let's Encrypt HTTPS certificates no longer works currently, giving the error above.

The users reported that they have Windows updates enabled, so their machines should have the newest root certificates.

I installed a clean Windows 10 virtual machine and setup TortoiseSVN to investigate the problem myself, and I can reproduce it with latest, updated, Windows 10 and latest TortoiseSVN, getting the same validation error. I have verified that my Windows has the "ISRG Root X1" in the certificate store, and Edge can connect normally to the HTTPS site - it is specifically TortoiseSVN that fails to validate.

I am not a Windows user myself, but it is my assumption that TortoiseSVN uses the system's certificate store. Therefore it should still trust Let's Encrypt certificates, and I am puzzled why the HTTPS validation fails in some cases.

As can be seen on Qualys SSL test, our server is sending the correct (default) Let's Encrypt certificate chain, leading up to the ISRG Root X1: https://www.ssllabs.com/ssltest/analyze.html?d=svn3.sliksvn.com&hideResults=on

Therefore my question is, could there be an issue with TortoiseSVN itself, leading it to not successfully trust Let's Encrypt's new root certificate?

Any help would be appreciated. I would be happy to test, since I have a reliable reproduce.

If someone is interested in testing HTTPS in a live scenario, we provide free Subversion accounts, one can be created here: https://sliksvn.com/signup/

For now, I have advised our users to verify the fingerprint and accept it manually - fortunately that option is there, so the users can keep on being productive... Only every few months, our certificate will renew, and I expect the users will get the error again.

Kind regards,

Walter Hop

Slik BV

---

TortoiseSVN version and bundled libraries info:

TortoiseSVN 1.14.1, Build 29085 - 64 Bit , 2021/02/09 16:17:02

ipv6 enabled

Subversion 1.14.1, -release

apr 1.6.5

apr-util 1.6.1

serf 1.3.9

OpenSSL 1.1.1i 8 Dec 2020

zlib 1.2.11

SQLite 3.29.0

[Daniel Sahlberg]

Hi,

I have received similar reports in $DAYJOB but in a quite different setup (Windows 2012, IIS, asp.net site, Apple clients). I'm not sure if you are hosting Subversion on Windows or Unix (using Unix in a broad sense including Linux and *BSD).

For me the solution was to restart the server. It seems that Windows was serving a certificate including both the old chain rooted in DST Root CA X3 (which is expired) and the new chain rooted in ISRG Root X1 (however the latter chain without one of the intermediate certificates). Apple clients choose the old chain and displayed errors while Windows clients used the new chain and downloaded the intermediate certificate as needed. After the restart, the server is (probably) only serving the new chain.

It is possible that TortoiseSVN is displaying the same behaviour as the Apple clients if the server is serving the old certificate chain.

There is also a thread in the group TortoiseSVN-dev (https://groups.google.com/g/tortoisesvn-dev/c/ZIzJPJjqW8A) regarding this problem.

I found a blog article (https://borncity.com/win/2021/09/30/lets-encrypt-zertifikate-rger-mit-windows-sophos-utm-macos-ios-30-9-2021/) containing a collection of different applications having problems. This contained the suggestion to restart the server.

SSL Labs has a good SSL server test (https://www.ssllabs.com/ssltest/) where you can see the chains sent from the server. On svn3.sliksvn.com (as well as the site mentioned in -dev) there are two Certification paths. On my $DAYJOB site there is only one chain after the server restart (I don't have a copy of the test I did yesterday but I'm quite sure that it had two chains).

Hope this gives some input. Please report back since this is probably affecting more users.

Kind regards,

Daniel Sahlberg

[Matthew Trescott]

Hi all,

Followup from https://groups.google.com/g/tortoisesvn-dev/c/ZIzJPJjqW8A/m/yhtPaOX_AgAJ

It sounds like the fact that Microsoft IIS won't serve the old chain after it expires is more a limitation than feature (https://docs.certifytheweb.com/docs/kb/kb-202109-letsencrypt/). The old chain is still used to support older Android devices because Android does not enforce the expiration date on DST Root CA X3.

Since I run mod_dav_svn on Apache, my only option (assuming this is the right solution for Tortoise compatibility) would be to remove support for these old Android devices by configuring Certbot to get a chain that ends with a self-signed ISRG Root X1 certificate (instead of an ISRG Root X1 certificate cross-signed by DST Root CA X3). Unfortunately there's no such thing as User-Agent detection with SSL/TLS...

I will try changing the chain and report back.

Best regards,

Matt

[Daniel Sahlberg]

Hi,

This is getting interesting :-)

I agree that it might be a problem for older Android clients if only one chain is served (but it will also affect equally old iOS devices), in my case it doesn't matter but YMMW.

I have a personal server using Apache and mod_dav_svn on Linux (Fedora 34) and I don't have this issue with TortoiseSVN 1.14.1.

When checking with SSLLabs, I see that all intermediate certificates are sent by the server (for both my server and the two mentioned in the thread). I'm quite certain this was not the case yesterday. Do you still experience issues?

Kind regards,

Daniel

[Matthew Trescott]

When checking with SSLLabs, I see that all intermediate certificates are sent by the server (for both my server and the two mentioned in the thread). I'm quite certain this was not the case yesterday. Do you still experience issues?

I don't believe anything changed on its own since yesterday; TortoiseSVN was still giving errors this morning. However, I completed the certificate chain switch (certbot renew --preferred-chain 'ISRG Root X1' --force-renew) and Tortoise is now satisfied, although my old Android now complains :)

So indeed the problem is that Tortoise is only checking whether the highest certificate in the chain is a trust anchor, and not considering that an intermediary could be a trust anchor as well.

[Walter Hop]

On 2 Oct 2021, at 20:44, Matthew Trescott via TortoiseSVN <torto...@googlegroups.com> wrote:

I don't believe anything changed on its own since yesterday; TortoiseSVN was still giving errors this morning. However, I completed the certificate chain switch (certbot renew --preferred-chain 'ISRG Root X1' --force-renew) and Tortoise is now satisfied, although my old Android now complains :)

That is great! It’s a fair trade-off. I’m pretty sure that SVN clients on old Android phones are extremely rare. I am definitely going to try the alternate chain. Sadly our ACME client (acme_tiny) does not support it yet, but we created a PR for it - it was rejected then, on the premise that it wouldn’t be necessary. But now that the DST expiration has finally happened, it turns out there are various corner cases where a sysop would want to choose the alternate chain.

So indeed the problem is that Tortoise is only checking whether the highest certificate in the chain is a trust anchor, and not considering that an intermediary could be a trust anchor as well.

That is important information. thank you. I don’t know the TortoiseSVN source, but this MIGHT be caused by the way that the bundled OpenSSL is configured. The OpenSSL version is new enough, and this behavior is switchable. A patch was circulated on FreeBSD-security to change OpenSSL’s default behavior so that it should succeed at the first verified certificate and no longer bother looking at the expired DST root.

--- crypto/openssl/crypto/x509/x509_vpm.c.prev  2021-10-01
09:16:51.753533000 -0400
+++ crypto/openssl/crypto/x509/x509_vpm.c       2021-10-01
09:19:39.708106000 -0400
@@ -537,7 +537,7 @@
      "default",                 /* X509 default parameters */
      0,                         /* Check time */
      0,                         /* internal flags */
-     0,                         /* flags */
+     X509_V_FLAG_TRUSTED_FIRST, /* flags */
      0,                         /* purpose */
      0,                         /* trust */
      100,                       /* depth */

Perhaps (but this is absolutely above my pay grade) a similar configuration in TortoiseSVN’s build makes OpenSSL fail, and a change to OpenSSL’s configuration might be a structural solution for TortoiseSVN.

But first I am going to experiment with the short chain.

Thanks to all for your input!

[Daniel Sahlberg]

Unless I'm mistaken, this flag is already set, at least in the "default" parameter section (as in the patch). It is not set in the "ssl_client" section but I don't know which section might be used by the TortoiseSVN connection.

However as far as I understand the workaround is only required on OpenSSL 1.0.2. TortoiseSVN is, AFAIK, using OpenSSL 1.1.1. 

Kind regards,

Daniel Sahlberg

[Tony Rietwyk]

I assume that everyone is aware that a single certificate expiry has caused issues all over the place since Thu Sep 30.  See https://www.theregister.com/2021/09/30/lets_encrypt_xero_slack_outages/

[Christian Herget]

Hello,

I searched for a workaround for this issue and the following worked for me (Windows 10).

Go to https://letsencrypt.org/certificates/ and get the active Let’s Encrypt R3 certificate (https://letsencrypt.org/certs/lets-encrypt-r3.der).

Add this certificate to your Windows certificate store. For this you just need to right click the file and select Install Certificate.

Best regards,
Christian

[Bartholomew Gander]

For the future readers of the thread - solution suggested by Christian Herget works like a charm!

Regards,

Bartek