CA Survey Responses on Removing Old Roots

[Ben Wilson]

In Item 8 of the April 2021 CA Survey, we asked CAs about the challenges, strategies, and information gaps involved with removing old root CA certificates from the root store. 

Below is a summary of the suggestions we received.

• Pre-2006/pre-2012 roots should not be automatically considered non-compliant with the Baseline Requirements. Instead, Mozilla should remove roots that cannot provide an unbroken sequence of period-of-time audits. Alternatively, Mozilla should focus on industry timeframes (e.g. NIST) for the removal of 2048-bit RSA roots.

• Mozilla should provide data on how many clients and third parties are still using older roots that might be seriously impacted by their removal from NSS.

• Mozilla should balance the risks it intends to address by removing the roots with the disruptions it will cause.

• Mozilla should publish its timelines for root removals, including root submission, review, inclusion, and deprecation of the older roots.

• Mozilla needs to coordinate the effort with other root programs (e.g. Microsoft, Apple and Google) and should streamline the process to ensure sufficient ubiquity of new roots.

• An older root CA should only be distrusted 15 months after the new root CA is included in all of Mozilla, Microsoft, Apple and Google.

• The inclusion process is too long, and this effort will clog inclusion requests. Mozilla and the other root stores need an abbreviated process for this effort.

• Cross-certification adds another certificate to the chain, which is sometimes difficult for customers to configure, and S/MIME clients do not process cross certificates very well.

• Mozilla should distinguish between root CAs supporting serverAuth certificates vs. S/MIME certificates, and Mozilla should keep the email trust bit and remove the websites trust bit to help bridge the transition.

• Mozilla should publish guidance, in coordination with other root stores, consisting of:

• important aspects of root inclusion and requirements applicable to roots to help facilitate the removal and inclusion process; and

• good and bad practices from past experience and mistakes to help CAs avoid future non-compliance and delays.

What are everyone's thoughts regarding these suggestions?

Thanks,

Ben

[Ryan Sleevi]

On Thu, May 13, 2021 at 3:45 PM Ben Wilson <bwi...@mozilla.com> wrote:

In Item 8 of the April 2021 CA Survey, we asked CAs about the challenges, strategies, and information gaps involved with removing old root CA certificates from the root store. 

Below is a summary of the suggestions we received.

• Pre-2006/pre-2012 roots should not be automatically considered non-compliant with the Baseline Requirements. Instead, Mozilla should remove roots that cannot provide an unbroken sequence of period-of-time audits. Alternatively, Mozilla should focus on industry timeframes (e.g. NIST) for the removal of 2048-bit RSA roots.

This is nonsense. While it's understandable for CA self-interest to take this position, we have enough academic research and lived experience to know that this simply doesn't pass the basic sniff test. With respect to audits, it assumes that everything we know now, in 2021, was also known in pre-2006 and pre-2012 and considered then - but that's simply not true. The amount of clarifications or absolute BS we've seen from CAs and auditors in the period is fairly strong evidence that mistakes were made (and, arguably, pre-2016, the very real norm for all CAs).

That's not to say that some CAs weren't doing their very best, or that some CAs were substantially better than others, but holistically, this doesn't hold, and in looking to apply a policy that provides some assurance for users, and some degree of objectivity, means that either we accept known-garbage or we treat all CAs equally.

 

• Mozilla should provide data on how many clients and third parties are still using older roots that might be seriously impacted by their removal from NSS.

This is also nonsense. The CA can transition to new roots and new hierarchies largely without issue. The CA is the one responsible for knowing what clients their roots are involved in, and for knowing what capabilities those clients support, and for communicating any challenges back.

If this were to be taken seriously, then it's the same as the CA saying it's impossible for them to issue new intermediates. That's obviously farcical, and so there's no basis for that.

As discussed below, this is something that absolutely can be quantified, today, voluntarily, and so it is within CAs interests to do so, before it becomes mandatory.

 

• Mozilla should balance the risks it intends to address by removing the roots with the disruptions it will cause.

The CA bears the burden to establish what, if any, disruption. This is functionally no different than adding a new intermediate, and any CA even suggesting disruption bears the burden of evidence to concretely articulate that.

 

• Mozilla should publish its timelines for root removals, including root submission, review, inclusion, and deprecation of the older roots.

This is reasonable, to a degree - any transition basically has to consider the extant, still-valid certificates. In general, this means a period as such:

T=0: New root is created. The CA performs testing (e.g. their own, allowing customers to opt-in)

T=X: Last certificate issued from OLD.

T=X+1: The CA MUST transition new certificate issuance to this new root. This is accomplished by cross-signing NEW with OLD, creating a chain "OLD -> NEW -> INTERMEDIATE -> LEAF"

At X+{max validity at time X}, you can then remove OLD, because all the OLD certificates have expired, and all new certificates have a valid chain to NEW (which also happens to be a valid chain to OLD)

So while it's definitely true that Mozilla shouldn't want to remove old roots while there are still certificates chaining to those old roots that *only* work with those old roots, that situation will always be true unless and until the CA takes steps to create new roots and transition issuance to these new roots. It's in both Mozilla's, and CA's, interests for CAs to do that sooner than later, to work out concretely what (if any) issues there are.

Functionally, this is the same as the SHA-1 migration or the RSA-1024 migration. The sooner CAs begin the work, the sooner the entire ecosystem can move. If CAs don't voluntarily do it, they may be required to do so.

So what does that timeframe concretely look like? Well, had CA/B Forum Ballot 185 been adopted when it was proposed, it would be easier. Instead, because this was delayed until Ballot SC31, it's complicated.

• Prior to 2020-09-01, certificates could be valid up to 825 days.
• On/After 2020-09-01, certificates are valid for up to 398 days

This means that the last two year certificate expires on 2022-12-04 (825 days from 2020-08-31). Assuming that requiring a replacement of such certificates is wholly off the table, that sets the lower-bound on when a zero-disruption replacement of an existing root can be made, assuming CAs began creating new roots today.

If we ignore certificates issued prior to 2020-09-01 (simply for sake of discussion), then what we're saying is the removal of OLD happens at T=X+398 days. Adding NEW can happen at any point from T=0 onwards, because Mozilla products will handle that just fine. The CA can transition to issuing new certs from NEW at any point from T=0 on. Since no doubt CAs will want some time to test, there's some sense in having the usual MAY/SHOULD/MUST transition (where T=0 is MAY and T=X+1 is MUST).

As to the certificates issued prior to 2020-09-01, it might be asked "Why even discuss this challenge now, if nothing can be done until 2022-12-04?" The answer is simple: If a CA has not fully migrated (MUST) to NEW by 2021-11-01, then each day after that they continue to issue from OLD is another day added to that 2022-12-04 transition timeline.

So if we want to transition in, say, 2023 at the latest, then we need CAs to be moving before the fall freeze. And if we want them to be able to get good feedback from their users and minimize disruption, we need them to start creating NEW now, so they can be working out these issues.

The worst case would be saying something like:

• By 2021-07-01, every CA MUST have created a NEW to replace (one or more) of their OLD
• We want zero disruption and assume the worst case
• This means every user who holds a certificate under OLD should be able to naturally renew (meaning the transition does not start until 2022-12-04, when the last of OLD expires)
• When they naturally renew, we should give them a certificate from NEW.
• If they run into issues, they can get a certificate from OLD
• We'll allow zero disruption, meaning they have the full lifetime of OLD (398 days) to fix whatever issues with NEW there were. They also can continue to obtain OLD (without a new lifetime) in the event of things like revocation. This means the MUST doesn't start until 2023-12-04, which is also the upper-bound for removing CAs.

No data supports the timeline needing to be that long, and its primarily that long because of the existing 2y certs. Being willing to require an earlier renewal specifically for those legacy 2y certs would help bring that timeline in. Similarly, there are a number of CAs that have already created NEW and begun this process, so there's no need to believe 2023-12-04 should be the goal - we can and should expect sooner transition.

 

• Mozilla needs to coordinate the effort with other root programs (e.g. Microsoft, Apple and Google) and should streamline the process to ensure sufficient ubiquity of new roots.

This is nonsense. Any CA with an extant root has zero issues with ubiquity, because they can simply cross-sign new-with-old, as CAs have been doing now for the better part of 25 years. This is just FUD.

 

• An older root CA should only be distrusted 15 months after the new root CA is included in all of Mozilla, Microsoft, Apple and Google.

This is nonsense as well, for the same reason above.

The 15 months here is at least unsupported as presented, but presumably this was highlighting either the "to test in parallel" or "to allow certificates to expire". Testing in parallel is entirely reasonable, but they can do that independent of NEW being added (because of the cross-sign), so it's not reasonable here specifically. "To allow certificates to expire" may be reasonable, but as the timeline above captures, that's a trade-off, and one best mitigated by CAs starting sooner than later.

 

• The inclusion process is too long, and this effort will clog inclusion requests. Mozilla and the other root stores need an abbreviated process for this effort.

This is unreasonable, primarily because there is no operational gating on the inclusion process of NEW. Even if Mozilla was completely clogged through 2025, the transition outline above would provide zero disruption for CAs.

Certainly, there's benefit in prioritization, but the question of abbreviation or not is not really something that I think we should consider CAs' views on that heavily. That's because what they're really asking for is to not be held to the same standard, of review or discussion, as other CAs, and while that's no doubt appealing, that's probably not a great outcome. Ultimately, however, that's for Mozilla and the community to decide, based on the community's needs, because the CA is otherwise largely unimpacted by this.

Certainly, however, the "and other root stores" is nonsense. There's no need for the coordination/synchronization, precisely because they can proceed independently. And there's no guarantee that NEW will even be accepted by other root stores - it may very well be that the root store decides to end its relationship with the CA, both NEW and OLD alike, based on facts of such a review. That shouldn't block Mozilla, for example, if they decide to continue the relationship but others do not, just as adding to Mozilla now does not guarantee inclusion in other root stores.

• Cross-certification adds another certificate to the chain, which is sometimes difficult for customers to configure, and S/MIME clients do not process cross certificates very well.

• Mozilla should distinguish between root CAs supporting serverAuth certificates vs. S/MIME certificates, and Mozilla should keep the email trust bit and remove the websites trust bit to help bridge the transition.

These are somewhat reasonable, although the statement about S/MIME clients is one that demands evidence, and is a little suspect based on the major implementations I've examined in the past. Supported with concrete data, it might be reasonable to treat a TLS transition independent of an S/MIME transition.

 

• Mozilla should publish guidance, in coordination with other root stores, consisting of:

• important aspects of root inclusion and requirements applicable to roots to help facilitate the removal and inclusion process; and

• good and bad practices from past experience and mistakes to help CAs avoid future non-compliance and delays.

This is entirely reasonable, although I would argue non-blocking. Existing CAs already have a reasonable expectation placed on them to be aware of m.d.s.p. and CA incidents, and so existing CAs should already be considering these good and bad practices. Certainly, it's within the realm of reasonableness to expect CAs to themselves perform an analysis of incidents and inclusions from, say, the past X years, as part of the application process, and this is something that is good.

I'm concerned about the amount of feedback that seems to try and shift the responsibility to Mozilla - in terms of evidence of claims CAs are making, in terms of existing expectations. I do believe some of these suggestions were made in earnest good faith, and not as "We can't/won't if you don't", and so while I have strong words about some of the points raised, there's definitely opportunities for greater collaboration. However, that collaboration can't be seen as shifting the burden of expectations from CAs, who play an important role in protecting users and thus need to be capable of, and are expected to be capable of, doing some of these tasks themselves.

[Wojtek Porczyk]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

On Thu, May 13, 2021 at 07:58:17PM -0400, Ryan Sleevi wrote:
> > -

> >
> > Mozilla should publish its timelines for root removals, including root
> > submission, review, inclusion, and deprecation of the older roots.
> >
> >
> This is reasonable, to a degree - any transition basically has to consider
> the extant, still-valid certificates. In general, this means a period as
> such:

[snip]

The explanation is right, but missed the part about review and inclusion.
I'd like to add I strongly object to publishing any timelines for inclusion,
especially if that would mean an upper limit on time period in which the
application may be reviewed or else the application is accepted. Quite the
opposite, the application should be accepted when community feels it's OK and
not before.

The current system of 3-week public discussion with a limit of 1 CA under
discussion at a time is reasonable, especially for people who don't read this
list daily.


- --
pozdrawiam / best regards
Wojtek Porczyk

I do not fear computers,
I fear lack of them.
-- Isaac Asimov
-----BEGIN PGP SIGNATURE-----

iQIzBAEBCgAdFiEEaO0VFfpr0tEF6hYkv2vZMhA6I1EFAmCeQB8ACgkQv2vZMhA6
I1G3Lg/9GLDuOAGaY0Oxg8tkxfjdaUSFm5zTf7LVkE/GaCYzcMtebWM01XZu91d8
0Sk32D7XTopeXI9VCnefr6IzFAbEw/ZeeL/sqlU2PGkhvxHpMr1dKGj2O6r9elC8
EvWI9B7xBrap+bzQTB7eEXIxLcwvm+B1eicZ6C6s8q8g2BbF8mz5+gyw7C/GfdRR
ggPXXUa+G2rVvY/IIfBhR8RIr6yvFJj/gDYq8XLQoNItre0U4nA2Oi+08nmDIIrC
bJDoTFGQU/lC/e+G4xKaK0ZRn8tO5t5evHdOHYLv0BKE5o8CJmIvV+sWU4WgY+YV
3h8lUZQhd/qLnZzfjZhtrUDUgr6FCXPsueTlL+1h+uw3nG+YTEGALiiHYZdRFdVU
zOFhOatRUK1/DW32QNydY4RKE+jBr6YyjknDZn8DYzha2HU+sXdUgtnsWNZ3mlzA
GibFbpI+ARLx/d7FD9xG9VMwZmpHRmI7cmmDKJUEpyxosf31zmsxWs1ef2CHdvmp
jqsblB7EcZHz6tWAV6yGCE7wdM8y5vq9gTcXzXkkOD/pfp3IWFew/XQcSiZeTno3
IDtvKlyQjFcEi3byrT0Xqnw7hE+ghh2gLBelP2vOwGd87bP+HfuICOKXzKkEYqAf
hoKaWqts5cXpj3WS78HQZ9jPIoj9a6ZJYsI6EYS9IHGm7KBm9Fc=
=lsfu
-----END PGP SIGNATURE-----

[Dimitris Zacharopoulos]

On 14/5/2021 2:58 π.μ., Ryan Sleevi wrote:

• Cross-certification adds another certificate to the chain, which is sometimes difficult for customers to configure, and S/MIME clients do not process cross certificates very well.

• Mozilla should distinguish between root CAs supporting serverAuth certificates vs. S/MIME certificates, and Mozilla should keep the email trust bit and remove the websites trust bit to help bridge the transition.

These are somewhat reasonable, although the statement about S/MIME clients is one that demands evidence, and is a little suspect based on the major implementations I've examined in the past. Supported with concrete data, it might be reasonable to treat a TLS transition independent of an S/MIME transition.

We were able to reproduce and confirm past findings on this issue regarding the S/MIME agents not using cross-certificates in the chain.

Our tests were performed using Thunderbird as sending agent and TB/Outlook as receiving agents.

Sending agent trusted old and new Root and included the cross-certificate in local certificate store.

The signed email contained only the Intermediate Certificate that chains to the new Root. There was no way to "dictate" TB to include the cross-certificate in the signature.

The receiving agents trusted only the old Root.

The receiving agents did not attempt to follow the AIA CAIssuers URI in order to get the cross-certificate, therefore the path validation failed and the signature was marked invalid.


Dimitris.

[Dimitris Zacharopoulos]

On 9/8/2021 8:25 μ.μ., Dimitris Zacharopoulos wrote:

On 14/5/2021 2:58 π.μ., Ryan Sleevi wrote:

• Cross-certification adds another certificate to the chain, which is sometimes difficult for customers to configure, and S/MIME clients do not process cross certificates very well.

• Mozilla should distinguish between root CAs supporting serverAuth certificates vs. S/MIME certificates, and Mozilla should keep the email trust bit and remove the websites trust bit to help bridge the transition.

These are somewhat reasonable, although the statement about S/MIME clients is one that demands evidence, and is a little suspect based on the major implementations I've examined in the past. Supported with concrete data, it might be reasonable to treat a TLS transition independent of an S/MIME transition.

We were able to reproduce and confirm past findings on this issue regarding the S/MIME agents not using cross-certificates in the chain.

Minor correction. "not using cross-certificates in the chain" --> "not adding cross-certificates in the chain" during the signing operation.

The signer would need to do weird "tricks" with the local trust store to make the agent add the cross-certificate in the chain.

Our tests were performed using Thunderbird as sending agent and TB/Outlook as receiving agents.

Sending agent trusted old and new Root and included the cross-certificate in local certificate store.

The signed email contained only the Intermediate Certificate that chains to the new Root. There was no way to "dictate" TB to include the cross-certificate in the signature.

The receiving agents trusted only the old Root.

The receiving agents did not attempt to follow the AIA CAIssuers URI in order to get the cross-certificate, therefore the path validation failed and the signature was marked invalid.


Dimitris.

--
You received this message because you are subscribed to the Google Groups "dev-secur...@mozilla.org" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dev-security-po...@mozilla.org.
To view this discussion on the web visit https://groups.google.com/a/mozilla.org/d/msgid/dev-security-policy/b8891dcf-b042-4514-2324-61d0cc101c13%40it.auth.gr.

[Ryan Sleevi]

Thanks. This, at least, is a little more concrete as to what you meant by "do not process well" - it seems what you were actually testing here is whether or not AIA fetching occurs, correct?

It's certainly not unexpected that Thunderbird does not do AIA chasing, especially since Firefox does not. At a minimum, the Thunderbird behaviour will be affected by what path builder it's using (which depends on which NSS version as well as the build flags for your distribution - e.g. if it's Mozilla-provided Thunderbird vs a distro-provided version), but even there, I suspect Thunderbird-proper hasn't wired up AIA. It should, however, properly handle the scenario where sender sends both, as well as paths that chain to "new certified by old".

It's unclear, however, from your description on your test methodology for the certificate chain. It sounds like you did "(new leaf) <- (new intermediate)" as the actual message contents, but it's unclear if you constructed a chain "(new leaf) <- (new intermediate) <- (new root) <- (old root)" or a chain of "(new leaf) <- (new intermediate) <- (new root), (new leaf) <- (old intermediate) <- (old root)", the latter of which is expected to be problematic.

The test here is:

- Sending agent sends certificate bundle that contains their cert, intermediate, and new-signed-by-old

That should work. Outlook's calls through WinVerifyTrust/CAPI will handle that, as should Thunderbird, provided the notBefore/notAfter dates are set appropriately for (new root signed by old root).

Maybe I'm further misunderstanding what you're meaning though by "do not process cross certificates well", but it seems this was a test methodology issue more than anything?

[Dimitris Zacharopoulos]

On 9/8/2021 10:18 μ.μ., Ryan Sleevi wrote:

On Mon, Aug 9, 2021 at 1:25 PM Dimitris Zacharopoulos <ji...@it.auth.gr> wrote:

On 14/5/2021 2:58 π.μ., Ryan Sleevi wrote:

• Cross-certification adds another certificate to the chain, which is sometimes difficult for customers to configure, and S/MIME clients do not process cross certificates very well.

• Mozilla should distinguish between root CAs supporting serverAuth certificates vs. S/MIME certificates, and Mozilla should keep the email trust bit and remove the websites trust bit to help bridge the transition.

These are somewhat reasonable, although the statement about S/MIME clients is one that demands evidence, and is a little suspect based on the major implementations I've examined in the past. Supported with concrete data, it might be reasonable to treat a TLS transition independent of an S/MIME transition.

We were able to reproduce and confirm past findings on this issue regarding the S/MIME agents not using cross-certificates in the chain.

Our tests were performed using Thunderbird as sending agent and TB/Outlook as receiving agents.

Sending agent trusted old and new Root and included the cross-certificate in local certificate store.

The signed email contained only the Intermediate Certificate that chains to the new Root. There was no way to "dictate" TB to include the cross-certificate in the signature.

The receiving agents trusted only the old Root.

The receiving agents did not attempt to follow the AIA CAIssuers URI in order to get the cross-certificate, therefore the path validation failed and the signature was marked invalid.

Thanks. This, at least, is a little more concrete as to what you meant by "do not process well" - it seems what you were actually testing here is whether or not AIA fetching occurs, correct?

Not quite, we first ensured that the sending agent did not add the cross-certificate in the certificate chain, which we knew it would make it impossible for the receiving agent to successfully validate the chain, and then "hoped" for AIA fetching to occur.

It's certainly not unexpected that Thunderbird does not do AIA chasing, especially since Firefox does not. At a minimum, the Thunderbird behaviour will be affected by what path builder it's using (which depends on which NSS version as well as the build flags for your distribution - e.g. if it's Mozilla-provided Thunderbird vs a distro-provided version), but even there, I suspect Thunderbird-proper hasn't wired up AIA. It should, however, properly handle the scenario where sender sends both, as well as paths that chain to "new certified by old".

It's unclear, however, from your description on your test methodology for the certificate chain. It sounds like you did "(new leaf) <- (new intermediate)" as the actual message contents, but it's unclear if you constructed a chain "(new leaf) <- (new intermediate) <- (new root) <- (old root)" or a chain of "(new leaf) <- (new intermediate) <- (new root), (new leaf) <- (old intermediate) <- (old root)", the latter of which is expected to be problematic.

We constructed the "(new leaf) <- (new intermediate) <- (new root) <- (old root)".

The test here is:

- Sending agent sends certificate bundle that contains their cert, intermediate, and new-signed-by-old

This is where the sending mail agents fail. As explained in the test, TB and Outlook do not have a way to "inject" the cross-certificate in the chain, therefore the default behavior is to only send (new leaf) <- (new intermediate).

If the cross-certificate is included in the chain, the receiving agents successfully validate the chain. In order to do that, the sender must manually delete the (new root) from the local trust store so the sending agent can build a path to the (old root), using the cross-certificate, but that's not something we want to enourage.

I hope this makes things a bit clearer.


Thanks,
Dimitris.

[Jesper Kristensen]

Hi Ryan

You seem to only focus on how the recipient email application will validate a certificate chain it receives. Do you disagree that it is important how the sending email application chooses what chain it will put in the message?

In TLS, most servers will send the chain exactly as you configured it, so it might be less relevant for TLS. Though the discussion here indicate that it can also be relevant for TLS: https://community.letsencrypt.org/t/potential-problem-with-r3-intermediates-on-windows-servers/157164

--

You received this message because you are subscribed to the Google Groups "dev-secur...@mozilla.org" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dev-security-po...@mozilla.org.

To view this discussion on the web visit https://groups.google.com/a/mozilla.org/d/msgid/dev-security-policy/CAErg%3DHHpnzu2OWdPZ_9535Z0tiztgXMnn3%2BBqVKQA4fr%3DxqNng%40mail.gmail.com.

[Ryan Sleevi]

On Tue, Aug 10, 2021 at 5:05 AM Jesper Kristensen <jespe...@gmail.com> wrote:

Hi Ryan

You seem to only focus on how the recipient email application will validate a certificate chain it receives. Do you disagree that it is important how the sending email application chooses what chain it will put in the message?

I am focused on that point, because that is the current point of discussion. That doesn’t detract from discussing other things, but with respect to the original statement, the follow-up question, the subsequent testing, and the follow-ups there, the focus was, and is, how the recipient will behave.

Certainly, sender behavior can have relevance, but given that the specific concern here is new software at the sender (“new” in some unavoidable sense, as the concern captured this far is about the existence of the new root in the sender) needing to communicate with a recipient on an older version, the primary point of discussion is simply trying to be explicit about what in particular is concerning about the recipient.