CA Survey Responses on Removing Old Roots

140 views
Skip to first unread message

Ben Wilson

unread,
May 13, 2021, 3:45:45 PMMay 13
to dev-secur...@mozilla.org

In Item 8 of the April 2021 CA Survey, we asked CAs about the challenges, strategies, and information gaps involved with removing old root CA certificates from the root store. 

Below is a summary of the suggestions we received.

  • Pre-2006/pre-2012 roots should not be automatically considered non-compliant with the Baseline Requirements. Instead, Mozilla should remove roots that cannot provide an unbroken sequence of period-of-time audits. Alternatively, Mozilla should focus on industry timeframes (e.g. NIST) for the removal of 2048-bit RSA roots.

  • Mozilla should provide data on how many clients and third parties are still using older roots that might be seriously impacted by their removal from NSS.

  • Mozilla should balance the risks it intends to address by removing the roots with the disruptions it will cause.

  • Mozilla should publish its timelines for root removals, including root submission, review, inclusion, and deprecation of the older roots.

  • Mozilla needs to coordinate the effort with other root programs (e.g. Microsoft, Apple and Google) and should streamline the process to ensure sufficient ubiquity of new roots.

  • An older root CA should only be distrusted 15 months after the new root CA is included in all of Mozilla, Microsoft, Apple and Google.

  • The inclusion process is too long, and this effort will clog inclusion requests. Mozilla and the other root stores need an abbreviated process for this effort.

  • Cross-certification adds another certificate to the chain, which is sometimes difficult for customers to configure, and S/MIME clients do not process cross certificates very well.

  • Mozilla should distinguish between root CAs supporting serverAuth certificates vs. S/MIME certificates, and Mozilla should keep the email trust bit and remove the websites trust bit to help bridge the transition.

  • Mozilla should publish guidance, in coordination with other root stores, consisting of:
  • important aspects of root inclusion and requirements applicable to roots to help facilitate the removal and inclusion process; and

  • good and bad practices from past experience and mistakes to help CAs avoid future non-compliance and delays.


What are everyone's thoughts regarding these suggestions?

Thanks,

Ben

Ryan Sleevi

unread,
May 13, 2021, 7:58:32 PMMay 13
to Ben Wilson, dev-secur...@mozilla.org
On Thu, May 13, 2021 at 3:45 PM Ben Wilson <bwi...@mozilla.com> wrote:

In Item 8 of the April 2021 CA Survey, we asked CAs about the challenges, strategies, and information gaps involved with removing old root CA certificates from the root store. 

Below is a summary of the suggestions we received.

  • Pre-2006/pre-2012 roots should not be automatically considered non-compliant with the Baseline Requirements. Instead, Mozilla should remove roots that cannot provide an unbroken sequence of period-of-time audits. Alternatively, Mozilla should focus on industry timeframes (e.g. NIST) for the removal of 2048-bit RSA roots.


This is nonsense. While it's understandable for CA self-interest to take this position, we have enough academic research and lived experience to know that this simply doesn't pass the basic sniff test. With respect to audits, it assumes that everything we know now, in 2021, was also known in pre-2006 and pre-2012 and considered then - but that's simply not true. The amount of clarifications or absolute BS we've seen from CAs and auditors in the period is fairly strong evidence that mistakes were made (and, arguably, pre-2016, the very real norm for all CAs).

That's not to say that some CAs weren't doing their very best, or that some CAs were substantially better than others, but holistically, this doesn't hold, and in looking to apply a policy that provides some assurance for users, and some degree of objectivity, means that either we accept known-garbage or we treat all CAs equally.
 
  • Mozilla should provide data on how many clients and third parties are still using older roots that might be seriously impacted by their removal from NSS.


This is also nonsense. The CA can transition to new roots and new hierarchies largely without issue. The CA is the one responsible for knowing what clients their roots are involved in, and for knowing what capabilities those clients support, and for communicating any challenges back.

If this were to be taken seriously, then it's the same as the CA saying it's impossible for them to issue new intermediates. That's obviously farcical, and so there's no basis for that.

As discussed below, this is something that absolutely can be quantified, today, voluntarily, and so it is within CAs interests to do so, before it becomes mandatory.
 
  • Mozilla should balance the risks it intends to address by removing the roots with the disruptions it will cause.


The CA bears the burden to establish what, if any, disruption. This is functionally no different than adding a new intermediate, and any CA even suggesting disruption bears the burden of evidence to concretely articulate that.
 
  • Mozilla should publish its timelines for root removals, including root submission, review, inclusion, and deprecation of the older roots.


This is reasonable, to a degree - any transition basically has to consider the extant, still-valid certificates. In general, this means a period as such:

T=0: New root is created. The CA performs testing (e.g. their own, allowing customers to opt-in)
T=X: Last certificate issued from OLD.
T=X+1: The CA MUST transition new certificate issuance to this new root. This is accomplished by cross-signing NEW with OLD, creating a chain "OLD -> NEW -> INTERMEDIATE -> LEAF"

At X+{max validity at time X}, you can then remove OLD, because all the OLD certificates have expired, and all new certificates have a valid chain to NEW (which also happens to be a valid chain to OLD)

So while it's definitely true that Mozilla shouldn't want to remove old roots while there are still certificates chaining to those old roots that *only* work with those old roots, that situation will always be true unless and until the CA takes steps to create new roots and transition issuance to these new roots. It's in both Mozilla's, and CA's, interests for CAs to do that sooner than later, to work out concretely what (if any) issues there are.

Functionally, this is the same as the SHA-1 migration or the RSA-1024 migration. The sooner CAs begin the work, the sooner the entire ecosystem can move. If CAs don't voluntarily do it, they may be required to do so.

So what does that timeframe concretely look like? Well, had CA/B Forum Ballot 185 been adopted when it was proposed, it would be easier. Instead, because this was delayed until Ballot SC31, it's complicated.
  • Prior to 2020-09-01, certificates could be valid up to 825 days.
  • On/After 2020-09-01, certificates are valid for up to 398 days
This means that the last two year certificate expires on 2022-12-04 (825 days from 2020-08-31). Assuming that requiring a replacement of such certificates is wholly off the table, that sets the lower-bound on when a zero-disruption replacement of an existing root can be made, assuming CAs began creating new roots today.

If we ignore certificates issued prior to 2020-09-01 (simply for sake of discussion), then what we're saying is the removal of OLD happens at T=X+398 days. Adding NEW can happen at any point from T=0 onwards, because Mozilla products will handle that just fine. The CA can transition to issuing new certs from NEW at any point from T=0 on. Since no doubt CAs will want some time to test, there's some sense in having the usual MAY/SHOULD/MUST transition (where T=0 is MAY and T=X+1 is MUST).

As to the certificates issued prior to 2020-09-01, it might be asked "Why even discuss this challenge now, if nothing can be done until 2022-12-04?" The answer is simple: If a CA has not fully migrated (MUST) to NEW by 2021-11-01, then each day after that they continue to issue from OLD is another day added to that 2022-12-04 transition timeline.

So if we want to transition in, say, 2023 at the latest, then we need CAs to be moving before the fall freeze. And if we want them to be able to get good feedback from their users and minimize disruption, we need them to start creating NEW now, so they can be working out these issues.

The worst case would be saying something like:
  • By 2021-07-01, every CA MUST have created a NEW to replace (one or more) of their OLD
  • We want zero disruption and assume the worst case
    • This means every user who holds a certificate under OLD should be able to naturally renew (meaning the transition does not start until 2022-12-04, when the last of OLD expires)
    • When they naturally renew, we should give them a certificate from NEW.
    • If they run into issues, they can get a certificate from OLD
    • We'll allow zero disruption, meaning they have the full lifetime of OLD (398 days) to fix whatever issues with NEW there were. They also can continue to obtain OLD (without a new lifetime) in the event of things like revocation. This means the MUST doesn't start until 2023-12-04, which is also the upper-bound for removing CAs.
No data supports the timeline needing to be that long, and its primarily that long because of the existing 2y certs. Being willing to require an earlier renewal specifically for those legacy 2y certs would help bring that timeline in. Similarly, there are a number of CAs that have already created NEW and begun this process, so there's no need to believe 2023-12-04 should be the goal - we can and should expect sooner transition.
 
  • Mozilla needs to coordinate the effort with other root programs (e.g. Microsoft, Apple and Google) and should streamline the process to ensure sufficient ubiquity of new roots.


This is nonsense. Any CA with an extant root has zero issues with ubiquity, because they can simply cross-sign new-with-old, as CAs have been doing now for the better part of 25 years. This is just FUD.
 
  • An older root CA should only be distrusted 15 months after the new root CA is included in all of Mozilla, Microsoft, Apple and Google.


This is nonsense as well, for the same reason above.

The 15 months here is at least unsupported as presented, but presumably this was highlighting either the "to test in parallel" or "to allow certificates to expire". Testing in parallel is entirely reasonable, but they can do that independent of NEW being added (because of the cross-sign), so it's not reasonable here specifically. "To allow certificates to expire" may be reasonable, but as the timeline above captures, that's a trade-off, and one best mitigated by CAs starting sooner than later.
 
  • The inclusion process is too long, and this effort will clog inclusion requests. Mozilla and the other root stores need an abbreviated process for this effort.


This is unreasonable, primarily because there is no operational gating on the inclusion process of NEW. Even if Mozilla was completely clogged through 2025, the transition outline above would provide zero disruption for CAs.

Certainly, there's benefit in prioritization, but the question of abbreviation or not is not really something that I think we should consider CAs' views on that heavily. That's because what they're really asking for is to not be held to the same standard, of review or discussion, as other CAs, and while that's no doubt appealing, that's probably not a great outcome. Ultimately, however, that's for Mozilla and the community to decide, based on the community's needs, because the CA is otherwise largely unimpacted by this.

Certainly, however, the "and other root stores" is nonsense. There's no need for the coordination/synchronization, precisely because they can proceed independently. And there's no guarantee that NEW will even be accepted by other root stores - it may very well be that the root store decides to end its relationship with the CA, both NEW and OLD alike, based on facts of such a review. That shouldn't block Mozilla, for example, if they decide to continue the relationship but others do not, just as adding to Mozilla now does not guarantee inclusion in other root stores.

  • Cross-certification adds another certificate to the chain, which is sometimes difficult for customers to configure, and S/MIME clients do not process cross certificates very well.

  • Mozilla should distinguish between root CAs supporting serverAuth certificates vs. S/MIME certificates, and Mozilla should keep the email trust bit and remove the websites trust bit to help bridge the transition.


These are somewhat reasonable, although the statement about S/MIME clients is one that demands evidence, and is a little suspect based on the major implementations I've examined in the past. Supported with concrete data, it might be reasonable to treat a TLS transition independent of an S/MIME transition.
 
  • Mozilla should publish guidance, in coordination with other root stores, consisting of:
  • important aspects of root inclusion and requirements applicable to roots to help facilitate the removal and inclusion process; and

  • good and bad practices from past experience and mistakes to help CAs avoid future non-compliance and delays.


This is entirely reasonable, although I would argue non-blocking. Existing CAs already have a reasonable expectation placed on them to be aware of m.d.s.p. and CA incidents, and so existing CAs should already be considering these good and bad practices. Certainly, it's within the realm of reasonableness to expect CAs to themselves perform an analysis of incidents and inclusions from, say, the past X years, as part of the application process, and this is something that is good.

I'm concerned about the amount of feedback that seems to try and shift the responsibility to Mozilla - in terms of evidence of claims CAs are making, in terms of existing expectations. I do believe some of these suggestions were made in earnest good faith, and not as "We can't/won't if you don't", and so while I have strong words about some of the points raised, there's definitely opportunities for greater collaboration. However, that collaboration can't be seen as shifting the burden of expectations from CAs, who play an important role in protecting users and thus need to be capable of, and are expected to be capable of, doing some of these tasks themselves.

Wojtek Porczyk

unread,
May 14, 2021, 5:17:26 AMMay 14
to Ryan Sleevi, Ben Wilson, dev-secur...@mozilla.org
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

On Thu, May 13, 2021 at 07:58:17PM -0400, Ryan Sleevi wrote:
> > -
> >
> > Mozilla should publish its timelines for root removals, including root
> > submission, review, inclusion, and deprecation of the older roots.
> >
> >
> This is reasonable, to a degree - any transition basically has to consider
> the extant, still-valid certificates. In general, this means a period as
> such:

[snip]

The explanation is right, but missed the part about review and inclusion.
I'd like to add I strongly object to publishing any timelines for inclusion,
especially if that would mean an upper limit on time period in which the
application may be reviewed or else the application is accepted. Quite the
opposite, the application should be accepted when community feels it's OK and
not before.

The current system of 3-week public discussion with a limit of 1 CA under
discussion at a time is reasonable, especially for people who don't read this
list daily.


- --
pozdrawiam / best regards
Wojtek Porczyk

I do not fear computers,
I fear lack of them.
-- Isaac Asimov
-----BEGIN PGP SIGNATURE-----

iQIzBAEBCgAdFiEEaO0VFfpr0tEF6hYkv2vZMhA6I1EFAmCeQB8ACgkQv2vZMhA6
I1G3Lg/9GLDuOAGaY0Oxg8tkxfjdaUSFm5zTf7LVkE/GaCYzcMtebWM01XZu91d8
0Sk32D7XTopeXI9VCnefr6IzFAbEw/ZeeL/sqlU2PGkhvxHpMr1dKGj2O6r9elC8
EvWI9B7xBrap+bzQTB7eEXIxLcwvm+B1eicZ6C6s8q8g2BbF8mz5+gyw7C/GfdRR
ggPXXUa+G2rVvY/IIfBhR8RIr6yvFJj/gDYq8XLQoNItre0U4nA2Oi+08nmDIIrC
bJDoTFGQU/lC/e+G4xKaK0ZRn8tO5t5evHdOHYLv0BKE5o8CJmIvV+sWU4WgY+YV
3h8lUZQhd/qLnZzfjZhtrUDUgr6FCXPsueTlL+1h+uw3nG+YTEGALiiHYZdRFdVU
zOFhOatRUK1/DW32QNydY4RKE+jBr6YyjknDZn8DYzha2HU+sXdUgtnsWNZ3mlzA
GibFbpI+ARLx/d7FD9xG9VMwZmpHRmI7cmmDKJUEpyxosf31zmsxWs1ef2CHdvmp
jqsblB7EcZHz6tWAV6yGCE7wdM8y5vq9gTcXzXkkOD/pfp3IWFew/XQcSiZeTno3
IDtvKlyQjFcEi3byrT0Xqnw7hE+ghh2gLBelP2vOwGd87bP+HfuICOKXzKkEYqAf
hoKaWqts5cXpj3WS78HQZ9jPIoj9a6ZJYsI6EYS9IHGm7KBm9Fc=
=lsfu
-----END PGP SIGNATURE-----
Reply all
Reply to author
Forward
0 new messages