Groups keyboard shortcuts have been updated
Dismiss
See shortcuts

MRSP 3.0: Issue #276: Delayed Revocation

4,377 views
Skip to first unread message

Ben Wilson

unread,
Dec 15, 2024, 3:51:43 PM12/15/24
to dev-secur...@mozilla.org

All,

The purpose of this email is to start discussion of Mozilla GitHub Issue #276 ("Address Delayed Revocation"). We would like to collect comments and feedback on a proposal to address delayed certificate revocation from a Mozilla perspective. It builds on prior discussions and feedback regarding delayed revocation, and the proposal is meant to replace guidance currently provided on the Mozilla CA wiki.

Here is the comparison link for a proposed new section 6.1.3 in the MRSP:

https://github.com/mozilla/pkipolicy/compare/51b2f702accd54cb70d52081a9e814298433495b%E2%80%A6efa8ac40ac341fb813620938ef72328a53858038

Summary

Here are the highlights of the proposal:

  • Revocation must occur promptly in compliance with the timelines set in section 4.9.1 of the TLS Baseline Requirements (TLS BRs). Mozilla does not grant exceptions to these timelines.
  • New CA Obligations:
    • Educate subscribers on revocation timelines and discourage reliance on certificates in systems that cannot tolerate timely revocation.
    • Include contractual language requiring subscriber cooperation with revocation timelines.
    • Maintain and test mass revocation plans annually, including the revocation of 30 randomly chosen certificates within a 5-day period.
  • Beginning April 15, 2026, CA audit reports must attest to compliance with the mass revocation planning requirements.
  • Delayed revocation incidents must be reported per version 2.1 of the CCADB's Incident Reporting Guidelines (as currently proposed)
  • Repeated delayed revocation incidents will result in heightened scrutiny or sanctions, which may include root removal.

Background

Earlier this year, on this list, I proposed an Interim Policy to Address Delayed Revocation. While the proposed interim policy provided clarity, it faced criticism regarding implementation complexity, burden on subscribers and CAs, and the feasibility of associated measures, such as transitioning delayed revocation domains to 90-day certificates. Also, there were subsequent proposals aimed at reducing certificate lifetimes and encouraging automation. See e.g. https://github.com/cabforum/servercert/pull/553.

This new proposal drops proposed measures such as domain-specific tracking and subscriber attestations and instead focuses on subscriber education, mass revocation preparedness, and robust incident reporting as the primary mechanisms for improving agility and transparency regarding delayed revocation.

If adopted, the proposed MRSP § 6.1.3 would replace the current guidance on delayed revocation in Mozilla’s wiki and ensure consistency with the CCADB's Incident Reporting Guidelines.

I welcome your feedback on this draft proposal. Please share your thoughts, questions, or concerns to help us refine and improve it further.

Thanks,

Ben Wilson

Mozilla Root Store

Pedro Fuentes

unread,
Dec 16, 2024, 5:02:35 AM12/16/24
to dev-secur...@mozilla.org, Ben Wilson
Hi Ben,
About " annual plan testing by revoking 30 randomly chosen certificates within a 5-day timeframe; and"...
We understand that this mean that a CA will need to randomly revoke 30 certificates that most likely don't have other reason for being revoked than being randomly chosen and customers will just need to "happily" accept the situation... Harsh, but doable...

About "audit report submitted under section 3.1 SHALL include an attestation that the CA operator has met these mass revocation planning requirements", it must be considered that the attestation letters of Webtrust audit reports have a fixed format, so such addition would be added most likely as a "Other matters" section, that audits can take each differently.

My question here would be if you think it's there any chance that these requirements become part of the TLS BRs instead of the Mozilla Policy, I see several benefits here:
- Checking the mass-revocation plan would be integral part of the audit scope, so auditors don't need to figure out how to include it in the reports... it just needs to be added to the audit criteria.
- The "inverse-lottery" thing could be added in the revocation timelines of the BR, so there's an entry in the 5-day deadline adding a new category "The certificate has been randomly chosen for revocation during an internal audit". This should facilitate the contractual language to add in the subscriber agreement.

Ben Wilson

unread,
Dec 18, 2024, 12:17:46 PM12/18/24
to dev-secur...@mozilla.org

All,

As part of the discussions on this proposal, namely that CAs “maintain and test mass revocation plans annually, including the revocation of 30 randomly chosen certificates within a 5-day period,” I’ve received a few comments via private channels, and to ensure transparency and foster discussion, I am sharing them here anonymously:

1. “Mozilla does not grant exceptions…” -- this is the most important signal that Mozilla can provide.

2. If certificate consumers want to prohibit delayed revocation, then they need to make it clear to CAs that they won't accept it and that they will kick them out of the root stores if they still do it. Don't try to solve this issue with indirect measures like random revocations. Just be straight about it and make it clear that there will be consequences for the very first delayed revocation and onward.

3. We will face big problems in revoking productive customer certificates just to test our mass-revocation plan and procedures. Our current customer contracts do not foresee this. While we can revoke at any time for security or compliance reasons, this authorization should not be used just to test mass-revocation. This will also require us to push out contract changes to our complete TLS customer base, which will take a considerable amount of time and effort.

4. This part of the proposal should occur within the CA/Browser Forum through amendments to the TLS Baseline Requirements, and not via Mozilla Root Store Policy.

5. Why was the number 30 chosen as a sample?  Some CA operators issue very few certificates, while some CAs issue millions of certificates.

I welcome your feedback on these points, the random sampling proposal, and any others.

Thanks,

Ben

Mike Shaver

unread,
Dec 18, 2024, 12:46:01 PM12/18/24
to Ben Wilson, dev-secur...@mozilla.org
Thanks for this summary, Ben.

WRT point 3, I think a good way to roll out randomly-sampled certificate revocation is as an amendment to the CPS of all CAs; CAs amend their CPSes all the time without major contractual upheaval, so it seems like a useful tool here. Coupled with an appropriate effective-after date, we could be sure that all issued certificates were governed by a CPS version that includes random revocation as audit mechanism.

WRT the random selection itself, I would like to see them chosen via an externally-transparent process rather than leaving it to CAs to be appropriately random. Whether by Mozilla or a CCADB process, something that samples from crt.sh in an appropriate way, with a public randomness seed, would give higher confidence in the process. CAs will otherwise have a strong incentive to “roll the dice again” if the random selection includes a major or stubborn customer, and those are the cases that most need to be tested IMO.

I think for all of these proposals, it’s important to state what the consequences of violation will be.

I don’t think that Mozilla should feel constrained to only have policies that exactly match the TLS BRs. They are *baseline* requirements, not exhaustive requirements, and each cert consumer has its own context in which to make tradeoffs between inclusiveness and rigour.

More broadly, it would be good to state what the main purpose and values of the MRSP are: for example, is the goal to permit as many CAs as can meet the requirements to encourage broad competition, or to optimize for the security of Mozilla’s users (which might mean having fewer CAs for which oversight by Mozilla can be more thorough)?

Mike

--
You received this message because you are subscribed to the Google Groups "dev-secur...@mozilla.org" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dev-security-po...@mozilla.org.
To view this discussion visit https://groups.google.com/a/mozilla.org/d/msgid/dev-security-policy/71b07640-d425-4f2f-8da4-d97a9475b9f6n%40mozilla.org.

Jeremy Rowley

unread,
Dec 18, 2024, 12:53:51 PM12/18/24
to dev-secur...@mozilla.org, Ben Wilson
I agree with point number 2. I think Mozilla should just make it clear delayed revocations are not permitted and call it good. This is the only industry I know where we'd propose breaking availability in the CIA triangle just in case integrity might be compromised. Better to pull off the band-aid and, assuming we really like 5 day revocations, either a) require that companies use ACME to deploy certificates or b) simply state that CAs delaying revocation past 5 days will likely be distrusted. 

Wayne

unread,
Dec 18, 2024, 2:02:26 PM12/18/24
to dev-secur...@mozilla.org
Thanks for the breakdown of responses so far Ben.

Fully agreed on point 1. Partially agree with point 2 but there are benefits for proving that the revocation system functions in practice and having a random revocation cycle creates better habits in the consumer space.

Point 3 is unusual as revocation for compliance reasons is explicitly outlined, and that's all that this is? There isn't any new changes to the legal language required that I can see.

Point 4 has been covered, being above the baseline requirements is barely noteworthy but encouraging these practices in more programs is good advice.

Point 5 is a matter of scale, in prior discussions % of total active issuance were put forward as ideas. Would the CAs handling 100s of certificates per year be the ones to push the idea of it being 10% of all issuance? I'm sure the CAs can decide on which matter would suit their practices best as long as the figure is above 0. It is ultimately arbitrary unless we're moving to a % where we can be assured of coverage across a given number of years.

I'm of mixed minds as to Jeremy's comment though. All regulatory practices I'm aware of propose that availability will be broken when integrity is at risk, it's also fundamental to the WebPKI as an ecosystem. Putting that aside, I would consider a CA exception to the revocation audit minimums if short-lived certificates were all that were issued. Same substance for revocation exemption in general as per the BRs.

- Wayne

Rich Salz

unread,
Dec 18, 2024, 3:05:54 PM12/18/24
to Ben Wilson, dev-secur...@mozilla.org

As part of the discussions on this proposal, namely that CAs “maintain and test mass revocation plans annually, including the revocation of 30 randomly chosen certificates within a 5-day period,” I’ve received a few comments via private channels, and to ensure transparency and foster discussion, I am sharing them here anonymously:

Would a CA be allowed to pre-notify customers whose certs were randomly selected and {pre/re}-issue them replacements?

Ben Wilson

unread,
Dec 18, 2024, 10:48:15 PM12/18/24
to Rich Salz, dev-secur...@mozilla.org

Dear Rich,

Thank you for your question.

I think it would be advisable for a CA operator’s mass-revocation testing plan to include an immediate notice to all customers whose certificates were randomly selected because we would want to minimize disruption to server operations while testing the CA’s ability to revoke and replace certificates promptly.

That said, CAs should consider performing occasional tests that go beyond providing pre-generated replacement certificates, in which subscribers generate and submit new public keys. That would address the risks from a widespread incident, like Heartbleed, where the potential compromise of private keys necessitated key pair replacement. Preparing for such scenarios ensures that subscribers will be able to quickly perform the tasks of key pair generation, public key submission, and certificate installation.

Much to discuss.

Thanks again,

Ben

Matt Palmer

unread,
Dec 19, 2024, 5:02:43 AM12/19/24
to dev-secur...@mozilla.org
On Wed, Dec 18, 2024 at 09:17:45AM -0800, 'Ben Wilson' via dev-secur...@mozilla.org wrote:
> As part of the discussions on this proposal, namely that CAs “maintain and
> test mass revocation plans annually, including the revocation of 30
> randomly chosen certificates within a 5-day period,” I’ve received a few
> comments via private channels, and to ensure transparency and foster
> discussion, I am sharing them here anonymously:
>
> 1. “Mozilla does not grant exceptions…” -- this is the most important
> signal that Mozilla can provide.

This has been fairly unambiguous from my perspective, and yet it
appears that this wording has been open to misinterpretation.

> 2. If certificate consumers want to prohibit delayed revocation, then they
> need to make it clear to CAs that they won't accept it and that they will
> kick them out of the root stores if they still do it. Don't try to solve
> this issue with indirect measures like random revocations. Just be straight
> about it and make it clear that there will be consequences for the very
> first delayed revocation and onward.

I do believe that clearly spelled-out consequences for non-compliance
would go some way to encouraging good behaviour. The current approach,
where non-compliance may or may not have any repercussions for the CA,
doesn't help those within the CA who are fighting the good fight to push
back against terrible ideas. "It might lead to bad things happening" is
a very weak argument compared to "if we do this, Mozilla will definitely
remove us from their trust store".

> 3. We will face big problems in revoking productive customer certificates
> just to test our mass-revocation plan and procedures. Our current customer
> contracts do not foresee this. While we can revoke at any time for security
> or compliance reasons,

What is "Mozilla makes us revoke 30 randomly-chosen certificates,
RNJesus has decided that today's your turn in the barrel" if not a
compliance reason?

> 4. This part of the proposal should occur within the CA/Browser Forum
> through amendments to the TLS Baseline Requirements, and not via Mozilla
> Root Store Policy.

... so that all the CAs can vote against it and kill it. As the name
suggests, *Baseline* Requirements are the very lowest common denominator
that all parties are willing to accept; hog-tying one trust store that
wishes to innovate is completely unacceptable.

> 5. Why was the number 30 chosen as a sample? Some CA operators issue very
> few certificates, while some CAs issue millions of certificates.

I was considering responding to this part of the original proposal, to
suggest "30 or 0.N% of annual issuance volume, whichever is lower" type
of language, but when I thought about it further, if an entire CA is
issuing so few certificates that revoking 30 is an unreasonable burden,
I'd be very concerned about their operational practices in general,
given how little practice they get with them.

On the subject of the selection of certificates, I'd like to echo the
other comments expressing concerns about the degree to which CAs might
seek to "game" the random selection. It's one of those "it's too easy
to do, and too hard to get caught" situations where shenanigans (or even
the appearance of shenanigans) like to live.

- Matt

Roman Fischer

unread,
Dec 19, 2024, 10:43:45 AM12/19/24
to dev-secur...@mozilla.org

Dear everybody,

 

>Point 3 is unusual as revocation for compliance reasons is explicitly outlined, and that's all that this is? There isn't any new changes to the legal language required that I can see.

 

I checked with our legal department and they clearly stated that we will have to amend the contracts if we want to be sure that we can't be sued for damages inflicted by random revocation. Current language (at least in our contracts) allows us to revoke in case of security incidents, non-compliance and some other cases. But a preventative revocation would currently not be covered.

 

Also, I'm not convinced that it will motivate many customers towards automation. Most will probably just gamble and bet on not being in the unlucky 30, especially if they get their certs from a large CA... 😉

 

 

>Point 4 has been covered, being above the baseline requirements is barely noteworthy but encouraging these practices in more programs is good advice.

 

Regarding a comment made in another mail, I think it's fair to say that most public trust CAs -need- to conform to all major root store policies to be of any value for their customers. The more fragmented these root store policies are, the more complex (and error prone!) complying to them becomes. For this reason, we would really encourage the root store operators to converge on requirements set forward.

 

Kind regards
Roman

--

You received this message because you are subscribed to the Google Groups "dev-secur...@mozilla.org" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dev-security-po...@mozilla.org.

Andrew Chen

unread,
Dec 19, 2024, 11:00:33 AM12/19/24
to Ben Wilson, Rich Salz, dev-secur...@mozilla.org
Ben -- thanks for the proposal. Random revocation is likely to be painful for subscribers without automation around eminent revocation, which ARI is expected to address. Would it make sense to couple this proposal to a specific adoption rate of ARI amongst subscribers?

Andrew

--
You received this message because you are subscribed to the Google Groups "dev-secur...@mozilla.org" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dev-security-po...@mozilla.org.

Mike Shaver

unread,
Dec 19, 2024, 11:30:03 AM12/19/24
to Andrew Chen, Ben Wilson, Rich Salz, dev-secur...@mozilla.org
Do you mean that CAs could avoid having to do random revocation by discouraging ARI adoption among their subscribers?

The point is that right now revocation is so painful that it’s causing CAs to side with subscriber convenience over the integrity of the web PKI. Sampled, controlled revocations let us identify points of pain before they have security implications, and motivate Subscribers to prepare their systems—whether through automation or not, up to them, I’m not their dad—to tolerate on-time revocation. We care about the likely outcomes of automation, such as tolerance of short revocation or expiry timelines, really, but if BigSlowCo wants to staff a 24-hour cert maintenance squad such that they don’t (successfully) pressure their CA into blowing revocation deadlines, that’s their opex choice. Directly evaluating ecosystem capability around prompt revocation is the only way I can think of to identify areas of danger or weakness before they become issues for the web.

Mike

Andrew Chen

unread,
Dec 19, 2024, 11:45:20 AM12/19/24
to Mike Shaver, Ben Wilson, Rich Salz, dev-secur...@mozilla.org
Mike -- that's a reasonable point. I suppose making it a date OR percentage of ARI adoption isn't likely to change things, either. It would be nice if we could incentivize ARI adoption and make random revocation a non-event for many? most? people.

Andrew

Mike Shaver

unread,
Dec 19, 2024, 11:59:46 AM12/19/24
to Roman Fischer, dev-secur...@mozilla.org
On Thu, Dec 19, 2024 at 10:43 AM Roman Fischer <roman....@swisssign.com> wrote:

Dear everybody,

 

>Point 3 is unusual as revocation for compliance reasons is explicitly outlined, and that's all that this is? There isn't any new changes to the legal language required that I can see.

 

I checked with our legal department and they clearly stated that we will have to amend the contracts if we want to be sure that we can't be sued for damages inflicted by random revocation. Current language (at least in our contracts) allows us to revoke in case of security incidents, non-compliance and some other cases. But a preventative revocation would currently not be covered.

A surprisingly direct statement from inside counsel, given that the detailed language of the policy component hasn’t been determined! But of course as we know you can get sued for anything, and the question is whether you eventually prevail in court. We saw with the unfortunate DigiCert situation earlier this year that even clear contract language around revocation promptness can’t protect against misleading filings or overly-sympathetic judges, at least in terms of initial injunctions and bringing a case that’s not trivially dismissed.

If a CPS update is not sufficient—which would confuse me, but that’s not impossible!—your legal department would have ~18 months to roll out some (likely trivial) new language and get it adopted during renewals or through other means. I have confidence in them!

An alternative, of course, is that Mozilla or whoever just adds N sampled certs to OneCRL with 5 days notice to the CA. Given that, there’s nothing that a CA needs to do other than inform the subscriber and reissue when asked, but that only really tests Subscriber capability and not CA commitment to compliance. I think we have seen an unfortunate abundance of evidence that both are areas of concern around prompt revocation, not that they necessarily need to be addressed by the same mechanism.

Also, I'm not convinced that it will motivate many customers towards automation. Most will probably just gamble and bet on not being in the unlucky 30, especially if they get their certs from a large CA...

Yes, this is a clear weakness of the fixed-30 element of the proposal, but that’s an argument for proportionate sampling, not for avoiding the practice entirely.

I have a hard time reconciling “most will probably ignore this context” with “it would be too disruptive for subscribers”, I admit.

Regarding a comment made in another mail, I think it's fair to say that most public trust CAs -need- to conform to all major root store policies to be of any value for their customers. The more fragmented these root store policies are, the more complex (and error prone!) complying to them becomes. For this reason, we would really encourage the root store operators to converge on requirements set forward.

A requirement is no harder to meet because it’s from one root store than if it applies to multiple, surely. What is the source of additional error risk that goes away when something is more widely adopted by root programs? Does it become easier to participate in CT if that requirement from Chrome’s program is adopted by others?

Are you saying that it’s too hard for a CA to analyze a handful of different root store policies and determine what the requirements *are*? Within the limits of reasonableness, that’s a job for the quality of language in the program descriptions; this is an area where there could be some improvements, which is part of why Ben is driving this refresh of the MSRP I think.

Independent of that, I think it would be great for more root stores to join in this new practice, and I would like advocate to them for exactly that. That’s a separate question from whether it needs to be in the BRs via the CABF. I think it can be effective without inclusion in the BRs, just like Chrome’s policy on CT has been effective in driving adoption.

Mike

Mike Shaver

unread,
Dec 19, 2024, 11:59:46 AM12/19/24
to Andrew Chen, Ben Wilson, Rich Salz, dev-secur...@mozilla.org
Pressure that causes more frequent cert issuance and installation is pretty much the only thing that will incentivize adoption of things that make issuance and installation easier, I think. Certainly many CAs, per their incident Action Items, have been “educating subscribers” about the merits of automation for some time, with what I would say are unsatisfactory results.

Mike

Jeremy Rowley

unread,
Dec 19, 2024, 1:07:41 PM12/19/24
to Mike Shaver, Andrew Chen, Ben Wilson, Rich Salz, dev-secur...@mozilla.org
I agree that educating subscribers has been largely ineffective. However, randomly causing outages won't solve the issue . Random revocations will just make people hate the browsers and CAs more. It's a strange solution to the objective of faster certificate replacement when short-lived certificates force automation as does simply requiring CAs to mandate automation. Only allow CAs to deliver certificates via an automated solution and all of a sudden you have 100% automation adoption.

Mike Shaver

unread,
Dec 19, 2024, 1:15:26 PM12/19/24
to Jeremy Rowley, Andrew Chen, Ben Wilson, Rich Salz, dev-secur...@mozilla.org
No, limiting issuance to ACME and requiring ARI support would not really give us 100% automation adoption in a meaningful way. It would mean that the CAs just handled API traffic, but ultimately there remain a large number of TLS-performing systems that will simply never get updated to have direct ACME support. Instead, each time ARI twigs a renewal, The Cert Person will still have to pull the new cert from /var/certbot/renewal-inbox and go jam it in the 2010-vintage LB or the partner systems that have it pinned for dumb-to-us reasons or whatever else. Declaring victory on cert management automation (and what we really want, which is practical certificate agility) simply because all subscriber<->CA interaction was mediated by an API would be quite naive IMO.

Mik

Jeremy Rowley

unread,
Dec 19, 2024, 1:23:39 PM12/19/24
to Mike Shaver, Andrew Chen, Ben Wilson, Rich Salz, dev-secur...@mozilla.org
Yeah - that was some hyperbole, but you'll get broader adoption of automation with short-lived certificates and requiring ACME (or some automation in general) rather than randomly breaking sites. The general public will not know this a random sampling of revocations and this plan re-inserts the education part of the plan, which does not work. For random revocations to promote faster replacement, the public would need to know a) that the revocations are happening randomly and b) know that they need automation to help deal with the random revocations. If education hasn't worked to date, adding a layer of complexity won't help much. 

 

Jeremy Rowley

unread,
Dec 19, 2024, 1:40:28 PM12/19/24
to Mike Shaver, Andrew Chen, Ben Wilson, Rich Salz, dev-secur...@mozilla.org

> 2. If certificate consumers want to prohibit delayed revocation, then they
> need to make it clear to CAs that they won't accept it and that they will
> kick them out of the root stores if they still do it. Don't try to solve
> this issue with indirect measures like random revocations. Just be straight
> about it and make it clear that there will be consequences for the very
> first delayed revocation and onward.

> I do believe that clearly spelled-out consequences for non-compliance
> would go some way to encouraging good behaviour.  The current approach,
>  where non-compliance may or may not have any repercussions for the CA,
>  doesn't help those within the CA who are fighting the good fight to push
>  back against terrible ideas.  "It might lead to bad things happening" is
>  a very weak argument compared to "if we do this, Mozilla will definitely
> remove us from their trust store".


+1000 to this. The current framework is to let the CAs guess what will happen if they delay revocation or violate the BRs. When you talk to subscribers with the browsers, there is a weird tension from both sides to have the CA allow a delay or allow a violation but then accept responsibility. Having been in the middle of "exceptional circumstances" before, I can tell you that knowing whether a violation is a distrust-worthy event or just a bad idea is difficult. Understanding clear consequences will make every internal discussion and external discussion far easier. 


> 3. We will face big problems in revoking productive customer certificates
> just to test our mass-revocation plan and procedures. Our current customer
> contracts do not foresee this. While we can revoke at any time for security
> or compliance reasons,

> What is "Mozilla makes us revoke 30 randomly-chosen certificates,
>   RNJesus has decided that today's your turn in the barrel" if not a
>   compliance reason?


I agree with Matt - it is a compliance-based revocation. Plus, Section 4.9.1.1 states: "Revocation is required by the CA’s Certificate Policy and/or Certification Practice Statement for a reason that is not otherwise required to be specified by this section 4.9.1.1 (CRLReason “unspecified (0)” which results in no reasonCode extension being provided in the CRL);". Just add the revocation reason of "random sampling" to the CPS and the problem is solved.

> 5. Why was the number 30 chosen as a sample?  Some CA operators issue very
> few certificates, while some CAs issue millions of certificates.

> I was considering responding to this part of the original proposal, to
>  suggest "30 or 0.N% of annual issuance volume, whichever is lower" type
>  of language, but when I thought about it further, if an entire CA is
>  issuing so few certificates that revoking 30 is an unreasonable burden,
>  I'd be very concerned about their operational practices in general,
>  given how little practice they get with them.

>  On the subject of the selection of certificates, I'd like to echo the
>  other comments expressing concerns about the degree to which CAs might
>  seek to "game" the random selection.  It's one of those "it's too easy
>  to do, and too hard to get caught" situations where shenanigans (or even
>  the appearance of shenanigans) like to live.

I agree that if a CA isn't issuing 30 certificates a year, they probably shouldn't be a publicly trusted CA for TLS.  At that point, the CA has limited value to the Mozilla community compared to the risks of mis-issuance (imo). However, 30 does seem awfully low considering the volume of issuance for some CAs. With only 30 certificates, some CAs will revoke only certs from 1 large client, even if the sample is randomly pulled (for example, Cloudflare is a huge number of certificates for whomever has that agreement). With 30 certificates, you won't get the sampling you hope.  

Rich Salz

unread,
Dec 19, 2024, 3:24:27 PM12/19/24
to Ben Wilson, dev-secur...@mozilla.org

I think it would be advisable for a CA operator’s mass-revocation testing plan to include an immediate notice to all customers whose certificates were randomly selected because we would want to minimize disruption to server operations while testing the CA’s ability to revoke and replace certificates promptly.

That's not quite the question I was asking.  I said "pre-notify". Imagine a timeline like this:
  N pick enough certs randomly. Generate replacement certs for those being revoked.
  N+1 notify those customers they will be revoked ("this is a test of the emergency broadcasting system" as it were) and that you have replacement certs
  N + 1 + x Do the revocation

Would that be valid? If not, then as a reasonably large subscriber, I think Akamai would expect to have a cert in the mass-revocation plan, and if we have to respond at incident speed so that our customers are not impacted by such a test, we would probably take that into consideration about which CAs we use.

Ben Wilson

unread,
Dec 19, 2024, 4:28:22 PM12/19/24
to Rich Salz, dev-secur...@mozilla.org
Rich,
That option should be considered.
Thanks,
Ben


Matt Palmer

unread,
Dec 19, 2024, 6:52:52 PM12/19/24
to dev-secur...@mozilla.org
On Thu, Dec 19, 2024 at 08:44:54AM -0800, 'Andrew Chen' via dev-secur...@mozilla.org wrote:
> Mike -- that's a reasonable point. I suppose making it a date OR percentage
> of ARI adoption isn't likely to change things, either. It would be nice if
> we could incentivize ARI adoption and make random revocation a non-event
> for many? most? people.

CAs could incentivize ARI adoption by randomly selecting certificates
and revoking them, with five days advanced notice provided via ARI.

- Matt

Matt Palmer

unread,
Dec 19, 2024, 6:59:44 PM12/19/24
to dev-secur...@mozilla.org
On Thu, Dec 19, 2024 at 11:07:22AM -0700, Jeremy Rowley wrote:
> I agree that educating subscribers has been largely ineffective. However,
> randomly causing outages won't solve the issue.

Oh, I don't know... ransomware has been far more effective at improving
DR practices than several decades of education was.

> automation. Only allow CAs to deliver certificates via an automated
> solution and all of a sudden you have 100% automation adoption.

You'd have 100% *issuance* automation adoption, but not 100% *lifecycle*
automation adoption. The evidence I've collected
(https://www.hezmatt.org/~mpalmer/blog/2024/01/30/why-certificate-automation-matters.html)
suggests to me that some fraction of people who use ACME for certificate
issuance are still manually handling at least some part of the
certificate lifecycle, and it's the whole lifecycle that matters when
determining whether prompt certificate replacement is feasible, not just
issuance.

- Matt

Rich Salz

unread,
Dec 19, 2024, 8:24:24 PM12/19/24
to Matt Palmer, dev-secur...@mozilla.org


On Thu, Dec 19, 2024, 6:59 PM Matt Palmer <mpa...@hezmatt.org> wrote:
Oh, I don't know... ransomware has been far more effective at improving
DR practices than several decades of education was.

An important difference is that I am a customer of the CA, and often paying for a service. I'm not paying you to DoS me.

In the venues where I mainly work( the ietf) we often say that revocation does not work on the web. Which part of the Web PKI needs to become more responsive? My hunch is that it is not the CAs.

Dimitris Zacharopoulos

unread,
Dec 23, 2024, 1:32:13 AM12/23/24
to Rich Salz, Ben Wilson, dev-secur...@mozilla.org
Hi Rich,

The CA cannot issue a replacement certificate if the Domain and/or Identity Validation reuse period has expired.

Some CAs even choose to request fresh domain validations at every issuance.

DZ.

Dec 19, 2024 22:24:31 Rich Salz <rich...@gmail.com>:

--
You received this message because you are subscribed to the Google Groups "dev-secur...@mozilla.org" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dev-security-po...@mozilla.org.

Rich Salz

unread,
Dec 23, 2024, 8:30:45 AM12/23/24
to Dimitris Zacharopoulos, Ben Wilson, dev-secur...@mozilla.org
Yes i know. The key point is to not DoS customers, so do what you have to make sure they have certs before the revocation experiment.

Mike Shaver

unread,
Dec 23, 2024, 11:20:56 AM12/23/24
to Rich Salz, Matt Palmer, dev-secur...@mozilla.org
There are certainly CAs who could improve the overall state of the Web PKI by being more responsive to their commitments. We had an example this year of a CA sitting on revocation of invalid certs for more than a month because the vendor of their CA-management software said it would take that long to get the fix to their issuance (rather than directing their subscribers to get certs from another CA who could correctly issue certificates at that time—because it would cause inconvenience for the subscribers).

However, in many delayed revocation cases, it is indeed the Subscriber who needs to be more responsive to their (legally binding, as required per the BRs) commitments to tolerate prompt revocation. CAs in these circumstances are pressured to violate their covenants under the BRs and related policies, and defer revocation to accommodate Subscriber convenience (both immediate and historical, in the form of underinvestment in certificate management or automation). A random revocation test is really a test of the *Subscriber* population, and how they are influenced by the policies and (more importantly) actions of their chosen CA.

To emphasize, as a customer of the CA operating under the TLS BRs, you have legally agreed that the CA can blow away the cert in 24/120 hours and that you will tolerate that without unacceptable-to-the-world-and-the-web consequences. That legal commitment is often not reflected operationally, unfortunately, and the desired result of a random revocation audit is to smoke out those cases, and if that results in some operational distress for underprepared Subscribers, then hopefully their distress is helpful /pour encourager les autres/ as well as motivating the Subscribers in question to improve their operations.

IMO the random revocation should take exactly the same operational form as a real revocation under the 5-day clock: CA is notified that they need to revoke, identically to a CPR being filed, and then they inform the subscriber and the clock has started. When the clock expires, the certificate is revoked. I don't think that the process of issuing the replacement certificate is typically the long pole, and rather it's available as soon as the Subscriber asks for it. If the DCV is still in its validity period then the CA could proactively reissue at the point of the CPR arriving, if that proves to be a meaningful assistance to the Subscriber in replacing things.

Mike

--
You received this message because you are subscribed to the Google Groups "dev-secur...@mozilla.org" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dev-security-po...@mozilla.org.

Watson Ladd

unread,
Dec 23, 2024, 12:32:56 PM12/23/24
to Rich Salz, Matt Palmer, dev-secur...@mozilla.org
Revocation doesn't work for non-browser clients. Thanks to OneCRL it
does work decently well for browsers.

>
> --
> You received this message because you are subscribed to the Google Groups "dev-secur...@mozilla.org" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to dev-security-po...@mozilla.org.
> To view this discussion visit https://groups.google.com/a/mozilla.org/d/msgid/dev-security-policy/CAFH29trsULk%3D%2BTjyMkZBkr716vkXomrq0mOD0_Tpb8TMLF7-AQ%40mail.gmail.com.



--
Astra mortemque praestare gradatim

Dimitris Zacharopoulos

unread,
Dec 23, 2024, 12:54:54 PM12/23/24
to Rich Salz, Ben Wilson, dev-secur...@mozilla.org
FWIW, I don't like the approach of random revocations. I would prefer to focus on re-issuance and certificate rollover exercises by each CA, providing statistics of how subscribers perform, without any real disruptions. Instead of selecting certificates for revocation, select certificates for re-issuance and rollover in 5 days.

DZ.

Dec 23, 2024 13:30:45 Rich Salz <rich...@gmail.com>:

Matt Palmer

unread,
Dec 23, 2024, 4:52:21 PM12/23/24
to dev-secur...@mozilla.org
On Mon, Dec 23, 2024 at 05:54:43PM +0000, 'Dimitris Zacharopoulos' via dev-secur...@mozilla.org wrote:
> FWIW, I don't like the approach of random revocations. I would prefer
> to focus on re-issuance and certificate rollover exercises by each CA,
> providing statistics of how subscribers perform, without any real
> disruptions. Instead of selecting certificates for revocation, select
> certificates for re-issuance and rollover in 5 days.

Except that there is no way to objectively measure that rollover has
actually taken place.

- Matt

Dimitris Zacharopoulos

unread,
Dec 24, 2024, 2:56:11 AM12/24/24
to Matt Palmer, dev-secur...@mozilla.org
True, but it's a start. You could objectively measure only the replacement certificates for publicly accessible websites, not the ones blocked by firewalls.

Dimitris.

Dec 23, 2024 21:52:24 Matt Palmer <mpa...@hezmatt.org>:
> --
> You received this message because you are subscribed to the Google Groups "dev-secur...@mozilla.org" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to dev-security-po...@mozilla.org.
> To view this discussion visit https://groups.google.com/a/mozilla.org/d/msgid/dev-security-policy/a0c795c8-7f1e-4ff6-a5ee-8ae8c3e95d84%40mtasv.net.

Mike Shaver

unread,
Dec 24, 2024, 9:43:37 AM12/24/24
to Dimitris Zacharopoulos, Matt Palmer, dev-secur...@mozilla.org
How is a root program or observer to know all the servers that should be checked for a given certificate, given the wide prevalence of load balancing?

We know that subscribers will (successfully) pressure CAs to extend revocation timelines even for systems that are locked to a limited peer IP range or are entirely internal like test deployments.

I don’t think it would really meet the goal of the exercise.

Mike

Roman Fischer

unread,
Jan 3, 2025, 4:49:59 AMJan 3
to dev-secur...@mozilla.org

I think the problem of delayed revocation is automatically solved once the proposed clarification by Mozilla (and other root store operators) clearly states that delayed revocation will unconditionally to removal from the trust store. Why invent a whole new mechanism when the same goal can be achieved much easier?

 

Kind regards & welcome to 2025

Roman

Dimitris Zacharopoulos

unread,
Jan 3, 2025, 5:09:16 AMJan 3
to dev-secur...@mozilla.org


On 1/3/2025 11:49 AM, Roman Fischer wrote:

I think the problem of delayed revocation is automatically solved once the proposed clarification by Mozilla (and other root store operators) clearly states that delayed revocation will unconditionally to removal from the trust store. Why invent a whole new mechanism when the same goal can be achieved much easier?


Because in some cases it can have a huge impact that puts the stability of the entire WebPKI ecosystem at risk, causing economic and social impact at a Global scale. What would happen if Let's Encrypt revoked all their Certificates in https://bugzilla.mozilla.org/show_bug.cgi?id=1715455?


Best regards,
Dimitris.

Roman Fischer

unread,
Jan 3, 2025, 5:27:52 AMJan 3