MRSP 3.0: Issue #276: Delayed Revocation

[Ben Wilson]

All,

The purpose of this email is to start discussion of Mozilla GitHub Issue #276 ("Address Delayed Revocation"). We would like to collect comments and feedback on a proposal to address delayed certificate revocation from a Mozilla perspective. It builds on prior discussions and feedback regarding delayed revocation, and the proposal is meant to replace guidance currently provided on the Mozilla CA wiki.

Here is the comparison link for a proposed new section 6.1.3 in the MRSP:

https://github.com/mozilla/pkipolicy/compare/51b2f702accd54cb70d52081a9e814298433495b%E2%80%A6efa8ac40ac341fb813620938ef72328a53858038

Summary

Here are the highlights of the proposal:

• Revocation must occur promptly in compliance with the timelines set in section 4.9.1 of the TLS Baseline Requirements (TLS BRs). Mozilla does not grant exceptions to these timelines.
• New CA Obligations:
• Educate subscribers on revocation timelines and discourage reliance on certificates in systems that cannot tolerate timely revocation.
• Include contractual language requiring subscriber cooperation with revocation timelines.
• Maintain and test mass revocation plans annually, including the revocation of 30 randomly chosen certificates within a 5-day period.
• Beginning April 15, 2026, CA audit reports must attest to compliance with the mass revocation planning requirements.
• Delayed revocation incidents must be reported per version 2.1 of the CCADB's Incident Reporting Guidelines (as currently proposed)
• Repeated delayed revocation incidents will result in heightened scrutiny or sanctions, which may include root removal.

Background

Earlier this year, on this list, I proposed an Interim Policy to Address Delayed Revocation. While the proposed interim policy provided clarity, it faced criticism regarding implementation complexity, burden on subscribers and CAs, and the feasibility of associated measures, such as transitioning delayed revocation domains to 90-day certificates. Also, there were subsequent proposals aimed at reducing certificate lifetimes and encouraging automation. See e.g. https://github.com/cabforum/servercert/pull/553.

This new proposal drops proposed measures such as domain-specific tracking and subscriber attestations and instead focuses on subscriber education, mass revocation preparedness, and robust incident reporting as the primary mechanisms for improving agility and transparency regarding delayed revocation.

If adopted, the proposed MRSP § 6.1.3 would replace the current guidance on delayed revocation in Mozilla’s wiki and ensure consistency with the CCADB's Incident Reporting Guidelines.

I welcome your feedback on this draft proposal. Please share your thoughts, questions, or concerns to help us refine and improve it further.

Thanks,

Ben Wilson

Mozilla Root Store

[Pedro Fuentes]

Hi Ben,

About " annual plan testing by revoking 30 randomly chosen certificates within a 5-day timeframe; and"...

We understand that this mean that a CA will need to randomly revoke 30 certificates that most likely don't have other reason for being revoked than being randomly chosen and customers will just need to "happily" accept the situation... Harsh, but doable...

About "audit report submitted under section 3.1 SHALL include an attestation that the CA operator has met these mass revocation planning requirements", it must be considered that the attestation letters of Webtrust audit reports have a fixed format, so such addition would be added most likely as a "Other matters" section, that audits can take each differently.

My question here would be if you think it's there any chance that these requirements become part of the TLS BRs instead of the Mozilla Policy, I see several benefits here:

- Checking the mass-revocation plan would be integral part of the audit scope, so auditors don't need to figure out how to include it in the reports... it just needs to be added to the audit criteria.

- The "inverse-lottery" thing could be added in the revocation timelines of the BR, so there's an entry in the 5-day deadline adding a new category "The certificate has been randomly chosen for revocation during an internal audit". This should facilitate the contractual language to add in the subscriber agreement.

[Ben Wilson]

All,

As part of the discussions on this proposal, namely that CAs “maintain and test mass revocation plans annually, including the revocation of 30 randomly chosen certificates within a 5-day period,” I’ve received a few comments via private channels, and to ensure transparency and foster discussion, I am sharing them here anonymously:

1. “Mozilla does not grant exceptions…” -- this is the most important signal that Mozilla can provide.

2. If certificate consumers want to prohibit delayed revocation, then they need to make it clear to CAs that they won't accept it and that they will kick them out of the root stores if they still do it. Don't try to solve this issue with indirect measures like random revocations. Just be straight about it and make it clear that there will be consequences for the very first delayed revocation and onward.

3. We will face big problems in revoking productive customer certificates just to test our mass-revocation plan and procedures. Our current customer contracts do not foresee this. While we can revoke at any time for security or compliance reasons, this authorization should not be used just to test mass-revocation. This will also require us to push out contract changes to our complete TLS customer base, which will take a considerable amount of time and effort.

4. This part of the proposal should occur within the CA/Browser Forum through amendments to the TLS Baseline Requirements, and not via Mozilla Root Store Policy.

5. Why was the number 30 chosen as a sample?  Some CA operators issue very few certificates, while some CAs issue millions of certificates.

I welcome your feedback on these points, the random sampling proposal, and any others.

Thanks,

Ben

[Mike Shaver]

Thanks for this summary, Ben.

WRT point 3, I think a good way to roll out randomly-sampled certificate revocation is as an amendment to the CPS of all CAs; CAs amend their CPSes all the time without major contractual upheaval, so it seems like a useful tool here. Coupled with an appropriate effective-after date, we could be sure that all issued certificates were governed by a CPS version that includes random revocation as audit mechanism.

WRT the random selection itself, I would like to see them chosen via an externally-transparent process rather than leaving it to CAs to be appropriately random. Whether by Mozilla or a CCADB process, something that samples from crt.sh in an appropriate way, with a public randomness seed, would give higher confidence in the process. CAs will otherwise have a strong incentive to “roll the dice again” if the random selection includes a major or stubborn customer, and those are the cases that most need to be tested IMO.

I think for all of these proposals, it’s important to state what the consequences of violation will be.

I don’t think that Mozilla should feel constrained to only have policies that exactly match the TLS BRs. They are *baseline* requirements, not exhaustive requirements, and each cert consumer has its own context in which to make tradeoffs between inclusiveness and rigour.

More broadly, it would be good to state what the main purpose and values of the MRSP are: for example, is the goal to permit as many CAs as can meet the requirements to encourage broad competition, or to optimize for the security of Mozilla’s users (which might mean having fewer CAs for which oversight by Mozilla can be more thorough)?

Mike

--
You received this message because you are subscribed to the Google Groups "dev-secur...@mozilla.org" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dev-security-po...@mozilla.org.
To view this discussion visit https://groups.google.com/a/mozilla.org/d/msgid/dev-security-policy/71b07640-d425-4f2f-8da4-d97a9475b9f6n%40mozilla.org.

[Jeremy Rowley]

I agree with point number 2. I think Mozilla should just make it clear delayed revocations are not permitted and call it good. This is the only industry I know where we'd propose breaking availability in the CIA triangle just in case integrity might be compromised. Better to pull off the band-aid and, assuming we really like 5 day revocations, either a) require that companies use ACME to deploy certificates or b) simply state that CAs delaying revocation past 5 days will likely be distrusted. 

[Wayne]

Thanks for the breakdown of responses so far Ben.

Fully agreed on point 1. Partially agree with point 2 but there are benefits for proving that the revocation system functions in practice and having a random revocation cycle creates better habits in the consumer space.

Point 3 is unusual as revocation for compliance reasons is explicitly outlined, and that's all that this is? There isn't any new changes to the legal language required that I can see.

Point 4 has been covered, being above the baseline requirements is barely noteworthy but encouraging these practices in more programs is good advice.

Point 5 is a matter of scale, in prior discussions % of total active issuance were put forward as ideas. Would the CAs handling 100s of certificates per year be the ones to push the idea of it being 10% of all issuance? I'm sure the CAs can decide on which matter would suit their practices best as long as the figure is above 0. It is ultimately arbitrary unless we're moving to a % where we can be assured of coverage across a given number of years.

I'm of mixed minds as to Jeremy's comment though. All regulatory practices I'm aware of propose that availability will be broken when integrity is at risk, it's also fundamental to the WebPKI as an ecosystem. Putting that aside, I would consider a CA exception to the revocation audit minimums if short-lived certificates were all that were issued. Same substance for revocation exemption in general as per the BRs.

- Wayne

[Rich Salz]

As part of the discussions on this proposal, namely that CAs “maintain and test mass revocation plans annually, including the revocation of 30 randomly chosen certificates within a 5-day period,” I’ve received a few comments via private channels, and to ensure transparency and foster discussion, I am sharing them here anonymously:

Would a CA be allowed to pre-notify customers whose certs were randomly selected and {pre/re}-issue them replacements?

[Ben Wilson]

Dear Rich,

Thank you for your question.

I think it would be advisable for a CA operator’s mass-revocation testing plan to include an immediate notice to all customers whose certificates were randomly selected because we would want to minimize disruption to server operations while testing the CA’s ability to revoke and replace certificates promptly.

That said, CAs should consider performing occasional tests that go beyond providing pre-generated replacement certificates, in which subscribers generate and submit new public keys. That would address the risks from a widespread incident, like Heartbleed, where the potential compromise of private keys necessitated key pair replacement. Preparing for such scenarios ensures that subscribers will be able to quickly perform the tasks of key pair generation, public key submission, and certificate installation.

Much to discuss.

Thanks again,

Ben

[Matt Palmer]

On Wed, Dec 18, 2024 at 09:17:45AM -0800, 'Ben Wilson' via dev-secur...@mozilla.org wrote:
> As part of the discussions on this proposal, namely that CAs “maintain and
> test mass revocation plans annually, including the revocation of 30
> randomly chosen certificates within a 5-day period,” I’ve received a few
> comments via private channels, and to ensure transparency and foster
> discussion, I am sharing them here anonymously:
>
> 1. “Mozilla does not grant exceptions…” -- this is the most important
> signal that Mozilla can provide.

This has been fairly unambiguous from my perspective, and yet it
appears that this wording has been open to misinterpretation.

> 2. If certificate consumers want to prohibit delayed revocation, then they
> need to make it clear to CAs that they won't accept it and that they will
> kick them out of the root stores if they still do it. Don't try to solve
> this issue with indirect measures like random revocations. Just be straight
> about it and make it clear that there will be consequences for the very
> first delayed revocation and onward.

I do believe that clearly spelled-out consequences for non-compliance
would go some way to encouraging good behaviour. The current approach,
where non-compliance may or may not have any repercussions for the CA,
doesn't help those within the CA who are fighting the good fight to push
back against terrible ideas. "It might lead to bad things happening" is
a very weak argument compared to "if we do this, Mozilla will definitely
remove us from their trust store".

> 3. We will face big problems in revoking productive customer certificates
> just to test our mass-revocation plan and procedures. Our current customer
> contracts do not foresee this. While we can revoke at any time for security
> or compliance reasons,

What is "Mozilla makes us revoke 30 randomly-chosen certificates,
RNJesus has decided that today's your turn in the barrel" if not a
compliance reason?

> 4. This part of the proposal should occur within the CA/Browser Forum
> through amendments to the TLS Baseline Requirements, and not via Mozilla
> Root Store Policy.

... so that all the CAs can vote against it and kill it. As the name
suggests, *Baseline* Requirements are the very lowest common denominator
that all parties are willing to accept; hog-tying one trust store that
wishes to innovate is completely unacceptable.

> 5. Why was the number 30 chosen as a sample? Some CA operators issue very
> few certificates, while some CAs issue millions of certificates.

I was considering responding to this part of the original proposal, to
suggest "30 or 0.N% of annual issuance volume, whichever is lower" type
of language, but when I thought about it further, if an entire CA is
issuing so few certificates that revoking 30 is an unreasonable burden,
I'd be very concerned about their operational practices in general,
given how little practice they get with them.

On the subject of the selection of certificates, I'd like to echo the
other comments expressing concerns about the degree to which CAs might
seek to "game" the random selection. It's one of those "it's too easy
to do, and too hard to get caught" situations where shenanigans (or even
the appearance of shenanigans) like to live.

- Matt

[Roman Fischer]

Dear everybody,

 

>Point 3 is unusual as revocation for compliance reasons is explicitly outlined, and that's all that this is? There isn't any new changes to the legal language required that I can see.

 

I checked with our legal department and they clearly stated that we will have to amend the contracts if we want to be sure that we can't be sued for damages inflicted by random revocation. Current language (at least in our contracts) allows us to revoke in case of security incidents, non-compliance and some other cases. But a preventative revocation would currently not be covered.

 

Also, I'm not convinced that it will motivate many customers towards automation. Most will probably just gamble and bet on not being in the unlucky 30, especially if they get their certs from a large CA... 😉

 

 

>Point 4 has been covered, being above the baseline requirements is barely noteworthy but encouraging these practices in more programs is good advice.

 

Regarding a comment made in another mail, I think it's fair to say that most public trust CAs -need- to conform to all major root store policies to be of any value for their customers. The more fragmented these root store policies are, the more complex (and error prone!) complying to them becomes. For this reason, we would really encourage the root store operators to converge on requirements set forward.

 

Kind regards
Roman

--

You received this message because you are subscribed to the Google Groups "dev-secur...@mozilla.org" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dev-security-po...@mozilla.org.

To view this discussion visit https://groups.google.com/a/mozilla.org/d/msgid/dev-security-policy/c2290409-2ef1-4295-8f5d-13369f2dbbe1n%40mozilla.org.

[Andrew Chen]

Ben -- thanks for the proposal. Random revocation is likely to be painful for subscribers without automation around eminent revocation, which ARI is expected to address. Would it make sense to couple this proposal to a specific adoption rate of ARI amongst subscribers?

Andrew

--

You received this message because you are subscribed to the Google Groups "dev-secur...@mozilla.org" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dev-security-po...@mozilla.org.

To view this discussion visit https://groups.google.com/a/mozilla.org/d/msgid/dev-security-policy/CA%2B1gtaZjqvpanXnx-eX7_%3D3E3jE5isKW1h4cD9-_jCmmBS8DCQ%40mail.gmail.com.

[Mike Shaver]

Do you mean that CAs could avoid having to do random revocation by discouraging ARI adoption among their subscribers?

The point is that right now revocation is so painful that it’s causing CAs to side with subscriber convenience over the integrity of the web PKI. Sampled, controlled revocations let us identify points of pain before they have security implications, and motivate Subscribers to prepare their systems—whether through automation or not, up to them, I’m not their dad—to tolerate on-time revocation. We care about the likely outcomes of automation, such as tolerance of short revocation or expiry timelines, really, but if BigSlowCo wants to staff a 24-hour cert maintenance squad such that they don’t (successfully) pressure their CA into blowing revocation deadlines, that’s their opex choice. Directly evaluating ecosystem capability around prompt revocation is the only way I can think of to identify areas of danger or weakness before they become issues for the web.

Mike

To view this discussion visit https://groups.google.com/a/mozilla.org/d/msgid/dev-security-policy/CAPdVL6UdzMNOC7DKX1De5VpYh604ge6u7cCBwqMSSphCChs1Kg%40mail.gmail.com.

[Andrew Chen]

Mike -- that's a reasonable point. I suppose making it a date OR percentage of ARI adoption isn't likely to change things, either. It would be nice if we could incentivize ARI adoption and make random revocation a non-event for many? most? people.

Andrew

[Mike Shaver]

On Thu, Dec 19, 2024 at 10:43 AM Roman Fischer <roman....@swisssign.com> wrote:

Dear everybody,

 

>Point 3 is unusual as revocation for compliance reasons is explicitly outlined, and that's all that this is? There isn't any new changes to the legal language required that I can see.

 

I checked with our legal department and they clearly stated that we will have to amend the contracts if we want to be sure that we can't be sued for damages inflicted by random revocation. Current language (at least in our contracts) allows us to revoke in case of security incidents, non-compliance and some other cases. But a preventative revocation would currently not be covered.

A surprisingly direct statement from inside counsel, given that the detailed language of the policy component hasn’t been determined! But of course as we know you can get sued for anything, and the question is whether you eventually prevail in court. We saw with the unfortunate DigiCert situation earlier this year that even clear contract language around revocation promptness can’t protect against misleading filings or overly-sympathetic judges, at least in terms of initial injunctions and bringing a case that’s not trivially dismissed.

If a CPS update is not sufficient—which would confuse me, but that’s not impossible!—your legal department would have ~18 months to roll out some (likely trivial) new language and get it adopted during renewals or through other means. I have confidence in them!

An alternative, of course, is that Mozilla or whoever just adds N sampled certs to OneCRL with 5 days notice to the CA. Given that, there’s nothing that a CA needs to do other than inform the subscriber and reissue when asked, but that only really tests Subscriber capability and not CA commitment to compliance. I think we have seen an unfortunate abundance of evidence that both are areas of concern around prompt revocation, not that they necessarily need to be addressed by the same mechanism.

Also, I'm not convinced that it will motivate many customers towards automation. Most will probably just gamble and bet on not being in the unlucky 30, especially if they get their certs from a large CA...

Yes, this is a clear weakness of the fixed-30 element of the proposal, but that’s an argument for proportionate sampling, not for avoiding the practice entirely.

I have a hard time reconciling “most will probably ignore this context” with “it would be too disruptive for subscribers”, I admit.

Regarding a comment made in another mail, I think it's fair to say that most public trust CAs -need- to conform to all major root store policies to be of any value for their customers. The more fragmented these root store policies are, the more complex (and error prone!) complying to them becomes. For this reason, we would really encourage the root store operators to converge on requirements set forward.

A requirement is no harder to meet because it’s from one root store than if it applies to multiple, surely. What is the source of additional error risk that goes away when something is more widely adopted by root programs? Does it become easier to participate in CT if that requirement from Chrome’s program is adopted by others?

Are you saying that it’s too hard for a CA to analyze a handful of different root store policies and determine what the requirements *are*? Within the limits of reasonableness, that’s a job for the quality of language in the program descriptions; this is an area where there could be some improvements, which is part of why Ben is driving this refresh of the MSRP I think.

Independent of that, I think it would be great for more root stores to join in this new practice, and I would like advocate to them for exactly that. That’s a separate question from whether it needs to be in the BRs via the CABF. I think it can be effective without inclusion in the BRs, just like Chrome’s policy on CT has been effective in driving adoption.

Mike

[Mike Shaver]

Pressure that causes more frequent cert issuance and installation is pretty much the only thing that will incentivize adoption of things that make issuance and installation easier, I think. Certainly many CAs, per their incident Action Items, have been “educating subscribers” about the merits of automation for some time, with what I would say are unsatisfactory results.

Mike

[Jeremy Rowley]

I agree that educating subscribers has been largely ineffective. However, randomly causing outages won't solve the issue . Random revocations will just make people hate the browsers and CAs more. It's a strange solution to the objective of faster certificate replacement when short-lived certificates force automation as does simply requiring CAs to mandate automation. Only allow CAs to deliver certificates via an automated solution and all of a sudden you have 100% automation adoption.

To view this discussion visit https://groups.google.com/a/mozilla.org/d/msgid/dev-security-policy/CADQzZqvuqw_odOeona27nmKpn%3DSFhKGrOT24Tkt0XvcG3apuyg%40mail.gmail.com.

[Mike Shaver]

No, limiting issuance to ACME and requiring ARI support would not really give us 100% automation adoption in a meaningful way. It would mean that the CAs just handled API traffic, but ultimately there remain a large number of TLS-performing systems that will simply never get updated to have direct ACME support. Instead, each time ARI twigs a renewal, The Cert Person will still have to pull the new cert from /var/certbot/renewal-inbox and go jam it in the 2010-vintage LB or the partner systems that have it pinned for dumb-to-us reasons or whatever else. Declaring victory on cert management automation (and what we really want, which is practical certificate agility) simply because all subscriber<->CA interaction was mediated by an API would be quite naive IMO.

Mik

[Jeremy Rowley]

Yeah - that was some hyperbole, but you'll get broader adoption of automation with short-lived certificates and requiring ACME (or some automation in general) rather than randomly breaking sites. The general public will not know this a random sampling of revocations and this plan re-inserts the education part of the plan, which does not work. For random revocations to promote faster replacement, the public would need to know a) that the revocations are happening randomly and b) know that they need automation to help deal with the random revocations. If education hasn't worked to date, adding a layer of complexity won't help much. 

 

[Jeremy Rowley]

> 2. If certificate consumers want to prohibit delayed revocation, then they
> need to make it clear to CAs that they won't accept it and that they will
> kick them out of the root stores if they still do it. Don't try to solve
> this issue with indirect measures like random revocations. Just be straight
> about it and make it clear that there will be consequences for the very
> first delayed revocation and onward.

> I do believe that clearly spelled-out consequences for non-compliance
> would go some way to encouraging good behaviour.  The current approach,
>  where non-compliance may or may not have any repercussions for the CA,
>  doesn't help those within the CA who are fighting the good fight to push
>  back against terrible ideas.  "It might lead to bad things happening" is
>  a very weak argument compared to "if we do this, Mozilla will definitely
> remove us from their trust store".

+1000 to this. The current framework is to let the CAs guess what will happen if they delay revocation or violate the BRs. When you talk to subscribers with the browsers, there is a weird tension from both sides to have the CA allow a delay or allow a violation but then accept responsibility. Having been in the middle of "exceptional circumstances" before, I can tell you that knowing whether a violation is a distrust-worthy event or just a bad idea is difficult. Understanding clear consequences will make every internal discussion and external discussion far easier. 

> 3. We will face big problems in revoking productive customer certificates
> just to test our mass-revocation plan and procedures. Our current customer
> contracts do not foresee this. While we can revoke at any time for security
> or compliance reasons,

> What is "Mozilla makes us revoke 30 randomly-chosen certificates,
>   RNJesus has decided that today's your turn in the barrel" if not a
>   compliance reason?

I agree with Matt - it is a compliance-based revocation. Plus, Section 4.9.1.1 states: "Revocation is required by the CA’s Certificate Policy and/or Certification Practice Statement for a reason that is not otherwise required to be specified by this section 4.9.1.1 (CRLReason “unspecified (0)” which results in no reasonCode extension being provided in the CRL);". Just add the revocation reason of "random sampling" to the CPS and the problem is solved.

> 5. Why was the number 30 chosen as a sample?  Some CA operators issue very
> few certificates, while some CAs issue millions of certificates.

> I was considering responding to this part of the original proposal, to
>  suggest "30 or 0.N% of annual issuance volume, whichever is lower" type
>  of language, but when I thought about it further, if an entire CA is
>  issuing so few certificates that revoking 30 is an unreasonable burden,
>  I'd be very concerned about their operational practices in general,
>  given how little practice they get with them.

>  On the subject of the selection of certificates, I'd like to echo the
>  other comments expressing concerns about the degree to which CAs might
>  seek to "game" the random selection.  It's one of those "it's too easy
>  to do, and too hard to get caught" situations where shenanigans (or even
>  the appearance of shenanigans) like to live.

I agree that if a CA isn't issuing 30 certificates a year, they probably shouldn't be a publicly trusted CA for TLS.  At that point, the CA has limited value to the Mozilla community compared to the risks of mis-issuance (imo). However, 30 does seem awfully low considering the volume of issuance for some CAs. With only 30 certificates, some CAs will revoke only certs from 1 large client, even if the sample is randomly pulled (for example, Cloudflare is a huge number of certificates for whomever has that agreement). With 30 certificates, you won't get the sampling you hope.  

[Rich Salz]

I think it would be advisable for a CA operator’s mass-revocation testing plan to include an immediate notice to all customers whose certificates were randomly selected because we would want to minimize disruption to server operations while testing the CA’s ability to revoke and replace certificates promptly.

That's not quite the question I was asking.  I said "pre-notify". Imagine a timeline like this:

  N pick enough certs randomly. Generate replacement certs for those being revoked.

  N+1 notify those customers they will be revoked ("this is a test of the emergency broadcasting system" as it were) and that you have replacement certs

  N + 1 + x Do the revocation

Would that be valid? If not, then as a reasonably large subscriber, I think Akamai would expect to have a cert in the mass-revocation plan, and if we have to respond at incident speed so that our customers are not impacted by such a test, we would probably take that into consideration about which CAs we use.

[Ben Wilson]

Rich,

That option should be considered.

Thanks,

Ben

[Matt Palmer]

On Thu, Dec 19, 2024 at 08:44:54AM -0800, 'Andrew Chen' via dev-secur...@mozilla.org wrote:
> Mike -- that's a reasonable point. I suppose making it a date OR percentage
> of ARI adoption isn't likely to change things, either. It would be nice if
> we could incentivize ARI adoption and make random revocation a non-event
> for many? most? people.

CAs could incentivize ARI adoption by randomly selecting certificates
and revoking them, with five days advanced notice provided via ARI.

- Matt

[Matt Palmer]

On Thu, Dec 19, 2024 at 11:07:22AM -0700, Jeremy Rowley wrote:
> I agree that educating subscribers has been largely ineffective. However,
> randomly causing outages won't solve the issue.

Oh, I don't know... ransomware has been far more effective at improving
DR practices than several decades of education was.

> automation. Only allow CAs to deliver certificates via an automated
> solution and all of a sudden you have 100% automation adoption.

You'd have 100% *issuance* automation adoption, but not 100% *lifecycle*
automation adoption. The evidence I've collected
(https://www.hezmatt.org/~mpalmer/blog/2024/01/30/why-certificate-automation-matters.html)
suggests to me that some fraction of people who use ACME for certificate
issuance are still manually handling at least some part of the
certificate lifecycle, and it's the whole lifecycle that matters when
determining whether prompt certificate replacement is feasible, not just
issuance.

- Matt

[Rich Salz]

On Thu, Dec 19, 2024, 6:59 PM Matt Palmer <mpa...@hezmatt.org> wrote:

Oh, I don't know... ransomware has been far more effective at improving
DR practices than several decades of education was.

An important difference is that I am a customer of the CA, and often paying for a service. I'm not paying you to DoS me.

In the venues where I mainly work( the ietf) we often say that revocation does not work on the web. Which part of the Web PKI needs to become more responsive? My hunch is that it is not the CAs.

[Dimitris Zacharopoulos]

Hi Rich,

The CA cannot issue a replacement certificate if the Domain and/or Identity Validation reuse period has expired.

Some CAs even choose to request fresh domain validations at every issuance.

DZ.

Dec 19, 2024 22:24:31 Rich Salz <rich...@gmail.com>:

--
You received this message because you are subscribed to the Google Groups "dev-secur...@mozilla.org" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dev-security-po...@mozilla.org.

To view this discussion visit https://groups.google.com/a/mozilla.org/d/msgid/dev-security-policy/CAFH29trVcYUbbWCE722Qu8qECvG%3DBBS9MswpK6%2B3YVQRAhnC2A%40mail.gmail.com.

[Rich Salz]

Yes i know. The key point is to not DoS customers, so do what you have to make sure they have certs before the revocation experiment.

[Mike Shaver]

There are certainly CAs who could improve the overall state of the Web PKI by being more responsive to their commitments. We had an example this year of a CA sitting on revocation of invalid certs for more than a month because the vendor of their CA-management software said it would take that long to get the fix to their issuance (rather than directing their subscribers to get certs from another CA who could correctly issue certificates at that time—because it would cause inconvenience for the subscribers).

However, in many delayed revocation cases, it is indeed the Subscriber who needs to be more responsive to their (legally binding, as required per the BRs) commitments to tolerate prompt revocation. CAs in these circumstances are pressured to violate their covenants under the BRs and related policies, and defer revocation to accommodate Subscriber convenience (both immediate and historical, in the form of underinvestment in certificate management or automation). A random revocation test is really a test of the *Subscriber* population, and how they are influenced by the policies and (more importantly) actions of their chosen CA.

To emphasize, as a customer of the CA operating under the TLS BRs, you have legally agreed that the CA can blow away the cert in 24/120 hours and that you will tolerate that without unacceptable-to-the-world-and-the-web consequences. That legal commitment is often not reflected operationally, unfortunately, and the desired result of a random revocation audit is to smoke out those cases, and if that results in some operational distress for underprepared Subscribers, then hopefully their distress is helpful /pour encourager les autres/ as well as motivating the Subscribers in question to improve their operations.

IMO the random revocation should take exactly the same operational form as a real revocation under the 5-day clock: CA is notified that they need to revoke, identically to a CPR being filed, and then they inform the subscriber and the clock has started. When the clock expires, the certificate is revoked. I don't think that the process of issuing the replacement certificate is typically the long pole, and rather it's available as soon as the Subscriber asks for it. If the DCV is still in its validity period then the CA could proactively reissue at the point of the CPR arriving, if that proves to be a meaningful assistance to the Subscriber in replacing things.

Mike

--

You received this message because you are subscribed to the Google Groups "dev-secur...@mozilla.org" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dev-security-po...@mozilla.org.

To view this discussion visit https://groups.google.com/a/mozilla.org/d/msgid/dev-security-policy/CAFH29trsULk%3D%2BTjyMkZBkr716vkXomrq0mOD0_Tpb8TMLF7-AQ%40mail.gmail.com.

[Watson Ladd]

Revocation doesn't work for non-browser clients. Thanks to OneCRL it
does work decently well for browsers.

>
> --
> You received this message because you are subscribed to the Google Groups "dev-secur...@mozilla.org" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to dev-security-po...@mozilla.org.

> To view this discussion visit https://groups.google.com/a/mozilla.org/d/msgid/dev-security-policy/CAFH29trsULk%3D%2BTjyMkZBkr716vkXomrq0mOD0_Tpb8TMLF7-AQ%40mail.gmail.com.



--
Astra mortemque praestare gradatim

[Dimitris Zacharopoulos]

FWIW, I don't like the approach of random revocations. I would prefer to focus on re-issuance and certificate rollover exercises by each CA, providing statistics of how subscribers perform, without any real disruptions. Instead of selecting certificates for revocation, select certificates for re-issuance and rollover in 5 days.

DZ.

Dec 23, 2024 13:30:45 Rich Salz <rich...@gmail.com>:

[Matt Palmer]

On Mon, Dec 23, 2024 at 05:54:43PM +0000, 'Dimitris Zacharopoulos' via dev-secur...@mozilla.org wrote:
> FWIW, I don't like the approach of random revocations. I would prefer
> to focus on re-issuance and certificate rollover exercises by each CA,
> providing statistics of how subscribers perform, without any real
> disruptions. Instead of selecting certificates for revocation, select
> certificates for re-issuance and rollover in 5 days.

Except that there is no way to objectively measure that rollover has
actually taken place.

- Matt

[Dimitris Zacharopoulos]

True, but it's a start. You could objectively measure only the replacement certificates for publicly accessible websites, not the ones blocked by firewalls.

Dimitris.

Dec 23, 2024 21:52:24 Matt Palmer <mpa...@hezmatt.org>:

> --
> You received this message because you are subscribed to the Google Groups "dev-secur...@mozilla.org" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to dev-security-po...@mozilla.org.

> To view this discussion visit https://groups.google.com/a/mozilla.org/d/msgid/dev-security-policy/a0c795c8-7f1e-4ff6-a5ee-8ae8c3e95d84%40mtasv.net.

[Mike Shaver]

How is a root program or observer to know all the servers that should be checked for a given certificate, given the wide prevalence of load balancing?

We know that subscribers will (successfully) pressure CAs to extend revocation timelines even for systems that are locked to a limited peer IP range or are entirely internal like test deployments.

I don’t think it would really meet the goal of the exercise.

Mike

To view this discussion visit https://groups.google.com/a/mozilla.org/d/msgid/dev-security-policy/4072d67e-6e30-4c01-a18b-993b94102a41%40it.auth.gr.

[Roman Fischer]

I think the problem of delayed revocation is automatically solved once the proposed clarification by Mozilla (and other root store operators) clearly states that delayed revocation will unconditionally to removal from the trust store. Why invent a whole new mechanism when the same goal can be achieved much easier?

 

Kind regards & welcome to 2025

Roman

To view this discussion visit https://groups.google.com/a/mozilla.org/d/msgid/dev-security-policy/CADQzZqtjanuDV1GEMhXzuEFG%2BnaC5ZVKfDgchYX4UfuxRtpwFA%40mail.gmail.com.

[Dimitris Zacharopoulos]

On 1/3/2025 11:49 AM, Roman Fischer wrote:

I think the problem of delayed revocation is automatically solved once the proposed clarification by Mozilla (and other root store operators) clearly states that delayed revocation will unconditionally to removal from the trust store. Why invent a whole new mechanism when the same goal can be achieved much easier?

Because in some cases it can have a huge impact that puts the stability of the entire WebPKI ecosystem at risk, causing economic and social impact at a Global scale. What would happen if Let's Encrypt revoked all their Certificates in https://bugzilla.mozilla.org/show_bug.cgi?id=1715455?


Best regards,
Dimitris.

To view this discussion visit https://groups.google.com/a/mozilla.org/d/msgid/dev-security-policy/ZR0P278MB017059F4DBB48A4B20BEA95AFA152%40ZR0P278MB0170.CHEP278.PROD.OUTLOOK.COM.

[Roman Fischer]

I think we can't have both (be able to revoke any certificate within 24h/5d *and* be considerate about possible negative effects).

 

As far as the discussion here seems to go: Revocation within 5 days seems to be more important. I *personally* don't agree but if that's the way things are going to be, then I'm in favor of strict and clear rules.

 

Cheers
Roman

To view this discussion visit https://groups.google.com/a/mozilla.org/d/msgid/dev-security-policy/5fe1e9c2-3598-4001-9447-2f6ec646770f%40it.auth.gr.

[Rich Salz]

As far as the discussion here seems to go: Revocation within 5 days seems to be more important. I *personally* don't agree but if that's the way things are going to be, then I'm in favor of strict and clear rules.

I don't think the discussion is going that way. Some of us have raised concerns, but most of the messages in the thread have been questions about clarification.

 

[Martijn Katerbarg]

Hi Ben, 

> Maintain and test mass revocation plans annually, including the revocation of 30 randomly chosen certificates within a 5-day period. 

We believe this part of the proposal will not have the intended effect, and would rather cause annoyance with Subscribers. Let me explain why:  

Over the years, Sectigo has done several mass revocations. We have procedures in place for handling these in a compliant fashion, and these procedures have worked. For a total of 30 certificates, we would not use our mass revocation procedures; rather, for 30 certificates it’s far more effective to pick a handful of employees to perform manual outreach to customers. Contacting customers directly and personally ensures that they are taking steps to replace each certificate before it is revoked, and it’s do-able for 30 certificates. It’s not do-able for 30,000 certificates.  
 

Question: What is this requirement hoping to accomplish? Is it (1) to confirm that CAs have a revocation plan available and tested; or (2) to make sure Subscribers have such a plan in order and are ready for it at any time? 

If the answer is number 2, we’re very skeptical that it would be successful. Going by https://crt.sh/cert-populations, we’re counting around 1 billion unexpired pre-certificates. Let’s take that as a ballpark number. Based on CCADB records, there’s about 55 distinct CA Owners included in the Mozilla Root Store. 55 times 30 certificates become 1650 revocations per year. That means, as a Subscriber, (ignoring math around which CA Owner I’d use) each of my unexpired certificates would have a 0.000165% chance of being randomly selected for revocation out of all unexpired certificates out there. That chance would further decline if I’d use one of the top 10 CAs based on number of issued certificates. I’m not sure how much that would spur Subscribers to change tactics. 

Additionally, we feel that several CAs have already shown that they are able to perform mass-revocations throughout the year, in full compliance with the BR mandated revocation deadlines. Wouldn’t it be fairer to only impose this random revocation requirement on those CAs that have actually had delayed revocation incidents? 

i.e., If a CA hasn’t had a delayed revocation incident in the last 12 months, then they MAY perform any random revocations; whereas if a CA has had a delayed revocation incident in the last 12 months, then they MUST.  

 

Regards,

 

Martijn Katerbarg
Sectigo

 

From: dev-secur...@mozilla.org <dev-secur...@mozilla.org> on behalf of Rich Salz <rich...@gmail.com>
Date: Friday, 3 January 2025 at 19:22
To: Roman Fischer <roman....@swisssign.com>
Cc: dev-secur...@mozilla.org <dev-secur...@mozilla.org>
Subject: Re: MRSP 3.0: Issue #276: Delayed Revocation

--

You received this message because you are subscribed to the Google Groups "dev-secur...@mozilla.org" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dev-security-po...@mozilla.org.

To view this discussion visit https://groups.google.com/a/mozilla.org/d/msgid/dev-security-policy/CAFH29triiDQbO%3DF2t5%2BBmDrAVmh%2BbzMydQ4DhgUyOHsE3QbJaQ%40mail.gmail.com.

[Rob Stradling]

[Posting in a personal capacity, per https://wiki.mozilla.org/CA/Policy_Participants]

Ben wrote:
> 1. “Mozilla does not grant exceptions…” -- this is the most important signal that Mozilla can provide.

I feel compelled to express the view that Mozilla should not provide any guidance beyond this “Mozilla does not grant exceptions” statement.  In other words, I think that the non-policy language at https://wiki.mozilla.org/CA/Responding_To_An_Incident#Revocation should be completely removed.  Not updated, not improved, and certainly not incorporated into the MRSP in some way.  Completely Removed.

I have formed this opinion somewhat reluctantly and primarily because I've observed one CA, despite their error being pointed out multiple times by multiple community members, continually misrepresent the non-policy language at https://wiki.mozilla.org/CA/Responding_To_An_Incident#Revocation as "Mozilla policy".  In one comment, that CA made the bogus claim that "the current Mozilla policy has language that allows for delayed revocation".  Despite now recognizing that delayed revocation "violates the Mozilla policy based on the fact that Mozilla states that CAs are expected to comply with the BRs", that CA seems to want to bury that uncomfortable truth of the "most important signal" beneath Issue #276, writing "We continue to appreciate Ben's efforts to lead a productive discussion on potential policy changes in this area, and would prefer that people spend their efforts on moving such discussions forward".

Furthermore, by my count there were 30 "leaf-revocation-delay" bugs opened against 20 CA Owners during 2024.  My recollection is that most of these revocations were delayed by choice rather than by accident.  I wonder how many of these CAs have misinterpreted https://wiki.mozilla.org/CA/Responding_To_An_Incident#Revocation as a "get out of jail free card", despite the already clear assertion on that wiki page that "Mozilla does not grant exceptions to the BR revocation requirements"?

Based on events so far, my conclusion is this: however well intentioned it might be, the mere existence of any language that attempts to regulate CA responses to delayed revocation incidents drowns out that "most important signal" and consequently risks doing more harm than good.  IMHO, that risk can only be mitigated if that language is Completely Removed.

> New CA Obligations:
> - Maintain and test mass revocation plans annually, including the revocation of 30 randomly chosen certificates within a 5-day period.

Please note that there are CAs that do correctly understand that "Mozilla does not grant exceptions" and that do always strive to adhere to the mandatory BR revocation deadlines.  Why should these rule-abiding CAs and their subscribers be burdened with this proposed random revocation requirement?  This seems unfair, in my view.

Martijn asked if it would be "fairer to only impose this random revocation requirement on those CAs that have actually had delayed revocation incidents". I think that this would not only be fairer but might also act as a deterrent against CAs delaying revocations in the first place!

---

From: 'Ben Wilson' via dev-secur...@mozilla.org <dev-secur...@mozilla.org>
Sent: 18 December 2024 17:17
To: dev-secur...@mozilla.org <dev-secur...@mozilla.org>

Subject: Re: MRSP 3.0: Issue #276: Delayed Revocation

 

This Message Is From an External Sender

This message came from outside your organization.

Report Suspicious

 

All,

As part of the discussions on this proposal, namely that CAs “maintain and test mass revocation plans annually, including the revocation of 30 randomly chosen certificates within a 5-day period,” I’ve received a few comments via private channels, and to ensure transparency and foster discussion, I am sharing them here anonymously:

1. “Mozilla does not grant exceptions…” -- this is the most important signal that Mozilla can provide.

2. If certificate consumers want to prohibit delayed revocation, then they need to make it clear to CAs that they won't accept it and that they will kick them out of the root stores if they still do it. Don't try to solve this issue with indirect measures like random revocations. Just be straight about it and make it clear that there will be consequences for the very first delayed revocation and onward.

3. We will face big problems in revoking productive customer certificates just to test our mass-revocation plan and procedures. Our current customer contracts do not foresee this. While we can revoke at any time for security or compliance reasons, this authorization should not be used just to test mass-revocation. This will also require us to push out contract changes to our complete TLS customer base, which will take a considerable amount of time and effort.

4. This part of the proposal should occur within the CA/Browser Forum through amendments to the TLS Baseline Requirements, and not via Mozilla Root Store Policy.

5. Why was the number 30 chosen as a sample?  Some CA operators issue very few certificates, while some CAs issue millions of certificates.

I welcome your feedback on these points, the random sampling proposal, and any others.

Thanks,

Ben

On Monday, December 16, 2024 at 3:02:35 AM UTC-7 pfuen...@gmail.com wrote:

Hi Ben,

About " annual plan testing by revoking 30 randomly chosen certificates within a 5-day timeframe; and"...

We understand that this mean that a CA will need to randomly revoke 30 certificates that most likely don't have other reason for being revoked than being randomly chosen and customers will just need to "happily" accept the situation... Harsh, but doable...

About "audit report submitted under section 3.1 SHALL include an attestation that the CA operator has met these mass revocation planning requirements", it must be considered that the attestation letters of Webtrust audit reports have a fixed format, so such addition would be added most likely as a "Other matters" section, that audits can take each differently.

My question here would be if you think it's there any chance that these requirements become part of the TLS BRs instead of the Mozilla Policy, I see several benefits here:

- Checking the mass-revocation plan would be integral part of the audit scope, so auditors don't need to figure out how to include it in the reports... it just needs to be added to the audit criteria.

- The "inverse-lottery" thing could be added in the revocation timelines of the BR, so there's an entry in the 5-day deadline adding a new category "The certificate has been randomly chosen for revocation during an internal audit". This should facilitate the contractual language to add in the subscriber agreement.

El domingo, 15 de diciembre de 2024 a las 21:51:43 UTC+1, Ben Wilson escribió:

All,

The purpose of this email is to start discussion of Mozilla GitHub Issue #276 ("Address Delayed Revocation"). We would like to collect comments and feedback on a proposal to address delayed certificate revocation from a Mozilla perspective. It builds on prior discussions and feedback regarding delayed revocation, and the proposal is meant to replace guidance currently provided on the Mozilla CA wiki.

Here is the comparison link for a proposed new section 6.1.3 in the MRSP:

https://github.com/mozilla/pkipolicy/compare/51b2f702accd54cb70d52081a9e814298433495b%E2%80%A6efa8ac40ac341fb813620938ef72328a53858038

Summary

Here are the highlights of the proposal:

• Revocation must occur promptly in compliance with the timelines set in section 4.9.1 of the TLS Baseline Requirements (TLS BRs). Mozilla does not grant exceptions to these timelines.
• New CA Obligations:
• Educate subscribers on revocation timelines and discourage reliance on certificates in systems that cannot tolerate timely revocation.
• Include contractual language requiring subscriber cooperation with revocation timelines.

• Maintain and test mass revocation plans annually, including the revocation of 30 randomly chosen certificates within a 5-day period.

• Beginning April 15, 2026, CA audit reports must attest to compliance with the mass revocation planning requirements.
• Delayed revocation incidents must be reported per version 2.1 of the CCADB's Incident Reporting Guidelines (as currently proposed)
• Repeated delayed revocation incidents will result in heightened scrutiny or sanctions, which may include root removal.

Background

Earlier this year, on this list, I proposed an Interim Policy to Address Delayed Revocation. While the proposed interim policy provided clarity, it faced criticism regarding implementation complexity, burden on subscribers and CAs, and the feasibility of associated measures, such as transitioning delayed revocation domains to 90-day certificates. Also, there were subsequent proposals aimed at reducing certificate lifetimes and encouraging automation. See e.g. https://github.com/cabforum/servercert/pull/553.

This new proposal drops proposed measures such as domain-specific tracking and subscriber attestations and instead focuses on subscriber education, mass revocation preparedness, and robust incident reporting as the primary mechanisms for improving agility and transparency regarding delayed revocation.

If adopted, the proposed MRSP § 6.1.3 would replace the current guidance on delayed revocation in Mozilla’s wiki and ensure consistency with the CCADB's Incident Reporting Guidelines.

I welcome your feedback on this draft proposal. Please share your thoughts, questions, or concerns to help us refine and improve it further.

Thanks,

Ben Wilson

Mozilla Root Store

--
You received this message because you are subscribed to the Google Groups "dev-secur...@mozilla.org" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dev-security-po...@mozilla.org.

To view this discussion visit https://groups.google.com/a/mozilla.org/d/msgid/dev-security-policy/71b07640-d425-4f2f-8da4-d97a9475b9f6n%40mozilla.org.

[Ben Wilson]

All,

Thanks for your responses thus far.

We agree about the need to ensure that our messaging around timely revocation remains clear and unambiguous.

In response to your comments, we might want to develop and adopt a graduated list of proportionate penalties that CA operators will face in response to delayed revocation (rather than imposing across-the-board requirements on all CAs). This approach will not only be more effective, but also will provide CA operators with clear expectations about the consequences of delayed revocation.  A structured list of enforcement measures would create a strong incentive for CAs to adhere to revocation timelines.  A "menu of measures" could graduate from minor, first-time incidents to more stringent measures for repeated or egregious violations.

I invite the community to revisit and review all previously suggested measures, suggest new ones, and advise us about this idea of adopting a structured, ordered list of sanctions for CAs that delay revocation.

I look forward to your additional comments and suggestions.

Thanks again,

Ben

 

 

On Thursday, January 9, 2025 at 2:45:36 AM UTC-7 r...@sectigo.com wrote:

[Posting in a personal capacity, per https://wiki.mozilla.org/CA/Policy_Participants]

Ben wrote:
> 1. “Mozilla does not grant exceptions…” -- this is the most important signal that Mozilla can provide.

I feel compelled to express the view that Mozilla should not provide any guidance beyond this “Mozilla does not grant exceptions” statement.  In other words, I think that the non-policy language at https://wiki.mozilla.org/CA/Responding_To_An_Incident#Revocation should be completely removed.  Not updated, not improved, and certainly not incorporated into the MRSP in some way.  Completely Removed.

I have formed this opinion somewhat reluctantly and primarily because I've observed one CA, despite their error being pointed out multiple times by multiple community members, continually misrepresent the non-policy language at https://wiki.mozilla.org/CA/Responding_To_An_Incident#Revocation as "Mozilla policy".  In one comment, that CA made the bogus claim that "the current Mozilla policy has language that allows for delayed revocation".  Despite now recognizing that delayed revocation "violates the Mozilla policy based on the fact that Mozilla states that CAs are expected to comply with the BRs", that CA seems to want to bury that uncomfortable truth of the "most important signal" beneath Issue #276, writing "We continue to appreciate Ben's efforts to lead a productive discussion on potential policy changes in this area, and would prefer that people spend their efforts on moving such discussions forward".

Furthermore, by my count there were 30 "leaf-revocation-delay" bugs opened against 20 CA Owners during 2024.  My recollection is that most of these revocations were delayed by choice rather than by accident.  I wonder how many of these CAs have misinterpreted https://wiki.mozilla.org/CA/Responding_To_An_Incident#Revocation as a "get out of jail free card", despite the already clear assertion on that wiki page that "Mozilla does not grant exceptions to the BR revocation requirements"?

Based on events so far, my conclusion is this: however well intentioned it might be, the mere existence of any language that attempts to regulate CA responses to delayed revocation incidents drowns out that "most important signal" and consequently risks doing more harm than good.  IMHO, that risk can only be mitigated if that language is Completely Removed.

> New CA Obligations:
> - Maintain and test mass revocation plans annually, including the revocation of 30 randomly chosen certificates within a 5-day period.

Please note that there are CAs that do correctly understand that "Mozilla does not grant exceptions" and that do always strive to adhere to the mandatory BR revocation deadlines.  Why should these rule-abiding CAs and their subscribers be burdened with this proposed random revocation requirement?  This seems unfair, in my view.

Martijn asked if it would be "fairer to only impose this random revocation requirement on those CAs that have actually had delayed revocation incidents". I think that this would not only be fairer but might also act as a deterrent against CAs delaying revocations in the first place!

---

From: 'Ben Wilson' via dev-security-policy@mozilla.org <dev-security-policy@mozilla.org>

Sent: 18 December 2024 17:17

To: dev-security-policy@mozilla.org <dev-security-policy@mozilla.org>

--
You received this message because you are subscribed to the Google Groups "dev-security-policy@mozilla.org" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dev-security-policy+unsub...@mozilla.org.

[Matt Palmer]

On Thu, Jan 09, 2025 at 09:45:26AM +0000, 'Rob Stradling' via dev-secur...@mozilla.org wrote:
> [Posting in a personal capacity, per https://wiki.mozilla.org/CA/Policy_Participants]
> Ben wrote:
> > New CA Obligations:
> > - Maintain and test mass revocation plans annually, including the revocation of 30 randomly chosen certificates within a 5-day period.
>
> Please note that there are CAs that do correctly understand that
> "Mozilla does not grant exceptions" and that do always strive to
> adhere to the mandatory BR revocation deadlines.

I'm sure all CAs "strive", to greater or lesser degrees of both effort
and effectiveness, to adhere to the revocation requirements.
Nevertheless, issues do occur across even CAs that are presumably
striving hard, as the breadth of CAs represented in the list of historic
revocation failure bugzilla issues highlights. The list of prominent
CAs who have had a delayed revocation incident appears much longer than
the list of prominent CAs that haven't. I would not presume to accuse
every CA on that list of not striving to adhere to the mandatory
deadlines at the time of their respective incidents.

> Why should these
> rule-abiding CAs and their subscribers be burdened with this proposed
> random revocation requirement? This seems unfair, in my view.

In general, anything that does not get exercised regularly tends to
atrophy, and so when it turns out to be needed, it does not perform as
well as would be hoped.

During the time I was actively monitoring for BR violations identified
by the Pwnedkeys Revokinator, I found (and reported) multiple cases,
from otherwise (presumably) rule-abiding CAs. It was from this
experience that I first came up with the idea of proactive revocation
requests. Even the best CAs have deficiencies and limitations in their
processes and systems, and it is only by regularly exercising the
processes and systems in a variety of ways will those deficiencies and
limitations be identified, and hence (hopefully) fixed.

That's why I support random revocation requirements for *all* CAs --
because it is practically axiomatic that all CAs' systems will be
less-than-perfect, with problems that have lain dormant, and are only
identified by real-world, end-to-end testing. And that's even before we
start considering the subscriber-level problems out there...

On the subject of being "burdened" by additional requirements, I'd
respond that all regulation is a "burden", and that's just part and
parcel of participating in a regulated industry. CA behaviour has
highlighted a deficiency in the regulatory framework, and that is now
being addressed. If there is a "lighter touch" regulation that could
achieve the same outcomes (identify deficiencies in CA revocation
processes before they turn into massive, real-world problems), I'm sure
Mozilla would seriously consider it. But "don't enhance the regulations
because they would be a burden to the regulated" is not an effective
means of achieving safe and reliable systems.

> Martijn asked if it would be "fairer to only impose this random
> revocation requirement on those CAs that have actually had delayed
> revocation

> incidents"<https://www.mail-archive.com/dev-secur...@mozilla.org/msg01951.html>.

> I think that this would not only be fairer but might also act as a
> deterrent against CAs delaying revocations in the first place!

There is definitely something to be said for additional scrutiny on CAs
that have historically had problems with their revocation processes, in
a similar vein to "consent decrees" -- once you've demonstrated a
problem, we're going to watch you a bit more closely for a while
afterwards. Along those lines, I'd suggest something like:

-----8<-----

In the 12 months following the closure of each delayed revocation
incident report, Mozilla will verify the CA's process and system
improvements by making a (verified in some appropriate manner)
revocation request to the CA, enumerating a selection of certificates or
other relevant identifiers, chosen in a manner at Mozilla's sole
discretion to reflect the characteristics of the original delayed
revocation incident, of the same magnitude of impact as the original
delayed revocation.

The CA will be expected to process all of the listed identifiers within
the BR-required timeframe which applied to the original delayed
revocation, and failure to do so will be considered a delayed revocation
incident like any other.

----->8-----

For example, if a CA delays revocation on about 10,000 EV certs that
were improperly issued due to improper domain-control validation, then
Mozilla would, at a time of its own choosing within the 12 months after
the original incident is closed, email the CA a list of about 10,000 EV
certs issued by that CA and expect to see them all show up in CRLs and
OCSP responses as "revoked" within 24 hours of the email being received
by the CA's problem reporting address.

As another, slightly more convoluted example, if a CA delays revocation on
some number of certificates due to 10 compromised keys, then Mozilla
would pick 10 public keys used in certificates issued by that CA and
require all certificates that use those key to be revoked, and that the
new certificates not use any of the identified keys.

Mirroring the magnitude of the original delayed revocation both tests
the improvements made to handling the volume of the original incident,
and motivates CAs to put effort into minimising the scope of the
original incident, even if they can't prevent it entirely.

Similarly, requiring the same revocation period as the original incident
tests the same "kind" of incident again, on the presumption that a CA
that couldn't meet a 24 hour deadline the first time *might* still have
been able to meet a five day deadline, so testing a different scenario
is not nearly as useful a test as replicating the original scenario as
closely as possible.

Waiting until after the incident report is closed ensures that the test
a "fair" one, in that it isn't hitting a CA while it's still getting the
ship afloat again. While CAs might try to delay the test by holding the
incident open forever, it'll be compensated for by the poor optics of
having a big list of unresolved incidents -- if an incident's sole
action item was "deploy enhanced certificate linting tool", for example,
and the incident report is still open two years later, one might start
to question the CA's capacity to execute. Eventually that incident's
going to get closed, and if a bunch of big incidents all get closed at
once, that CA's going to have a busy year ahead of them...

I have deliberately made it Mozilla's role to select the certificates
and choose the timing, because as others have mentioned, I have zero
faith that all CAs will honestly select the certificates and timing of
the revocation in a random manner. It is just too easy to put the thumb
on the scales with practically zero chance of being detected, to think
that *someone* won't do it.

As an aside, if Mozilla would like to outsource the reporting and
analysis of test revocations, I happen to have a high-volume revocation
monitoring service that's just itching to be more widely used...

- Matt

[Ryan Hurst]

I agree with Rob’s point: Mozilla’s clear stance—"Mozilla does not grant exceptions"—is critical. The non-policy guidance on Responding to an Incident has been repeatedly misused by some CAs to justify delayed revocations, despite corrections. Removing this language entirely is the best way to reinforce Mozilla’s expectations and prevent further misuse.

Ryan Hurst

To view this discussion visit https://groups.google.com/a/mozilla.org/d/msgid/dev-security-policy/MW4PR17MB47290C4E22FC2F389E320EADAA132%40MW4PR17MB4729.namprd17.prod.outlook.com.

[Roman Fischer]

Dear Matt,

>In general, anything that does not get exercised regularly tends to atrophy, and so when it turns out to be needed, it does not perform as well as would be hoped.

I agree. However, as far as I remember at least the delayed revocations in the past 1-2 years were not due to CAs incapability to do mass-revocation but due to CAs believing that weighing customer's claims of negative effects of revocation against complying to the mandatory revocation times was acceptable (I'm not in the industry long enough to say if at any previous time such weighing was accepted by root store operators or the community). IMHO, the problem is not the mass-revocation but the believe that delayed revocation -might- be acceptable under certain circumstances.

>That's why I support random revocation requirements for *all* CAs -- because it is practically axiomatic that all CAs' systems will be less-than-perfect, with problems that have lain dormant, and are only identified by real-world, end-to-end testing. And that's even before we start considering the subscriber-level problems out there...

If testing CAs mass-revocation process is the goal, then we could just put a requirement in the BRs. Auditors would then check if CAs did mass-revocation tests. Such tests don't have to be done on productive public trusted certificates to prove that the process works. 😊

Kind regards
Roman

[Matt Palmer]

On Fri, Jan 10, 2025 at 07:08:07AM +0000, Roman Fischer wrote:
> Dear Matt,
>
> >In general, anything that does not get exercised regularly tends to
> >atrophy, and so when it turns out to be needed, it does not perform
> >as well as would be hoped.
>
> I agree. However, as far as I remember at least the delayed
> revocations in the past 1-2 years were not due to CAs incapability to
> do mass-revocation but due to CAs believing that weighing customer's
> claims of negative effects of revocation against complying to the
> mandatory revocation times was acceptable

While that has been the cause of a few of the more high-profile
mass-revocation incidents, there are plenty of others that have
different causes.

> >That's why I support random revocation requirements for *all* CAs --
> >because it is practically axiomatic that all CAs' systems will be
> >less-than-perfect, with problems that have lain dormant, and are only
> >identified by real-world, end-to-end testing. And that's even before
> >we start considering the subscriber-level problems out there...
>
> If testing CAs mass-revocation process is the goal, then we could just
> put a requirement in the BRs.

The word "just" is doing quite a bit of work in that sentence. In any
event, as the word "Baseline" in the name "Baseline Requirements"
implies, the existence of the BRs does not preclude Mozilla from
imposing additional requirements on its program participants.

> Auditors would then check if CAs did mass-revocation tests.

You have an extremely optimistic opinion of the effectiveness of
auditors to prevent incidents, one which is not borne out by history.

> Such tests don't have to be done on productive
> public trusted certificates to prove that the process works. 😊

On the contrary, they *do* have to be done on productive public trusted
certificates, because otherwise you're not doing a full end-to-end test
of the process as used by real revocation requests. And I can tell you,
categorically, with receipts[1], that real world third-party end-to-end
testing of such processes finds real problems.

- Matt

[1] https://bugzilla.mozilla.org/buglist.cgi?email1=Palmer&emailreporter1=1&resolution=FIXED&classification=Client%20Software&classification=Developer%20Infrastructure&classification=Components&classification=Server%20Software&classification=Other&order=Importance&bug_status=RESOLVED&component=CA%20Certificate%20Compliance&product=CA%20Program&query_format=advanced&emailtype1=substring

[Suchan Seo]

Wouldn't fixed 30 certificates per CA cause smaller CA just spam load genenrate test certificates on production envirement to delude chance to it actually hit acutal client?

2025년 1월 10일 금요일 오후 5시 5분 55초 UTC+9에 Matt Palmer님이 작성:

[Matt Palmer]

On Fri, Jan 10, 2025 at 06:21:02PM -0800, Suchan Seo wrote:
> Wouldn't fixed 30 certificates per CA cause smaller CA just spam load
> genenrate test certificates on production envirement to delude chance to it
> actually hit acutal client?

It is certainly a possibility. However, that sort of thing leaves
extensive evidence (which, I will note, a CA-driven "random" choice
protocol does not), and if such evidence was found and brought to light
publicly, I expect it would not reflect well on the CA involved.

Also, in the protocol I proposed, as Mozilla has full discretion over
the the criteria by which certificates are chosen for a revocation test,
it does not have to select those certificates entirely at random. It
could identify certificates issued for the purposes you describe, and
just not include such certificates in the list of those to be revoked.

- Matt

[Ben Wilson]

All,

It has been recommended that Mozilla replace its wiki page on delayed revocation incident reporting.  Here is a draft for your review and comment.

Ben

URL:  https://wiki.mozilla.org/CA/Responding_To_An_Incident#Revocation

Goals:  Mozilla’s goals are: to ensure that revocation occurs as swiftly as possible while maintaining the overall security and stability of the web; to reduce delayed revocation incidents to the greatest extent possible; to improve delayed revocation incident reports; and to gather better information on those cases where delayed revocation occurs. 

Date to Go Live:  January 17, 2025

Revocation

Mozilla’s Expectations on Revocation

CA operators MUST revoke misissued or otherwise problematic TLS server certificates within 24 hours or 5 days, depending on the circumstances set forth in section 4.9.1 of the CA/Browser Forum’s TLS Baseline Requirements (TLS BRs).  Mozilla does not grant exceptions to the revocation requirements of the TLS BRs. 

To ensure compliance with the TLS BRs, Mozilla requires that CA operators:

• engage in proactive communication and advise subscribers well in advance about the revocation timelines and explicitly warn them against using publicly-trusted TLS server certificates on systems that cannot tolerate timely revocation;

• include appropriate language in customer agreements requiring subscribers’ timely cooperation in meeting revocation timelines and acknowledging the CA’s obligations to adhere to applicable policies and standards;

• prepare and maintain credible plans to address mass revocation events, including detailed procedures for handling mass revocations effectively, including rapid communication with affected parties and conducting annual plan testing; and 

• engage a third party assessor to evaluate whether the CA Operator has: 

  • credible plans to handle mass revocation events; 

  • tested the operational effectiveness of the plans, including the accuracy and adequacy of documentation of plan testing, including timelines, results, and remediation steps; and 

  • incorporated feedback from such exercises to improve future readiness. 

Incident Reporting

The CCADB incident reporting process ensures the Web PKI community is informed and that issues are tracked and resolved effectively. Clear and timely communication fosters trust and accountability, mitigating risks to the ecosystem.

If a CA operator determines that it might delay revocation of certificates beyond the time period required by the TLS BRs, it MUST file a preliminary incident report with a Summary section immediately in Bugzilla, even if the delay has not yet occurred.

Such delayed revocation incident reports SHOULD be updated at least every 3 days and MUST be updated no less frequently than every 7 days to describe a summary of:

• the number of certificates that have been revoked; 

• the number of certificates that have not yet been revoked;

• the number of certificates planned for revocation that have expired; and

• an estimate for when all remaining revocations will be completed. 

Consistent with CCADB incident reporting requirements, in the “Analysis” section, the CA operator SHALL explain in the Analysis section of the incident report those factors and rationales behind the decision to delay revocation (including detailed and substantiated explanations of how extensive harm would result to third parties–such as essential public services or widely relied-upon systems–and why the situation is exceptionally rare and unavoidable).

Also, the Timeline section should include the time(s) at which the CA Operator actually completed revocation of affected certificates, and the Action Items list MUST include steps reasonably calculated to prevent or reduce future revocation delays.

Consequences of Delayed Revocations

Failing to meet the standards of timely revocation erodes trust in the Web PKI and poses risks to global internet security.  Delayed revocation is a measure of last resort and MUST not be used routinely.  Repeated incidents of delayed revocation without sufficient justification will result in heightened scrutiny and sanctions, including removal of the CA from the Mozilla Root Store. CA operators must also adhere to the policies and revocation requirements of other Root Store Programs that include their CA certificates.  Additionally, all delayed revocation incidents MUST be listed as findings in the CA operator’s next TLS BR audit statement.

[Rob Stradling]

> Delayed revocation is a measure of last resort and MUST not be used routinely.

Hi Ben.  One nit: That "not" should be capitalized.

---

From: 'Ben Wilson' via dev-secur...@mozilla.org <dev-secur...@mozilla.org>

Sent: 13 January 2025 17:00

To: dev-secur...@mozilla.org <dev-secur...@mozilla.org>
Subject: Re: MRSP 3.0: Issue #276: Delayed Revocation

--
You received this message because you are subscribed to the Google Groups "dev-secur...@mozilla.org" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dev-security-po...@mozilla.org.

To view this discussion visit https://groups.google.com/a/mozilla.org/d/msgid/dev-security-policy/fcff6b2b-605c-4cea-8e56-310e1e3edd21n%40mozilla.org.

[Ben Wilson]

All,

I have posted a replacement wiki page and created a GitHub commit to address this issue.

https://wiki.mozilla.org/CA/Responding_To_An_Incident#Revocation

https://github.com/BenWilson-Mozilla/pkipolicy/commit/76d7e31f00581c6b9cad38b08e125932b8b2828e

Please let me know of any comments or suggestions.

Thanks,

Ben

On Monday, January 13, 2025 at 11:31:36 AM UTC-7 r...@sectigo.com wrote:

> Delayed revocation is a measure of last resort and MUST not be used routinely.

Hi Ben.  One nit: That "not" should be capitalized.

---

From: 'Ben Wilson' via dev-security-policy@mozilla.org <dev-security-policy@mozilla.org>

Sent: 13 January 2025 17:00

To: dev-security-policy@mozilla.org <dev-security-policy@mozilla.org>

--
You received this message because you are subscribed to the Google Groups "dev-security-policy@mozilla.org" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dev-security-policy+unsub...@mozilla.org.

[Ben Wilson]

All,

I have edited proposed section 6.1.3 of the MRSP to add/allow "annual plan testing through tabletop exercises, simulations, parallel testing, or use of test environments, which do not involve the revocation of active certificates."

Ben

On Wednesday, January 15, 2025 at 8:10:52 PM UTC-7 Ben Wilson wrote:

All,

I have posted a replacement wiki page and created a GitHub commit to address this issue.

https://wiki.mozilla.org/CA/Responding_To_An_Incident#Revocation

https://github.com/BenWilson-Mozilla/pkipolicy/commit/76d7e31f00581c6b9cad38b08e125932b8b2828e

Please let me know of any comments or suggestions.

Thanks,

Ben

On Monday, January 13, 2025 at 11:31:36 AM UTC-7 r...@sectigo.com wrote:

> Delayed revocation is a measure of last resort and MUST not be used routinely.

Hi Ben.  One nit: That "not" should be capitalized.

---

From: 'Ben Wilson' via dev-secur...@mozilla.org <dev-security-policy@mozilla.org>

Sent: 13 January 2025 17:00

To: dev-secur...@mozilla.org <dev-security-policy@mozilla.org>

To unsubscribe from this group and stop receiving emails from it, send an email to dev-security-policy+unsubscribe...@mozilla.org.