Groups keyboard shortcuts have been updated
Dismiss
See shortcuts

Proposal to Close Delayed Revocation Incidents

1,222 views
Skip to first unread message

Ben Wilson

unread,
Feb 2, 2025, 2:25:39 PMFeb 2
to dev-secur...@mozilla.org

All,

I am considering closing the following delayed revocation Bugzilla incidents later this week (Friday, 7-Feb-2025), as listed in meta Bug 1911183:

1872738, 1877388, 1884568, 1885568, 1886110, 1886532, 1886665, 1887110, 1887888, 1888882, 1889062, 1890685, 1891331, 1892419, 1896053, 1896553, 1898848, 1903066, 1910805, 1916478, and 1924385,

provided that the CA operator has completed its stated Action Items and filed a Closure Summary.

My reasoning is as follows:

I kept these bugs open while we navigated towards a solution for handling delayed revocations going forward. I believe that the new language in Mozilla Root Store Policy (MRSP) 3.0, effective March 1, 2025, introduces significant measures to improve compliance with revocation requirements and enhance delayed revocation incident reporting.

Under MRSP section 6.1.3, CA operators will be explicitly required to engage in proactive subscriber communication, more specific contractual provisions, and mass revocation preparedness to ensure timely certificate revocation. Additionally, CAs must undergo third-party assessments to validate their readiness for large-scale revocations and that they have documented the outcomes of the testing of their mass revocation plans.

MRSP section 2.4 incorporates the updated incident reporting requirements of the CCADB, and mandates that CAs provide detailed and substantiated justifications in incident reports, explaining delays and impacts on third parties. It notes that Mozilla does not have any exceptions for delayed revocation, and that repeated unjustified delays will result in heightened scrutiny and potential removal from the Mozilla Root Store.

Also, MRSP section 7.1 will require that new TLS-issuing root certificates have at least the ability for automated domain control validation, certificate issuance, and retrieval. This ensures that certificate management processes are efficient, scalable, and less prone to human error, aligning with modern security best practices.

CAs and stakeholders should recognize these changes only as first, yet important, steps in addressing delayed revocation and reporting.

Thoughts?

Thanks,

Ben


Tim Callan

unread,
Feb 6, 2025, 4:18:45 PMFeb 6
to dev-secur...@mozilla.org, Ben Wilson

Ben,

You may see from my recent posts on a few of these individual bugs that at least some of these incidents in my opinion aren’t ready to close.  There has been and continues to be a problem with CAs failing to recognize that BR-mandated revocation deadlines are not theirs to take or leave, and we still have CAs, sometimes nearly a year later, who have not acknowledged this fact nor taken steps to fix their errors.  I appreciate your response to one of my recent posts in which you say that a hard commitment to never delay revocation is too much.  Though I do not agree, I understand your viewpoint and am taking your guidance.

That said, my recollection is that some of these CAs have simply never publicly accepted any culpability nor made an earnest show of changing their priorities.  I put it to you that a CA who refuses to accept responsibility for its free choice to ignore the BRs and has no expressed intention to change its decision making has not remediated the problem. I believe such a bug should remain open until the message sinks in.

If you concur on this last point, then I request you leave these bugs open just a little longer so the community has a chance to review them and weigh in on which are ready to close and which are not.  I don’t imagine this requires a great deal of time, but I don’t see it getting done by tomorrow either.

Cheers,

-tlc

Ben Wilson

unread,
Feb 6, 2025, 4:28:42 PMFeb 6
to Tim Callan, dev-secur...@mozilla.org
Thanks, Tim.  Sounds good. I'll hold off until we have looked at the degree to which each CA has acknowledged the importance of complying with the Baseline Requirements and resetting their priorities on timely revocation.
Ben

Jeremy Rowley

unread,
Feb 6, 2025, 4:33:20 PMFeb 6
to Tim Callan, dev-secur...@mozilla.org, Ben Wilson
I disagree. I think the language was unclear before, and, based on my previous work. I think the public language did lead to confusion on what circumstances delayed revocation could occur, but I have not seen anyone state that the delay was in compliance with the BRs. This seems like a misrepresentation of most of the CAs, and a misrepresentation that is particular sus given that it is coming from a competitor to the other CAs. I think you should close the current bugs given that the language has changed, even if the expectations around delayed revocation remain constant. 

Keeping a bug open just to make a CA look bad for Sectigo seems like an improper purpose for having a bug reporting mechanism. Isn't the purpose to learn why it happened and rectify the issue? Seems like this has happened on all of the bugs I've seen and Mozilla policy has become better for it.



On Thu, Feb 6, 2025 at 2:18 PM Tim Callan <ptimc...@gmail.com> wrote:
--
You received this message because you are subscribed to the Google Groups "dev-secur...@mozilla.org" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dev-security-po...@mozilla.org.
To view this discussion visit https://groups.google.com/a/mozilla.org/d/msgid/dev-security-policy/16701b3c-e0f1-4bfe-9123-79d2273cc4b9n%40mozilla.org.

Jeremy Rowley

unread,
Feb 6, 2025, 4:58:37 PMFeb 6
to Tim Callan, dev-secur...@mozilla.org, Ben Wilson
Perhaps it would be better to look at each delayed revocation bug: 

https://bugzilla.mozilla.org/show_bug.cgi?id=1943528 - Entrust was distrusted so no need to keep it open
https://bugzilla.mozilla.org/show_bug.cgi?id=1942879 - Issue identified and the delay was unintentional
https://bugzilla.mozilla.org/show_bug.cgi?id=1942877 - Issued identified and delay was unintentional 
https://bugzilla.mozilla.org/show_bug.cgi?id=1922572 - No comments after KIRs explanation
https://bugzilla.mozilla.org/show_bug.cgi?id=1916478 - Delay appears unintentional and was an operational issue
https://bugzilla.mozilla.org/show_bug.cgi?id=1910805 - Delay was explained in more detail than other bugs
https://bugzilla.mozilla.org/show_bug.cgi?id=1903066 - Comment 6 acknowledges the delay was wrong
 https://bugzilla.mozilla.org/show_bug.cgi?id=1898848 - Acknowleded that it was a mistake to delay yet Tim keeps pushing
https://bugzilla.mozilla.org/show_bug.cgi?id=1891331 - Netlock talks about the EU difficulties in revocation
https://bugzilla.mozilla.org/show_bug.cgi?id=1889062 - Acknowledges that it was against the BRs to delay
https://bugzilla.mozilla.org/show_bug.cgi?id=1888882 - Comment 28 pretty much states that will ensure awareness of teh 5 day requirement
https://bugzilla.mozilla.org/show_bug.cgi?id=1887888 - This one never answered the question of what they will do going forward.
https://bugzilla.mozilla.org/show_bug.cgi?id=1886665 - States that htey will not delay revocation 

Given the bugs, the lengthy discussion on the bugs, and the revised language in the Mozilla revocation expectation document, what is the value to the community in keeping these open? The value to Sectigo and the Sectigo podcast is pretty obvious, but why would Mozilla or the Mozilla community want to keep going on these items? If there is a clear ask on what should be done, what is it? 

If we want each CA to say "We will never delay revocation" again, we could say that must be stated in the closing. As that is expressly not stated as a requirement, I'm baffled on what more most of the bugs need before closing. 

Amir Omidi

unread,
Feb 6, 2025, 8:12:40 PMFeb 6
to Jeremy Rowley, Tim Callan, dev-secur...@mozilla.org, Ben Wilson
> I think the public language did lead to confusion on what circumstances delayed revocation could occur,

Did it though? One of the expectations for CAs has been to watch and monitor Bugzilla. If they had done so, I don't think this confusion would exist.


> If we want each CA to say "We will never delay revocation" again, we could say that must be stated in the closing. As that is expressly not stated as a requirement, I'm baffled on what more most of the bugs need before closing

What do you mean that is not a requirement? As far as I know practically all the root programs and the BRs are clear about this being a requirement.


> Keeping a bug open just to make a CA look bad for Sectigo seems like an improper purpose for having a bug reporting mechanism.
> The value to Sectigo and the Sectigo podcast is pretty obvious

These statements do not belong in this mailing list.


Jeremy Rowley

unread,
Feb 6, 2025, 8:35:41 PMFeb 6
to Amir Omidi, Tim Callan, dev-secur...@mozilla.org, Ben Wilson
I think this statement from Ben makes it clear that the expectation is not that delayed revocation will never happen: 

You are right that I should not have made the statement alleging that sectigo might have ulterior motives. I apologize for that and retract that statement.

I do think think it would be helpful to know which exact bugs Tim is objecting to closing and why? There are a couple that don’t have statements but some of them have been open for a year with non additional action items of note

Jeremy Rowley

unread,
Feb 6, 2025, 9:20:06 PMFeb 6
to Amir Omidi, Tim Callan, dev-secur...@mozilla.org, Ben Wilson
To clarify, revocation within the prescribed timeline is a requirement of the BRs for sure, but does Mozilla expect that there will never be a violation of the BR revocation requirement? I don’t think that’s quite as clear given the comments from the browsers. I do agree with Tim that such a commitment should be a requirement though as it absolutely removes an potential future ambiguity.

Mike Shaver

unread,
Feb 7, 2025, 10:41:36 AMFeb 7
to Jeremy Rowley, Tim Callan, dev-secur...@mozilla.org, Ben Wilson
Hi Jeremy,

I don't agree with your listed reasoning for these bugs all being suitable for closure, though I do genuinely appreciate you taking the time to enumerate them like this. Some responses to selected entries below!

On Thu, Feb 6, 2025 at 4:58 PM Jeremy Rowley <rowl...@gmail.com> wrote:
Perhaps it would be better to look at each delayed revocation bug: 


To me this doesn't mean "there is nothing to do with this bug" but more likely "in the 5 days since it opened, nobody has bothered to ask anything ahead of the full incident report". These days I'm trying to focus my limited root-meddling time on inadequate action items, so I generally don't bother with preliminary incident reports because I would have to guess what the flaws in the future action items would be.
 
https://bugzilla.mozilla.org/show_bug.cgi?id=1943528 - Entrust was distrusted so no need to keep it open

The purpose of a delayed revocation incident is not only to provide root programs with visibility into the practices and reliability of CAs. It is also to ensure that all (ahem, remaining) CAs have the opportunity to learn from the incident such that they might avoid a similar incident in the future. Entrust being distrusted does not necessarily mean that there is no value in further discussion of the incident, and indeed I think generally many incidents are pretty poorly reasoned and explained in those matters.
 
https://bugzilla.mozilla.org/show_bug.cgi?id=1942879 - Issue identified and the delay was unintentional

Note that in this case the issue was "malware filtering blocked CPR", which was sufficiently distinct from "spam filter blocked CPR" that multiple CAs did not extrapolate from the latter to the former when monitoring previous incidents. This makes me feel that perhaps more attention could be paid by the root programs to explaining how they expect CAs to address classes of issue, rather than just the most narrowly-interpreted case specifically implicated in an incident. Closing the bug ahead of that clarity coming from root programs or, failing that, peer CAs or other community members, seems like a missed opportunity to avoid future incidents like "we block things that have a non-ASCII sender name" or whatever the next fine speciation would be.
 
https://bugzilla.mozilla.org/show_bug.cgi?id=1916478 - Delay appears unintentional and was an operational issue

This is a great example of where the work hasn't been done to make the incident useful for other CAs. It contains the old canards about "improving procedures" and "training staff", but doesn't detail exactly what was wrong with the previous policies or training, or—more importantly—what caused the training or policies to have those defects in the first place. Without that, all a CA can take away is "have procedures and train staff" which, I still permit myself to hope, they all understand at that level of detail already.
 
https://bugzilla.mozilla.org/show_bug.cgi?id=1898848 - Acknowleded that it was a mistake to delay yet Tim keeps pushing

I don't see Tim continuing to push in this bug at all. The last comment was some time ago, and the discussion seems to mostly have been between Entrust representatives and Wayne.
 
https://bugzilla.mozilla.org/show_bug.cgi?id=1889062 - Acknowledges that it was against the BRs to delay

This one is definitely not done, given that the subscriber rationale was given as "these agencies take too long to replace stuff and they're very important in some way", and there was nothing specific in the action items or otherwise to indicate how GDCA will navigate that tension in the future other than "encouragement" or "make a policy (of unspecified contents)".
 
https://bugzilla.mozilla.org/show_bug.cgi?id=1888882 - Comment 28 pretty much states that will ensure awareness of teh 5 day requirement

Yep, for sure *this* time, "customer education" will be sufficient. :P "When users apply for or obtain certificates, they will be clearly informed of the revocation scenarios and time limits" is the sort of thing that's hard for me to take seriously when the BRs already required that the Subscriber make a legally binding commitment to that effect.

Also, that report describes an action item (maybe? a commitment at least) that was to be completed a month ago, and for which no status has been given:

> This action plan will take a long time, but we will still accelerate the progress to encourage as many of our existing customers with larger certificate volumes to use the automated certificate deployment system as much as possible by 2025.

Unless the goal is just to complete the *encouragement* by 2025, in which case it's a nothing-burger again.

https://bugzilla.mozilla.org/show_bug.cgi?id=1887888 - This one never answered the question of what they will do going forward.

Indeed. Seems like closing in that situation would send an unfortunate message that you can just wait it out and the awkward questions will go away.
 
The value to Sectigo and the Sectigo podcast is pretty obvious,

This is bullshit, unwelcome on the list, and quite surprising. I know you apologized but I still can't let it just go by when I'm replying to the message.

Mike

Ben Wilson

unread,
Feb 7, 2025, 11:15:35 AMFeb 7
to Mike Shaver, Jeremy Rowley, Tim Callan, dev-secur...@mozilla.org
All,
Today I'll go through all delayed revocation bugs, and starting from the information provided here, begin to triage them for closure.
Thanks,
Ben

Jeremy Rowley

unread,
Feb 7, 2025, 12:34:00 PMFeb 7
to Mike Shaver, Tim Callan, dev-secur...@mozilla.org, Ben Wilson
Thanks Mike! Sorry again about the Sectio comment. My own interest (because my name is on them) in closing these bugs is pretty obvious.... Not an excuse by any means, but I know it is the main reason I reacted so poorly. 

Overall, I think a lot of the CAs are confused about what they need to do to close the bug. I think with the delayed revocation bugs in particular, there's a lack of clarity on what the community needs to close the bug as resolved considering it was a conscious decision by the CA to delay revocation. Although I do not represent a CA, I cannot think of what the CA can say or do to ensure that delayed revocation doesn't happen again, especially when you can't get browsers to say "Delayed revocation is never acceptable".  Is there a bug with intentional delayed revocation where the action items are ones that were good technical controls to ensure it didn't happen again? I'm struggling to think of any that give real good reasons on how delayed revocation won't happen again outside of just promising "No more delayed revocation". 


>> To me this doesn't mean "there is nothing to do with this bug" but more likely "in the 5 days since it opened, nobody has bothered to ask anything ahead of the full incident report". These days I'm trying to focus my limited root-meddling time on inadequate action items, so I generally don't bother with preliminary incident reports because I would have to guess what the flaws in the future action items would be.

Yeah - I agree this one needs more time and attention. It isn't old and should not be closed. 
 
https://bugzilla.mozilla.org/show_bug.cgi?id=1943528 - Entrust was distrusted so no need to keep it open

>> The purpose of a delayed revocation incident is not only to provide root programs with visibility into the practices and reliability of CAs. It is also to ensure that all (ahem, remaining) CAs have the opportunity to learn from the incident such that they might avoid a similar incident in the future. Entrust being distrusted does not necessarily mean that there is no value in further discussion of the incident, and indeed I think generally many incidents are pretty poorly reasoned and explained in those matters.
 
Makes sense. Is Entrust going to keep responding now that they have sold the business to Sectigo? I realise Entrust still owns the roots which means we need them to reply to the comments going forward. 

https://bugzilla.mozilla.org/show_bug.cgi?id=1942879 - Issue identified and the delay was unintentional

>> Note that in this case the issue was "malware filtering blocked CPR", which was sufficiently distinct from "spam filter blocked CPR" that multiple CAs did not extrapolate from the latter to the former when monitoring previous incidents. This makes me feel that perhaps more attention could be paid by the root programs to explaining how they expect CAs to address classes of issue, rather than just the most narrowly-interpreted case specifically implicated in an incident. Closing the bug ahead of that clarity coming from root programs or, failing that, peer CAs or other community members, seems like a missed opportunity to avoid future incidents like "we block things that have a non-ASCII sender name" or whatever the next fine speciation would be.
 
I think so too. Be good to get clarity on how to address different situations. IMO, an intentional delay of revocation is more egregious than something like this where a tool blocked the CPR. I think it's far more acceptable where a technical solution caused the issue instead of a human doing something wrong as it shows the CA has technology in place to control security and (hopefully) automate a lot of the process. 

https://bugzilla.mozilla.org/show_bug.cgi?id=1916478 - Delay appears unintentional and was an operational issue

>> This is a great example of where the work hasn't been done to make the incident useful for other CAs. It contains the old canards about "improving procedures" and "training staff", but doesn't detail exactly what was wrong with the previous policies or training, or—more importantly—what caused the training or policies to have those defects in the first place. Without that, all a CA can take away is "have procedures and train staff" which, I still permit myself to hope, they all understand at that level of detail already.
 
Agreed on this one. Unintentional delays need technical controls to prevent them from reoccuring. This one doesn't have technical controls. 

https://bugzilla.mozilla.org/show_bug.cgi?id=1898848 - Acknowleded that it was a mistake to delay yet Tim keeps pushing

>> I don't see Tim continuing to push in this bug at all. The last comment was some time ago, and the discussion seems to mostly have been between Entrust representatives and Wayne.

You're right. This one is one I would close though.

https://bugzilla.mozilla.org/show_bug.cgi?id=1889062 - Acknowledges that it was against the BRs to delay

>> This one is definitely not done, given that the subscriber rationale was given as "these agencies take too long to replace stuff and they're very important in some way", and there was nothing specific in the action items or otherwise to indicate how GDCA will navigate that tension in the future other than "encouragement" or "make a policy (of unspecified contents)".
 
Okay - what action item should they have?

https://bugzilla.mozilla.org/show_bug.cgi?id=1888882 - Comment 28 pretty much states that will ensure awareness of teh 5 day requirement

>> Yep, for sure *this* time, "customer education" will be sufficient. :P "When users apply for or obtain certificates, they will be clearly informed of the revocation scenarios and time limits" is the sort of thing that's hard for me to take seriously when the BRs already required that the Subscriber make a legally binding commitment to that effect.

I think these are hard as I know customers cite the previous Mozilla policy as allowing delayed revocation. Getting rid of that language will be a significant boon. 

https://bugzilla.mozilla.org/show_bug.cgi?id=1887888 - This one never answered the question of what they will do going forward.

>> Indeed. Seems like closing in that situation would send an unfortunate message that you can just wait it out and the awkward questions will go away.

Yeah - I definitely agree with this one.

On Fri, Feb 7, 2025 at 8:41 AM Mike Shaver <mike....@gmail.com> wrote:

Tim Callan

unread,
Feb 10, 2025, 2:22:46 PMFeb 10
to dev-secur...@mozilla.org, Jeremy Rowley, Tim Callan, dev-secur...@mozilla.org, Ben Wilson, Mike Shaver

I appreciate that this matter already has been discussed and accept Jeremy’s explanation and retraction, and I have no interest in reopening that conversation.  Nonetheless, based on this thread it feels to me that we would be wise to articulate our motivations.

 

About five years ago, Sectigo was in a bad place as a public CA. The company’s commitment to quality of execution, accurate issuance, responsibility to the WebPKI, and continuous improvement was abysmal. This manifested itself in a series of severe errors that should not have occurred accompanied by terrible management of the resulting Bugzilla incidents.

 

I sincerely believe that had Sectigo continued down that path, our roots would be distrusted today.  Fortunately, a few thought leaders inside the company had this epiphany, made their voices heard, and ultimately convinced those around them.  This led to clear policy changes inside Sectigo and began a process that in the long term required widespread technical, cultural, procedural, and leadership reform.  We forged new (for us at least) philosophies about information, quality control, operational efficiency, certificate morphology, policy governance, internal communication, external communication, automation, and the proper role of humans in our processes. There is pretty much no aspect of our public CA operation that we did not scrutinize and renovate in the past five years. While we don’t think we’re perfect by a long shot, we know beyond doubt that we have come a great distance.

 

It's been a lengthy, hard journey that has required deep thinking, unflinching self-examination, guts, persistence, and lots of sweat.  So we appreciate that it can be difficult for a CA to commit to the kind of improvement that, unfortunately, many of us still clearly need.

 

Sectigo would like to help.  Looking at the behaviors of the WebPKI CA community, we perceive a common tendency for CAs not to understand the importance of self-improvement, or know how to go about it, or even recognize the incredible privilege they have as stewards of public trust.  We see CAs treating the short-term and short-sighted desires of their corporate owners and local governments as more important than five-and-a-half billion internet users around the globe.  We routinely see CAs placing their Subscribers’ convenience over the good of the WebPKI.

 

We see these things, and we have chosen to act.  We are trying to show CAs that self-improvement not only is possible but is an asset to the strength of your operation.  We are trying to spur improvement through encouragement and information sharing and where necessary strong rhetoric.  We try to provide concrete coaching while challenging our fellows to become their better selves.

 

This is a learning process for us.  A sad fact is that many CAs appear quite intractable and unreceptive.  But we also see wins, instances where some CA will experience an authentic “light bulb” moment or a change of heart.  So we know it’s possible to connect in that way.  At least on occasion we see the idea of “CAs helping CAs” actually work.

 

Doubtless we have made and will continue to make errors along the way, as we try to figure out how to be sherpas to other CAs who don’t have the benefit of our last five years’ experience.  Surely some will never appreciate it, or value it, or care in any way.  Or for that matter improve.  But some already have.  This makes us think that, however preliminary our efforts and inconsistent the results, there is something here worth pursuing.

Ben Wilson

unread,
Feb 11, 2025, 11:09:00 AMFeb 11
to Tim Callan, dev-secur...@mozilla.org, Jeremy Rowley, Mike Shaver
Reply all
Reply to author
Forward
0 new messages