GoDaddy Revocation Disclosure

Daymion Reynolds

unread,

Aug 17, 2018, 6:22:14 PM8/17/18

to mozilla-dev-s...@lists.mozilla.org

Ben Laurie

unread,

Aug 18, 2018, 5:27:05 PM8/18/18

to drey...@godaddy.com, mozilla-dev-s...@lists.mozilla.org

On Fri, 17 Aug 2018 at 18:22, Daymion Reynolds via dev-security-policy <
dev-secur...@lists.mozilla.org> wrote:

> Revoke Disclosure
>
> GoDaddy has been proactively performing self-audits. As part of this
> process, we identified a vulnerability in our code that would allow our
> validation controls to be bypassed. This bug would allow for a Random Value
> that was generated for intended use with Method 3.2.2.4.6 and 3.2.2.4.7 and
> was validated using Method 3.2.2.4.2 by persons who were not confirmed as
> the domain contact. This bug was introduced November 2014 and was leveraged
> to issue a total of 865 certificates. The bug was closed hours after
> identification, and in parallel we started the scope and revocation
> activities.
>
> In accordance with CA/B Forum BR, section 4.9.1.1, all miss-issued
> certificates were revoked within 24 hours of identification.
>
> A timeline of the Events for Revocation are as follows:
>
> 8/13 9:30am – Exploit issue surfaced as possible revocation event.
> 8/13 9:30-4pm – Issue scope identification (at this point it was unknown),
> gathering certificate list
> 8/13 4pm – Certificate list finalized for revoke total 825 certs, Revoke
> notification sent to cert owners.
>

I presume you mean domain owners?

Do we know if any of these certs were used? If so, how?

> <https://crt.sh/?id=576003106www.hostile-environment-training.com,https://crt.sh/?id=606496635>

> <https://crt.sh/?id=637598060desktop.ems-uk.org,https://crt.sh/?id=631958613>
> _______________________________________________
> dev-security-policy mailing list
> dev-secur...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-security-policy
>

Daymion Reynolds

unread,

Aug 20, 2018, 12:26:49 PM8/20/18

to mozilla-dev-s...@lists.mozilla.org

Yes, domain owners.

Yes, some of the certs were being used as typical server certs. We have not detected any nefarious activities.

Wayne Thayer

unread,

Aug 20, 2018, 1:40:15 PM8/20/18

to Daymion Reynolds, mozilla-dev-security-policy

Thank you for the disclosure Daymion. I have created bug 1484766 to track
this issue. I've requested an incident report to help the community better
understand what happened and what can and is being done to prevent similar
problems in the future, as described in the last two topics [1]:

6. Explanation about how and why the mistakes were made or bugs introduced,
and how they avoided detection until now.
7. List of steps your CA is taking to resolve the situation and ensure such
issuance will not be repeated in the future, accompanied with a timeline of
when your CA expects to accomplish these things.

- Wayne

[1] https://wiki.mozilla.org/CA/Responding_To_An_Incident#Incident_Report

On Mon, Aug 20, 2018 at 9:26 AM Daymion Reynolds via dev-security-policy <
dev-secur...@lists.mozilla.org> wrote:

> On Saturday, August 18, 2018 at 2:27:05 PM UTC-7, Ben Laurie wrote:

> Yes, domain owners.
>
> Yes, some of the certs were being used as typical server certs. We have
> not detected any nefarious activities.

Daymion Reynolds

unread,

Aug 20, 2018, 5:32:19 PM8/20/18

to mozilla-dev-s...@lists.mozilla.org

Wayne, I have found the bug. Will add information to it soon. -Daymion

Jeremy Rowley

unread,

Mar 12, 2019, 4:17:15 PM3/12/19

to mozilla-dev-s...@lists.mozilla.org

One item that I think could bear a useful discussion from these incident reports is how the community can get more involved in discussing and helping with incident reports. For example, the 63 bit serial number issue is leading to a lot of certs potentially being revoked with little benefit to the community (IMO of course). There are definite downsides to revocation that may or may not be fully considered when people are responding to incidents. For example, adding a bunch of certs to a CRL for a minor issue seems like a pointless increase in CRL size. There's also the customer disruption and other issues to consider that are probably important for the community to know when looking at incident reports.

I'm wondering if we (the community or CABForum) should have some mechanism of evaluating these risks and proposed incident plans before/while the plan is executed. For example, the pros and cons of revocation of the certs could be discussed. Actual revocation would be up to the CA , course, and any non-compliances would be noted on the audit report, but this part of the policy could be a community effort: "That you will perform an analysis to determine the factors that prevented timely revocation of the certificates, and include a set of remediation actions in the final incident report that aim to prevent future revocation delays." (https://wiki.mozilla.org/CA/Responding_To_An_Incident#Revocation). We could have the CA propose a rough draft to the community where they engage in a QA about the incident and then have members make a recommendation to the CA about remediation. All voluntary on the advice. This is probably the way it is supposed to currently work, but right now the flow seems like:
1) Post to Mozilla
2) Create an incident report
3) Community discussion about compliance and why CAs need to do better 😊
4) Update incident report until Wayne closes it

A new flow that includes the community more fully could be:
1) Post to Mozilla, the post must include an initial proposed plan of action
2) Create an incident report (to track bugs)
3) Discuss on the Mozilla forum the proposed plan and post updated plans based on member suggestions
4) Post a final draft to Bugzilla
5) Post updates per a timeline set in the incident report
6) Wayne closes the bug.

This is probably a lot more work for the CA, but I know we'd find the community feedback on how to resolve issues useful. Maybe it could also change into a continuous flow of "How can X CA do better - here's some suggestions" instead of "Better put up the lightning rod and get through this".

Thoughts? Again, probably how this is supposed to work already, but if we can turn it into more actionable feedback about what's next, then I'd find that super useful.

Jeremy

-----Original Message-----
From: dev-security-policy <dev-security-...@lists.mozilla.org> On Behalf Of Daymion Reynolds via dev-security-policy
Sent: Monday, August 20, 2018 10:27 AM
To: mozilla-dev-s...@lists.mozilla.org
Subject: Re: GoDaddy Revocation Disclosure

On Saturday, August 18, 2018 at 2:27:05 PM UTC-7, Ben Laurie wrote:

Yes, domain owners.

Yes, some of the certs were being used as typical server certs. We have not detected any nefarious activities.

_______________________________________________
dev-security-policy mailing list
dev-secur...@lists.mozilla.org

https://clicktime.symantec.com/a/1/SBQDHoNcfxIDo-iEQJuT-dmr7_pW60OdL-qV1EsnpHA=?d=bqzo5p3oDP-sRSQId_bPtHZigaw6daIpv_RsZQkwY-iHzzEA9WJYCCgyej3-L1Kn1kdzI7y7RWimDSsQZGlm9Fzw9NuG4hz0W-b_SuJ7uL9yiouYvIx6Wu4BsWwhF0FfeXN8L8dCJ3oaQPf_L2RelGC9xWRYrBkZWSGYN_1-HcPQIVhUYElwCGv09MRZjSB9vCm27aJXNSf1EPNSQX588qKM3jvO66sFZV2eO86SP2Jtmj1tW8eq1S8dSuu4_27dgMB7fnu9BaDMAsD6324YyDIlTGrTKRblMwbO3piSWYwiVoPg3Oh-XWcs8D9oVhjs8TWpvBcNUUt6zuHrA_ieaEzCik7_j1K3AEwVGrtmeh8eB7RhvP8OuyoXM4XBbf0AI8RySn6OrgkCnq22P3R2fuNooB4-S5I5WmM%3D&u=https%3A%2F%2Flists.mozilla.org%2Flistinfo%2Fdev-security-policy

Ryan Sleevi

unread,

Mar 12, 2019, 4:34:17 PM3/12/19

to Jeremy Rowley, mozilla-dev-s...@lists.mozilla.org

On Tue, Mar 12, 2019 at 4:17 PM Jeremy Rowley via dev-security-policy <
dev-secur...@lists.mozilla.org> wrote:

> A new flow that includes the community more fully could be:
> 1) Post to Mozilla, the post must include an initial proposed plan of
> action
> 2) Create an incident report (to track bugs)
> 3) Discuss on the Mozilla forum the proposed plan and post updated plans
> based on member suggestions
> 4) Post a final draft to Bugzilla
> 5) Post updates per a timeline set in the incident report
> 6) Wayne closes the bug.
>
> This is probably a lot more work for the CA, but I know we'd find the
> community feedback on how to resolve issues useful. Maybe it could also
> change into a continuous flow of "How can X CA do better - here's some
> suggestions" instead of "Better put up the lightning rod and get through
> this".
>
> Thoughts? Again, probably how this is supposed to work already, but if we
> can turn it into more actionable feedback about what's next, then I'd find
> that super useful.
>

So, I think many of these elements are already captured in the current
process, as the lengthy discussion with DigiCert regarding underscores [1],
and this provides a model for engaging with the community and gathering
feedback and concerns about the response.

CAs are responsible for drafting their initial incident reports, gathering
feedback, and making a decision - much as DigiCert did with underscores.
The CA is judged based on how well they considered and balanced the risks,
there is opportunity for concerns about improving (an area DigiCert
encountered with its own reports), and we move forward.

It would seem, from your broader message, that this is looking for some
sort of blanket approval, independent of the CA or facts specific to that
CA, and I think that's something that we've been explicitly trying to avoid
- as the context matters. There are a number of hazards, which Matt Palmer
highlighted during the discussion of underscores [2][3][4], and I think
those still apply now as much as they did two and a half months ago.

[1]
https://groups.google.com/d/msg/mozilla.dev.security.policy/0oy4uTEVnus/pnywuWbmBwAJ

[2]
https://groups.google.com/d/msg/mozilla.dev.security.policy/0oy4uTEVnus/APSWO4SYCgAJ
[3]
https://groups.google.com/d/msg/mozilla.dev.security.policy/0oy4uTEVnus/voFCTMFVAwAJ
[4]
https://groups.google.com/d/msg/mozilla.dev.security.policy/0oy4uTEVnus/ZqO9fHZMAwAJ

Jeremy Rowley

unread,

Mar 12, 2019, 4:38:25 PM3/12/19

to mozilla-dev-s...@lists.mozilla.org

Not looking for blanket approval – I stated it’d still be part of the audit report. We also aren’t directly impacted by this particular incident (which is why I brought it up here). The actual evaluation of the CA would remain up to Mozilla of course, but the really good discussion about 63 bits (especially the proposed ballot language) got me thinking about how we could apply this more generally to incident reports and how CAs can use them before deciding on a course of action. The underscore discussion was definitely good as well, and I felt had a great outcome.

I think the primary change I’m proposing is that the initial report shouldn’t be an incident report. Instead, the initial report can be short blurb posted to Mozilla along with a description on what the Ca plans to do. Then the community can talk about the plan in addition to the incident, rather than just the incident.

Jeremy

From: Ryan Sleevi <ryan....@gmail.com>
Sent: Tuesday, March 12, 2019 2:31 PM
To: Jeremy Rowley <jeremy...@digicert.com>
Cc: mozilla-dev-s...@lists.mozilla.org
Subject: Re: GoDaddy Revocation Disclosure

On Tue, Mar 12, 2019 at 4:17 PM Jeremy Rowley via dev-security-policy <dev-secur...@lists.mozilla.org <mailto:dev-secur...@lists.mozilla.org> > wrote:

A new flow that includes the community more fully could be:
1) Post to Mozilla, the post must include an initial proposed plan of action
2) Create an incident report (to track bugs)
3) Discuss on the Mozilla forum the proposed plan and post updated plans based on member suggestions
4) Post a final draft to Bugzilla
5) Post updates per a timeline set in the incident report
6) Wayne closes the bug.

This is probably a lot more work for the CA, but I know we'd find the community feedback on how to resolve issues useful. Maybe it could also change into a continuous flow of "How can X CA do better - here's some suggestions" instead of "Better put up the lightning rod and get through this".

Ryan Sleevi

unread,

Mar 12, 2019, 4:57:58 PM3/12/19

to Jeremy Rowley, mozilla-dev-s...@lists.mozilla.org

On Tue, Mar 12, 2019 at 4:38 PM Jeremy Rowley via dev-security-policy <
dev-secur...@lists.mozilla.org> wrote:

> I think the primary change I’m proposing is that the initial report
> shouldn’t be an incident report. Instead, the initial report can be short
> blurb posted to Mozilla along with a description on what the Ca plans to
> do. Then the community can talk about the plan in addition to the incident,
> rather than just the incident.
>

Thanks for clarifying, and hopefully I'm not reducing the context too much.

I think if it's before a CA has missed a revocation deadline, that's
exactly what's possible.
However, once a CA has missed the deadline captured in the Baseline
Requirements, it's expected to be an incident report and it's expected that
the CA will have a plan on how to resolve it.

I can see a number of ways in which things could go wrong if the CA isn't
required to have a plan until they've discussed it with m.d.s.p. CAs are
trusted, in theory, because they're able to apply meaningful judgement and
to comply with Root Program policies and the Baseline Requirements.

As an example of where this absolutely could backfire, imagine that a CA
waits to take action for a given incident, because they're hoping some
other CA is affected and that will somehow alter their own need to be
responsive. Alternatively, imagine a CA that is not adequately staffed and
simply seeks to crib from other CA's responses - not really providing the
community any assurances that the particular CA understands the issues or
their own need to be responsive. Imagine a CA that tries to sockpuppet
their way into suggesting revocation isn't "really" necessary.

We trust CAs to be responsive and to take corrective steps when they're
non-compliant. The Incident Reports provide an avenue of transparency for
that, helping the community develop assurance and mitigate concerns that
might exist or be introduced by a given plan. However, I would much rather
be in a place where we're seeing CAs take meaningful corrective actions as
quickly as possible, and I worry that this proposal would fundamentally
discourage it, because it benefits those who wait the longest. I don't
think that's the intent, but I think that's a natural consequence.