GoDaddy: Failure to revoke certificate with compromised key within 24 hours

[sandy...@gmail.com]

On 7 May 2020 at 12:07:07 PM UTC I reported a certificate to GoDaddy at prac...@starfieldtech.com as having its private key compromised.

I received the automated acknowledgement confirmation, however, as of 2020-05-09 03:39:36 UTC (well after 24 hours), OCSP still shows the certificate as being "Good"

The unrevoked certificate is https://crt.sh/?id=2366734355

I believe this is a breach of the CA-BR [4.9.1.1. Reasons for Revoking a Subscriber Certificate] -

"The CA SHALL revoke a Certificate within 24 hours if one or more of the following occurs"...."The CA obtains evidence that the Subscriber's Private Key corresponding to the Public Key in the Certificate suffered a Key Compromise"

I would like to request GoDaddy revoke the certificate and provide an incident report on this matter.

[Ryan Sleevi]

Do you have a copy of the OCSP response?

With such issues, we may need signed artifacts to demonstrate
non-compliance. For example, it shows as revoked via both OCSP and CRL
for me.

> _______________________________________________
> dev-security-policy mailing list
> dev-secur...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-security-policy

[sandy...@gmail.com]

I actually submitted this post 6 days ago and was only just approved today.. is there a lack of resources approving blog posts? just don't see how it's helpful when posts show up so late.

As noted, I sampled the OCSP responder well after 24 hours and the cert had not been revoked yet. I don't have a signed copy to share as i didn't save it but I don't think it's necessary since it still took GoDaddy over 24 hours to revoke. If you compare report timestamp with ocsp timestamp the difference is approximately 28hrs and 48mins.

[Ryan Sleevi]

On Tue, May 19, 2020 at 12:38 PM sandybar497--- via
dev-security-policy <dev-secur...@lists.mozilla.org> wrote:
> I actually submitted this post 6 days ago and was only just approved today.. is there a lack of resources approving blog posts? just don't see how it's helpful when posts show up so late.

It looks like you may be posting through Google Groups, which can
cause moderation delays if you're not signed up through
https://lists.mozilla.org/listinfo/dev-security-policy (Groups is
largely Archives, with some mirroring for posting that can have
hiccups, as you can see)

Certainly, you can always report issues through Bugzilla, as noted at
https://wiki.mozilla.org/CA/Incident_Dashboard , which doesn't have
the same moderation queue.

> As noted, I sampled the OCSP responder well after 24 hours and the cert had not been revoked yet. I don't have a signed copy to share as i didn't save it but I don't think it's necessary since it still took GoDaddy over 24 hours to revoke.

Not trying to suggest it's not the case, but these statements alone
aren't necessarily enough to demonstrate non-compliance. Signed
responses or other evidence are useful, especially when things are "on
the cusp"

> If you compare report timestamp with ocsp timestamp the difference is approximately 28hrs and 48mins.

Can you provide the original message with headers? Either to this or
as an attachment to Bugzilla?

[sandy...@gmail.com]

Here are the original headers (omitting my email)

***

MIME-Version: 1.0
Date: Thu, 7 May 2020 12:07:07 +0000
Message-ID: <CANb+OL=25wrEtLMXSgEbv=6EuDRHgdugr+FY...@mail.gmail.com>
Subject: Certificate Problem Report - compromised key
From: sandy <sandy...@gmail.com>
To: prac...@starfieldtech.com
Content-Type: multipart/mixed; boundary="00000000000092dbd705a50db8c4"
--00000000000092dbd705a50db8c4
Content-Type: multipart/alternative; boundary="00000000000092dbd505a50db8c2"
--00000000000092dbd505a50db8c2
Content-Type: text/plain; charset="UTF-8"
Hello,
Request you revoke the all certificate associated with this
compromised key.
https://crt.sh/?spkisha256=e92984ace6f80c75b092df972962f2d3f1365ba08c8bbf9b98cdf3aec20d2d2d
Attached is a valid CSR produced from the original key as evidence of
compromise. The CSR is referenced with the spki sha256 fingerprint as the
filename.
Per cab-forum guidelines, the cert should be revoked within 24 hours.
- Sandy
--00000000000092dbd505a50db8c2
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
<div dir=3D"ltr">Hello,<br><br>Request you revoke the all certificate assoc=
iated with this compromised=C2=A0key.=C2=A0=C2=A0<br><br><a href=3D"https:/=
/crt.sh/?spkisha256=3De92984ace6f80c75b092df972962f2d3f1365ba08c8bbf9b98cdf=
3aec20d2d2d">https://crt.sh/?spkisha256=3De92984ace6f80c75b092df972962f2d3f=
1365ba08c8bbf9b98cdf3aec20d2d2d</a>=C2=A0=C2=A0<br><br>Attached is a valid =
CSR produced from the original key as evidence of compromise. The CSR is re=
ferenced with the spki sha256 fingerprint as the filename.<br><br>Per cab-f=
orum guidelines, the cert should be revoked within 24 hours.<br><br>- Sandy=
<br></div>
--00000000000092dbd505a50db8c2--
--00000000000092dbd705a50db8c4
Content-Type: application/octet-stream; name="e92984ace6f80c75b092df972962f2d3f1365ba08c8bbf9b98cdf3aec20d2d2d.pem"
Content-Disposition: attachment; filename="e92984ace6f80c75b092df972962f2d3f1365ba08c8bbf9b98cdf3aec20d2d2d.pem"
Content-Transfer-Encoding: base64
X-Attachment-Id: f_k9wq5sjj0
Content-ID: <f_k9wq5sjj0>
--00000000000092dbd705a50db8c4--

***

- sandy

[Matt Palmer]

On Tue, May 19, 2020 at 07:33:00PM -0700, sandybar497--- via dev-security-policy wrote:
> Here are the original headers (omitting my email)
>
> ***
>
> MIME-Version: 1.0
> Date: Thu, 7 May 2020 12:07:07 +0000
> Message-ID: <CANb+OL=25wrEtLMXSgEbv=6EuDRHgdugr+FY...@mail.gmail.com>
> Subject: Certificate Problem Report - compromised key
> From: sandy <sandy...@gmail.com>

[...]
> https://crt.sh/?spkisha256=e92984ace6f80c75b092df972962f2d3f1365ba08c8bbf9b98cdf3aec20d2d2d

crt.sh sez:

Revoked (cessationOfOperation) 2020-05-08 16:55:17 UTC

Got to say, that definitely does look like over 24 hours from e-mail to
revocation. Unfortunately, because you're using gmail, it's tricky to be
able to demonstrate when GoDaddy *actually* received the e-mail -- I don't
know of a way to get at the MTA logs to show when it was delivered to the
remote MTA.

I'd be curious to hear from GoDaddy as to why the revocation reason here is
marked as "cessationOfOperation", rather than "keyCompromise". That
seems... fishy.

> Content-Type: application/octet-stream; name="e92984ace6f80c75b092df972962f2d3f1365ba08c8bbf9b98cdf3aec20d2d2d.pem"
> Content-Disposition: attachment; filename="e92984ace6f80c75b092df972962f2d3f1365ba08c8bbf9b98cdf3aec20d2d2d.pem"
> Content-Transfer-Encoding: base64
> X-Attachment-Id: f_k9wq5sjj0
> Content-ID: <f_k9wq5sjj0>

Somewhere along the line this got lost. It'd be good to have a copy of it,
for completeness. Since it's in PEM format, you can include it in the body
of an e-mail -- the Mozilla lists are a bit finicky with attachments.

- Matt

[sandy...@gmail.com]

I had received a auto-confirmation email from GoDaddy [donot...@secureserver.net] just one minute after sending my report, the email reply contained case incident id 41854028.

Here is a copy of the evidence of compromise sent along with my report (PEM encoded CSR signed from original private key).

-----BEGIN CERTIFICATE REQUEST-----
MIICozCCAYsCAQAwXjEYMBYGA1UECgwPQ29tcHJvbWlzZWQgS2V5MUIwQAYDVQQD
DDlUaGUga2V5IHRoYXQgc2lnbmVkIHRoaXMgQ1NSIGhhcyBiZWVuIHB1YmxpY2x5
IGRpc2Nsb3NlZC4wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDuGNUD
DTHpFfAEJj5h9bDHitmui7uJGaVybhxYzdoEvxzeNAhBESQHMfRGyhr2cvHeWlfX
G8j1ZjimEEdzF1E14Jqx6duWYyowe4Crc3lFZduisw149ASzwu4A6CDR00zyeb7L
xpnthpvSSGzJ8iMZEEC4odsMxOlO0yoEwd7ketlybn6jLNpUIMii/bolbLvY9bMg
5wPMTVyrhLoum+KP+DSP7TuZx41LAeBjhRaYZAXHtrcQAjKIJ+6YjKv/uYdDREKq
dw2accMGrsWcSKM/bKuA+l/8+Pye/aMnSo4b7dNzILWGkJC0Ipdg99bkPtx/bWTX
NXZfe+EcsQdJK5rNAgMBAAGgADANBgkqhkiG9w0BAQsFAAOCAQEAKYleYx/U6n2v
Xai5ckvujoodT5rrINzjI/wuohioys0M8keN5Iq9zbcfX1orHPBhG8+c1pFTzmjh
TNhAyz/aur3LqXJ8wijZIDky27WFvjw98jQB6n6Di+LHWHFbFmwz/mHwGIDDqo7c
Oy8yG0gXOPOnMwL7VDctgu7/Kk/JX8mcWLbISyCr2CnljOH4nQOEz3j3+MhLZPg7
NcQSq52oiGCPWAEnQ4aJI7vdhY8TWab82sLDO6qy61wek4hp7z1nVctpJkQvBORi
F76ayXlgL4G6oCG12VVloK52Ti8kk15HB6YFhD/1mz0fUyOTe/PzedOBaPhiAvv2
FPDcLgBXlg==
-----END CERTIFICATE REQUEST-----

Requesting GoDaddy to provide an incident report for this matter.

- sandy

[Daniela Hood]

Hello Sandy,

GoDaddy received an email on Friday, May 7, 2020 12:06 UTC, reporting a key compromise, by Sandy. Once received our team started working on making sure that the certificate had indeed a compromised key, the investigation on the certificate finished at that same day Friday, May 7th between 16:54 UTC and 16:55 UTC.
After that we followed the Baseline Requirements 4.9.1 That says: "The CA obtains evidence that the Subscriber's Private Key corresponding to the Public Key in the Certificate suffered a Key Compromise;" We obtained the evidence that the key was compromised when we finished our investigation at 16:55 UTC, that was the time we set 24 hours revocation of the certificate, the same was revoked at May 8th at 16:55 UTC.
We communicated with the reporter as soon as we completed our investigation and informed that the affected certificate would be revoked strictly within 24 hours which we have done and can be confirmed here: https://crt.sh/?id=2366734355

Lastly, GoDaddy take key compromises very seriously and recognize the importance to the industry and health of the ecosystem.

Thank you,

Daniela Hood
GoDaddy

[Kurt Roeckx]

On Thu, May 21, 2020 at 02:01:49PM -0700, Daniela Hood via dev-security-policy wrote:
> Hello Sandy,
>
> GoDaddy received an email on Friday, May 7, 2020 12:06 UTC, reporting a key compromise, by Sandy. Once received our team started working on making sure that the certificate had indeed a compromised key, the investigation on the certificate finished at that same day Friday, May 7th between 16:54 UTC and 16:55 UTC.
> After that we followed the Baseline Requirements 4.9.1 That says: "The CA obtains evidence that the Subscriber's Private Key corresponding to the Public Key in the Certificate suffered a Key Compromise;" We obtained the evidence that the key was compromised when we finished our investigation at 16:55 UTC, that was the time we set 24 hours revocation of the certificate, the same was revoked at May 8th at 16:55 UTC.
> We communicated with the reporter as soon as we completed our investigation and informed that the affected certificate would be revoked strictly within 24 hours which we have done and can be confirmed here: https://crt.sh/?id=2366734355

>From what I understand, you received the evidence at May 7, 2020
12:06 UTC, but it took you until 16:55 UTC to confirm that the
evidence you've received was valid.

I think the 24 hour starts at the time you receive the evidence, not
the time that you confirm the evidence is valid. Otherwise you can
just delay looking at the mail for say a week, and still claim
that you revoked it in 24 hours.


Kurt

[Jeremy Rowley]

Yes - that's been well established. See https://bugzilla.mozilla.org/show_bug.cgi?id=1639801 (where Ryan reminded me that this has been discussed and resolved with actual language in the BRs)

[Matt Palmer]

On Thu, May 21, 2020 at 02:01:49PM -0700, Daniela Hood via dev-security-policy wrote:

> After that we followed the Baseline Requirements 4.9.1 That says: "The CA

> obtains evidence that the Subscriber's Private Key corresponding to the

> Public Key in the Certificate suffered a Key Compromise;" We obtained the
> evidence that the key was compromised when we finished our investigation
> at 16:55 UTC, that was the time we set 24 hours revocation of the
> certificate, the same was revoked at May 8th at 16:55 UTC.

BRs 4.9.5:

"The period from receipt of the Certificate Problem Report or
revocation-related notice to published revocation MUST NOT exceed the time
frame set forth in Section 4.9.1.1".

> can be confirmed here: https://crt.sh/?id=2366734355

Can you explain why the revocation reason is "cessationOfOperation", rather
than "keyCompromise"? To not provide a revocation reason at all is one
thing, but to indicate a factually incorrect one is... something else
entirely.

- Matt

[Daniela Hood]

Hello,

Thank you for all the comments in this thread. We filed an incident report related to the revocation timing that can be followed here: https://bugzilla.mozilla.org/show_bug.cgi?id=1640310. We also identified the error in revocation reason as a user error, corrected the error and provided feedback to the employee.

Daniela Hood
GoDaddy

-----Original Message-----
From: dev-security-policy <dev-security-...@lists.mozilla.org> On Behalf Of Matt Palmer via dev-security-policy
Sent: Thursday, May 21, 2020 6:32 PM
To: dev-secur...@lists.mozilla.org
Subject: Re: GoDaddy: Failure to revoke certificate with compromised key within 24 hours

Notice: This email is from an external sender.

On Thu, May 21, 2020 at 02:01:49PM -0700, Daniela Hood via dev-security-policy wrote:

> After that we followed the Baseline Requirements 4.9.1 That says: "The

> CA obtains evidence that the Subscriber's Private Key corresponding to

> the Public Key in the Certificate suffered a Key Compromise;" We
> obtained the evidence that the key was compromised when we finished
> our investigation at 16:55 UTC, that was the time we set 24 hours
> revocation of the certificate, the same was revoked at May 8th at 16:55 UTC.

BRs 4.9.5:

"The period from receipt of the Certificate Problem Report or revocation-related notice to published revocation MUST NOT exceed the time frame set forth in Section 4.9.1.1".

> can be confirmed here: https://crt.sh/?id=2366734355

Can you explain why the revocation reason is "cessationOfOperation", rather than "keyCompromise"? To not provide a revocation reason at all is one thing, but to indicate a factually incorrect one is... something else entirely.

- Matt

[Nick Lamb]

On Fri, 22 May 2020 22:48:42 +0000
Daniela Hood via dev-security-policy

<dev-secur...@lists.mozilla.org> wrote:

> Hello,
>
> Thank you for all the comments in this thread. We filed an incident
> report related to the revocation timing that can be followed here:
> https://bugzilla.mozilla.org/show_bug.cgi?id=1640310. We also
> identified the error in revocation reason as a user error, corrected
> the error and provided feedback to the employee.

In addition to Ryan's concerns about the supposed ambiguity of a
pretty clear rule in the BRs I am as always interested in what can be
learned from incidents that might help everybody else.


What mechanism, if any, would have detected this "user error" in the
absence of a report by a third party to m.d.s.policy ?

Every CA has humans doing stuff, and humans make mistakes. Whether
that's a Let's Encrypt team member fat-fingering a server configuration
or a Symantec employee using google.com rather than a Symantec name for
a test. But even though it's expected for humans to make mistakes, we
demand more of the Certificate Authority than we could ask of one human.

Where humans are necessary they will make mistakes and so you need
compensating controls. In this case that might mean reviewing critical
work done by humans. Depending on volume that might mean a second
person looks at every revocation, or it might mean a sample is examined
once a week for example.

I'd like to see incident reports like this not stop at "user error" for
this reason. Why wasn't the "user error" caught? What (other than
"feedback to the employee") prevents it happening again ?


Nick.

[Daniela Hood]

Hello,

We have made improvements on our problem reporting process, provided more training to our agents and made changes in our system to assure that such error would not happen again. We will keep on working with the community to find solutions that could benefit the industry, in hopes to avoid such errors from occurring.

Daniela Hood
GoDaddy


-----Original Message-----
From: Nick Lamb <n...@tlrmx.org>
Sent: Friday, May 22, 2020 4:50 PM
To: dev-secur...@lists.mozilla.org
Cc: Daniela Hood <dxh...@godaddy.com>
Subject: Re: GoDaddy: Failure to revoke certificate with compromised key within 24 hours

Notice: This email is from an external sender.

[Ryan Sleevi]

Thank you for your update.

This does not appear to answer the questions raised. I would appreciate if
GoDaddy shared a more meaningful response, in line with addressing the
concerns Nick raised, as well as the outstanding matters on the bug.

In particular, this response fails to analyze what went wrong, what has
been done systemically by GoDaddy to prevent future incidents, and what are
practices other CAs should consider to prevent similar incidents.

In addition to the outstanding question from Nick, for this sort of
response to be acceptable at all, more detail is needed:
- What improvements were made, and why?
- What training was done? What was changed? How is the training performed
and evaluated? Why did the previous training fail to address this? Why is
training seen as an acceptable answer, given previous training failed? What
is done to support and monitor compliance to training?
- What changes were made to the system? Why would they address this issue?
How does that relate to why the issue?

In 2020, publicly trusted CAs should not be expecting that such “incident
reports”, if this can even be called that, are sufficient. As stewards of
the trust placed in them by Mozilla and the broader community, and by other
root stores, substantive and highly detailed, technical responses are
expected. The goal of these reports is to both simultaneously address
concerns caused by the failure to adhere to expectations and to help ensure
that all CAs have an opportunity to learn from and implement similar
mechanisms. If the response does not have sufficient information to allow
for an independent implementation of whatever mitigations, and to the same
level of assurance, it quite simply is not a response that meets
expectations. We need to be able to work together, as an industry, to move
things forward, and that requires complete transparency.

On Thu, May 28, 2020 at 7:55 PM Daniela Hood via dev-security-policy <

[Daniela Hood]

GoDaddy acknowledges the inquiry and we will work to have a response to the community by Wednesday, June 3rd.


Daniela Hood
GoDaddy | Compliance Manager
[https://email-sig.gd-resources.net/img/godaddy-logo-outline.png]
+16026881766
Gilbert, Arizona, United States
dxh...@godaddy.com<mailto:dxh...@godaddy.com>

From: Ryan Sleevi <ry...@sleevi.com>
Sent: Friday, May 29, 2020 7:52 AM
To: Daniela Hood <dxh...@godaddy.com>

Cc: dev-secur...@lists.mozilla.org
Subject: Re: GoDaddy: Failure to revoke certificate with compromised key within 24 hours

Notice: This email is from an external sender.

Thank you for your update.

This does not appear to answer the questions raised. I would appreciate if GoDaddy shared a more meaningful response, in line with addressing the concerns Nick raised, as well as the outstanding matters on the bug.

In particular, this response fails to analyze what went wrong, what has been done systemically by GoDaddy to prevent future incidents, and what are practices other CAs should consider to prevent similar incidents.

In addition to the outstanding question from Nick, for this sort of response to be acceptable at all, more detail is needed:
- What improvements were made, and why?
- What training was done? What was changed? How is the training performed and evaluated? Why did the previous training fail to address this? Why is training seen as an acceptable answer, given previous training failed? What is done to support and monitor compliance to training?
- What changes were made to the system? Why would they address this issue? How does that relate to why the issue?

In 2020, publicly trusted CAs should not be expecting that such “incident reports”, if this can even be called that, are sufficient. As stewards of the trust placed in them by Mozilla and the broader community, and by other root stores, substantive and highly detailed, technical responses are expected. The goal of these reports is to both simultaneously address concerns caused by the failure to adhere to expectations and to help ensure that all CAs have an opportunity to learn from and implement similar mechanisms. If the response does not have sufficient information to allow for an independent implementation of whatever mitigations, and to the same level of assurance, it quite simply is not a response that meets expectations. We need to be able to work together, as an industry, to move things forward, and that requires complete transparency.

On Thu, May 28, 2020 at 7:55 PM Daniela Hood via dev-security-policy <dev-secur...@lists.mozilla.org<mailto:dev-secur...@lists.mozilla.org>> wrote:
Hello,

We have made improvements on our problem reporting process, provided more training to our agents and made changes in our system to assure that such error would not happen again. We will keep on working with the community to find solutions that could benefit the industry, in hopes to avoid such errors from occurring.

Daniela Hood
GoDaddy


-----Original Message-----
From: Nick Lamb <n...@tlrmx.org<mailto:n...@tlrmx.org>>
Sent: Friday, May 22, 2020 4:50 PM
To: dev-secur...@lists.mozilla.org<mailto:dev-secur...@lists.mozilla.org>
Cc: Daniela Hood <dxh...@godaddy.com<mailto:dxh...@godaddy.com>>
Subject: Re: GoDaddy: Failure to revoke certificate with compromised key within 24 hours

Notice: This email is from an external sender.



On Fri, 22 May 2020 22:48:42 +0000
Daniela Hood via dev-security-policy

<dev-secur...@lists.mozilla.org<mailto:dev-secur...@lists.mozilla.org>> wrote:

> Hello,
>
> Thank you for all the comments in this thread. We filed an incident
> report related to the revocation timing that can be followed here:
> https://bugzilla.mozilla.org/show_bug.cgi?id=1640310. We also
> identified the error in revocation reason as a user error, corrected
> the error and provided feedback to the employee.

In addition to Ryan's concerns about the supposed ambiguity of a pretty clear rule in the BRs I am as always interested in what can be learned from incidents that might help everybody else.


What mechanism, if any, would have detected this "user error" in the absence of a report by a third party to m.d.s.policy ?

Every CA has humans doing stuff, and humans make mistakes. Whether that's a Let's Encrypt team member fat-fingering a server configuration or a Symantec employee using google.com<http://google.com> rather than a Symantec name for a test. But even though it's expected for humans to make mistakes, we demand more of the Certificate Authority than we could ask of one human.

Where humans are necessary they will make mistakes and so you need compensating controls. In this case that might mean reviewing critical work done by humans. Depending on volume that might mean a second person looks at every revocation, or it might mean a sample is examined once a week for example.

I'd like to see incident reports like this not stop at "user error" for this reason. Why wasn't the "user error" caught? What (other than "feedback to the employee") prevents it happening again ?


Nick.

_______________________________________________
dev-security-policy mailing list
dev-secur...@lists.mozilla.org<mailto:dev-secur...@lists.mozilla.org>
https://lists.mozilla.org/listinfo/dev-security-policy

[Daniela Hood]

Hello all,

We appreciate the concerns and your questions to this thread. GoDaddy takes such issues very seriously and recognize the importance to the industry and health of the ecosystem.

​In the case where the user selected the incorrect revocation reason, we identified the error shortly before it was reported as part of a standard review. We reviewed the error with the user and corrected it the same day, per our procedure. Upon reviewing with the user, we identified an opportunity to enhance our process through additional visual cues to enable the agent to perform a final review prior to committing a revocation. Additionally, our process includes team calibrations where prior errors are used as training opportunities for the entire team. We also include any areas that have changed or where we notice an increased instance of errors in our annual training program. All of these efforts in combination help us to keep the instance of errors down and address situations as they arise.

We hope this information helps address concerns regarding this error.

Thank you,

_______________________________________________
dev-security-policy mailing list
dev-secur...@lists.mozilla.org<mailto:dev-secur...@lists.mozilla.org>
https://lists.mozilla.org/listinfo/dev-security-policy

[Cynthia Revström]

Hi Daniela,

Sorry if I am missing something, but what do you mean by "incorrect
revocation reason"?
The first sentence in the email sent to you by Sandy sounds pretty clear to
me "Request you revoke the all certificate associated with this
compromised key".

Also I don't see how any of what you have said you have done would help to
prevent it from taking over 24h again, could you please elaborate?

- Cynthia

> _______________________________________________
> dev-security-policy mailing list
> dev-secur...@lists.mozilla.org<mailto:
> dev-secur...@lists.mozilla.org>
> https://lists.mozilla.org/listinfo/dev-security-policy

[Daniela Hood]

Hello Cynthia,
Our last post was intended to respond to the question immediately above it with regard to why the revoked certificate showed a revocation reason of 'cessation of operations' rather than 'compromise.' Additional information regarding what we are doing to ensure certificates are revoked within 24 hours of receipt of notice that a certificate is compromised can be found in the bug report: https://bugzilla.mozilla.org/show_bug.cgi?id=1640310
Daniela Hood
GoDaddy

Daniela Hood
GoDaddy | Compliance Manager
[https://email-sig.gd-resources.net/img/godaddy-logo-outline.png]
+16026881766
Gilbert, Arizona, United States
dxh...@godaddy.com<mailto:dxh...@godaddy.com>

From: Cynthia Revström <m...@cynthia.re>
Sent: Wednesday, June 3, 2020 1:52 PM
To: Daniela Hood <dxh...@godaddy.com>
Cc: ry...@sleevi.com; dev-secur...@lists.mozilla.org
Subject: Re: GoDaddy: Failure to revoke certificate with compromised key within 24 hours

Notice: This email is from an external sender.

Hi Daniela,

Sorry if I am missing something, but what do you mean by "incorrect revocation reason"?
The first sentence in the email sent to you by Sandy sounds pretty clear to me "Request you revoke the all certificate associated with this
compromised key".

Also I don't see how any of what you have said you have done would help to prevent it from taking over 24h again, could you please elaborate?

- Cynthia

On Wed, Jun 3, 2020 at 8:13 PM Daniela Hood via dev-security-policy <dev-secur...@lists.mozilla.org<mailto:dev-secur...@lists.mozilla.org>> wrote:
Hello all,

We appreciate the concerns and your questions to this thread. GoDaddy takes such issues very seriously and recognize the importance to the industry and health of the ecosystem.

​In the case where the user selected the incorrect revocation reason, we identified the error shortly before it was reported as part of a standard review. We reviewed the error with the user and corrected it the same day, per our procedure. Upon reviewing with the user, we identified an opportunity to enhance our process through additional visual cues to enable the agent to perform a final review prior to committing a revocation. Additionally, our process includes team calibrations where prior errors are used as training opportunities for the entire team. We also include any areas that have changed or where we notice an increased instance of errors in our annual training program. All of these efforts in combination help us to keep the instance of errors down and address situations as they arise.

We hope this information helps address concerns regarding this error.

Thank you,

Daniela Hood
GoDaddy


-----Original Message-----
From: dev-security-policy <dev-security-...@lists.mozilla.org<mailto:dev-security-...@lists.mozilla.org>> On Behalf Of Daniela Hood via dev-security-policy
Sent: Friday, May 29, 2020 9:16 PM
To: 'ry...@sleevi.com<mailto:ry...@sleevi.com>' <ry...@sleevi.com<mailto:ry...@sleevi.com>>
Cc: dev-secur...@lists.mozilla.org<mailto:dev-secur...@lists.mozilla.org>
Subject: RE: GoDaddy: Failure to revoke certificate with compromised key within 24 hours

Notice: This email is from an external sender.



GoDaddy acknowledges the inquiry and we will work to have a response to the community by Wednesday, June 3rd.


Daniela Hood
GoDaddy | Compliance Manager
[https://email-sig.gd-resources.net/img/godaddy-logo-outline.png]
+16026881766
Gilbert, Arizona, United States

dxh...@godaddy.com<mailto:dxh...@godaddy.com><mailto:dxh...@godaddy.com<mailto:dxh...@godaddy.com>>

From: Ryan Sleevi <ry...@sleevi.com<mailto:ry...@sleevi.com>>
Sent: Friday, May 29, 2020 7:52 AM
To: Daniela Hood <dxh...@godaddy.com<mailto:dxh...@godaddy.com>>
Cc: dev-secur...@lists.mozilla.org<mailto:dev-secur...@lists.mozilla.org>
Subject: Re: GoDaddy: Failure to revoke certificate with compromised key within 24 hours

Notice: This email is from an external sender.


Thank you for your update.

This does not appear to answer the questions raised. I would appreciate if GoDaddy shared a more meaningful response, in line with addressing the concerns Nick raised, as well as the outstanding matters on the bug.

In particular, this response fails to analyze what went wrong, what has been done systemically by GoDaddy to prevent future incidents, and what are practices other CAs should consider to prevent similar incidents.

In addition to the outstanding question from Nick, for this sort of response to be acceptable at all, more detail is needed:
- What improvements were made, and why?
- What training was done? What was changed? How is the training performed and evaluated? Why did the previous training fail to address this? Why is training seen as an acceptable answer, given previous training failed? What is done to support and monitor compliance to training?
- What changes were made to the system? Why would they address this issue? How does that relate to why the issue?

In 2020, publicly trusted CAs should not be expecting that such “incident reports”, if this can even be called that, are sufficient. As stewards of the trust placed in them by Mozilla and the broader community, and by other root stores, substantive and highly detailed, technical responses are expected. The goal of these reports is to both simultaneously address concerns caused by the failure to adhere to expectations and to help ensure that all CAs have an opportunity to learn from and implement similar mechanisms. If the response does not have sufficient information to allow for an independent implementation of whatever mitigations, and to the same level of assurance, it quite simply is not a response that meets expectations. We need to be able to work together, as an industry, to move things forward, and that requires complete transparency.

On Thu, May 28, 2020 at 7:55 PM Daniela Hood via dev-security-policy <dev-secur...@lists.mozilla.org<mailto:dev-secur...@lists.mozilla.org><mailto:dev-secur...@lists.mozilla.org<mailto:dev-secur...@lists.mozilla.org>>> wrote:
Hello,

We have made improvements on our problem reporting process, provided more training to our agents and made changes in our system to assure that such error would not happen again. We will keep on working with the community to find solutions that could benefit the industry, in hopes to avoid such errors from occurring.

Daniela Hood
GoDaddy


-----Original Message-----
From: Nick Lamb <n...@tlrmx.org<mailto:n...@tlrmx.org><mailto:n...@tlrmx.org<mailto:n...@tlrmx.org>>>
Sent: Friday, May 22, 2020 4:50 PM
To: dev-secur...@lists.mozilla.org<mailto:dev-secur...@lists.mozilla.org><mailto:dev-secur...@lists.mozilla.org<mailto:dev-secur...@lists.mozilla.org>>
Cc: Daniela Hood <dxh...@godaddy.com<mailto:dxh...@godaddy.com><mailto:dxh...@godaddy.com<mailto:dxh...@godaddy.com>>>
Subject: Re: GoDaddy: Failure to revoke certificate with compromised key within 24 hours

Notice: This email is from an external sender.



On Fri, 22 May 2020 22:48:42 +0000
Daniela Hood via dev-security-policy

<dev-secur...@lists.mozilla.org<mailto:dev-secur...@lists.mozilla.org><mailto:dev-secur...@lists.mozilla.org<mailto:dev-secur...@lists.mozilla.org>>> wrote:

> Hello,
>
> Thank you for all the comments in this thread. We filed an incident
> report related to the revocation timing that can be followed here:
> https://bugzilla.mozilla.org/show_bug.cgi?id=1640310. We also
> identified the error in revocation reason as a user error, corrected
> the error and provided feedback to the employee.

In addition to Ryan's concerns about the supposed ambiguity of a pretty clear rule in the BRs I am as always interested in what can be learned from incidents that might help everybody else.


What mechanism, if any, would have detected this "user error" in the absence of a report by a third party to m.d.s.policy ?

Every CA has humans doing stuff, and humans make mistakes. Whether that's a Let's Encrypt team member fat-fingering a server configuration or a Symantec employee using google.com<http://google.com><http://google.com> rather than a Symantec name for a test. But even though it's expected for humans to make mistakes, we demand more of the Certificate Authority than we could ask of one human.

Where humans are necessary they will make mistakes and so you need compensating controls. In this case that might mean reviewing critical work done by humans. Depending on volume that might mean a second person looks at every revocation, or it might mean a sample is examined once a week for example.

I'd like to see incident reports like this not stop at "user error" for this reason. Why wasn't the "user error" caught? What (other than "feedback to the employee") prevents it happening again ?


Nick.

_______________________________________________
dev-security-policy mailing list
dev-secur...@lists.mozilla.org<mailto:dev-secur...@lists.mozilla.org><mailto:dev-secur...@lists.mozilla.org<mailto:dev-secur...@lists.mozilla.org>>
https://lists.mozilla.org/listinfo/dev-security-policy
_______________________________________________
dev-security-policy mailing list
dev-secur...@lists.mozilla.org<mailto:dev-secur...@lists.mozilla.org>
https://lists.mozilla.org/listinfo/dev-security-policy
_______________________________________________
dev-security-policy mailing list
dev-secur...@lists.mozilla.org<mailto:dev-secur...@lists.mozilla.org>
https://lists.mozilla.org/listinfo/dev-security-policy