GoDaddy: Failure to revoke key-compromised certificate within 24 hours

[Matt Palmer]

Following a problem report sent to the CPS-specified address, evidence of
key compromise for a private key used in a certificate issued by GoDaddy has
not been revoked within the 24 hour timeframe required by the Baseline
Requirements s4.9.1.1.

Whilst GoDaddy did provide a response within 24 hours, I do not believe it
meets the spirit of the requirement to provide a "preliminary report", as
per BRs s4.9.5.

A detailed timeline of communications, as well as my specific concerns about
GoDaddy's response, are given below.

2020-03-08 02:47:54Z Problem report sent:

-----8<-----
Date: Sun, 8 Mar 2020 13:47:54 +1100
From: Matt Palmer <mpa...@hezmatt.org>
To: prac...@starfieldtech.com
Subject: Problem Report for certificate(s) with compromised private key

One or more certificates issued by your CA are using a private key which has
been publicly disclosed. The list of affected certificates can be retrieved
from

https://crt.sh/?spkisha256=c7701317f3cdab8d47ae581dc9e92ac249e9cea65765a4e4ef1100246d003f16

Included below is a CSR, signed by the compromised private key,
demonstrating proof of possession:

-----BEGIN CERTIFICATE REQUEST-----
MIIC0TCCAbkCAQAwgYsxaTBnBgNVBAMMYFRoZSBrZXkgdGhhdCBzaWduZWQgdGhp
cyBDU1IgaGFzIGJlZW4gcHVibGljbHkgZGlzY2xvc2VkLiBJdCBzaG91bGQgbm90
IGJlIHVzZWQgZm9yIGFueSBwdXJwb3NlLjEeMBwGA1UECgwVaHR0cHM6Ly9wd25l
ZGtleXMuY29tMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAmIykJVXM
hnFnYQjuDD3uQw2DU7x4taco5sV0s5yzhVWoEOuBRjikAfXU0umzYv1mx80SzXy8
092U9E0noLVtRbZS1Z53WJonUcvtxyqYt1syCM2OuUEYwXoSMmBIK11YZSaBK91/
HECnJbJ5gAqddUUNcsVcKj9LZOEwiwQnVan42FCqLYJT6w9t2rbcMPJyajRCHaHI
do0KkzWY1cg4oc4S21/7sszVz4IFLMNvT5NoJ768D6atkqTVduMsOPEaourni8kl
4ANE3dCkyYh24/Y0tbrB6Vo2lTTVOUpYEfMPDgkaVMtEPtU03SfmfqZrJMp2KwmG
G4rgFne6SZ1yMQIDAQABoAAwDQYJKoZIhvcNAQELBQADggEBADTZKcOXd6cf/Zs/
keZv8hoTPalYsIjfBlsDyhkdzIRXlBhow+bJQxgA1X49E1qxht7YAuKuzJqH/1KE
18S7+e4krWSh6Thp9bv8UrRWVCyGGoXe0DQrmVQuA9jb32i0S8SioyQ/rrnQZM5L
gb0sFb+keHxOYFK7vaK+Iq0GTVOLq2M0dOzvsvzw1u1BfXLxXI9UryxDkKb6I9uy
QWfigG50G7CifguPyxeRnK4Js/8cAzXREJF1D9UUDj6rByEWSxEEnc9NyAqOmjsS
1Q8DxrSK6uMOPy3icngD+Ykq6PIWKeXe0eTme0W7feGjrsmVZChhvrbFjkPaFmrZ
83FAPKc=
-----END CERTIFICATE REQUEST-----

Please revoke all affected certificates within 24 hours, as per the Baseline
Requirements.

- Matt
----->8-----

2020-03-08 02:48:01Z E-mail is accepted by remote mail server:

-----8<-----
Mar 8 02:48:01 minotaur postfix/smtp[2204]: 9D7421859AB:
to=<prac...@starfieldtech.com>,
relay=smtp.secureserver.net[68.178.213.37]:25, delay=3.2,
delays=0.45/0.01/1.4/1.3, dsn=2.0.0, status=sent (250 2.0.0 Alyxjg6mq8nXu -
Alyxjg6mq8nXuAlyyjakiR mail accepted for delivery)
----->8-----

2020-03-08 02:48:42Z Auto-ack response from dono...@secureserver.net,
whose substantive content, once the HTML gak has been filed off, is as
follows:

-----8<-----
Your inquiry has been received. You should expect a response
within 72 hours.

This is your Incident ID: 41360505

If you wish to speak with a customer service representative, call +1 (480)
505-8825 and reference the Incident ID above.

Thanks,
Customer Service
----->8-----

2020-03-08 22:33:51Z Followup e-mail received from
donot...@secureserver.net:

-----8<-----
We have received a certificate problem report for tell.gao.gov and are
required to investigate the facts and circumstances related to this problem
report. Contingent on our findings, the affected certificate(s) may require
revocation within a strict timeline of 24 hours or up to 5 days, depending
upon severity. Consequently, the current certificate(s) will cease to
function immediately upon revocation.

Our findings regarding this problem report conclude that we are unable to
proceed with a revocation as the provided CSR is not conclusive. We are
able to duplicate this CSR without having access to the private key. At
this time, we are requesting additional information from you in the form of
either Option A or Option B below. Please respond within 24 hours to ensure
timely resolution.

Option A: Send us the private key in PEM format.

Option B: Run the command below and send us the output. Please replace
PRIVATEKEY-FILE with the path to the compromised private key. Please be
sure to use the exact same string so we can verify accordingly.
openssl dgst -sha256 -sign PRIVATEKEY-FILE <(echo -n 'compromised') openssl enc -base64

[my original e-mail, with formatting mangled, elided]

If you need to follow up, please reference incident ID 41360505 when you
contact our support team at +1 (480) 505-8825.

Please do not reply to this email. Emails sent to this address will not be
answered.
----->8-----

At of the time of writing, which is approximately 25-1/2 hours after the
initial problem report was sent, https://crt.sh/?id=1166030780&opt=ocsp
shows the certificate as having not been revoked.

I have three concerns about the nature of the GoDaddy response to this
problem report:

1. The CSR I provided in the original report is functionally identical to the
signature that is being requested, except it is stronger because a CSR
contains a signature over the public key which is compromised, rather
than just being a signature over a single very short, generic string. It
most certainly can be validated using openssl.

2. Even if I hadn't provided verifiable proof of compromise initially, the
command which was provided by GoDaddy support does not actually work.
Providing plausible but ineffective methods of providing evidence of key
compromise would hypothetically be a fantastic way to drag out the
revocation process, were any CA willing to engage in such shenanigans.

(In case anyone's wondering, I checked *very* carefully that the command
I have pasted from the response above is exactly what is rendered in my
browser).

3. Even if I were to fix the command provided, there is no practical way to
provide the information requested to GoDaddy. The e-mail was sent from
`donot...@secureserver.net`, and the body of the e-mail specifically
states that I should not reply. The only approved method of response is
to call a US phone number. Making an international phone call to read
out a string of base64 characters does not strike me as an appropriate
means of providing evidence of key compromise.

I would appreciate a response from GoDaddy regarding this failure to abide
by the BRs, if one is considered warranted by the CA Policy module owner
and/or peers.

- Matt

[Joanna Fox]

Matt,

Thank you for sharing your experience with our problem reporting mechanism on this forum. It is due to this that we were able to get to the root of the issue. Here is some detail into what we saw.

Yesterday, we launched an investigation which included various members of the team researching this issue. We took this investigation as far as we could with the information we had and concluded that the CSR provided, as we read it, was malformed. We ran this CSR through various tools but were unable to successfully confirm validity.

This morning, based on the statements in this forum, we discovered that our email system had misinterpreted the CSR formatting due to it being pasted in the body of the email. When we fix Base64 encoding, the CSR verifies.

Upon this discovery we have initiated revocation to occur within the guidelines of 24 hours from obtaining evidence that the private key was compromised. We take key compromises very seriously and recognize the importance to the industry and health of the ecosystem.

Lastly, we also noticed that the email you received was malformed, missing some of the required content for the OpenSSL command. This event has led to a review of our email system to learn how we can avoid malformed encoding issues in the future.

Thank you,
Joanna Fox
GoDaddy

[Matt Palmer]

Hi Joanna,

Thanks for responding. When can this list, or Bugzilla, expect GoDaddy's
incident report? Also, for the avoidance of further doubt, can you give an
exact timestamp at which GoDaddy considers that evidence of key compromise
was "obtained" for this certificate?

- Matt

[bif]

Matt,

Voluntarily providing CSR is not an ideal way to prove key compromise, because you could've simply found this CSR somewhere (I know, I know, super unlikely with your Subject... but still could happen.)

And while "compromised" is way too short (one can sign up to 32 bytes using it as a nonce in regular TLS session) to prove the key compromise, in the absence of the actual compromised private key, about the only way to ensure the possession is to get the reporter to sign some data chosen by the CA. It very well may be a random CN in the CSR, or plain old openssl dgst.

[Ryan Sleevi]

On Tue, Mar 10, 2020 at 4:25 PM bif via dev-security-policy <
dev-secur...@lists.mozilla.org> wrote:

> Matt,
>
> Voluntarily providing CSR is not an ideal way to prove key compromise,
> because you could've simply found this CSR somewhere (I know, I know, super
> unlikely with your Subject... but still could happen.)
>

That seems to be making the perfect the enemy of the good. This seems like
a perfectly reasonable thing to allow and accept. If we're concerned about
someone "might" having done something as a reason to reject it, it seems
like we'd need another pass through the BRs validation methods - e.g. to
remove the non-IANA-reserved mail addresses.

I don't disagree that this is a possibility, but the probability of this
possibility seems incredibly low, and the risk of a CA deciding to revoke
on this basis seems immeasurably better than the risk of deciding to not
revoke.

> And while "compromised" is way too short (one can sign up to 32 bytes
> using it as a nonce in regular TLS session) to prove the key compromise, in
> the absence of the actual compromised private key, about the only way to
> ensure the possession is to get the reporter to sign some data chosen by
> the CA. It very well may be a random CN in the CSR, or plain old openssl
> dgst.
>

I'm concerned about CAs disproportionately adding burdens to reporters of
compromise. For example, your 'compromised' case doesn't really make sense.
The string 'compromised', or even the hash of the string 'compromised',
would not in and of itself be a sufficient TLS transcript. I agree with you
that one can't simply /look/ for the string 'compromised' in the unsigned
data, where that signed data is a TLS transcript, but that's not what you
really described, and that distinction seems important.

I also disgree with, and have concerns with, the CA being able to direct
the signing of the data. I think the ecosystem learned a lot from the
Heartbleed scenario, of "We're not affected, we're not affected... oh crap,
we're affected".

I'm sympathetic to CAs wanting to filter out the noise of shoddy reports
and shenanigans, but I'm also highly suspicious of CAs that put too
unreasonable an onus on reporters. It seems, in the key compromise case,
the benefit of the doubt should largely deal with the reporter. If we saw
some quantifiable increase in hijinks and misrevocations, there are a
myriad of ways to deal with that. The most effective of these reasons seems
to be facilitating rapid replacement of certificates, rather than
preferring ossification.

[Nicholas Knight]

On Tuesday, March 10, 2020 at 1:25:21 PM UTC-7, bif wrote:
> Matt,
>
> Voluntarily providing CSR is not an ideal way to prove key compromise, because you could've simply found this CSR somewhere (I know, I know, super unlikely with your Subject... but still could happen.)
>

While a CSR isn't particularly sensitive in itself, people don't often go around posting them publicly. One of the most likely places to find it is on the machine it was generated on, indicating that the machine holding the private key was probably breached, or somewhere in a CA's infrastructure, indicating the CA was breached. Even if the person providing the CSR *doesn't*, in fact, have the private key, I'd be extremely worried about who *does*.

[Piotr Kucharski]

On Tue, 10 Mar 2020 at 22:19, Ryan Sleevi <ry...@sleevi.com> wrote:

>
>
> On Tue, Mar 10, 2020 at 4:25 PM bif via dev-security-policy <

> dev-secur...@lists.mozilla.org> wrote:
>
>> Matt,
>>
>> Voluntarily providing CSR is not an ideal way to prove key compromise,
>> because you could've simply found this CSR somewhere (I know, I know, super
>> unlikely with your Subject... but still could happen.)
>>
>

> That seems to be making the perfect the enemy of the good.
>

Most likely :) Take it as more general musings for having generally
accepted recipes for providing proof of key possession, using which would
not allow CAs to wiggle out of responsibility of revocation.

> This seems like a perfectly reasonable thing to allow and accept. If we're
> concerned about someone "might" having done something as a reason to reject
> it, it seems like we'd need another pass through the BRs validation methods
> - e.g. to remove the non-IANA-reserved mail addresses.
>

Sure, as long as CN is long enough, and CSR is clean, ie. no
extensions allowing to fit, say, collision attacks.

>
> I don't disagree that this is a possibility, but the probability of this
> possibility seems incredibly low, and the risk of a CA deciding to revoke
> on this basis seems immeasurably better than the risk of deciding to not
> revoke.
>

As a CA I would most likely accept this proof, considering the incredibly
low false positive possibility. :)
All I'm saying is that proof of key possession is not clearly defined in
the absence of said key being shared with a CA.

I am totally against putting unreasonable onus on reporters! But hopefully
you agree that CAs should strive for zero false positives in revocations.

[Ryan Sleevi]

On Tue, Mar 10, 2020 at 5:56 PM Piotr Kucharski <b...@google.com> wrote:

> I'm sympathetic to CAs wanting to filter out the noise of shoddy reports
>> and shenanigans, but I'm also highly suspicious of CAs that put too
>> unreasonable an onus on reporters. It seems, in the key compromise case,
>> the benefit of the doubt should largely deal with the reporter. If we saw
>> some quantifiable increase in hijinks and misrevocations, there are a
>> myriad of ways to deal with that. The most effective of these reasons seems
>> to be facilitating rapid replacement of certificates, rather than
>> preferring ossification.
>>
>
> I am totally against putting unreasonable onus on reporters! But hopefully
> you agree that CAs should strive for zero false positives in revocations.
>

I'd happily take a 95% false positive of revocations if there were 0%
impact in the revocation (e.g. due to easy replacement). I'm mainly
hesitant to setting up a system of 0% false positives but which has a 5%
false negative.

That's why I'm less excited for standard systems of signaling revocation
(not that there isn't some value!), and more keen on systems that make
revocation easier, quicker, and less-impactful. That's obviously Hard Work,
but that's the exciting part of working in PKI. Everything is Hard Work
these days :D

[Matt Palmer]

On Tue, Mar 10, 2020 at 01:25:11PM -0700, bif via dev-security-policy wrote:
> Voluntarily providing CSR is not an ideal way to prove key compromise,
> because you could've simply found this CSR somewhere (I know, I know,
> super unlikely with your Subject... but still could happen.)

Feel free to trawl the Internet looking for CSRs with a subject that
indicates that the private key is compromised, where that key is used in a
publicly-trusted certificate issued to a third party, and for which the
private key hasn't been compromised (feel free to confirm that last part via
the pwnedkeys.com search API).

When you find one -- just one -- we can continue debating the merits of
using a static CSR as a method of attesting key compromise. Until then,
I'll keep using them as the least-worst available option.

> And while "compromised" is way too short (one can sign up to 32 bytes
> using it as a nonce in regular TLS session) to prove the key compromise,
> in the absence of the actual compromised private key, about the only way
> to ensure the possession is to get the reporter to sign some data chosen
> by the CA. It very well may be a random CN in the CSR, or plain old
> openssl dgst.

That would require the database of all pwnedkeys to be available live, to
permit real-time generation of signatures in response to CA requests.
Having that many keys online is a *spectacular* security risk, given the
importance of some of the certificates whose private keys have been publicly
disclosed (there's another gao.gov certificate yet to be revoked, not to
mention the one for *.avast.com!). I deliberately and *very* purposely keep
the database of actual private keys out of harm's way, and requiring the
database to be kept online to respond to challenge-response interactions
with CAs seems like a spectacularly bad security compromise.

Further, given the lack of any sort of standardisation of revocation
workflows, even within a single CA, a challenge-response mechanism for key
compromise attestation would almost certainly require manual intervention on
my part. Given the large backlog of certificates still to be revoked, and
the ongoing stream of new certificates and private keys that will,
seemingly, never stop, requiring manual steps from me for every one of those
is, I'm sure you can understand, not something I'd be keen on.

- Matt

[Matt Palmer]

On Tue, Mar 10, 2020 at 05:18:51PM -0400, Ryan Sleevi via dev-security-policy wrote:
> I'm sympathetic to CAs wanting to filter out the noise of shoddy reports
> and shenanigans, but I'm also highly suspicious of CAs that put too
> unreasonable an onus on reporters.

If CAs want a 100% reliable and trustworthy means of receiving key
compromise reports, they can stand up a server which implements RFC8555
s7.6. The backend doesn't have to immediately revoke the cert; it can
create a ticket in the CA's workflow management system saying "this cert has
been demonstrated to have a compromised private key, do the needful". No
need for compliance specialists, PKI experts, or anyone to be on hand to
check what's going on. Put a link to the ACME directory in s4.9.12 of their
CPS and in CCADB, Job done.

- Matt

[Piotr Kucharski]

For 0% of impact the FPs do not matter that much, so agreed!

Of course for now reality is not that... yet!
https://github.com/certbot/certbot/issues/1028 seems so appropriate :)

PS I was definitely not advocating for 5% false negative, no; we must
strive for 0% false negatives as well; all I was saying was exercising
caution for the perhaps-not-so-clear-cut 5% cases. (Probably closer to 1%)

On Tue, 10 Mar 2020 at 23:08, Ryan Sleevi <ry...@sleevi.com> wrote:

>
>
> On Tue, Mar 10, 2020 at 5:56 PM Piotr Kucharski <b...@google.com> wrote:
>
>> I'm sympathetic to CAs wanting to filter out the noise of shoddy reports
>>> and shenanigans, but I'm also highly suspicious of CAs that put too

[Matthew Hardeman]

Isn't the evident answer, if reasonable compromise is not forthcoming, just
to publish the compromised private key. There's no proof of a compromised
private key quite as good as providing a copy of it.

I understand the downsides, but I think that capricious burdens encourage
stripping the issue bare. You can't dodge a copy of a key.

> _______________________________________________
> dev-security-policy mailing list
> dev-secur...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-security-policy
>

[Matt Palmer]

On Tue, Mar 10, 2020 at 05:53:13PM -0500, Matthew Hardeman via dev-security-policy wrote:
> Isn't the evident answer, if reasonable compromise is not forthcoming, just
> to publish the compromised private key. There's no proof of a compromised
> private key quite as good as providing a copy of it.

Yes, going full-disc is one option. I'm hopeful that there is a happy
middle ground somewhere that means I don't have to drop keys in full public
view, though.

- Matt

[Joanna Fox]

Matt,

Our investigation reopened at March 9, 9:36 AM based on the information you provided in this forum. We were able to research and run appropriate testing which led to evidence of key compromise being determined March 9, 10:24 AM. We continued our diligence in accordance with the Baseline Requirements and revocation completed March 10, 10:24 AM.

Joanna Fox