Let's Encrypt: Failure to revoke key-compromised certificates within 24 hours

[Matt Palmer]

These two certificates:

https://crt.sh/?id=2602048478&opt=ocsp
https://crt.sh/?id=2601324532&opt=ocsp

Were issued by Let's Encrypt more than 24 hours ago, and remain unrevoked,
despite the revocation of the below two certificates, which use the same
private key, for keyCompromise prior to the above two certificates being
issued:

https://crt.sh/?id=2602048478&opt=ocsp
https://crt.sh/?id=2599226028&opt=ocsp

As per recent discussions here on m.d.s.p, I believe this is a breach of BR
s4.9.1.1.

- Matt

[Nick Lamb]

On Sat, 21 Mar 2020 09:25:26 +1100
Matt Palmer via dev-security-policy

Hi Matt,

I haven't looked at the substance of your concern yet, but the 1st and
3rd links you gave above both look identical to me whereas your text
implies they should differ. Perhaps this is a copy-paste error?

Nick.

[Matt Palmer]

On Sat, Mar 21, 2020 at 01:53:31AM +0000, Nick Lamb wrote:
> On Sat, 21 Mar 2020 09:25:26 +1100
> Matt Palmer via dev-security-policy
> <dev-secur...@lists.mozilla.org> wrote:
>
> > These two certificates:
> >
> > https://crt.sh/?id=2602048478&opt=ocsp
> > https://crt.sh/?id=2601324532&opt=ocsp
> >
> > Were issued by Let's Encrypt more than 24 hours ago, and remain
> > unrevoked, despite the revocation of the below two certificates,
> > which use the same private key, for keyCompromise prior to the above
> > two certificates being issued:
> >
> > https://crt.sh/?id=2602048478&opt=ocsp
> > https://crt.sh/?id=2599226028&opt=ocsp
> >
> > As per recent discussions here on m.d.s.p, I believe this is a breach
> > of BR s4.9.1.1.
>

> I haven't looked at the substance of your concern yet, but the 1st and
> 3rd links you gave above both look identical to me whereas your text
> implies they should differ. Perhaps this is a copy-paste error?

Oh the facepalm, it burns (probably too much hand sanitizer)... let me try
that again.

Recently issued and as-yet-unrevoked certificate, the first:

https://crt.sh/?id=2602048478&opt=ocsp

Previously revoked certificate for the same key:

https://crt.sh/?id=2599363087&opt=ocsp

Recently issued and as-yet-unrevoked certificate, the second:

https://crt.sh/?id=2601324532&opt=ocsp

Previously revoked certificate for the same key:

https://crt.sh/?id=2599226028&opt=ocsp

I've also, since my initial report, come across some more keys that have
been successfully re-used by Let's Encrypt customers after being revoked for
key compromise. You can pull the details out of the recent history:

https://crt.sh/?spkisha256=c5b2c5acc5a35409cb18c7f820b93a3d53e2fd17d99df165875881d60ff91ca2
https://crt.sh/?spkisha256=35e61785dc449d235568dc5919f9f4bca31a234f0768e6c057f1d9e39491d76d
https://crt.sh/?spkisha256=bb84a7d81dafd4e59877bb31595545eb5a205a4cc7db881b027fa499c5086c1c

There's also this one, which is another reuse-after-revocation, but the
prior history of this key suggests that there's something *far* more
interesting going on, given the variety of CAs and domain names it has been
used for (and its current residence, on a Taiwanese traffic stats server):

https://crt.sh/?spkisha256=69fc5edbd904577629121b09c49b711e201c46213e5b175bbee08a4d1d30b3c7

If anyone figures out the story with that last key, I'd be most pleased to
hear about it.

- Matt

[Ryan Sleevi]

Apologies for the delay here. I filed
https://bugzilla.mozilla.org/show_bug.cgi?id=1625322 for this.

[Josh Aas]

On Thursday, March 26, 2020 at 6:27:10 PM UTC-4, Ryan Sleevi wrote:
> Apologies for the delay here. I filed
> https://bugzilla.mozilla.org/show_bug.cgi?id=1625322 for this.

We are looking into this.

Matt - It would be helpful if you could report issues like this to the CA in question, not just to mdsp. We can review and respond faster. So far as I know we were never contacted. You can use our secu...@letsencrypt.org email address in the future. Thanks.

[Josh Aas]

It seems that Matt did report to our community forums. I'd like to encourage people to report known or potential security issues to our security@ address.

[Matt Palmer]

On Mon, Mar 30, 2020 at 01:48:28PM -0700, Josh Aas via dev-security-policy wrote:
> Matt - It would be helpful if you could report issues like this to the CA
> in question, not just to mdsp.

Helpful to *whom*, exactly? I don't write up these reports to be helpful to
the CA in question; I write them to be helpful to the community. I don't
see how reporting these problems to an individual CA is helpful to anyone
except that one CA -- which, as I said, is not a goal I am aiming for here.

At any rate, since (as I understand it) all CAs are supposed to be watching
mdsp anyway, sending a report here should be equivalent to sending it to all
CAs -- including Let's Encrypt -- anyway.

- Matt

[Ryan Sleevi]

On Mon, Mar 30, 2020 at 5:43 PM Matt Palmer via dev-security-policy
<dev-secur...@lists.mozilla.org> wrote:
>
> On Mon, Mar 30, 2020 at 01:48:28PM -0700, Josh Aas via dev-security-policy wrote:
> > Matt - It would be helpful if you could report issues like this to the CA
> > in question, not just to mdsp.
>
> Helpful to *whom*, exactly? I don't write up these reports to be helpful to
> the CA in question; I write them to be helpful to the community. I don't
> see how reporting these problems to an individual CA is helpful to anyone
> except that one CA -- which, as I said, is not a goal I am aiming for here.

I don't think that's quite a particularly helpful stance to take :)
Or, put differently, "why not both"

That said, your specific incident was in the gray area, where you'd
already previously reported compromise and the CA issued certs with
known compromised keys. You shouldn't "have" to report those new keys,
but it's still good form.

> At any rate, since (as I understand it) all CAs are supposed to be watching
> mdsp anyway, sending a report here should be equivalent to sending it to all
> CAs -- including Let's Encrypt -- anyway.

Ish? https://wiki.mozilla.org/CA/Incident_Dashboard specifically
encourages reporters to file a new incident bug.

https://wiki.mozilla.org/CA/Responding_To_An_Incident#Incident_Report
allows CAs to post to m.d.s.p. and a member will convert to a bug, but
I don't think it should be, nor do I want, m.d.s.p. to be the general
catch-all reporting mechanism for general users :) For one, as you can
see by my timeliness to the threads, it makes it hard to respond and
triage appropriately.

[Matt Palmer]

On Mon, Mar 30, 2020 at 06:01:58PM -0400, Ryan Sleevi wrote:
> On Mon, Mar 30, 2020 at 5:43 PM Matt Palmer via dev-security-policy
> <dev-secur...@lists.mozilla.org> wrote:
> >
> > On Mon, Mar 30, 2020 at 01:48:28PM -0700, Josh Aas via dev-security-policy wrote:
> > > Matt - It would be helpful if you could report issues like this to the CA
> > > in question, not just to mdsp.
> >
> > Helpful to *whom*, exactly? I don't write up these reports to be helpful to
> > the CA in question; I write them to be helpful to the community. I don't
> > see how reporting these problems to an individual CA is helpful to anyone
> > except that one CA -- which, as I said, is not a goal I am aiming for here.
>
> I don't think that's quite a particularly helpful stance to take :)
> Or, put differently, "why not both"

Yes, I might have put it a bit harshly, and I apologise for that. I was
somewhat taken aback by the implication of hiding problems by reporting to
them to the CA, rather than to the community.

> That said, your specific incident was in the gray area, where you'd
> already previously reported compromise and the CA issued certs with
> known compromised keys. You shouldn't "have" to report those new keys,
> but it's still good form.

I *have* been reporting additional certificates using the same compromised
key, using the ACME revocation endpoint provided by the CA, and indicating
that the reason for requesting certificate revocation was key compromise. I
don't see how it's a "gray area", though, except insofar as multiple CAs
have misinterpreted the BRs in roughly the same way.

If someone would like to make the argument that it's a gray area because I
submitted the revocation requests via ACME, rather than the CPS-provided
e-mail address, well, I can switch to sending e-mails, but having a human
process all the revocation requests is unlikely to be a better use of
everyone's time.

> > At any rate, since (as I understand it) all CAs are supposed to be watching
> > mdsp anyway, sending a report here should be equivalent to sending it to all
> > CAs -- including Let's Encrypt -- anyway.
>
> Ish? https://wiki.mozilla.org/CA/Incident_Dashboard specifically
> encourages reporters to file a new incident bug.

I don't see that that page *discourages* reporters from posting to mdsp. My
point, though, is that Josh asked for a report to be sent to Let's Encrypt,
all CAs are supposed to watch mdsp, therefore sending to mdsp should satisfy
Josh's request for a copy to be sent to Let's Encrypt.

> https://wiki.mozilla.org/CA/Responding_To_An_Incident#Incident_Report
> allows CAs to post to m.d.s.p. and a member will convert to a bug, but
> I don't think it should be, nor do I want, m.d.s.p. to be the general
> catch-all reporting mechanism for general users :) For one, as you can
> see by my timeliness to the threads, it makes it hard to respond and
> triage appropriately.

I did look into creating CA Compliance bugs directly, however I'm not 100%
confident of what counts as a compliance issue (as you've seen with some of
my past posts). Also, bugzilla uses a mail relay which is blocked on my
mail server (AWS' Spam Emission Service), and there's nothing in the SMTP
transaction I can use to whitelist *just* Bugzilla's emails. So I can't
create an account, and so I can't create bugs (or make snarky comments about
why CAs haven't updated their open bugs in three weeks -- which you may
count as a positive, perhaps?)

- Matt

[Matt Palmer]

On Tue, Mar 31, 2020 at 01:34:27PM +1100, Matt Palmer wrote:
> If someone would like to make the argument that it's a gray area because I
> submitted the revocation requests via ACME, rather than the CPS-provided
> e-mail address, well, I can switch to sending e-mails, but having a human
> process all the revocation requests is unlikely to be a better use of
> everyone's time.

... aaaaaaand they did
(https://bugzilla.mozilla.org/show_bug.cgi?id=1625322#c2). Oh well, I guess
someone at Let's Encrypt is going to have a bit more to do from now on.

- Matt