Digicert: failure to revoke certificate with previously compromised key

[Matt Palmer]

Certificate https://crt.sh/?id=2606438724, issued either at 2020-03-21
00:00:00 UTC (going by notBefore) or 2020-03-21 01:56:31 UTC (going by
SCTs), is using a private key with SPKI
4310b6bc0841efd7fcec6ba0ed1f36e7a28bf9a707ae7f7771e2cd4b6f31b5af, which was
reported to Digicert as compromised on 2020-03-20 02:05:49 UTC (and for
which https://crt.sh/?id=1760024320 was revoked for keyCompromise soon after
certificate 2606438724 was issued).

As previously discussed on this list, the visible consensus is that,
according to the BRs, certificates for which the CA already had evidence of
key compromise must be revoked within 24 hours of issuance. That 24 hour
period has passed for the above certificate, and thus it would appear that
Digicert has failed to abide by the BRs.

- Matt

[Jeremy Rowley]

That's not the visible consensus IMO. The visible consensus is we need to revoke a cert that is key compromised once we're informed the key is compromised for that cert (https://groups.google.com/forum/m/#!topic/mozilla.dev.security.policy/1ftkqbsnEU4). The certificate you mentioned was issued before the keys were blacklisted and not part of a certificate problem report. When revoking a cert we scan to see if additional certs are issued with the same key t, but this particular cert one was issued after the scan but before the revocation, largely because the way you are submitting certificate problem reports breaks automation. We currently don't have a way to blacklist private keys until a certificate is revoked, although that would be a nice enhancement for us to add in the future. Anyway, I don't think anything reported violated the BR since 1) this cert was not part of a certificate problem report and 2) we will be revoking within 24 hours of your Mozilla posting.

I support the idea of swift revocation of compromised private keys and do appreciate you reporting them. I think this is helpful in ensuring the safety of users online. However, using the SPKI to submit information breaks our automation, making finding and revoking certs difficult. The more standards way (IMO) is the SHA2 thumbprint or serial number or a good old CSR. Because submitting the SPKI breaks automation, getting evidence of key compromise took an additional 5 hours after you submitted the report. We still revoked all of the current certs with submitted keys within 24 hours of the report (since compromised private keys are bad and there is nothing that says we can't revoke earlier than 24 hours), but I did want to clarify that I don't think the time starts until we can actually get the information necessary to do an investigation (because there is not sufficient evidence of a key compromise until then).

Going to the previous discussion, I'd definitely support seeing a standardized way to report key compromise. Trying to account for the various formats they come in and through the various channels creates a lot of manual work on a process that can easily be automated.

Jeremy

_______________________________________________
dev-security-policy mailing list
dev-secur...@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy

[Matt Palmer]

On Mon, Mar 23, 2020 at 06:14:29AM +0000, Jeremy Rowley wrote:
> That's not the visible consensus IMO. The visible consensus is we need to
> revoke a cert that is key compromised once we're informed the key is
> compromised for that cert
> (https://groups.google.com/forum/m/#!topic/mozilla.dev.security.policy/1ftkqbsnEU4).

I think that link might not be doing what you expect, as it (at least for
me) is collapsing all the replies in that topic before Doug Beattie's post.
The only response that seems relevant in that topic to was Ryan's reply to
me up-thread from Doug's post, which was, in (I believe) relevant part, when
I asked the question:

> 3. Can a CA be deemed to have "obtained evidence" of key compromise prior
> to the issuance of a certificate, via a previously-submitted key
> compromise problem report for the same private key? If so, it would
> seem that, even if the issuance of the certificate is OK, it is a
> failure-to-revoke incident if the cert doesn't get revoked within 24
> hours...

To which Ryan replied:

> Correct, that was indeed the previous conclusion around this. The CA can
> issue, but then are obligated to revoke within 24 hours. There’s not a
> statute of limitation on “obtains evidence” here, precisely because it
> could allow a host of shenanigans, such as CAs arguing the require per-cert
> evidence rather than systemic demonstrations.

I don't think that supports your point, though, so I wonder if I've got the
wrong part. That last part of Ryan's: "shenanigans, such as CAs arguing
the[y?] require per-cert evidence rather than systemic demonstrations",
seems to me like it's describing your statement, above, that you (only?)
"need to revoke a cert that is key compromised once we're the key is
compromised *for that cert*" (emphasis added). I don't read Ryan's use of
"shenanigans" as approving of that sort of thing.

> The certificate you mentioned was issued before the keys were blacklisted
> and not part of a certificate problem report. When revoking a cert we
> scan to see if additional certs are issued with the same key t, but this
> particular cert one was issued after the scan but before the revocation,
> largely because the way you are submitting certificate problem reports
> breaks automation. We currently don't have a way to blacklist private
> keys until a certificate is revoked, although that would be a nice
> enhancement for us to add in the future. Anyway, I don't think anything
> reported violated the BR since 1) this cert was not part of a certificate
> problem report and 2) we will be revoking within 24 hours of your Mozilla
> posting.

The way I see it, once a CA is provided with accepted evidence of key
compromise, every certificate issued by that CA using the same key -- past,
present, and future -- is implicitly part of that certificate problem
report. *I* think that's supported by Ryan's confirmation of my question,
quoted above, but presumably you disagree?

At the end of the day the only call that matters is that of the CA module
owner and peers, so I guess we'll have to leave it up to them to make the
call as to whether Digicert's behaviour that I described -- and the facts of
which you don't seem to dispute -- constitutes a BR violation.

Honestly, I'm a bit surprised you're trying so hard to argue against taking
responsibility for this occurrence (probably should use "incident" yet, I
guess). Based on purely on what you wrote in the above paragraph, it seems
like it was a simple oversight in your systems for what is, I won't deny, a
bit of a corner case. Even I, as gung-ho as I am about throwing out
misbehaving CAs, wouldn't argue that this in and of itself is anything worth
a rap over the knuckles for.

As far as I can see, the meat of an incident report for this occurrence
could be something like "our key blacklist is only populated at the time
revocation occurs; the certificate in question was issued between problem
report and revocation; we've added a story to the backlog to modify our CA
systems to allow keys to be blacklisted before revocation, and until then
we've modified our procedures to do a sweep of our certificate archive after
the initial revocation has taken place, to catch a similarly situated
certificate in the future."

Bim bam bom, all done and dusted, and we can get back to washing our hands.
That you're *not* doing that is perplexing, and a little disconcerting.

> I support the idea of swift revocation of compromised private keys and do
> appreciate you reporting them. I think this is helpful in ensuring the
> safety of users online. However, using the SPKI to submit information
> breaks our automation, making finding and revoking certs difficult. The
> more standards way (IMO) is the SHA2 thumbprint or serial number or a good
> old CSR.

This confuses me a bit; an SPKI *is* a "SHA2[56] thumbprint" of the
compromised key, and keys don't have serial numbers. As for a CSR, I
provide one, signed by the compromised key, which contains (as is the nature
of the beast) the entire public key, which you can extract and wield in
whichever manner you choose.

If you're referring to the SHA2 thumbprint or serial number of a
*certificate*, then we're back to "CAs arguing the[y?] require per-cert
evidence", which comes under the (disputed?) territory of "shenanigan".
Meanwhile, I don't understand how *any* CSR -- even the original CSR which
was submitted to the CA (which I wouldn't necessarily have, nor would I be
expected to) -- could even uniquely identify a certificate, since it doesn't
have a serial number or SHA2 thumbprint or *any* unique identifier that ties
it to the certificate.

> Because submitting the SPKI breaks automation, getting evidence
> of key compromise took an additional 5 hours after you submitted the
> report.

What automation *does* Digicert have in place to process key compromise
reports? If I'd, say, just e-mailed a PEM format private key to
rev...@digicert.com (or even 800 of them), how much quicker would that have
been processed, as compared to the SPKIs I provided?

I collect so many private keys of your customers' certs (I've got, IIRC,
nine queued up that have come in over the last few hours that I'm yet to
process), that it's worth my while spending an hour or two coding up
something if it means I can just squirt key compromise reports into your
systems somehow and avoid the whole e-mail round-trip dance.

> We still revoked all of the current certs with submitted keys
> within 24 hours of the report (since compromised private keys are bad and
> there is nothing that says we can't revoke earlier than 24 hours), but I
> did want to clarify that I don't think the time starts until we can
> actually get the information necessary to do an investigation (because
> there is not sufficient evidence of a key compromise until then).

I certainly agree that the wording in the BRs, that says
that the 24 hour period begins when the CA "obtains" evidence of key
compromise, could stand to be improved. "Obtains" is a word open to much
interpretation, and one could, potentially, argue that a CA only "obtains"
evidence when it takes an active step -- such as opening an e-mail,
retrieving a provided link, or you could even say that the evidence isn't
really "obtained" until the CA actually validates the signature on a CSR and
compares public keys.

However, by that argument, a CA could delay revocation indefinitely by
claiming that they hadn't yet "obtained" evidence, before finally saying
"oh, yes, OK, now we've got it" and doing the revocation. Hopefully we
agree that a CA which deliberately did that would be worthy of a trip to the
naughty step, which means that the BRs should not be interpreted that way,
based on my recollection of precedent.

So, let's talk quality of evidence provided. If Digicert had a legitimate
belief that the evidence I provided was insufficient, or was otherwise
problematic, the correct action, IMO, would have been to mention that in the
preliminary report, which has to come within the first 24 hours. I don't
know how closely you watch the queues in the Digicert salt mines, so you
might already be aware of this, but the "your evidence sucks" situation
already happened, with the first problem reports I submitted. I was
providing a JWS attesting to compromise, your people said "we can't handle
this", I checked with m.d.s.p as to whether that was reasonable, Ryan called
me a goose for choosing to use JWS, and we moved on. CSRs for everyone!

So I started submitting problem reports with links to crt.sh and a CSR,
and everything seemed to be going swimmingly. I did dump a whole pile of
compromised keys on y'all at once, I admit, but I had an 800+ key backlog,
and at the rate new keys were coming in, the backlog would have *never* been
cleared if I'd kept reporting just a few a day. (See above, re: finding
nine in the past few hours).

The time to say "hey, while we *can* process these SPKIs and CSRs, they are
kinda teh suck, what else you got?" would have been, ideally, when I sent
that first trickle of compromise reports that were acted upon. At worst,
when the Big Bomb of Keys dropped, you stick your hand up and say "uhm,
these kinda suck to process, any chance it's easy for you to send us X
instead?", while simultaneously working on the pile as best you can. Worst
case I've gone to bed, and you do what you did anyway, but depending on what
X was, I might have been able to oblige.

Only raising this *now*, after something bad (may have) happened, strikes me
as a bit of a cop-out. (Is that an Australianism?)

Even if Digicert would just prefer that I did something differently in my
problem reports in the future -- something that would make it legitimately
easier for Digicert to process my reports, without making my life a misery
(after all, y'all have a lot more resources than I do) -- all you have to do
is ask. My e-mail address is right there in every problem report I send.

> Going to the previous discussion, I'd definitely support seeing a
> standardized way to report key compromise. Trying to account for the
> various formats they come in and through the various channels creates a
> lot of manual work on a process that can easily be automated.

Weeeeeeeell... I'm in two minds here. Given the volume of reports I'm
doing, a standard format that I can easily automate into, that I *know* all
CAs would accept, would be awesome -- *for me*. There's no need to develop
any new protocols, even, because a quite serviceable one already exists --
the revokeCert endpoint in ACME. (And don't think I'm not a bit salty that
Digicert implements ACME but hides it behind customer registration, either
<grin>).

However, if someone wants to report a compromised key as a one-off, making
them jump through unnecessary hoops to beat an ACME client into doing their
bidding just so they can report a compromised key is *massively*
counter-productive. Whilst you might say that a CA should also continue to
accept key compromise reports via e-mail -- and I would agree -- I have
little-to-no confidence that there would be at least one CA who would try to
force everyone to use their not-particularly-casual-user-friendly ACME
endpoint to submit key compromise notifications.

So, if someone does manage to wedge "all CAs must provide an ACME revokeCert
endpoint, *and* publish the directory URL for that ACME server in CCADB"
into the BRs (and my oh my, how I pray for that day), I hope that the same
amendment will also include language saying that CAs must also accept,
without kicking, key compromise reports via e-mail (at an e-mail address
also recorded in its own field in CCADB).

- Matt

[Jeremy Rowley]

Hey Matt,

Ryan's post was the part I thought was relevant, but I understood it differently. The cert was issued, but we should have now revoked it (24 hours after receiving notice). I do see your interpretation though, and the language does support 24 hours after issuing the new cert. What I need is a tool that scans after revocation to ensure there are no additional certs with the same key. The frustration is that this was where the cert was issued after our scan of all keys but just before revocation. As a side note, our system blacklists the keys when a cert is revoked for key compromise, which means I don't have a way to blacklist a key before a cert is ever issued.

>> I don't think that supports your point, though, so I wonder if I've got the wrong part. That last part of Ryan's: "shenanigans, such as CAs arguing the[y?] require per-cert evidence rather than systemic demonstrations", seems to me like it's describing your statement, above, that you (only?) "need to revoke a cert that is key compromised once we're the key is compromised *for that cert*" (emphasis added). I don't read Ryan's use of "shenanigans" as approving of that sort of thing.

I don't think its shenanigans, but I do think it's a pretty common interpretation. Such information would help determine the common interpretation of this section. I agree that CAs should scan all certs for the same key and revoke where they find them. Is that actually happening? Do other CAs object to there being a lack of specificity if you give the keys without a cert attached?

>> Bim bam bom, all done and dusted, and we can get back to washing our hands.
That you're *not* doing that is perplexing, and a little disconcerting.

That's an oversimplification of the incident report process. I'm not resisting the incident report itself since incident reports are a good and healthy way for the public to have transparency into a CAs operation. In fact, I wouldn't mind filing one just to give public documentation on what DigiCert is doing for revocation. I was objecting to actually calling this an incident/breach of the guidelines since I disagreed with that, and there's a trend where the interpretation of these sections seems to evolve over time. My main emphasis was that the guidelines are ambiguous and need refinement. I'd support refining them to be more clear, but calling something shenanigans when the language is unclear is unfair.

For the SPKI, you're right. The slow part is downloading it from your website. I guess I should have said "a link to the key compromise proof" rather than the SPKI. The additional time required to do that was 5 hours. Although the automation isn't quite complete yet, the end-state goal is to do the following when we get a key compromise notification:
1) Run the CSR through a tool and confirm key compromise
2) Have the tool email the impacted subscriber
3) Have the tool process the revocation at the 24 hour mark.

The two parts that are done is having the tool process the revocation and parse out all the key compromises and processing the revocation within 24 hours. Unfortunately, we haven't automated the cert problem report yet so we can only accept an email. Once I get the revocation tool working the way I want it, we'll add a tool where you can submit a CSR directly and avoid the round trip.

>> Even if Digicert would just prefer that I did something differently in my problem reports in the future -- something that would make it legitimately easier for Digicert to process my reports, without making my life a misery (after all, y'all have a lot more resources than I do) -- all you have to do is ask. My e-mail address is right there in every problem report I send.

Nah - although I'd prefer just to get the CSRs in a zip file, I can work with what you send us. In fact, I wouldn't change your process at all since I want to accommodate all sorts of ways for people to send us private keys. I'd rather not force you to do extra work to send us the keys in a different way as anything that creates barriers to key compromise reports is a bad thing. The reason I brought up the format is that I'm not sure when the 24 hour clock actually kicks off for revocation. Is it 24 hours after we get your email or 24 hours after we can confirm key compromise? I've always regarded these as a certificate problem report under 4.9.5 (requiring us to kick off the investigation) but then revocation should happen 24 hours after we have reason to suspect the key compromise is legit. Usually those are the same time, but with the email link (and number of keys in the last batch) it may take longer to actually do the investigation to show the key is compromised. I think there's ambiguity in when that timer starts, and clearing up that ambiguity in the CAB forum would be good. As you pointed out "obtains" can mean different things.

>> The time to say "hey, while we *can* process these SPKIs and CSRs, they are kinda teh suck, what else you got?" would have been, ideally, when I sent that first trickle of compromise reports that were acted upon. At worst, when the Big Bomb of Keys dropped, you stick your hand up and say "uhm, these kinda suck to process, any chance it's easy for you to send us X instead?", while simultaneously working on the pile as best you can. Worst case I've gone to bed, and you do what you did anyway, but depending on what X was, I might have been able to oblige.

To be clear, I'm not complaining about the format. I'm wondering when we obtained the private key for the 24 hour purposes. With automation, the time between when we get the email and when we confirm key compromise should be nearly zero. However, with a more manual process, that time is not insignificant. What I don't like about the interpretation that the revocation event is 24 hours from when we get an email is some emails are very vague about key compromise. With that reading, if we get an email without proof that is later followed up by proof, the 24 hour period could start when we get the initial email even if the proof is provided 25 hours later. That does happen, which is why I think the time period should be 24 hours from after the CA receives proof of key compromise. But even that is ambiguous. When did we receive proof of key compromise? I'd say it's when all the CSRs finished downloading. If that's not the case, then you are encouraging CAs to be myopic in the way they accept key compromise information.

Jeremy


-----Original Message-----
From: dev-security-policy <dev-security-...@lists.mozilla.org> On Behalf Of Matt Palmer via dev-security-policy

[Ryan Sleevi]

On Mon, Mar 23, 2020 at 11:01 AM Jeremy Rowley via dev-security-policy <
dev-secur...@lists.mozilla.org> wrote:

> Hey Matt,
>
> Ryan's post was the part I thought was relevant, but I understood it
> differently. The cert was issued, but we should have now revoked it (24
> hours after receiving notice). I do see your interpretation though, and the
> language does support 24 hours after issuing the new cert. What I need is
> a tool that scans after revocation to ensure there are no additional certs
> with the same key. The frustration is that this was where the cert was
> issued after our scan of all keys but just before revocation. As a side
> note, our system blacklists the keys when a cert is revoked for key
> compromise, which means I don't have a way to blacklist a key before a cert
> is ever issued.
>

Matt, Jeremy,

To make sure I understand the timeline correctly:
2020-03-20 02:05:49 UTC - Matt reports SPKI 4310b6bc0841efd7fcec6ba0ed1f36
e7a28bf9a707ae7f7771e2cd4b6f31b5af, associated with
https://crt.sh/?id=1760024320 , as compromised
2020-03-21 01:56:31 UTC - DigiCert issues https://crt.sh/?id=2606438724 with
that same SPKI
2020-03-21 02:09:12 UTC - DigiCert revokes https://crt.sh/?id=1760024320
2020-03-23 03:16:18 UTC - DigiCert revokes https://crt.sh/?id=2606438724

Is that roughly correct?

If so, it does seem like an Incident Report is warranted here, so we can
understand why:
a) https://crt.sh/?id=2606438724 wasn't revoked when
https://crt.sh/?id=1760024320 was revoked (assuming those timestamps in the
CRL are accurate)
b) The key wasn't blocklisted as known compromised (if the timestamps are
incorrect)

That is, it doesn't seem unreasonable that, for situations of key
compromise, the CA has the necessary data to scan their systems for
potential reuse of that key. Given DigiCert's data lake
<https://bugzilla.mozilla.org/show_bug.cgi?id=1526154>, it should be
possible to scan for issues.

If I've misunderstood the timing here, please feel free to correct. This is
where the incident report process is useful, and Resolved/Invalid is a
perfectly fine state to end in.

[Jeremy Rowley]

Yeah - that’s about the sum of it. I’ll file an incident report.

There are two things worth discussing in general:

1. I’m very interested in seeing the Let’s Encrypt response to this issue since the biggest obstacle in trying to find all of the keys with the same private key is the sheer volume of the certs. Trying to do a comprehensive search when a private key is provided leaves some window between when we start the analysis and when we revoke.


1. Another issue in trying to report keys that aren’t affiliated with any cert is that the process becomes subject to abuse. Without knowing a cert affiliated with a key, someone can continuously generate keys and submit them as compromised. You end up just blacklisting random keys, DDOSing the revocation system as it kicks off another request to search for those keys. I don’t think it’s feasible. This is why the disclosures need to be affiliated with actual certs.

From: Ryan Sleevi <ry...@sleevi.com>
Sent: Monday, March 23, 2020 10:54 AM
To: Jeremy Rowley <jeremy...@digicert.com>
Cc: Matt Palmer <mpa...@hezmatt.org>; Mozilla <mozilla-dev-s...@lists.mozilla.org>
Subject: Re: Digicert: failure to revoke certificate with previously compromised key

On Mon, Mar 23, 2020 at 11:01 AM Jeremy Rowley via dev-security-policy <dev-secur...@lists.mozilla.org<mailto:dev-secur...@lists.mozilla.org>> wrote:
Hey Matt,

Ryan's post was the part I thought was relevant, but I understood it differently. The cert was issued, but we should have now revoked it (24 hours after receiving notice). I do see your interpretation though, and the language does support 24 hours after issuing the new cert. What I need is a tool that scans after revocation to ensure there are no additional certs with the same key. The frustration is that this was where the cert was issued after our scan of all keys but just before revocation. As a side note, our system blacklists the keys when a cert is revoked for key compromise, which means I don't have a way to blacklist a key before a cert is ever issued.

Matt, Jeremy,

To make sure I understand the timeline correctly:

2020-03-20 02:05:49 UTC - Matt reports SPKI 4310b6bc0841efd7fcec6ba0ed1f36e7a28bf9a707ae7f7771e2cd4b6f31b5af, associated with https://crt.sh/?id=1760024320 , as compromised

[Matt Palmer]

On Mon, Mar 23, 2020 at 03:01:34PM +0000, Jeremy Rowley wrote:
> Ryan's post was the part I thought was relevant, but I understood it
> differently. The cert was issued, but we should have now revoked it (24
> hours after receiving notice). I do see your interpretation though, and
> the language does support 24 hours after issuing the new cert.

Aha, righto. Glad we've gotten on the same page there.

> What I need is a tool that scans after revocation to ensure there are no
> additional certs with the same key.

I can give you a certwatch SQL query that'll do that, if you like -- "show
me all certs with this SPKI (or set of SPKIs) which aren't expired or
OCSP-revoked". I use pretty much that query to get periodic reports of new
certificates that have appeared with keys already in the dungeon. It's not
ideal for your purposes, though -- you might get some false positives
because your OCSP responders aren't up-to-date, and false negatives are
possible if certwatch is backlogged (or a cert wasn't logged).

Beyond that, though, if your internal certificate archive isn't indexed on
SPKI fingerprint, and updated in near-real-time, those are problems that I
think you'll have to fix, because the Internet's propensity to post their
private keys on the Internet, and then reuse them to get new certs after the
old one got revoked for key compromise, does not seem to be one that is
going away any time soon.

> The frustration is that this was
> where the cert was issued after our scan of all keys but just before
> revocation. As a side note, our system blacklists the keys when a cert is
> revoked for key compromise, which means I don't have a way to blacklist a
> key before a cert is ever issued.

To the software developers! *blows trumpets*

> >> I don't think that supports your point, though, so I wonder if I've got
> >> the wrong part. That last part of Ryan's: "shenanigans, such as CAs
> >> arguing the[y?] require per-cert evidence rather than systemic
> >> demonstrations", seems to me like it's describing your statement,
> >> above, that you (only?) "need to revoke a cert that is key compromised
> >> once we're the key is compromised *for that cert*" (emphasis added). I
> >> don't read Ryan's use of "shenanigans" as approving of that sort of
> >> thing.
>
> I don't think its shenanigans, but I do think it's a pretty common
> interpretation. Such information would help determine the common
> interpretation of this section. I agree that CAs should scan all certs
> for the same key and revoke where they find them. Is that actually
> happening?

A lot to unpack here, let me make up some specific questions and answer them
as best I can.

"Have any other CAs failed to revoke certificates issued for keys for
which they had previously revoked a certificate for key compromise?"

Yes, one CA has failed to do so, and I've reported that to this list.

"Have any other CAs successfully revoked a certificate within the
BR-mandated (am I OK using that phrase now?) time period, that they issued
for a private key for which they had previously revoked a certificate due to
key compromise?"

I don't know, I haven't checked (yet). It's on my (lengthy) list of Interesting
Questions For Which I Need To Write Insanely Complicated SQL Queries.

"Have any other CAs blacklisted a private key that was reported as
compromised and prevented issuance before they had issued a certificate for
that key?"

Naturally I can't answer that one directly, because I don't have internal
access to CA systems. *But*, I have one test case, in which a private key
known to have "hopped" between CAs after revocation was reported to a number
of CAs proactively, but there hasn't been a resolution to that test case one
way or the other. No new certs have been issued for the same name, with the
compromised key or any other, so it's impossible to infer what the outcome
may be.

Other very specific questions welcomed. The answers will probably be "I
haven't looked yet", but there are a lot of questions I've got at least a
vague idea of how to answer, once I have time2SQL.

> Do other CAs object to there being a lack of specificity if you give the
> keys without a cert attached?

Since I have been sending (links to) CSR format compromise attestations, no
CA has communicated an objection to the format of my reports, nor have they
failed to act (in some fashion) to any of my reports, as far as I am aware.

> >> Bim bam bom, all done and dusted, and we can get back to washing our hands.
>
> That you're *not* doing that is perplexing, and a little disconcerting.

My wife is immuno-suppressed -- about the only time I'm not washing my
hands is when I'm typing at my keyboard (the suds get in the switches and
cause all sorts of problems). If I could get a reliable supply of
sanitizer, I'd be swimming in the stuff.

> That's an oversimplification of the incident report process. I'm not
> resisting the incident report itself since incident reports are a good and
> healthy way for the public to have transparency into a CAs operation. In
> fact, I wouldn't mind filing one just to give public documentation on what
> DigiCert is doing for revocation. I was objecting to actually calling
> this an incident/breach of the guidelines since I disagreed with that, and
> there's a trend where the interpretation of these sections seems to evolve
> over time. My main emphasis was that the guidelines are ambiguous and
> need refinement. I'd support refining them to be more clear, but calling
> something shenanigans when the language is unclear is unfair.

I can see your point there -- there's no denying that the BR language can be
misinterpreted. Given a suitably motivated reader, of course, *anything*
can be creatively misinterpreted, but that use of "obtains evidence" is
undoubtedly tricky.

>From what Ryan said in the previous thread, my impression is that attempts
were made to improve the language, but were blocked. Unfortunately, in that
sense, you're beholden to the most reactionary majority of your fellow CAs
in improving the clarity of the language in the BRs.

I suppose these sorts of enduring ambiguity are part of the reason why all
CAs are expected to follow this list -- you could think of it as the "judiciary
branch", interpreting the "laws" and providing precedent.

> For the SPKI, you're right. The slow part is downloading it from your
> website. I guess I should have said "a link to the key compromise proof"
> rather than the SPKI.

Just for clarity, was it the HTTP requests themselves that were slow (which
is something I want to know about, and fix) or was it that your front-line
people aren't tooled up for ad-hoc automation? (To be clear, I'm not
criticising you for having front-line people who don't know shell scripting,
just trying to clarify the problem)

> 3) Have the tool process the revocation at the 24 hour mark.

You'll want to factor in the delays in getting the message to your OCSP
responders and CRLs. I don't think that arguing that "publishing" doesn't
need to include actually making the revocation public would get very far.

> >> Even if Digicert would just prefer that I did something differently in
> >> my problem reports in the future -- something that would make it
> >> legitimately easier for Digicert to process my reports, without making
> >> my life a misery (after all, y'all have a lot more resources than I do)
> >> -- all you have to do is ask. My e-mail address is right there in
> >> every problem report I send.
>
> Nah - although I'd prefer just to get the CSRs in a zip file, I can work
> with what you send us.

Well, there shouldn't be any more mass key compromise notifications coming
through, at least not of the scale of that last round, so "CSRs in a zip"
shouldn't be necessary.

I'm intending to get to the point in the near future where the notifications
go out one by one, as soon as the key is discovered to be compromised.

> The reason I brought up the format is that I'm not sure when the 24 hour
> clock actually kicks off for revocation. Is it 24 hours after we get your
> email or 24 hours after we can confirm key compromise? I've always
> regarded these as a certificate problem report under 4.9.5 (requiring us
> to kick off the investigation) but then revocation should happen 24 hours
> after we have reason to suspect the key compromise is legit.

It's surprising that you say that, because to me, 4.9.5 isn't nearly as
ambiguous as the relevant point in 4.9.1.1. 4.9.5 says "The period from
receipt of the Certificate Problem Report or revocation-related notice to
published revocation". While "receipt" could, I suppose, still be argued as
meaning "when we checked the mailbox", I don't think it's reasonable to
claim that "receipt" means "when we validated the CSR" -- especially since
it's "receipt of the Certificate Problem Report", not "receipt of concrete
evidence" or anything like that.

At any rate, for the specific issue we're discussing -- failure to revoke a
cert for a previously-reported compromised private key -- I don't think
4.9.5 really matters. 4.9.1.1 says the CA has to revoke within 24 hours of
"obtaining evidence of key compromise". For a certificate issued after a
report of key compromise has been sent to the CA, the moment at which it can
be said that the CA has "obtained evidence of key compromise" *for that
specific certificate*, depending on your interpretation, could be:

* the time at which the problem report which initially reported the compromised key
landed on the CA's mail server (which is the last moment that an external
observer can know what's going on);

* the time at which that the CA validates that the evidence is valid;

* the time at which the new certificate using the compromised private key is
issued; or

* when someone says "hey, you issued another cert for that key I mentioned
before!"

(If you think there's another one, please add it. Again, remember, this is
*only* for the case of a certificate issued after an initial key compromise
report is received by the CA.)

The last of those moments is what it looks (to me) like you were relying on,
but which Ryan's statement seems to describe as "shenanigans". It implies
to me that CAs have no memory of the past when it comes to key compromises,
which isn't a situation that is reasonable, to my mind. The same "obtains
evidence" phrasing is used in the next point (regarding domain control
validation that cannot be relied upon), and the idea that a CA would argue
that they could legitimately issue a certificate for a name which they had
previously accepted had not been properly validated is, I hope you'd agree,
disconcerting in the extreme.

> To be clear, I'm not complaining about the format. I'm wondering when we
> obtained the private key for the 24 hour purposes. With automation, the
> time between when we get the email and when we confirm key compromise
> should be nearly zero. However, with a more manual process, that time is
> not insignificant. What I don't like about the interpretation that the
> revocation event is 24 hours from when we get an email is some emails are
> very vague about key compromise. With that reading, if we get an email
> without proof that is later followed up by proof, the 24 hour period could
> start when we get the initial email even if the proof is provided 25 hours
> later.

Well, if that were the case, then within 24 hours of the receipt of the
problem report you should have provided a preliminary report which says
"evidence is insufficient", and thus that problem report could reasonably be
considered to have been handled. The "followup" e-mail that did contain
useful evidence would reasonably be considered a "new" problem report.

Even when the useful evidence is provided after the initial e-mail, but
before you put out the preliminary compromise report, if the initial e-mail
didn't contain anything actionable I doubt that a root store is going to
class it as an incident if you don't revoke within 24 hours of the initial
e-mail -- as long as you got it done within 24 hours of when the actual,
useful evidence landed on your MTA.

Certainly, if someone came here and tried to drop a CA in it with the above
set of facts, I'd be in the front row of people saying "pull the other one,
it plays jingle bells".

> That does happen, which is why I think the time period should be
> 24 hours from after the CA receives proof of key compromise. But even
> that is ambiguous. When did we receive proof of key compromise? I'd say
> it's when all the CSRs finished downloading.

You could have started processing revocations for the first CSRs downloaded
-- waiting for the last CSR to download before starting to process the first
is unlikely to be a valid argument for delayed revocation.

It would almost certainly be worse for you, though, if it *was* a valid
argument, because the solution to that, from my perspective, would be to
send 800-odd separate certificate problem reports, each with one key. I
can't imagine a situation in which that pile of e-mails would be *easier* to
process than one big list of compromised keys, but it would prevent a CA
from arguing that it look a long time to download all the CSRs.

Claiming that you didn't have enough staff to process that volume of key
compromise reports would be unlikely to be accepted, because if "lack of
staff" were seen as a valid excuse it would encourage CAs to understaff
their problem report processing, which would be Extremely Terribad.

> If that's not the case, then you are encouraging CAs to be myopic in the
> way they accept key compromise information.

Yes, it is possible that being strict on revocation deadlines might cause
some CAs to react by trying to reject valid-but-tricky certificate problem
reports. There's all *sorts* of reasons why CAs might react in ways that
the root stores (and the surrounding community) would find problematic.
At the end of the day, the ultimate arbiters of the reasonableness of those
behaviours are the root stores.

As I have already done once before, if a CA rejected one of my problem
reports for what I felt were spurious reasons, I'd check with Mozilla and
the community, via this list, to see if my expectations were unreasonable.
If the response was "your problem report was reasonable", then I'd lay out
the full details of the case, and the CA gets to explain their behaviour,
and root stores get to act as they see fit. If, on the other hand, the
response is "stop being a goose, Matt" then I don't make a report, and I do
something else in the future.

- Matt

[Matt Palmer]

On Mon, Mar 23, 2020 at 12:53:43PM -0400, Ryan Sleevi wrote:
> To make sure I understand the timeline correctly:
> 2020-03-20 02:05:49 UTC - Matt reports SPKI 4310b6bc0841efd7fcec6ba0ed1f36
> e7a28bf9a707ae7f7771e2cd4b6f31b5af, associated with
> https://crt.sh/?id=1760024320 , as compromised
> 2020-03-21 01:56:31 UTC - DigiCert issues https://crt.sh/?id=2606438724 with
> that same SPKI
> 2020-03-21 02:09:12 UTC - DigiCert revokes https://crt.sh/?id=1760024320
> 2020-03-23 03:16:18 UTC - DigiCert revokes https://crt.sh/?id=2606438724
>
> Is that roughly correct?

Yes, that appears to be a correct summary of the timeline as I see it.

- Matt

[Matt Palmer]

On Mon, Mar 23, 2020 at 06:15:00PM +0000, Jeremy Rowley wrote:
> There are two things worth discussing in general:
>
> 1. I’m very interested in seeing the Let’s Encrypt response to this issue
> since the biggest obstacle in trying to find all of the keys with the same
> private key is the sheer volume of the certs. Trying to do a
> comprehensive search when a private key is provided leaves some window
> between when we start the analysis and when we revoke.

Well, Let's Encrypt has committed to automatically blacklisting keys
reported for keyCompromise in its CA software
(https://community.letsencrypt.org/t/116762), so it won't be nearly as
problematic for them as it appears to be for Digicert. At any rate, though,
I can get a list of all certs with a certain SPKI out of crt.sh in a matter
of a couple of seconds, and crt.sh has a *lot* more certs in it than
Digicert has issued. The schema for the certwatch database is publicly
available, so, if nothing else, you could stand up a copy of that database,
skip deploying the CT scrapers, and just stuff a copy of every cert you
issue into the certificate table, and you're off to the races.

> 1. Another issue in trying to report keys that aren’t affiliated with
> any cert is that the process becomes subject to abuse. Without knowing a
> cert affiliated with a key, someone can continuously generate keys and
> submit them as compromised. You end up just blacklisting random keys,
> DDOSing the revocation system as it kicks off another request to search
> for those keys. I don’t think it’s feasible. This is why the disclosures
> need to be affiliated with actual certs.

I don't think that anyone has, as yet, claimed that it is a BR violation for
a CA to issue a certificate with a key for which they have not yet received
a valid certificate problem report. Nor do I believe that anyone has
claimed that a certificate problem report without any indication of a
problematic certificate is valid. So, it appears to me that you're arguing
against something that nobody has proposed.

- Matt