Let's Encrypt Experience with Upgrading Revocation Reason in CRLs

[Kiel Christofferson]

Let's Encrypt likes to learn about the ecosystem from the experiences of the community. Today we want to bring some information about an event that may be interesting or useful to other CAs in precisely that way.

One of our engineers discovered a bug in Boulder (our Certificate Authority software) which caused revoked certificates included in one of our CRL shards to retain their original revocation reason code despite a subsequent ACME revoke-certificate request which "upgraded" the reason code to keyCompromise.

In succinct technical detail, our engineer described the bug in this way:

For explicitly sharded certificates, CRL status is read from the revokedCertificates table. This table gets written at revocation time. At re-revocation time (for key compromise), it only gets written by the SA if the caller passes a nonzero ShardIdx to UpdateRevokedCertificate. The RA was never passing a nonzero ShardIdx to UpdateRevokedCertificate.

Note: “SA” and “RA” here are referencing our Boulder components, and “RA” is not a typical “Registration Authority”.

This prompted an investigation covering two primary points:

  1) What language in the Baseline Requirements and Root Programs covers the "upgrade" of reason codes in CRLs, or the expectations around re-revocation?
  2) Since this bug did not affect our OCSP infrastructure, what are the implications of CRLs and OCSP responses containing different reason codes for a single revoked certificate?

For the first point, we found what we believe to be clear language in Mozilla's Root Store Policy and the BRs:

- First paragraph of Mozilla's Root Store Policy 6.1.1

When an end entity TLS certificate (i.e. a certificate capable of being used for TLS-enabled servers) is revoked for one of the reasons below, the specified CRLReason MUST be included in the reasonCode extension of the CRL entry corresponding to the end entity TLS certificate, as described in sections 4.9.1 and 7.2.2 of the TLS BRs.

- Final paragraph of Baseline Requirements 7.2.2

When a CA obtains verifiable evidence of Key Compromise for a Certificate whose CRL entry does not contain a reasonCode extension or has a reasonCode extension with a non‐keyCompromise reason, the CA SHOULD update the CRL entry to enter keyCompromise as the CRLReason in the reasonCode extension.

For the second point, language specifically discussing the possibility of a mismatch between CRL and OCSP revocation reasons was harder to find. What we did find seemed to represent a design of eventual consistency, which makes operational sense given the very different nextUpdate requirements for CRLs vs OCSP, and possible variability for software that is publishing CRLs vs software responding to OCSP requests.

Even if "SHOULD update the CRL entry" appears to leave room for this sort of thing, it did not match our expectations on our system. We fixed the bug, deployed the new version, and we tracked-down three certificates in our database which would have shown as Revoked in both CRLs and OCSP, but which would have shown reason code "keyCompromise" in OCSP while retaining their original reason code in CRLs (all three happened to be "unspecified"). Re-revocation is pretty rare, and re-revocation with the new reason being keyCompromise is even more rare. We started using the previously-mentioned revokedCertificates table in early March, and only accumulated these three. We updated the affected rows, and all is consistent.

Thank you for reading, we welcome questions and discussion.

--
Kiel C
SRE at Let's Encrypt

[Ben Wilson]

Thanks, Kiel. This is very helpful for the community, and it's important to Mozilla that we are able to alert our users to revocations due to key compromise. While we're on the topic—if anyone has suggestions for improving Mozilla's guidance on revocation reason codes, especially around key compromise, we'd love your input. We are always open to suggestions on how we can improve the use of revocation reason codes.

Thanks again,

Ben

--
You received this message because you are subscribed to the Google Groups "CCADB Public" group.
To unsubscribe from this group and stop receiving emails from it, send an email to public+un...@ccadb.org.
To view this discussion visit https://groups.google.com/a/ccadb.org/d/msgid/public/CABK0L0038%2Bo235fz7oM2YkQDNhdBVdF8DQmKsQ%2B--bq%3DHbtC-Q%40mail.gmail.com.

[Jeremy Rowley]

This is super interesting. What does it mean when OCSP and CRL entries don't match?  You're right that this isn't a compliance issue that I can tell (and 6960 only says they should match). 

Can I ask a tangential question? How are you defining sharted certs? I'm familiar with sharded CT logs but what is a CRL shard? Are you just using standard CRL partitioning or is it something more? 

On Thu, Apr 17, 2025 at 1:00 PM 'Kiel Christofferson' via CCADB Public <pub...@ccadb.org> wrote:

--

[Aaron Gable]

We use "sharding" and "partitioning" as synonyms. Our issuing intermediates (e.g. E5 and R10) each issue 128 different CRLs. Each subscriber certificate is uniquely mapped to one of those 128 shards/partitions. We've recently been changing how that mapping works (from a computation based on the cert's notAfter date to a computation based on the cert's serial number), and this bug was introduced as part of that changeover. The key phrase in that technical paragraph is "explicitly sharded", which is how we internally refer to certificates using the new serial-based mapping scheme.

Aaron

To view this discussion visit https://groups.google.com/a/ccadb.org/d/msgid/public/CAFK%3DoS9skMUBynCUjgfRFO9Y4w8uzQ_bLbvhMrZzxDdVzW%3DO%3DQ%40mail.gmail.com.