Retiring DigiCert Log Server (aka "CT1") in Chrome

[Devon O'Brien]

DigiCert turned down their DigiCert Log Server (also known as “CT1”) on January 4, 2022, and as such, Chrome will be transitioning this log to Retired 2 weeks from this announcement. This log’s turn down was announced in December on the certificate-...@googlegroups.com group, which is usually reserved for technical discussions about CT implementation; however, we are now actively monitoring that group for log announcements as well. Going forward, we hope to streamline future log Retirements by clarifying our communications expectations with an update to https://goo.gl/chrome/ct-log-policy/. 

Effective 2022-01-24, the following log(s) will transition to Retired, with the last ‘Qualified’ SCT having a timestamp no later than 1642982400, or 2022-01-24T00:00:00Z in ISO 8601 format:

• DigiCert Log Server (https://ct1.digicert-ct.com/log/ct/v1/)

After 2022-01-24, SCTs from DigiCert Log Server will no longer count towards the CT Compliance requirements stating that at least one SCT must come from a CT log that was Qualified, Usable, or ReadOnly at time of check. As such, after this point, it is no longer appropriate to serve SCTs from this log in the TLS handshake, in OCSP responses, or embedded in certificates issued on-or-after 2022-01-24T00:00:00Z. 

Embedded SCTs dated prior to 2022-01-24T00:00:00Z will still satisfy CT Compliance requirements that permit SCTs to come from CT logs that are Qualified, Usable, ReadOnly, or Retired at time of check; however, at least one other SCT must come from a non-Retired CT log in order for the certificate to successfully validate in Chrome.

What does this mean for site operators?

If you are delivering SCTs embedded in the certificate, this should require no action on your part. All previously-issued certificates containing SCTs from this log that complied with the Chrome CT Policy will continue to do so.

If you are currently serving SCTs via OCSP, then your CA must take appropriate action to update their OCSP pipeline to include at least one SCT from a non-Google operated log that is Qualified, Usable, or ReadOnly. Once done, you must refresh the OCSP response stapled to the connection. Alternatively, you may choose to provide a policy-satisfying set of SCTs via another mechanism outlined in the Chrome CT Policy.

If you are currently delivering SCTs via a TLS extension, SCTs issued by this log will no longer contribute towards CT Compliance. As such, you must begin to serve SCTs from a separate non-Retired, non-Google log by 2022-01-24, in order to satisfy the CT requirements for SCTs delivered via TLS.

What does this mean for CAs?

If you are embedding SCTs in your certificates, SCTs from this log for newly-issued certificates will no longer meet CT Compliance requirements stating that at least one SCT must come from a CT log that was Qualified, Usable, or ReadOnly at time of check. In order to ensure that newly-issued certificates will be CT-compliant, you should update your CT log configuration to remove the DigiCert Log Server while also ensuring that you are still logging a policy-satisfying set of CT logs after the removal.

While it is not required by policy, CAs with existing certificates embedding SCTs from this log may wish to proactively reissue affected certificates to increase the resilience of these certificates to possible future log incidents.

If you are embedding SCTs from this log in your OCSP responses, you must issue new OCSP responses before 2022-01-24, which replace these SCTs with SCTs issued from another non-Retired, non-Google log. Your customers must then begin to serve these new responses or provide a policy-satisfying set of SCTs via another mechanism.

What does this mean for Log Operators?

If you are operating a CT log not listed in the above Retirement announcement, you do not need to take any action.

[Andrew Ayer]

Between 2022-01-11 00:15 UTC and 2022-01-11 02:30 UTC, this log
came back online and published a new STH with a larger tree size:

{
"sth_version": 0,
"tree_size": 24899054,
"timestamp": 1641816035152,
"sha256_root_hash": "PzzYDT5nw9gfxheu+yp0WkJskPPkVYnZD/MtLBLSqok=",
"tree_head_signature": "BAMARzBFAiEAh0wEZLdEi6GOos7uDgq+Oi6enSdCa/OSV7NnZDxjtmMCIBqH53V8/XAHtTemn/lfa4GRRVd7JFr3JHbp8k+xDgSz"
}

The previous tree size was 24897274.

The newly-published log entries all have timestamps from 2022-01-04.
What this means is that for 7 days, there were qualified-at-time-of-check
SCTs in circulation for log entries which no monitor could
access. Furthermore, any monitor which stopped monitoring this
log in response to the shutdown announcement, or wasn't running
while the log briefly rematerialized, still hasn't seen these log
entries. For example, neither crt.sh nor the Google-operated mirror
(https://ct.googleapis.com/logs/us1/mirrors/digicert_ct1/) have seen
these entries.

There's nothing stopping this log from issuing further SCTs in the next
two weeks which can be used to satisfy the qualified-at-time-of-check SCT
requirement.

I realize that the multi-week delay retiring logs isn't new, but it
seems like something that needs to be fixed, especially as the
Google Log requirement goes away. Even if no log operator behaves
maliciously, this delay can still be exploited to get browsers to
accept effectively-unlogged certificates:

1. Log A is already retired because its private key was compromised.

2. Log B (operated by a different operator) announces a shutdown time.

3. Immediately before the shutdown, an attacker with a misissued
certificate/precertificate submits the precertificate to Log B and
obtains an SCT. The log shuts down before any monitor has a chance to
see the precertificate.

4. The attacker presents the certificate to browsers with a forged SCT
from Log A and the SCT from Log B. Since Log B is not yet retired,
the certificate will comply with both Apple and Chrome CT policy (modulo
Chrome's Google Log requirement).

Regards,
Andrew

[Jeremy Rowley]

Just FYI on what you say yesterday - We spun up the log to grab some information to answer a couple of inquiries. While it was online, the log generated a new treehead and logged the pending CT requests. We shut down the log after grabbing the information. 

[Alex Cohn]

Why were there pending CT requests? Were these submitted and issued SCTs before the log was shut down the first time, and sat pending for a week while the log was down? Or were these submitted after the log was brought back up yesterday?

It seems like it's standard practice when decommissioning a log to empty its list of allowed roots at least 1 MMD before pulling the plug, to ensure monitors can see that either 1. all entries given a SCT have been incorporated into the Merkle tree or 2. the log's MMD has been blown. Was this done with CT1?

--
You received this message because you are subscribed to the Google Groups "Certificate Transparency Policy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ct-policy+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/ct-policy/6011b114-1a58-45a8-ab83-88cc7050e2d2n%40chromium.org.

[Jeremy Rowley]

There's always pending requests to a log. Anyone can submit to a log until it is shut down so we didn't worry about it too much. We announced the shut down in 2019, warning CAs that they should use a different log for a quorum of proofs.

I don't think there is a standard practice for retiring a log is there? I haven't seen any sort of documentation on it anyway.  We didn't clear out the queue, letting people submit until the posted shut down date. We probably should have cleared out the pending requests after shut down, but that is in hindsight. 

[Rob Stradling]

> For example, neither crt.sh nor the Google-operated mirror have seen these entries.

crt.sh hasn't yet stopped attempting to fetch STHs and entries from CT1, so I can't explain this.

Jeremy: Would it be possible for DigiCert to turn CT1 back on again (in some sort of read-only manner) so that all the monitors can catch up?

---

From: ct-p...@chromium.org <ct-p...@chromium.org> on behalf of Andrew Ayer <ag...@andrewayer.name>
Sent: 12 January 2022 17:13
To: ct-p...@chromium.org <ct-p...@chromium.org>
Subject: Re: [ct-policy] Retiring DigiCert Log Server (aka "CT1") in Chrome

 

CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.

Between 2022-01-11 00:15 UTC and 2022-01-11 02:30 UTC, this log
came back online and published a new STH with a larger tree size:

    {
      "sth_version": 0,
      "tree_size": 24899054,
      "timestamp": 1641816035152,
      "sha256_root_hash": "PzzYDT5nw9gfxheu+yp0WkJskPPkVYnZD/MtLBLSqok=",
      "tree_head_signature": "BAMARzBFAiEAh0wEZLdEi6GOos7uDgq+Oi6enSdCa/OSV7NnZDxjtmMCIBqH53V8/XAHtTemn/lfa4GRRVd7JFr3JHbp8k+xDgSz"
    }

The previous tree size was 24897274.

The newly-published log entries all have timestamps from 2022-01-04.
What this means is that for 7 days, there were qualified-at-time-of-check
SCTs in circulation for log entries which no monitor could
access.  Furthermore, any monitor which stopped monitoring this
log in response to the shutdown announcement, or wasn't running
while the log briefly rematerialized, still hasn't seen these log
entries.  For example, neither crt.sh nor the Google-operated mirror

(https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fct.googleapis.com%2Flogs%2Fus1%2Fmirrors%2Fdigicert_ct1%2F&amp;data=04%7C01%7Crob%40comodoca.com%7C2c981f4021114cc2897408d9d5ef139f%7C0e9c48946caa465d96604b6968b49fb7%7C0%7C0%7C637776045085205904%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=FS8KH1HqNYYZFxp67JZZv41ybQIAAwC7Yz2%2FwhxgjXw%3D&amp;reserved=0) have seen

these entries.

There's nothing stopping this log from issuing further SCTs in the next
two weeks which can be used to satisfy the qualified-at-time-of-check SCT
requirement.

I realize that the multi-week delay retiring logs isn't new, but it
seems like something that needs to be fixed, especially as the
Google Log requirement goes away.  Even if no log operator behaves
maliciously, this delay can still be exploited to get browsers to
accept effectively-unlogged certificates:

1. Log A is already retired because its private key was compromised.

2. Log B (operated by a different operator) announces a shutdown time.

3. Immediately before the shutdown, an attacker with a misissued
certificate/precertificate submits the precertificate to Log B and
obtains an SCT.  The log shuts down before any monitor has a chance to
see the precertificate.

4. The attacker presents the certificate to browsers with a forged SCT
from Log A and the SCT from Log B.  Since Log B is not yet retired,
the certificate will comply with both Apple and Chrome CT policy (modulo
Chrome's Google Log requirement).

Regards,
Andrew

--
You received this message because you are subscribed to the Google Groups "Certificate Transparency Policy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ct-policy+...@chromium.org.

To view this discussion on the web visit https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgroups.google.com%2Fa%2Fchromium.org%2Fd%2Fmsgid%2Fct-policy%2F20220112101324.4eafeee3522ad48b05330d6f%2540andrewayer.name&amp;data=04%7C01%7Crob%40comodoca.com%7C2c981f4021114cc2897408d9d5ef139f%7C0e9c48946caa465d96604b6968b49fb7%7C0%7C0%7C637776045085205904%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=iWpcNB%2BpBYw32gr8TITImjjXCZw8dWAVdwZfGoldP%2Fs%3D&amp;reserved=0.

[Jeremy Rowley]

Sure. We'll turn it back on for roughly 24 hours so the monitors can catch up. Let us turn off tree head signing so the last tree head is valid longer than 12 hours and turn off new logging so we don't get a flood of new certs when we turn it back on. 

[Rob Stradling]

> crt.sh hasn't yet stopped attempting to fetch STHs and entries from CT1, so I can't explain this.

Ah, I see why.  crt.sh doesn't like the fact that the log's server cert has expired (https://crt.sh/?id=2133814869).

---

From: ct-p...@chromium.org <ct-p...@chromium.org> on behalf of Rob Stradling <r...@sectigo.com>
Sent: 12 January 2022 20:45
To: Andrew Ayer <ag...@andrewayer.name>; ct-p...@chromium.org <ct-p...@chromium.org>; Jeremy Rowley <jeremy...@digicert.com>

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/ct-policy/MW4PR17MB4729BDD52950C1A821AE836FAA529%40MW4PR17MB4729.namprd17.prod.outlook.com.

[Jeremy Rowley]

Yeah - we actually chose the shutdown date to be before the cert expired. The log is up and running now if you want to grab any info.

[Jeremy Rowley]

The last good head we issued should be:
{ "tree_size": 24899054, "timestamp": 1641978042733, "sha256_root_hash": "PzzYDT5nw9gfxheu+yp0WkJskPPkVYnZD/MtLBLSqok=", "tree_head_signature": "BAMARzBFAiEA3qec2WrkRPXPgOfB1NjJFtdvDEoWhqk0P1U7KQDJurkCIExr3Ipqe2ym32NIBo6XUbf3k4s+ev8vCdecevP0IjqH" }

We checked the queue and there is nothing in it.

I can leave the log up for longer than 24 hours. What would you like?

[Alex Cohn]

To be clear, by "pending requests" we're talking about add-chain/add-pre-chain requests that resulted in a SCT issuance, but have not yet been sequenced into the log's Merkle tree and covered by a STH, right? 

Chromium's CT Policy requires a log to "Verifiably incorporat[e] all certificates into the Log for which SCTs have been issued, within the Log’s MMD". As a relying party, imo that's the core role of a CT log: if I have a SCT, I know the corresponding certificate is available for public inspection. 

A log intending to cease operation gracefully can stop accepting new submissions beforehand: it can return '{"certificates":[]}' from get-roots and reject all incoming add-chain and add-pre-chain requests with an appropriate HTTP 4xx error.

As an example concern, a future CT auditor attempting to verify the embedded CT1 SCT in [1] using a mirror would have been unable to do so, had you not turned the log back up today. 

I've been unable to find where I saw the "let 1MMD pass after you stop accepting certificates before you turn down the log entirely" advice, and I'm now thinking it might have applied instead to read-only transitions? (I might have even subconsciously reverse-engineered it from the Aviator freeze: the timestamp on the most recent Aviator entry is 2016-11-29 12:51:16.316 UTC, but the last STH was issued just over 24h later at 2016-11-30 13:24:18.330 UTC)

Maybe Chromium should publish a cradle-to-grave set of best practices for log operations?

Alex

1. https://crt.sh/?sha256=D565FA1FD75467B013765555295A33DE25DBA1780C0BA09A0F6C2C1AE5A47743

[Rob Stradling]

Thanks.  crt.sh now has all the entries from CT1.

---

From: Jeremy Rowley <rowl...@gmail.com>
Sent: 12 January 2022 23:08
To: Certificate Transparency Policy <ct-p...@chromium.org>
Cc: Rob Stradling <r...@sectigo.com>; Andrew Ayer <ag...@andrewayer.name>; Jeremy Rowley <ct-p...@chromium.org>

[Jeremy Rowley]

Okay cool. I'll plan on turning off the log again on Monday.

[Rasmus Dahlberg]

Hi Jeremy,

Could you state the log's final signed tree head so that there is no
ambiguity about that?  If you are turning the log back on in a read-only
mode, could you keep it available for more than 24 hours?  I suspect
some monitors need more time than that to notice this issue and respond.

Is it correct to assume that there are no more pending entries now?

-Rasmus

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/ct-policy/217e6000-96d2-49b8-aba8-1210e558314an%40chromium.org.

[Rasmus Dahlberg]

(Looks like my earlier message appeared a bit later on the list.)

Thank you Jeremy for answering my questions and keeping the log
available for longer than 24 hours!  Unless anyone else says Monday is
too short I am not going to nudge you further on this matter.  I just
wanted to increase the likelihood that monitors can catch up. :)

-Rasmus

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/ct-policy/10fbc80e-309c-4302-988e-1ad672604d79n%40chromium.org.

[Kurt Roeckx]

On Wed, Jan 12, 2022 at 03:08:24PM -0800, Jeremy Rowley wrote:
> Yeah - we actually chose the shutdown date to be before the cert expired.
> The log is up and running now if you want to grab any info.

I failed to get any info because it's still using an expired
certificate.


Kurt

[Andrew Ayer]

It's a bit annoying, but logs are not required to use valid
certificates, and some logs have used certificates issued by CAs that
are not widely trusted.

In my experience, you need to disable certificate validation to
effectively monitor CT logs. Arguably, CT doesn't need transport-layer
authentication since most[1] of the stuff that needs authentication
is signed directly or indirectly by the log's key. If the monitor is
well-written - checking the signature of STHs and SCTs, and verifying
that get-entries produces the expected STH root hash - then the worst
an active attacker can do is cause a denial-of-service, which they can
do anyways. Monitors need to do these checks in any case, because
the log itself might be the attacker, returning get-entries responses
that don't match the STH. In fact, every time my
non-certificate-validating monitor has detected an invalid get-entries
response, it has not been a MitM, but the log operator screwing up!

[1] The only exception is get-roots, but log operators are required
to report changes to their accepted roots out-of-band via the Chromium
bug tracker.

Regards,
Andrew

[Andrew Ayer]

Unfortunately, the Google mirror
(https://ct.googleapis.com/logs/us1/mirrors/digicert_ct1) didn't
pick up the additional entries before CT1 shut down again.

I've attached a get-entries response for the range [24897274,
24899053] in case anyone wants the additional entries.

Andrew