Upcoming Sabre CT Log Maintenance Window (switch to Trillian)

[Nicolas Dufour]

Hello everyone,

We are planning to switch Sabre CT log (sabre.ct.comodo.com) to Trillian on the following date: Wednesday March 15th 2023.

The operation will start around 11am EST: the log will be read-only (any POST request will be rejected), and will last 2-4 hours. We will send notifications when it starts and ends.

In order to offer a more reliable Sabre back to the CT ecosystem, we believe performing this migration sooner is better.
We welcome any differing opinions and are open to change the date / time if this is found to be too short notice.

We plan to build brand new temporal sharded logs for Sabre and Mammoth. The current timeline is slated for two weeks after Sabre's switch to Trillian. This is subject to change based on Sabre's Trillian implementation.

Please forward this message to people who might be impacted.

Thank you,
Nicolas on behalf of Sectigo.

[Nicolas Dufour]

Hello all,

Sabre CT log (sabre.ct.comodo.com) is now switched to read-only mode: any POST request will be rejected.

I will send another email once it's back accepting requests.

Thank you,

Nicolas

[Nicolas Dufour]

Hello all,

Sabre CT log (sabre.ct.comodo.com) is now back online and accepting certificates.

It's now on Trillian.

Thank you,

Nicolas

[Nicolas Dufour]

Hello,

I want to add a big thank you to the Let's Encrypt folks who were a precious help.

Especially Phil Porada on his support both at the database level and metrics.

Sabre seems to run smoothly and we'll use it as a guide for the future logs for sure.

Thank you,

Nicolas

[Phil Porada]

:) Glad to be of assistance. Here's to a long life for the Sabre log!

[Bineesh]

Will there be any change to the CT Log api end point for clients to submit certificate - https://sabre.ct.comodo.com/ct/v1/add-pre-chain ?

[Bineesh]

Also any chance your log id changed from  VYHUwhaQNgFK6gubVzxT8MDkOHhwJQgXL6OqHQcT0ww= to

23b9raxl59CVCIhuIVm9i5A1L1/q0+PcXiLrNQrMe5g=

[Bineesh]

It seems Google chrome is rejecting SCTs from this log

[Nicolas Dufour]

Hello Bineesh,

Thanks a lot for mentioning it.

We did see a discrepancy this morning on the keypair that sabre used to have on super dupper: precisely the private key that was used by the CTFE was different, whereas the public key was identical.

I fixed it this morning around 9-10am EST.

On top of that, it seems that the default max_get_entries (1000) was too high for the server to handle. So GetEntries requests with large range were all failing.

We then changed it to 500 and no ill behavior is seen since.

Thank you,

Nicolas

[Bineesh]

Thank you for looking into it. Then all the certificates submitted to Sabre during this issuer are not failing at Google/Apple CT log validation. 

What's the time window were Sabre signing SCT tokens with wrong key? Aren't all the CAs using Sabre impacted with this issue ?

[Rasmus Dahlberg]

Hi Nicolas,

I think you need to look further into the get-entries endpoint. Still failing:

    $ curl -G --data-urlencode "start=2000" --data-urlencode "end=3000" https://sabre.ct.comodo.com/ct/v1/get-entries
    Forbidden
    backend GetLeavesByRange request failed: rpc error: code = ResourceExhausted desc = grpc: received message larger than max (6586447 vs. 4194304)

-Rasmus

--
You received this message because you are subscribed to the Google Groups "Certificate Transparency Policy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ct-policy+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/ct-policy/7317fed5-2b4f-4be8-9745-18aacf377b6bn%40chromium.org.

[Nicolas Dufour]

Hello Rasmus,

Thanks a lot, I ended up testing the sweet spot for that request and it happens around 310.

So I lowered the max entries once more to 256.

Hopefully, this value would be lower enough to be under the 4MB buffer size of the grpc call.

Nicolas

[Nicolas Dufour]

Hello Bineesh,

For the time window, I would say from the moment it was live on Trillian, so yesterday (Wednesday March 15th) at around 12:30pm EST to today (Thursday March 16th) at 8:28am EST.

Nicolas

[Andrew Ayer]

On Thu, 16 Mar 2023 15:02:45 -0700 (PDT)
Bineesh <bin...@gmail.com> wrote:

> Thank you for looking into it. Then all the certificates submitted to
> Sabre during this issuer are not failing at Google/Apple CT log
> validation. What's the time window were Sabre signing SCT tokens with
> wrong key? Aren't all the CAs using Sabre impacted with this issue ?

CAs that don't validate SCT signatures are impacted. CAs that are
doing it right would have rejected the SCTs with invalid signatures and
gotten SCTs from other logs.

Cert Spotter authenticates and audits every SCT that it sees embedded in a
certificate. So far we have not observed any embedded SCTs with this
other log ID. However. not all CAs submit final certificates so we
don't have a complete picture.

Regards,
Andrew

[Aaron Gable]

It's worth noting that the certificate-transparency-go library automatically checks the signature on the SCT as part of all add-chain requests: https://github.com/google/certificate-transparency-go/blob/9f7f7b6d171455621e130e25312efec7f8b16dd2/client/logclient.go#L106-L108

--
You received this message because you are subscribed to the Google Groups "Certificate Transparency Policy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ct-policy+...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/ct-policy/20230316200118.1b1a81e265abe747090f9994%40andrewayer.name.

[Jeremy Rowley]

This seems like a pretty big failing, especially where there hasn't been any communication to CAs on why their certs might be failing in Chrome. There probably should be some CT policy around communicating to impacted CAs and monitors when an event like this happens. Searching Censys, there's about a dozen CAs with impacted certs that might not know why they're having issues.

[Joe DeBlasio]

Hi folks,

Chrome is treating the period where Sabre was returning SCTs signed with the wrong key as a period of reduced availability. This is consistent with how we would treat a log returning any other form of invalid response.

Log operators and other interested parties are expected to report issues to ct-policy@ (as has happened in this case), and CAs are expected to monitor this forum for relevant announcements. We'll add additional language to https://goo.gl/chrome/ct-log-policy to clarify our reporting expectations for log operators.

Ultimately, CAs are responsible for the contents of certificates they issue -- CAs should validate returned SCTs (validating both the signature and that it corresponds to the correct certificate) before relying on that SCT or forwarding it on to their subscribers.

Best,
Joe, Devon, Carlos, and the rest of the Chrome CT team

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/ct-policy/460caa8c-5de7-46d6-8089-faee9124c3bcn%40chromium.org.

[Jeremy Rowley]

The practical effect is worse than a typical log outage because certs were issued with incorrectly signed SCTs received from a trusted log. Sure CAs should have checked the signature of the SCT, but that wasn't a policy or tools (until now) for that.  Calling it the same as a traditional outage doesn't address that the real world impact was quite different. Mostly, I think there should have been some easy way to tell "these are the certs that had the wrong SCTs" from the log operator. Something to make it easy to find an replace certs that didn't get the correct SCTs back from the log.

[Rob Stradling]

> Mostly, I think there should have been some easy way to tell "these are the certs that had the wrong SCTs" from the log operator.

Hi Jeremy.  I won't comment on whether or not that's a reasonable expectation.

I will say that we have not yet been able to find the time/resources to complete (as far as possible(*)) that analysis.

(*) Quoting Andrew: "not all CAs submit final certificates so we don't have a complete picture."

---

From: ct-p...@chromium.org <ct-p...@chromium.org> on behalf of Jeremy Rowley <rowl...@gmail.com>
Sent: 22 March 2023 22:36
To: Certificate Transparency Policy <ct-p...@chromium.org>
Cc: Joe DeBlasio <jdeb...@chromium.org>; Certificate Transparency Policy <ct-p...@chromium.org>; Aaron Gable <aa...@letsencrypt.org>; Bineesh <bin...@gmail.com>; Nicolas Dufour <nrdu...@gmail.com>; Phil Porada <ppo...@letsencrypt.org>; Andrew Ayer <ag...@andrewayer.name>; Jeremy Rowley <rowl...@gmail.com>
Subject: Re: [ct-policy] Re: Upcoming Sabre CT Log Maintenance Window (switch to Trillian)

 

CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/ct-policy/88f0c5cd-4a56-4c61-91b3-ee855e8dd54an%40chromium.org.

[Rob Stradling]

> I will say that we have not yet been able to find the time/resources to complete (as far as possible(*)) that analysis.

Hi Jeremy.

I've hacked together a script (see https://gist.github.com/robstradling/e494d1f8ff561812782f6616be373ff4) that has scanned the crt.sh DB to look for all logged certificates that contain Dodo's Log ID, since that is the Log ID that Sabre would have been emitting in SCTs during the incident.  Since Dodo is not a trusted log, it seems likely that this will match the data you're looking for.  See below for a summary of the results.

Caveats:

  - crt.sh's CT monitor has fallen considerably behind on fetching entries from argon2023 and xenon2023 (see https://crt.sh/monitored-logs), but following some performance improvements I implemented a few weeks ago it is gradually catching up.  It has yet to reach the time period of the Sabre incident though.

  - As previously stated, there is no particular reason why any of these certificates should have been submitted to any logs.  There may well be many more certificates affected that have not been logged.

┌───────┬──────────────────────────────────────────────────────────────────────────────────────────────────────┬──────────────────────────────────────────────────────┐

│ count │                                                 name                                                 │                       ca_owner                       │

├───────┼──────────────────────────────────────────────────────────────────────────────────────────────────────┼──────────────────────────────────────────────────────┤

│     1 │ C=AT, O=e-commerce monitoring GmbH, CN=GLOBALTRUST 2020 SERVER OV 1                                  │ e-commerce monitoring GmbH                           │

│     9 │ C=CH, O=SwissSign AG, CN=SwissSign RSA TLS DV ICA 2021 - 1                                           │ SwissSign AG                                         │

│     1 │ C=CH, O=SwissSign AG, CN=SwissSign RSA TLS DV ICA 2022 - 1                                           │ SwissSign AG                                         │

│     3 │ C=CH, O=SwissSign AG, CN=SwissSign RSA TLS EV ICA 2021 - 1                                           │ SwissSign AG                                         │

│    16 │ C=CH, O=SwissSign AG, CN=SwissSign RSA TLS OV ICA 2021 - 1                                           │ SwissSign AG                                         │

│     5 │ C=CH, O=SwissSign AG, CN=SwissSign RSA TLS OV ICA 2022 - 1                                           │ SwissSign AG                                         │

│     1 │ C=HU, L=Budapest, O=NETLOCK Kft., OU=Tanúsítványkiadók (Certification Services), CN=NETLOCK DVSSL CA │ Netlock                                              │

│     1 │ C=KR, O=NAVER BUSINESS PLATFORM Corp., CN=NAVER Secure Certification Authority 1                     │ NAVER Cloud                                          │

│     1 │ C=NO, O=Buypass AS-983163327, CN=Buypass Class 2 CA 5                                                │ Buypass                                              │

│     4 │ C=US, O="Cloudflare, Inc.", CN=Cloudflare Inc ECC CA-3                                               │ DigiCert                                             │

│     2 │ C=US, O="Cloudflare, Inc.", CN=Cloudflare Inc RSA CA-2                                               │ DigiCert                                             │

│     3 │ C=US, O=Microsoft Corporation, CN=Microsoft Azure TLS Issuing CA 01                                  │ Microsoft Corporation                                │

│     1 │ C=US, O="Root Networks, LLC", CN=Root CA - G3                                                        │ Asseco Data Systems S.A. (previously Unizeto Certum) │

└───────┴──────────────────────────────────────────────────────────────────────────────────────────────────────┴──────────────────────────────────────────────────────┘

---

From: 'Rob Stradling' via Certificate Transparency Policy <ct-p...@chromium.org>
Sent: 23 March 2023 16:39
To: Jeremy Rowley <rowl...@gmail.com>; Certificate Transparency Policy <ct-p...@chromium.org>
Cc: Joe DeBlasio <jdeb...@chromium.org>; Aaron Gable <aa...@letsencrypt.org>; Bineesh <bin...@gmail.com>; Nicolas Dufour <nrdu...@gmail.com>; Phil Porada <ppo...@letsencrypt.org>; Andrew Ayer <ag...@andrewayer.name>

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/ct-policy/MW4PR17MB47295BF8112EB22E5C47BCFBAA879%40MW4PR17MB4729.namprd17.prod.outlook.com.

[Rob Stradling]

I've done a port 443 scan for all the dNSNames in certificates that were logged to Sabre during the incident.  This scan found another 566 certificates that contain Dodo's Log ID, and I've submitted all of these certificates to the Dodo log itself (which crt.sh monitors).  Below is an updated per-issuing-CA summary of the number of affected certificates that have been observed so far.

┌───────┬──────────────────────────────────────────────────────┬──────────────────────────────────────────────────────┐
│ count │                       ca_name                        │                       ca_owner                       │
├───────┼──────────────────────────────────────────────────────┼──────────────────────────────────────────────────────┤
│    17 │ Shoper® SSL                                          │ Asseco Data Systems S.A. (previously Unizeto Certum) │
│    87 │ Certyfikat SSL                                       │ Asseco Data Systems S.A. (previously Unizeto Certum) │
│    17 │ www.lh.pl                                            │ Asseco Data Systems S.A. (previously Unizeto Certum) │
│   100 │ Certum Domain Validation CA SHA2                     │ Asseco Data Systems S.A. (previously Unizeto Certum) │
│     9 │ Root CA - G3                                         │ Asseco Data Systems S.A. (previously Unizeto Certum) │
│     1 │ Buypass Class 2 CA 5                                 │ Buypass                                              │
│     1 │ certSIGN Enterprise CA Class 3 G2                    │ certSIGN                                             │
│     1 │ CFCA OV OCA                                          │ China Financial Certification Authority (CFCA)       │
│     3 │ Public Certification Authority - G2                  │ Chunghwa Telecom                                     │
│    36 │ Cybertrust Japan SureServer CA G4                    │ Cybertrust Japan Co., Ltd.                           │
│    13 │ Cybertrust Japan SureServer EV CA G3                 │ Cybertrust Japan Co., Ltd.                           │
│     4 │ Cloudflare Inc ECC CA-3                              │ DigiCert                                             │
│     2 │ Cloudflare Inc RSA CA-2                              │ DigiCert                                             │
│     8 │ DigiCert Global G2 TLS RSA SHA256 2020 CA1           │ DigiCert                                             │
│     2 │ DigiCert TLS RSA SHA256 2020 CA1                     │ DigiCert                                             │
│    27 │ GeoTrust Global TLS RSA4096 SHA256 2022 CA1          │ DigiCert                                             │
│     3 │ Encryption Everywhere DV TLS CA - G2                 │ DigiCert                                             │
│     1 │ GLOBALTRUST 2020 SERVER OV 1                         │ e-commerce monitoring GmbH                           │
│   282 │ Microsoft Azure TLS Issuing CA 01                    │ Microsoft Corporation                                │
│   258 │ Microsoft Azure TLS Issuing CA 02                    │ Microsoft Corporation                                │
│    66 │ Microsoft Azure TLS Issuing CA 05                    │ Microsoft Corporation                                │
│    75 │ Microsoft Azure TLS Issuing CA 06                    │ Microsoft Corporation                                │
│    19 │ 政府伺服器數位憑證管理中心 - G1                      │ National Development Council                         │
│     1 │ NAVER Secure Certification Authority 1               │ NAVER Cloud                                          │
│     1 │ NETLOCK DVSSL CA                                     │ Netlock                                              │
│     9 │ SwissSign RSA TLS DV ICA 2021 - 1                    │ SwissSign AG                                         │
│     1 │ SwissSign RSA TLS DV ICA 2022 - 1                    │ SwissSign AG                                         │
│     3 │ SwissSign RSA TLS EV ICA 2021 - 1                    │ SwissSign AG                                         │
│    16 │ SwissSign RSA TLS OV ICA 2021 - 1                    │ SwissSign AG                                         │
│     5 │ SwissSign RSA TLS OV ICA 2022 - 1                    │ SwissSign AG                                         │
│    10 │ Trustwave Organization Validation SHA256 CA, Level 1 │ Viking Cloud, Inc.                                   │
└───────┴──────────────────────────────────────────────────────┴──────────────────────────────────────────────────────┘

---

From: Rob Stradling <r...@sectigo.com>
Sent: 23 March 2023 18:33
To: Jeremy Rowley <rowl...@gmail.com>; Certificate Transparency Policy <ct-p...@chromium.org>; Rob Stradling <r...@sectigo.com>

[Seo Suchan]

How many of those certificates will fail to verify on chrome because not have enough valid SCTs?

(they always able to use other protocol (OCSP and TLS) for SCT yet, so I don't think this is misinsurance or something

2023년 3월 24일 금요일 오후 8시 55분 27초 UTC+9에 Rob Stradling님이 작성:

[Rob Stradling]

> How many of those certificates will fail to verify on chrome because not have enough valid SCTs?

I haven't done that analysis, but I'd be surprised if it's less than all of them.

> (they always able to use other protocol (OCSP and TLS) for SCT yet, so I don't think this is misinsurance or something

Correct, it's not misissuance.  However, some of the affected certificates may need to be reissued anyway.  OCSP Stapling and the signed_certificate_timestamp TLS extension probably aren't available to all affected Subscribers.

BTW, for some more analysis on this incident, the SSLMate blog is worth a read: https://sslmate.com/blog/post/march_2023_ct_blunders

---

From: Seo Suchan <tjt...@gmail.com>
Sent: 24 March 2023 22:01

To: Certificate Transparency Policy <ct-p...@chromium.org>

Cc: Rob Stradling <r...@sectigo.com>; Joe DeBlasio <jdeb...@chromium.org>; Aaron Gable <aa...@letsencrypt.org>; Bineesh <bin...@gmail.com>; Nicolas Dufour <nrdu...@gmail.com>; Phil Porada <ppo...@letsencrypt.org>; Andrew Ayer <ag...@andrewayer.name>; Jeremy Rowley <rowl...@gmail.com>

[Jeremy Rowley]

It’s not all of them because some of the certs contain more than the minimum number of logs. Anything with extra scts in the certs works fine.