Sectigo Sabre Has Grown

Andrew Ayer

unread,

Jun 4, 2025, 1:32:37 PM6/4/25

to ct-p...@chromium.org, ct...@sectigo.com

At 2025-06-04 16:10 UTC, I was gossiped the following STH from Sectigo Sabre:

{"log_id": "VYHUwhaQNgFK6gubVzxT8MDkOHhwJQgXL6OqHQcT0ww=", "timestamp": 1749053231869, "tree_size": 546099881, "sth_version": 0, "sha256_root_hash": "gPvc2thVykvJA4yjn9d/LdVK5EEFMrjnZTbhdO0MMdc=", "tree_head_signature": "BAMARzBFAiEA57zG8W8OugBiNozsz9wdqT+sm6lp1unDFFSyez2+kjwCIFjL7fnWlq56Cas7u5U22u7x5zSc1rm4Hd7+VG98p3bi"}

This STH has a tree size 3 entries larger than the size Sabre was frozen at, and a timestamp more than 2 hours after the announced shutdown time of the log. Since the log has been shut down, my monitor has been unable to download these entries.

This concerns me, because this log is still considered ReadOnly by Apple and Chrome, meaning that the SCTs for these entries will be considered qualified-at-time-of-check, and could be paired with a backdated and unincorporated SCT from a Retired log to satisfy CT policy, a scenario which I previously described here: https://groups.google.com/a/chromium.org/g/ct-policy/c/P5aj4JEBFPM/m/9AEcvY01EQAJ

I hope Sectigo can bring this log back online so monitors can retrieve the new entries, and explain why a supposedly read-only log grew in size after it was shut down.

UAs should be transitioning this log to Retired or Rejected as soon as possible. The log should only contain expired certificates at this point (ignoring these 3 new mystery entries). Ideally the retirement would have already been planned to coincide with the log's shut down. Logs which are Qualified, Usable, or ReadOnly need to be accessible by monitors.

Regards,
Andrew

Martijn Katerbarg

unread,

Jun 4, 2025, 4:15:59 PM6/4/25

to Andrew Ayer, ct-p...@chromium.org, #CTOps

Hi Andrew,

We've observed the same on our end, with crt.sh having been able to ingest these 3 certificates. We are currently investigating how some POST requests have made it past the existing block we have had in place, essentially during the shutdown procedure.

The database has not yet been destroyed. As such, we will bring Sabre back online tomorrow, and re-schedule its extinction, while we investigate what has happened.

Regards,

Martijn Katerbarg
Sectigo

From: Andrew Ayer <ag...@andrewayer.name>
Date: Wednesday, 4 June 2025 at 19:32
To: ct-p...@chromium.org <ct-p...@chromium.org>, #CTOps <ct...@sectigo.com>
Subject: Sectigo Sabre Has Grown

At 2025-06-04 16: 10 UTC, I was gossiped the following STH from Sectigo Sabre: {"log_id": "VYHUwhaQNgFK6gubVzxT8MDkOHhwJQgXL6OqHQcT0ww=", "timestamp": 1749053231869, "tree_size": 546099881, "sth_version": 0, "sha256_root_hash": "gPvc2thVykvJA4yjn9d/LdVK5EEFMrjnZTbhdO0MMdc=",

ZjQcmQRYFpfptBannerStart

This Message Is From an External Sender

This message came from outside your organization.

Report Suspicious

ZjQcmQRYFpfptBannerEnd

At 2025-06-04 16:10 UTC, I was gossiped the following STH from Sectigo Sabre:

{"log_id": "VYHUwhaQNgFK6gubVzxT8MDkOHhwJQgXL6OqHQcT0ww=", "timestamp": 1749053231869, "tree_size": 546099881, "sth_version": 0, "sha256_root_hash": "gPvc2thVykvJA4yjn9d/LdVK5EEFMrjnZTbhdO0MMdc=", "tree_head_signature": "BAMARzBFAiEA57zG8W8OugBiNozsz9wdqT+sm6lp1unDFFSyez2+kjwCIFjL7fnWlq56Cas7u5U22u7x5zSc1rm4Hd7+VG98p3bi"}

This STH has a tree size 3 entries larger than the size Sabre was frozen at, and a timestamp more than 2 hours after the announced shutdown time of the log.  Since the log has been shut down, my monitor has been unable to download these entries.

This concerns me, because this log is still considered ReadOnly by Apple and Chrome, meaning that the SCTs for these entries will be considered qualified-at-time-of-check, and could be paired with a backdated and unincorporated SCT from a Retired log to satisfy CT policy, a scenario which I previously described here: https://urldefense.com/v3/__https://groups.google.com/a/chromium.org/g/ct-policy/c/P5aj4JEBFPM/m/9AEcvY01EQAJ__;!!J5K_pWsD!wO7_N35IdtGqYrQRwayBXAlk7Epo3sS3R2AqGhUq7MbMy0Qfbi6ejo4ob7e82Z9Di6J3P2k-IfCcFYmb$

Martijn Katerbarg

unread,

Jun 5, 2025, 6:53:37 AM6/5/25

to Andrew Ayer, ct-p...@chromium.org

All,

Just a quick update that Sabre has been brought back online just a few minutes ago.

Regards,

Martijn Katerbarg
Sectigo

Andrew Ayer

unread,

Jun 5, 2025, 12:12:19 PM6/5/25

to Martijn Katerbarg, ct-p...@chromium.org

Thanks, Martijn.

The tree grew yet again, up to 546,099,887 entries. I've been able to retrieve all of them.

Good news is that the extra 9 entries are all certificates, not precertificates, so no embedded SCTs are impacted. (The certificates are all recently-issued and from Let's Encrypt; I guess they're still trying to submit final certificates here?)

Regards,
Andrew

On Thu, 5 Jun 2025 10:53:23 +0000
"'Martijn Katerbarg' via Certificate Transparency Policy"

> <https://us-phishalarm-ewt.proofpoint.com/EWT/v1/J5K_pWsD!CGYXkEc1_eA4S3AguRidoMVvtJ-9Mie1YYRS71qdR3OOGcFdA7QAEdvnVJv9VW6EE_btHNXPSDnsZHOKG6VkaWBrsxY0cukQipN3BxtQATY$>

>
>
> ZjQcmQRYFpfptBannerEnd
>
> At 2025-06-04 16:10 UTC, I was gossiped the following STH from
> Sectigo Sabre:
>
>
>
> {"log_id": "VYHUwhaQNgFK6gubVzxT8MDkOHhwJQgXL6OqHQcT0ww=",
> "timestamp": 1749053231869, "tree_size": 546099881, "sth_version": 0,
> "sha256_root_hash": "gPvc2thVykvJA4yjn9d/LdVK5EEFMrjnZTbhdO0MMdc=",
> "tree_head_signature":
> "BAMARzBFAiEA57zG8W8OugBiNozsz9wdqT+sm6lp1unDFFSyez2+kjwCIFjL7fnWlq56Cas7u5U22u7x5zSc1rm4Hd7+VG98p3bi"}
>
>
>
> This STH has a tree size 3 entries larger than the size Sabre was
> frozen at, and a timestamp more than 2 hours after the announced
> shutdown time of the log. Since the log has been shut down, my
> monitor has been unable to download these entries.
>
>
>
> This concerns me, because this log is still considered ReadOnly by
> Apple and Chrome, meaning that the SCTs for these entries will be
> considered qualified-at-time-of-check, and could be paired with a
> backdated and unincorporated SCT from a Retired log to satisfy CT
> policy, a scenario which I previously described here:

> https://urldefense.com/v3/__https://groups.google.com/a/chromium.org/g/ct-policy/c/P5aj4JEBFPM/m/9AEcvY01EQAJ__;!!J5K_pWsD!wO7_N35IdtGqYrQRwayBXAlk7Epo3sS3R2AqGhUq7MbMy0Qfbi6ejo4ob7e82Z9Di6J3P2k-IfCcFYmb$<https://urldefense.com/v3/__https:/groups.google.com/a/chromium.org/g/ct-policy/c/P5aj4JEBFPM/m/9AEcvY01EQAJ__;!!J5K_pWsD!wO7_N35IdtGqYrQRwayBXAlk7Epo3sS3R2AqGhUq7MbMy0Qfbi6ejo4ob7e82Z9Di6J3P2k-IfCcFYmb$>

>
>
>
> I hope Sectigo can bring this log back online so monitors can
> retrieve the new entries, and explain why a supposedly read-only log
> grew in size after it was shut down.
>
>
>
> UAs should be transitioning this log to Retired or Rejected as soon
> as possible. The log should only contain expired certificates at
> this point (ignoring these 3 new mystery entries). Ideally the
> retirement would have already been planned to coincide with the log's
> shut down. Logs which are Qualified, Usable, or ReadOnly need to be
> accessible by monitors.
>
>
>
> Regards,
>
> Andrew
>

> --
> You received this message because you are subscribed to the Google
> Groups "Certificate Transparency Policy" group. To unsubscribe from
> this group and stop receiving emails from it, send an email to
> ct-policy+...@chromium.org. To view this discussion visit
> https://groups.google.com/a/chromium.org/d/msgid/ct-policy/SA1PR17MB65035245BEACD54365014677E36FA%40SA1PR17MB6503.namprd17.prod.outlook.com.

Matthew McPherrin

unread,

Jun 5, 2025, 1:14:45 PM6/5/25

to Andrew Ayer, Martijn Katerbarg, ct-p...@chromium.org

Let's Encrypt does not submit final certs to Sabre. This is presumably somebody else cross-submitting from another log, network scanner, etc.

Let's Encrypt uses Google's v3/log_list.json to determine shard windows and won't try to submit outside of what is specified in that file.

To view this discussion visit https://groups.google.com/a/chromium.org/d/msgid/ct-policy/20250605091215.498fda01de1a633cee50e3f2%40andrewayer.name.

Rob Stradling

unread,

Jun 5, 2025, 5:47:57 PM6/5/25

to Andrew Ayer, Martijn Katerbarg, ct-p...@chromium.org

Thanks for following up, Andrew.

We're now planning to shutdown Sabre (again!) at 2025-06-12 14:00 UTC.

> (The certificates are all recently-issued and from Let's Encrypt; I guess they're still trying to submit final certificates here?)

Entries 546099878, 546099879, and 546099880 were submitted from one IP address (apparently in California), with User-Agent "Java/17.0.15".
The other 6 entries (546099881..546099886) were submitted from a different IP address (apparently in Tamil Nadu), with User-Agent "Java/11.0.25".

It's currently still a mystery how these 9 add-chain requests were able to reach Sabre's CTFE pods. The block on POST requests was implemented in our Nginx front-end load balancers. As part of shutting down Sabre yesterday, this POST block and the sabre.ct.comodo.com site were removed in a single update to the Nginx configuration; so in theory it should not have been possible for those 9 add-chain requests to be routed through to the CTFE pods.

When we next shutdown Sabre, I've asked our team to make sure we shutdown the CTFE pods first, before updating the Nginx configuration. This should prevent the problem from recurring, even if we can't explain why the problem occurred.

From: ct-p...@chromium.org <ct-p...@chromium.org> on behalf of Andrew Ayer <ag...@andrewayer.name>
Sent: 05 June 2025 17:12
To: Martijn Katerbarg <martijn....@sectigo.com>
Cc: ct-p...@chromium.org <ct-p...@chromium.org>
Subject: Re: [ct-policy] Re: Sectigo Sabre Has Grown

. To view this discussion visit
> https://urldefense.com/v3/__https://groups.google.com/a/chromium.org/d/msgid/ct-policy/SA1PR17MB65035245BEACD54365014677E36FA*40SA1PR17MB6503.namprd17.prod.outlook.com__;JQ!!J5K_pWsD!04OqNuNHBYPy4PTUSuvvvFFBF5yK0_NKGT45HaKbV97iGT4JPKMkeWbmxWHd-5ROYQt4Unn7m_isrJo$.

-- 
You received this message because you are subscribed to the Google Groups "Certificate Transparency Policy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ct-policy+...@chromium.org.
To view this discussion visit https://urldefense.com/v3/__https://groups.google.com/a/chromium.org/d/msgid/ct-policy/20250605091215.498fda01de1a633cee50e3f2*40andrewayer.name__;JQ!!J5K_pWsD!04OqNuNHBYPy4PTUSuvvvFFBF5yK0_NKGT45HaKbV97iGT4JPKMkeWbmxWHd-5ROYQt4Unn7vorGlu4$.

Rob Stradling

unread,

Jun 12, 2025, 1:39:34 PM6/12/25

to Certificate Transparency Policy, Rob Stradling, ct-p...@chromium.org, Andrew Ayer, Martijn Katerbarg

Our sabre.ct.comodo.com log has been shut down once again.

Reply all

Reply to author

Forward