MMD violations in mammoth2025h2

Joe DeBlasio

unread,

Aug 25, 2025, 1:20:54 PM (13 days ago) Aug 25

to #CTOps, Certificate Transparency Policy

Hi Sectigo CTOps,

(CCing ct-policy@ for awareness.)

We've seen a very large number of SCTs not included within Mammoth2025h2's MMD this morning. I've included one cert below with an example embedded SCT that hasn't been included as of sending this email.

We'd appreciate it if you could temporarily suspend submissions to mammoth2025h2 ASAP to limit the potential impact, and then investigate and report back.

Thank you,

Joe on behalf of the Chrome CT Team

----

Rob Stradling

unread,

Aug 26, 2025, 2:07:55 PM (12 days ago) Aug 26

to Joe DeBlasio, #CTOps, Certificate Transparency Policy

Hi Joe. We temporarily suspended POST requests to Mammoth2025h2 at 10:42 UTC today. We will re-enable once the sequencing backlog has gone.

There's not really anything new that we can investigate here. The poor sequencing performance of our MariaDB-based logs is well known and has bitten us several times before.

Per https://groups.google.com/a/chromium.org/g/ct-policy/c/-N1YTYNeVl4/m/aqcgMUoJGwAJ, we intend to permanently suspend POST requests to all of the Sabre and Mammoth shards at approximately 2025-09-18 15:00 UTC. We are looking forward to this day!

From: Joe DeBlasio <jdeb...@chromium.org>
Sent: 25 August 2025 18:20
To: #CTOps <ct...@sectigo.com>; Certificate Transparency Policy <ct-p...@chromium.org>
Subject: MMD violations in mammoth2025h2

This Message Is From an External Sender

This message came from outside your organization.

Report Suspicious

Joe DeBlasio

unread,

Aug 27, 2025, 2:03:53 PM (11 days ago) Aug 27

to Rob Stradling, #CTOps, Certificate Transparency Policy

Thank you, Rob.

I appreciate that our (Chrome's) thoughts have been shifting a bit here as log behavior has continued to evolve.

I think it would be reasonable to never re-enable submissions to mammoth2025h2, instead transitioning it to readonly now. Other mammoth and sabre shards could move to readonly at the previously-discussed date. The prior plan (everything waiting until September) was an attempt to maximize capacity in the ecosystem, but if mammoth2025h2 can't safely offer that capacity, there's no benefit in trying to postpone the transition for that shard. That also does not detract from the value of the other shards waiting until Tiger* are fully Usable.

Even this plan presupposes that the log is able to fully clear its backlog. I notice that despite having more than a full day of no submissions, mammoth2025h2 still has unincluded certificates -- I've attached one certificate with an SCT from 6:47 UTC yesterday that I can't validate as of this writing. If the log hasn't cleared the backlog in the next day or two, we'll instead need to move to Retire the log.

Joe

2025-08-27-mammoth2025h2-inclusion-failure.pem

Rob Stradling

unread,

Aug 28, 2025, 7:24:47 AM (11 days ago) Aug 28

to Certificate Transparency Policy, Joe DeBlasio, #CTOps, Certificate Transparency Policy, Rob Stradling

Hi Joe. Mammoth2025h2's submission backlog cleared at approximately 23:00 UTC yesterday.

We won't re-enable submissions to this shard. Per your suggestion, please transition it to Readonly now.

Reply all

Reply to author

Forward