Sunsetting Tuscolo2025h2

[Filippo Valsorda]

Hello carbon-based lifeforms,

It's allegedly 2026, so we are all busy archiving our 2025h2 shards.

To make the process as boring as possible, I am updating Sunlight to automatically turn read-only a week after the NotAfter limit. You can review the PR at https://github.com/FiloSottile/sunlight/pull/53. It's already running on Navigli.

We will rollout this change and sunset Tuscolo2025h2 in the following stages:

1. Tuesday 2026-01-13, 1300–1400Z — roll out #53 to Tuscolo
   Tuscolo2025h2 will become automatically read-only, returning 410s to add-[pre-]chain
2. Tuesday/Wednesday 2026-01-13/2026-01-14 — follow the "Archiving a log shard" public playbook
   Tuscolo2025h2 will become available on the Internet Archive and on ct-archive
3. Monday 2026-01-16, 1600–1800Z — follow the "Decommissioning a log shard" public playbook
   Tuscolo2025h2 will become unavailable
   

Let us know if you have any feedback on the proposed change or plan.

Cheers,

Filippo

[Filippo Valsorda]

2026-01-12 23:40 GMT+01:00 Filippo Valsorda <fil...@ml.filippo.io>:

Monday 2026-01-16, 1600–1800Z — follow the "Decommissioning a log shard" public playbook

Monday 2026-01-19, 1600–1800Z.

sigh

[Filippo Valsorda]

2026-01-12 23:40 GMT+01:00 Filippo Valsorda <fil...@ml.filippo.io>:

1. Tuesday 2026-01-13, 1300–1400Z — roll out #53 to Tuscolo
   Tuscolo2025h2 will become automatically read-only, returning 410s to add-[pre-]chain
   

This step was completed uneventfully. Tuscolo2025h2 is now read-only, and the final checkpoint is listed at

https://tuscolo2025h2.sunlight.geomys.org/log.v3.json

[Filippo Valsorda]

Hello everyone,

Unfortunately, the Internet Archive has a (very reasonable) 1100GB limit on item size, which Tuscolo2025h2 exceeds. I don't want to circumvent it by splitting the archive over multiple items, both for convenience and to respect the spirit of the limit.

Instead, we created a torrent file and will be seeding it for the foreseeable future.

https://raw.githubusercontent.com/geomys/ct-archive/refs/heads/main/torrents/ct_geomys_tuscolo2025h2.torrent

If anyone has suggestions for long-term homes for larger log archives, we'd love to hear them!

Cheers,

Filippo

--

You received this message because you are subscribed to the Google Groups "Certificate Transparency Policy" group.

To unsubscribe from this group and stop receiving emails from it, send an email to ct-policy+...@chromium.org.

To view this discussion visit https://groups.google.com/a/chromium.org/d/msgid/ct-policy/16f56aec-0e48-4077-bae6-ba2ac104fac3%40app.fastmail.com.

[Filippo Valsorda]

2026-01-12 23:40 GMT+01:00 Filippo Valsorda <fil...@ml.filippo.io>:

Monday 2026-01-16, 1600–1800Z — follow the "Decommissioning a log shard" public playbook

Tuscolo2025h2 will become unavailable

This step was completed uneventfully. Tuscolo2025h2 is now unavailable 🫡

Tuscolo2025h2 completes its run with 100% uptime as measured by Google.

https://tuscolo2025h2.sunlight.geomys.org/,add-chain,100.0000

https://tuscolo2025h2.sunlight.geomys.org/,add-pre-chain,100.0000

https://tuscolo2025h2.sunlight.geomys.org/,checkpoint,100.0000

https://tuscolo2025h2.sunlight.geomys.org/,get-roots,100.0000

https://tuscolo2025h2.sunlight.geomys.org/,tile,100.0000

The log's archive is available as a torrent.

In related news, ct-archive now includes phosphorescent-seeder, the simple tool being used to seed the read-only archive, and instructions on how to generate and seed a torrent, in case other operators need to archive logs too large for the Internet Archive.

[Andrew Ayer]

On Mon, 19 Jan 2026 18:15:01 +0100
"Filippo Valsorda" <fil...@ml.filippo.io> wrote:

> This step was completed uneventfully. Tuscolo2025h2 is now
> unavailable

This log is still Usable in Chrome, which does not enforce that a certificate's expiration lies within the log's expiry range. The log needs to be transitioned to Retired ASAP.

Regards,
Andrew

[Andrew Ayer]

Or Rejected.

[Filippo Valsorda]

The timeline mismatch is a bit unfortunate, but what is the difference to security between a frozen log that’s not producing new STHs (like Tuscolo2025h2 hasn’t since last week) and an unavailable log?

In both cases the log operator is asserting no new entries are being added to the log beyond those included in the final STH.

I don’t think today’s change requires any expedited change by Chrome.

(If that’s wrong, we can bring this log back temporarily, in case that helps.)

-- 

You received this message because you are subscribed to the Google Groups "Certificate Transparency Policy" group.

To unsubscribe from this group and stop receiving emails from it, send an email to ct-policy+...@chromium.org.

To view this discussion visit https://groups.google.com/a/chromium.org/d/msgid/ct-policy/20260119103338.0cb19cfb339fb66928c2a1b1%40andrewayer.name.

[Andrew Ayer]

On Mon, 19 Jan 2026 19:52:07 +0100
"Filippo Valsorda" <fil...@ml.filippo.io> wrote:

> The timeline mismatch is a bit unfortunate, but what is the
> difference to security between a frozen log that's not producing new
> STHs (like Tuscolo2025h2 hasn't since last week) and an unavailable
> log?
>
> In both cases the log operator is asserting no new entries are being
> added to the log beyond those included in the final STH.

The difference is that a new monitor can't confirm that the final
STH doesn't contain unexpired certificates for domains that they're
interested in. (Unless they use BitTorrent to download the archive,
which I don't think is reasonable.)

In this case, the final STH doesn't contain any unexpired certificates,
so the unavailability can't actually lead to monitors missing
certificates. However, it will cause alarms in monitors whose policy
is to try to monitor any Qualified, Usable, or ReadOnly log, which will
blunt the security value of those alarms, which are meant to signal the
possibility that the monitor is missing certificates.

More broadly, this isn't the first time that a log was shut down before
it was Retired, and at some point the lack of coordination could lead to
trusted certificates being unavailable to monitors, depending on the
set of Retired logs at the time. Also, log operators have sometimes
mishandled shutdowns (e.g. [1] [2]) which caused issues that could have
been avoided if the logs were Retired on schedule.

> I don’t think today’s change requires any expedited change by Chrome.

I apologize for making this seem like an emergency, especially on a US
holiday; it certainly doesn't require any action today. I still think
it's pretty important for logs to be transitioned to Retired/Rejected
on or before their shutdown date. To that end, I suggest that Let's
Encrypt's logs be scheduled for retirement ahead of their February 28,
2026 shutdown.

> (If that’s wrong, we can bring this log back temporarily, in case that helps.)

If it's not too much trouble, it would help with not making monitors
alarm.

Regards,
Andrew

[1] https://groups.google.com/a/chromium.org/g/ct-policy/c/P5aj4JEBFPM/m/9AEcvY01EQAJ
[2] https://groups.google.com/a/chromium.org/g/ct-policy/c/W0GAdMRwZuA

[Filippo Valsorda]

We have brought the read path of Tuscolo2025h2 back online to help prevent alert fatigue. (However, I am still confused as to how read-only mode didn't trigger the alerts: an STH older than the MMD is an RFC 6962 violation as much as being offline. If the answer is that expired shards are special-cased to allow them to be stale, why not also allow them to be offline? In general, I still feel like read-only and offline are equivalent, except for the ability of new monitors to verify the absence of unexpired certificates, which however can be collectively verified by all existing monitors.)

We will not be bringing the write path back, because although a monitor might arguably be monitoring the get-roots endpoint, that doesn't justify the risk of removing the readonly seal from the ZFS dataset, or of running Sunlight in an untested configuration.

We're not in a rush to turn the log down (we have plenty of space), but to avoid similar issues and to plan maintenance work in the future it would indeed be very useful if Chrome published expected timelines for when logs can be expected to be transitioned to Retired/Rejected.

[Pim van Pelt]

Hoi,

Andrew is asking for Chromium to list the log as Retired (seems more appropriate than Rejected) before the read-path is removed, which sounds reasonable to me.
I'll take this thread as having pushed a big-red-button for IPng and I will not continue with Gouda and Halloumi decommission as per my plan in [1].

How shall we coordinate this ? Proposed timing for static logs:

1) +14d after write window is closed, remove write path
2) +14d make photocopy and archive the log on S3 or IA
3) (anywhere between +0 and +28d) transition the log to Retired
4) +28d but strictly after (3) remove read path
5) +45d delete / repurpose ZFS storage

Is it reasonable to expect the transition to happen within 28d of the write window closing?

I will pause the Gouda/Halloumi turndown process (they are at step 3 now).

groet,
Pim

[1] https://groups.google.com/a/chromium.org/d/msgid/ct-policy/a38833f3ec76a5ce12b446261d22a9f0%40ipng.ch

--
You received this message because you are subscribed to the Google Groups "Certificate Transparency Policy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ct-policy+...@chromium.org.

To view this discussion visit https://groups.google.com/a/chromium.org/d/msgid/ct-policy/07b268f9-88c7-40d2-b691-f5becde2be5e%40app.fastmail.com.

-- 
Pim van Pelt <p...@ipng.ch>
PBVP1-RIPE https://ipng.ch/

[Andrew Ayer]

Thank you for bringing back the read path!

You're right that not transitioning a log to ReadOnly when it stops issuing STHs is also a potential cause of monitor alert fatigue. Ideally, that would be avoided as well. I didn't notice because Cert Spotter doesn't raise an alarm for stale STHs. I guess since no one complained, there aren't any extant monitors checking STH freshness? (Graham Edgecombe's monitor did, but it's not running anymore.)

(My reason for alerting on one but not the other is more pragmatic than principled: a stale STH most likely means the log is truly not growing (if the log is malicious, it'll just bump the timestamp and only gossip can help) whereas a retrieval failure could indicate a non-malicious condition such as a network problem between the monitor and log. Arguably, if monitors had reliable gossip with UAs, they wouldn't need to raise an alarm on either STH retrieval failure or staleness, since gossip would tell them about larger trees. Unfortunately, Google's STH feed is not reliable enough for this.)

Regards,
Andrew

On Tue, 20 Jan 2026 00:46:29 +0100
"Filippo Valsorda" <fil...@ml.filippo.io> wrote:

> We have brought the read path of Tuscolo2025h2 back online to help
> prevent alert fatigue. (However, I am still confused as to how
> read-only mode didn't trigger the alerts: an STH older than the MMD
> is an RFC 6962 violation as much as being offline. If the answer is
> that expired shards are special-cased to allow them to be stale, why
> not also allow them to be offline? In general, I still feel like

> read-only and offline are equivalent, except for the ability of *new*

> > > I don___t think today___s change requires any expedited change by

> > > Chrome.
> >
> > I apologize for making this seem like an emergency, especially on a
> > US holiday; it certainly doesn't require any action today. I still
> > think it's pretty important for logs to be transitioned to
> > Retired/Rejected on or before their shutdown date. To that end, I
> > suggest that Let's Encrypt's logs be scheduled for retirement ahead
> > of their February 28, 2026 shutdown.
> >

> > > (If that___s wrong, we can bring this log back temporarily, in case

> > > that helps.)
> >
> > If it's not too much trouble, it would help with not making monitors
> > alarm.
> >
> > Regards,
> > Andrew
> >
> > [1]
> > https://groups.google.com/a/chromium.org/g/ct-policy/c/P5aj4JEBFPM/m/9AEcvY01EQAJ
> > [2]
> > https://groups.google.com/a/chromium.org/g/ct-policy/c/W0GAdMRwZuA
> >
>

> --
> You received this message because you are subscribed to the Google
> Groups "Certificate Transparency Policy" group. To unsubscribe from
> this group and stop receiving emails from it, send an email to
> ct-policy+...@chromium.org. To view this discussion visit

> https://groups.google.com/a/chromium.org/d/msgid/ct-policy/07b268f9-88c7-40d2-b691-f5becde2be5e%40app.fastmail.com.

[Filippo Valsorda]

While we don't mind delaying taking the log offline, we are pretty happy with the automatic read-only transition recently added to Sunlight, and wouldn't want to make that a manual process that needs to be coordinated with the UAs.

Maybe the medium-term solution is for UAs to schedule state transitions based on the planned state change metadata proposed by Apple some time ago.

[Joe DeBlasio]

Hi folks,

Apologies for doing a bad job of communicating our thoughts and expectations.

Log transitions are currently a manual process for us. As a result, we try to batch changes when we can, and in particular we try to migrate all shards whose expiry ranges have passed to Rejected at once. As some log operators extend their temporal intervals a few weeks beyond the strict calendar boundary that the shard names imply, we typically wait at least 20 days into the year / half year. In this case, we expect to transition all of the 2025h2 shards to Rejected in the next two weeks.

Until we make that transition, the logs are listed in our log list as Usable, and are thus still subject to the expectations of Chrome's CT log policy (https://goo.gle/chrome-ct-log-policy). We should have proactively communicated this, but that means that it's not OK with the policy to turn down a log while it is still listed as Usable. That also means that logs should also not stop accepting valid submissions while Usable, either (even though this is something I was explicitly excited about in slack). I definitely think we should find a way to simplify and ease this situation, but "coordinat[ing] with the UAs" is the policy's current expectation for included operators.

On the Chrome side, I'm optimistic that in the next few weeks we'll have automated state transitions after a log's expiry window passes. That will significantly reduce the lag where logs are considered Usable but whose windows have closed. I personally do not want the logs to transition to Rejected immediately when the expiry window closes, however, as I want to ensure that our policy sets the expectation that logs give monitors to fully ingest the log before the log turns down. If folks have ideas for what the optimal schedule would be, let's try to align on it here, and we can ensure that our policies and practices align with that schedule.

Apropos of the per-operator metadata proposal: we will be picking this proposal up again soon. I'm not entirely convinced that the additional complexity of supporting planned state changes is worth it, but I'm open to it, and encourage folks to leave comments on Clint's Google Doc. 

Joe, on behalf of the Chrome CT team

To view this discussion visit https://groups.google.com/a/chromium.org/d/msgid/ct-policy/13852bda-1374-476d-9300-48f4dce0057e%40app.fastmail.com.

[Filippo Valsorda]

Hi Joe,

Thank you for clarifying the expectations of the Chrome policy. The miscommunication is unfortunate, but I still think reverting the read-only seal on Tuscolo2025h2 and doing an untested transition back from read-only to operational is unnecessary risk for the ecosystem. What do you think?

As for future transitions, I don’t have a strong opinion on the actual length of the timelines, and I’m happy to align the Sunlight timelines with Chrome’s preferences. (I believe that monitors being more than a few hours behind is actually a severe issue, and I hope Tuscolo never caused that to happen.)

As a log operator I’d like to ask that Chrome publish a fixed duration past the end of the temporal period after which it is safe to stop accepting submissions and producing fresh STHs, so it can be safely automated.

Cheers,

Filippo

[Andrew Ayer]

On Thu, 22 Jan 2026 12:53:48 +0100
"Filippo Valsorda" <fil...@ml.filippo.io> wrote:

> As a log operator I’d like to ask that Chrome publish a fixed
> duration past the end of the temporal period after which it is safe
> to stop accepting submissions and producing fresh STHs, so it can be
> safely automated.

What's your motivation for the automatic transition to read-only? Would rejecting expired certificates (allowed by both Chrome and Apple log policy) satisfy it? You'd still have to sign fresh STHs, but at least the logs would never grow after the end of their temporal period.

Regards,
Andrew

[Filippo Valsorda]

It's operational risk mitigation: every time I edit the config file, I risk causing downtime. If Sunlight automatically and safely turns itself down, I can then simply log in, make the ZFS dataset read-only, and start the archival process.

It's a good point about rejecting expired certificates being allowed, but then what is the benefit to the ecosystem of waiting to transition to ReadOnly after the end of the temporal period?

[Andrew Ayer]

On Thu, 22 Jan 2026 17:18:34 +0100

I get the value of automating config changes. I'm not clear on the benefit of going read-only (as in not accepting new entries or signing new STHs) or ReadOnly (as published in a log list) as an intermediate stage before the log becomes Rejected.

Let's say Chrome commits to automatically transitioning logs to Rejected $N days after their temporal period ends. A Sunlight log could automatically turn itself down $N days (or maybe $N + 1 to be safe) after its temporal period ends. Would that work for you?

Regards,
Andrew

[Joe DeBlasio]

> I still think reverting the read-only seal on Tuscolo2025h2 and doing an untested transition back from read-only to operational is unnecessary risk for the ecosystem. What do you think?

I completely agree. I'm not particularly worried about Tuscolo2025h2 -- I mostly want to ensure we all agree on the right steps moving forward.

> As a log operator I’d like to ask that Chrome publish a fixed duration past the end of the temporal period after which it is safe to stop accepting submissions and producing fresh STHs, so it can be safely automated.

That's very reasonable. Literally just yesterday I was reviewing code that will eventually transition logs from Usable to Rejected a week after the end of their expiry windows. I don't think we feel it's necessary to have an explicit period in Chrome's ReadOnly state, though of course logs could choose to stay up in whatever form they'd like after the log moves to Rejected.

I'll follow up here when that automation is solidified, and we can then update our policy to be more explicit about this timeline.

Joe

--

You received this message because you are subscribed to the Google Groups "Certificate Transparency Policy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ct-policy+...@chromium.org.

To view this discussion visit https://groups.google.com/a/chromium.org/d/msgid/ct-policy/31b61629-9f43-430c-a0f9-b6dc0edfafc5%40app.fastmail.com.

[Filippo Valsorda]

Yes to both, Chrome transitioning to Rejected directly is perfectly fine for us as well, and I agree there is no particular value to a ReadOnly stage on the Chrome side.

Sunlight transitions to read-only automatically instead of going offline automatically because it felt a bit safer and essentially 90% of the benefit: once the log is read-only we can freeze the ZFS dataset, start the archival, even remove the key. It'd be maybe a bit annoying for some clients if the log went offline automatically at $N + 1 and the archive only materialized later on.

[Kurt Roeckx]

Hi,

Some of the monitors might need some time to catch up. I'm not sure what a reasonable time is between going read only and offline.

I'm still busy with some of the 2025 logs. But I think I'm going to stop that part, since I don't have the resources to keep up with it anymore.

Kurt

[Filippo Valsorda]

Hi Kurt,

That's unfortunate, and even if the issues might not be fixable right away, I would like to learn more about what causes them, and what we might do to help. What is it that causes the backlog to accumulate? What resources are you constrained on? Also, what 2025 logs are you behind on?

Cheers,

Filippo