Gouda maintenance Weds Aug 12th
109 views
Skip to first unread message
Pim van Pelt
Aug 10, 2026, 3:24:02 AM (3 days ago) Aug 10
to Ct Policy
Hoi folks,
TL/DR: Gouda will be unavailable on 2026-08-12 between 0800-0900Z
IPng has deployed a new machine for our Sunlight logs. They are
currently running on a VM with passthrough ZFS storage, but we're
intending to run them on bare metal going forward. Now that we've
submitted our 2028 log shards, this is a good time to make the move.
Yesterday I moved the staging logs (Rennet 2027h1+h2 and 2028h1+h2) to
this new machine. It took about 30min, mostly due to steps 4) and 5)
giving me a learning moment.
On Wednesday 2026-08-12 from 0800-0900Z I will move the Gouda production
log shards for 2027+2028 to this new machine also. The procedure to
safely move logs between Sunlight instances that I landed on is roughly:
0) rsync the data to new machine
1) remove the shard(s) from sunlight on original VM; keep them serving
in Skylight for step 5
2) freeze ZFS dataset(s); extract checkpoints from sqlite3 (see below)
3) rsync the delta to new machine
4) add the shard(s) to sunlight on the new machine; insert the snapshots
into sqlite3
5) start skylight + sunlight on the new machine; sunlight will read
skylight checkpoint(s) from original VM via monitoringprefix in addition
to its localdirectory
6) move nginx frontends for the shard to point to the new machine
7) remove the shard from skylight on original VM
At the end of this window, Gouda and Rennet shard 2026h2 will remain on
the original VM; and all eight 2027+2028 shards will be on the new machine.
For future readers: watch out with Sunlight's shared checkpoints.db; it
contains blob of logID -> checkpoint so copying logs between two running
Sunlight machines cannot simply rsync the checkpoints.db; instead,
extract the to-be-moved logID->checkpoints from the source and upsert
them to the destination.
I will be using this on the original VM (after stopping Sunlight and
setting ZFS datasets readonly):
sqlite3 -batch -noheader checkpoints.db "SELECT 'INSERT INTO
checkpoints(logID,body) VALUES(' || quote(logID) || ',' || quote(body)
|| ') ON CONFLICT(logID) DO UPDATE SET body=excluded.body;' FROM
checkpoints WHERE substr(body,1,9) IN (CAST('gouda2027' AS BLOB),
CAST('gouda2028' AS BLOB));" > /tmp/upsert-gouda.sql
grep -c '^INSERT INTO' /tmp/upsert-gouda.sql # expect 4
And then on the new machine (which is already running Rennet shards):
sqlite3 /ssd-vol0/shared/checkpoints.db < /tmp/upsert-gouda.sql
groet,
Pim obo ct-...@ipng.ch
--
Pim van Pelt
PBVP1-RIPE - https://ipng.ch/
TL/DR: Gouda will be unavailable on 2026-08-12 between 0800-0900Z
IPng has deployed a new machine for our Sunlight logs. They are
currently running on a VM with passthrough ZFS storage, but we're
intending to run them on bare metal going forward. Now that we've
submitted our 2028 log shards, this is a good time to make the move.
Yesterday I moved the staging logs (Rennet 2027h1+h2 and 2028h1+h2) to
this new machine. It took about 30min, mostly due to steps 4) and 5)
giving me a learning moment.
On Wednesday 2026-08-12 from 0800-0900Z I will move the Gouda production
log shards for 2027+2028 to this new machine also. The procedure to
safely move logs between Sunlight instances that I landed on is roughly:
0) rsync the data to new machine
1) remove the shard(s) from sunlight on original VM; keep them serving
in Skylight for step 5
2) freeze ZFS dataset(s); extract checkpoints from sqlite3 (see below)
3) rsync the delta to new machine
4) add the shard(s) to sunlight on the new machine; insert the snapshots
into sqlite3
5) start skylight + sunlight on the new machine; sunlight will read
skylight checkpoint(s) from original VM via monitoringprefix in addition
to its localdirectory
6) move nginx frontends for the shard to point to the new machine
7) remove the shard from skylight on original VM
At the end of this window, Gouda and Rennet shard 2026h2 will remain on
the original VM; and all eight 2027+2028 shards will be on the new machine.
For future readers: watch out with Sunlight's shared checkpoints.db; it
contains blob of logID -> checkpoint so copying logs between two running
Sunlight machines cannot simply rsync the checkpoints.db; instead,
extract the to-be-moved logID->checkpoints from the source and upsert
them to the destination.
I will be using this on the original VM (after stopping Sunlight and
setting ZFS datasets readonly):
sqlite3 -batch -noheader checkpoints.db "SELECT 'INSERT INTO
checkpoints(logID,body) VALUES(' || quote(logID) || ',' || quote(body)
|| ') ON CONFLICT(logID) DO UPDATE SET body=excluded.body;' FROM
checkpoints WHERE substr(body,1,9) IN (CAST('gouda2027' AS BLOB),
CAST('gouda2028' AS BLOB));" > /tmp/upsert-gouda.sql
grep -c '^INSERT INTO' /tmp/upsert-gouda.sql # expect 4
And then on the new machine (which is already running Rennet shards):
sqlite3 /ssd-vol0/shared/checkpoints.db < /tmp/upsert-gouda.sql
groet,
Pim obo ct-...@ipng.ch
--
Pim van Pelt
PBVP1-RIPE - https://ipng.ch/
Andrew Ayer
Aug 10, 2026, 10:31:29 AM (2 days ago) Aug 10
to Pim van Pelt, 'Pim van Pelt' via Certificate Transparency Policy
Hi Pim,
Thanks for sharing your plan with the community.
I recommend against using upsert on the checkpoint DB. If a checkpoint for a log already exists in the new DB, it indicates something has gone awry and replacing it could lead to a split view. As Sunlight's README says, the checkpoint DB must never be changed or modified since it's Sunlight's safety mechanism for preventing split views. A plain, non-upserting, INSERT should be OK, provided the old Sunlight instance is read-only.
I've opened https://github.com/FiloSottile/sunlight/issues/72 with an idea for a migration tool so that operators don't have to manipulate the DB file directly.
Regards,
Andrew
On Mon, 10 Aug 2026 09:23:49 +0200
"'Pim van Pelt' via Certificate Transparency Policy"
> --
> You received this message because you are subscribed to the Google
> Groups "Certificate Transparency Policy" group. To unsubscribe from
> this group and stop receiving emails from it, send an email to
> ct-policy+...@chromium.org. To view this discussion visit
> https://groups.google.com/a/chromium.org/d/msgid/ct-policy/fa7b416c-566d-4268-8f84-104a23fdaab4%40ipng.ch.
Thanks for sharing your plan with the community.
I recommend against using upsert on the checkpoint DB. If a checkpoint for a log already exists in the new DB, it indicates something has gone awry and replacing it could lead to a split view. As Sunlight's README says, the checkpoint DB must never be changed or modified since it's Sunlight's safety mechanism for preventing split views. A plain, non-upserting, INSERT should be OK, provided the old Sunlight instance is read-only.
I've opened https://github.com/FiloSottile/sunlight/issues/72 with an idea for a migration tool so that operators don't have to manipulate the DB file directly.
Regards,
Andrew
On Mon, 10 Aug 2026 09:23:49 +0200
"'Pim van Pelt' via Certificate Transparency Policy"
> You received this message because you are subscribed to the Google
> Groups "Certificate Transparency Policy" group. To unsubscribe from
> this group and stop receiving emails from it, send an email to
> ct-policy+...@chromium.org. To view this discussion visit
> https://groups.google.com/a/chromium.org/d/msgid/ct-policy/fa7b416c-566d-4268-8f84-104a23fdaab4%40ipng.ch.
Pim van Pelt
5:50 AM (13 hours ago) 5:50 AM
to ct-p...@chromium.org
Hoi folks,
Andrew, thanks for the advice on UPSERTs and for filing #72. I agree with the advice that we should minimize changes to checkpoints, and I was certainly uncomfortable performing surgery on them. However, checkpoints on ctlog0 were shared between staging and production (this was probably a bad idea). When migrating Rennet staging shards, I did not INSERT but rather literally copied the checkpoints.db over, which allowed Rennet to start in my learning moment (#4 from the original post). But now, of course all checkpoints.db entries were on ctlog2, including gouda202[78]*, so I needed to UPSERT not INSERT them.
I thought about your comment a bit, and decided to split the monolithic checkpoints.db (from original VM) into checkpoints-staging.db (Rennet) and checkpoints.db (Gouda), which kind of made the whole INSERT/UPSERT moot, but the result is that we have now isolated the checkpoints sqlite3 database for IPng logs.
This
is now done, with pertinent timestamps (in
UTC):
Aug 12 08:04:24 ctlog0 systemd[1]: Stopping sunlight.service - Sunlight Certificate Transparency Log (prod)... Aug 12 08:04:25 ctlog0 sunlight[3715286]: {"time":"2026-08-12T08:04:25.209614131Z","level":"INFO","source":{"function":"filippo.io/sunlight/internal/ctlog.LoadLog","file":"/home/jeroen/go/pkg/mod/filippo.io/sunl...@v0.9.0/internal/ctlog/ctlog.go","line":326},"msg":"loaded log","log":"gouda2026h2","logID":"Goudaw/+v4G0eTnG0jEKhtbRAtTwRuIYLJ3jX14mJe8=","size":871034168,"timestamp":1786521864011} Aug 12 08:14:33 ctlog2 sunlight[2310047]: time=2026-08-12T08:14:33.848Z level=INFO msg="sequenced pool" log=gouda2027h1 old_tree_size=44820540 entries=8 start=2026-08-12T08:14:33.823Z tree_size=44820548 tiles=3 timestamp=1786522473823 elapsed=24.850573ms Aug 12 08:14:34 ctlog2 sunlight[2310047]: time=2026-08-12T08:14:34.842Z level=INFO msg="sequenced pool" log=gouda2027h1 old_tree_size=44820548 entries=1 start=2026-08-12T08:14:34.824Z tree_size=44820549 tiles=3 timestamp=1786522474824 elapsed=18.015412ms
08:04:24Z
restarted Sunlight on original VM, it
continued to serve 2026h2 one second later
... copied 2027h1/2027h2 and 2028h1/2028h2
from ctlog0 to ctlog2 ...
08:14:34Z ansible finished the frontend
rollout and the first writes for 2027h1 were
sequenced.
Andrew, thanks for the advice on UPSERTs and for filing #72. I agree with the advice that we should minimize changes to checkpoints, and I was certainly uncomfortable performing surgery on them. However, checkpoints on ctlog0 were shared between staging and production (this was probably a bad idea). When migrating Rennet staging shards, I did not INSERT but rather literally copied the checkpoints.db over, which allowed Rennet to start in my learning moment (#4 from the original post). But now, of course all checkpoints.db entries were on ctlog2, including gouda202[78]*, so I needed to UPSERT not INSERT them.
I thought about your comment a bit, and decided to split the monolithic checkpoints.db (from original VM) into checkpoints-staging.db (Rennet) and checkpoints.db (Gouda), which kind of made the whole INSERT/UPSERT moot, but the result is that we have now isolated the checkpoints sqlite3 database for IPng logs.
We
have a few surplus datasets now on ctlog0:
root@ctlog0:/ssd-vol0/enc/sunlight# zfs get all | grep readonly ssd-vol0 readonly off default ssd-vol0/enc readonly off default ssd-vol0/logs readonly off default ssd-vol0/logs/gouda2026h1 readonly on local ssd-vol0/logs/gouda2026h2 readonly off default ssd-vol0/logs/gouda2027h1 readonly on local ssd-vol0/logs/gouda2027h2 readonly on local ssd-vol0/logs/gouda2028h1 readonly on local ssd-vol0/logs/gouda2028h2 readonly on local ssd-vol0/logs/rennet2026h1 readonly on local ssd-vol0/logs/rennet2026h2 readonly off default ssd-vol0/logs/rennet2027h1 readonly on local ssd-vol0/logs/rennet2027h2 readonly on local ssd-vol0/logs/rennet2028h1 readonly on local ssd-vol0/logs/rennet2028h2 readonly on local ssd-vol0/shared readonly off default
I
propose to silently destroy those datasets
from ctlog0 by midcycle (Oct). In the mean
time, let us know here or on ct-...@ipng.ch
if you see anything out of the ordinary with
any of the logshards.
groet,
Pim
groet,
Pim
Filippo Valsorda
6:09 AM (12 hours ago) 6:09 AM
to Certificate Transparency Policy
2026-08-12 11:50 GMT+02:00 'Pim van Pelt' via Certificate Transparency Policy <ct-p...@chromium.org>:
However, checkpoints on ctlog0 were shared between staging and production (this was probably a bad idea).
FWIW, this was not a bad idea, and it's how Sunlight is designed to be operated and how Geomys operates it.
The checkpoints db is tiny, but it's the ultimate backstop against catastrophic log forks, for example if a prod key is erroneously configured in a staging log. If the database is not shared, it can't help.
Reply all
Reply to author
Forward
0 new messages