Mammoth2025h1 log entry 1021714 later timestamp than next STH

[Luke Valenta]

Hi folks,

We noticed an entry in the mammoth2025h1 log which had a timestamp ahead of the bounding tree head. We only picked this up since we happened to poll the latest STH at exactly the right time to catch the discrepancy. We reported to the Sectigo team earlier this week and they're starting an investigation.

Mammoth2025h1’s entry 1021714 has timestamp 1717928700088 (2024-06-09 10:25:00.088 GMT), but we recorded a STH with tree_size 1021715 above it with timestamp 1717928699927 (2024-06-09 10:24:59.927 GMT) [1], 161 milliseconds prior to the entry’s timestamp.

[1]
{"tree_size":1021715,"timestamp":1717928699927,"sha256_root_hash":"Wa3eDh/g7KyE4CdzEJzw6RglnFGwBQH54OUupZTRb7M=","tree_head_signature":"BAMARzBFAiEAjg+6uNUUSfmbg9nQDc5XX4J36EJfzF4nUdm0dDxHBx4CIFTUCEt+atRig9pOSy40\nq8hWUj5u1ppwnZG+gnIOPxE1"}

Best,
Luke

--

Luke Valenta

Systems Engineer - Research

[Rob Stradling]

Thanks Luke for posting this report.

RFC6962 section 3.5 says that an STH's "timestamp MUST be at least as recent as the most recent SCT timestamp in the tree", and it's clear that the observed behaviour from mammoth2025h1 did not meet that requirement.  The only plausible explanation that we can come up with is clock skew on one or more of the nodes that produces SCTs / STHs, although sadly we no longer have logs from 2024-06-09 from which to prove or disprove that hypothesis.

We sought advice from the Trillian / CTFE (CT Front End) engineers on the transparency-dev Slack (public invitation here).  Quoting Al Cutter from that conversation:

'Yeah, clock skew seems likely.

Unfortunately this is very tricky to check for - the CTFE chooses the timestamp, and that’s opaque by the time it gets to Trillian (it’s simply bytes in the Merkle Leaf at that point), so if the block on the server running CTFE is far enough ahead of the one on the machine running the log_server then this could potentially happen.

Trillian’s log_signer binary does have a --sequencer_guard_window flag which is intended to help with this; it more or less says “entries in the queue must have been there for at least this long before they’re eligible for integration into the tree”.'

We reviewed our log_signer configuration and found that we were already setting sequencer_guard_window=1s.  In the hope of avoiding any future occurrence of this type of problem, we intend to increase that to sequencer_guard_window=10s across all of our logs.

---

From: 'Luke Valenta' via Certificate Transparency Policy <ct-p...@chromium.org>
Sent: 14 June 2024 14:14
To: ct-p...@chromium.org <ct-p...@chromium.org>
Cc: #CTOps <ct...@sectigo.com>; Carlos Rodrigues <crodr...@cloudflare.com>; Bas Westerbaan <b...@cloudflare.com>
Subject: [ct-policy] Mammoth2025h1 log entry 1021714 later timestamp than next STH

 

CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.

--
You received this message because you are subscribed to the Google Groups "Certificate Transparency Policy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ct-policy+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/ct-policy/CAAUDTJhH%3Duu%2B7wyL0Xji_JOwR__VoC%3DtSb5ZQmTVX%3DVUi%3DEpUw%40mail.gmail.com.