oak2022 entries 135227797 and 135923917

[Bas Westerbaan]

Hi all,

We noticed two entries in Oak2022 which at the time of merger had timestamps a few weeks in the future.

Oak2022's entry 135227797 has timestamp 1639042196214 (2021-12-09 GMT), but we recorded a STH with tree_size 135228431 above it with timestamp 1637728775590 (2021-11-24 GMT) [1].

Similarly, entry 135923917 has timestamp 1639116187750 (2021-12-10 GMT), but we recorded a STH with tree_size 135924849 above it with timestamp 1637752211893 (2021-11-24 GMT) [2].

Best,

 Bas

[1] {"tree_size":135228431,"timestamp":1637728775590,"sha256_root_hash":"5CxC06vRe9dI3EYPKD5VanXSMtDBbW042PQpXquiElw=","tree_head_signature":"BAMARzBFAiEAr7zgzDmYlv+B43dh+qO70CRdrPC0BJcggey+MHcOxHUCIC3bVNU/irr7CUG7ic6BpULELX7QWe9RNFJdaap2HZJp"}

[2] {"tree_size":135924849,"timestamp":1637752211893,"sha256_root_hash":"0mhw/MA13V3GCL1xpOE7AWEvigu1wSrACJYiUGXHYNM=","tree_head_signature":"BAMARzBFAiEAks6qHpLmU0RC/WyIEJV0uFkLOIniS1wo+qnqxjIxzKoCIGZELoH3FUbYykXoiCzMkalGYUXUoPVpJ4qC0LtusUep"}

[Aaron Gable]

Hi Bas,

Thanks for the notification. We've confirmed the finding and are investigating. We'll provide an update on our findings so far in the next 24 hours.

Thanks,

Aaron

[Aaron Gable]

Based on our investigation so far, we're confident both that a) this discrepancy is additional fallout from the December 9th incident, not from normal operations; and b) we have the data to reconstruct the incorrect entries without invalidating any of the cryptographic properties of the log.

We intend to provide a full incident report with root cause timeline and remediation items by the end of the week.

Thanks again,

Aaron

[Aaron Gable]

We have shared our incident report in a standalone thread for easier searching.

Thanks,

Aaron

[Bas Westerbaan]

Hi Aaron,

Thank you, this is great. I'm impressed by the speed and thoroughness of your response.

Best,

 Bas

[Devon O'Brien]

I'd like to add to Bas's sentiment and thank you for such a thorough and well-written post-mortem about this incident. Reports of this quality provide new and existing log operators useful insights into avoiding this situation in the future, and also enable improvements to CT log implementations to avoid future failure modes. The Chrome CT team will be discussing these proposed improvements to Trillian + CTFE with the Google CT log team, which maintains these software packages. Additionally, we will look into possible improvements to our CT log compliance monitoring solution to see if we can ensure more rapid detection of issues such as this across the CT log ecosystem.

Many thanks,

Devon