Yeti 2022 not furnishing entries for STH 65569149

[Andrew Ayer]

Yeti 2022 has produced the following STHs:

{
"tree_size": 65551225,
"timestamp": 1624968008035,
"sha256_root_hash": "QLoUSs76wgHT7RyyJDdJPbPokyQnNwsVBOMq/gIuEMk=",
"tree_head_signature": "BAMASDBGAiEAgpQ9zzpRo8NZHfw/lXk0g9YvvaNtALinJoaqxK+tUosCIQDYsuAJbqiTv5/CZechrVOjk3C1SxURGlNibEKW5iQ0zA=="
}

{
"tree_size": 65569149,
"timestamp": 1624971609346,
"sha256_root_hash": "ogxKC1kdkMoDxTYnEkdHVt/UotIeRpip7x6QljoOGuY=",
"tree_head_signature": "BAMASDBGAiEA7gQwSrLcn6JzY9USfUyQjzdifF6ojGztYaCvcYWZFLcCIQDMlxZitnji0l5mcclFCS0C6FpFEITWOqJYEJCnBB6rIg=="
}

Although Yeti 2022 can produce a valid consistency proof between these
two STHs, the entries in the range [65551225, 65569149) returned from
get-entries produce the root hash
KTFRWeiy7n+HvsK6lZ7AM1RInOUeYHBhFHMJRO5iGcs= when they are appended to
the tree with root hash QLoUSs76wgHT7RyyJDdJPbPokyQnNwsVBOMq/gIuEMk=.
I've attached a list of the leaf hashes of the log entries in
this range which Yeti 2022 furnished to my monitor.

A similar problem previously occurred with Nimbus 2018:
https://bugs.chromium.org/p/chromium/issues/detail?id=780654#c10

Regards,
Andrew

[Devon O'Brien]

Andrew, thank you as always for your timely and detailed posts to ct-policy@ regarding observed issues with CT Logs. We can confirm that we've observed unexpected behavior from Yeti2022 today as well and are continuing to investigate the scope of this issue.

DigiCert CT Ops -- Can you please look into this matter and report your findings and fix/timeline on this thread?

-Devon

[Andrew Ayer]

The problem seems to be with entry 65562066.

If you download this entry with get-entries
(https://yeti2022.ct.digicert.com/log/ct/v1/get-entries?start=65562066&end=65562066)
you get an entry with leaf hash
ty69nN62xOf6NHpRjmsw7GxIa3IfKVUK/2EBQDUsNU8=.

However, if you try to request an inclusion proof for this leaf hash
(https://yeti2022.ct.digicert.com/log/ct/v1/get-proof-by-hash?hash=ty69nN62xOf6NHpRjmsw7GxIa3IfKVUK/2EBQDUsNU8=&tree_size=65569149),
you get an error:

{ "error_message": "The leaf hash was not found", "error_code": "hash unknown" }

Regards,
Andrew

[Jeremy Rowley]

I can confirm that the log is not operated correctly. The last good treehead was signed on June 29th around noon GMT.  Somehow entry  65562066 shifted one bit after the June 29th signing, which is causing the issue. If you shift the bit back, the tree signs correctly. We are still investigating why the first happened and have only hit deadends so far. The cert is logged in other DigiCert logs. We replicated the timestamp in our dev environment (with a separate private key). Neither seem to be causing the error. 

Jeremy

--
You received this message because you are subscribed to the Google Groups "Certificate Transparency Policy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ct-policy+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/ct-policy/20210630174304.133c9a1dc9263cd84e568165%40andrewayer.name.

[Andrew Ayer]

It looks like the leaf hash added to the Merkle tree is
ti69nN62xOf6NHpRjmsw7GxIa3IfKVUK/2EBQDUsNU8= - a one-bit difference from
the correct hash of ty69nN62xOf6NHpRjmsw7GxIa3IfKVUK/2EBQDUsNU8=.

I think the most likely explanation is that this was a hardware error
caused by a cosmic ray or the like, rather than a software bug. It's just
very bad luck :-(

Unfortunately, it's not possible for the log to recover from this event.
In the case of Nimbus 2018, the log operator was able to fix the
get-entries response to return entries matching the STH. In this case,
it can't be done because it would require breaking SHA-2's preimage
resistance to find the preimage of the bit-flipped hash. Yeti 2022
will never be able to return a get-entries response that matches STHs
with tree_size > 65562066.

Additionally, the SCT issued for entry 65562066 can't be audited,
because the client will attempt to audit the correct leaf hash, which
is not in the Merkle tree.

There are still entries being added to Yeti 2022. CAs should
immediately cease using this log, and ideally it would be made
read-only.

Regards,
Andrew

> https://groups.google.com/a/chromium.org/d/msgid/ct-policy/CAFK%3DoS8jrOyxtnndoJ55pO-pxE0pAtvV8jY2p98tXt862fQ18A%40mail.gmail.com.

[Devon O'Brien]

This bit flip does appear to be an unfortunate random event, almost certainly caused by hardware and outside the control of the Log Operator (DigiCert). 

That said, the inability to recover this entry will continue to interfere with monitoring and auditing Yeti2022 for as long as it remains Qualified, Usable, or Read-Only,  and will be making a policy announcement to that effect very soon. In the meantime, as Andrew mentioned, if DigiCert could configure Yeti2022 to stop accepting new entries as soon as possible, it will help minimize the impact to the ecosystem in the interim.

I'd also like to reiterate that we do not believe that this failure is the fault of DigiCert, and we've long speculated that extremely low chances of hardware bit flips, combined with the append-only nature of CT Logs might eventually manifest in a situation like this at some point. We would gladly accept (and encourage!) the application of a new 2022 shard in place of Yeti2022. 

-Devon

[Jeremy Rowley]

Thanks Devon. We've moved the log to read-only mode. No additional certs can be logged to the Yeti 2022 shart.