It's totally dead, and I don't think we can recover it. The sheer
number of records in the 2022 shard is making recovery of the database
pretty much impossible.
Here's the incident report:
The morning of Sept 28, SRE was performing work setting up a new
Cassandra cluster for the Yeti CTLogs to migrate to. During this, a
blanket delete command was inadvertently issued to all Cassandra
servers of both the new and old clusters. This resulted in the
deletion of all files under /var/lib/cassandra/data.
The mistake was immediately caught, but not before the command was
completed running on all Cassandra servers. At 19:22 GMT the Yeti
CTLog applications were configured to not accept new entries to
prevent any further data loss. After stopping the damage, we attempted
to recover the files and started exporting data for the 2023-2025
shards. We also stopped signer applications for all Yeti shards. 2024
and 2025 were quite fast as they had little to no certs. We managed to
re-stand up those logs within 2 hours of starting the export, although
they weren't live for new signing for about 3 hours. 2023 had more
data and the attempts to export failed. We figured out the issue with
the export, fixed it, and started exporting again.
Please let me know if there are any follow up questions I can answer.
Jeremy