Mariadb HA with Trillian

[Nicolas Dufour]

Hello,

I'm Nicolas Dufour, I'm working at Sectigo on setting up the new Sabre log based on Trillian.

So far, it's been using a mariadb server as its main storage and I was wondering if some of you could chime in on your experience on using mariadb (or mysql) in HA configuration.

A backup of the db is of course useful, but obviously if the backup is once a day, then the log can't recover.

Any suggestion on using galera or other approach?

Thank you,

Nicolas Dufour

[Phil Porada]

Here's some blog posts we've written about running our CT logs. Additionally, if you're going to be running servers you control instead of say Amazon RDS like we use, check out our blog post about how we run the Boulder CA database on MariaDB with ZFS. Almost all of the problems you will run into with Trillian are going to be database related. There's risk mitigation strategies that exist, but you'll have to figure out what's right for your log. The nuturing CT log growth has our updated/current sharding strategy laid out.
1) https://letsencrypt.org/2019/11/20/how-le-runs-ct-logs.html
2) https://letsencrypt.org/2022/05/19/nurturing-ct-log-growth.html
3) https://letsencrypt.org/2021/01/21/next-gen-database-servers.html

This google doc contains a bunch of notes from when we toyed with using Galera and physical servers for our non-existent Birch and Elm logs. Looking back, I still don't think that I would choose Galera unless I had a stable network with >=10GB links between database servers. Sticking with standard MariaDB, a tool like `openark/orchestrator` or something else that can intelligently handle MariaDB failovers would be very handy.
1) https://docs.google.com/document/d/1h6W6XXpFkfGC6udG0GsQBmRKwy06RTAWlsKxfms49No/edit

Here's some post-mortems we've had as a result of database shenanigans we've encountered (or caused) along the way. You may find them helpful.
1) https://groups.google.com/a/chromium.org/g/ct-policy/c/madcUcQZ1IQ/m/AeYuAvc6BAAJ
2) https://groups.google.com/a/chromium.org/g/ct-policy/c/9fTOoC4UzmE/m/f6ij84uECAAJ