Let's Encrypt Static Logs reset

[Matthew McPherrin]

In the year that Let's Encrypt has run our new Sunlight CT logs, we've learned a lot.

In preparation for applying to be trusted logs, we'll be shutting our current shards down and spinning up new ones. Since these logs aren't trusted, this shouldn't have any impact, but monitors will want to switch to the new shards once available.

Since I know many people are interested in operating static CT logs, I thought I'd share a few details.

At a high level, we'll be making a few changes:

* Switching to a single S3 bucket, instead of 1-per-shard. We think this will work better with AWS's behaviour around scaling up the underlying infra for S3. We saw degraded S3 performance when 90-day certs switched to a new bucket, which we hope we can avoid going forward. This is mostly invisible to the end-user as we only expose the storage via CDN, but felt it was worth mentioning for the collective awareness.

* Splitting the submission and monitoring prefixes. When I first set up Sunlight, it wasn't clear to me that we'd publicly expose two prefixes, so there's a bit of extra complexity in our CDN configuration that we can drop. We can also reduce a bit of traffic through the CDN that isn't really needed.

Once the new shards are up, we'll be starting the process with the CT log programs of applying to have them be trusted Static CT logs.

[Jeremy Rowley]

Hi Matthew - thank you for sharing this! how much better was the performance of sunlight compared to your trillian logs. I assume the tiling (genius idea) was a huge pay off?  Was the goal in eliminating merge delays successful? I would love to read more about the results of your year long operation if you've published it anywhere.

--
You received this message because you are subscribed to the Google Groups "Certificate Transparency Policy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ct-policy+...@chromium.org.
To view this discussion visit https://groups.google.com/a/chromium.org/d/msgid/ct-policy/CAKh5S0bt03dMTFzXz0krcK39-OzH1%2BbgGrzE6WZ7vYCeOAqnvQ%40mail.gmail.com.

[Fernando 🐼]

For the longest time, AWS required specific bucket path prefix to distribute data in a balanced way. 

That has been dramatically improved a few years back. 

But anyone with a TAM should consult them on the ideal strategy to store large amount of data in buckets for performance reasons. Also for their infra to scale as fast as you need. 

--
Fernando 🐼

--

[Matthew McPherrin]

I have good news! We're writing a blog post on this subject. I'll be sure to share with this list once it's ready.