Hi all,
Late last year multiple WebPKI community participants noticed the same set of issues burdening the CT ecosystem, and came up with overlapping solutions. Today, I am happy to announce Sunlight, a design developed in collaboration with Let’s Encrypt, and borne out of discussions with the Chrome and TrustFabric teams at Google, the Sigsum project, and other CT log and monitor operators.
You can find a design document at https://filippo.io/a-different-CT-log, a formal specification of the API at https://c2sp.org/sunlight, the implementation at https://github.com/FiloSottile/sunlight, and additional resources including running logs at https://sunlight.dev.
This design is probably not perfect. The question we’re asking today is whether it is moving in the right direction, and if the migration path is sustainable for every stakeholder. If so, we hope we can incorporate feedback and go ahead with deploying it, and then iterate further in the future.
We invite everyone to read the more in depth resources above, but here is a summary of the technical details.
No Merge Delay. add-chain requests are pooled and held until the next sequencing, which happens every second. No SCT is returned until the leaf is merged. add-chain requests complete in ~700ms on average, 1.5s at most.
SCT leaf_index extension. SCTs include an RFC 6962 extension with the index of the incorporated leaf.
Object storage backed. All critical data is stored in object or file storage, like S3 or a filesystem. There is no relational database.
Static tile-based read path. Monitors can fetch the tree directly from object storage, where it’s laid out as “tiles”. (A Go client library is coming soon, and we’d love to hear from monitors what they need from it.)
Single node. Instances operate with a single node, relying on the object store for durability, and on the distributed nature of CT for overall system availability.
Compare-and-swap safety mechanism. As an additional safety measure against operational issues, each sequencing performs a compare-and-swap operation of the STH against an auxiliary storage backend.
Sunlight was designed to address pain points of log operators and monitors.
Reduced cost. Object storage is cheaper than database storage, Sunlight can run with trivial amounts of CPU and memory, and bandwidth costs are reduced by tile compression. Operating the write path in the cloud should cost less than $5k/year. The read path in the cloud is dominated by bandwidth costs, which will benefit significantly. Off-cloud any 1Gbps racked server will do.
Reduced operational complexity. No RDBMS, no merge delay, no consensus. Object storage is readily available in both cloud and self-hosted environments. Porting it to a new infrastructure (Fly.io and Tigris) took a couple days.
Nearly bulletproof. As long as the compare-and-swap backend (which requires no manual maintenance) is not tampered with, it should be impossible to cause irreversible incidents such as consistency failures. The operator can run parallel instances, rollback local or object storage, run out of disk, or even reuse keys, and Sunlight will fail safe.
No read rate-limits. If S3 has rate-limits, I couldn’t find them. With a 10Gbps connection monitors should be able to fetch a whole log in a couple hours, and certainly won’t be rate-limited tailing the log.
Easily scalable. Object storage scales effortlessly, and Sunlight was designed to handle 30x the current WebPKI issuance rate without load balancing across logs. Essentially all CPU goes towards chain verification and SCT signing, and currently consumes ~40% of a single CPU core, suggesting we can get to 128x the current rate on a single node.
A step towards killing SCTs. We are not proposing any changes now, but embedding the leaf index in the SCT is a step towards removing that indirection altogether and verifying proofs on the clients.
You can read more about how these qualities relate to the experience of a log operator in Let’s Encrypt’s announcement.
To sum up, Sunlight aims to make CT log operation more accessible by reducing cost, complexity, and risk. As a side effect, we also hope the API changes will remove “rust” from the protocol, allowing it to iterate more often in the next ten years than it did in the past ten.
We look forward to collecting feedback and answering questions here, or in the #sunlight channel of the transparency-dev Slack. PRs and issues are also welcome at the spec and implementation repositories.
A presto,
Filippo
--
You received this message because you are subscribed to the Google Groups "Certificate Transparency Policy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ct-policy+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/ct-policy/e452b894-d691-4f6f-aecc-454c77654c05%40app.fastmail.com.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/ct-policy/CAFZs0S4Z0%2B%3DOK75-en%2Bt9Uy0wRLP7ahcZkAhp5-%2BmjHJOcsrOA%40mail.gmail.com.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/ct-policy/CABrd9SSY1nCbL_EqxkGgz2o4nSkzeTHLQM-cMe6%2BjbHdmBMx-Q%40mail.gmail.com.
I acknowledge the Sunlight design is trading off the write path availability, but not the read path. I would expect that S3 + Cloudfront to have similar or higher availability than Trillian on RDS. AWS advertises a 99.95% sla for RDS, and 99.99% for S3. In practice we have had more problems with RDS than S3, primarily related to scaling under load.
Given that, I'm not sure that increasing the number of required SCTs follows from that.The concern about multiple operators on a single object store leading to shared outages is a real concern. I believe based on their public IP addresses, the current Let's Encrypt, Digicert, and TrustAsia's logs are all backed by AWS.I don't know what the best path is to de-risking that, but it is not a new risk.
It's extremely exciting to see the launch of Sunlight! It should makeCT logs significantly easier to operate reliably, which will improvethe health of the CT ecosystem.I've updated SSLMate's monitor to ingest STHs from the 15 publicSunlight logs, and it has already detected an issue. The Sunlightspecification says:"Note that all cryptographic operations (such as hashes and signatures)are as specified by RFC 6962, so these APIs can be thought of as analternative encoding format for the same data"However, at least 10 Sunlight logs have produced STHs with a tree sizeof 0 and an all-zero root hash, while RFC 6962 specifies that the roothash of an empty tree is the SHA-256 hash of an empty string. I'veattached the STHs to this email in the form of an STH pollination JSONdocument.
I do like this design, not least because it is a step towards a design I've noodled for many years now: RAIL (Redundant Array of Inexpensive Logs), the core idea there being you accept an even lower availability and thus accept that you need to get more SCTs (or, in fact, inclusion proofs) in your cert.
One of the design goals of introducing a new read-path API is that the tiles can be served directly from a web server or object storage (either directly, or via CDN).The application server handles the write APIs and stores tiles, but isn't involved in the read path at all.Thus Monitors/Auditors should be able to continue to process entries even if the application server handling writes is down, or gone.
I anticipate this will make keeping historical log archives or mirrors much easier as well.
I don't anticipate that the current design will reduce service providers very much, either.The cloud providers all have object storage available as a commodity item (At least: AWS, Google, Azure, Oracle, DigitalOcean, OVH, Linode).While the current Sunlight implementation uses object storage, the design can be adapted to standard file storage.That means any classical "web servers with a storage cluster" type on-prem/bare-metal deployment will work as well.Or if you just have four servers in a rack, Minio or Ceph are options.
The case of "we only have databases" isn't currently supported, but I'm not sure that's a significant fraction of plausible service providers.And of course, we don't want a single CT log implementation running all logs either.Anyone who wants to run a database-backed log can continue to run Trillian or other implementations.
2024-03-18 21:48 GMT+01:00 Ben Laurie <be...@google.com>:I do like this design, not least because it is a step towards a design I've noodled for many years now: RAIL (Redundant Array of Inexpensive Logs), the core idea there being you accept an even lower availability and thus accept that you need to get more SCTs (or, in fact, inclusion proofs) in your cert.Hi Ben, very glad to hear you like Sunlight! Hopefully that means you don't mind that we used the name of your initial draft ^^ (If you actually do mind, the name would be my fault, for the record.)Indeed, I do think it's a step closer to the ideas of RAIL, and there is an availability tradeoff. However, I want to stress that Sunlight was explicitly designed not to require additional SCTs, as that would be a significant change in browser policy, which could slow down rollout.The CT system has two somewhat orthogonal redundancy mechanisms: multiple qualified logs in the ecosystem, and multiple SCTs in a certificate.If a log's add-chain endpoint is unavailable, CAs need to submit to other logs. This is where Sunlight's single node architecture trades off availability: it can't have as many nines as a Trillian instance can have. This will be ok as long as there are enough logs overall, and Sunlight will hopefully increase the overall number of logs.On the other hand, if a log becomes Rejected, certificates need to include other SCTs to remain valid. Sunlight does not suffer a tradeoff here. The kind of issues that cause logs to become Rejected are not any more likely to occur to a Sunlight log than they are to a Trillian log. Even if the 99% SLO is strictly enforced on the write path, 22h of downtime (1% of three months) is the stuff of major incidents or software bugs, not the effect of updating or migrating a single-node system. Major incidents and software bugs spare no one, neither single nodes nor distributed systems, as we’ve seen. Hell, my maybe unpopular opinion is that distributed systems are less likely to have brief downtimes but more likely to have long downtimes, because when they do go down they are often more complex to bring back up.
Anyway, to sum up Sunlight does make an availability tradeoff, but doesn’t increase log rejection chances, so doesn’t need additional SCTs.
P.S. We could even have a conversation about decoupling write-path and read-path SLO, because the former is compensated by log redundancy, while the latter is also about making sure certificates make it to monitors in a timely fashion. However, I prefer not to put that conversation on the critical path of deploying Sunlight logs, as there's no need.
Thank you Andrew for bringing this up, it's an intentional design choice that's worth discussing with the community.Monitors indeed need a simple reproducible process to reconstruct chains.My preference would be to specify byte-for-byte equality of Issuer and Subject as a requirement for Sunlight-logged chains.
--You received this message because you are subscribed to the Google Groups "Certificate Transparency Policy" group.To unsubscribe from this group and stop receiving emails from it, send an email to ct-policy+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/ct-policy/c1734f89-d5fa-4342-8542-04c5cbf5bfb3%40app.fastmail.com.
That might prove OK in the end, but it makes me nervous. The Sunlightspec claims "these APIs can be thought of as an alternative encodingformat for the same data". I find this an extremely compellingproperty that justifies accelerated adoption of Sunlight, because itmeans consumers can ultimately get the same bytes from a Sunlight logas they would from an RFC6962 log, giving me confidence that monitoringSunlight logs will work just like monitoring RFC6962 logs. But ifmonitors and translation proxies have to perform non-reproducible chainbuilding, then the property claimed in the Sunlight spec does not hold,and I worry that there is some new edge case waiting to be discovered.
(By the way, I support requiring byte-for-byte subject/issuer matchingduring acceptance, and this can still be required. But it should notbe a load-bearing part of the protocol.)
On Fri, 07 Jun 2024 17:27:46 +0200"Filippo Valsorda" <fil...@ml.filippo.io> wrote:> I have prepared a revision to the Sunlight spec with the following> changes:I just finished writing an RFC 6962 compatibility proxy against the newSunlight spec: https://github.com/AGWA/sunglasses/
I found the exercise very straightforward. The logic is simple - muchsimpler than it would have been without the change to certificate chains.I'm feeling good about the protocol.
I wanted to flag one aspect of the protocol for discussion. The logentries in the data tile are not prefixed by their length, so skippingan entry requires parsing it successfully. Consequentially, if alog publishes a malformed entry, monitors won't be able to parse anysubsequent entry in the tile. In contrast, a single malformed entry inRFC 6962 doesn't prevent other entries from being parsed. But perhapsa single malformed entry necessitates disqualifying the log anyways,so this is not a big deal? This has never happened with RFC 6962 logsso there's no precedent. I'm curious what others think.