The names tiles Static CT API extension

[Filippo Valsorda]

Hello fellow mortals,

I'm reluctantly proposing an optional extension to the Static CT API which acknowledges the fact that a large portion of CT log clients are not participating in the transparency or WebPKI ecosystem, but are purely interested in discovering domain names.

Last week I reviewed two new RFC 6962 clients, and found that neither verified the STH signature or the inclusion of leaves in the root. One of them even turned off TLS verification. All they care about is the list of SANs (presumably for subdomain discovery purposes). I've seen more like this in the past, and my guess is that a large portion—maybe a majority—of the read traffic of CT logs is from similar clients.

Even with Static CT, serving full data tiles to these clients is a colossal waste of bandwidth. I propose to document (in c2sp.org/static-ct-api? somewhere else?) that Static CT logs MAY optionally expose gzip-compressed tiles of the following structure in JSON lines format.

type TrimmedEntry struct {

    // Index is the zero-based index of the leaf in the log.

   Index int64

   // Timestamp is the UNIX timestamp in milliseconds of when the entry was

   // added to the log.

   Timestamp int64

   // Subject is a DER encoded RDNSequence.

   //

   // It is omitted if it includes only a CommonName that matches one of the

   // DNS or IP entries. That is the case for all Domain Validated WebPKI

   // certificates.

   Subject []byte `json:",omitempty"`

   // DNS and IP are the Subject Alternative Names of the certificate.

   DNS []string `json:",omitempty"`

   IP []string `json:",omitempty"`

}

I rolled this out to navigli2025h2, and the compressed names tiles are more than 20x smaller than the compressed data tiles. There might be some induced demand from making it easier to access this data, but it's unlikely to cause a 20-fold increase. This has the potential to drastically reduce the bandwidth requirements of running a log, probably taking it below 1 Gbps, which significantly expands accessibility.

Here's a PR that implements support in Sunlight's write path, and in the Static CT API Go client. In particular, note that the Go client falls back to data tiles, so we don't need universal log support if clients use the library (which is a big if).

The clients I reviewed don't support Static CT yet, so if we want them to adopt this, now is the time to roll it out. (The User-Agent requirement might also help discover and contact them afterwards, but it's going to be more of a slog.)

To be clear, we should not steer security-conscious clients to this; I did my best to make the secure, inclusion- and signature-checking client API as easy to use as possible so that hopefully more Static CT clients will check the log than they did with RFC 6962. Also, this should not become part of browser policies, and logs should not be assessed or disqualified based on its correctness. It's just an optional side-API for not-really-CT clients who demonstrably don't care about the transparency properties.

I'm looking forward to any feedback. I plan to merge the PR and roll this out to Tuscolo at the end of the week if there are no strong objections.

Ciao,

Filippo

[Ivan Ristic]

As a thought, it might be useful to put this API endpoint on a different URL, where different rate limiting policies can apply. I imagine that operators with a good load balancer can do this anyhow, based on the path, but for some it might be useful to be able deploying on a separate hostname altogether.

I am with you on the "reluctant" part, but this could be a good way to distinguish those who care about certificates from those who don't.

--
You received this message because you are subscribed to the Google Groups "Certificate Transparency Policy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ct-policy+...@chromium.org.
To view this discussion visit https://groups.google.com/a/chromium.org/d/msgid/ct-policy/bcfca3d1-7547-4dc4-b43d-7bf1549f6815%40app.fastmail.com.

--

Ivan

[Jeroen Massar]

> On 4 Aug 2025, at 13:17, Filippo Valsorda <fil...@ml.filippo.io> wrote:
>
> Hello fellow mortals,
>
> I'm reluctantly proposing an optional extension to the Static CT API which acknowledges the fact that a large portion of CT log clients are not participating in the transparency or WebPKI ecosystem, but are purely interested in discovering domain names.
>
> Last week I reviewed two new RFC 6962 clients, and found that neither verified the STH signature or the inclusion of leaves in the root. One of them even turned off TLS verification. All they care about is the list of SANs (presumably for subdomain discovery purposes). I've seen more like this in the past, and my guess is that a large portion—maybe a majority—of the read traffic of CT logs is from similar clients.

My two small rappli: maybe we should have guidance for client authors noting that if one just want the names, but do not care if they actually are valid that they can use this path and if they actually care about the cryptographic part that they use that properly.

Fully agree on introducing that now, because yes, lots of consumers are likely purely using CT for discovering hostnames, for which it is a great technique if one does not have access to a Public DNS DB, which actually will only see leaked queries and not hostnames for which a SSL cert is created with everything as good as needing one.

Private CAs are therefore still heavy in use for many enterprises to not leak that kind of information to the public.

(Other can of worms: ability to say "Private CA with public key X can only sign certs for domain example.net <http://example.net/> and can log certificates in CT while stripping any more specific labels..." thus allowing browsers etc to see 'yeah that CA was involved and we know about the cert, thus must be real' and also allow the private CA to verify that nothing is mis-signed as they have the list of names and can compare to the internal DB thus possibly exposing leak of the private key and certs being signed outside of the real system...)

Greets,
Jeroen

[Seo Suchan]

I think this endpoint may (ab)used by CT users that only want to monitor about certificates of their own domain, by only monitoring this normally than lookup main tiles only when they see domain they care about: but unlike full ct client this doesn't ensure certificate named by it didn't created, because this name tiles are not cryptographically bind to tree, so ct log can skip any name they want, right? This looks like major footgun and feels like what most of name-only clients are trying to do. I think this tile need to bind to main tree some way, for example as an entry in main tile. so it can verified by full monitor that they didn't skiped name in it to be actually used.

2025년 8월 5일 화요일 오전 4시 38분 34초 UTC+9에 Jeroen Massar님이 작성:

[Bas Westerbaan]

I agree it makes total sense to cater to this use case.

Perhaps to signal that this is not cryptographically bound or part of the proper API, we could just make it be newline separated dnsNames?

Best,

 Bas

--

[Filippo Valsorda]

Thanks everyone for the feedback, comments inline.

2025-08-04 14:23 GMT+02:00 Jeroen Massar <jer...@massar.ch>:

My two small rappli: maybe we should have guidance for client authors noting that if one just want the names, but do not care if they actually are valid that they can use this path and if they actually care about the cryptographic part that they use that properly.

Definitely, the docs should be very clear about that.

In the Go client API I called the method UnauthenticatedTrimmedEntries() so that there'd be no way not to notice even without reading the docs.

2025-08-04 16:00 GMT+02:00 Ivan Ristic <ivan....@gmail.com>:

As a thought, it might be useful to put this API endpoint on a different URL, where different rate limiting policies can apply. I imagine that operators with a good load balancer can do this anyhow, based on the path, but for some it might be useful to be able deploying on a separate hostname altogether.

That would be nice, but has a discoverability issue. I don't think we should put these endpoints in log_list.json, so how does a client learn where the endpoint is?

Keep in mind we have the somewhat unusual task of intercepting relatively lazy clients (they are not checking inclusion proofs!), so if we add too many hoops they will just use data tiles.

2025-08-05 17:28 GMT+02:00 Seo Suchan <tjt...@gmail.com>:

I think this endpoint may (ab)used by CT users that only want to monitor about certificates of their own domain, by only monitoring this normally than lookup main tiles only when they see domain they care about: but unlike full ct client this doesn't ensure certificate named by it didn't created, because this name tiles are not cryptographically bind to tree, so ct log can skip any name they want, right? This looks like major footgun and feels like what most of name-only clients are trying to do. I think this tile need to bind to main tree some way, for example as an entry in main tile. so it can verified by full monitor that they didn't skiped name in it to be actually used.

Who would check that binding? Again, these are for clients that are already using the current system without verification. There's no reason they would check this binding if they don't check inclusion in the root. They already made the choice to trust the log. Full monitors can't help them either, because logs could just serve different names tiles to full monitors and to non-verifying clients.

2025-08-05 18:14 GMT+02:00 Bas Westerbaan <b...@cloudflare.com>:

Perhaps to signal that this is not cryptographically bound or part of the proper API, we could just make it be newline separated dnsNames?

I'd be happy to add more signposts hinting at the insecurity of this. However, it also has to be useful to most non-verifying clients, or they will just go back to (unverified) data tiles.

I think some of them do care about the relationship between names on the same certificate, so it kinda has to be one line per entry, and JSON is less annoying than figuring out what to do if there's a space in a SAN.

Same argument for the optional OV/EV Subject, some might care about tying domain names to entities. (Weak opinion on this, to be honest.)

Then I put the timestamp in there because some of the dashboards I've seen have a "first seen" column.

We could remove the index? It's easy to reconstruct, but I guess there's no point in making it easy to map back to the specific log entry.

[Filippo Valsorda]

2025-08-05 19:02 GMT+02:00 Filippo Valsorda <fil...@ml.filippo.io>:

We could remove the index? It's easy to reconstruct, but I guess there's no point in making it easy to map back to the specific log entry.

I have removed the Index field from https://github.com/FiloSottile/sunlight/pull/45.

[Martin Hutchinson]

Hi all,

First up, thanks to Filippo for your awareness. We all benefit from you looking at the bigger picture to spot this usage pattern. I can see why this makes sense for a log operator to host, but I have concerns across a few dimensions.

Concern: Peer pressure (or policy) leads to non-optionality

If this goes into the Static CT spec as an optional extension, that’s a bit funky, but I can see the argument. I’d personally prefer to see this API take shape in its own separate spec, but there’s initial overhead to that. It does make versioning easier in the future though.

My main concern is that while it may be listed as optional, in reality, it may become enforced via policy, or simply peer pressure. This proposal was discussed on Slack with the observation that “it only works if all production Static CT logs expose it, or lazy clients will just use the ubiquitous option”. If this is true, then any log operator turning up a Static CT log will be the black sheep if they choose not to implement this “optional” API.

Concern: Non verifiable endpoints on CT logs is a dubious role model

CT is the poster child for transparency. Newcomers to transparency often take the stance that if CT does it, then it’s the right approach. I’ve spent countless hours over my time in transparency telling people that “yes CT uses SCTs, but that doesn’t mean you should, and here’s why...”. I worry about the message that this sends out to the wider transparency community, especially as Static CT is touted as CT redone with modern best practices.

Such a change seems likely to embolden those that are liable to believe that “verifiable logs always tell the truth, so you don’t need to verify them”.

Concern: The contents of the response are reactive to current usage

It’s great to see that the content of the response has been slimmed down since the original proposal. However, this has a flip-side. Imagine if in 6 months, we spot a high usage from people needing to know about the Issuer, perhaps to train AI models to look for unusual issuance patterns. The contents of the proposed TrimmedEntry wouldn’t allow such users to do this, so they would need to fall back on the full tiles. Would we then version up the Static CT spec to add new fields to this response body? Versioning this and rolling it out across log operators and clients seems like a challenge.

Concern: Consumers need to know about CT policy (and pull many duplicates)

Clients harvesting SANs from logs will need to have a fresh copy of all current CT logs, which means somehow acquiring a copy of the OTLL. Either they pull directly from, e.g. Chrome, or the client library maintainer needs to propagate changes into the library. We’ve seen problems with staleness and brittleness from these kinds of dependencies before.

The consumer library then needs to determine which API to invoke on the log. Does this mean that the OTLL needs to publish a field to indicate whether this optional API is available?

The consumer then needs to poll every log on the policy, knowing that every cert will appear in multiple logs (potentially in both the pre-cert and final cert form). This is more duplicate data, though the cost is at least spread across the ecosystem.

Proposal

In my eyes, the correct approach is that one or more actors step up to effectively run a SAN announcement service. There are already actors that consume all of the logs (e.g. monitors), and so have the data to do this. They would also be able to effectively de-dupe the same cert in multiple logs, and between the pre- and final-cert forms.

I like this because it maintains the verifiable properties further down the pipe, and addresses all of the concerns I’ve outlined above.

There is one downside however: finding one or more parties to host such a service. The cynic in me understands that you probably considered this, ruled it out, and thus made your proposal that logs should run the service. However, I still have a spark of optimism, and maybe someone will come forward and allow my optimism to grow.

Cheers,

Martin

--

[Filippo Valsorda]

Hi Martin,

Thank you for thinking this through.

I think the first two concerns boil down to "we don't want this to be too official" and I agree. How about we make the "specification" just a Markdown file in the Sunlight repository, with a non-normative link from c2sp.org/static-ct-api, which specifies it should not be emulated?

About the ubiquity requirement, I realized after making that comment that the client can just fallback to data tiles, and that's what I implemented in UnauthenticatedTrimmedEntries.

In general, we also need to figure out better ways to guide new transparency ecosystems than by correcting them after they emulate CT. Even without this, Static CT is not a good example, with its SCTs and extra data, as you said.

The latter two concerns, as well as the proposal, seem to boil down to "we can do better than this" and we definitely can! I don't think it's worth it though. This is a targeted optimization to hopefully offload clients that are not the primary concern of the CT ecosystem. This line of thought might be worth a week of our collective time, but I don't think it'd be worth a month. No amount of optimism will convince me that a sustainable, reliable public collating service can be stood up in less than a month.

As to discovery specifically, these clients are already using all_logs.json, so they already have the monitoring prefix. Remember we don't really care about breaking these clients, so I don't think that's a problem.

To sum up, this was a timeboxed performance optimization exercise. If the consensus is that #43 is a net negative, I will drop it and probably not think about those clients again soon. Otherwise, I will merge it as-is or with minor tweaks at the end of this week. If in six months it turns out to be (or become) useless, we'll just turn it off.

I obviously can't dispose of anyone else's time, so if someone else disagrees and thinks it's worth dedicating more resources to a proper solution, I am also happy to turn this off once that proper solution exists.

Cheers,

Filippo

[Martin Hutchinson]

Hey Filippo,

On Wed, 6 Aug 2025 at 14:48, Filippo Valsorda <fil...@ml.filippo.io> wrote:

Hi Martin,

Thank you for thinking this through.

I think the first two concerns boil down to "we don't want this to be too official" and I agree. How about we make the "specification" just a Markdown file in the Sunlight repository, with a non-normative link from c2sp.org/static-ct-api, which specifies it should not be emulated?

Making it less official would largely mitigate my immediate concerns with it. Playing around with endpoints in a less official capacity amongst log operators that choose to do so sounds useful, and avoids the concerns that I laid out in the previous email.

About the ubiquity requirement, I realized after making that comment that the client can just fallback to data tiles, and that's what I implemented in UnauthenticatedTrimmedEntries.

A fallback like this makes sense, and allows log operators to turn this on or off as they experiment, and without needing to communicate this via any official mechanism. SGTM.

 

In general, we also need to figure out better ways to guide new transparency ecosystems than by correcting them after they emulate CT. Even without this, Static CT is not a good example, with its SCTs and extra data, as you said.

Neat idea! Have I ever told you about the Claimant Model? ;-)

 

The latter two concerns, as well as the proposal, seem to boil down to "we can do better than this" and we definitely can! I don't think it's worth it though. This is a targeted optimization to hopefully offload clients that are not the primary concern of the CT ecosystem. This line of thought might be worth a week of our collective time, but I don't think it'd be worth a month. No amount of optimism will convince me that a sustainable, reliable public collating service can be stood up in less than a month.

As to discovery specifically, these clients are already using all_logs.json, so they already have the monitoring prefix. Remember we don't really care about breaking these clients, so I don't think that's a problem.

To sum up, this was a timeboxed performance optimization exercise. If the consensus is that #43 is a net negative, I will drop it and probably not think about those clients again soon. Otherwise, I will merge it as-is or with minor tweaks at the end of this week. If in six months it turns out to be (or become) useless, we'll just turn it off.

I obviously can't dispose of anyone else's time, so if someone else disagrees and thinks it's worth dedicating more resources to a proper solution, I am also happy to turn this off once that proper solution exists.

I don't object to any other log operators turning up a side API. On the whole I think the PR is a net positive, so long as it doesn't come with any burden on other log implementations to also do this. Opportunity is cool, burden is not.

I also don't have the power to dispose of anyone else's time, but I have had a couple of responses to say that people are thinking of doing this -- my optimism lives another day! The ideal outcome could be that this side API is launched, assuming that it will be transient and replaced N months from now. And then we trust that the Right Thing will happen, and turn down any experiments that are no longer useful.

Grazie mille,

Martin

[Filippo Valsorda]

The names tiles experiment has landed in Sunlight, and was rolled out to Tuscolo and the Let's Encrypt Sunlight logs.

The format is documented at https://github.com/FiloSottile/sunlight/blob/main/names-tiles.md, with a scary warning at the top. I opened a PR to add a link from the Static CT API specification (since folks need to find this somehow), but that's also full of "go away" signs and spiked traps.

Sunlight v0.6.0's new Client.UnauthenticatedTrimmedEntries method automatically fetches names tiles if available, and falls back to full Static CT data tiles otherwise.

Unrelatedly, sunlight.Client now has Checkpoint(), CheckInclusion(SCT), and Issuer(fingerprint) methods, too. I believe this completes the set of required features to implement both tailing and pecking Static CT monitors.