The Tuscolo Static CT log

[Filippo Valsorda]

Ciao a tutti,

I’m excited to announce the Tuscolo Static Certificate Transparency log, operated by Geomys.

https://tuscolo.sunlight.geomys.org

Tuscolo is a production log, hosted on a single dedicated server by Port 179 LTD (better known as the operator of bgp.tools, AS206924, and Behind the Sofa). It is backed by Sunlight and a new local filesystem backend.

We wish for this log to be both public good infrastructure for the WebPKI, and a learning resource for the CT community. To that end, we are happy to share administrative and operational details.

The total yearly amortized cost to Geomys is £7,760 ($10,350): £5,250 of hardware price, and £500/month of colocation and operational support. This includes 500Mbps of P95 bandwidth (~160TB of traffic per month), which we estimate should be 2-3x what we’ll need, based on data from other operators and accounting for Static CT savings of up to 78%. We can easily scale bandwidth at increasingly cheap marginal costs.

The hardware is a an overprovisioned refurbished Supermicro H12SSW-AN6 with new enterprise SSDs for the main storage pool.

• 24-core Zen 3 (Milan) AMD EPYC 7443P
• 128GB of ECC DDR4 2666 MHz
• 25Gb/s Mellanox ConnectX-5
• 2x 512GB SK hynix PC801 NVMe SSDs (md raid1, rootfs)
• 4x 7.68TB Samsung PM9A3 U.2 SSDs (ZFS raidz2)
• Redundant power / PDUs
• Colocated at Digital Realty's LON1
  

Sunlight is configured with a pool size of 750 certificates per shard, selected somewhat arbitrarily, and sequences the pool every second. Tuscolo is currently receiving all Let’s Encrypt pre- and final certificates, with 1.05 system load, 41ms P50 / 101ms P99 sequencing time, and 1.04s P99 add-chain duration (including up to 1s of pooling).

The new local Sunlight backend does carefully fsync()ed writes to a POSIX filesystem for tiles, and checkpoints are tracked in a separate global SQLite database to avoid accidental dataset management-related rollbacks.

Since the local backend lays out tiles according to the Static CT hierarchy, the read path could have been any HTTP file server. However, we implemented a custom read path, Skylight, to get better observability and rate limits.

In particular, the Tuscolo logs apply a global rate limit of 75 requests / second from clients that don’t include an email address in the User-Agent, and an additional global rate limit of 75 requests / second for partial data tile requests. A well-behaved client that identifies itself and opportunistically waits for tiles to fill up will not incur in any rate limit. We think this will be an important tool to shape and iterate on client behavior, but we’re open to feedback from the community.

A couple fun Skylight details:

• Rate limits are implemented with the GCRA, taking a page out of Let’s Encrypt’s book.
• Heavy hitter User-Agent strings are tracked with the Space Saving algorithm, and exposed internally, as they include private email addresses. We might publish redacted stats occasionally.
• Copious metrics are exposed publicly, just like Sunlight’s, including connection reuse stats. (Please reuse connections!) Feel free to scrape them, but please use an interval of at least 60s.
• The /health endpoint that’s linked to our pagers loads the checkpoint of each log, verifies their signatures, and returns a 500 status code if any checkpoint is older than 5s.
• All endpoints allow Cross-Origin Resource Sharing. I am not sure what this is for, but maybe someone will come up with something interesting!
  

We are internally targeting 99.9% uptime across endpoints, and are confident we can exceed the 99% threshold. You can track our uptime on the automated status page.

Currently, we accept the same roots as Let’s Encrypt’s Oak log, but we plan to ensure we accept all logs in the Google, Apple, and Mozilla root programs before we submit this log for inclusion.

We also set up a staging Sunlight instance on the same box, called Navigli. It only accepts staging and testing roots, and we’re happy to add more. Let’s Encrypt’s staging certificates are including SCTs from it. We will test Sunlight development builds on this log, and we offer no SLO. With the introduction of Navigli, I will be decommissioning the Rome development log.

A selection of configuration files and system details is available here. You can also see our (short) playbook.

As you can imagine, I am running this log out of pocket in part to prove a point about how cheap operating a Static CT log can be. I am fully committed to run this log long-term regardless of funding, but it would be great to spin up another instance in a different location for redundancy, potentially on behalf of a generous CA or UA. wink

Looking forward to any feedback,

Filippo

P.S.: Fun fact, the first commit of Sunlight was written at Tuscolo, on this red bench, which is pictured in the Sunlight logo.

[David Cook]

Congratulations, and thanks, for setting this log up. The low operational costs are an important advancement for the ecosystem.

I spent some time this weekend testing the durability of Sunlight's local storage backend with the ALICE tool from Pillai et. al., OSDI '14, and the results look positive. This tool records system calls made by a program with strace, models their effect on file descriptors and the file system, reconstructs a variety of possible post-crash filesystem states, and checks how the application behaves when presented with those recovered states. In this case, I have the program alternate between adding a few entries and making checkpoints. After crash recovery, I confirm that the log doesn't get smaller, and I confirm that Sunlight's built in consistency checks pass (comparing checkpoints in the lock backend and storage backend; checking the right edge of the tree). My test harness is available on GitHub.

Currently, the tool is only reporting issues that could arise if rename operations are not persisted atomically by the filesystem. The paper mentions that many filesystems ensure renaming is an atomic operation through journaling or other techniques, so this seems like a reasonable assumption to make. Sunlight has fared better in terms of durability issues than other software I've looked at previously. I attribute this to the fact that all write operations go through a small number of code paths, either SQLite or in internal/durable/path.go, and the write operations in the latter file already make use of the usual durability tricks like writing to temporary files, swapping via renaming, and fsyncing directories.

Well done, and thanks again,

--David Cook

[Andrew Ayer]

First of all, thank you so much for doing this! In addition to adding more log capacity, demonstrating the feasibility of running low-cost logs will be enormously valuable to the ecosystem.

On Thu, 08 May 2025 18:40:02 +0200
"Filippo Valsorda" <fil...@ml.filippo.io> wrote:

> In particular, the Tuscolo logs apply a global rate limit of 75 requests / second _from clients that don't include an email address in the User-Agent_, and an additional global rate limit of 75 requests / second for partial data tile requests. A well-behaved client that identifies itself and opportunistically waits for tiles to fill up <https://c2sp.org/static-ct-api#partial-tiles> will not incur in any rate limit. We think this will be an important tool to shape and iterate on client behavior, but we're open to feedback from the community.

I have some concerns about the rate limiting of partial tiles. The spec permits (in fact, requires) clients to fetch a partial tile if it remains partial for "too long (as defined by client policy)". It doesn't say what a reasonable client policy is (technically, 1 second would be fully compliant!), but given the 60 second MMD, I would say that 60 seconds is reasonable. But if a log is growing slowly enough (such as the 2025h1 shards, which are currently growing at rates as low as 3 entries per second), then tiles would regularly remain partial for longer than 60 seconds, and monitors trying to retrieve certificates within the MMD would be subject to the rate limit. At that point, I'm not sure the log truly has a 60 second MMD. An entry should not be considered incorporated until it's available for unencumbered download from the log.

It seems wrong for a log to publish a checkpoint containing entries that it doesn't want clients to download yet. I'm curious how feasible it would be for logs to delay publication of checkpoints for partial tiles. For Sunlight/Skylight, I think Sunlight could sign two checkpoints when sequencing a pool: one containing all entries in the log ("checkpoint", as it does now), and another for the closest full tile ("checkpoint_fulltile"). Skylight would rewrite requests for "checkpoint" to "checkpoint_fulltile" if "checkpoint_fulltile" is less than 60 seconds old. (I realize this would not be fully "static" anymore, but Skylight exists to add smarts to the read path, and I chose the filenames intentionally so that it would still be possible to serve from a normal HTTP server, albeit without the partial tile avoidance.)

Or maybe Skylight could apply the rate limit only if every entry in the partial tile is less than 60 seconds old?

As for requiring an email address, I'm not opposed to the concept, but I do believe it violates both Chrome CT Log Policy and RFC 6962, which say "Log operators MUST NOT impose any conditions on retrieving or sharing data from the log." It should be precisely documented in log policy and/or the static-ct-api spec what clients have to do to get unencumbered access to a log. I think this is important for ensuring that the conditions are reasonable and not detrimental to transparency, and for avoiding a situation where every log has its own conditions which clients must know about.

Regards,
Andrew

[Luke Valenta]

Hi Andrew,

> I'm curious how feasible it would be for logs to delay publication of checkpoints for partial tiles.

We recently added the ability to delay sequencing partial tiles to the Azul implementation, and there's some discussion at https://github.com/FiloSottile/sunlight/issues/33 about how to get the right tuning to avoid CA timeouts. I'd be interested to hear your thoughts!

Best,

Luke

--
You received this message because you are subscribed to the Google Groups "Certificate Transparency Policy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ct-policy+...@chromium.org.
To view this discussion visit https://groups.google.com/a/chromium.org/d/msgid/ct-policy/20250522073125.15299c88ad4901f972535b95%40andrewayer.name.

--

Luke Valenta

Systems Engineer - Research

[Filippo Valsorda]

2025-05-22 16:31 GMT+02:00 Andrew Ayer <ag...@andrewayer.name>:

I have some concerns about the rate limiting of partial tiles.

We did some thinking on this and came to the conclusion that you're right, unsurprisingly :)

We have removed the partial tile rate limit. We still have metrics for how much bandwidth goes towards partial tiles, and if the metrics suggest it's worth it we can consider the more complicated solutions, but for now it might just not be.

Clients are still encouraged to perform the optimization for fast-growing logs.

As for requiring an email address, I'm not opposed to the concept, but I do believe it violates both Chrome CT Log Policy and RFC 6962, which say "Log operators MUST NOT impose any conditions on retrieving or sharing data from the log."  It should be precisely documented in log policy and/or the static-ct-api spec what clients have to do to get unencumbered access to a log.  I think this is important for ensuring that the conditions are reasonable and not detrimental to transparency, and for avoiding a situation where every log has its own conditions which clients must know about.

This we think is more of a grey area, and we care more strongly about keeping it. We agree there's a risk of proliferation of different rules in different logs, though.

How would folks feel about enshrining this as a client SHOULD in c2sp.org/static...@v1.1.0?

"Clients SHOULD include a way for the log operator to contact them in the User-Agent header, as an email address and/or a URL (with a `+https://` prefix), and logs MAY rate-limit anonymous or unreachable clients."

[Pierre Barre]

I believe that authenticating monitors, particularly with such weak authentication mechanisms, is counterproductive. A log operator acting in bad faith could exploit this information to their advantage by serving a different tree to a specific monitor.

If we want proper authentication and communication channels with consumers, we need a robust authentication system—not email addresses embedded in user agents. That approach will inevitably become unwieldy and create trouble.

Best,

Pierre

--

You received this message because you are subscribed to the Google Groups "Certificate Transparency Policy" group.

To unsubscribe from this group and stop receiving emails from it, send an email to ct-policy+...@chromium.org.

To view this discussion visit https://groups.google.com/a/chromium.org/d/msgid/ct-policy/02c308a9-e978-4078-af7d-6b7e94b4f815%40app.fastmail.com.

[Ben Cartwright-Cox]

Pierre

> A log operator acting in bad faith could exploit this information to their advantage by serving a different tree to a specific monitor.

That is ridiculous. If I wanted to do that (and to be clear, I am not
suggesting that our logs should or will ever do this) I would be using
a much more precise identifier than a user agent, notably, a ASN or IP
address itself.

> If we want proper authentication and communication channels with consumers, we need a robust authentication system—not email addresses embedded in user agents. That approach will inevitably become unwieldy and create trouble.

It is important to not have "perfect become the enemy of good", user
agents with basic identification have worked well in other areas of
tech, and mostly serves to ensure there is some kind of contact path
for a misbehaving client, so things do not have to suffer in silence.

A "proper authentication and communication channels" would likely make
the whole process so complex and cumbersome that people here will
either burn out trying to set it up, or we will just spend the next
set of years trying to define what that even means. The user agent
there is here to provide a path for contacting people, not as an
ironclad authentication mechanism.

Regards
Ben

[Filippo Valsorda]

I think it's important not to think of this as an authentication mechanism. It's a way for well-behaved clients to provide a contact point for the log operator to reach out to in case of innocent misbehavior, and for the log operator to cap the bandwidth of non-malicious but unreachable (and so potentially misbehaving) clients.

To view this discussion visit https://groups.google.com/a/chromium.org/d/msgid/ct-policy/CAL%3D9YSVjoASqw2Xb8r%2BPzXGn4NY3q7woDuNQnRH5yegwDV2iBg%40mail.gmail.com.

[Pierre Barre]

Hi Ben,

I'm not advocating for authentication at all. My point is that requiring an email address in the user agent string is authentication, despite your claim that it isn't. You can't have it both ways - either you're implementing an authentication mechanism (however weak) or you're not.

You are claiming this isn't authentication while simultaneously saying it's meant to identify and establish communication with specific clients.
That's literally what authentication is.

If log operators need to communicate with misbehaving consumers, we already have perfectly good mechanisms for this:

- Return appropriate HTTP error codes (429 for rate limiting, 403 for forbidden, etc.)
- Include descriptive error messages in the response body
- Use standard HTTP headers like Retry-After

These mechanisms are:

1. Already standardised
2. Immediately visible to the consumer
3. Don't require parsing user agent strings
4. Don't create a false authentication layer
5. Actually work in practice

There's no need to implement crazy schemes like embedding contact information in user agents. Any competent monitoring system will log and handle HTTP errors appropriately. If someone is ignoring 429 responses, they're going to ignore your user agent requirements too.

What's more, this approach opens up obvious abuse vectors. What happens when someone starts issuing requests with random email addresses in the user agent string? Or worse, using someone else's email address from multiple IPs across different ASNs? You've just created a mechanism for harassment and false attribution with no way to verify legitimacy.

Also, calling my concerns "ridiculous" isn't helpful. Your argument that you'd use IP addresses or ASNs instead of user agents misses the point - both are equally trivial to change. Anyone can use VPNs, proxies, or cloud providers to get new IPs. These aren't sophisticated fingerprinting techniques, they're just as useless as user agent strings for identifying bad actors.

If you're so adamant that this isn't authentication, then let's commit to that position fully and not implement authentication at all - not even badly.

Best,
Pierre

> https://groups.google.com/a/chromium.org/d/msgid/ct-policy/CAL%3D9YSVjoASqw2Xb8r%2BPzXGn4NY3q7woDuNQnRH5yegwDV2iBg%40mail.gmail.com.

[Pierre Barre]

Hi Filippo,

I'm struggling to understand how this isn't authentication when you're literally describing using identity information to make access control decisions. You say we should "cap the bandwidth" of certain clients based on whether they provide contact info - that's textbook authentication and authorization.

The "non-malicious but unreachable" category doesn't make sense to me. If they're truly non-malicious, why do they need bandwidth capping? And if they're causing problems that require rate limiting, then by definition they're being malicious to your service.

None of this addresses the actual problems I raised:

- Anyone can stick fil...@example.com in their User-Agent and now that person gets harassment emails

- Malicious actors won't use real contact info anyway

- We already have HTTP 429 responses that work perfectly fine

I get that you want some way to contact misbehaving clients, but this approach just doesn't work. It provides zero security benefit while creating new problems.

Can we please either commit to doing real authentication (and have that discussion) or just use the standard HTTP mechanisms that already exist? This halfway approach helps no one.

Best,

Pierre

To view this discussion visit https://groups.google.com/a/chromium.org/d/msgid/ct-policy/a24e3657-1c91-4c6b-b529-06fc718d523d%40app.fastmail.com.

[Ben Cartwright-Cox]

But this is not designed for malicious actors, this is a mechanism
that is designed for well meaning clients who make mistakes (a common
occurrence in the years of deploying similar types of machine readable
interfaces)

If someone maliciously puts random emails in there, then ¯\_(ツ)_/¯ ,
we will email a dead email once, and move on with our lives. We do not
harass users of APIs.

Operationally speaking from different widely used things in my career,
while such a method is not bulletproof, it is simple enough that a
well meaning user who is encouraged to put a contact method down will
do so, and if their client goes wrong, they are typically happy they
get an email about it.

The way I am reading this (kinda low stakes detail?) discussion, is
more an operational experience vs academical viewpoints. I clearly
can't convince you here, but it's also hard to argue against the
(good) things I have first hand witnessed over the last 10+ years of
deploying similar patterns

I don't really think this is a productive discussion at this point (to
be blunt, I can't tell if you are arguing here in bad faith or not),
so this is the last I will be replying on this (Filippo is his own
guy, so he may still)

> To view this discussion visit https://groups.google.com/a/chromium.org/d/msgid/ct-policy/7b63e800-26c5-4fe5-825b-35f420414410%40app.fastmail.com.

[Pierre Barre]

Ben,

First you dismiss my concerns as "ridiculous," now you're questioning whether I'm arguing in bad faith.

It seems what you want is agreement, not discussion. I'm raising legitimate technical concerns about a proposed specification - that's not bad faith, that's exactly what these discussions are for.

Best,
Pierre

[Filippo Valsorda]

Pierre,

I hope you will take this as constructive criticism.

We are here telling you that based on our experience non-malicious but misbehaving clients are a thing, and this is an effective countermeasure. Instead of assuming competence and acknowledging our experience and trying to understand where it diverges from yours, or discussing tradeoffs and iterating on the solution, you are flatly denying it, confidently and categorically stating that it “just doesn't work”, “provides zero security”, and “helps no one”.

Frankly, that’s neither productive nor enjoyable, and it’s a pattern in our interactions. I simply don’t feel the need to engage with it, so although I would prefer to get your input and proceed with consensus, I’m also content with deferring to user agent policymakers.

Filippo

[Pierre Barre]

Filippo,

No technical defense, just "we have experience" and threats to proceed without consensus. Got it.

You asked "How would folks feel about enshrining this as a client SHOULD in c2sp.org/static...@v1.1.0?" When I raised technical concerns, I'm told I'm not being constructive and my input isn't enjoyable.

This is exactly why I suggested going through IETF - your personal spec repository where "decisions are not based on consensus" isn't the right venue for infrastructure that Chrome, Firefox, and Safari might depend on.

The fact that we're debating user agent contact info in 2025 instead of using HTTP status codes is surreal. But rather than explain why existing mechanisms don't work, you dismiss critics as "unproductive" - classic manipulative groupthink where disagreement is reframed as a personality flaw.

At this point, this looks more like a resume-building operation than serious infrastructure work. On the transparency.dev Slack, you're already discussing invoicing companies for SLAs on your log, while simultaneously proposing "the next step after Static CT is an AT Proto Jetstream-like trusted filtered feed" - constant churn, new specs, new complexity, all flowing through your personal projects.

I hope serious players freeze this spec if they ever adopt it. Having core infrastructure depend on someone's personal project where they can add arbitrary requirements while shutting down criticism is untenable.

I understand now that technical input from outside your circle isn't welcome. I'll keep that in mind going forward.

Best,

Pierre

[Filippo Valsorda]

2025-06-23 15:33 GMT+02:00 Pierre Barre <pierre...@barre.sh>:

I hope serious players freeze this spec if they ever adopt it.

Sigh. I'm not addressing the rest, but yes, adopters of C2SP specifications should refer to a fixed version. The Chrome CT Log Policy refers to "c2sp.org/static-ct-api v1.0" already. The Apple one doesn't, and I emailed certificate-tran...@group.apple.com about it.

[Rob Stradling]

> This is exactly why I suggested going through IETF - your personal spec repository where "decisions are not based on consensus" isn't the right venue for infrastructure that Chrome, Firefox, and Safari might depend on.

Pierre,

"Going through IETF" didn't exactly work out well for CTv2 (RFC9162).  It took several years longer [1] than originally anticipated [2] for us to get the document completed and published, by which time CTv1 (RFC6962) was firmly entrenched and CTv2 was "too little, too late".  And along the way, even the original aim of RFC9162 becoming a Standards track RFC had to be abandoned, hence why (like RFC6962 before it) RFC9162 is Experimental.

Please let's not do that again.

Restricting innovation to a snail's pace does not suit this community, in my view.

[1] https://datatracker.ietf.org/doc/html/rfc9162

"Published: December 2021"

[2] https://web.archive.org/web/20180913070728/https://datatracker.ietf.org/wg/trans/about/

"Milestones

Dec 2015    6962bis to IESG"

---

From: ct-p...@chromium.org <ct-p...@chromium.org> on behalf of Pierre Barre <pierre...@barre.sh>
Sent: 23 June 2025 14:33
To: Filippo Valsorda <fil...@ml.filippo.io>; Ben Cartwright-Cox <b...@benjojo.co.uk>
Cc: Certificate Transparency Policy <ct-p...@chromium.org>
Subject: Re: [ct-policy] The Tuscolo Static CT log

 

This Message Is From an External Sender

This message came from outside your organization.

Report Suspicious

 

To view this discussion visit https://groups.google.com/a/chromium.org/d/msgid/ct-policy/3439ee35-7e20-4d55-8cc0-7228b8b83f73%40app.fastmail.com.

[Pierre Barre]

Hi Rob,

Thanks for the context on CTv2. I wasn’t aware of the IETF timeline issues. Fair point.

If IETF is too slow for this community, perhaps we could explore other models? The current approach where specs can be updated like any OSS library using a BDFL governance model is untenable. 

There must be something between IETF’s pace and no process at all.

Best,

Pierre

[Andrew Ayer]

On Mon, 23 Jun 2025 10:28:48 +0200
"Filippo Valsorda" <fil...@ml.filippo.io> wrote:

> How would folks feel about enshrining this as a client SHOULD in
> c2sp.org/static...@v1.1.0?
>
> "Clients SHOULD include a way for the log operator to contact them in
> the User-Agent header, as an email address and/or a URL (with a
> `+https://` prefix), and logs MAY rate-limit anonymous or unreachable
> clients."

I would support this, with some wordsmithing to make it more precise what a log may do:

"Clients SHOULD include a way for the log operator to contact them in the User-Agent header, as an email address and/or a URL (with a `+https://` prefix), and logs MAY rate-limit clients which do not provide a reachable email address and/or reachable URL."

Regards,
Andrew

[Joe DeBlasio - Google]

Hi friends,

Sorry for the delay in chiming in -- today has brought some other CT-related distractions that have taken precedence.

First off, a reminder for everyone to please stay professional and civil. I'd ask that everyone err towards granting grace. ct-policy@ has a pretty good track record recently of keeping the emotional level down, and I'd like to keep it that way. We won't hesitate to step in more aggressively if things really do get off the rails, but, frankly, everyone just be cool.

Regarding the IETF-vs-C2SP-vs-other avenues debate, there was significant discussion over the past few years regarding how we should move forward as a group. Chrome was and is supportive of C2SP for static-ct-api because we felt there was broad (though not universal) consensus that it was worth exploring a model. There was wide recognition that there was value in being more nimble and reactive than venues like the IETF could provide. While I'm very sympathetic to not everyone loving this experiment, the time for arguing that we shouldn't try it at all has passed (at least from Chrome's perspective). While we're committed to experimenting with C2SP, if it turns out that that approach can't meet the needs of the ecosystem, we all can and will adapt. 

Though I don't think describing the C2SP model as "BDFL" is accurate, it is important for Chrome that there is a wider array of maintainers of the static-ct-api spec than there is currently, and I've previously articulated that that's a blocker before Chrome's policy became agnostic to log spec (between RFC6962 and static-ct-api).

Regarding the email address / rate limiting discussion, Chrome's on record saying that we don't like rate limiting (and see it as a form of reduced availability), but acknowledge that it's a necessary evil. I don't think including an email address in a request header is harmful, and wouldn't oppose some form of "SHOULD" language. I agree that it is not, as has been noted, a form of authentication.

Joe

> > To unsubscribe from this group and stop receiving emails from it, send an email to ct-policy+unsubscribe@chromium.org.

> > To view this discussion visit https://groups.google.com/a/chromium.org/d/msgid/ct-policy/02c308a9-e978-4078-af7d-6b7e94b4f815%40app.fastmail.com.

> >

> >

>

> --

> You received this message because you are subscribed to the Google Groups "Certificate Transparency Policy" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to ct-policy+unsubscribe@chromium.org.

> To view this discussion visit https://groups.google.com/a/chromium.org/d/msgid/ct-policy/CAL%3D9YSVjoASqw2Xb8r%2BPzXGn4NY3q7woDuNQnRH5yegwDV2iBg%40mail.gmail.com.

>

>

>

> --

> You received this message because you are subscribed to the Google Groups "Certificate Transparency Policy" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to ct-policy+unsubscribe@chromium.org.

> To view this discussion visit https://groups.google.com/a/chromium.org/d/msgid/ct-policy/a24e3657-1c91-4c6b-b529-06fc718d523d%40app.fastmail.com.

>

>

> --

> You received this message because you are subscribed to the Google Groups "Certificate Transparency Policy" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to ct-policy+unsubscribe@chromium.org.

> To view this discussion visit https://groups.google.com/a/chromium.org/d/msgid/ct-policy/7b63e800-26c5-4fe5-825b-35f420414410%40app.fastmail.com.

--

You received this message because you are subscribed to the Google Groups "Certificate Transparency Policy" group.

To unsubscribe from this group and stop receiving emails from it, send an email to ct-policy+unsubscribe@chromium.org.

[Pierre Barre]

Hi Joe,

Thanks for weighing in. Quick question: How is using identity information to make rate limiting decisions not authentication? That's literally what authentication is - making access control decisions based on identity.

I'm genuinely confused by the distinction being made here.

Best,

Pierre

[Winston de Greef]

Hi Pierre,

You're focusing a lot on that you believe that including an email in the user agent and using that for rate limiting is authentication. It seems that other people in this thread don't feel like authentication is the right word to describe this.

It seems to me that this is arguing about a definition, and I find that this in general isn't helpful. Instead of talking about whether or not it is authentication, I think it's better to just directly talk about the consequences. If this is or is not authentication really doesn't matter. That this might give log operators an easier way to show different log consumers conflicting views is completely disconnected from if this is or isn't authentication. I think by just not talking about whether or not this is authentication, you would communicate your worries more clearly.

I'd like to reiterate that the definitions we use are very arbitrary. To illustrate this point I'd like to provide the following definition from wikipedia:

Authentication is the act of proving an assertion, such as the identity of a computer system user. In contrast with identification, the act of indicating a person or thing's identity, authentication is the process of verifying that identity.

Using this definition, I could say that because the identity is not verified, there is no authentication.

The definition you provide: "making access control decisions based on identity" isn't necessarily more or less correct (depending on the context it might be more or less useful to communicate ideas, which makes it a better or worse definition). Under this definition what's happening here is authentication.

Sincerely,

Winston de Greef

> > To unsubscribe from this group and stop receiving emails from it, send an email to ct-policy+...@chromium.org.

> > To view this discussion visit https://groups.google.com/a/chromium.org/d/msgid/ct-policy/02c308a9-e978-4078-af7d-6b7e94b4f815%40app.fastmail.com.

> >

> >

>

> --

> You received this message because you are subscribed to the Google Groups "Certificate Transparency Policy" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to ct-policy+...@chromium.org.

> To view this discussion visit https://groups.google.com/a/chromium.org/d/msgid/ct-policy/CAL%3D9YSVjoASqw2Xb8r%2BPzXGn4NY3q7woDuNQnRH5yegwDV2iBg%40mail.gmail.com.

>

>

>

> --

> You received this message because you are subscribed to the Google Groups "Certificate Transparency Policy" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to ct-policy+...@chromium.org.

> To view this discussion visit https://groups.google.com/a/chromium.org/d/msgid/ct-policy/a24e3657-1c91-4c6b-b529-06fc718d523d%40app.fastmail.com.

>

>

> --

> You received this message because you are subscribed to the Google Groups "Certificate Transparency Policy" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to ct-policy+...@chromium.org.

> To view this discussion visit https://groups.google.com/a/chromium.org/d/msgid/ct-policy/7b63e800-26c5-4fe5-825b-35f420414410%40app.fastmail.com.

--

You received this message because you are subscribed to the Google Groups "Certificate Transparency Policy" group.

To unsubscribe from this group and stop receiving emails from it, send an email to ct-policy+...@chromium.org.

To view this discussion visit https://groups.google.com/a/chromium.org/d/msgid/ct-policy/3439ee35-7e20-4d55-8cc0-7228b8b83f73%40app.fastmail.com.

--
You received this message because you are subscribed to the Google Groups "Certificate Transparency Policy" group.

To unsubscribe from this group and stop receiving emails from it, send an email to ct-policy+...@chromium.org.
To view this discussion visit https://groups.google.com/a/chromium.org/d/msgid/ct-policy/d4206007-4918-4e60-835b-db2bcce80922%40app.fastmail.com.

[Pierre Barre]

Hi Winston,

I think the distinction matters because if this is just about contact info, then rate limiting shouldn’t depend on it. And if rate limiting does depend on it, then we’re doing access control.

Best,

Pierre

[Winston de Greef]

Hi all,

On the wording to add to the static ct spec:

1. It's a good thing if ct monitors include a way to contact them in the user agent (when they are consuming the logged certificates).

2. Providing ct monitors with a higher rate limit will motivate them to include a way to contact them in the user agent.

Still, using rate limits to encourage this behavior makes me feel a bit uncomfortable. I would like to propose the following two options that I feel are better:

1. Only include the recommendation for clients to include a way to contact them in the user-agent. the static-ct standard doesn't mention rate limiting in it anywhere, so don't explicitly sanction the use of a rate limit to get clients to include this info in the user-agent.

2. Include an opt out email, that logs must handle the same way as an actual email. Bad faith actors can just lie about email addresses, so this doesn't really impact that. This would be a wording like: "... logs MAY rate-limit clients which do not provide a reachable email address (except for no-co...@ct.invalid) and/or reachable URL. If a client includes the email no-co...@ct.invalid, this MUST be treated as any other contact method"

Some other comments on the wording:

Clients SHOULD include a way for the log operator to contact them in the User-Agent header, as an email address and/or a URL (with a `+https://` prefix), and logs MAY rate-limit clients which do not provide a reachable email address and/or reachable URL.

The  "and/or" in "MAY rate-limit clients which do not provide a reachable email address and/or reachable URL" should just be "or".

Also, I think there should be a distinction between consuming the logged certificates (for monitoring) and verifying inclusion. Something like:

# User Agent

When ingesting logged certificates, clients SHOULD include a way for the log operator to contact them in the User-Agent header, as an email address and/or a URL (with a `+https://` prefix), Logs MAY rate-limit clients which do not provide a reachable email address or reachable URL.

Clients that are auditing the log, or are verifying inclusion of SCT SHOULD use the user-agent "static-ct client" to make serving conflicting merkle trees harder

Sincerely,

Winston de Greef

[Filippo Valsorda]

2025-06-23 23:28 GMT+02:00 Winston de Greef <winston...@gmail.com>:

Hi all,

On the wording to add to the static ct spec:

1. It's a good thing if ct monitors include a way to contact them in the user agent (when they are consuming the logged certificates).

2. Providing ct monitors with a higher rate limit will motivate them to include a way to contact them in the user agent.

Still, using rate limits to encourage this behavior makes me feel a bit uncomfortable. I would like to propose the following two options that I feel are better:

1. Only include the recommendation for clients to include a way to contact them in the user-agent. the static-ct standard doesn't mention rate limiting in it anywhere, so don't explicitly sanction the use of a rate limit to get clients to include this info in the user-agent.

2. Include an opt out email, that logs must handle the same way as an actual email. Bad faith actors can just lie about email addresses, so this doesn't really impact that. This would be a wording like: "... logs MAY rate-limit clients which do not provide a reachable email address (except for no-co...@ct.invalid) and/or reachable URL. If a client includes the email no-co...@ct.invalid, this MUST be treated as any other contact method"

Hi Winston, thank you for your feedback!

I want to mention that the rate limit is not just a stick or carrot to encourage providing contact information, but a way to protect well-behaved clients from non-malicious misbehaving ones.

For example, imagine two clients that don't implement Retry-After or exponential backoff, and cause thundering herd problems. One of them provides a contact in the User-Agent, so after noticing the first time we can reach out and ask them to fix it, or send a PR. The other doesn't, so remains unfixed. The rate limit will naturally cap the impact of the unfixed one, letting the fixed one through while the log recovers.

However, you make a very good point that the spec doesn't mention rate limiting and maybe shouldn't start now. I think it's a good idea to keep rate limiting choices in the realm of UA policies and log operational details. The spec just needs to encourage and allow the behavior we want to see from clients. Logs then take responsibility for the rate limiting compromises they choose to make.

That makes the text simply "Clients SHOULD include a way for the log operator to contact them in the User-Agent header, as an email address and/or a URL (with a `+https://` prefix)."

Some other comments on the wording:

Clients SHOULD include a way for the log operator to contact them in the User-Agent header, as an email address and/or a URL (with a `+https://` prefix), and logs MAY rate-limit clients which do not provide a reachable email address and/or reachable URL.

The  "and/or" in "MAY rate-limit clients which do not provide a reachable email address and/or reachable URL" should just be "or".

Also, I think there should be a distinction between consuming the logged certificates (for monitoring) and verifying inclusion. Something like:

# User Agent

When ingesting logged certificates, clients SHOULD include a way for the log operator to contact them in the User-Agent header, as an email address and/or a URL (with a `+https://` prefix), Logs MAY rate-limit clients which do not provide a reachable email address or reachable URL.

Clients that are auditing the log, or are verifying inclusion of SCT SHOULD use the user-agent "static-ct client" to make serving conflicting merkle trees harder

I am less convinced by this. First, because having clients directly audit SCTs against a log has privacy implications. Second, because I want to avoid any fingerprinting cat-and-mouse games. Third, because a log serving split views to monitors is just as bad as serving them to clients IMHO.

SCT inclusion checks are a complicated topic, but I think the best answer we have in CT so far is to request inclusion proofs from trusted monitors, who ideally gossip STHs between them. (This is not at all optimal, which is why I believe in the witness ecosystem.)

[Pierre Barre]

Hi Filippo,

Once you "fix" the client with contact info, how do you distinguish it from other clients claiming the same identity? User-Agent strings aren't unique identifiers,  and emails are not secrets. Anyone can claim to be "WellBehaved/1.0 (ad...@example.com)".

At scale, misbehaving clients will just copy well-known User-Agents with contact info - the same way scrapers pretend to be Googlebot today.

I suppose you could add unique tokens to the User-Agent, though at that point we'd be - actually, never mind.

Best,

Pierre

To view this discussion visit https://groups.google.com/a/chromium.org/d/msgid/ct-policy/6414934a-792f-4f89-a3de-4cb1bfc16a47%40app.fastmail.com.

[Clint Wilson]

Hi Filippo,

The Apple CT Log Program (https://support.apple.com/103703) has been updated to refer to v1.0.0 of the specification.

Thank you!

-Clint

--
You received this message because you are subscribed to the Google Groups "Certificate Transparency Policy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ct-policy+...@chromium.org.

To view this discussion visit https://groups.google.com/a/chromium.org/d/msgid/ct-policy/50a89a87-f810-456e-bed7-af9359e8df20%40app.fastmail.com.

[Matt Palmer]

On Mon, Jun 23, 2025 at 10:28:48AM +0200, Filippo Valsorda wrote:
> How would folks feel about enshrining this as a client SHOULD in
> c2sp.org/static...@v1.1.0?
>
> "Clients SHOULD include a way for the log operator to contact them in
> the User-Agent header, as an email address and/or a URL (with a
> `+https://` prefix), and logs MAY rate-limit anonymous or unreachable
> clients."

I do not feel good about the rate-limiting of clients which choose to
not provide contact information. Certificate Transparency information
is intended to be widely accessible, and any deliberate policy that
restricts that must be, prima facie, viewed with the deepest skepticism.

The argument that log operators need to have contact information for
clients so that they can communicate issues with the client
implementation ignores the fact that there is already a mechanism for
communicating with the client: HTTP status codes, and error messages.
Log operators would already need to use this mechanism to communicate
the fact that a client is being denied service for lack of contact
details (rather than for some other reason), thus it seems reasonable to
use that same mechanism to communicate other reasons for denying
service. Hence, contact information is not necessary for the argued
purpose.

- Matt

[Winston de Greef]

Hi all,

I wanted to share some thoughts I had about some of the topics mentioned in this email chain,

1. Filipo brings up that the rate limit isn't just a purely carrot/stick to encourage providing contact information. However, I feel like it is solely the responsibility of the log operator to ensure the log remains available. It feels like a false dichotomy to say: choose between more restrictions on non-malicious misbehaving log clients or lower availability. To me the natural answer is that the log operator should increase capacity.
2. This idea was originally mentioned to me by Pierre (thanks!): if a few non-malicious misbehaving clients can significantly impact the ability for a log to remain available, that implies that said log is woefully incapable of handling a malicious attack.
3. Imagine a hypothetical client that requests the most recent partial static-ct log twice, instead of the partial tile and the full tile. This is wasting resources, however, it seems silly to restrict their access to the log by returning error pages for some (or all) of those requests. Having an email to send a bug report to is clearly better.
4. Matt brings up:

1. Certificate Transparency information is intended to be widely accessible, and any deliberate policy that restricts that must be, prima facie, viewed with the deepest skepticism

1. This is something I agree with, but I feel that the decision to not make a change should be equally viewed with deep skepticism. While I don't find this argument convincing myself (see point 1), we should also consider something like:
1.  Not rate limiting clients without contact information makes it harder for log operators to inform them about bugs that waste resources.
2. This leads to these bugs not being fixed (or being mixed more slowly).
3. This increases strain on log operators.
4. This increases the chance that the log becomes less available for everyone, even well behaving clients that include contact information.
5. Deciding not to implement these rate limits might restrict log access for everyone, which is way worse (if it happens, or you might say the expected value of the damage is higher) than that some clients that don't want to include contact information/aren't aware of this requirement getting a lower rate limit

Sincerely,

Winston de Greef

--
You received this message because you are subscribed to the Google Groups "Certificate Transparency Policy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ct-policy+...@chromium.org.

To view this discussion visit https://groups.google.com/a/chromium.org/d/msgid/ct-policy/473f9b4e-c5e6-4410-b738-89a1aefd388f%40mtasv.net.

[Matt Palmer]

On Thu, Jun 26, 2025 at 08:07:25PM -0400, Winston de Greef wrote:
> 4. Matt brings up:

>
> > Certificate Transparency information is intended to be widely
> > accessible, and any deliberate policy that restricts that must be, prima
> > facie, viewed with the deepest skepticism
>
> This is something I agree with, but I feel that the decision to not make
> a change should be equally viewed with deep skepticism.
>
> While I don't find
> this argument convincing myself (see point 1), we should also consider
> something like:

> 1. Not rate limiting clients without contact information makes it

> harder for log operators to inform them about bugs that waste resources.

I don't believe that it has been suggested that any current protocol
specification or policy contains a complete prohibition on rate
limiting. Indeed, current practice amongst both RFC6962 and static-ct
log operators appears to be to rate limit early and rate limit often,
implying that log operators do not believe that rate limiting is
prohibited.

I also don't recall anyone arguing that rate limiting MUST NOT occur.
Indeed, such a prohibition would be ludicrous, given that resources are
always finite.

As such, your hypothesised consequence does not follow:

> 2. This leads to these bugs not being fixed (or being mixed more
> slowly).

because "misbehaving" clients (however you might wish to define
"misbehaving") MAY be rate limited or otherwise denied service under
current policies and protocol specification. Subsequently, the rest of
your hypothetical consequences are not shown to follow, either.

- Matt

[K. York]

One important question to consider here: are we talking about contact information for the client operator or for the author of the client implementation code?

If an expectation is set for the latter, some Good Old User Agent Regex /CT Monitor/ && /\+https:/ could suffice and is exactly how browser user agents are treated in the real world.

~Kane

Sent with Shortwave

To view this discussion visit https://groups.google.com/a/chromium.org/d/msgid/ct-policy/CAD2RJqaTUANpobB0h3mmX1feOVZD4%2BwFjrsH%2B8eEzHBMVYnf9Q%40mail.gmail.com.

[Winston de Greef]

Hi Matt,

I don't believe that it has been suggested that any current protocol specification or policy contains a complete prohibition on rate limiting

Yep. What I meant to say was: the current situation is that log operators will have the same rate limits for clients that include contact information. You say (Matt) that a decision to limit access by setting lower rate limits for clients without contact information should be viewed with a lot of skepticism. The idea I was trying to communicate is that just continuing with the status quo could result in a situation that is actually more limiting for clients.

because "misbehaving" clients (however you might wish to define "misbehaving") MAY be rate limited or otherwise denied service under current policies and protocol specification.  Subsequently, the rest of your hypothetical consequences are not shown to follow, either.

I understand under misbehaving essentially that clients are using more log resources than reasonably necessary for their use case. (ie a monitor requesting every certificate twice).

Also I'd say that yes I agree with you, misbehaving logs could receive stricter rate limits, and that might work. (I did say that I wasn't convinced by this argument). But, if a client notices that they are getting lower rate limits, they might decide to start using multiple ips to evade the rate limit. (Or maybe if there is a specific error message, they will realize their mistake and fix their client.) But this is essentially Filippo's reason for implementing the rate limit, so at least one person thinks this is a pretty good argument.

Sincerely,

Winston de Greef

[Winston de Greef]

Hi Kane,

As far as I can tell, Tuscolo only requires a valid email. Maybe they will look at emails like c...@example.com and say well that doesn't fit the requirements we clearly communicated, so this user agent gets the rate limit (or maybe they apply some logic like this automatically).

However, it is in the monitor's best interest for log operators to contact them. They don't want to waste their own resources making useless requests. I think the best approach is to include the name of the software, a version number, contact information for the software maintainer, and contact information for the operator. (If the software used isn't known, probably a way to contact the maintainers as well.)

Something like:

User-Agent: Cert Spotter v0.20.1 <+https://github.com/SSLMate/certspotter>, operated by winston...@gmail.com

Or:

User-Agent: Winston's custom client v0.1, maintained and operated by winston...@gmail.com

This makes it possible for a log operator to see if the client is on the most recent version of the software, and file a bug report if it is. If the client isn't on the most recent version, the log operator can contact the client operator and ask them to update (if the bug is fixed in the newest version). Including a version number also makes it easier for an operator to see when you might have fixed the bug.

Sincerely,

Winston de Greef

[Pierre Barre]

Hi Winston,

Thanks for your analysis. 

Quick math on capacity: A 1.5B entry log is ~1.5TB total (~5.9M tile requests). At 10Gb/s bandwidth, that's ~4,880 tiles/second. Over 6 months, you have ~19.7PB total capacity - enough for thousands of monitors doing full syncs plus ongoing updates.

The real issue: Why would legitimate monitors "fetch the same random tiles over and over"? That's not monitoring, that's broken behavior.

Smart rate limiting should target actual abuse:

```

Fetching unchanged tiles repeatedly -> 429 + "Use If-Modified-Since"

Ignoring caching headers -> Rate limit

Random access patterns -> Not legitimate monitoring

```

This protects resources based on behavior, not identity. User-Agent emails are trivially spoofed and won't stop actual attacks.

If your log can't handle legitimate monitoring traffic without requiring email addresses, you need better infrastructure, not authentication theater.

Best,

Pierre

To view this discussion visit https://groups.google.com/a/chromium.org/d/msgid/ct-policy/CAD2RJqaTUANpobB0h3mmX1feOVZD4%2BwFjrsH%2B8eEzHBMVYnf9Q%40mail.gmail.com.

[Filippo Valsorda]

2025-06-27 03:40 GMT+02:00 K. York <kane...@gmail.com>:

One important question to consider here: are we talking about contact information for the client operator or for the author of the client implementation code?

If an expectation is set for the latter, some Good Old User Agent Regex /CT Monitor/ && /\+https:/ could suffice and is exactly how browser user agents are treated in the real world.

Yes, Skylight is open source, so anyone can see that we consider not anonymous any client with a @, a +https://, or even a github.com in the User-Agent.

https://github.com/FiloSottile/sunlight/blob/244b8b1a10/cmd/skylight/main.go#L174-L184

This is a very common courtesy of automated clients, which makes it easier to deal with non-malicious clients that cause trouble. We also know how to deal with malicious clients if necessary, but that requires a lot more time and effort which we are hoping to be able to dedicate to more productive tasks.

To view this discussion visit https://groups.google.com/a/chromium.org/d/msgid/ct-policy/7ec9302d-f62e-4b59-8110-b4e6be8a9137%40mail.shortwave.com.

[Pierre Barre]

So despite insisting throughout this thread that this isn't authentication, you've implemented a system that checks for identity markers (@, +https://, github.com) to make access control decisions?

That's literally authentication - just incredibly weak authentication that's defeated by a single character.

Your comment about dealing with malicious clients requiring "a lot more time and effort which we are hoping to be able to dedicate to more productive tasks" is concerning. This is infrastructure that major browsers may depend on - security isn't optional overhead, it's the job.

```

curl -H "User-Agent: MyMonitor/1.0" https://ct.example.com/ # Rate limited?

curl -H "User-Agent: MyMonitor/1.0 @" https://ct.example.com/  # Full access?

```

This approach might have been common practice in the early 2000s, but I haven't seen a production system in the past decade that relies on User-Agent decoration for access control. Modern systems use behavior-based rate limiting for good reason.

Best,

Pierre

To view this discussion visit https://groups.google.com/a/chromium.org/d/msgid/ct-policy/9a2a613f-ec3a-4d07-af5f-4251f0ec41b0%40app.fastmail.com.

[Pierre Barre]

P.S. - Your insistence on this User-Agent approach is starting to make sense. You COULD implement proper behavior-based rate limiting with the static CT API (that's not that complicated for CT, we know the access patterns) - but that would require additional infrastructure like reverse proxies or edge computing to analyze request patterns. That would make your system more complex, slower, harder to operate, and more expensive.

And here's the kicker: it would remove the only purported advantage of the static CT API - that you only need a "normal" web server to serve the read path. Once you add the infrastructure needed for real security, low operating costs and performance you've lost your entire architectural selling point.

So you're stuck defending "@" as a security control because admitting you need real rate limiting would mean admitting the static API design's core selling points (simplicity, speed, low operational cost) - which are already doubtful, see the benchmarks I published a few days back - are incompatible with basic security requirements.

You're literally choosing "security by @" over proper infrastructure because it would undermine your architectural narrative. That's not engineering, it's ideology.

Best,

Pierre

To view this discussion visit https://groups.google.com/a/chromium.org/d/msgid/ct-policy/70fc1792-a575-44eb-80cf-5db890da5ca3%40app.fastmail.com.

[Pierre Barre]

Hi all,

Here's behavioral rate limiting for static CT using nothing but nginx configuration:

https://gist.github.com/Barre/3903d2195546ad21a4ac71309d028bf6

It's 80 lines of config that:

- Tracks clients repeatedly fetching tiles without If-Modified-Since/If-None-Match

- Rate limits checkpoint spam (>1/minute)

- Returns proper 429s with Retry-After

This protects against the exact misbehaviors described earlier. No email addresses, no User-Agent parsing, no "@" symbols. Just actual rate limiting based on actual behavior.

Note that this demonstrates static CT requires a full-featured web server in production - you can't safely serve it from S3 or a minimal static file server without risking bankruptcy from misbehaving clients. Once you need nginx with rate limiting modules, the operational complexity is similar to RFC6962...

I hope this will settle the authentication-by-punctuation debate... :)

Best,

Pierre

To view this discussion visit https://groups.google.com/a/chromium.org/d/msgid/ct-policy/213f243d-e7d3-4974-bd44-f37252985b84%40app.fastmail.com.

[Joe DeBlasio - Google]

Pierre,

Much of this message is ad hominem, and several of your preceding messages are also drifting far from treating others with the respect expected in this forum.

If you are not able to participate respectfully, you will lose the ability to post to ct-policy@.  Expressing and defending a differing opinion is welcome; being rude, or attacking people rather than their ideas, is not. This is your one and only direct warning.

Joe

~Kane

Sent with Shortwave

To unsubscribe from this group and stop receiving emails from it, send an email to ct-policy+unsubscribe@chromium.org.

To view this discussion visit https://groups.google.com/a/chromium.org/d/msgid/ct-policy/473f9b4e-c5e6-4410-b738-89a1aefd388f%40mtasv.net.

--

You received this message because you are subscribed to the Google Groups "Certificate Transparency Policy" group.

To unsubscribe from this group and stop receiving emails from it, send an email to ct-policy+unsubscribe@chromium.org.

To view this discussion visit https://groups.google.com/a/chromium.org/d/msgid/ct-policy/CAD2RJqaTUANpobB0h3mmX1feOVZD4%2BwFjrsH%2B8eEzHBMVYnf9Q%40mail.gmail.com.

--

You received this message because you are subscribed to the Google Groups "Certificate Transparency Policy" group.

To unsubscribe from this group and stop receiving emails from it, send an email to ct-policy+unsubscribe@chromium.org.

To view this discussion visit https://groups.google.com/a/chromium.org/d/msgid/ct-policy/7ec9302d-f62e-4b59-8110-b4e6be8a9137%40mail.shortwave.com.

--

You received this message because you are subscribed to the Google Groups "Certificate Transparency Policy" group.

To unsubscribe from this group and stop receiving emails from it, send an email to ct-policy+unsubscribe@chromium.org.

To view this discussion visit https://groups.google.com/a/chromium.org/d/msgid/ct-policy/9a2a613f-ec3a-4d07-af5f-4251f0ec41b0%40app.fastmail.com.

--

You received this message because you are subscribed to the Google Groups "Certificate Transparency Policy" group.

To unsubscribe from this group and stop receiving emails from it, send an email to ct-policy+unsubscribe@chromium.org.

[Pierre Barre]

Joe,

When I was called "ridiculous" and accused of arguing in "bad faith," there was no moderation. Now that I've provided:

- Working nginx rate limiting config instead of "security by @"

- A full dual CT implementation that massively outperforms existing implementations

- Comprehensive benchmarks with hard performance data

- Technical proposals for improving the ecosystem

I'm threatened with a ban.

What's disrespectful? Building better implementations? Showing performance data? Providing working solutions? Pointing out that using "@" for security decisions is inadequate?

Or is the real issue that I worked autonomously without asking permission from the "in group" first? That I didn't coordinate with the established players before demonstrating better approaches? Perhaps I should have asked for approval before my benchmarks showed superior performance?

I am severely disappointed in Google's position to shut down technical criticism. Your reaction is completely out of place. This looks like shutting down technical criticism rather than addressing it. The "school teacher" tone when confronted with inconvenient technical facts doesn't help either.

I'll leave and continue working with groups that value technical merit instead of requiring political pre-approval for innovation. Your email will not look the way you think it looks for anyone.

Bye!

Best,

Pierre

[Matt Palmer]

On Thu, Jun 26, 2025 at 09:55:47PM -0400, Winston de Greef wrote:
> Hi Matt,
>
> I don't believe that it has been suggested that any current protocol
> > specification or policy contains a complete prohibition on rate limiting
>
> Yep. What I meant to say was: the current situation is that log operators
> will have the same rate limits for clients that include contact
> information. You say (Matt) that a decision to limit access by setting
> lower rate limits for clients without contact information should be viewed
> with a lot of skepticism. The idea I was trying to communicate is that just
> continuing with the status quo could result in a situation that is actually
> more limiting for clients.

I don't see how that follows. In the status quo, clients that operate
in a manner which does not unduly impact the ability of a log will not
be rate-limited, while clients that do will be.

- Matt

[Winston de Greef]

Hi Matt,

This is clearly not true. I've seen multiple clients complaining about Sectigo's rate limits being way too low for a well behaving monitor to keep up (Also my argument doesn't depend on rate limits to be high enough in general, because getting it lowered to ie 1 req/s once a month still impacts log availability). Also I'd like to reiterate that this argument is not necessarily a very good argument (I'm not convinced!) but I want to express that "any deliberate policy that restricts that must be, prima facie, viewed with the deepest skepticism" is a bar that is too high. (Or alternatively that staying with the status quo must be viewed with the same skepticism), because holding on to the status quo just because it explicitly limits clients could result in a worse situation.

 So what I'm saying is that while I think this isn't a particularly convincing argument:

1.  Not rate limiting clients without contact information makes it harder for log operators to inform them about bugs that waste resources.
2. This leads to these bugs not being fixed (or being mixed more slowly).
3. This increases strain on log operators.
4. This increases the chance that the log becomes less available for everyone, even well behaving clients that include contact information.

1. Deciding not to implement these rate limits might restrict log access for everyone, which is way worse (if it happens, or you might say the expected value of the damage is higher) than that some clients that don't want to include contact information/aren't aware of this requirement getting a lower rate limit.

Something like this might not pass the bar you provide, but I think should still be applied:

1. There is something that limits log clients, but increases log availability. (this is instead of points 1 through 4 in the previous list).
2. This should still be implemented, because the limit this places on clients is less than the limit of worse log availability because this isn't applied. (Of course this depends on the limit and how much it improves availability, but I believe that log availability is very important to clients. So if including emails in user agents increases log availability by 5%, this clearly outways the drag of putting an email in the user agent.)

Sincerely,

Winston de Greef

--
You received this message because you are subscribed to the Google Groups "Certificate Transparency Policy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ct-policy+...@chromium.org.

To view this discussion visit https://groups.google.com/a/chromium.org/d/msgid/ct-policy/4e4ec040-225f-4f79-98bd-4f5efe459604%40mtasv.net.

[Winston de Greef]

Oops: ".... because it explicitly limits clients ..." should be ".... because it doesn't explicitly limit clients ..."

Sincerely,

Winston de Greef

[Matt Palmer]

On Fri, Jun 27, 2025 at 10:54:25PM -0400, Winston de Greef wrote:
> This is clearly not true. I've seen multiple clients complaining about

> Sectigo's rate limits being way too low for ***a well behaving monitor to keep
> up*** [Emphasis added by MP]

Exactly -- Sectigo's problems have _nothing_ to do with not being able to
contact misbehaving clients to tell them what they're doing wrong. As
such, it's not relevant to a discussion about a possible policy or
specification change that would enable providing differential service
based on whether or not a misbehaving client can be contacted via email.

> I want to
> express that "any deliberate policy that restricts that must be, prima
> facie, viewed with the deepest skepticism" is a bar that is too high.

We're going to have to agree to disagree on that, then. The entire
security of CT is predicated on a diverse population of monitors being
able to freely access the log contents. Thus, anything that impacts
availability directly impacts the broader security guarantees of CT,
hence my "deepest skepticism".

> (Or
> alternatively that staying with the status quo must be viewed with the same
> skepticism), because holding on to the status quo just because it
> explicitly limits clients could result in a worse situation.

That seems to be largely a restatement of the Politicians' Syllogism:
"Something must be done. This is something. Therefore we must do it."

> So what I'm saying is that while I think this isn't a particularly
> convincing argument:
>

> 1. Not rate limiting clients without contact information makes it

> harder for log operators to inform them about bugs that waste resources.

> 2. This leads to these bugs not being fixed (or being mixed more slowly).
> 3. This increases strain on log operators.
> 4. This increases the chance that the log becomes less available for

> everyone, even well behaving clients that include contact information.

> 5. Deciding not to implement these rate limits might restrict log access

> for everyone, which is way worse (if it happens, or you might say the
> expected value of the damage is higher) than that some clients that don't
> want to include contact information/aren't aware of this
> requirement getting a lower rate limit.
>
> Something like this might not pass the bar you provide, but I think should
> still be applied:
>

> 1. There is something that limits log clients, but increases log

> availability. (this is instead of points 1 through 4 in the previous list).

For any log that complies with the Chrome CT Log Policy -- specifically,
"Rate limiting is a form of reduced log availability" -- there is, in
fact, _not_ "something that limits log clients, but increases log
availability".

Even absent that policy, though, "something that limits log clients, but
increases log availability" is unlikely to exist. The most that
limiting some log clients could do is to make resources available to
serve other clients.

> 2. This should still be implemented, because the limit this places on

> clients is less than the limit of worse log availability because this isn't
> applied. (Of course this depends on the limit and how much it improves
> availability, but I believe that log availability is very important to
> clients. So if including emails in user agents increases log availability
> by 5%, this clearly outways the drag of putting an email in the user agent.)

Comparing "5% more availability" with "putting an email in the user
agent" is invalid. Putting an email address in a user agent
does not, _in and of itself_, provide a single extra CPU cycle or bit of
network bandwidth with which to serve clients (in fact, as it increases
the size of all requests, it ever-so-slightly does the opposite).

Rather, what including emails in user agents allows, relative to
"something that limits log clients", is providing differential
rate-limiting. Such rate-limiting might increase availability for a
_subset_ of well-behaved clients (those which provide emails) by 5%. It
would, however, necessarily _degrade_ availability for those
well-behaved clients that don't provide emails, by an equal-or-greater
amount.

One might consider trying to argue that "non-email-providing clients
that are sufficiently well-behaved won't be rate-limited", but if a log
has enough resources to serve all well-behaved clients without
rate-limiting, you don't need this policy in the first place. It only
provides value when rate-limiting is necessary, by providing a mechanism
for discriminating between clients based on something other than their
actual impact on the log's availability, and that is something I do not
believe is warranted.

- Matt

[Winston de Greef]

Hi Matt,

This is clearly not true. I've seen multiple clients complaining about Sectigo's rate limits being way too low for a well behaving monitor to keep up. 

Exactly -- Sectigo's problems have _nothing_ to do with not being able to contact misbehaving clients to tell them what they're doing wrong.  As such, it's not relevant to a discussion about a possible policy or specification change that would enable providing differential service based on whether or not a misbehaving client can be contacted via email.

But would Sectigo be able to implement a higher rate limit if there was contact information? The example argument I've provided would result in the answer being yes, so it is relevant.

We're going to have to agree to disagree on that, then.  The entire security of CT is predicated on a diverse population of monitors being able to freely access the log contents.  Thus, anything that impacts availability directly impacts the broader security guarantees of CT, hence my "deepest skepticism".

As I understand it, the security benefits of CT comes from the following two ideas:

1. There is a thriving ecosystem of auditors that gossip to each other and to clients that want to verify SCTs (ie browsers) so that clients that verify SCTs can know that someone has taken a close look at the certificate..
2. There is a thriving ecosystem of monitors that see certificates as soon as possible, and can raise the alarm if they find a misissued certificate.

Auditors and clients verifying SCTs will not hit a rate limit of 75 requests per second. Monitors can better do their job if they have stable and fast access to issued certificates (or pre-certificates). Filipo's proposed contact-information based rate limit does not impact auditors, and would be a benefit (if the example argument is to be believed) for monitors, because they get a more stable log. (A more stable log is worth a lot to a monitor, because that directly impacts how well they can look for misissued certificates, and they would certainly be willing to trade the drag of having to include contact information in the user agent for better log availability.)

I feel like this example presents a situation where increasing explicit limitations "you must include contact information if you want a higher rate limit" while reducing implicit limitations "log availability" has a net positive on the security benefits of CT.

Maybe more specifically: this policy "impacts availability" by making it a bit harder for monitors to set up access, but increases log availability with increased stability.

 

That seems to be largely a restatement of the Politicians' Syllogism: "Something must be done.  This is something.  Therefore we must do it."

This feels like a significant mischaracterization of my line of reasoning. I am not saying that something must be done, just that even though a change has downsides, it can lead to a situation that is net better.

there is, in fact, _not_ "something that limits log clients, but increases log availability".

and 

"something that limits log clients, but increases log availability" is unlikely to exist.

Well, this might be true. I can't say I've come across anything that I'm convinced does this. But the point of my reasoning is that if something like this were to exist, it would be a good idea to apply it (I tried to communicate this by starting with an assertion that there is something like this. If there is something like this, then we should do ...) However, it seems to me that your bar of "deepest skepticism" would result in the conclusion that it's best not to apply this hypothetical improvement. Therefore, I don't think you are using the right bar.

The most that limiting some log clients could do is to make resources available to serve other clients.

I don't think these limits are a zero sum game: if a log requires clients to include contact information for a higher rate limit, this places a limit on the clients. If however, this means that the log operator can prevent resources from being wasted by contacting a client and convince them to fix a bug causing misbehavior, this is net positive.

Another possible situation that leads to a positive result is that a log operator can achieve better performance when get-entries requests are made with start % 64 == 0 and end=start+64. If the log operator configures a higher rate limit for requests that match that restriction, this increases the amount of requests all monitors can make.

Comparing "5% more availability" with "putting an email in the user agent" is invalid.  Putting an email address in a user agent does not, _in and of itself_, provide a single extra CPU cycle or bit of network bandwidth with which to serve clients (in fact, as it increases the size of all requests, it ever-so-slightly does the opposite).

Rather, what including emails in user agents allows, relative to "something that limits log clients", is providing differential rate-limiting.  Such rate-limiting might increase availability for a _subset_ of well-behaved clients (those which provide emails) by 5%.  It would, however, necessarily _degrade_ availability for those well-behaved clients that don't provide emails, by an equal-or-greater amount.

I describe a hypothetical situation wherein including an email actually does result in increasing availability by 5%. It doesn't actually matter where this improvement comes from. (Maybe in this hypothetical situation including an email in the user agent does magically free up cpu-cycles.) My point is that if including emails does this, monitors would want it implemented.

Also, in a less magical world, there is a mechanism where including an email improves availability beyond just shuffling capacity around. Namely, that log operators can contact monitors and ask them to fix a bug in their implementation. This reduces the use of the log's resources by this monitor, leaving more resources for the other monitors (or more buffer resources), increasing availability.

I'd also like to reiterate that I don't believe that the example I keep referring to (requiring contact info in user agents) is a magic example for which my assertions hold (This is something that limits log clients, but increases log availability.) It's just an example I use in the hope that I can illustrate why there might be something that could have this result. (The mechanism for how it could increase log availability exists! And, I think it has a non-zero effect. However, I think there are other factors that make the non-zero impact negligible.) I also don't think that just dismissing this idea (or something like it) because it limits log accessibility isn't the right choice.

Now that I'm writing the end of this email, I realize that static-ct could also be an example. It limits log availability, because now monitors have to implement two APIs instead of one (a significantly larger burden than including an email address in the user agent), but it's been implemented because it's easier to host (at least that's the idea) and therefore increases log availability. Maybe you'll find the reasonings I give more convincing if you replace my user-agent based examples with equivalent static-ct based examples.

 

Sincerely,

Winston de Greef

--
You received this message because you are subscribed to the Google Groups "Certificate Transparency Policy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ct-policy+...@chromium.org.

To view this discussion visit https://groups.google.com/a/chromium.org/d/msgid/ct-policy/e0da8686-14b2-4a9e-b9d3-a0e1934740db%40mtasv.net.

[Winston de Greef]

Hi,

Just noticed some mistakes in my email I wanted to correct:

"something that limits log clients, but increases log availability" is unlikely to exist.

Well, this might be true. I can't say I've come across anything that I'm convinced does this.

I forgot to update this after realising that static-ct is a good example. Static ct limits log clients, but provides increased log availability. Tuscolo Static CT log claims to provide access with no rate limits to well behaving clients with contact information, something which as far as I know no other logs are promising.

Another possible situation that leads to a positive result is that a log operator can achieve better performance when get-entries requests are made with start % 64 == 0 and end=start+64.

This is a hypothetical situation. There should be an if here: ... is that if a log operator can achieve better...

I also don't think that just dismissing this idea (or something like it) because it limits log accessibility isn't the right choice.

Double negative here when there should only be one. I forgot to change "isn't" to "is" while editing my email. It should be: ... limits log accessibility is the right choice.

Sincerely,

Winston de Greef

[Matt Palmer]

On Sat, Jun 28, 2025 at 02:59:18AM -0400, Winston de Greef wrote:
> This is clearly not true. I've seen multiple clients complaining about

> >> Sectigo's rate limits being way too low for a well behaving monitor to keep
> >> up.
> >
> > Exactly -- Sectigo's problems have _nothing_ to do with not being able to
> > contact misbehaving clients to tell them what they're doing wrong. As
> > such, it's not relevant to a discussion about a possible policy or
> > specification change that would enable providing differential service based
> > on whether or not a misbehaving client can be contacted via email.
>
> But would Sectigo be able to implement a higher rate limit if there was
> contact information?

No.

Sectigo's problem isn't misbehaving clients, it's just having
insufficient capacity and/or a poor implementation, such that the logs
are unable to support the existing load of the well-behaved ecosystem.
(This statement is based on my interpretation of information given by
Sectigo representatives in this group; if my characterisation is
incorrect, I'd appreciate clarification from a Sectigo representative)

> The example argument I've provided would result in the
> answer being yes, so it is relevant.

Your "example argument" relating to emails-in-user-agents is, by your
own admission, "magic":

> where this improvement comes from. (Maybe in this hypothetical situation
> including an email in the user agent does magically free up cpu-cycles.) My

The statement "would Sectigo be able to implement a higher rate limit if
there was magic?" is, indeed, likely to be true, but absent evidence
supporting the existence of said magic, it seems a stretch to call such
a statement "relevant".

> We're going to have to agree to disagree on that, then. The entire
> > security of CT is predicated on a diverse population of monitors being able
> > to freely access the log contents. Thus, anything that impacts
> > availability directly impacts the broader security guarantees of CT, hence
> > my "deepest skepticism".
>
> As I understand it, the security benefits of CT comes from the following
> two ideas:
>

> 1. There is a thriving ecosystem of auditors that gossip to each

> other and to clients that want to verify SCTs (ie browsers) so that clients
> that verify SCTs can know that someone has taken a close look at
> the certificate..

> 2. There is a thriving ecosystem of monitors that see certificates as

> soon as possible, and can raise the alarm if they find a misissued
> certificate.
>
> Auditors and clients verifying SCTs will not hit a rate limit of 75
> requests per second.

Objection m'lud, assertion not in evidence.

> Monitors can better do their job if they have stable
> and fast access to issued certificates (or pre-certificates).

Stipulated.

> Filipo's
> proposed contact-information based rate limit does not impact auditors,

Assertion not in evidence.

> and
> would be a benefit (if the example argument is to be believed)

It's an argument whose existence is predicated on "magic", so you'll
hopefully forgive me if I don't uncritically accept it as a given.

> I feel like this example presents a situation where increasing explicit
> limitations "you must include contact information if you want a higher rate
> limit" while reducing implicit limitations "log availability" has a net
> positive on the security benefits of CT.

I disagree with the premise of the assertion, and hence do not accept
the conclusion.

> Maybe more specifically: this policy "impacts availability" by making it a
> bit harder for monitors to set up access, but increases log availability
> with increased stability.

It does not increase log availability.

> > "something that limits log clients, but increases log availability" is
> > unlikely to exist.
>
> Well, this might be true. I can't say I've come across anything that I'm
> convinced does this. But the point of my reasoning is that if something
> like this were to exist, it would be a good idea to apply it (I tried to
> communicate this by starting with an assertion that there is something like
> this. If there is something like this, then we should do ...) However, it
> seems to me that your bar of "deepest skepticism" would result in the
> conclusion that it's best not to apply this hypothetical improvement.
> Therefore, I don't think you are using the right bar.

No, "deepest skepticism" would not result in a hypothetical improvement
which was actually shown to be an improvement from being rejected,
merely rejecting hypothetical improvements which did not have evidence
of improvement.

> The most that limiting some log clients could do is to make resources
> > available to serve other clients.
>
> I don't think these limits are a zero sum game: if a log requires clients
> to include contact information for a higher rate limit, this places a limit
> on the clients. If however, this means that the log operator can prevent
> resources from being wasted by contacting a client and convince them to fix
> a bug causing misbehavior, this is net positive.

This is a different mechanism to the one you previously argued for,
where reducing availability to some clients by X% somehow allows the log
to magically increase availability to other clients by X+N%.

Enabling communication between logs and clients allows the system to run
with feedback, an essential part of producing a stable operating point.
You won't get any argument on that point from me, I survived my control
theory course.

There are many potential mechanisms for feedback in the client/log
communication pipeline other than "provide an email address and have a
human manually reach out". The basis for evaluating a proposed solution
should not ever be solely "is this better than not doing it?", but
rather "is this better than the other known solutions?". New medicines
aren't brought to market solely because they're "better than nothing";
new medicines are better than what we've already got.

HTTP status codes and response bodies, for instance, are already in use
(in fact, they're even a necessary component of the proposed "email in
the user agent" mechanism), and provide an admirable mechanism for
providing feedback to clients. In addition to already largely existing,
this feedback mechanism has the additional benefits of being
privacy-preserving, discriminating between clients _solely_ on their
impact on the log's availability (rather than a multi-order derivative),
and is also extremely amenable to automation.

> Another possible situation that leads to a positive result is that a log
> operator can achieve better performance when get-entries requests are made
> with start % 64 == 0 and end=start+64. If the log operator configures a
> higher rate limit for requests that match that restriction, this increases
> the amount of requests all monitors can make.

Feel free to propose that, then, with appropriate evidence, so that it
can be evaluated as a potential policy or specification change.

> > Comparing "5% more availability" with "putting an email in the user agent"
> > is invalid. Putting an email address in a user agent does not, _in and of
> > itself_, provide a single extra CPU cycle or bit of network bandwidth with
> > which to serve clients (in fact, as it increases the size of all requests,
> > it ever-so-slightly does the opposite).
> >
> > Rather, what including emails in user agents allows, relative to
> > "something that limits log clients", is providing differential
> > rate-limiting. Such rate-limiting might increase availability for a
> > _subset_ of well-behaved clients (those which provide emails) by 5%. It
> > would, however, necessarily _degrade_ availability for those well-behaved
> > clients that don't provide emails, by an equal-or-greater amount.
>
> I describe a hypothetical situation wherein including an email actually
> does result in increasing availability by 5%. It doesn't actually matter
> where this improvement comes from. (Maybe in this hypothetical situation
> including an email in the user agent does magically free up cpu-cycles.) My
> point is that if including emails does this, monitors would want it
> implemented.

And if there were a goose that laid golden eggs, I'd want that, too.
But I'm not going to have a conversation with my bank manager which asks
them to assume, for the sake of argument, that I've got a flock of them
in my backyard.

> Also, in a less magical world, there is a mechanism where including an
> email improves availability beyond just shuffling capacity around. Namely,
> that log operators can contact monitors and ask them to fix a bug in their
> implementation. This reduces the use of the log's resources by this
> monitor, leaving more resources for the other monitors (or more buffer
> resources), increasing availability.

There is already a mechanism by which log operators can contact
monitors: HTTP status codes and response bodies.

- Matt