Undocumented and variable get-entries limits across CT logs

[Michel Le Bihan]

Hello,

I'm monitoring CT logs to inspect newly issued certificates and I've hit a wall with the undocumented limits on the get-entries endpoint.

RFC 6962 Section 4.6 says "Logs MAY restrict the number of entries that can be retrieved per get-entries request," but doesn't explain how clients should discover what those limits are or how logs should communicate when a limit has been exceeded.

I've measured the following limits across different log operators:

• Google: ~32 entries
• TrustAsia: ~32 entries
• Let's Encrypt: ~256 entries
• DigiCert: ~256 entries
• Sectigo: ~256 entries
• Cloudflare: ~1024 entries

The issues I'm running into:

1. Silent truncation: When more entries than a log permits are requested, fewer entries are simply returned with no error message, no header, and no way to tell "that's all the entries in this range" from "you hit the limit."

2. No discovery mechanism: There's no standardized way to query a log for its get-entries limit. Each log operator sets different limits with no documentation.

3. Rate limiting conflicts: Using a conservative approach (requesting only 32 entries per call to work with all logs) gets me rate limited by DigiCert with 429 errors.

4. No sustainable workaround: Hardcoding per-log limits doesn't seem viable since new logs are added regularly, operators may change their limits, and there's no authoritative source for this information.

Has anyone else run into this? How are you handling it? Am I missing something obvious?

Thanks,
Michel Le Bihan

[Winston de Greef]

Hi Michel,

1 is standard behavior specified in RFC 6962 and isn't actually that bad. 4 isn't a problem, and actually silently truncating is a good solution. Someone else on this email thread recommended always setting last to the log size from get-sth and updating first as the last one received plus one. This way you don't have to worry at all about what restrictions logs implement. If you use this method 2 isn't a problem either.

I believe the 429 your seeing is just a rate limit based on request count/throughput, and not related to the server not liking how you format your requests. Rate limits that are too low to download all entries are a complaint that is regularly posted to this email list. Some people have mentioned implementing sending requests from multiple op adresses to be able to keep up.

Sincerely,

Winston de Greef

--
You received this message because you are subscribed to the Google Groups "Certificate Transparency Policy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ct-policy+...@chromium.org.
To view this discussion visit https://groups.google.com/a/chromium.org/d/msgid/ct-policy/0f0c4b3c-acd0-49e0-a72d-cea3be18dce8n%40chromium.org.

[Matt Palmer]

On Sat, Oct 04, 2025 at 11:54:12AM -0700, Michel Le Bihan wrote:
> The issues I'm running into:
>

> 1.
>
> *Silent truncation*: When more entries than a log permits are requested,

> fewer entries are simply returned with no error message, no header, and no
> way to tell "that's all the entries in this range" from "you hit the limit."

Differentiating between the two cases requires you to compare the ID of
the last entry you received with the tree_size from the STH.

> 2.
>
> *No discovery mechanism*: There's no standardized way to query a log for

> its get-entries limit. Each log operator sets different limits with no
> documentation.

Requiring a log operator to document a limit would, presumably, require
that they return that many entries, but there are
operational reasons why doing that is sometimes not preferred.

> 3.
>
> *Rate limiting conflicts*: Using a conservative approach (requesting

> only 32 entries per call to work with all logs) gets me rate limited by
> DigiCert with 429 errors.

Yeah, some logs are too conservative with their rate limits. It's
technically a violation of Chrome's log policy, but until Chrome starts
wielding the bonk-stick, nobody's going to care. I doubt it'll happen,
because 6962 logs are basically on life support at this point anyway,
with static CT being the new hawtness that everyone is moving to.

> 4.
>
> *No sustainable workaround*: Hardcoding per-log limits doesn't seem

> viable since new logs are added regularly, operators may change their
> limits, and there's no authoritative source for this information.

Closely related to issue #2, but there's no need to hard-code, discover,
or otherwise care about log fetch size, as the basic fetch algorithm of
"fetch end to tree_size; advance start by the number of entries received
last time" works just fine in all scenarios.

> Has anyone else run into this? How are you handling it? Am I missing
> something obvious?

Yes. See above (or [1]). Yes.

- Matt

[1] https://github.com/mpalmer/scrape-ct-log/blob/main/src/fetcher/mod.rs#L153

[Michel Le Bihan]

---

Hi Alex, Winston and Matt,

Thanks for the quick responses!

I've updated my implementation to always set end to tree_size - 1 and just advance start by the number of entries received in each response. Works perfectly and completely sidesteps the whole issue.

Thanks for taking the time to explain this, and for the code reference. It was really helpful to see a working implementation.

Best,
Michel

[Philippe Boneff]

Hi Michel,

You might find this piece of documentation useful: it explains a few of these behaviours, and links to several reference implementations. PRs welcomed if anything is unclear or missing.

Cheers,
Philippe

--

You received this message because you are subscribed to the Google Groups "Certificate Transparency Policy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ct-policy+...@chromium.org.

To view this discussion visit https://groups.google.com/a/chromium.org/d/msgid/ct-policy/24c96cc6-82be-4906-bfcc-2a8ed907276cn%40chromium.org.