Rate Limiting Issues with DigiCert CT Logs

[Michel Le Bihan]

Hi all,

I'm monitoring CT logs to inspect newly issued certificates and running into issues with 429 errors on DigiCert logs.

Their rate limit appears to be only 0.15-0.18 req/s. Meanwhile, their https://wyvern.ct.digicert.com/2026h1/ log is growing by approximately 2,700 entries per minute (~44.56/s). Since each request returns at most 256 entries, I'd need roughly 0.174 req/s just to keep up in the ideal case. This margin is extremely tight, and my monitor occasionally falls behind. Once that happens, it becomes impossible to catch up given the rate limit.

Additionally, they don't set a `Retry-After` header, so I'm forced to heuristically guess at retry timing, which is suboptimal.

Has anyone else experienced similar issues or found workarounds for monitoring high-volume DigiCert logs?

Best regards,
Michel Le Bihan

[Rob Stradling]

Due to frequent HTTP 429 responses, crt.sh's monitor is unable to get entries from https://wyvern.ct.digicert.com/2026h2 and https://sphinx.ct.digicert.com/2026h2 fast enough to keep up with the growth rate of those logs.

I've tried dialing down the request rate from my side, but even 1 req/sec triggers 429s.

DigiCert CTOps team: Any tips?  Anything you can improve at your side?

[Rick Roos]

Hi Rob, thanks for reaching out. Quick question for you. Are you by chance using persistent connections and if so can you turn that off and see if your overall rate improves?

Thanks, 

Rick 

--
You received this message because you are subscribed to the Google Groups "Certificate Transparency Policy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ct-policy+...@chromium.org.
To view this discussion visit https://groups.google.com/a/chromium.org/d/msgid/ct-policy/97f9d307-a647-47c1-9914-41382a935847n%40chromium.org.

[Rob Stradling]

Thanks Rick.  Yes, I was using persistent connections.  Disabling that has eliminated the 429s, and my monitor seems to be achieving a slightly higher ingestion rate.

What's the optimal configuration that a monitor should use for DigiCert's logs at the moment?  (i.e., requests per second, concurrent requests, batch size, etc).

---

From: Rick Roos <rick...@gmail.com>
Sent: 28 May 2026 18:29
To: Rob Stradling <r...@sectigo.com>
Cc: Certificate Transparency Policy <ct-p...@chromium.org>; Michel Le Bihan <michel.le...@gmail.com>; Certificate Transparency Operations <ct...@digicert.com>
Subject: Re: [ct-policy] Re: Rate Limiting Issues with DigiCert CT Logs

 

Hi Rob, thanks for reaching out. Quick question for you. Are you by chance using persistent connections and if so can you turn that off and see if your overall rate improves? Thanks, Rick On Thu, May 28, 2026, 9: 41 AM 'Rob Stradling'

ZjQcmQRYFpfptBannerStart

This Message Is From an External Sender

This message came from outside your organization.

Report Suspicious

 

ZjQcmQRYFpfptBannerEnd

[Rick Roos]

Hi Rob,

As requests come in they go through a load balancer and then into a cluster of Nginx servers. For these logs we do not have a centralized rate limiter and instead each Nginx server is configured individually.  Thus to achieve maximum throughput, you need to spread your requests across all Nginx servers by making parallel concurrent requests. That is why turning off persistent connections improved your overall throughput when your client supports concurrent requests. 

Each log shard has 10 Nginx servers and each is configured to allow 1 request per second, resulting in a total theoretical throughput of 10 requests per second.  The maximum batch size is 256 entries if it aligns on a 256 boundary offset (0-255, 256-511, etc), otherwise the entries will be truncated to the next boundary.

I hope this helps.

Thanks,

Rick 

To view this discussion visit https://groups.google.com/a/chromium.org/d/msgid/ct-policy/MW4PR17MB47293DE7D9A50FCC0F809D0DAA092%40MW4PR17MB4729.namprd17.prod.outlook.com.