Rate Limit Change for Yeti2024

[Alessandro Maurizio]

Hello,
I can't find any news around so I'll try asking.

Tonight, our scanning systems reported an influx of "new" (actually, old) certs from Yeti2024. This Log, along Yeti2025, was suffering from an extremely high ratelimiting which we found impossible to recover. Our systems worked fine with all the others CT logs to date.

Since we never actually disabled the Yeti fetcher (it's just... 500 millions behind..), tonight we noticed an almost x50 throughtput and we started actually recovering the CT log - even if this means we'll be out for the long game on this CT.

What I've seen it's that a lot more of our requests are actually going in, almost on par with a Google CT in terms of throughtput.

Anyone knows if this was announced/is expected?
The new throughtput slowly increased around 02/05/2024, 20.00 UTC to 01.00 UTC, to a steady rate.

NOTE: Yeti2025 still has the old ratelimits.

Thanks!

[Luke Valenta]

Hi Alessandro,

I'm not sure of any changes in Yeti2024's rate-limiting policy, but you may find this thread useful for crawling logs (including Cloudflare's Nimbus logs) that have rate-limits configured: https://groups.google.com/a/chromium.org/g/ct-policy/c/36gB0Z4EbWE/m/iTRmqP_lAAAJ.

Best,

Luke

--
You received this message because you are subscribed to the Google Groups "certificate-transparency" group.
To unsubscribe from this group and stop receiving emails from it, send an email to certificate-transp...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/certificate-transparency/b37a174b-224a-45c3-9fbb-3f979948a2f4n%40googlegroups.com.

--

Luke Valenta

Systems Engineer - Research

[Alessandro Maurizio]

Hello Luke,
Thanks for your answer.

As of today I can confirm that the Rate Limit is indeed changed on Yeti2024 (since we're still recovering at a steady rate after the weekend) without any application code change.

Thanks for the link, it provides some good advice on how not to hit rate limiting. Sadly, it's a bit hard to implement that logic with our current application because we do automatically detect the batch size using clamping, then we mass parallelize a block of Get-Entries between the min-max, as "chunked requests". This means while the other calls are done sequentially, there is a specific time when mass Get-Entries are done in parallel from the same IP and that would trigger the 1-minute ban.

We'll try to see if we can adapt the algorithm (but we have currently no issues in fetching Nimbus, even if we hit the 429, we have more than enough fetching space to stay on par even after limit hits) but we'll see what we can do.

As of today, using some mirrors and some retires, we're able to fetch all currently usable CTs except Yeti2025.

Thank you!

Alessandro

[Luke Valenta]

> there is a specific time when mass Get-Entries are done in parallel from the same IP and that would trigger the 1-minute ban

I suspect that you'll get better performance by using a single well-tuned thread (per log) for issuing get-entries requests rather than parallelizing that step. This is how Cloudflare's CT monitor is configured and it hasn't been a problem to keep up or catch up with logs.

Best,

Luke

To view this discussion on the web visit https://groups.google.com/d/msgid/certificate-transparency/5dd70233-637f-42c6-986b-ad3be4ef6881n%40googlegroups.com.