Does heavy rate limiting count as CT Log unavailability?
583 views
Skip to first unread message
Ben Cartwright-Cox
Mar 25, 2024, 12:15:32 PMMar 25
to ct-p...@chromium.org
Hey everybody,
Given that CT logs have uptime requirements, I was curious on how
those uptime requirements interact with HTTP rate limiting, and at
what point does a HTTP rate limit become so strict that the log might
be considered down?
For context the reason I am looking/discussing this in the first place
is that I run a website that consumes CT logs entries by querying the
“v1/get-entries” API, However the problem is that some logs ( mostly
speaking the cloudflare ones, but digicert also have similar issues )
have very strict rate limits that mean catching up and auditing these
logs is extraordinarily difficult, as these logs are growing faster
than their HTTP rate limits allow me to fetch.
At the time of writing I'm only able to fetch approximately 70,000
entries per hour from a couple of the previously mentioned logs due to
HTTP 429 responses. While that log is growing well over (sometimes
400,000+ entries per hour) that.
As far as I can observe the CT specification does not have a very good
answer for this, I was curious on what other people's thoughts were,
given that this issue does actually provide a problem around
validating the integrity of a log.
Obviously if there is something I've missed here around limits to
fetching data, I would greatly appreciate someone could call out my
mistake!
Thanks
Ben Cartwright-Cox
Given that CT logs have uptime requirements, I was curious on how
those uptime requirements interact with HTTP rate limiting, and at
what point does a HTTP rate limit become so strict that the log might
be considered down?
For context the reason I am looking/discussing this in the first place
is that I run a website that consumes CT logs entries by querying the
“v1/get-entries” API, However the problem is that some logs ( mostly
speaking the cloudflare ones, but digicert also have similar issues )
have very strict rate limits that mean catching up and auditing these
logs is extraordinarily difficult, as these logs are growing faster
than their HTTP rate limits allow me to fetch.
At the time of writing I'm only able to fetch approximately 70,000
entries per hour from a couple of the previously mentioned logs due to
HTTP 429 responses. While that log is growing well over (sometimes
400,000+ entries per hour) that.
As far as I can observe the CT specification does not have a very good
answer for this, I was curious on what other people's thoughts were,
given that this issue does actually provide a problem around
validating the integrity of a log.
Obviously if there is something I've missed here around limits to
fetching data, I would greatly appreciate someone could call out my
mistake!
Thanks
Ben Cartwright-Cox
Tom Ritter
Mar 25, 2024, 1:51:57 PMMar 25
to Ben Cartwright-Cox, ct-p...@chromium.org
This is reminiscent of the blocking I experienced from China-based CT
logs. I tried running an auditor against these logs, but was blocked
because my server also hosted a Tor (middle) node. I was not trying
to access the logs over Tor (although I think a blanket block of Tor
is equally problematic), just from an IP address that hosted the node.
-tom
> --
> You received this message because you are subscribed to the Google Groups "Certificate Transparency Policy" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to ct-policy+...@chromium.org.
> To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/ct-policy/CAL%3D9YSV6STEnKM-yuOkkzT64hTKwHogmabf04gwib5_yw2H_aw%40mail.gmail.com.
logs. I tried running an auditor against these logs, but was blocked
because my server also hosted a Tor (middle) node. I was not trying
to access the logs over Tor (although I think a blanket block of Tor
is equally problematic), just from an IP address that hosted the node.
-tom
> You received this message because you are subscribed to the Google Groups "Certificate Transparency Policy" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to ct-policy+...@chromium.org.
> To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/ct-policy/CAL%3D9YSV6STEnKM-yuOkkzT64hTKwHogmabf04gwib5_yw2H_aw%40mail.gmail.com.
Joe DeBlasio
Mar 25, 2024, 6:53:22 PMMar 25
to Tom Ritter, Ben Cartwright-Cox, ct-p...@chromium.org
The purpose of running a log is to ensure that certificate data is widely available to monitors and other consumers on the web, and overly-aggressive rate limiting absolutely inhibits certificate availability. At the same time, rate limiting is used by many log operators to ensure their logs stay healthy, accessible to as many users as possible, and available for new certificate submissions.
This is a hard balance to get right, and the ecosystem as a whole isn't where we want it to be -- Chrome's own infrastructure has hit rate limits while monitoring logs in the past, and we know of several other monitors who have struggled to keep up with log growth, too. Thanks for bringing this back up on ct-policy@ -- anyone else struggling with this should feel free to do the same. Doing so allows us to explore remedies or other strategies together, and even if no good options are available, it's useful to document the challenges to shape future work across the ecosystem. To my knowledge, past issues have been addressed by logs increasing rate limits, or readers increasing the volume of IPs they request data from to circumvent the limits. Neither of those workarounds holistically addresses the issue, but have been enough until now.
One of the reasons I'm enthusiastic about efforts like Sunlight is that the design removes server processing from the read path and allows data to be easily cached, which should substantially reduce the need for rate limiting. Once such changes are widely deployed (either through Sunlight or some future effort), there will likely be better opportunities to improve expectations and offer more sophisticated policy on data availability.
-Joe (and Carlos, who had to deal with Chrome encountering this issue first-hand not long ago)
This is a hard balance to get right, and the ecosystem as a whole isn't where we want it to be -- Chrome's own infrastructure has hit rate limits while monitoring logs in the past, and we know of several other monitors who have struggled to keep up with log growth, too. Thanks for bringing this back up on ct-policy@ -- anyone else struggling with this should feel free to do the same. Doing so allows us to explore remedies or other strategies together, and even if no good options are available, it's useful to document the challenges to shape future work across the ecosystem. To my knowledge, past issues have been addressed by logs increasing rate limits, or readers increasing the volume of IPs they request data from to circumvent the limits. Neither of those workarounds holistically addresses the issue, but have been enough until now.
One of the reasons I'm enthusiastic about efforts like Sunlight is that the design removes server processing from the read path and allows data to be easily cached, which should substantially reduce the need for rate limiting. Once such changes are widely deployed (either through Sunlight or some future effort), there will likely be better opportunities to improve expectations and offer more sophisticated policy on data availability.
-Joe (and Carlos, who had to deal with Chrome encountering this issue first-hand not long ago)
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/ct-policy/CA%2BcU71mK%3Dvap32iYzYBnoNhAkz%3Dmy7cUxdsYLu902oq6aBPDwA%40mail.gmail.com.
杨小明
Mar 25, 2024, 10:15:08 PMMar 25
to Certificate Transparency Policy, Ben Cartwright-Cox
hi
The certificate-transparency-go added a default behaviour align_getentries at v1.1.1. If you only get-entries API fetch certs, you should align the start and end parameters. Some ct server cached the get-entries API, because of the backend database query very expensive. Some ct server rate limit 8qps. If you crazy request, someone maybe detected your are malware or loss control user. If you got 429, maybe slow the request rate useful.
align_getentries detail see: https://github.com/google/certificate-transparency-go/releases/tag/v1.1.1
Luke Valenta
Apr 3, 2024, 2:08:59 PMApr 3
to Ben Cartwright-Cox, ct-p...@chromium.org
Hi Ben,
Apologies for the slow response. For the Nimbus logs, the current rate-limiting policy allows 100 requests per 10 seconds per IP, with a 1-minute block for IPs that exceed that limit. We currently allow a batch size of 1024 for get-entries requests, so this should allow one to crawl 10k entries per minute (or 36M entries per hour) as long as the client waits 100ms in between get-entries requests to avoid the 1-minute block.
Similarly, we haven't run into any issues crawling DigiCert's logs with a 500ms delay in between get-entries requests (of batch size 256). See https://groups.google.com/a/chromium.org/g/ct-policy/c/AJ7msx2aWac/m/oz9kh8HVAgAJ for discussion on their limits.
For some history, we set these rate-limits in coordination with the Chrome team to ensure we didn't violate their uptime requirements. Please do let us know if you see continued issues following the logs, and we'll take that into consideration for adjusting the rate limits.
Similarly, we haven't run into any issues crawling DigiCert's logs with a 500ms delay in between get-entries requests (of batch size 256). See https://groups.google.com/a/chromium.org/g/ct-policy/c/AJ7msx2aWac/m/oz9kh8HVAgAJ for discussion on their limits.
For some history, we set these rate-limits in coordination with the Chrome team to ensure we didn't violate their uptime requirements. Please do let us know if you see continued issues following the logs, and we'll take that into consideration for adjusting the rate limits.
Best,
Luke
Luke
--
You received this message because you are subscribed to the Google Groups "Certificate Transparency Policy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ct-policy+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/ct-policy/CAL%3D9YSV6STEnKM-yuOkkzT64hTKwHogmabf04gwib5_yw2H_aw%40mail.gmail.com.
Luke Valenta
Systems Engineer - Research
Reply all
Reply to author
Forward
0 new messages