Yeti 2022-2 Rate Limits

[Andrew Ayer]

When Cert Spotter receives a 429 Too Many Requests error, it backs off
exponentially with random jitter. Before giving up, it waits at least 2
minutes before sending another request. Despite this, Cert Spotter still
sometimes gets rate limited by Yeti 2022-2 when calling get-entries.
For example, in the last few weeks, the following requests failed with a
429 error:

2022-07-31 15:38:48.789248+00: https://yeti2022-2.ct.digicert.com/log/ct/v1/get-entries?start=710936103&end=710937102

2022-07-31 12:28:40.883322+00: https://yeti2022-2.ct.digicert.com/log/ct/v1/get-entries?start=710366503&end=710367502

2022-07-28 15:18:56.384171+00: https://yeti2022-2.ct.digicert.com/log/ct/v1/get-entries?start=703523219&end=703524218

2022-07-26 01:11:02.465644+00: https://yeti2022-2.ct.digicert.com/log/ct/v1/get-entries?start=696552444&end=696553443

2022-07-23 05:09:00.258214+00: https://yeti2022-2.ct.digicert.com/log/ct/v1/get-entries?start=689207790&end=689208789

2022-07-21 09:00:11.039766+00: https://yeti2022-2.ct.digicert.com/log/ct/v1/get-entries?start=684282835&end=684283834

2022-07-21 09:00:11.039766+00: https://yeti2022-2.ct.digicert.com/log/ct/v1/get-entries?start=684282835&end=684283834

2022-07-20 13:25:21.71001+00: https://yeti2022-2.ct.digicert.com/log/ct/v1/get-entries?start=682086640&end=682087639

2022-07-20 00:12:41.77653+00: https://yeti2022-2.ct.digicert.com/log/ct/v1/get-entries?start=680535873&end=680536872

2022-07-19 01:02:16.120108+00: https://yeti2022-2.ct.digicert.com/log/ct/v1/get-entries?start=677775845&end=677776844

A user of the open source Cert Spotter has also
reported being rate limited by Yeti 2022-2 recently:
https://github.com/SSLMate/certspotter/issues/49#issuecomment-1191272177

Additionally, a recent thread on the certificate-transparency@ mailing
list has drawn attention to the fact that the rate limits of
Yeti 2022-2 make it challenging if not impossible to download
this log from scratch within a reasonable amount of time:
https://groups.google.com/g/certificate-transparency/c/1aDoJkjgbuw

All this makes me think that Yeti 2022-2's rate limits are too
restrictive. In particular, rejecting requests that occur less
frequently than once every 2 minutes doesn't seem reasonable.
Could DigiCert increase the rate limits on this log?

Regards,
Andrew

[Devon O'Brien]

Hi Andrew,

Thanks for raising this point; while rate limiting by CT logs is an important protection mechanism to ensure they are able to remain in good standing with the CT Log Policy (https://goo.gl/chrome/ct-log-policy), it also has the ability to diminish the general availability and utility of a CT log to the broader ecosystem if triggering too often.

Concretely, I have a couple questions about DigiCert's rate limiting on their logs:

1. What are DigiCert Yeti 2022-2's rate limits and how does it apply (per IP address, per CA, etc.)?

2. Do DigiCert's other CT logs have the same or a different rate limit as Yeti 2022-2?

3. How often do you see the rate limiting for these logs take effect and to what extent could this be safely relaxed?

-Devon

[Jeremy Rowley]

Hey all, 

The rate limit is set only for the get-entries endpoint. There is no rate limiting on people submitting certs to the log. The limit is 2 request per second per IP address and a limit of 30 requests per second overall. Those limits were set based on previous I/O issues with Amazon’s EBS storage. We think the issues are getting worse now that the shard is at 730M certs. We can increase the limit if you’d like us to see what happens. Yeti and Nessie have the same rate limits. They are built on the same code-base and have a similar infrastructure setup. We’re seeing the rate limit hit pretty much constantly.

Let me know if you want us to see what happens if we increase the rate limit. It could end up impairing a CA's ability to log certs.

Jeremy

[Fernando 🐼]

Hi Jeremy, 

Were the past AWS EBS issues performance related or of another nature? 

If so, are you able to share if they are volume types GP2, GP3 or Provisioned io2? 

Both GP3 and io2 should allow you a better ability to set IOPS and throughput values independently of the EBS size. 

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-volume-types.html

Hope that helps. 

-- 
Fernando 🐼

--
You received this message because you are subscribed to the Google Groups "Certificate Transparency Policy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ct-policy+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/ct-policy/4e3669c0-120f-4144-81f0-e38fd699b375n%40chromium.org.

[Ivan Ristic]

I just want to chime in that we're also seeing backlogs because of the rate limiting [on DigiCert's CT logs], and we're just following the logs using sequential fetching. In fact, there's a small backlog at the moment (which is why I came here :). I am worried that we will fall behind and won't be able to catch up.

--

You received this message because you are subscribed to the Google Groups "Certificate Transparency Policy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ct-policy+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/ct-policy/4e3669c0-120f-4144-81f0-e38fd699b375n%40chromium.org.

--

Ivan

[Jeremy Rowley]

Thanks for your feedback everyone. We increased the rate limit to 120
requests per second. We're seeing peaks of up to 3000 requests per
minute, which means no one should be rate limited right now. We're
monitoring the log to make sure it can sustain the additional load.
Longer term, we want to move to a new cluster. We'll need to take an
outage to do so - I'll give a heads up here before we do. I'm planning
the move for Q4 or (more likely) Q1 next year.

[David Huang]

Hi, I'm not sure if I'm missing something but our monitor is still being rate limited by DigiCert's CT logs.

Even on my laptop, I can pretty consistently hit the "429 Too Many Requests" response just by reloading https://yeti2022-2.ct.digicert.com/log/ct/v1/get-entries?start=710936103&end=710937102 several times by hand (<5 requests in 10 seconds) which is way lower than "120 requests per second". Is that expected?

David

[Aaron Gable]

I believe the 120rps rate limit (raised from the previous 30rps) is a global rate limit, not per IP. This thread has not included a mention of the per-IP 2rps rate limit being increased.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/ct-policy/0c411a54-b32d-46f3-96ea-dd3c31e35b75n%40chromium.org.

[Matthew Hickford]

Hi all. If it helps anyone, the Google mirror has caught up:

https://yeti2022-2.ct.digicert.com/log/

https://ct.googleapis.com/logs/eu1/mirrors/digicert_yeti2022_2/

https://yeti2022-2.ct.digicert.com/log/ct/v1/get-sth

https://ct.googleapis.com/logs/eu1/mirrors/digicert_yeti2022_2/ct/v1/get-sth