Rate limits

[Doug Beattie (Globalsign)]

Hi Devon and all,

We've been hitting rate limits recently and wanted to know what the rate limits are set at.  I looked the initial ticket:

https://bugs.chromium.org/p/chromium/issues/detail?id=833350

and it just says that rate limits are set based on IP address.  If we supply our IP address, can we have our rate limit increased, or should we get a larger block of IP addresses and use those to spread the apparent load?  LE apparently does 12qps without issue and I'm sure we're no where near that.

Here are some logs:

    Oct 20 14:16:22 eleanor2 jboss_audit_log: 14:16:15,115 INFO  [org.cesecore.certificates.certificatetransparency.HttpPostTimeoutInvoker] (pool-15-thread-87) Error content from CT log (https://ct.googleapis.com/logs/xenon2021/ct/v1/add-pre-chain) was: Too Many Requests

    Oct 20 14:17:52 eleanor2 jboss_audit_log: 14:17:44,757 INFO  [org.cesecore.certificates.certificatetransparency.HttpPostTimeoutInvoker] (pool-15-thread-107) Error content from CT log (https://ct.googleapis.com/logs/xenon2020/ct/v1/add-pre-chain) was: Too Many Requests

    Oct 20 14:18:12 eleanor2 jboss_audit_log: 14:18:07,967 INFO  [org.cesecore.certificates.certificatetransparency.HttpPostTimeoutInvoker] (pool-20-thread-51) Error content from CT log (https://ct.googleapis.com/logs/xenon2020/ct/v1/add-pre-chain) was: Too Many Requests

    Oct 20 14:18:21 eleanor3 jboss_audit_log: 14:18:21,307 INFO  [org.cesecore.certificates.certificatetransparency.HttpPostTimeoutInvoker] (pool-16-thread-109) Error content from CT log (https://ct.googleapis.com/logs/xenon2020/ct/v1/add-pre-chain) was: Too Many Requests

[Kat Joyce]

Hi Doug,

We are currently investigating and rolling out a fix for the issue you have been hitting - hang tight!

Also, just as an FYI for the future, the fastest way to get intouch with the Google Log operators if you notice any issue with our Logs is via google-...@googlegroups.com.

We will let you know as soon as we have nailed down the issue, and will be publishing details of what happened in due course.

Kind regards,

Kat and the CT Team at Google.

--
You received this message because you are subscribed to the Google Groups "Certificate Transparency Policy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ct-policy+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/ct-policy/169733d8-c3d3-4a2b-a4da-5124dc43604cn%40chromium.org.

[Mohammadamin Karbasforushan]

Hi all,

> On Oct 20, 2020, at 15:53, 'Kat Joyce' via Certificate Transparency Policy <ct-p...@chromium.org> wrote:
>
> Hi Doug,
>
> We are currently investigating and rolling out a fix for the issue you have been hitting - hang tight!
>
> Also, just as an FYI for the future, the fastest way to get intouch with the Google Log operators if you notice any issue with our Logs is via google-...@googlegroups.com.
>
> We will let you know as soon as we have nailed down the issue, and will be publishing details of what happened in due course.

I would also love to see the rate limits documented.

> Kind regards,
> Kat and the CT Team at Google.
>
> On Tue, Oct 20, 2020 at 1:07 PM Doug Beattie (Globalsign) <douglas...@gmail.com> wrote:
> Hi Devon and all,
>
> We've been hitting rate limits recently and wanted to know what the rate limits are set at. I looked the initial ticket:
>
> https://bugs.chromium.org/p/chromium/issues/detail?id=833350
>
> and it just says that rate limits are set based on IP address. If we supply our IP address, can we have our rate limit increased, or should we get a larger block of IP addresses and use those to spread the apparent load? LE apparently does 12qps without issue and I'm sure we're no where near that.

We were hitting the the same limits a while back and in the end went with a larger block of IPs, which did the job. I don’t have the QPS, but with six or seven source IPs we managed to get ~750Mbps of download bandwidth which was enough in our case.

>
> Here are some logs:
>
> Oct 20 14:16:22 eleanor2 jboss_audit_log: 14:16:15,115 INFO [org.cesecore.certificates.certificatetransparency.HttpPostTimeoutInvoker] (pool-15-thread-87) Error content from CT log (https://ct.googleapis.com/logs/xenon2021/ct/v1/add-pre-chain) was: Too Many Requests
>
> Oct 20 14:17:52 eleanor2 jboss_audit_log: 14:17:44,757 INFO [org.cesecore.certificates.certificatetransparency.HttpPostTimeoutInvoker] (pool-15-thread-107) Error content from CT log (https://ct.googleapis.com/logs/xenon2020/ct/v1/add-pre-chain) was: Too Many Requests
>
> Oct 20 14:18:12 eleanor2 jboss_audit_log: 14:18:07,967 INFO [org.cesecore.certificates.certificatetransparency.HttpPostTimeoutInvoker] (pool-20-thread-51) Error content from CT log (https://ct.googleapis.com/logs/xenon2020/ct/v1/add-pre-chain) was: Too Many Requests
>
> Oct 20 14:18:21 eleanor3 jboss_audit_log: 14:18:21,307 INFO [org.cesecore.certificates.certificatetransparency.HttpPostTimeoutInvoker] (pool-16-thread-109) Error content from CT log (https://ct.googleapis.com/logs/xenon2020/ct/v1/add-pre-chain) was: Too Many Requests
>
>
> --
> You received this message because you are subscribed to the Google Groups "Certificate Transparency Policy" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to ct-policy+...@chromium.org.
> To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/ct-policy/169733d8-c3d3-4a2b-a4da-5124dc43604cn%40chromium.org.
>
> --
> You received this message because you are subscribed to the Google Groups "Certificate Transparency Policy" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to ct-policy+...@chromium.org.

> To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/ct-policy/CAO%2BqTAkD_Yn05g4yS%2BFjdXo-%3DRA8MWy8FynfrU%3Dd74FMk%2B7KLg%40mail.gmail.com.

Hope this helps.

Ps. Sorry, I first replied directly (and another time sent via an address not in the mailing list). Didn’t mean to spam.

Cheers,
Amin

[Kat Joyce]

Hi everyone,

I have just realised I wasn't clear in my first response.  This was first brought to our attention by Let's Encrypt, who also experienced problems that appear to the outside world as rate limiting.  We began the investigation and remediation earlier today as a result.  However, the underlying cause is more intricate and unexpected than simple rate limiting.  As I mentioned, we will publish incident details in due course, once we're done getting things running smoothly again.

With regards to publishing/documenting rate limits in general, for us, that is not a trivial task.  We have quite a complex quota system in place, that sits underneath various other pieces of Google infrastructure that are each doing their own thing.  We strive to never block CAs from submitting, and today's case is due to something separate from our quota system, so even having that system documented would not have helped avoid this issue.  If you do ever have issues accessing our Logs, please reach out to us at google-...@googlegroups.com, so we can look at why that may be happening.

For now, keep an eye out for the incident report to follow.

Kind regards,

Kat

On Tue, Oct 20, 2020 at 1:30 PM mka...@mpi-inf.mpg.de <mka...@mpi-inf.mpg.de> wrote:

Hi all,

> On Oct 20, 2020, at 15:53, 'Kat Joyce' via Certificate Transparency Policy <ct-p...@chromium.org> wrote:
>

> Hi Doug,
>
> We are currently investigating and rolling out a fix for the issue you have been hitting - hang tight!
>
> Also, just as an FYI for the future, the fastest way to get intouch with the Google Log operators if you notice any issue with our Logs is via google-...@googlegroups.com.
>
> We will let you know as soon as we have nailed down the issue, and will be publishing details of what happened in due course.

I would also love to see the rate limits documented. :)

>
> Kind regards,
> Kat and the CT Team at Google.
>
> On Tue, Oct 20, 2020 at 1:07 PM Doug Beattie (Globalsign) <douglas...@gmail.com> wrote:
> Hi Devon and all,
>
> We've been hitting rate limits recently and wanted to know what the rate limits are set at.  I looked the initial ticket:
>
> https://bugs.chromium.org/p/chromium/issues/detail?id=833350
>
> and it just says that rate limits are set based on IP address.  If we supply our IP address, can we have our rate limit increased, or should we get a larger block of IP addresses and use those to spread the apparent load?  LE apparently does 12qps without issue and I'm sure we're no where near that.

We were hitting the the same limits a while back and in the end went with a larger block of IPs, which did the job. I don’t have the QPS, but with six or seven source IPs we managed to get ~750Mbps of download bandwidth which was enough in our case.

>

> Here are some logs:
>
>     Oct 20 14:16:22 eleanor2 jboss_audit_log: 14:16:15,115 INFO  [org.cesecore.certificates.certificatetransparency.HttpPostTimeoutInvoker] (pool-15-thread-87) Error content from CT log (https://ct.googleapis.com/logs/xenon2021/ct/v1/add-pre-chain) was: Too Many Requests
>
>     Oct 20 14:17:52 eleanor2 jboss_audit_log: 14:17:44,757 INFO  [org.cesecore.certificates.certificatetransparency.HttpPostTimeoutInvoker] (pool-15-thread-107) Error content from CT log (https://ct.googleapis.com/logs/xenon2020/ct/v1/add-pre-chain) was: Too Many Requests
>
>     Oct 20 14:18:12 eleanor2 jboss_audit_log: 14:18:07,967 INFO  [org.cesecore.certificates.certificatetransparency.HttpPostTimeoutInvoker] (pool-20-thread-51) Error content from CT log (https://ct.googleapis.com/logs/xenon2020/ct/v1/add-pre-chain) was: Too Many Requests
>
>     Oct 20 14:18:21 eleanor3 jboss_audit_log: 14:18:21,307 INFO  [org.cesecore.certificates.certificatetransparency.HttpPostTimeoutInvoker] (pool-16-thread-109) Error content from CT log (https://ct.googleapis.com/logs/xenon2020/ct/v1/add-pre-chain) was: Too Many Requests
>
>
> --
> You received this message because you are subscribed to the Google Groups "Certificate Transparency Policy" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to ct-policy+...@chromium.org.
> To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/ct-policy/169733d8-c3d3-4a2b-a4da-5124dc43604cn%40chromium.org.
>
> --
> You received this message because you are subscribed to the Google Groups "Certificate Transparency Policy" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to ct-policy+...@chromium.org.

> To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/ct-policy/CAO%2BqTAkD_Yn05g4yS%2BFjdXo-%3DRA8MWy8FynfrU%3Dd74FMk%2B7KLg%40mail.gmail.com.

Hope this helps.

Cheers,
Amin

--
You received this message because you are subscribed to the Google Groups "Google CT Logs" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-ct-log...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/google-ct-logs/2F5E2EEC-4E18-4DA3-B53B-2EAC20F80EB3%40mpi-inf.mpg.de.

[Kurt Roeckx]

On Tue, Oct 20, 2020 at 04:09:47PM +0330, Mohammadamin Karbasforushan wrote:
> We were hitting the the same limits a while back and in the end went with a larger block of IPs, which did the job. I don’t have the QPS, but with six or seven source IPs we managed to get ~750Mbps of download bandwidth which was enough in our case.

I'm limited to 1 IP address, and don't nearly get anything close
to 100 Mbit/s for all of Google's log comibined. I currently see
about 10 Mbit/s, while I still have a large backlog for some
Google logs.

The performance seems to mostly depend on the logs. For instance
pilot, icarus and argon are really slow, rocketeer is faster, while
xenon is much faster.

Anyway, today there were periods I got http status code 429's, while
I never triggered them before as far as I know. I assume this is the
incident Kat is talking about.


Kurt