Rate Limit Exceeded Questions

[Tom Davis]

Hello,

We have been running into an error when querying the Docs API: 403 - Rate limit exceeded, lower query rate. We have a couple questions regarding this error:

1. What exactly is the rate limit? I've heard whispers of 10 req/sec but I can't find a confirmed number.
2. What combination of Maximum Rate and Bucket Size for a queue would result in reliably not hitting said limit?

Thanks!

[Tom Davis]

Additionally, we've also found that this results in 503 errors <http://www.google.com/support/websearch/bin/answer.py?&answer=86640> afterwards for an indeterminate amount of time. Apparently our "computer or network may be sending automated queries," which is hardly unexpected given that it's querying an API...

[Vic Fryzel]

Hi Tom,

Yes, there is a qps limit per user.  We don't document the limit in case we need to change it, and applications should exponentially back-off as needed.  That is, if you get a 403 or 503, wait slightly longer, then try again.

Thanks,

-Vic

[Tom Davis]

Thanks for the fast reply, Vic. Do you have a recommended starting point for backing off? Like, should we wait 5 seconds or 50? This only effects us when doing jobs that take a while to test so we'd really appreciate any rules of thumb you can provide.

[Vic Fryzel]

So in general I recommend exponential back-off.  That goes something like this:

Request rejected

Wait 1

Request rejected

Wait 2

Request rejected

Wait 4

Request rejected

Wait 8

Request rejected

Wait 16

...

So you just multiply your delay by 2 each time.

Thanks,

-Vic