Lowering rate limits for unidentified Search API traffic

[Doug Williams]

All --

Last month we sent out the following call to developers [1] to add identifying User Agents and HTTP Referrer strings to their Search API traffic. This is part of a drawn out push to incent as much of our search traffic as possible to include this identifying data.

To identify your application to the Search API, you should always include a uniquely identifying User Agent. A HTTP Referrer is also expected, but not required.

Within the next week, we will begin to throttle unidentified search traffic with a lower rate limit. The rate limit will still be sufficient for most applications (as seen in our logs) but will be lower than the default that is given to consumers identifying themselves. This change should not break properly developed applications that are expecting the HTTP 503 responses that are returned during rate limiting [2].

To be clear, applications using the Search API and including a User Agent (and hopefully an HTTP Referrer) will receive the same rate limit. Applications failing to include a User Agent will receive a lower limit. 

Our goal here is to gain understanding into search traffic. With this data, we will be better informed to build and support the Search API into the future.

1. http://groups.google.com/group/twitter-development-talk/browse_thread/thread/f7cd82f2c43a77d0

2. http://apiwiki.twitter.com/Rate-limiting

Thanks,
Doug