Heads up: rate limiting coming soon
Alex Payne
Our API is responsible for a staggering amount of daily traffic; many
times all of our web, IM, and SMS traffic put together. That's great
news.
The bad news is that we have to take steps against those users that
are abusing the API to ensure its continued availability. Starting
early next week we'll be deploying some simple API rate limiting.
Every authenticated user gets 1440 (one thousand four hundred and
forty) requests per 24 hour period, starting from their first request.
To put this in perspective, that's enough to make one authenticated
request to the Twitter API every minute, all day, every day. Requests
to API methods that don't require authentication are not rate limited
because they're easily cached, and thus not a big load on our servers.
If, for some terrifying reason, you're concerned about needing more
than 1440 requests per day, you have two options:
1. Start using Jabber/XMPP to receive updates and direct messages.
We'll be writing up more information on this shortly, but a number of
developers are already using it for bots and other projects.
2. Email me privately and ask to be put on a white list. Please
include the screen name of the user you're authenticating with and an
explanation of why you need to talk to the Twitter API more than once
per minute.
I'm happy to answer any questions you may have about our API rate
limiting. We're just trying to make sure that our service is highly
available for legitimate uses and highly unavailable to abusers.
Thanks for your understanding.
--
Alex Payne
http://twitter.com/al3x
James Seng
This is one of the problems I was concerned about in the early days of
twitter API.
Using XMPP is a solution but it does requires Twitter to maintain a
reliable jabber connection. Sadly, from experience polling from
Jabber, I see a very high "drop rates" while receiving message from
XMPP. Would twitter provide a more reliable XMPP interface or alt, one
that is specifically for API developers?
I hope these issues are resolved before the rate limiting is put into
operation (incidentally, when?)
-James Seng
kev...@tinytwitter.com
the heads up and keep up the great work!
Mihai Parparita
this limit is hit? Will a specific HTTP status code be returned? (403?
503?)
Mihai
Cameron Kaiser
> are abusing the API to ensure its continued availability.
Is it people who are just hitting the servers over and over? Or ... ?
> Starting
> early next week we'll be deploying some simple API rate limiting.
> Every authenticated user gets 1440 (one thousand four hundred and
> forty) requests per 24 hour period, starting from their first request.
I think this is a good idea, but I can see one population getting
dinged inadvertently -- if they leave, for example, a client running on
one computer and then go and start up another one somewhere else, they
could easily hit that number and more if they forget. I, too, am guilty
of that, but people may wonder why Twitterrific suddenly stopped working
and start blaming clients.
--
------------------------------------ personal: http://www.cameronkaiser.com/ --
Cameron Kaiser * Floodgap Systems * www.floodgap.com * cka...@floodgap.com
-- Everywhere is within walking distance if you have the time. ----------------
Thomas Freudenberg
> [...]
> I think this is a good idea, but I can see one population getting
> dinged inadvertently -- if they leave, for example, a client running on
> one computer and then go and start up another one somewhere else, they
> could easily hit that number and more if they forget. I, too, am guilty
> of that, but people may wonder why Twitterrific suddenly stopped working
> and start blaming clients.
Maybe that encourages developers of Twitter clients to implement some
intelligence in the client, e.g. stall hitting the server as long as
the user is idle or the screensaver is running.
-Thomas
Lode
> are abusing the API to ensure its continued availability. Starting
> early next week we'll be deploying some simple API rate limiting.
> Every authenticated user gets 1440 (one thousand four hundred and
> forty) requests per 24 hour period, starting from their first request.
>
Hi Alex,
I surely think this is a *good* thing, so keep up the good work. Any
chance this rate limiting mechanism can also be used to implement this
request:
http://groups.google.com/group/twitter-development-talk/browse_thread/thread/be06e7cceeba3fb4/2318e7a0a2a2ec36
(getting more than 20 updates via the api/rss)
The objection to releasing this was that it would have a big impact on
the database, but by limiting the amount of requests over a given time
would surely be a way to enable this functionality without overloading
the database servers, not?
Best,
Lode
Cameron Kaiser
> > dinged inadvertently -- if they leave, for example, a client running on
> > one computer and then go and start up another one somewhere else, they
> > could easily hit that number and more if they forget. I, too, am guilty
> > of that, but people may wonder why Twitterrific suddenly stopped working
> > and start blaming clients.
>
> Maybe that encourages developers of Twitter clients to implement some
> intelligence in the client, e.g. stall hitting the server as long as
> the user is idle or the screensaver is running.
This would be very hard to do in a web browser instance or running remotely.
Someone suggested ratelimiting over a shorter period of time rather than over
a whole day. I think I agree with that because if you accidentally screw up
under the currently proposed ratelimiting arrangement, you're toast on Twitter
for the remainder of the 24 hour period. Restricting people to, say, 60
requests an hour would mean you're only down for *that* count.
This would also mean that checking DMs and regular posts would count as two,
using the count up faster than usual.
I still want to reiterate that this is a good idea and no doubt necessary --
as long as people had a chance to modify their behaviour to stay under the
limit, and that the message returned was something greppable or at least
machine grokkable.
Alternatively, could something be added to the API to query "accesses left"?
As long as it doesn't count to the total, of course :)
--
------------------------------------ personal: http://www.cameronkaiser.com/ --
Cameron Kaiser * Floodgap Systems * www.floodgap.com * cka...@floodgap.com
-- I like being single. I'm always there when I need me. -- Art Leo -----------
Alex Payne
Brad W
-Brad
On Jun 25, 7:02 pm, "Alex Payne" <a...@al3x.net> wrote:
> I like the 60 requests an hour idea. Going with that instead.
>
> > Cameron Kaiser * Floodgap Systems *www.floodgap.com* ckai...@floodgap.com
> > -- I like being single. I'm always there when I need me. -- Art Leo -----------
>
> --
> Alex Paynehttp://twitter.com/al3x- Hide quoted text -
>
> - Show quoted text -
bear
min limit of the anonymous sources.
My own twitter bot pulls all of the public feed but anonymously - so
for me it doesn't make a difference.
> > Alex Paynehttp://twitter.com/al3x-Hide quoted text -
Cameron Kaiser
The only other question I have (dunno about others) is if this applies to
hitting personal timeline or all authenticated actions (the initial
announcement implied the latter). Since there are I'm sure many people
with their Twitterrifics and TTYtters set to one-hit-a-minnit, posting at
the beginning of a particular hour would mean that they would not be able
to get updates at the end proportional to their activity.
I see a couple solutions to this:
- Make the recommended interval, say, two minutes, and keep the 60 req/hr.
Reinforce this with two minute caching so people get no benefit from
banging the timeline more rapidly. This means that most people will hit it
30 times an hour, and thus have 30 tweets to make in that time.
On the other hand, this will need some way of letting the user know how
much "time" they have left with some sort of API call (which hopefully will
not count towards the 60).
- Make tweets "free" (and only hits to view their timeline will actually
cost). I support this option, and I imagine that timeline hits are by far
a worse offender to load than actual tweets sent; I know there's a side
benefit in muzzling spammy tweeters but I think this will annoy more than
it will delight.
Naturally neither solution is exclusive ... both could be implemented
simultaneously.
--
------------------------------------ personal: http://www.cameronkaiser.com/ --
Cameron Kaiser * Floodgap Systems * www.floodgap.com * cka...@floodgap.com
-- Absence of evidence is not evidence of absence. -- Carl Sagan --------------
Colm McMullan
I'm a bit concerned about this. I have a bot that responds (as close
to real time as possible) to direct messages, at the minute we run a
script every 15 seconds which pulls in the new direct messages and
sends whatever number of appropriate responses there are. Currently
we're doing this with HTTP requests and as you can tell, we could be
doing anything between 4 (to get the direct messages) and any larger
number of requests a minute (to send whatever responses to new
requests).
We tried moving to being a Jabber bot recently and had immense
difficulty, in fact I even posted to the group and got this response
from yourself:
"If you'd like some official word: we're working on Jabber and
delivery
stability, but it's an uphill battle. For that reason, Jabber/XMPP is
not presently a supported platform for third-party development. I'm
hoping this changes real soon :) "
That was on June 1st, has this changed?
Thanks,
Colm
On Jun 23, 3:53 am, "James Seng" <j...@seng.sg> wrote:
> Hi Alex,
>
> This is one of the problems I was concerned about in the early days of
> twitter API.
>
> Using XMPP is a solution but it does requires Twitter to maintain a
> reliable jabber connection. Sadly, from experience polling from
> Jabber, I see a very high "drop rates" while receiving message from
> XMPP. Would twitter provide a more reliable XMPP interface or alt, one
> that is specifically for API developers?
>
> I hope these issues are resolved before the rate limiting is put into
> operation (incidentally, when?)
>
> -James Seng
>
Alex Payne
The rate limiting is currently disabled while we field requests from
users like you. Jabber stability has improved greatly since June 1st,
and we'll be writing up a document on how to best make use of it for
development purposes.
If you contact me off-list, I can place you on a white list until
we've sorted out a better strategy for your particular application's
API needs.
Alex Payne
http://groups.google.com/group/twitter-development-talk/web/api-documentation
Andy
On 26 Jun, 11:02, "Alex Payne" <a...@al3x.net> wrote:
> I like the 60 requests an hour idea. Going with that instead.
I think the problem with 60/hour as opposed to 1440/24h is that most
people sleep or are away from a computer during the 24hours. A typical
working day I may be logged into twitter on two computers, both
updating... on my mac with Twitterrific and Twitbin on my pc. I would
much rather use my 1440 to keep updated during the hours I'm awake and
online then to be 'wasting' the allocation for the hours I'm not.
My own twitter utility (a plugin for Yahoo! Messenger that keeps your
status in sync as well as notify of friends updates
http://www.wackyb.co.nz/Yahoo_Messenger_Twitter_Sync_Plugin ) makes
two requests.. once for the friends (to notify of the latest tweets
from your friends) and again for your own latest tweet (to update and
sync your status). What would make this easier would be having a
method that retrieves your latest friends tweets AND your own last
tweet.
As you can see... for the 8 or so hours I may be using my computers
and logged into the twitter service it's possible that with
Twitterrific, Twitbin and plugins running that even with them all set
to a low interval it would exceed that 1/min limit but would still be
within the 1440/24h
- Andy
Cameron Kaiser
>
> I think the problem with 60/hour as opposed to 1440/24h is that most
> people sleep or are away from a computer during the 24hours.
This is a valid criticism, but my original objection to the 1440/24h is
that if you accidentally blow it, you're dead for the whole 24 hours.
--
------------------------------------ personal: http://www.cameronkaiser.com/ --
Cameron Kaiser * Floodgap Systems * www.floodgap.com * cka...@floodgap.com
-- What an incredible thing we did. -- R. J. Mical, Commodore-Amiga -----------
Mark Phillip
Cameron Kaiser
> probably happen on a small percentage of days.
>
> I'd much rather be able to give users better response times during peak
> hours, and slow things down when traffic tends to be low.
However, the 1440/24h doesn't achieve this either. Yes, it lets you spread
the access throughout the day, but it also lets everyone else do it, so
peak hours will still have peak activity since people have access to burn
and no penalty to do so.
--
------------------------------------ personal: http://www.cameronkaiser.com/ --
Cameron Kaiser * Floodgap Systems * www.floodgap.com * cka...@floodgap.com
-- Homestar has a web site? -- Strong Bad -------------------------------------