Web Images Videos Maps News Shopping Gmail more »
Recently Visited Groups | Help | Sign in
Google Groups Home
Heads up: rate limiting coming soon
There are currently too many topics in this group that display first. To make this topic appear first, remove this option from another topic.
There was an error processing your request. Please try again.
flag
  20 messages - Collapse all  -  Translate all to Translated (View all originals)
The group you are posting to is a Usenet group. Messages posted to this group will make your email address visible to anyone on the Internet.
Your reply message has not been sent.
Your post was successful
 
From:
To:
Cc:
Followup To:
Add Cc | Add Followup-to | Edit Subject
Subject:
Validation:
For verification purposes please type the characters you see in the picture below or the numbers you hear by clicking the accessibility icon. Listen and type the numbers you hear
 
Alex Payne  
View profile  
 More options Jun 22 2007, 10:16 pm
From: "Alex Payne" <a...@al3x.net>
Date: Fri, 22 Jun 2007 19:16:27 -0700
Local: Fri, Jun 22 2007 10:16 pm
Subject: Heads up: rate limiting coming soon
First off, thank you all for making the Twitter API the success it is.
 Our API is responsible for a staggering amount of daily traffic; many
times all of our web, IM, and SMS traffic put together.  That's great
news.

The bad news is that we have to take steps against those users that
are abusing the API to ensure its continued availability.  Starting
early next week we'll be deploying some simple API rate limiting.
Every authenticated user gets 1440 (one thousand four hundred and
forty) requests per 24 hour period, starting from their first request.

To put this in perspective, that's enough to make one authenticated
request to the Twitter API every minute, all day, every day.  Requests
to API methods that don't require authentication are not rate limited
because they're easily cached, and thus not a big load on our servers.

If, for some terrifying reason, you're concerned about needing more
than 1440 requests per day, you have two options:

1. Start using Jabber/XMPP to receive updates and direct messages.
We'll be writing up more information on this shortly, but a number of
developers are already using it for bots and other projects.

2. Email me privately and ask to be put on a white list.  Please
include the screen name of the user you're authenticating with and an
explanation of why you need to talk to the Twitter API more than once
per minute.

I'm happy to answer any questions you may have about our API rate
limiting.  We're just trying to make sure that our service is highly
available for legitimate uses and highly unavailable to abusers.
Thanks for your understanding.

--
Alex Payne
http://twitter.com/al3x


    Reply to author    Forward  
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
James Seng  
View profile  
 More options Jun 22 2007, 10:53 pm
From: "James Seng" <ja...@seng.sg>
Date: Sat, 23 Jun 2007 10:53:19 +0800
Local: Fri, Jun 22 2007 10:53 pm
Subject: Re: Heads up: rate limiting coming soon
Hi Alex,

This is one of the problems I was concerned about in the early days of
twitter API.

Using XMPP is a solution but it does requires Twitter to maintain a
reliable jabber connection. Sadly, from experience polling from
Jabber, I see a very high "drop rates" while receiving message from
XMPP. Would twitter provide a more reliable XMPP interface or alt, one
that is specifically for API developers?

I hope these issues are resolved before the rate limiting is put into
operation (incidentally, when?)

-James Seng

On 6/23/07, Alex Payne <a...@al3x.net> wrote:


    Reply to author    Forward  
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
kev...@tinytwitter.com  
View profile  
 More options Jun 23 2007, 1:03 am
From: kev...@tinytwitter.com
Date: Fri, 22 Jun 2007 22:03:23 -0700
Local: Sat, Jun 23 2007 1:03 am
Subject: Re: Heads up: rate limiting coming soon
Hey Alex, sounds like a completely reasonable thing to do!  Thanks for
the heads up and keep up the great work!

On Jun 22, 8:16 pm, "Alex Payne" <a...@al3x.net> wrote:


    Reply to author    Forward  
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Mihai Parparita  
View profile  
 More options Jun 23 2007, 1:27 am
From: Mihai Parparita <mihai.parpar...@gmail.com>
Date: Sat, 23 Jun 2007 05:27:28 -0000
Local: Sat, Jun 23 2007 1:27 am
Subject: Re: Heads up: rate limiting coming soon
Thanks for the heads up. What will be the behavior of the API when
this limit is hit? Will a specific HTTP status code be returned? (403?
503?)

Mihai

On Jun 22, 10:16 pm, "Alex Payne" <a...@al3x.net> wrote:


    Reply to author    Forward  
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Cameron Kaiser  
View profile  
 More options Jun 23 2007, 1:27 am
From: Cameron Kaiser <spec...@floodgap.com>
Date: Fri, 22 Jun 2007 22:27:44 -0700 (PDT)
Local: Sat, Jun 23 2007 1:27 am
Subject: Re: Heads up: rate limiting coming soon

> The bad news is that we have to take steps against those users that
> are abusing the API to ensure its continued availability.

Is it people who are just hitting the servers over and over? Or ... ?

> Starting
> early next week we'll be deploying some simple API rate limiting.
> Every authenticated user gets 1440 (one thousand four hundred and
> forty) requests per 24 hour period, starting from their first request.

I think this is a good idea, but I can see one population getting
dinged inadvertently -- if they leave, for example, a client running on
one computer and then go and start up another one somewhere else, they
could easily hit that number and more if they forget. I, too, am guilty
of that, but people may wonder why Twitterrific suddenly stopped working
and start blaming clients.

--
------------------------------------ personal: http://www.cameronkaiser.com/ --
  Cameron Kaiser * Floodgap Systems * www.floodgap.com * ckai...@floodgap.com
-- Everywhere is within walking distance if you have the time. ----------------


    Reply to author    Forward  
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Thomas Freudenberg  
View profile  
 More options Jun 23 2007, 4:09 am
From: Thomas Freudenberg <thomas.freudenb...@gmail.com>
Date: Sat, 23 Jun 2007 08:09:31 -0000
Local: Sat, Jun 23 2007 4:09 am
Subject: Re: Heads up: rate limiting coming soon
On Jun 23, 7:27 am, Cameron Kaiser <spec...@floodgap.com> wrote:

> [...]
> I think this is a good idea, but I can see one population getting
> dinged inadvertently -- if they leave, for example, a client running on
> one computer and then go and start up another one somewhere else, they
> could easily hit that number and more if they forget. I, too, am guilty
> of that, but people may wonder why Twitterrific suddenly stopped working
> and start blaming clients.

Maybe that encourages developers of Twitter clients to implement some
intelligence in the client, e.g. stall hitting the server as long as
the user is idle or the screensaver is running.

-Thomas


    Reply to author    Forward  
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Lode  
View profile  
 More options Jun 23 2007, 10:34 am
From: Lode <lodevermei...@gmail.com>
Date: Sat, 23 Jun 2007 07:34:36 -0700
Local: Sat, Jun 23 2007 10:34 am
Subject: Re: Heads up: rate limiting coming soon

> The bad news is that we have to take steps against those users that
> are abusing the API to ensure its continued availability.  Starting
> early next week we'll be deploying some simple API rate limiting.
> Every authenticated user gets 1440 (one thousand four hundred and
> forty) requests per 24 hour period, starting from their first request.

Hi Alex,

I surely think this is a *good* thing, so keep up the good work. Any
chance this rate limiting mechanism can also be used to implement this
request:
http://groups.google.com/group/twitter-development-talk/browse_thread...
(getting more than 20 updates via the api/rss)

The objection to releasing this was that it would have a big impact on
the database, but by limiting the amount of requests over a given time
would surely be a way to enable this functionality without overloading
the database servers, not?

Best,
Lode

--
twitter.com/lode
www.lodev.name


    Reply to author    Forward  
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Cameron Kaiser  
View profile  
 More options Jun 23 2007, 11:24 am
From: Cameron Kaiser <spec...@floodgap.com>
Date: Sat, 23 Jun 2007 08:24:36 -0700 (PDT)
Local: Sat, Jun 23 2007 11:24 am
Subject: Re: Heads up: rate limiting coming soon

> > I think this is a good idea, but I can see one population getting
> > dinged inadvertently -- if they leave, for example, a client running on
> > one computer and then go and start up another one somewhere else, they
> > could easily hit that number and more if they forget. I, too, am guilty
> > of that, but people may wonder why Twitterrific suddenly stopped working
> > and start blaming clients.

> Maybe that encourages developers of Twitter clients to implement some
> intelligence in the client, e.g. stall hitting the server as long as
> the user is idle or the screensaver is running.

This would be very hard to do in a web browser instance or running remotely.

Someone suggested ratelimiting over a shorter period of time rather than over
a whole day. I think I agree with that because if you accidentally screw up
under the currently proposed ratelimiting arrangement, you're toast on Twitter
for the remainder of the 24 hour period. Restricting people to, say, 60
requests an hour would mean you're only down for *that* count.

This would also mean that checking DMs and regular posts would count as two,
using the count up faster than usual.

I still want to reiterate that this is a good idea and no doubt necessary --
as long as people had a chance to modify their behaviour to stay under the
limit, and that the message returned was something greppable or at least
machine grokkable.

Alternatively, could something be added to the API to query "accesses left"?
As long as it doesn't count to the total, of course :)

--
------------------------------------ personal: http://www.cameronkaiser.com/ --
  Cameron Kaiser * Floodgap Systems * www.floodgap.com * ckai...@floodgap.com
-- I like being single. I'm always there when I need me. -- Art Leo -----------


    Reply to author    Forward  
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Alex Payne  
View profile  
 More options Jun 25 2007, 7:02 pm
From: "Alex Payne" <a...@al3x.net>
Date: Mon, 25 Jun 2007 16:02:30 -0700
Local: Mon, Jun 25 2007 7:02 pm
Subject: Re: Heads up: rate limiting coming soon
I like the 60 requests an hour idea.  Going with that instead.

On 6/23/07, Cameron Kaiser <spec...@floodgap.com> wrote:

--
Alex Payne
http://twitter.com/al3x

    Reply to author    Forward  
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Brad W  
View profile  
 More options Jun 25 2007, 9:48 pm
From: Brad W <j.mongol...@gmail.com>
Date: Mon, 25 Jun 2007 18:48:33 -0700
Local: Mon, Jun 25 2007 9:48 pm
Subject: Re: Heads up: rate limiting coming soon
Looks like TwitterTroll.com is going to take a hit.... :(

-Brad

On Jun 25, 7:02 pm, "Alex Payne" <a...@al3x.net> wrote:


    Reply to author    Forward  
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
bear  
View profile  
 More options Jun 25 2007, 10:24 pm
From: bear <bea...@gmail.com>
Date: Tue, 26 Jun 2007 02:24:52 -0000
Local: Mon, Jun 25 2007 10:24 pm
Subject: Re: Heads up: rate limiting coming soon
Brad - only if your using an authenticated source to get around the 20/
min limit of the anonymous sources.

My own twitter bot pulls all of the public feed but anonymously - so
for me it doesn't make a difference.

On Jun 25, 9:48 pm, Brad W <j.mongol...@gmail.com> wrote:


    Reply to author    Forward  
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Cameron Kaiser  
View profile  
 More options Jun 25 2007, 10:33 pm
From: Cameron Kaiser <spec...@floodgap.com>
Date: Mon, 25 Jun 2007 19:33:57 -0700 (PDT)
Local: Mon, Jun 25 2007 10:33 pm
Subject: Re: Heads up: rate limiting coming soon

> I like the 60 requests an hour idea.  Going with that instead.

The only other question I have (dunno about others) is if this applies to
hitting personal timeline or all authenticated actions (the initial
announcement implied the latter). Since there are I'm sure many people
with their Twitterrifics and TTYtters set to one-hit-a-minnit, posting at
the beginning of a particular hour would mean that they would not be able
to get updates at the end proportional to their activity.

I see a couple solutions to this:

- Make the recommended interval, say, two minutes, and keep the 60 req/hr.
Reinforce this with two minute caching so people get no benefit from
banging the timeline more rapidly. This means that most people will hit it
30 times an hour, and thus have 30 tweets to make in that time.

  On the other hand, this will need some way of letting the user know how
much "time" they have left with some sort of API call (which hopefully will
not count towards the 60).

- Make tweets "free" (and only hits to view their timeline will actually
cost). I support this option, and I imagine that timeline hits are by far
a worse offender to load than actual tweets sent; I know there's a side
benefit in muzzling spammy tweeters but I think this will annoy more than
it will delight.

Naturally neither solution is exclusive ... both could be implemented
simultaneously.

--
------------------------------------ personal: http://www.cameronkaiser.com/ --
  Cameron Kaiser * Floodgap Systems * www.floodgap.com * ckai...@floodgap.com
-- Absence of evidence is not evidence of absence. -- Carl Sagan --------------


    Reply to author    Forward  
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Colm McMullan  
View profile  
 More options Jun 26 2007, 4:19 am
From: Colm McMullan <colm.mcmul...@gmail.com>
Date: Tue, 26 Jun 2007 08:19:37 -0000
Local: Tues, Jun 26 2007 4:19 am
Subject: Re: Heads up: rate limiting coming soon
Hi,

I'm a bit concerned about this.  I have a bot that responds (as close
to real time as possible) to direct messages, at the minute we run a
script every 15 seconds which pulls in the new direct messages and
sends whatever number of appropriate responses there are.  Currently
we're doing this with HTTP requests and as you can tell, we could be
doing anything between 4 (to get the direct messages) and any larger
number of requests a minute (to send whatever responses to new
requests).

We tried moving to being a Jabber bot recently and had immense
difficulty, in fact I even posted to the group and got this response
from yourself:

"If you'd like some official word: we're working on Jabber and
delivery
stability, but it's an uphill battle.  For that reason, Jabber/XMPP is
not presently a supported platform for third-party development.  I'm
hoping this changes real soon :) "

http://groups.google.com/group/twitter-development-talk/browse_thread...

That was on June 1st, has this changed?

Thanks,
Colm

On Jun 23, 3:53 am, "James Seng" <j...@seng.sg> wrote:


    Reply to author    Forward  
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Alex Payne  
View profile  
(1 user)  More options Jun 26 2007, 1:52 pm
From: "Alex Payne" <a...@al3x.net>
Date: Tue, 26 Jun 2007 10:52:13 -0700
Local: Tues, Jun 26 2007 1:52 pm
Subject: Re: Heads up: rate limiting coming soon
Colm,

The rate limiting is currently disabled while we field requests from
users like you.  Jabber stability has improved greatly since June 1st,
and we'll be writing up a document on how to best make use of it for
development purposes.

If you contact me off-list, I can place you on a white list until
we've sorted out a better strategy for your particular application's
API needs.

On 6/26/07, Colm McMullan <colm.mcmul...@gmail.com> wrote:

--
Alex Payne
http://twitter.com/al3x

    Reply to author    Forward  
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Alex Payne  
View profile  
 More options Jun 26 2007, 2:00 pm
From: "Alex Payne" <a...@al3x.net>
Date: Tue, 26 Jun 2007 11:00:07 -0700
Local: Tues, Jun 26 2007 2:00 pm
Subject: Re: Heads up: rate limiting coming soon
I'm keeping the details current at
http://groups.google.com/group/twitter-development-talk/web/api-docum...

On 6/25/07, Cameron Kaiser <spec...@floodgap.com> wrote:

--
Alex Payne
http://twitter.com/al3x

    Reply to author    Forward  
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Andy  
View profile  
 More options Jul 5 2007, 11:49 pm
From: Andy <andrew.spar...@gmail.com>
Date: Thu, 05 Jul 2007 20:49:40 -0700
Local: Thurs, Jul 5 2007 11:49 pm
Subject: Re: Heads up: rate limiting coming soon

On 26 Jun, 11:02, "Alex Payne" <a...@al3x.net> wrote:

> I like the 60 requests an hour idea.  Going with that instead.

I think the problem with 60/hour as opposed to 1440/24h is that most
people sleep or are away from a computer during the 24hours. A typical
working day I may be logged into twitter on two computers, both
updating... on my mac with Twitterrific and Twitbin on my pc. I would
much rather use my 1440 to keep updated during the hours I'm awake and
online then to be 'wasting' the allocation for the hours I'm not.

My own twitter utility (a plugin for Yahoo! Messenger that keeps your
status in sync as well as notify of friends updates
http://www.wackyb.co.nz/Yahoo_Messenger_Twitter_Sync_Plugin ) makes
two requests.. once for the friends (to notify of the latest tweets
from your friends) and again for your own latest tweet (to update and
sync your status). What would make this easier would be having a
method that retrieves your latest friends tweets AND your own last
tweet.

As you can see... for the 8 or so hours I may be using my computers
and logged into the twitter service it's possible that with
Twitterrific, Twitbin and plugins running that even with them all set
to a low interval it would exceed that 1/min limit but would still be
within the 1440/24h

 - Andy

( http://www.wackyb.co.nz )


    Reply to author    Forward  
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Cameron Kaiser  
View profile  
 More options Jul 5 2007, 11:56 pm
From: Cameron Kaiser <spec...@floodgap.com>
Date: Thu, 5 Jul 2007 20:56:58 -0700 (PDT)
Local: Thurs, Jul 5 2007 11:56 pm
Subject: Re: Heads up: rate limiting coming soon

> > I like the 60 requests an hour idea.  Going with that instead.

> I think the problem with 60/hour as opposed to 1440/24h is that most
> people sleep or are away from a computer during the 24hours.

This is a valid criticism, but my original objection to the 1440/24h is
that if you accidentally blow it, you're dead for the whole 24 hours.

--
------------------------------------ personal: http://www.cameronkaiser.com/ --
  Cameron Kaiser * Floodgap Systems * www.floodgap.com * ckai...@floodgap.com
-- What an incredible thing we did. -- R. J. Mical, Commodore-Amiga -----------


    Reply to author    Forward  
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Mark Phillip  
View profile  
 More options Jul 6 2007, 1:27 am
From: "Mark Phillip" <markphil...@gmail.com>
Date: Fri, 6 Jul 2007 00:27:42 -0500
Local: Fri, Jul 6 2007 1:27 am
Subject: Re: Heads up: rate limiting coming soon

I agree with Andy.  I understand the fear of "blowing it", but that will
probably happen on a small percentage of days.

I'd much rather be able to give users better response times during peak
hours, and slow things down when traffic tends to be low.

Thanks,
Mark
http://areyouwatchingthis.com

On 7/5/07, Cameron Kaiser <spec...@floodgap.com> wrote:


    Reply to author    Forward  
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Cameron Kaiser  
View profile  
 More options Jul 6 2007, 1:31 am
From: Cameron Kaiser <spec...@floodgap.com>
Date: Thu, 5 Jul 2007 22:31:22 -0700 (PDT)
Local: Fri, Jul 6 2007 1:31 am
Subject: Re: Heads up: rate limiting coming soon

> I agree with Andy.  I understand the fear of "blowing it", but that will
> probably happen on a small percentage of days.

> I'd much rather be able to give users better response times during peak
> hours, and slow things down when traffic tends to be low.

However, the 1440/24h doesn't achieve this either. Yes, it lets you spread
the access throughout the day, but it also lets everyone else do it, so
peak hours will still have peak activity since people have access to burn
and no penalty to do so.

--
------------------------------------ personal: http://www.cameronkaiser.com/ --
  Cameron Kaiser * Floodgap Systems * www.floodgap.com * ckai...@floodgap.com
-- Homestar has a web site? -- Strong Bad -------------------------------------


    Reply to author    Forward  
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Mark Phillip  
View profile  
 More options Jul 6 2007, 1:57 am
From: "Mark Phillip" <markphil...@gmail.com>
Date: Fri, 6 Jul 2007 00:57:01 -0500
Local: Fri, Jul 6 2007 1:57 am
Subject: Re: Heads up: rate limiting coming soon

Well there are two different issues here: what is best for Twitter and what
is best for developers and users.

Alex's first proposal was 1440/24h, so it seems that they don't have a
problem with that.

Users will have a better experience with bots that tend to answer more
quickly.  If Twitter can handle the load, we should let each developer
tailor their service to be as enjoyable for users as possible.

Thanks,
Mark
http://areyouwatchingthis.com

On 7/6/07, Cameron Kaiser <spec...@floodgap.com> wrote:


    Reply to author    Forward  
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
End of messages
« Back to Discussions « Newer topic     Older topic »

Google Groups - Google Home - Terms of Service - Privacy Policy
©2009 Google