API limit confusion

[sjespers]

Hi there,

I am a little bit confused by the API limits.

The server for my application is whitelisted. So it's limit is 20000
API hits.
I use oAuth to authorize Twitter users.
When I check an oAuth'd user's rate limit, he also seems gets 20000
API hits. Is that true?

Also, when I call the Twitter API using the user's oAuth credentials,
which API limit gets that hit? The user's? Or the server's?

Thanks,
Serge

[srikanth reddy]

Hi
I am also looking for this. The following post says there is no limit on calls from application

http://tweetdeck.posterous.com/what-does-rate-limit-exceeded

Rate limit is applicable on Get methods from ip/client.
Can someone confirm if one can make unlimited calls (from an app) to get request token? What is this 20000 limit? Is it for GET calls for authorized client/ip

Regards
Srikanth

[Abraham Williams]

I recommend that you both read: http://apiwiki.twitter.com/Rate-limiting

Serge: If you have an IP that is white listed all applicable calls from that IP will count against the 20000 limit.

Srikanth: That blog post says that twitter.com has no limit. It says nothing about anybody else not having a limit. The 20k is for GET requests however POST request have their own limits.

Abraham

--
Abraham Williams | Community Evangelist | http://web608.org
Hacker | http://abrah.am | http://twitter.com/abraham
Project | http://fireeagle.labs.poseurtech.com
This email is: [ ] blogable [x] ask first [ ] private.
Sent from Madison, WI, United States

[sjespers]

@Abraham: If that were true then calling rate_limit_status should give
the same result... which it doesn't!

On Jul 22, 3:26 pm, Abraham Williams <4bra...@gmail.com> wrote:
> I recommend that you both read:http://apiwiki.twitter.com/Rate-limiting
>
> Serge: If you have an IP that is white listed all applicable calls from that
> IP will count against the 20000 limit.
>
> Srikanth: That blog post says that twitter.com has no limit. It says nothing
> about anybody else not having a limit. The 20k is for GET requests however
> POST request have their own limits.
>
> Abraham
>

[Abraham Williams]

In your first email you said "When I check an oAuth'd user's rate limit, he also seems gets 20000 API hits." so I'm not sure what you are seeing. 

Also it used to be that user requests from a whitelisted IP would reflect on the users limit unless they had hit their rate limit at which point it would count against the IP. I'm not sure if it still works this way though.

Abraham

[srikanth reddy]

@Abraham: Does it mean my consumer app (not Desktop client) cannot serve more than 150 authorized users/hour(if it is not white listed). It is hard to believe.
If it is desktop client the 150 limit is understandable.

The blog post says

"This limit applies to your Twitter account rather than the applications which make the calls to the API i.e. you have 100 API calls per hour in total regardless of which Twitter applications you use - it is NOT 100 API calls per application"

As you said

"Also it used to be that user requests from a whitelisted IP would reflect on the users limit unless they had hit their rate limit at which point it would count against the IP."

its probably first user and then IP.

" POST request have their own limits"
yes i do not mean infinite calls but my consumer app should be able to get more than 20k request tokens

Thanks for your time. Really helpful
Srikanth

[Abraham Williams]

On Wed, Jul 22, 2009 at 15:06, srikanth reddy <srikanth...@gmail.com> wrote:

@Abraham: Does it mean my consumer app (not Desktop client) cannot serve more than 150 authorized users/hour(if it is not white listed). It is hard to believe.
If it is desktop client the 150 limit is understandable.

Each user and each IP has 150 calls/hour. If five applications (desktop or web) are making calls on behalf of a single user or IP they count against the same 150. Rate limiting has no connection to applications.

 

The blog post says

"This limit applies to your Twitter account rather than the applications which make the calls to the API i.e. you have 100 API calls per hour in total regardless of which Twitter applications you use - it is NOT 100 API calls per application"

As you said

"Also it used to be that user requests from a whitelisted IP would reflect on the users limit unless they had hit their rate limit at which point it would count against the IP."

its probably first user and then IP.

Yes. User then IP.

[srikanth reddy]

On Thu, Jul 23, 2009 at 12:07 PM, Abraham Williams <4br...@gmail.com> wrote:

<<
Each user and each IP has 150 calls/hour. If five applications (desktop or web) are making calls on behalf of a single user or IP they count against the same 150. Rate limiting has no connection to applications.
>>

Agreed. i have no issues with desktop apps as each user owns one (in which case ip/user does not matter and am pretty happy with 150 limit).

But i am trying to understand this ip limit for web apps


The  whole confusion is regarding this statement in
http://apiwiki.twitter.com/Rate-limiting

"IP whitelisting takes precedence to account rate limits. GET requests from a whitelisted IP address made on a user's behalf will be deducted from the whitelisted IP's limit, not the users. Therefore, IP-based whitelisting is a best practice for applications that request many users' data"

If the above holds true my consumer web app could end up serving very few authenticated users. As you said it should be the other way.
May be some one who has developed and encountered this problem with a webapp (with out being whitelisted) can confirm.

Thanks
Srikanth


 

[Abraham Williams]

On Thu, Jul 23, 2009 at 02:02, srikanth reddy <srikanth...@gmail.com> wrote:

The  whole confusion is regarding this statement in

http://apiwiki.twitter.com/Rate-limiting

"IP whitelisting takes precedence to account rate limits. GET requests from a whitelisted IP address made on a user's behalf will be deducted from the whitelisted IP's limit, not the users. Therefore, IP-based whitelisting is a best practice for applications that request many users' data"

If the above holds true my consumer web app could end up serving very few authenticated users. As you said it should be the other way.
May be some one who has developed and encountered this problem with a webapp (with out being whitelisted) can confirm.

I guess it is not the same as it used to be with how it does not effect user limits first. With 20k/h you can accomplish a lot. If you hit that limit that you should contact a...@twitter.com and talk with them about higher limits or more efficient methods to use.

Abraham

[Hwee-Boon Yar]

It's working like you want it to be.

In other words, you have a web app running on a single server with a
single IP. You make authenticated requests using each user's account.
If your IP is whitelisted, the calls go towards your 20k limit, if it
is not whitelisted, it goes against the current 150 limit for the
respective accounts. That's what it means by "IP whitelisting takes

precedence to account rate limits".

--
Hwee-Boon

On Jul 23, 3:02 pm, srikanth reddy <srikanth.yara...@gmail.com> wrote:

> On Thu, Jul 23, 2009 at 12:07 PM, Abraham Williams <4bra...@gmail.com>wrote:
>
> <<
> Each user and each IP has 150 calls/hour. If five applications (desktop or
> web) are making calls on behalf of a single user or IP they count against
> the same 150. Rate limiting has no connection to applications.
>
>
>
> Agreed. i have no issues with desktop apps as each user owns one (in which
> case ip/user does not matter and am pretty happy with 150 limit).
>
> But i am trying to understand this ip limit for web apps
>

> The  whole confusion is regarding this statement inhttp://apiwiki.twitter.com/Rate-limiting
>
> "IP whitelisting takes precedence to account rate limits. *GET requests from

> a whitelisted IP address made on a user's behalf will be deducted from the

> whitelisted IP's limit, not the users*. Therefore, IP-based whitelisting is

[srikanth reddy]

Ohh. Then one user can make 150 authorized calls via consumer and deny service to others :(

[jmathai]

> In other words, you have a web app running on a single server with a
> single IP. You make authenticated requests using each user's account.
> If your IP is whitelisted, the calls go towards your 20k limit, if it
> is not whitelisted, it goes against the current 150 limit for the
> respective accounts. That's what it means by "IP whitelisting takes
> precedence to account rate limits".

I don't believe that is true. If your web app is running on a
whitelisted IP then you get up to 20k GET calls per hour. POST
requests (status or DM) are counted against the user being
authenticated. You CANNOT retrieve a user's rate limit status.

[jim.renkel]

My experience with this is, I think, a little bit different than what
you describe.

It appears to me that each user of a white-listed site gets 20k
requests per hour, independent of any other users of that site or any
other uses of the twitter API at other sites by that user.

I didn't think this was what twitter intended and reported it as a bug
(See: http://code.google.com/p/twitter-api/issues/detail?id=617), but
the twitter folk said "Yup, working as intended".

After you log in at http://twxlate.com, the site reports rate limit
information on every page view, so you can see how this works there.

Comments expected and welcome.

Jim Renkel

[srikanth reddy]

@jim.renkel. Thanks a ton. I think now it is clear.

<<It appears to me that each user of a white-listed site gets 20k
requests per hour, independent of any other users of that site or any
other uses of the twitter API at other sites by that user >>

probably this is what they mean by

"IP whitelisting takes precedence to account rate limits. GET requests from a whitelisted IP address made on a user's behalf will be deducted from the whitelisted IP's limit, not the users."

If the IP limit  is for the consumer then it will lead to denial of service attacks.
This is how we wanted it to work.

Srikanth

[Hwee-Boon Yar]

Isn't this what I said?

--
Hwee-Boon

On Jul 24, 2:36 pm, srikanth reddy <srikanth.yara...@gmail.com> wrote:
> @jim.renkel. Thanks a ton. I think now it is clear.
>
> <<It appears to me that each user of a white-listed site gets 20k

> requests per hour, independent of any other users of that site or *any
> other uses of the twitter API at other sites by that user *>>

>
> probably this is what they mean by
>

> *"IP whitelisting takes precedence to account rate limits*. *GET requests

> from a whitelisted IP address made on a user's behalf will be deducted from

> the whitelisted IP's limit, not the users*."

>
> If the IP limit  is for the consumer then it will lead to denial of service
> attacks.
> This is how we wanted it to work.
>
> Srikanth
>
>
>
> On Fri, Jul 24, 2009 at 12:52 AM, jim.renkel <james.ren...@gmail.com> wrote:
>
> > My experience with this is, I think, a little bit different than what
> > you describe.
>
> > It appears to me that each user of a white-listed site gets 20k
> > requests per hour, independent of any other users of that site or any
> > other uses of the twitter API at other sites by that user.
>
> > I didn't think this was what twitter intended and reported it as a bug
> > (See:http://code.google.com/p/twitter-api/issues/detail?id=617), but
> > the twitter folk said "Yup, working as intended".
>

> > After you log in athttp://twxlate.com, the site reports rate limit

[Bill Kocik]

If this is correct (and I don't think it is), then it's very different
from what has always been my understanding. I've stated a few times on
this list my belief that if you're going to be supporting a
significant number of simultaneous users, whitelisting works against
you. No one has ever challenged that assertion.

This is in the rate limit documentation Abraham linked to: "Each
whitelisted entity, whether an account or IP address, is allowed 20000
requests per hour."

It is my understanding that if your IP is whitelisted, you get one
pool of 20,000 requests per each hour to divide up amongst all your
users - NOT 20,000 requests for each of them. I could be wrong, but I
don't think I am. If you have a whitelisted IP, here's an experiment
you can run:

1. Check the current rate limit using two different Twitter accounts
from your IP (using curl or whichever tool you choose). You should get
the same number (probably 20,000 unless you've been making requests
recently from that IP)
2. Make a request that counts against the rate limit from ONE and only
ONE of the two accounts (grab their statuses/friends_timeline, for
example)
3. Repeat step 1. Do both users now see 19,999? Or does one see 19,999
and one see 20,000?

If one user still sees 20,000, then I'm wrong, and you've got 20,000
requests per hour per user (and I'm gonna go apply for
whitelisting :). If they both see 19,999, then I'm right - your IP has
a single pool of 20,000 requests from which all of your users draw.

On Jul 24, 2:36 am, srikanth reddy <srikanth.yara...@gmail.com> wrote:
> @jim.renkel. Thanks a ton. I think now it is clear.
>
> <<It appears to me that each user of a white-listed site gets 20k

> requests per hour, independent of any other users of that site or *any
> other uses of the twitter API at other sites by that user *>>

>
> probably this is what they mean by
>

> *"IP whitelisting takes precedence to account rate limits*. *GET requests

> from a whitelisted IP address made on a user's behalf will be deducted from

> the whitelisted IP's limit, not the users*."

>
> If the IP limit  is for the consumer then it will lead to denial of service
> attacks.
> This is how we wanted it to work.
>
> Srikanth
>
>
>
> On Fri, Jul 24, 2009 at 12:52 AM, jim.renkel <james.ren...@gmail.com> wrote:
>
> > My experience with this is, I think, a little bit different than what
> > you describe.
>
> > It appears to me that each user of a white-listed site gets 20k
> > requests per hour, independent of any other users of that site or any
> > other uses of the twitter API at other sites by that user.
>
> > I didn't think this was what twitter intended and reported it as a bug
> > (See:http://code.google.com/p/twitter-api/issues/detail?id=617), but
> > the twitter folk said "Yup, working as intended".
>

> > After you log in athttp://twxlate.com, the site reports rate limit

[Bill Kocik]

On Jul 24, 4:13 am, Hwee-Boon Yar <hweeb...@gmail.com> wrote:
> Isn't this what I said?

I don't think it is. I think your take is correct. What's telling is
this bit of text from up the chain: "It appears to me that each user
of a white-listed site gets 20k requests per hour".

I don't believe it's true that each user on a site whitelisted by IP
gets their own 20k requests to use. I think it's true that *all* users
of that whitelisted site are drawing from the *same single pool* of
20k requests allocated to that IP. I think that's where Srikanth is
confused.

But I'm always willing concede that I might be wrong. I'm wrong a lot.
I would actually like to be wrong on this one.

[srikanth reddy]

@ Hwee-Boon

> Isn't this what I said?

i dont think so. I (and i think everyone) interpreted it as 20k limit to IP for all users if ip is whitelisted else the limit is 150 per user.

@Bill Kocik

<< 3. Repeat step 1. Do both users now see 19,999? Or does one see 19,999
and one see 20,000?
>>

 jim renkel and sjepers have already tested this.I also verified with two different accounts.
on http://twxlate.com
Guess what ? it is working as intended :) (20k for each user)

Srikanth

[Bill Kocik]

On Jul 25, 4:47 am, srikanth reddy <srikanth.yara...@gmail.com> wrote:

> @Bill Kocik
>
> << 3. Repeat step 1. Do both users now see 19,999? Or does one see 19,999
> and one see 20,000?
>
>  jim renkel and sjepers have already tested this.I also verified with two
> different accounts.

> onhttp://twxlate.com

> Guess what ? it is working as intended :) (20k for each user)

Well I'll be. I've been wrong all this time. I wish someone had told
me. And I still really wish someone from Twitter would step in and
definitively say "Yes, it's 20K per whitelisted IP for EACH user"
before I go apply for whitelisting, but that kind of test is difficult
to argue with.

[srikanth reddy]

Probably they think what they mentioned here is enough??
http://code.google.com/p/twitter-api/issues/detail?id=617

Testing 20k calls for each user might be difficult .May be we should verify this when IP is not whitelisted.I believe for many sites, it is not uncommon to expect 200 users/hour with each user making 100 authorized calls. This would easily make up 20k calls/hour even when IP is not whitelisted.But I dont know  any such popular app.:(
If they are popular they get their IP whitelisted. :(