Putting a ceiling on requests from users and IPs on the whitelist
40 views
Skip to first unread message
Alex Payne
Jan 20, 2009, 6:48:35 PM1/20/09
to Twitter Development Talk, twitter-ap...@googlegroups.com
Up until now we've allowed users and IPs on our whitelist an unlimited
number of requests per hour. When our whitelist was in the tens and
low hundreds, this made sense. Now that we have more developers on the
whitelist than we can reasonably maintain close communication with, we
need to put a ceiling on the number of requests per hour whitelisted
accounts and IPs can make.
number of requests per hour. When our whitelist was in the tens and
low hundreds, this made sense. Now that we have more developers on the
whitelist than we can reasonably maintain close communication with, we
need to put a ceiling on the number of requests per hour whitelisted
accounts and IPs can make.
Starting later this week we'll be limiting those on the whitelist to
20,000 requests per hour. Yes, you read that right: twenty THOUSAND
requests per hour. According to our logs, this accounts for all but
the very largest consumers of our API. This is essentially a
preventative measure to ensure that no one API client, even a
whitelisted account or IP, can consume an inordinate amount of our
resoures.
If you run one of the services that routinely exceed 20k
requests/hour, please get in contact with us (a...@twitter.com) as soon
as possible. Chances are good that you'll simply need to slow your
crawl rates, implement more caching on your end, and limit requests to
only active accounts. We're happy to work with you to find solutions.
--
Alex Payne - API Lead, Twitter, Inc.
http://twitter.com/al3x
Jesse Stay
Jan 21, 2009, 12:19:10 AM1/21/09
to Twitter Development Talk
Alex, you and I have discussed this, but I still think this is a bad
decision until some sort of better method is placed around getting the
list of followers of a user. This basically limits how big any
application on your platform can get. Right now it takes 400 requests
alone to get Robert Scoble's followers. It takes 350 requests to get
Guy Kawasaki's followers. It takes similar to get Chris Pirillo's
followers. Does this mean we just exclude allowing them on our apps
now? Why develop for the Twitter platform any more if we know we can
only grow to your limit?
Jesse
Jesse
Mario Menti
Jan 21, 2009, 7:07:42 AM1/21/09
to twitter-deve...@googlegroups.com
Alex,
are status update POSTs still excluded from this limit?
Mario.
are status update POSTs still excluded from this limit?
Mario.
Alex Payne
Jan 21, 2009, 12:46:33 PM1/21/09
to twitter-deve...@googlegroups.com
Yes.
iematthew
Jan 21, 2009, 1:16:18 PM1/21/09
to Twitter Development Talk
Perhaps a leaner version for requesting a user's followers and friends
would help? Say, a method that only returns the ID and screen name for
the user's followers or friends, but in lots of a thousand or ten
thousand at a time.
On Jan 21, 12:19 am, Jesse Stay <jesses...@gmail.com> wrote:
> Alex, you and I have discussed this, but I still think this is a bad
> decision until some sort of better method is placed around getting the list
> of followers of a user. This basically limits how big any application on
> your platform can get. Right now it takes 400 requests alone to get Robert
> Scoble's followers. It takes 350 requests to get Guy Kawasaki's followers.
> It takes similar to get Chris Pirillo's followers. Does this mean we just
> exclude allowing them on our apps now? Why develop for the Twitter platform
> any more if we know we can only grow to your limit?
>
> Jesse
>
would help? Say, a method that only returns the ID and screen name for
the user's followers or friends, but in lots of a thousand or ten
thousand at a time.
On Jan 21, 12:19 am, Jesse Stay <jesses...@gmail.com> wrote:
> Alex, you and I have discussed this, but I still think this is a bad
> decision until some sort of better method is placed around getting the list
> of followers of a user. This basically limits how big any application on
> your platform can get. Right now it takes 400 requests alone to get Robert
> Scoble's followers. It takes 350 requests to get Guy Kawasaki's followers.
> It takes similar to get Chris Pirillo's followers. Does this mean we just
> exclude allowing them on our apps now? Why develop for the Twitter platform
> any more if we know we can only grow to your limit?
>
> Jesse
>
Jesse Stay
Jan 21, 2009, 1:23:32 PM1/21/09
to twitter-deve...@googlegroups.com
That would reduce our requests by ten-fold.
Jesse
Jesse
rhysmeister
Jan 21, 2009, 4:28:44 PM1/21/09
to Twitter Development Talk
Yeah, somehow being able to periodically fully replicate your twitter
data would be really good and reduce requests considerably.
data would be really good and reduce requests considerably.
greg schoen
Jan 21, 2009, 5:43:19 PM1/21/09
to twitter-deve...@googlegroups.com
I agree, ID/ScreenName only responses would cut down on A LOT of traffic.
Jode...@gmail.com
Jan 21, 2009, 9:06:13 PM1/21/09
to Twitter Development Talk
Alex,
Look forward to working with you on this.
As you know PeopleBrowsr passes all requests via a cache to minimise
API requests on Twitter.
Our requests which are coming from many clients will appear as coming
from one IP address.
Clients like Tweetdeck that dont cache, will make many more API calls
on Twitter (per user), though will not suffer the API ceiling.
Regards
Look forward to working with you on this.
As you know PeopleBrowsr passes all requests via a cache to minimise
API requests on Twitter.
Our requests which are coming from many clients will appear as coming
from one IP address.
Clients like Tweetdeck that dont cache, will make many more API calls
on Twitter (per user), though will not suffer the API ceiling.
Regards
Scott C. Lemon
Jan 21, 2009, 7:45:04 PM1/21/09
to Twitter Development Talk
Alex,
Can you please address the issue of how you would recommend that an
application be able to fetch the total list of a users followers? I'm
hearing all sorts of accusations and bluster, but want to understand
from your perspective how you would propose an application do this?
It's obvious there is tremendous value that can be provided to a user,
if an application were able to get the list, or changes to the list,
quickly and efficiently. I do know that for most users, they are able
to use the existing API, and due to the limited number of followers it
is fairly painless. However, I can also understand the burden on your
servers of requesting the list of followers ... which then also
requires that the current status of each follower be included.
Even here on the list there are people suggesting that you enhance the
API to simply drop the current status ... and only return the list of
followers ... which would seem to be a much simpler, and less
intensive, query.
Can you shed some light so that we can better understand your
thinking, and so that we can better understand how to architect our
twitter applications for the future?
As a side note, we are currently working on three twitter
applications ... two that are hosted services, and one that is a
desktop application. I want to ensure that we continue to build these
fully understanding how to work cooperatively with you!
Thanks!
Scott
Can you please address the issue of how you would recommend that an
application be able to fetch the total list of a users followers? I'm
hearing all sorts of accusations and bluster, but want to understand
from your perspective how you would propose an application do this?
It's obvious there is tremendous value that can be provided to a user,
if an application were able to get the list, or changes to the list,
quickly and efficiently. I do know that for most users, they are able
to use the existing API, and due to the limited number of followers it
is fairly painless. However, I can also understand the burden on your
servers of requesting the list of followers ... which then also
requires that the current status of each follower be included.
Even here on the list there are people suggesting that you enhance the
API to simply drop the current status ... and only return the list of
followers ... which would seem to be a much simpler, and less
intensive, query.
Can you shed some light so that we can better understand your
thinking, and so that we can better understand how to architect our
twitter applications for the future?
As a side note, we are currently working on three twitter
applications ... two that are hosted services, and one that is a
desktop application. I want to ensure that we continue to build these
fully understanding how to work cooperatively with you!
Thanks!
Scott
Aditya
Jan 21, 2009, 10:20:18 PM1/21/09
to Twitter Development Talk
In light of this, do we get a "rate_limit_status" type feed for a
given IP, or do we keep an internal track of our calls (which could be
inaccurate)?
given IP, or do we keep an internal track of our calls (which could be
inaccurate)?
Steve Brunton
Jan 21, 2009, 11:16:12 PM1/21/09
to twitter-deve...@googlegroups.com
On Wed, Jan 21, 2009 at 10:20 PM, Aditya <adi...@adityamukherjee.com> wrote:
>
> In light of this, do we get a "rate_limit_status" type feed for a
> given IP, or do we keep an internal track of our calls (which could be
> inaccurate)?
>
>
> In light of this, do we get a "rate_limit_status" type feed for a
> given IP, or do we keep an internal track of our calls (which could be
> inaccurate)?
>
You mean like the "rate_limit_status" API call?
http://apiwiki.twitter.com/REST+API+Documentation#ratelimitstatus
-steve
Alex Payne
Jan 22, 2009, 1:49:06 AM1/22/09
to twitter-deve...@googlegroups.com
> Can you please address the issue of how you would recommend that an
> application be able to fetch the total list of a users followers? I'm
> hearing all sorts of accusations and bluster, but want to understand
> from your perspective how you would propose an application do this?
> application be able to fetch the total list of a users followers? I'm
> hearing all sorts of accusations and bluster, but want to understand
> from your perspective how you would propose an application do this?
Right now, you can page through the lists of friends and followers.
We'll see about ways to get the friends list in one shot. I think our
best bet for a method that returns the entire set of friends or
followers would be simply returning an array of user IDs.
> Even here on the list there are people suggesting that you enhance the
> API to simply drop the current status ... and only return the list of
> followers ... which would seem to be a much simpler, and less
> intensive, query.
We'll certainly consider that.
> As a side note, we are currently working on three twitter
> applications ... two that are hosted services, and one that is a
> desktop application. I want to ensure that we continue to build these
> fully understanding how to work cooperatively with you!
All you need to do is send us an email, and we'll find solutions. It
may take some time if the solution you need isn't on our development
roadmap, but we're here to make it happen.
Jesse Stay
Jan 22, 2009, 2:02:31 AM1/22/09
to twitter-deve...@googlegroups.com
Alex, thanks - this is what we were looking for. In our defense, I did send you several personal e-mails and you mentioned this wasn't on your roadmap. While the limit will still eventually get us, this will get us much further. Is it necessary to implement the limit before these changes are in place, and when you do implement the limit, how do we know we're getting close to hitting it?
Thanks,
Jesse
Thanks,
Jesse
Matthias Bauer
Jan 22, 2009, 4:20:28 AM1/22/09
to twitter-deve...@googlegroups.com
iematthew wrote:
> Perhaps a leaner version for requesting a user's followers and friends
> would help? Say, a method that only returns the ID and screen name for
> the user's followers or friends, but in lots of a thousand or ten
> thousand at a time.
> Perhaps a leaner version for requesting a user's followers and friends
> would help? Say, a method that only returns the ID and screen name for
> the user's followers or friends, but in lots of a thousand or ten
> thousand at a time.
Yes, please!
-Matt
Scott C. Lemon
Jan 22, 2009, 1:15:03 PM1/22/09
to Twitter Development Talk
Alex,
Thank you for the quick reply ... and I agree ... it seems that
returning the list of followers or friends as an array of IDs would be
most effective. This is somewhat how Facebook does it with their
API. That would allow us to gain access to the information in a much
more compact way.
Obviously this would also make the amount of data being returned much
less, as it wouldn't have to include the status of those people or
anything else ... we can get that in subsequent requests, or by
monitoring and caching the feed.
So maybe there could be two new APIs?
http://twitter.com/statuses/friendIDs.format
http://twitter.com/statuses/followerIDs.format
What do you propose as next steps for this? Do I/we have to enter a
bug/issue to make the request? Or are you already working on a
solution for this? We would be very interested in seeing this new API
so that we can fine tune our operation and minimize the number of
requests we'll be making ...
Scott
Thank you for the quick reply ... and I agree ... it seems that
returning the list of followers or friends as an array of IDs would be
most effective. This is somewhat how Facebook does it with their
API. That would allow us to gain access to the information in a much
more compact way.
Obviously this would also make the amount of data being returned much
less, as it wouldn't have to include the status of those people or
anything else ... we can get that in subsequent requests, or by
monitoring and caching the feed.
So maybe there could be two new APIs?
http://twitter.com/statuses/friendIDs.format
http://twitter.com/statuses/followerIDs.format
What do you propose as next steps for this? Do I/we have to enter a
bug/issue to make the request? Or are you already working on a
solution for this? We would be very interested in seeing this new API
so that we can fine tune our operation and minimize the number of
requests we'll be making ...
Scott
Alex Payne
Jan 22, 2009, 5:46:41 PM1/22/09
to twitter-deve...@googlegroups.com
I'm already working on this. It'll be new methods, I think. Should be
ready to deploy by early next week.
ready to deploy by early next week.
Damon C
Jan 22, 2009, 8:28:20 PM1/22/09
to Twitter Development Talk
The only reply I can think of for this is HOT!
I just spent the past couple weeks building a service to search ones
followers (http://tweepsearch.com) and I still feel bad about pounding
the API. This will make it _much_ easier.
Thanks for making this change!
dpc
On Jan 22, 2:46 pm, Alex Payne <a...@twitter.com> wrote:
> I'm already working on this. It'll be new methods, I think. Should be
> ready to deploy by early next week.
>
I just spent the past couple weeks building a service to search ones
followers (http://tweepsearch.com) and I still feel bad about pounding
the API. This will make it _much_ easier.
Thanks for making this change!
dpc
On Jan 22, 2:46 pm, Alex Payne <a...@twitter.com> wrote:
> I'm already working on this. It'll be new methods, I think. Should be
> ready to deploy by early next week.
>
jstrellner
Jan 23, 2009, 5:16:42 AM1/23/09
to Twitter Development Talk
I am really looking forward to this too. I have one request, can
there be a variable in the URL to specify that we want BOTH the
username and the user ID?
We need both of these to make sense for our users because we use the
ID in our backend, but display the username in the front end. Without
both, we'd still end up making another call to get that info.
-Joel
On Jan 22, 2:46 pm, Alex Payne <a...@twitter.com> wrote:
> I'm already working on this. It'll be new methods, I think. Should be
> ready to deploy by early next week.
>
there be a variable in the URL to specify that we want BOTH the
username and the user ID?
We need both of these to make sense for our users because we use the
ID in our backend, but display the username in the front end. Without
both, we'd still end up making another call to get that info.
-Joel
On Jan 22, 2:46 pm, Alex Payne <a...@twitter.com> wrote:
> I'm already working on this. It'll be new methods, I think. Should be
> ready to deploy by early next week.
>
Swap
Jan 23, 2009, 4:47:19 AM1/23/09
to Twitter Development Talk
Thank you, thank you!
On Jan 23, 3:46 am, Alex Payne <a...@twitter.com> wrote:
> I'm already working on this. It'll be new methods, I think. Should be
> ready to deploy by early next week.
>
On Jan 23, 3:46 am, Alex Payne <a...@twitter.com> wrote:
> I'm already working on this. It'll be new methods, I think. Should be
> ready to deploy by early next week.
>
Alex Payne
Jan 23, 2009, 1:19:00 PM1/23/09
to btan, Twitter Development Talk, twitter-ap...@googlegroups.com
The image link is a singleton tag; it does not require a closing tag.
You can request whitelisting at http://twitter.com/help/request_whitelisting.
On Fri, Jan 23, 2009 at 06:22, btan <btan...@gmail.com> wrote:
> Hi Alex,
>
> I am trying to read twitter search API. I am able to fetch XML feed
> and parse all elements of an entry except link href for html and
> image. I am using SimpleXML in php and curl to read XML feed and parse
> data. I can easily get author name, title, content because it has
> standard tag <title> .... </title>.
> I can not read image link because it does not have the same open and
> close tage for image link.
>
> So, could you please provide a link for image in you search API.
> Twitter provides an image profile entity in public status. I would
> also like to know how to add our IP in your whitelist.
>
> regards
> btan
> www.explorewww.com
btan
Jan 23, 2009, 9:22:04 AM1/23/09
to Twitter Development Talk, Alex Payne, twitter-ap...@googlegroups.com
Hi Alex,
I am trying to read twitter search API. I am able to fetch XML feed
and parse all elements of an entry except link href for html and
image. I am using SimpleXML in php and curl to read XML feed and parse
data. I can easily get author name, title, content because it has
standard tag <title> .... </title>.
I can not read image link because it does not have the same open and
close tage for image link.
So, could you please provide a link for image in you search API.
Twitter provides an image profile entity in public status. I would
also like to know how to add our IP in your whitelist.
regards
btan
www.explorewww.com
On Jan 20, 6:48 pm, "Alex Payne" <al...@twitter.com> wrote:
I am trying to read twitter search API. I am able to fetch XML feed
and parse all elements of an entry except link href for html and
image. I am using SimpleXML in php and curl to read XML feed and parse
data. I can easily get author name, title, content because it has
standard tag <title> .... </title>.
I can not read image link because it does not have the same open and
close tage for image link.
So, could you please provide a link for image in you search API.
Twitter provides an image profile entity in public status. I would
also like to know how to add our IP in your whitelist.
regards
btan
www.explorewww.com
On Jan 20, 6:48 pm, "Alex Payne" <al...@twitter.com> wrote:
Scott C. Lemon
Jan 23, 2009, 12:22:21 PM1/23/09
to Twitter Development Talk
Alex,
You rock ... thank you for supporting the community, and continuing to
improve the platform! This is awesome news! I'll look forward to
seeing the new methods announced ... and we'll see if we can keep you
from being deafened by the applause of the developer community when
that happens! :-)
Thanks again ...
Scott
On Jan 22, 3:46 pm, Alex Payne <a...@twitter.com> wrote:
> I'm already working on this. It'll be new methods, I think. Should be
> ready to deploy by early next week.
>
You rock ... thank you for supporting the community, and continuing to
improve the platform! This is awesome news! I'll look forward to
seeing the new methods announced ... and we'll see if we can keep you
from being deafened by the applause of the developer community when
that happens! :-)
Thanks again ...
Scott
On Jan 22, 3:46 pm, Alex Payne <a...@twitter.com> wrote:
> I'm already working on this. It'll be new methods, I think. Should be
> ready to deploy by early next week.
>
Karthik
Jan 29, 2009, 1:51:29 PM1/29/09
to Twitter Development Talk
Alex,
Any updates on these new API's?
We are eagerly awaiting for its release.
Thanks,
Karthik
Any updates on these new API's?
We are eagerly awaiting for its release.
Thanks,
Karthik
> I'm already working on this. It'll be new methods, I think. Should be
> ready to deploy by early next week.
>
> ready to deploy by early next week.
>
Alex Payne
Jan 29, 2009, 5:27:19 PM1/29/09
to twitter-deve...@googlegroups.com
Shipping tomorrow, Monday if it slips.
Message has been deleted
Sylvain Munaut
Feb 2, 2009, 2:24:21 PM2/2/09
to Twitter Development Talk
up
...
lots of people waiting :)
...
lots of people waiting :)
Reply all
Reply to author
Forward
This conversation is locked
You cannot reply and perform actions on locked conversations.
0 new messages