Heads up: per-IP rate limits for unauthenticated API requests are pending

265 views
Skip to first unread message

Alex Payne

unread,
Jul 10, 2008, 7:23:08 PM7/10/08
to twitter-deve...@googlegroups.com
As we hope you've noticed, the site has been feeling much snappier for
both web and API requests over the past several days, even despite the
increased rate limit. In our continued effort to keep things fast and
prevent abuse we're planning on introducing rate-limiting by IP for
unauthenticated API requests. We'll allow 100 unauthenticated
requests per IP per hour, just as we currently do with authenticated
requests.

Please let us know if you foresee any ghastly issues with this change.
It won't go into production until early next week at the soonest.

--
Alex Payne
http://twitter.com/al3x

Brett Morgan

unread,
Jul 10, 2008, 8:11:55 PM7/10/08
to twitter-deve...@googlegroups.com
How are you planning on dealing with people behind large ISP proxies?

--

Brett Morgan http://brett.morgan.googlepages.com/

Alex Payne

unread,
Jul 10, 2008, 8:16:49 PM7/10/08
to twitter-deve...@googlegroups.com
Well, we can't trust cookies or user agents, so that's a tough one.
We're basically assuming that if a given IP is hammering on us, it's
far more likely to be an abusive user than a bunch of users behind a
proxy. We've never once seen a bunch of users behind a proxy show up
in our abuse logs to the best of my knowledge.

Rob Iles

unread,
Jul 10, 2008, 9:55:52 PM7/10/08
to twitter-deve...@googlegroups.com
Hi Alex,

I'm new to Twitter, but have a lot of experience in administrating large scale web sites (commercial and otherwise).

Some of the issues you're likely to come across, as I believe Brett was alluding to, are the likes of AOL who implement "transparent" web proxies. Here in the UK, NTL (Cable provider) and others do a similar thing - automagically intercept all traffic on port 80, and proxy/cache it - no doubt with the intention of serving content more quickly and incurring fewer transit charges or somesuch.  The trouble is, the end user, the ISP's customer, is generally unaware of this - hence the name  "transparent proxy" - - with such malfeasants in the pipeline, it's entirely possible that you will see multiple hits from a single IP (that of the prox(y/ies)) - and may end up blocking "innocent" users.

It might be worth considering adding a Whitelist to your rate-limiter, so that you / the team can look at the IP's that are flooding you, and if they turn out you be "megaproxies", do $something_different; (either allow all, or be more generous).

As you'll have full access to all the data that's hammering you, perhaps some heuristic analysis could be employed, if it's the same username, it's clearly abuse, otherwise consider X, Y, Z etc...

Hope this is of some help, keep up the good work :)

Rob

 

2008/7/11 Alex Payne <al...@twitter.com>:



--


Rob Iles

RMIDevelopment

Web: http://www.rob-iles.co.uk/rmidevelopment
Twitter: http://twitter.com/Rob_Iles
Mobile: 079 6666 1092
Skype: rob_iles

Alex Payne

unread,
Jul 10, 2008, 9:58:56 PM7/10/08
to twitter-deve...@googlegroups.com
Thanks for the suggestions, Rob!

tweetip

unread,
Jul 10, 2008, 11:05:18 PM7/10/08
to Twitter Development Talk
> megaproxies

A few more examples

- AT&T iPhone
- Verizon cell phones
- etc


Cameron Kaiser

unread,
Jul 10, 2008, 11:45:20 PM7/10/08
to twitter-deve...@googlegroups.com
> > megaproxies
>
> A few more examples
>
> - AT&T iPhone

No. The AT&T network has multiple IP exit points, and the iPhone does not use
a proxy.

--
------------------------------------ personal: http://www.cameronkaiser.com/ --
Cameron Kaiser * Floodgap Systems * www.floodgap.com * cka...@floodgap.com
-- The only thing to fear is fearlessness -- R. E. M. -------------------------

tweetip

unread,
Jul 11, 2008, 12:22:52 AM7/11/08
to Twitter Development Talk
> No. The AT&T network has multiple IP exit points, and the iPhone does not use
> a proxy.

Our testing shows a range - as growth of the iPhone ramps up, it could
create an issue.

Kee Hinckley

unread,
Jul 11, 2008, 1:44:50 AM7/11/08
to twitter-deve...@googlegroups.com
On Jul 10, 2008, at 7:23 PM, Alex Payne wrote:
> unauthenticated API requests. We'll allow 100 unauthenticated
> requests per IP per hour, just as we currently do with authenticated
> requests.
>
> Please let us know if you foresee any ghastly issues with this change.
> It won't go into production until early next week at the soonest.

We are building a web-based twitter reader. In particular, that means
we are proxying all requests from the user (which is a win for you,
because we're caching things like friendship relationships, tweets
read by multiple users...). The calls we make that currently don't
require authentication (e.g. getting lists of who someone follows,
getting status on the "@foo" references in a tweet) are made on behalf
of multiple twitter users. We're going to blow past 100 requests per
hour without even trying.

So yes... I have ghastly issues with the change.

This isn't even just a matter of our exceeding the limits. Writing
code to work inside of them is as much of an issue. For instance, we
are caching tweets. When we fetched the tweet, we got the URL for the
user's image. But that gets stale. So we need to periodically query so
that when the user looks at old tweets, they don't get broken icons.
We can (and are) smart about updating the info if we see a new tweet
from the user, but that doesn't handle all the cases. Previously
getting that info was a "free" call. You're making it have a cost. So
now we need to figure out how to juggle those calls and spread them
out so we don't exceed the limit. It's a nice barrier-to-entry for
other developers, but frankly, I'd rather work on features. These
constant changes are making it very difficult to develop applications,
let alone plan ahead.

I also agree with others who point out that proxies and NATs are going
to cause problems for you. Although I actually suspect the major ones
won't be with ISPs, but with companies with multiple Twitter users.

One suggestion. It won't help us very much, but it might help some of
the other cases. That's to make the limit per-ip-per-user. Which is to
say, give a user 100 authenticated calls (current set), plus 100
didn't-have-to-be-authenticated-but-are calls (new set). Odds are that
most clients are making all on-behalf-of-a-user calls authenticated
anyway--even if they don't have to be. When that is the case, you
don't really care about the IP address.

Kee Hinckley
CEO/CTO Somewhere Inc.
Somewhere: http://www.somewhere.com/
TechnoSocial: http://xrl.us/bh35i
I'm not sure which upsets me more; that people are so unwilling to
accept responsibility for their own actions, or that they are so eager
to regulate those of everybody else.


Evan Weaver

unread,
Jul 11, 2008, 1:52:16 AM7/11/08
to twitter-deve...@googlegroups.com
Yes, authenticated users will not be limited by ip for any request.

Evan

--
Evan Weaver

E.B.

unread,
Jul 11, 2008, 2:01:04 AM7/11/08
to twitter-deve...@googlegroups.com
I think he means, 100 unathenticated per-ip-per-user.

If you can't do it, that's fine, we all just have to work something out via
par...@twitter.com as you suggested before.

Thanks for the heads up.

E.B.

Evan Weaver

unread,
Jul 11, 2008, 2:12:09 AM7/11/08
to twitter-deve...@googlegroups.com
That would be the same as 200 requests of any type per authenticated
user; we can probably reach that goal.

Evan

--
Evan Weaver

Richard

unread,
Jul 11, 2008, 11:35:01 AM7/11/08
to Twitter Development Talk
This is very short notice.

I can see going past this number easily. Is the limit 100/per hour
every hour? or is it an average over a day or a week?
Are you assuming everyone is running desktop clients with one user,
what about web apps?
Will it matter how many users you have?
does this mean we must stay a tiny site forever because I can never
request more than 100/hour?

Currently when a user joins FriendBinder (my site) we have to do a
fetch for each of a user's friends that have updated with in the last
2 weeks (typically most of them).
This can often be 1000 requests for one user.

An alternative to the method we use is to use the user_timeline
method, which requires authentication which is annoying because it
means we have to ask each user for their password and this puts many
users off. When is twitter going to support oauth or similar. I think
oauth support is needed before you make this change and others like it
that push people onto authenticated requests.

Also it seems that the user_timeline method has only started allowing
paging and decent sized pages (20 results vs. 200) on the 7th of July
and I have based code on the fact that I couldn't use the
user_timeline properly and now very quickly you are changing it so
effectively I can *only* use it.

Is the following issue going to be fixed before this limit comes into
place:
http://groups.google.com/group/twitter-development-talk/browse_thread/thread/6c1b09be52e4c591

I think you are changing the way the API works too quickly for people
to keep up and raise concerns, bare in mind we have to write, debug
and test code too.

Am I only the only one who sees major problems with this other than
the megaproxies issue?

jstrellner

unread,
Jul 11, 2008, 12:52:00 PM7/11/08
to Twitter Development Talk
Alex,

This change will probably take Twitturly offline in the first hour.
We use the http://twitter.com/statuses/user_timeline/USER.json call
exclusively to get information about a user, somewhere to the tune of
about 7,000 per hour, depending on how many tweets per hour have URLs
in them.

We have always identified our user agent as "Twitturly / v0.5" (or
whatever the current version at that time is).

We usually run multiple servers to do the updates, but it would be way
to prohibitive to run 70 of them, especially when we have gotten our
system optimized to the point that we can run fine on one, or two
update server when in the middle of a peak.

Are you going to allow exceptions to this new rule?

Sincerely,
Joel Strellner

Evan Weaver

unread,
Jul 11, 2008, 1:08:26 PM7/11/08
to twitter-deve...@googlegroups.com
We will allow some exceptions, but this way at least we will know
about them, instead of getting hammered by anyone who wants to do 7000
requests an hour without warning.

Can't you get that user data inline from the tweets?

Evan

--
Evan Weaver

jungle

unread,
Jul 11, 2008, 2:14:57 PM7/11/08
to Twitter Development Talk
Both our sites, twist.flaptor.com and twittersearch.flaptor.com would
also be shut down by this.
If we can't get into the exception list for the public timeline either
through the API or the Jabber feed, we're history.
Please advise.

On Jul 11, 1:52 pm, jstrellner <jstrell...@urltrends.com> wrote:
> Alex,
>
> This change will probably take Twitturly offline in the first hour.
> We use thehttp://twitter.com/statuses/user_timeline/USER.jsoncall
> exclusively to get information about a user, somewhere to the tune of
> about 7,000 per hour, depending on how many tweets per hour have URLs
> in them.
>
> We have always identified our user agent as "Twitturly / v0.5" (or
> whatever the current version at that time is).
>
> We usually run multiple servers to do the updates, but it would be way
> to prohibitive to run 70 of them, especially when we have gotten our
> system optimized to the point that we can run fine on one, or two
> update server when in the middle of a peak.
>
> Are you going to allow exceptions to this new rule?
>
> Sincerely,
> Joel Strellner
>

Evan Weaver

unread,
Jul 11, 2008, 2:45:30 PM7/11/08
to twitter-deve...@googlegroups.com
What if we exempted *only* the public timeline?

Evan

--
Evan Weaver

tweetip

unread,
Jul 11, 2008, 3:06:10 PM7/11/08
to Twitter Development Talk
> What if we exempted *only* the public timeline?

We feel twitter assumes we developers don't include the timeline when
discussing rate limits. But with the Jabber fiasco, developers feel
out of the loop. Ask Alex about our endless emails this week wondering
what's going on :)

Michael

jstrellner

unread,
Jul 11, 2008, 3:09:56 PM7/11/08
to Twitter Development Talk
Evan,

I posted it earlier, but it doesn't look like it took.

We need that call primarily for spam reasons. We analyze their past
tweets as well as the information provided about the user in that call
to see if the current tweet that we are parsing is or could be spam.

We currently get the information from Summize to ease the load on the
twitter servers. We do not currently have access to the XMPP feed so
Summize was needed and in a lot of ways better for us since they can
add filters and we don't need to parse tons of tweets that we would
just have to ignore. The one downfall to using Summize that we found
was that they do not return the twitter user id, but an internal one
that they have assigned to that user. This call helps resolve that
issue too, without making any additional calls to the API.

Without this call, Twitturly would have a lot of spam showing in our
results and wouldn't be nearly as useful for our users.

FYI: you can see what Twitturly does by going here: http://twitturly.com

Sincerely,
Joel Strellner


On Jul 11, 10:08 am, "Evan Weaver" <ewea...@twitter.com> wrote:
> We will allow some exceptions, but this way at least we will know
> about them, instead of getting hammered by anyone who wants to do 7000
> requests an hour without warning.
>
> Can't you get that user data inline from the tweets?
>
> Evan
>
>
>
> On Fri, Jul 11, 2008 at 12:52 PM, jstrellner <jstrell...@urltrends.com> wrote:
>
> > Alex,
>
> > This change will probably take Twitturly offline in the first hour.
> > We use thehttp://twitter.com/statuses/user_timeline/USER.jsoncall
> > exclusively to get information about a user, somewhere to the tune of
> > about 7,000 per hour, depending on how many tweets per hour have URLs
> > in them.
>
> > We have always identified our user agent as "Twitturly / v0.5" (or
> > whatever the current version at that time is).
>
> > We usually run multiple servers to do the updates, but it would be way
> > to prohibitive to run 70 of them, especially when we have gotten our
> > system optimized to the point that we can run fine on one, or two
> > update server when in the middle of a peak.
>
> > Are you going to allow exceptions to this new rule?
>
> > Sincerely,
> > Joel Strellner
>

Evan Weaver

unread,
Jul 11, 2008, 3:19:08 PM7/11/08
to twitter-deve...@googlegroups.com
No, you're right that we're talking about limiting every request.

Evan

--
Evan Weaver

tweetip

unread,
Jul 11, 2008, 4:14:02 PM7/11/08
to Twitter Development Talk
Evan & Alex,

Fwiw, here's how each install of our desktop app plans to use the API:

- we poll the api up to 75 times/hour for non-public timeline data
(100 is plenty and will create a near realtime feel)

- we poll the api public timeline approx 3000 times/hour for client
specific mining. As tweet volume is increasing 5% per day via the api,
we'll need to hit the api more or fail under current api restrictions,
which is the since_id only returns 20 tweets. We stayed away from
coding for jabber - and hope we can continue to do what we need via
the api. Additionally the Summize data feed does not fulfill our
needs. This rate issue may come down to us setting up our own servers
for our clients to thump. But even that plan may be restricted in the
future as twitter refines their data use policy. Last night, we made a
decision to decrease our development efforts until we see concrete
policies coming from the biz side of twitter. We feel, for the first
time, what we're doing may not be ultimately acceptable to twitter.
And we're in final beta stage...

- Iow, we view twitter tech issues and twitter biz issues out of sync
with each other - enough so to cause us significant concern. This
doesn't imply we're mad - we rode into this corner of mirrors
voluntarily, knowing twitter policies on the api and the data would
change.

hth

Michael :)
Moab

Stut

unread,
Jul 11, 2008, 4:27:55 PM7/11/08
to twitter-deve...@googlegroups.com
On 11 Jul 2008, at 21:14, tweetip wrote:
> we poll the api public timeline approx 3000 times/hour for client
> specific mining.

Have you actually confirmed that you get more than 1200 tweets an hour
by doing that? Last month Alex confirmed[1] that they cache the public
timeline API response every 60 seconds so it shouldn't be possible to
get more than 1200 an hour and hitting it 3000 times is a massive
waste of your users resources as well as Twitter's.

Alex/Evan: Any word on when the Jabber feed will be open to all?

-Stut

--
http://stut.net/

[1] http://groups.google.com/group/twitter-development-talk/browse_thread/thread/f881564598a947a7/c5ee88b03b8d7faf?lnk=gst&q=waste+of+resources#c5ee88b03b8d7faf

Evan Weaver

unread,
Jul 11, 2008, 4:38:33 PM7/11/08
to twitter-deve...@googlegroups.com
There are many ways to bypass the cache right now. Most of them are
going away. After that we might speed up the timeout on the public
timeline cache to 15 or 30 seconds.

However, please note that the public timeline does not have all tweets
even when uncached.

Sorry for the disconnect. I definitely understand your frustration.

Evan

--
Evan Weaver

tweetip

unread,
Jul 11, 2008, 5:11:29 PM7/11/08
to Twitter Development Talk
Stut,

We're whitelisted - perhaps helping us bypass the cache? Today, at the
current rate, we'll filter & save 600,000+ tweets in realtime. Both
twitter & their vc's are aware of what we're doing. Here's a chart
from a few days ago:

http://tweetip.us/lkxh3

fyi, summize also accesses the api pub timeline when jabber is down.

Michael


ps: Evan-ok-thanks-we'll quietly wait for things to shut off :) and
for now shelving eight months of work :(

Evan Weaver

unread,
Jul 11, 2008, 5:15:53 PM7/11/08
to twitter-deve...@googlegroups.com
On Fri, Jul 11, 2008 at 5:11 PM, tweetip <twe...@mac.com> wrote:
> ps: Evan-ok-thanks-we'll quietly wait for things to shut off :) and
> for now shelving eight months of work :(

No, don't get discouraged. That's why we're having this conversation
before and not after.

However, I'm confused as to whether you guys think that you're getting
a full tweet stream or not.

Evan

--
Evan Weaver

tweetip

unread,
Jul 11, 2008, 5:35:08 PM7/11/08
to Twitter Development Talk
> However, I'm confused as to whether you guys think that you're getting
> a full tweet stream or not.

Evan,

We're not discouraged, just sad. The technology we've developed is the
value and can point at whatever data river.

We know we don't receive the full stream. In our discussions with
Summize, we've compared tweet count and they've said we're within a
few percent (not counting Asia). We've documented when we miss mission
critical tweets. To resolve this, we've lobbied for the api (not
jabber) to return as much as possible via since_id. For us, Jabber is
un-needed overkill.

Our understanding of "whitelisting" until recently: twitter would
allow Source "tweetip" to poll the public timeline as needed. Iow, not
based on IP or screenname, but on Source.

Michael

Evan Weaver

unread,
Jul 11, 2008, 5:50:17 PM7/11/08
to twitter-deve...@googlegroups.com
It looks to me like the Summize counts you got are wrong.

I am going to talk to Evan Williams about this and get back to you.

Evan

--
Evan Weaver

tweetip

unread,
Jul 11, 2008, 6:05:59 PM7/11/08
to Twitter Development Talk
> It looks to me like the Summize counts you got are wrong.

Evan,

Perhaps Summize was comparing their count from when they switch to the
api when jabber is offline. We've felt we're close to Summize because
of audits we do in major events, like the China quake. We recorded
tweets that Summize missed or didn't index. Also - we're counting
English tweets - not Asia nor Spanish nor ?

hope this convo helps.

Michael :)

Jode...@gmail.com

unread,
Jul 12, 2008, 7:16:57 AM7/12/08
to Twitter Development Talk, Evan Weaver, Andrew Maizels
Alex
As we do all our Twitter client requests server side this will blow us
out of the water.
We would have to rewrite our code so that all our client side code
makes direct calls on Twitter. This would exponentially increase API
call ie no caching on our side.
Regards

Andrew Maizels

unread,
Jul 12, 2008, 11:44:51 AM7/12/08
to Jode...@gmail.com, Twitter Development Talk, Evan Weaver
Here's what Evan said in that thread:


We will allow some exceptions, but this way at least we will know
about them, instead of getting hammered by anyone who wants to do 7000
requests an hour without warning.

Sounds reasonable.

Andrew M.

Chris Meller

unread,
Jul 12, 2008, 12:46:51 PM7/12/08
to twitter-deve...@googlegroups.com
Just out of curiosity, has anyone actually checked to see if a problem exists with people hammering the API in such a manner? All the comments have seemed along the lines of "at least we'll know if someone's hammering us", making it sound as if there's no clear idea of whether or not the problem exists yet.

I have no stake in the matter either way, as I just started lurking here to keep up with the API limit changes, I'm just curious.

Oh yeah, and supporting some form of authentication that doesn't require a user to hand their password over to a 3rd party service would be kick ass - for both users and developers. I don't mind giving my local copy of twhirl my password, but giving a 3rd party web service that has to store that password somewhere (hopefully securely) is different...

Alex Payne

unread,
Jul 12, 2008, 9:22:47 PM7/12/08
to twitter-deve...@googlegroups.com
Yes, we know that people hammering on our API - where it defeats a
reasonable caching strategy - is tough on our service.

There's been a lot of discussion around providing an alternative to
password-based authentication. Please search the group of "OAuth".
It's coming later this year.

--
Alex Payne
http://twitter.com/al3x

tweetip

unread,
Jul 13, 2008, 1:27:54 AM7/13/08
to Twitter Development Talk
I was asked to clarify my comments on another blog, and since no one
may ever see them I'll repost here. If twitter changes their minds and
has no desire to become a highly reliable worldwide comm utility, then
my comments are 100% invalid.

---
from a mission critical developer/end user point of view, and twitter
being a ---> worldwide comm utility <---

by not drafting an advanced specification, twitter failed to insure
the integrity of the data. Mobile app developers just started pumping
in whatever format/values they desired. This data "breaks" our app, if
we are filtering values based on historical analysis as is the case of
First Responder / Earthquake / Tornado / Emergency Services or for
that matter Market Movement / Breaking News / any realtime anything
where we feel Location is integral to our analysis.
---

Michael

tweetip

unread,
Jul 13, 2008, 1:44:20 AM7/13/08
to Twitter Development Talk
sorry - wrong forum - if this was disqus, i could just edit that last
post and skip happily away :) sort of

Jesse Stay

unread,
Jul 13, 2008, 2:07:46 AM7/13/08
to twitter-deve...@googlegroups.com

On Thu, Jul 10, 2008 at 5:23 PM, Alex Payne <al...@twitter.com> wrote:

As we hope you've noticed, the site has been feeling much snappier for
both web and API requests over the past several days, even despite the
increased rate limit.  In our continued effort to keep things fast and
prevent abuse we're planning on introducing rate-limiting by IP for
unauthenticated API requests.  We'll allow 100 unauthenticated
requests per IP per hour, just as we currently do with authenticated
requests.

What is the limit on authenticated requests?  I have to go through up to thousands of requests in a given hour just doing requests for a user's friends and follow requests in my socialtoo.com autofollow script - thus far it seems to be working.  Is the limit for authenticated users higher?  Or is this a really recent change and I just haven't noticed an issue yet?

Thanks,

Jesse

Dossy Shiobara

unread,
Jul 13, 2008, 10:02:16 AM7/13/08
to twitter-deve...@googlegroups.com
tweetip wrote:
[...]

> by not drafting an advanced specification, twitter failed to insure
> the integrity of the data. Mobile app developers just started pumping
> in whatever format/values they desired. This data "breaks" our app, [...]

Postel's Law: "Be conservative in what you do, be liberal in what you
accept from others."

If the data "breaks" your app, then your application is broken: it's a
defect in your app, and you should fix it.

--
Dossy Shiobara | do...@panoptic.com | http://dossy.org/
Panoptic Computer Network | http://panoptic.com/
"He realized the fastest way to change is to laugh at your own
folly -- then you can let go and quickly move on." (p. 70)

tweetip

unread,
Jul 13, 2008, 11:04:21 AM7/13/08
to Twitter Development Talk
> If the data "breaks" your app, then your application is broken: it's a
> defect in your app, and you should fix it.

Dossy,

There's simple elegant truth in that statement. But using this
reasoning in the real world of a communication utility just killed
someone, and/or created the juggernaut of legal reprisal. Many may not
understand this issue, but only the future will prove us right or
wrong.

Andrew Badera

unread,
Jul 13, 2008, 11:36:45 AM7/13/08
to twitter-deve...@googlegroups.com

tweetip

unread,
Jul 13, 2008, 11:45:22 AM7/13/08
to Twitter Development Talk
Andrew,

In five years, neither you nor I will have any influence on the
direction of twitter, nor will most of us chatting be developing for
twitter.

But right here, right now, people are listening.

Just state your opinion - leave it to others to compare.

thanks

funkatron

unread,
Jul 13, 2008, 6:14:02 PM7/13/08
to Twitter Development Talk
Will the "help/test" method be rate limited? If so, can it not be?

--
Ed Finkler
http://funkatron.com
AIM: funka7ron
ICQ: 3922133
Skype: funka7ron


On Jul 10, 7:23 pm, "Alex Payne" <a...@twitter.com> wrote:
> As we hope you've noticed, the site has been feeling much snappier for
> both web and API requests over the past several days, even despite the
> increased rate limit.  In our continued effort to keep things fast and
> prevent abuse we're planning on introducing rate-limiting by IP for
> unauthenticated API requests.  We'll allow 100 unauthenticated
> requests per IP per hour, just as we currently do with authenticated
> requests.
>

Dossy Shiobara

unread,
Jul 13, 2008, 6:15:34 PM7/13/08
to twitter-deve...@googlegroups.com
funkatron wrote:
> Will the "help/test" method be rate limited? If so, can it not be?

+1.

jstrellner

unread,
Jul 14, 2008, 11:24:16 AM7/14/08
to Twitter Development Talk
Alex,

I don't believe it has been mentioned yet, but the following questions
still remain:

- When will this new rate limiting go into effect?
- What will the process and requirements be to get approved for a
higher limit?

Sincerely,
Joel Strellner

Evan Weaver

unread,
Jul 14, 2008, 11:26:14 AM7/14/08
to twitter-deve...@googlegroups.com
If I can answer for Alex,

1. No timeline has been decided yet.

2. We're working on a better, clearer process for API consumer rate
exceptions. Because it requires coordination with business
development, it might be another week or two at least.

Evan

--
Evan Weaver

jstrellner

unread,
Jul 14, 2008, 11:31:23 AM7/14/08
to Twitter Development Talk
Thank you Evan. I look forward to hearing the outcome of your
internal discussions.

-Joel

tweetip

unread,
Jul 14, 2008, 11:58:50 AM7/14/08
to Twitter Development Talk
Evan,

> Because it requires coordination with business development...


A very big step towards that "worldwide comm utility".

thanks,

M :)

jstrellner

unread,
Jul 16, 2008, 9:16:10 PM7/16/08
to Twitter Development Talk
Evan / Alex,

What happened to future notice?

I started getting the following error a little bit ago:
"Rate limit exceeded. Clients may not make more than 100 requests per
hour."

Also, what happened to allowing us to get permission for additional
queries?

It is quite frustrating that you guys made this big of a change
without letting us know that it was going live...

-Joel


On Jul 14, 8:26 am, "Evan Weaver" <ewea...@twitter.com> wrote:

Jesse Stay

unread,
Jul 16, 2008, 9:26:56 PM7/16/08
to twitter-deve...@googlegroups.com
Again, this is where I think a QA environment that we could test our stuff in before you push would be extremely helpful, along with a regular push schedule.  It should be simply a matter of one additional step of pushing things out live, unless you guys are developing on your production servers!

Jesse

Ed Finkler

unread,
Jul 17, 2008, 12:19:07 AM7/17/08
to twitter-deve...@googlegroups.com
On Wed, Jul 16, 2008 at 9:16 PM, jstrellner <jstre...@urltrends.com> wrote:
>
> Evan / Alex,
>
> What happened to future notice?
>
> I started getting the following error a little bit ago:
> "Rate limit exceeded. Clients may not make more than 100 requests per
> hour."
>

Be *really* sure the error refers to unauthenticated requests. That's
exactly the message you get if you exceed the 100/hr limit on
*authenticated* requests. I suspect you're mistaken.

jstrellner

unread,
Jul 17, 2008, 12:25:46 AM7/17/08
to Twitter Development Talk
I am positive that these are unauthenticated requests. They flipped
the switch on at 5pm pst time.

Take a look at our queues here once they did that:
http://twitturly.com/status/

I am not mistaken.

-Joel


On Jul 16, 9:19 pm, "Ed Finkler" <funkat...@gmail.com> wrote:
> On Wed, Jul 16, 2008 at 9:16 PM, jstrellner <jstrell...@urltrends.com> wrote:
>
> > Evan / Alex,
>
> > What happened to future notice?
>
> > I started getting the following error a little bit ago:
> > "Rate limit exceeded. Clients may not make more than 100 requests per
> > hour."
>
> Be *really* sure the error refers to unauthenticated requests. That's
> exactly the message you get if you exceed the 100/hr limit on
> *authenticated* requests. I suspect you're mistaken.
>
> --
> Ed Finklerhttp://funkatron.com

Richard

unread,
Jul 17, 2008, 6:37:31 AM7/17/08
to Twitter Development Talk
Is this just on public_timeline method?

jstrellner

unread,
Jul 17, 2008, 8:18:43 AM7/17/08
to Twitter Development Talk
We are seeing it on the following call:

http://twitter.com/statuses/user_timeline/TWITTER USERNAME.json

The twitter username is different each time. They appear to be doing
it for ALL unauthenticated calls now, not just the public timeline.

-Joel

Funkatron

unread,
Jul 17, 2008, 9:49:39 AM7/17/08
to twitter-deve...@googlegroups.com
Sorry, I shouldn't have been so presumptive. I would have liked to at
least seen a "hey, we turned it on" post here, yeah.

- Ed Finkler

On Jul 17, 2008, at 8:18 AM, jstrellner <jstre...@urltrends.com>
wrote:

Alex Payne

unread,
Jul 17, 2008, 1:19:25 PM7/17/08
to twitter-deve...@googlegroups.com
Sorry, I've been a bit out-of-sync with our deployment schedule, so I
wasn't sure it was going out late yesterday, PDT. My apologies -
we'll be sure to send notice of future changes, although this is in
keeping with the "next week" timeline that I mentioned last week.

--
Alex Payne
http://twitter.com/al3x

Jason Follas

unread,
Jul 17, 2008, 4:03:11 PM7/17/08
to Twitter Development Talk
So this explains why any mashup created on Popfly that uses a Twitter
endpoint no longer works. Let the whitelisting begin!


On Jul 17, 1:19 pm, "Alex Payne" <a...@twitter.com> wrote:
> Sorry, I've been a bit out-of-sync with our deployment schedule, so I
> wasn't sure it was going out late yesterday, PDT.  My apologies -
> we'll be sure to send notice of future changes, although this is in
> keeping with the "next week" timeline that I mentioned last week.
>
>
>
>
>
> On Thu, Jul 17, 2008 at 6:49 AM, Funkatron <funkat...@gmail.com> wrote:
>
> > Sorry, I shouldn't have been so presumptive. I would have liked to at least
> > seen a "hey, we turned it on" post here, yeah.
>
> > - Ed Finkler
>
> Alex Paynehttp://twitter.com/al3x- Hide quoted text -
>
> - Show quoted text -

jstrellner

unread,
Jul 18, 2008, 1:55:07 AM7/18/08
to Twitter Development Talk
Yeah, good luck with that Jason.

We have been trying all day to get someone at Twitter to re-enable
us. This morning @biz said that Twitter white-listed us, but it
didn't seem to work. I let them know shortly after he posted it that
we were still getting blocked but no one has responded, despite dozens
of tweets from myself and our users asking them to re-enable us.

Twitturly is pretty much dead now and many other apps either already
are too, or they are going there very quickly.

-Joel
> > Alex Paynehttp://twitter.com/al3x-Hide quoted text -

jungle

unread,
Jul 18, 2008, 5:38:42 AM7/18/08
to Twitter Development Talk
Yes, and the silence from Twitter staff regarding this is worrying to
say the least. Alex and Evan are always there helping with all sorts
of questions, but this time, it seems, they got orders from above. And
that means this is a business decision, not a technical one. And it
all started roughly around the time Twitter acquired Summize. Every
app that remotely does something similar is doomed. That means search,
datamining, trends analysis, following conversations, whatever. The
whole ecosystem that formed around stuff that Twitter didn't do itself
has been considered dangerous competition.

So, fellas, we're pretty much screwed.
> > > Alex Paynehttp://twitter.com/al3x-Hidequoted text -

Ed Finkler

unread,
Jul 18, 2008, 1:46:33 PM7/18/08
to twitter-deve...@googlegroups.com
On Fri, Jul 18, 2008 at 5:38 AM, jungle <jha...@gmail.com> wrote:
>
> Yes, and the silence from Twitter staff regarding this is worrying to
> say the least. Alex and Evan are always there helping with all sorts
> of questions, but this time, it seems, they got orders from above.

That's pure speculation. Leave it.

--
Ed Finkler

Dossy Shiobara

unread,
Jul 18, 2008, 1:49:12 PM7/18/08
to twitter-deve...@googlegroups.com
Ed Finkler wrote:
> On Fri, Jul 18, 2008 at 5:38 AM, jungle <jha...@gmail.com> wrote:
>> Yes, and the silence from Twitter staff regarding this is worrying to
>> say the least. Alex and Evan are always there helping with all sorts
>> of questions, but this time, it seems, they got orders from above.
>
> That's pure speculation. Leave it.

Hanlon's razor: "Never attribute to malice that which can be adequately
explained by stupidity."

:-)

Kent Brewster

unread,
Jul 18, 2008, 1:40:13 PM7/18/08
to Twitter Development Talk
Sigh ... Twitterati is broken, again. Any chance you guys could
whitelist pipes.yahoo.com?

--Kent

Alex Payne

unread,
Jul 18, 2008, 2:08:52 PM7/18/08
to twitter-deve...@googlegroups.com
I'm kind of confused by language about "re-enabling". We haven't
*disabled* anyone's access to the RESTful API. If you need to be
whitelisted in order for your application to work, please fill out
http://twitter.com/help/request_whitelisting and put in a DETAILED
description of the API methods you call, how often you expect to call
them, etc. What we've done is change our code to enforce the
documented cache policies that have been in place for over a year.

We'll be announcing our partnership with Gnip today or early next
week, so that's another option for keeping up to date with the
activities of a large set of Twitter users. They expect to be able to
relay our XMPP feed in the near future as well, though that will
require an agreement signed by Twitter, Gnip, and the developer
consuming the feed.

Evan and I are engineers at Twitter; our job is to support the
technology, and by extension to support this community. While Twitter
has a fairly open internal conversation about our business direction,
at the end of the day we're not the ones making those decisions.
Clearly, there are some applications that directly compete with
Summize, now Twitter Search. The entire ecosystem of applications
that do "datamining, trends analysis, following conversations" and so
forth is not doomed, however.

When Apple and Intel form a partnership, they don't do so via
conversations on public forums. If you're trying to build a business
atop the Twitter API, the appropriate course of action is to talk to
us privately about an agreement between our two businesses so we can
provide you with the support and quality of service that your
application requires.

We'll be writing more on this topic at the Twitter Technology Blog.

--
Alex Payne
http://twitter.com/al3x

Alex Payne

unread,
Jul 18, 2008, 2:25:34 PM7/18/08
to twitter-deve...@googlegroups.com
We don't whitelist domains, IP addresses, countries, townships, or
potluck groups. Just users. Only users.

--
Alex Payne
http://twitter.com/al3x

jungle

unread,
Jul 18, 2008, 2:27:36 PM7/18/08
to Twitter Development Talk
On Jul 18, 2:49 pm, Dossy Shiobara <do...@panoptic.com> wrote:
> Hanlon's razor: "Never attribute to malice that which can be adequately
> explained by stupidity."

One thing I'm certain about is that they're not stupid, which doesn't
mean I think there's malice involved.
In fact, here's confirmation that this is a business decision:
http://tinyurl.com/68jfq8 [techcrunch]

jungle

unread,
Jul 18, 2008, 2:41:33 PM7/18/08
to Twitter Development Talk
On Jul 18, 3:08 pm, "Alex Payne" <a...@twitter.com> wrote:
> When Apple and Intel form a partnership, they don't do so via
> conversations on public forums.

I'm just glad that moving this conversation to the public forum
produced some answer. Of course I'll be glad to continue privately
with whom you consider appropiate.

Steve Brunton

unread,
Jul 18, 2008, 3:04:52 PM7/18/08
to twitter-deve...@googlegroups.com
On Fri, Jul 18, 2008 at 2:08 PM, Alex Payne <al...@twitter.com> wrote:
>
> We'll be announcing our partnership with Gnip today or early next
> week, so that's another option for keeping up to date with the
> activities of a large set of Twitter users. They expect to be able to
> relay our XMPP feed in the near future as well, though that will
> require an agreement signed by Twitter, Gnip, and the developer
> consuming the feed.
>

Not wanting to beat a dead horse, but I just took a look at the Gnip
API. It looks like that is just a feed of the Activity events and not
a full feed. The developer will still have to go back to the Twitter
API to get the full body of the tweet. This would still run up against
the API Rate limit either authenticated or unauthenticated correct?

-steve

Alex Payne

unread,
Jul 18, 2008, 3:10:54 PM7/18/08
to twitter-deve...@googlegroups.com
Indeed, it's Gnip's purpose to provide a mechanism for social content
that's similar to that of a blog ping service. It still requires
talking to our API, but ideally much less so, as Gnip can alert you
when a particular user your application cares about has updated. Less
polling, less likelihood to run up against our rate limits - or, an
easier justification for us to whitelist you.

--
Alex Payne
http://twitter.com/al3x

Richard

unread,
Jul 18, 2008, 3:25:28 PM7/18/08
to Twitter Development Talk
Whilst I agree that Twitter has to limit it's API in certain ways, I
think the release of this could have been handled better.

It is possible next time detrimental changes are made to the API it
could be:
- announced properly (i.e. on the blog)
- given adequate notice (at least 2 weeks IMHO - us consumers are
coding too and would like to test too)

Also given the Gnip deal was coming up should this not have been
delayed until after that had been announced?

Also, I would like to note that I expect if you ask the consumers of
your API, they probably have ideas on how they could reduce the number
of requests used by their application (I know I do) by additions to
methods available.

On Jul 18, 8:10 pm, "Alex Payne" <a...@twitter.com> wrote:
> Indeed, it's Gnip's purpose to provide a mechanism for social content
> that's similar to that of a blog ping service. It still requires
> talking to our API, but ideally much less so, as Gnip can alert you
> when a particular user your application cares about has updated. Less
> polling, less likelihood to run up against our rate limits - or, an
> easier justification for us to whitelist you.
>
>
>
> On Fri, Jul 18, 2008 at 12:04 PM, Steve Brunton <sbrun...@gmail.com> wrote:

Richard

unread,
Jul 19, 2008, 11:29:37 AM7/19/08
to Twitter Development Talk
Under these new per-IP limits what is the suggested way to get all of
the people someone is friends or followers? Since these methods are
limited to 100 results per request and it's not uncommon to encounter
people with more that 10,000 friends, followers or both. i.e.
100 friends * 100 requests = 10,000 people
hits the limit right away.

Kent Brewster

unread,
Jul 19, 2008, 12:44:22 PM7/19/08
to Twitter Development Talk
Last time this happened (May 31st) I requested that future API
deprecations be announced on the Twitter development blog, and got a
positive response from Alex:

http://groups.google.com/group/twitter-development-talk/msg/4406fae97a5a21dc

I'm going to guess that this isn't going to happen, simply because it
would be bad public relations.

--Kent

Richard

unread,
Jul 19, 2008, 2:07:22 PM7/19/08
to Twitter Development Talk
Well the current situation is bad for public relations IMHO. Reading
every post on this group in case there is announcement is a bit of
bind - especially if like me you have several other APIs to keep up
with equally well.

On Jul 19, 5:44 pm, Kent Brewster <kentb...@gmail.com> wrote:
> Last time this happened (May 31st) I requested that future API
> deprecations be announced on the Twitter development blog, and got a
> positive response from Alex:
>
> http://groups.google.com/group/twitter-development-talk/msg/4406fae97...
--
Richard
friendbinder.com
Reply all
Reply to author
Forward
0 new messages