Following up on anti-abuse and API limits

[Alex Payne]

All,

I apologize for giving the impression that we were going to start
instituting opaque and undocumented anti-abuse measures. I assure you
that as soon as we've surveyed our options and decided what to put in
place that the limits will be communicated to the list and documented.

We're not trying to single out anyone on this list. We greatly
appreciate the work that all of you have done expanding Twitter's
reach to the various platforms you develop for. What we're trying to
prevent are malicious and/or careless uses of our service. If we
identify are particular registered API clients that are misbehaving,
we'll contact the developers before taking any drastic steps. We're
just trying to ensure that we're available enough that legitimate use
of our service can continue.

As for the proposed decreased authenticated requests per hour: I'm
open to suggestions. If 50 will put too much strain on your clients
and there's some other way that we can provide you with the data you
need while decreasing the overall number of requests for this six day
period, please let us know.

We're not trying to pull the wool over anyone's eyes or shortchange
you after you've put in hard work on your applications. We're always
open to alternatives, and we're always listening.

--
Alex Payne
http://twitter.com/al3x

[Marco Kaiser]

Alex,

thanks for clarifying this, really appreciated.

The problem with 50 req/hour limit for desktop client is that many users following a lot of others will miss tweets. All clients I know use a certain number of requests to refresh the user timeline, replies and direct messages, leaving some requests for actions triggered by the user directly.

Consider a scenario where, per hour, 30 reqs are used for the timeline, 10 for replies, 10 for DMs, leaving 20 to user initiated actions. By updating the timeline every two minutes, users with a few hundred friends will miss tweets from time to time. And from a client's perspective, we don't even know and can tell the user when we miss tweets. This is a bad UX.

If you lower the limit to 50, much more users will run into this. They will blame it on us (the client) first, then on twitter.

From my point of view, what we definitely need is not a higher limit in general. The most important thing that really needs more capacity are user-timeline requests. Don't know if this is technically (and regarding your server's capacity) possible, but it would help a lot.

-Marco

[Andrew Badera]

100% agree with the quote below.

And/or a per-client limit ... though I know, that would require some pretty radical backend work to transition into.

On 3/1/08, Marco Kaiser <kaiser...@gmail.com> wrote:

From my point of view, what we definitely need is not a higher limit in general. The most important thing that really needs more capacity are user-timeline requests. Don't know if this is technically (and regarding your server's capacity) possible, but it would help a lot.

-Marco

--
--Andy Badera
http://andrew.badera.us/
and...@badera.us
(518) 641-1280
Google me: http://www.google.com/search?q=andrew+badera

[Ed Finkler]

On Sat, Mar 1, 2008 at 3:32 PM, Marco Kaiser <kaiser...@gmail.com> wrote:

> Consider a scenario where, per hour, 30 reqs are used for the timeline, 10
> for replies, 10 for DMs, leaving 20 to user initiated actions. By updating
> the timeline every two minutes, users with a few hundred friends will miss
> tweets from time to time. And from a client's perspective, we don't even
> know and can tell the user when we miss tweets. This is a bad UX.

I would like this as well. I do think that the actual number of users
who follow this many people is small (based on info from the last
Twitter newsletter that only 10% of users follow more than 70 other
users); they just complain the most. 8)

But, yes, more *data* per request would really be awesome, especially
if the request limits have to be reduced.

--
Ed Finkler
http://funkatron.com
AIM: funka7ron
ICQ: 3922133
Skype: funka7ron

[Ed Costello]

Have you considered placing different limits based on the type of request?  A read request should be less intensive than a write request, and (knowing nothing about twitter's backend) you could pre-generate and cache frequent read requests.

--
-ed costello
Artific Industries, e...@artific.com, http://artific.com/

[Alex Payne]

Reads and writes have different costs in our system depending on how
active a given user and their followers/followed are. We cache
heavily at multiple layers as it stands. But yes, different limits
based on the type of request make sense, although they introduce
conceptual complexity for developers and users.

--
Alex Payne
http://twitter.com/al3x

[Marco Kaiser]

Great news that you are considering this.

From a developers position, I'd love to take the complexity on me and find a way to hide it from my users. The time will easily be saved by fewer support requests.

Thanks!

[Dossy Shiobara]

On 2008.03.01, Alex Payne <al...@al3x.net> wrote:
> As for the proposed decreased authenticated requests per hour: I'm
> open to suggestions. If 50 will put too much strain on your clients
> and there's some other way that we can provide you with the data you
> need while decreasing the overall number of requests for this six day
> period, please let us know.

Fewer requests would be okay if each request delivered more value.
i.e., how about a friends/followers API request that returned _all_
friends or followers data, not just 100 at a time--thus, chewing through
a user's rate limit quickly if they have a lot of friends or followers?

-- Dossy

--
Dossy Shiobara | do...@panoptic.com | http://dossy.org/
Panoptic Computer Network | http://panoptic.com/
"He realized the fastest way to change is to laugh at your own
folly -- then you can let go and quickly move on." (p. 70)

[Bluespark]

Hi Alex,

I think the problem lies in the aggressive caching of the
friends_timeline which forces clients using the API to do an
authentication to make sure that a fresh copy is sent down the line
every time.

If clients did not have to use an authenticated request to get the
friends_timeline then 20 API calls would be enough (well not to get
flamed here 50 is heaps). I would even go so far as that if the
friends_timeline feed cache expired in 60s to 90s then this would sync
nicely with clients because currently you cannot really make more than
1 request every 70s.

So my question is which is more expensive, basic authentication or
user_timeline DB request?

I have also tried to mention this before. But as developers how can
we also help twitter?
Tweetr does client side profile image caching and while S3 is dirt
cheap if nearly everyone did this it would save twitter a decent
amount of bandwidth. Even thinking outside the box, is XML the best
way to send data back?

It would also be cool if the since tag actually worked efficiently
enough at twitter to actually reduce load. I remember asking this
before and you said the load is about the same. If it was not the
same then only the most recent tweets would be returned which would
mean less data sent. And most of the time users only get 1-2 new
tweets every 5 mins so why cant we make this more efficient?

Hope we can find a solution. Cheers, John.

[Marco Kaiser]

Authenticated access to the friends_timeline is definitely required, otherwise friends from users with protected profiles wouldn't be available to clients.

[Marco Kaiser]

Argh! Should have been:

"... otherwise tweets from friends with protected profiles wouldn't be available to clients"