abusive Twitter API usage question

[Dossy Shiobara]

I just want to come out and ask: is Twitter Karma one of those "abusive"
Twitter API users? I know lots of folks use the "bulk follow" option to
hammer Twitter with lots of follow requests. However, I was assuming
that the really low 70 req/hour rate limit would ensure that this was an
okay use of the API.

--
Dossy Shiobara | do...@panoptic.com | http://dossy.org/
Panoptic Computer Network | http://panoptic.com/
"He realized the fastest way to change is to laugh at your own
folly -- then you can let go and quickly move on." (p. 70)

[Marco Kaiser]

Dossy,

I don't know if this is of interest for you, but: some twhirl users reported that regularly run into limit exceeded problem after visiting TwitterKarma. I guess users of other desktop clients will experience this, too.

If it's not already there, maybe you can add a hint that using your site/service can lead to these problems?

-Marco

[Ed Finkler]

On Sat, Mar 1, 2008 at 2:29 PM, Dossy Shiobara <do...@panoptic.com> wrote:
>
> I just want to come out and ask: is Twitter Karma one of those "abusive"
> Twitter API users? I know lots of folks use the "bulk follow" option to
> hammer Twitter with lots of follow requests. However, I was assuming
> that the really low 70 req/hour rate limit would ensure that this was an
> okay use of the API.

"If you've been scraping Twitter or consuming
public API feeds unfairly..."

Doesn't seem like you would be, unless you're doing screen scraping or
hitting the public feeds (I'm reading this as "the public timeline")
excessively.

--
Ed Finkler
http://funkatron.com
AIM: funka7ron
ICQ: 3922133
Skype: funka7ron

[Dossy Shiobara]

On 2008.03.01, Marco Kaiser <kaiser...@gmail.com> wrote:
> I don't know if this is of interest for you, but: some twhirl users
> reported that regularly run into limit exceeded problem after visiting
> TwitterKarma. I guess users of other desktop clients will experience
> this, too.

Sure, since the rate limit is per user, not per application that uses
the API. Twitter Karma is a really quick way to burn through your rate
limit, for sure. :-)

> If it's not already there, maybe you can add a hint that using your
> site/service can lead to these problems?

I probably should ... but it's hard enough to explain to people how the
Twitter API rate limit works that I'm afraid that adding verbiage trying
to explain it will simply cause more confusion than resolve it.

-- Dossy

[Marco Kaiser]

On Sat, Mar 1, 2008 at 8:53 PM, Dossy Shiobara <do...@panoptic.com> wrote:

> If it's not already there, maybe you can add a hint that using your
> site/service can lead to these problems?

I probably should ... but it's hard enough to explain to people how the
Twitter API rate limit works that I'm afraid that adding verbiage trying
to explain it will simply cause more confusion than resolve it.

Whom do you tell? :-) I have to do this on a daily basis, as my twhirl users usually complain to me, not twitter :-)

By not explaining it to your users, you delegate resolving the confusion to other client developers. And desktop clients give a very bad user experience when limit is exceeded, as users are locked out for up to an hour :-/

[Alex Payne]

I don't have data on hand that singles out one client or IP from
another; that's part of the anti-abuse measures we're hoping to put
in. I'm going to address this again in another email to the list.

--
Alex Payne
http://twitter.com/al3x

[Ed Finkler]

I'd agree with MAK here. Users are REALLY confused by the request
limit, and error messages that explain what's happening (maybe linking
to a little FAQ pages on the limits and what can cause them) would go
a long way to reducing the # of questions and complaints. It's one of
the things I need to address in Spaz, for sure.

[Marco Kaiser]

FYI, I had written a post on twhirl's blog addressing this issue. Usually, this helps twhirl's users to understand the problem.

http://blog.twhirl.org/2008/01/04/pushing-the-limit/

[Dossy Shiobara]

On 2008.03.01, Ed Finkler <funk...@gmail.com> wrote:
> On Sat, Mar 1, 2008 at 2:29 PM, Dossy Shiobara <do...@panoptic.com> wrote:
> > I just want to come out and ask: is Twitter Karma one of those "abusive"

...

>
> Doesn't seem like you would be, unless you're doing screen scraping or
> hitting the public feeds (I'm reading this as "the public timeline")
> excessively.

I'm guessing what Alex is actually referring to is the public
"Following" data, i.e., twitter.com/username/friends, along with
people's public updates, i.e., twitter.com/username?page=n. The rate
limit doesn't apply to these requests. My guess is @dacort's Twitter
Stats uses these and probably hammers them pretty badly.

Twitter Karma uses the authenticated Twitter API, so all its requests
are affected by the rate limit. However, public requests where the HTML
is being parsed and no rate limit is enforced ... those are probably
what Alex is referring to. (I'm surprised there wasn't a per-IP
throttle on those requests as well, already, if there isn't.)

-- Dossy

[David HM Spector]

Perhaps we need a different approach entirely here... 

Some people (like me :)  are writing clients whose main goal will be to mine and visualize tweet data... hard coding the API call interval seems like a really bad idea, and of course, people beating the stuffing out of twitters infrastructure isn't too helpful either.. I propose the following

1) a "limits" ReST call.  limits would return the current dynamically generated values that clients should use "right now" when making calls to the API --  specifically (for starters) the max # of lookups per hour that are permitted.  This number should be good for about 1 hour and, of course, would be higher or lower based on projected loads.  

2) Every request for data older than some value (let's say 36 hrs for the point of argument) is considered archival and not retrievable via the main twitter infrastructure via the client API.  Instead twitter clients can get this data on a best effort (i.e. load driven) basis with the explicit caveat that you may get either a "try again later" failure or "here's a partial result, ask for more some other time" response. 

Does this make sense? Or, am I off in the weeds..?

regards,

  David

----------------------------------------------------------------------------------------------

                                          David HM Spector

spector at zeitgeist dot com                                   http://www.zeitgeist.com

w: +1 631.261.5013            m:+1 631.827.3132            f: +1 212.656.1443

                                                    ~ ~ ~

"New and stirring things are belittled because if they are not belittled, the 

humiliating question arises, 'Why then are you not taking part in them?'"

                                                                                       --H. G. Wells                                    

[Andrew Badera]

Makes sense here, but does it make sense from an existing infrastructure perspective? How do we get There from Here?

There is definitely a dichotomy (or more plural) when it comes to "client" uses of Twitter ... "conversation client" and "statistics client" are probably only the tip of the iceberg. I like the differentiation.

--
--Andy Badera
http://andrew.badera.us/
and...@badera.us
(518) 641-1280
Google me: http://www.google.com/search?q=andrew+badera

[Marco Kaiser]

On Sun, Mar 2, 2008 at 12:56 AM, Andrew Badera <and...@badera.us> wrote:

There is definitely a dichotomy (or more plural) when it comes to "client" uses of Twitter ... "conversation client" and "statistics client" are probably only the tip of the iceberg. I like the differentiation.

Yes, good point. Maybe introduce application keys that include some "client type" and have different limiting schemes, with respect to their API usage pattern? Also, would be nice if "statistics clients" wouldn't affect "conversation clients" limit.

But you are right, it's a long way from here to there.

[Myles Eftos]

Isn' that what http://twitter.com/account/archive is for? (See
http://groups.google.com/group/twitter-development-talk/browse_thread/thread
/91f8ac9437f1ba72/e6303f48ce7e0231?lnk=gst&q=read+only#e6303f48ce7e0231)

Would this not help out some what to the data-miners?

----------------------------------------------
Myles Eftos
Mobile: +61-409-293-183

MadPilot Productions
URL: http://www.madpilot.com.au <http://www.madpilot.com.au/>
Phone: +618-9467-7651
Fax: +618-9467-6289

Try our time tracking system: 88 Miles!
http://www.88miles.net <http://www.88miles.net/>

________________________________

[Cameron Kaiser]

> 1) a "limits" ReST call. limits would return the current dynamically
> generated values that clients should use "right now" when making calls
> to the API -- specifically (for starters) the max # of lookups per
> hour that are permitted. This number should be good for about 1 hour
> and, of course, would be higher or lower based on projected loads.

I don't like something like this for two reasons: first, it's no good for
users of either interactive or mining clients because in the first case
the user has to adjust his/her usage habits all the time and in the second
because a mining client will have no good way of estimating its time to
completion; and secondly, setting a limit based on perceived future load is
fraught with peril. There's no way Twitter could have predicted, for example,
that this year's Super Bowl wouldn't suck.

> 2) Every request for data older than some value (let's say 36 hrs for
> the point of argument) is considered archival and not retrievable via
> the main twitter infrastructure via the client API. Instead twitter
> clients can get this data on a best effort (i.e. load driven) basis
> with the explicit caveat that you may get either a "try again later"
> failure or "here's a partial result, ask for more some other time"
> response.

I think this would be more reasonable, assuming that it is computationally
cheap to get a gestalt view of the various slave database load averages.

--
------------------------------------ personal: http://www.cameronkaiser.com/ --
Cameron Kaiser * Floodgap Systems * www.floodgap.com * cka...@floodgap.com
-- "I'd love to go out with you, but I'll be at the opening of my garage door."