API limit issues

[Marco Kaiser]

Hi,

these days, many users of twitter desktop clients (twhirl for me, but I've seen problems reported for other clients too) run into limit exceeded problems. As tests for twhirl have shown, this is not a client issue, but the API reports back status 400 and the corresponding error message for these users, sometimes even on the first request when they start the application.

I assume this is not a bug in the twitter backend, and that these accounts are really locked due to over-using the API. As this problem has increased much over the past weeks, my assumption is that the many new social networking sites that import user feeds from twitter (like lifestreaming sites Socialthing, FriendFeed etc.) are consuming a lot more requests from API.

Twitter, have you seen any significant growth in API requests and accounts exceeding their limit? Can you give any comment on this, please?

Is there any chance for the near future that we can see the allowed number of requests be increased? Or maybe changing it to a client based instead of an account based limit?

Thanks in advance,
Marco

[Andy Bilodeau]

I this this information needs to be communicated to the end users. It
makes sense that all these Twitter hookins affect their limits. Is it
even possible to know where the requests from any given user are
coming from? And if so, could this be presented to the end user to
let them know that this is why you're getting a rate limit. this
would allow the end user to make better decisions before adding their
twitter data to just any old site.

I personally use Twitter IM which has been rock solid later (yay)
but Marco's point it spot on.

Excellent point!

Andy B.

(Twitter dev wanna be)

[Alex Payne]

It's been my intention to do per-application rate limiting once OAuth
is in place. In the meantime, I agree that users need to be informed
about when they're over the rate limit. I'll make sure that happens
sooner than later.

--
Alex Payne
http://twitter.com/al3x

[Ed Finkler]

On Wed, Mar 26, 2008 at 10:43 AM, Marco Kaiser <kaiser...@gmail.com> wrote:
> Hi,
>
> these days, many users of twitter desktop clients (twhirl for me, but I've
> seen problems reported for other clients too) run into limit exceeded
> problems. As tests for twhirl have shown, this is not a client issue, but
> the API reports back status 400 and the corresponding error message for
> these users, sometimes even on the first request when they start the
> application.

Just FYI, you're not alone in this -- I've seen it in Spaz and my
users also report it. I believe al3x is looking into what might be
causing this.

--
Ed Finkler
http://funkatron.com
AIM: funka7ron
ICQ: 3922133
Skype: funka7ron

[Cameron Kaiser]

> these days, many users of twitter desktop clients (twhirl for me, but I've
> seen problems reported for other clients too) run into limit exceeded
> problems. As tests for twhirl have shown, this is not a client issue, but
> the API reports back status 400 and the corresponding error message for
> these users, sometimes even on the first request when they start the
> application.

I've gotten bitten by this too. In my case, it turned out to be the Facebook
Twitter app that was the offender, and the problem went away when I got rid
of it. (Sorry, Alex. It was nice while it lasted ;-)

Smoking that out can be a full-time job if you have lots of things
authenticating against the API, so I'm happy to see per-app API settings,
as long as there is token prevention for malicious apps behaving badly.

--
------------------------------------ personal: http://www.cameronkaiser.com/ --
Cameron Kaiser * Floodgap Systems * www.floodgap.com * cka...@floodgap.com
-- People are weird. -- Law & Order SVU ---------------------------------------

[Ed Finkler]

On Wed, Mar 26, 2008 at 1:50 PM, Cameron Kaiser <spe...@floodgap.com> wrote:

> I've gotten bitten by this too. In my case, it turned out to be the Facebook
> Twitter app that was the offender, and the problem went away when I got rid
> of it. (Sorry, Alex. It was nice while it lasted ;-)

Is that the Twitter -> Facebook Status app? I figured it wouldn't need
to use the API to grab that data, since authentication isn't necessary
(unless your updates are private, that is).

[Alex Payne]

Our Facebook App doesn't require any API hits. No idea about the
other one(s) out there.

--
Alex Payne
http://twitter.com/al3x

[Cameron Kaiser]

> Our Facebook App doesn't require any API hits. No idea about the
> other one(s) out there.

Just to clarify: this is the little "Twitter box" that sits in your profile,
not the Twitter->FB Status app.

Dunno what to say, but the problem seemed to go away as soon as I junked it.
I might do some more snooping.

--
------------------------------------ personal: http://www.cameronkaiser.com/ --
Cameron Kaiser * Floodgap Systems * www.floodgap.com * cka...@floodgap.com

-- Make welfare as hard to get as building permits. ---------------------------

[foody]

Hi All,

Is it correct to assume that the Twitter Flash Badges do
not make API calls as well?

Cheers!
mdy

[Ed Finkler]

On Thu, Mar 27, 2008 at 6:06 AM, foody <Food4T...@gmail.com> wrote:
>
> Hi All,
>
> Is it correct to assume that the Twitter Flash Badges do
> not make API calls as well?

If you don't enter a username and pass, it certainly won't make any
that affect your rate limits. Limits only apply to authenticated
requests.

[foody]

Thanks, Ed! Appreciate your quick reply. That makes sense.

On Mar 27, 11:11 pm, "Ed Finkler" <funkat...@gmail.com> wrote:
>
> If you don't enter a username and pass, it certainly won't make any
> that affect your rate limits. Limits only apply to authenticated
> requests.
>
> --

> Ed Finklerhttp://funkatron.com

[Ed Finkler]

This is still an issue my users and I are running into, and I'm nearly
certain in at least some cases (like most of mine) that we're not
exceeding 70 reqs per hour. Alex, have you had any time to look into
this? Can I help in any way? Or can you confirm that I'm just nuts? 8)

--
Ed Finkler
http://funkatron.com
AIM: funka7ron
ICQ: 3922133
Skype: funka7ron

[Alex Payne]

Apologies, I haven't had time to look into it, but it's at the top of my list of bugs for this week.  Profuse apologies!

[JoeC]

And I think I need a little clarifiction on how the limit is enforced,
since I am running a web app. Is it requests by my app, or requests
from the login using my app?
Thanks,
JoeC

On Mar 29, 10:34 pm, "Alex Payne" <a...@al3x.net> wrote:
> Apologies, I haven't had time to look into it, but it's at the top of my
> list of bugs for this week. Profuse apologies!
>
>
>
> On Sat, Mar 29, 2008 at 5:35 PM, Ed Finkler <funkat...@gmail.com> wrote:
>
> > This is still an issue my users and I are running into, and I'm nearly
> > certain in at least some cases (like most of mine) that we're not
> > exceeding 70 reqs per hour. Alex, have you had any time to look into
> > this? Can I help in any way? Or can you confirm that I'm just nuts? 8)
>
> > --
> > Ed Finkler
> >http://funkatron.com
> > AIM: funka7ron
> > ICQ: 3922133
> > Skype: funka7ron
>

> > On Wed, Mar 26, 2008 at 1:42 PM, Ed Finkler <funkat...@gmail.com> wrote:
> > > On Wed, Mar 26, 2008 at 10:43 AM, Marco Kaiser <kaiser.ma...@gmail.com>

[Alex Payne]

The latter: the rate limit is enforced on a per-user basis.

[chesh2000pro]

Hey all --

I can confirm that a fair number of people are having this problem --
check this thread on Get Satisfaction:

http://getsatisfaction.com/twitter/topics/tweet_limit_exceeded

I can also confirm that my account is locked out but it isn't based on
authenticated requests. I changed my password yesterday (which would
have locked out anything I had given my password to) and then re-
launched Twhirl. I was asked for my new password (so it is
authenticating me), and still got the rate limit exceeded. Still
happening now.

Thanks for looking into it Alex!

Jay

[djvdm]

If this is the case, can you tell us if this scenario is going to
cause an issue:

We will be running a cron job that will send out direct messages to
our user base using one twitter account
We need to use direct messages because the information is confidential
in most cases.
The cron job will run every 5, 10, 15 or 30 minutes, depending on
their preferences.
If all users choose 5 minutes, then we can only send a batch of 70 DMs
in a minute?
Or one every second or so, for a total of 350 DMs in 5 muntes?

If this is the case, is there a way around this? This would not be
usable in this case.

Just want to be clear on how an application with thousands of users
can interact using twitter without running up against a limit. Or are
we missing something here?

Tks, Doug

[djvdm]

OK, I need to RTFM. After doing that, seems these POST requests will
not cause a rate limit issue.

However, 1 request per minute may not be enough in the long run with
the application we are planning. Could you point us in the right
direction re: Jabber/XMPP and how it is being used with Twitter to
deliver a near-realtime user experience? That is our ultimate goal.

Thanks (sorry for the duh moment)

Doug

[JoeC]

I don't think I'd be too happy with per-app rate-limiting. I think it
should still be done per user. Why penalize all my users because I
write a popular app, or because one user is a hog. Can you explain the
reasoning?

Thanks,
JoeC

[spec...@gmail.com]

I'm sure the Twitter people can correct me on this, but I believe they
intend it to be per-application-per-user, so if I use two clients, I
can make 2*max-api-calls per hour.

It's not based on a specific application being limited to any given
number of requests.

[Alex Payne]

The reason we don't do per-application rate limiting is that we don't
currently have a way of verifying the application a user is using.
When we move to OAuth, we will, but for that reason and others we're
being extremely careful about our move to OAuth. It changes the
impact to our system, the amount of work necessary for developers, and
a number of user expectations.

--
Alex Payne
http://twitter.com/al3x

[RoelandP]

Hi Al3x and other Twitter people,

There are still some occurences with the 400 'Rate limit exceeded.
Clients may not make more than 70 requests per hour'.
It is the same story as written over and over again here at
GetSatisfaction thread: http://getsatisfaction.com/twitter/topics/macworld_twitter_slowness_downtime

My app. @ http://m.slandr.net has some users who don't use any other
apps, neither have the Facebook App and did even change their
password, but still get the 400 rate limit exceeded.

In the GetSatisfaction thread people are talking about 'the mysterious
dark habit', but I know machines a bit and I think this is unintendly
due to some sort of cache or database row which wrongly says: "UserX
exceeded rate limit".
Can it be that there is some loophole in a script which causes the
'rate limit'-agent to not release these previously outrated/limited
users back into the 'free' zone?

thanks for looking into this!
all best, Roeland

[Alex Payne]

We've got this bug on our radar, and we'll get it fixed ASAP. Thanks!

--
Alex Payne
http://twitter.com/al3x