Updating the APIs authentication limiting policy

[Doug Williams]

Devs --

A change shipped last week that limited the number of times a user could access the account/verify_credentials method [1] in a given hour. This change proved hasty and short-sighted as pointed out by the subsequent discussion [2]. We apologize to any developer that was adversely affected. Given the problems, we want to fix this in a public and transparent manner.

Like most web services, we limit the number of attempts users can make to login to their accounts on Twitter.com to prevent brute force dictionary attacks. This same security is not extended to the platform and leaves accounts vulnerable to the same method of attack through the API.

The change we shipped to limit user accounts to 15 calls an hour to the account/verify_credentials method [1] was intended to mitigate this risk. It was thought to limit the number of tests a potential attack could run in the hour, even in a distributed fashion. However, we only protected a single resource which still leaves all other authenticated methods exposed as a vector of attack (limited only by the API rate limit).

Our thinking is now that we will limit the total number of unsuccessful attempts to access authenticated resources to 15 an hour per user per IP address. If a single IP address makes 15 attempts to access a protected resource unsuccessfully for a given user (as indicated by an HTTP 401), then the user will be locked out of authenticated resources from that IP address for 1 hour.

This scheme has all of the positive effects that we need, however we want to make sure that we have thought through all of the potential problems on the developer's side before we proceed with this change. Please contribute to the subsequent discussion if you have an opinion or concern. Once we come to an agreement, we will update with details and a timeline for shipping this update.

1. http://apiwiki.twitter.com/Twitter-REST-API-Method%3A-account%C2%A0verify_credentials

2. http://groups.google.com/group/twitter-development-talk/browse_thread/thread/b057014336ff502b

Regards,

Doug

[Scott]

Thanks for the update Doug. Does this still apply to OAuth apps?
Also, if a user goes through an app and unsuccessfully attempts to
login 15 times will that app be blocked from authenticating anybody
for an hour or just that user? The previous change seemed to block
the entire app from making an authentication request on anybody once
the limit had been hit.

[Josh Perry]

To be honest ever since the x-rate-limit HTTP headers were added we
removed the call to verify_credentials from our Twitter API layer.

Every time that our Twitter API layer does an HTTP request it
squirrels away the header values and any requests to our API from the
application for rate-limit information is just fulfilled from those
saved variables. So we don't need verify_credentials for rate-limit
information

Every time that our API does an HTTP request it watches for
unauthorized HTTP responses, so we don't need verify_credentials to
verify that our app is still authorized on the account or that the
user's password is still the same.

Every single twitter API method could be used to brute-force by
sending HTTP auth headers and watching the HTTP response, but you are
rate-limited to 150 requests/hour/ip, if this rate-limit is good
enough for all the other attack vectors it should probably be good
enough for verify_credentials. In fact verify_credentials is basically
a nop function, which IMHO really isn't needed any longer.

Josh

On Jul 21, 7:00 pm, Doug Williams <d...@twitter.com> wrote:
> Devs --A change shipped last week that limited the number of times a user

> 1.http://apiwiki.twitter.com/Twitter-REST-API-Method%3A-account%C2%A0ve...
> 2.http://groups.google.com/group/twitter-development-talk/browse_thread...
>
> Regards,
> Doug

[Jesse Stay]

Josh, is there a way, without verify_credentials, to identify that users have changed their Twitter passwords (and therefore you are no longer able to authenticate for them)?  For client apps, I don't see this being as much of a problem, but for server-based apps that run regular scripts on behalf of users this could become a regular issue, which is why we were running it.

In addition, what is the best way with OAuth to identify the screen name of an individual?  verify_credentials is the only way I'm aware of, unless there's something I'm missing (which is probably very likely).  I'd love to know if there's a better way.  A best practices doc on how to retrieve user information, and how to best verify users have not changed their passwords would certainly be useful I think.  I'd like to know how Twitter recommends we do this.

Jesse

[Goblin]

I've been updating my site to use OAuth and have found this to be a
big problem. Without the ability to call verify_credentials I haven’t
found a reliable way to ensure that logged in users with a valid
session are still authorised with Twitter. It's unlikely they will
revoke access in the middle of using the site, but the potential is
there for an action to fail because of this. With verify_credentials I
am able to check their OAuth tokens are still valid and also make sure
their profile info is up to date. Spent several hours having a
headache over this, especially since the API says these calls are
unlimited. I was looking all over for unescaped loops and all sorts :)

Agree that either making OAuth calls unlimited (since there shouldn't
be a security vulnerability there) or making valid calls unlimited
(same reason) whilst limiting invalid calls makes the most sense. As
you say, you're still getting the security you want but without
penalising legitimate users. If this is going to be a quick fix/
rollback, I'll go back to using the method and deal with the low limit
when testing for this week.

Stuart

[Abraham Williams]

Making calls to protected methods like direct_messages will fail if the token is no longer valid. It is also possible that the test method will fail with incorrect authentication info/oauth token. I have not tried it though.

Abraham

--
Abraham Williams | Community Evangelist | http://web608.org
Hacker | http://abrah.am | http://twitter.com/abraham
Project | http://fireeagle.labs.poseurtech.com
This email is: [ ] blogable [x] ask first [ ] private.
Sent from Madison, WI, United States

[jim.renkel]

My concern with this proposal is that it opens up denials of service,
not to twitter.com, but to "associated" sites such as twitpic, or my
site twxlate, among others

For example, Lance Armstrong is a heavy user of twitpic. It is very
easy for anyone to find Lance's twitter ID (@lancearmstrong), view his
status updates, and see that he is a frequent user of twitpic. Now,
someone that is "unhappy" with Lance, say one of George Hincapie's
ardent fans that really believes that Lance was a significant
contributor to George not winning the maillot jeune last Sunday,
could go to twitpic, fail to login as Lance the requisite number of
times, and deny Lance access to twitpic.

Not only celebrities would or could be subject to such denials of
service. I notice that @dougw occasionally uses twitpic! :-)

One solution to this problem is to add to each twitter account another
"private" ID. By default this private ID would be equal to the
existing (public) ID (If not equal to the account's public ID, it
would have to be unique among all twitter IDs, both public and
private.).

The public ID would be used just as the existing twitter ID is now:
others would use it to follow, mention, DM, etc., the user.

But the user MUST use their private ID for authenticated requests
through the API, and CAN also use it for non-authenticated requests.
In either case, twitter would treat a request from a private ID as if
it came from the corresponding public ID.

Blocking the public ID because of excessive authentication failures
would NOT block the associated private ID unless they were equal.
Changing your public ID would also change your private ID if the two
were the same before the change, i.e., they would remain the same
after the change.

It may seem onerous to require all users to also have a private ID,
but since it defaults to be the same as their public ID, only those
concerned about their service being denied would change it and
subsequently use it instead of their public ID to access associated
sites such as twitpic or twxlate.

In fact, I think this change, though potentially large on the twitter
side, could be implemented without any changes to users or associated
sites, with one small, obscure exception: now, if I attempt to create
a new twitter account or change the ID of an existing account, and
find that the ID I want is in use, I can view that account; if this
were implemented and I attempted to use a private ID that was not the
same as its associated public ID, I could not view the account using
the denied ID.

Comments expected and welcome.

Jim Renkel

On Jul 21, 6:00 pm, Doug Williams <d...@twitter.com> wrote:
> Devs --A change shipped last week that limited the number of times a user

[owkaye]

> One solution to this problem is to add to each twitter
> account another "private" ID.

Jim,

Wouldn't it make more sense to implement this "private id"
thing on your own server?

My thought here is that your service should maintain its own
database of users, and issue a unique "private id" for each
of these users.

Then when the visitor tries to login, your code can check to
see if the private id the visitor has entered is in your own
database. If so the person is allowed to login, and if not
they get an error.

Would this work to solve the problem of am I missing
something here?

Owkaye

[Doug Williams]

Scott,

This change will only affect Basic Auth, and will not affect OAuth applications.

Thanks,
Doug

[jim.renkel]

Owkaye,

Thanks for the comment and suggestion.

The problem with implementing this locally at associated web sites
rather than centrally at twitter is that:
- each site would have to implement it separately; and
- users would have to sign up and create a private ID at each site
they use.

That results in much confusion and duplication of effort both for web
site developers and users. It would be be much less confusing and
require much less total effort if it were done centrally.

That said and given twitter's priorities and available resources, I
don't expect them to implement this scheme or anything like it.

And, at this time, we don't even know if this is a real issue or just
a red-herring. I raised it because I saw it as a theoretical problem
with the proposal, not because anyone that I know has experienced this
problem.

Does anybody see this as a real or potential problem, or should we
just let the issue fade away?

Comments expected and welcome.

Jim

[Scott Aikin]

Jim,

What you're suggesting is basically what they offer with OAuth. Apps
are given a token to represent logins and a secret key to represent
passwords for their authenticated users. Both are very long and
impossible to guess. This mechanism works very well and basically
corrects all the issues with collecting actual logins and passwords
from users.

Scott

[TinBlue]

What do you mean the change won't affect OAuth? My application has
been suffering from this issue ever since you made the limit change.
My application has the ability to use either Basic or OAuth. My
twitter users get blocked with the 403 error after a few minutes of
usage because they reach the 15 limit Authentication limit. But It
does this wether I am using OAuth or Basic?

Can you just clarify that I am understanding correctly that your
rollback will fix OAuth problems too?


On Jul 23, 12:15 am, Doug Williams <d...@twitter.com> wrote:
> Scott,This change will only affect Basic Auth, and will not affect OAuth

[Dewald Pretorius]

Jim raised a huge weakness with the authentication rate limiting that
could essentially break third-party apps.

Anybody can try to add anybody else's Twitter account to a third-party
app using an invalid password. If they do that 15 times with a Twitter
account, the real owner of that Twitter account, who may have added
his account a long time ago with the correct password, is locked out
from using that app for an hour.

I believe you will absolutely have to reset / remove the lock as soon
as the Twitter account uses the correct password.

[Joshua Perry]

Jim's concern is valid, fortunately OAuth is immune to brute-force
attacks once the access key has been issued to an application. For this
reason alone I would urge people to switch to OAuth if at all possible.
I would hope (and assume) that if login attempts for an account are
locked out that a user would still be able to successfully use an
already authorized OAuth driven application.

Unfortunately allowing a successful un/pw login while an account is
locked out even when the correct password is presented effectively
bypasses the whole reason for a lockout in the first place, preventing
brute-force password attempts. If an attacker used a dictionary or
brute-force attack and the account was locked out after 15 attempts,
then they could continue trying even though the system replied "locked
out"; if they eventually sent the correct password it would just bypass
the lockout and they would then know the correct password.

Perhaps Twitter could implement a selective captcha, I know they are
annoying but if executed properly it could be effective protection
against brute-force and dictionary attacks. Say after 3 or 4 failed
attempts without a captch the API would then include a captcha image URL
in it's response that the application would then need to show to the
person and include the user's response with the next authentication
attempt as a header or POST variable. The site stackoverflow.com does
this to great effect, if you create posts quicker than a certain
threshold which a person would not exceed then they pop a captcha up, in
the normal use of the site you will never see one; I've only hit two
captchas in the last in the last 8 months using the site.

Josh

[Doug Williams]

Well said Joshua.

Dewald, you have identified the risk of using basic authentication. If
your users being locked out due to malicious behavior, you should
either implement further user-level rate limiting on your side or
adopt OAuth.

Are there any other glaring omissions in our thinking or should we
proceed with this as our solution?

Thanks,
Doug

[Marco Kaiser]

I think Dewald's concern is very valid - and even though OAuth might solve it, the reality is that most (if not all) desktop and mobile apps are using Basic Auth today for various reasons, so if you implement this policy as described, there's a pretty high risk that many users can be locked out of twitter from their usual ways to access it.

Also, again a reminder that AFAIK the last official status re: OAuth from Twitter was that it is still in beta, and therefore not recommended for production use - or has there been another announcement that I missed?

Marco

2009/7/24 Doug Williams <do...@twitter.com>

[Abraham Williams]

I would much rather have Twitter lock me out of my account for an hour then let some script kiddie brute force my password. Thanks Twitter.

Abraham

[Goblin]

Seems fine. Is there a timescale for rolling this out?

[Dewald Pretorius]

Doug,

I would prefer to adopt OAuth instead of writing code for Basic Auth.

So, you guys need to move OAuth out of public beta into full
production sooner rather than later. :-)

I manage 100,000+ Twitter accounts, and I simply cannot take on the
support workload of answering user tickets when there's a snag with
OAuth beta.

I monitor these forums and the API Issues and still see too many OAuth
issues being reported to give me a level of comfort that I can safely
switch over to OAuth.

On Jul 24, 5:46 pm, Doug Williams <d...@twitter.com> wrote:
> Well said Joshua.
>
> Dewald, you have identified the risk of using basic authentication. If
> your users being locked out due to malicious behavior, you should
> either implement further user-level rate limiting on your side or
> adopt OAuth.
>
> Are there any other glaring omissions in our thinking or should we
> proceed with this as our solution?
>
> Thanks,
> Doug
>

[TinBlue]

What is happening?

This rollback is taking far too long for something that has affected a
lot of people!

[Doug Williams]

Thanks to everyone who has contributed feedback. This approach is what we are going to take. 

Alex will be making this change shortly. I will update this thread when there is timeframe to share.

Thanks,
Doug

[Ray]

Doug,

I'm in a similar situation as that voiced by TinBlue. This change has
affected our iPhone App. We also want to encourage you to rollback
this change ASAP.

When you say "This approach is what we are going to take.", do you
mean rolling back the fix so as not to affect multiple, successful,
authorized logins? I'm hopeful that "this approach" means that our
apps will not be affected yet again by changing to a new auth
approach.

I appreciate you all keeping this thread informed.

Ray

> ...
>
> read more »

[Doug Williams]

Ray,

For clarity, we will roll back the current restriction of 15 calls per user per hour to account/verify_credentials, and implement the proposed scheme:

> ... we will limit the total number of unsuccessful

> attempts to access authenticated resources to 15 an hour per user per IP
> address. If a single IP address makes 15 attempts to access a
> protected resource unsuccessfully for a given user (as indicated by an HTTP 401),
> then the user will be locked out of authenticated resources from that
> IP address for 1 hour.

Thanks,

Doug

[Grant Emsley]

Locked out of authenticated resources for that account, or will that
IP not be able to login to any account?

On Jul 29, 1:14 pm, Doug Williams <d...@twitter.com> wrote:
> Ray,For clarity, we will roll back the current restriction of 15 calls per

> user per hour to account/verify_credentials, and implement the proposed
> scheme:
>
>
>
> > ... we will limit the total number of unsuccessful
> > attempts to access authenticated resources to 15 an hour per user per IP
> > address. If a single IP address makes 15 attempts to access a
> > protected resource unsuccessfully for a given user (as indicated by an
> HTTP 401),
> > then the user will be locked out of authenticated resources from that
> > IP address for 1 hour.
>
> Thanks,
> Doug
>

> ...
>
> read more »

[Bob Thomson]

Hi Doug,

Is there a timescale for rolling back / making the change to the new
scheme?

We're just putting the finishing touches to moving to OAuth and we're
experiencing the issue when using verify_credentials to get the users
basic details once we've got the token back from the authentication
process. We're experiencing the issue when:

1. Testing our login and authentication processes
2. When users login and logout of our application frequently

A heads up on when these changes will be made would be useful. Thanks,

Bob

> ...
>
> read more »

[Jesse Stay]

A timeframe would be very helpful. This is turning out to be a headache as I'm testing. If my own user is having to log in over and over to test my app, I'm quickly hitting the verify_credentials limit (and I'm even using OAuth).  I'm getting really frustrated.

Jesse

[Alex Payne]

The rollback should be deployed tomorrow. Sorry for the delay.

--
Alex Payne - Platform Lead, Twitter, Inc.
http://twitter.com/al3x

[Goblin]

Did the rollback happen?

On Aug 3, 6:56 pm, Alex Payne <a...@twitter.com> wrote:
> The rollback should be deployed tomorrow. Sorry for the delay.
>
>
>
> On Sat, Aug 1, 2009 at 23:36, Jesse Stay<jesses...@gmail.com> wrote:
> > A timeframe would be very helpful. This is turning out to be a headache as
> > I'm testing. If my own user is having to log in over and over to test my
> > app, I'm quickly hitting the verify_credentials limit (and I'm even using
> > OAuth).  I'm getting really frustrated.
> > Jesse
>

> > On Fri, Jul 31, 2009 at 8:01 PM, Bob Thomson <stormid...@googlemail.com>

> ...
>
> read more »

[Antony]

Would it be possible to have an update on this issue? It seems it's
stifling a lot of developers, me included.

[Antony]

It would appear not - just tried our app and after 15 page loads it
breaks. Our site uses OAuth to sign users in, so we use
verifycredentials on each page request to ensure the session is still
valid and return user details. On our system it is quite likely that
the user will see more than 15 pages within an hour, and I'm sure
there are many other similar situations like ours.

> ...
>
> read more »

[Dewald Pretorius]

Alex,

Did the change go live on Tuesday?

I have very irate users due to this issue. There are spam bots out
there that got hold of users' credentials. The users have changed
their Twitter passwords to get rid of the spam tweets published in
their timelines, but now those bots are locking them out 24x7 from all
apps that use the API.

On Aug 3, 2:56 pm, Alex Payne <a...@twitter.com> wrote:
> The rollback should be deployed tomorrow. Sorry for the delay.
>

> On Sat, Aug 1, 2009 at 23:36, Jesse Stay<jesses...@gmail.com> wrote:
> > A timeframe would be very helpful. This is turning out to be a headache as
> > I'm testing. If my own user is having to log in over and over to test my
> > app, I'm quickly hitting the verify_credentials limit (and I'm even using
> > OAuth).  I'm getting really frustrated.
> > Jesse
>

> > On Fri, Jul 31, 2009 at 8:01 PM, Bob Thomson <stormid...@googlemail.com>

> ...
>
> read more »

[Alex Payne]

The change did not go live yesterday due to some deploy issues. It's
not estimated to go out tomorrow. Once again, sorry for the delay.

--

[Goblin]

Alex, is that *not* estimated or was it an iPhone being daft and
changing now to not?

> ...
>
> read more »

[Alex Payne]

We've just heard from our operations and deploy staff that we won't be
able to deploy any code (for the API or otherwise) until Monday due to
the DDoS attack and other issues. That means that the revert to the
old rate limiting policy for this method won't go out this week. My
apologies.

--

[Grant Emsley]

Perhaps a better approach to the lockout:

Lock the account for x minutes after 15 *unique* bad passwords. So if
the user changes their password, and another program keeps trying with
their old password, that only counts as 1 attempt.
It still only gives them 15 guesses, but would cause fewer lockouts
because of badly behaved programs like the spam bots mentioned above.