JWT: maybe a better incarnation of "api tokens"

[Niphlod]

So, the idea is to have something issuing a token that can be validated at every request without ever hitting the database.
No additional tables, horizontal scalability, good security, not really another login-contrib-method, well suited for APIs, etc.
I'm taking a long road here in order to explain some pretty specific choices done in the code.


Now, the "bad" part: Auth.login_user() is absolutely NOT built for speed. For a good reason in the username-password verification part (pbkdf2 comparison for password matching
protects the stored password from being guessed with a bruteforce in case they get stolen) but not so much for the update_groups() part.

In there, we get an additional query for each group the user is into (guess the usual nosql 2% of users can benefit from that but that's another story).
Long story short, up until now APIs were mainly built around basic auth....it's a special case into core (should go along with other login methods IMHO)
but even with basic authentication, and actions protected by a mere requires_login, which doesn't really require groups, we check for username, password (plus empty registration id, plus cycling through every other login_method appended), plus membership recursive query to get groups. At EVERY request.
Nice (not really), but let's get along for the sake of elevating RBAC as our model for authorization: group membership is really important!
It gets cached in session.auth.user_groups (and auth.user_groups) but .... groan! it's hardly leveraged by any piece of code. Maybe we - experienced developers - already know that auth.requires() is really - really, really, really - the only validator to use in addition to requires_login().
But I guess most users don't know that.
After all the heavy lifting of storing auth.user_groups (and loading session.auth.user_groups), requires_membership(), that is the de-facto recommended way to "protect" resources, doesn't use "cached" user_groups, but instead checks at runtime the membership (another nifty two-separate queries) for each request. Ultra-groan.
Now, this is pretty "ultrasecure" because you can remove the membership to a logged in user and at runtime have the permission effectively revoked.
Bak to the real word....let's be straight, there should be a specific API in auth to do this instead of promoting it as a default (maybe with a self-explanatory "please_kill_the_performances" argument).
Then we have the ultra-extra-ultra management of sessions, with all the fancy "renew/delete/whatever" callback. Let's not go there, too many groans :-P

Real-word scenario: every "modern" identity and authorization provider now issues a "something" that carries your identification and authorization specs, that are valid for a little while. This pattern scales horizontally really well, and one can adjust the "please_kill_the_performances" in favour of security just narrowing the expiration.

web2py's auth has been largely engineered for seldomly identifying the user, storing the identification in the session, and the not-so-seldomly checking
authorizations through group membership. In that vision, an "api tokens" auth is completely unsuitable to scale horizontally to protect something, if we implement it bashing the database at every request for identification and membership.

Enter the attached file: granted, needs testing, refinement and your approval, but it lends authentication and authorization storing what gets stored into
session inside a JWT token.
To avoid bashing the db, the real deal about this implemetation is that the token then gets loaded and injected into auth, in order to make auth decorators work.
No db, no login_user(), nothing at all.

Now, a taddle bit of wording on "security" (as users are always so worried about that).
The token is comprised of a header, a payload and a signature.
JWT specs are large enough to permit payload encryption, but IMHO it's a useless PITA (both to implement and to debug).
The provided module doesn't do encryption, it just signs (and verifies accordingly) the payload.
If the database needs to be left alone, the - obvious - attack vector of the implemetation (this of whatever will it come up) relies on the attacker knowing the secret.
Users - and developers - may get scared, and ask for the "jti" claim to be added and stored to check that the token used by the attacker has been effectively issued, etc etc etc. But this bashes the db. Maybe only a redis-backed storage could resist the pressure.

I'm planning to add subclassing (or callbacking) extensions to the module to make that (and other intricacies) possible (like storing additional claims in the payload), but I'd not recommend it by default.
Granted, if the attacker guesses the secret, he can craft a perfectly valid token for any user (with any claim in it),  but the same goes for sessions stored in cookies (even if they are encrypted, as web2py uses a static secret by default).
Protection against bruteforcing should be done at a higher level that web2py, granted, but the implementation has a nice addition: users can salt the secret with their own function, maybe taking the hmac_key (a random uuid generated at "issuing token" time) as a starting point to generate the salt.
I hear you, "decoding the payload before verifying the signature is bad", but is a correct way to do that just from a theoretical standpoint. In this context, the worst thing happening is loading a json and not a pickle...so at most we loaded a Plain Old Python Dict.
No security issues there. There's also a max length defined for the header (to discourage DDOs) which some hardened webservers have already in place.
This makes the attack vector really smaller: an attacker needs to guess the secret AND the salt for every user. I don't think that we need to think about an additional layer of security (like encryption) to make the web2py's JWT api more secure.

I'm open to discussions and suggestions.
Please note that it's not bug free, nor tested (planned for this week).
It's just the first round of implemetation and, for all intents and purposes, a POC.

[Massimo DiPierro]

On Jul 20, 2015, at 3:58 PM, Niphlod <nip...@gmail.com> wrote:

So, the idea is to have something issuing a token that can be validated at every request without ever hitting the database.
No additional tables, horizontal scalability, good security, not really another login-contrib-method, well suited for APIs, etc.
I'm taking a long road here in order to explain some pretty specific choices done in the code.

I like this so far….

Now, the "bad" part: Auth.login_user() is absolutely NOT built for speed. For a good reason in the username-password verification part (pbkdf2 comparison for password matching
protects the stored password from being guessed with a bruteforce in case they get stolen) but not so much for the update_groups() part.

In there, we get an additional query for each group the user is into (guess the usual nosql 2% of users can benefit from that but that's another story).
Long story short, up until now APIs were mainly built around basic auth....it's a special case into core (should go along with other login methods IMHO)
but even with basic authentication, and actions protected by a mere requires_login, which doesn't really require groups, we check for username, password (plus empty registration id, plus cycling through every other login_method appended), plus membership recursive query to get groups. At EVERY request.
Nice (not really), but let's get along for the sake of elevating RBAC as our model for authorization: group membership is really important!
It gets cached in session.auth.user_groups (and auth.user_groups) but .... groan! it's hardly leveraged by any piece of code. Maybe we - experienced developers - already know that auth.requires() is really - really, really, really - the only validator to use in addition to requires_login().
But I guess most users don't know that.
After all the heavy lifting of storing auth.user_groups (and loading session.auth.user_groups), requires_membership(), that is the de-facto recommended way to "protect" resources, doesn't use "cached" user_groups, but instead checks at runtime the membership (another nifty two-separate queries) for each request. Ultra-groan.

auth.has_membership(groupname) # slow but correct

auth.requires(groupname in auth.user_groups.values()) # faster but occasionally incorrect

Now, this is pretty "ultrasecure" because you can remove the membership to a logged in user and at runtime have the permission effectively revoked.
Bak to the real word....let's be straight, there should be a specific API in auth to do this instead of promoting it as a default (maybe with a self-explanatory "please_kill_the_performances" argument).
Then we have the ultra-extra-ultra management of sessions, with all the fancy "renew/delete/whatever" callback. Let's not go there, too many groans :-P

Real-word scenario: every "modern" identity and authorization provider now issues a "something" that carries your identification and authorization specs, that are valid for a little while. This pattern scales horizontally really well, and one can adjust the "please_kill_the_performances" in favour of security just narrowing the expiration.

web2py's auth has been largely engineered for seldomly identifying the user, storing the identification in the session, and the not-so-seldomly checking
authorizations through group membership. In that vision, an "api tokens" auth is completely unsuitable to scale horizontally to protect something, if we implement it bashing the database at every request for identification and membership.

Enter the attached file: granted, needs testing, refinement and your approval, but it lends authentication and authorization storing what gets stored into
session inside a JWT token.
To avoid bashing the db, the real deal about this implemetation is that the token then gets loaded and injected into auth, in order to make auth decorators work.
No db, no login_user(), nothing at all.

Now, a taddle bit of wording on "security" (as users are always so worried about that).
The token is comprised of a header, a payload and a signature.
JWT specs are large enough to permit payload encryption, but IMHO it's a useless PITA (both to implement and to debug).
The provided module doesn't do encryption, it just signs (and verifies accordingly) the payload.
If the database needs to be left alone, the - obvious - attack vector of the implemetation (this of whatever will it come up) relies on the attacker knowing the secret.
Users - and developers - may get scared, and ask for the "jti" claim to be added and stored to check that the token used by the attacker has been effectively issued, etc etc etc. But this bashes the db. Maybe only a redis-backed storage could resist the pressure.

I'm planning to add subclassing (or callbacking) extensions to the module to make that (and other intricacies) possible (like storing additional claims in the payload), but I'd not recommend it by default.
Granted, if the attacker guesses the secret, he can craft a perfectly valid token for any user (with any claim in it),  but the same goes for sessions stored in cookies (even if they are encrypted, as web2py uses a static secret by default).

Which why I do not think that session in cookies are not a good idea and it is not the default.

Protection against bruteforcing should be done at a higher level that web2py, granted, but the implementation has a nice addition: users can salt the secret with their own function, maybe taking the hmac_key (a random uuid generated at "issuing token" time) as a starting point to generate the salt.
I hear you, "decoding the payload before verifying the signature is bad", but is a correct way to do that just from a theoretical standpoint. In this context, the worst thing happening is loading a json and not a pickle...so at most we loaded a Plain Old Python Dict.

if it is a json ok.

No security issues there. There's also a max length defined for the header (to discourage DDOs) which some hardened webservers have already in place.
This makes the attack vector really smaller: an attacker needs to guess the secret AND the salt for every user. I don't think that we need to think about an additional layer of security (like encryption) to make the web2py's JWT api more secure.

I'm open to discussions and suggestions.
Please note that it's not bug free, nor tested (planned for this week).
It's just the first round of implemetation and, for all intents and purposes, a POC.

I like the idea.

I would like it possible if this mechanism where as orthogonal to auth as possible. Auth is overkill and I can imagine replacing it with something better in the future. For example in most of my applications I never use auth_permission and have very few groups, so I handle them as a  boolean membership fields in auth_user. So really auth_user is the only table that most people need. So if the JWT implementation had an API that make it work with Auth and without Auth, it would be even better.

--
-- mail from:GoogleGroups "web2py-developers" mailing list
make speech: web2py-d...@googlegroups.com
unsubscribe: web2py-develop...@googlegroups.com
details : http://groups.google.com/group/web2py-developers
the project: http://code.google.com/p/web2py/
official : http://www.web2py.com/
---
You received this message because you are subscribed to the Google Groups "web2py-developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to web2py-develop...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
<web2py_jwt.py>

[Niphlod]

the beauty is that the payload can hold any number of claims. "Claims" in jwt are either key-values or key-mappings. A few of them are reserved, so as long as the naming convention doesn't clash, it's fully extensible at anyone's agenda. 

At a higher level, a payload is a POJO (plain old javascript object), so in python it is a PODO (plain old dict). You can put whatever you want in it.

As it is, inject_token() is the function that "takes the payload and logins the user" while serialize_auth_session() is the function that "takes the login and serializes the payload".

As long as you can subclass / customize these two, you can use it as orthogonal as you like.

[Massimo DiPierro]

I would stick with pojos

[Richard Vézina]

https://auth0.com/blog/2015/03/31/critical-vulnerabilities-in-json-web-token-libraries/

[Massimo DiPierro]

interesting. Clearly the none algorithm should not be allowed.

[Niphlod]

...and the module does not allow it. 

Guys (and girls?!), I may be a DBA in real life (yep, scary, sturdy, always whining about performances and optimizations) but I have a pretty track record on hacking on web2py and python in general :-P

It's perfectly fine to have a certain "return" for suggestments and ideas, but a really quick look in the docstring

     - algorithm : uses as they are in the JWT specs, HS256, HS384 or HS512 basically means
                   signing with HMAC with a 256, 284 or 512bit hash

or a few lines down below

        if self.algorithm not in ('HS256', 'HS384', 'HS512'):
            raise NotImplementedError('Algoritm %s not allowed' % algorithm)

should do the trick :P

[Richard Vézina]

Reading Simone post, I was under the impression that web2py RBAC make a lot of overhead compare to the same request without authentication required... Is this that much significative? Is a simple benchmark with ant without authenciation requirement enough to dertermine the overhead?

I make some tests didn't notice any performance issue between decorated auth.require_login() and plain no authentication requires and can't find significative differences... I may not have the proper way of testing it...

About the vulnerability, I wasn't pointing your code have an issue, just want to make us aware of the flaw...

Richard

[Anthony]

auth.requires_login() shouldn't add much overhead because there is no database lookup. Some of the other decorators, however, do involve checking the database.

Anthony

[Richard Vézina]

Is there advantage of using :

@auth.requires(lambda: 1 in auth.user_groups, requires_login=True)  # if 1 is my admin group id...

Instead of :

@auth.has_membership('admin')

Richard

[Anthony]

As Massimo pointed out, there are pros and cons to each...

@auth.requires(lambda: 1 in auth.user_groups, requires_login=True)  # if 1 is my admin group id...

The above is faster because there is no database lookup, but auth.user_groups may be out of date, as its value is set and stored in the session at time of login. If the user was added to or removed from some group since login, this authorization will be incorrect (this is particularly a problem when the "remember me" option is activated).
 

Instead of :

@auth.has_membership('admin')

Slower due to the database lookup, but always retrieves the up-to-date memberships.

Anthony

[Richard Vézina]

Thanks for explaining, make it clearer what Massimo has wrote...

Richard

--

[Massimo DiPierro]

Mind that  requires_login=True is default

[Massimo DiPierro]

Stupid me. Mac Mail did not show there was an attachment. Now I see the attachment because I am using gmail web. I will review and get back asap.

--

[Massimo DiPierro]

So I like this but I see it is very different than the token logic I had implemented and serves a very different purpose.

I was imagining a user of an app who wanted to give a token to third party so they could access some limited functionality of my account, if they abused the service they could simply expire a token. This requires database access. Example use case (use this token to upload data into my account by I retain permission to edit/delete uploaded data, I can expire the token when I do not want you to upload data any more).

With the jwt one is delegating authentication to the jwt token and tokens cannot be individually expired so they are not mean to give somebody else access. They are meant to authenticate the actual user.

I like the jwt and I think it should go in web2py (contrib?) but I do not think this replaces the other token logic. Did I misunderstand?

Massimo

On Mon, Jul 20, 2015 at 3:58 PM, Niphlod <nip...@gmail.com> wrote:

--

[Niphlod]

IMHO web2py shouldn't reinvent the wheel. 

The usecase scenario you described can be implemented with jwt too, using the jti claim which is included in the spec

The jti (JWT ID) claim provides a unique identifier for the JWT. The identifier value MUST be assigned in a manner that ensures that there is a negligible probability that the same value will be accidentally assigned to a different data object; if the application uses multiple issuers, collisions MUST be prevented among values produced by different issuers as well. The jti claim can be used to prevent the JWT from being replayed. The jti value is a case-sensitive string. Use of this claim is OPTIONAL.

as jwt can include any claim, it's really, really flexible an "orthongonal" as you want.

[Massimo DiPierro]

I am all for not reinventing the wheel but I do not do not see how this can be used to give a token a user and then expire it server side at will without hitting the DB.

Massimo

[Richard Vézina]

Could Redish be an option?

[Niphlod]

if you need to check for a random value your app generated, of course it needs to be stored somewhere and checked upon. 

The deal with JWT (and its specs) is that is extensible as much as we like it, with a good foundation and a relatively simple implemetation, that will enable this and other usecases all "under the same hat" 

[Massimo DiPierro]

Where should I put your file? contrib?

[Niphlod]

it isn't ready yet.

IMHO should go in contrib - plus the singleton wrapper, of course - although I didn't get why the previous implementation went on tools.py.

[Massimo DiPierro]

Do not worry. The token implementation will be removed

[Niphlod]

umpf, more than one month later, sorry for that, I return.
I think this version settles on the needed API for a good flexibility. Hopefully 200 LOC aren't that many to loose anyone's head on.

Docstrings are at the top, but just for discussion's sake, another - not so - brief outline...

JWT just do what session in cookies do in regards of authorization. They'll use these unreserved claims "by default":
- user
- user_groups
- hmac_key
plus some of the reserved ones
- exp
- iat

e.g.
{
  "user_groups": {
    "1": "user_1"
  },
  "hmac_key": "23355a3e-c78f-405c-ad13-66147e0e3863",
  "user": {
    "first_name": "ciao",
    "last_name": "ciao",
    "registration_id": "",
    "email": "ciao@localhost",
    "reset_password_key": "",
    "registration_key": "",
    "id": 1
  },
  "exp": 1441222830,
  "iat": 1441222530
}

Unreserved claims are then injected into auth, same as it happens in tools.py for Session.

Security is assured through signing the payload, with HMAC with 256, 284 or a 512bit hash. Users concerned about the fact that this module doesn't support encryption (hard to implement and requiring some crypto libraries not readily available on all systems) should be assured by:
- saltable secret
- DDoS prevention done on the length of the token itself
- custom additional checks are pluggable

Additionally:
- payload headers can be injected at runtime
- the payload can be verified with a custom check too, right after signature has been correctly verified

What does this module provide:
- endpoint that takes care of issuing tokens and refreshing them
- decorator that requires the token to be in an authorization header or in the _token var
Available customizations:
- algorithm: choose HMAC algo
- check expiration (can be turned off) via the exp claim
- configurable leeway for expiration checking
- expiration in seconds (can't be disabled, as a good practice. They can always disable expiration checking)
- token "refresher" availability
- header prefix for all header exchanges ("Bearer" seems to be emerging as a standard)
- possibility to add claims to the header (alg and typ are the only used by default)
- configurable "names" for parameters holding the username and the password to use to request a token
- configurable max length for the token, to avoid DDoS
Available hooks:
- salt: can be a function that takes the payload and computes the salt that gets appended to the secret
- additional_payload: can be a dict to be merged to the "standard" payload or a function that alters the payload before issuing the token
- before_authorization: a callable that checks the payload before authorizing the request

NB: the implementation tries to be as efficient as possible (I'm interested in dark corners for performance boost). I see it mainly to authorize zillions of requests to (micro)services and so it SHOULD be faster than the usual auth, without hitting the db if not strictly necessary.

As per "original" demand of covering one-time-issued tokens, the "jti" claim is the standard, and can be easily implemented, imagining to store valid tokens in a database table:

db.define_table('jwt_tokens', Field('token'), Field('user_id'), Field('inserted_on', 'datetime', default=request.now))

def myadditional_payload(payload):
     res = db(db.jwt_tokens.user_id == payload['user']['id']).select(orderby=~db.jwt_tokens.inserted_on).first()
     payload['jti'] = res.token
     return payload

def mybefore_authorization(tokend):
     res = db(
            (db.jwt_tokens.user_id == tokend['user']['id']) &
            (db.jwt_tokens.token == tokend['jti'])
     ).select().first()
     if not res:
         raise HTTP(400, u'Invalid JWT jti claim')

myjwt = Web2pyJwt('secret', auth,
                  additional_payload=additional_payload,
                  before_authorization=mybefore_authorization)

As always, naming APIs is not my best. @anthony: bring it on! :P

[Massimo DiPierro]

This is great. Thanks Simone. I have a terrible week but I will process this asap. 

[Massimo DiPierro]

So. I finally got the time to review this. I am fine with most of it but I would propose a change. if people use JWT it is because they want a standard protocol and that means that apps sharing auth info may not all be based on web2py. So I think this should be completely decoupled from web2py auth. It should have its own authentication/authorization decorators. If people want to write code that copies session info into a JWT token and then retrieve it is up to them to do so. They may copy all of parts of a session, object, including or not including session.auth. We should not mandate it.

Do you agree?

Massimo

On Mon, Jul 20, 2015 at 3:58 PM, Niphlod <nip...@gmail.com> wrote:

--

[Niphlod]

it's a shift from the standard "convention over configuration". I agree, though: if you can decouple more than I accomplished, while taking the same or less lines of code to set it up, I'm up for it.

I really don't know at that point what would be the plus on using web2py's one over the largely tested (and from which I took the code) already existing libraries for JWT, except that they don't ship in the zip with web2py.

[Massimo DiPierro]

How about we have something more generic in contrib that is not web2py specific and than we tweak tools.py to use it, the same way we use cas?

[Niphlod]

IMHO the same "issue" applies: this is not a revolutionary module and it doesn't shine in implementation details. There are FAR BETTER alternatives, but they don't ship with web2py.

This can pretty much apply to all contrib modules, specifically the "login" ones.

Up until now web2py's shipped contribs are easier to use because they leverages web2py's core conventions (and work around web2py 'weird' execution model). And more or less are dependancy-free.

If I/we reinvent the wheel to create something that is not web2py specific, I'm missing the plus side of using it vs external libraries.

Nobody is gonna use web2py's contrib module for jwt written by niphlod in flask.

If you want to tweak tools.py to use pyjwt or python-jose (the far better alternatives with their dependencies) and leave this contrib out, I'm +1.

[Massimo DiPierro]

The point is not that we want other people to use our contrib modules. The point is that some web2py users may want to use jwt with web2py without using the built-in auth.

[Niphlod]

uhm. I guess I didn't see any of those requirements ...

in any case, those should just need to override requires_jwt (the last line only) to match for their own authorization scheme and inject_token (with their own logic).

[Massimo DiPierro]

what do other people think about this. Once this is in, it stays so we need to make sure people understand it and are happy with it.

[Leonel Câmara]

I agree with Simone that integration with other web2py functionality such as auth makes sense. web2py is a fullstack framework, one of its big advantages is that you don't have to lose time integrating a bunch of different modules because most of what you want already works out of the box. I don't want to have to do my own auth integration. I think it's easy enough to make your custom class inheriting from this one (we could just add a comment to the class saying which methods they need to override) if you don't want to use auth.

[Michele Comitini]

I agree too, web2py is a framework that implements a lot of common patterns in a sane way. JWT integrated with web2py auth is something most would be confortable with.  Than if someone wants to take a different road there are external modules to do the job just fine.

2015-10-27 16:14 GMT+01:00 Leonel Câmara <leonel...@gmail.com>:

I agree with Simone that integration with other web2py functionality such as auth makes sense. web2py is a fullstack framework, one of its big advantages is that you don't have to lose time integrating a bunch of different modules because most of what you want already works out of the box. I don't want to have to do my own auth integration. I think it's easy enough to make your custom class inheriting from this one (we could just add a comment to the class saying which methods they need to override).

[Massimo DiPierro]

OK. So I think we should include Simone’s code as it is. In contrib? Simone, would you like to make any revision or do you consider it done?

[Michele Comitini]

My opinion is that "contrib" is misleading in this case since niphlod's code  would be a core feature not something taken ASIS from somewhere else, neither something that can be taken apart easily.

[Niphlod]

beware: depending on where it lands, it may be necessary to include the usual singleton to avoid reistantiating the class over and over in each request. 

There's a rogue "print initialized" statement to avoid "as it is" inclusions :P

[Massimo DiPierro]

this finally is in contrib. I do not think we should include this in Auth because it is simple enough as it is.