GSoC Check-in: Security Enhancements

[Rohan Jain]

Hi,

I am Rohan Jain, a student from Indian Institute of Technology,
Kharagpur. I'll be doing a Google Summer of Code project with django
this year under the title "Security Enhancements". As the title
suggests, it has something to do with Security Enhancements: like
improvements in CSRF protection and tokenization.

I have made some small updates to the proposal with the feedback it
got. It is under VC over here: http://gist.github.com/2203174
There isn't a direct way to diff gists, so here are the changes I did
if somebody has already read the proposal:

- The origin check will be an additional step to ensure a valid
request and not standalone. The conventional checks will still
exist.

- Add some issues Luke pointed out about signing and using sessions.

- Add info about my github fork and branches.

What I will be doing the following week:

- I haven't done any major contribution to django yet apart from a
tiny ticket some time ago. So, I'll be working on an ticket next
few weeks. It is related to filesystem backend of contrib.sessions,
was raised some time ago:
https://code.djangoproject.com/ticket/18194

- Cleanup and organize the proposal a bit more (Probably start
tracking it as the CSRF protection page -
https://code.djangoproject.com/wiki/CsrfProtection)

(I have also appended the current proposal below in this post)

--
Rohan


Proposal
--------

#Abstract

Django is a reasonably secure framework. It provides an API and
development patterns which transparently take care of the common web
security issues. But still there are security features which need
attention. I propose to work and improved CSRF checking without any
compromises and on integration of existing work on centralized token
system. If time permits I will also attempt on integration of
django-secure.

#Description
##CSRF Improvements

Cross-Origin Resource Sharing (CORS):
W3C has a working draft regarding [CORS][w3c-cors-draft], which opens
up the possibility for allowing client-side request cross-origin
requests. This directly triggers in mind the capability to develop
API which can be exposed directly to the web browser. This would let
us get rid of proxies and other hacks used to achieve this.
Currently all the major browsers support this: Chrome (all versions),
Firefox (> 3.0), IE (> 7.0), Safari (> 3.2), Opera (> 12.0). Firefox
and Chrome send the origin header for both AJAX and standard from POST
requests. Introduced it here as some further parts of the post refer
to this.

###Origin checking

With CORS around need for using CSRF token can be dropped, at least in
some browsers. [Ticket #16859][orig-check-ticket], is an attempt for
that. But this was rejected because of neglecting the case for
presence of `CSRF_COOKE_DOMAIN` (Refer to the closing comment on the
ticket for details). So to handle this we need to simulate checking of
CSRF cookie domain as web browsers do it. Maybe:

```python

reqest.META.get('HTTP_ORIGIN').endswith(settings.CSRF_COOKIE_DOMAIN)

```

In case the server receives an origin header in the request, it will
be used for an initial checking and then all the conventional checks
will be done. The general security will automatically be improved with
the increased market share of newer browsers which support Origin
Header.

As the closing comment points it out, we can't do this with secure
requests. They need to be essentially checked against the referrer or
origin, at least for now. We can not be sure that some untrusted or
insecure subdomain has not already set the cookie or cookie domain.
To deal with this, we have to consider https separately as it is
being done now. So it will be something like:

```python
def process_view(self, request, ....):

# Same initial setup

if request.method not in ('GET', 'HEAD', 'OPTIONS', 'TRACE'):

host = request.get_host()
origin = reqest.META.get('HTTP_ORIGIN', "")
cookie_domain = settings.CSRF_COOKIE_DOMAIN

if request.is_secure():
good_referer = 'https://%s/' % host
referer = origin or request.META.get('HTTP_REFERER')
# Do the same origin checks here

# We are insecure, so care less
# A better way for this check can be used if needed
elif origin.endswith(cookie_domain):
# Safe, continue conventional checking

# Do the conventional checks here
```

If the above were to be implemented, the setting `CSRF_COOKIE_DOMAIN`
should be deprecated for something like `CSRF_ALLOWED_DOMAIN` which
makes more sense.

###Multiple Allowed Domains (was Better CORS Support)
Since, already introducing Origin checking, we can go one step further
and try to provide better support for CORS for browsers supporting it.
A tuple/list setting, which specifies allowed domains will be
provided. Using this the various access control allowance response
headers will be set when the request origin is from amongst the
allowed domains. For CSRF check, just see if http origin is an allowed
domain.

```python

def set_cors_headers(response, origin):
response['Access-Control-Allow-Origin'] = origin

def process_response(self, request, response):

origin = reqest.META.get('HTTP_ORIGIN', "")

if origin in settings.CSRF_ALLOWED_DOMAINS:
set_cors_headers(response, origin)

def process_request(self, request, response):

# Use origin in settings.CSRF_ALLOWED_DOMAINS here instead of
# origin.endswith

```

Probably, something similar to the above will be needed to incorporate
the CORS support.

###Less restrictive secure requests

The current CSRF system is pretty much secure as it is. But CSRF
protection poses too much restriction to https. It says no to all the
request, without honouring any tokens. It kind of has to, thanks to
the way browsers allow cookie access. A cookie accessible through
subdomains mean that any subdomain secure or insecure can set the CSRF
token, which could be really serious for the site security. To get
around this, currently one has to completely exempt views from CSRF
and may or may not handle CSRF attacks. This can be dangerous. Also if
a person has a set of sites, which talk to each other through clients
and decides to run it over https, it would need some modifications.

Django should behave under https similarly as it does under http
without compromising any security. So, we need to make sure that the
CSRF token is always set by a trusted site. Signing the data with the
same key, probably `settings.SECRET_KEY`, across the sites looks apt
for this, using `django.core.signing`. We can have `get_token` and
`set_token` methods which abstract the signing process.
This can be done in two ways:

- Store CSRF data in sessions data in case `contrib.sessions` is
installed. Then the data will automatically be signed with the
secret key or will not be stored in the client as cookies at all.

- In case of it being absent from installed apps, revert to custom
signing

- Encryption?

```python
from django.core.signing import TimestampSigner

signer = TimestampSigner("csrf-token")
CSRF_COOKIE_MAX_AGE = 60 * 60 * 24 * 7 * 52


def get_unsigned_token(request):
# BadSignature exception needs to be handled somewhere
return signer.unsign(request.META.get("CSRF_COOKIE", None)
max_age = CSRF_COOKIE_MAX_AGE)

def set_signed_token(response, token):
response.set_cookie(settings.CSRF_COOKIE_NAME,
signer.sign(request.META["CSRF_COOKIE"]),
max_age = CSRF_COOKIE_MAX_AGE,
domain=settings.CSRF_COOKIE_DOMAIN,
path=settings.CSRF_COOKIE_PATH,
secure=settings.CSRF_COOKIE_SECURE
)


def get_token(request):
if 'django.contrib.sessions' in settings.INSTALLED_APPS:
return request.session.csrf_token
else:
return get_unsigned_token(request)

def set_token(response, token)
if 'django.contrib.sessions' in settings.INSTALLED_APPS:
request.session.csrf_token = token
else:
set_signed_token(response, token)

# Comparing to the token in the request
constant_time_compare(request_csrf_token, get_token(csrf_token))

```

Now, doing this is not as simple as the above code block makes it
look. There is a lot which can and probably will go wrong with this
approach:

- Even when the token is signed, other domains can completely replace
the CSRF token cookie, it won't grant them access through CSRF
check though. Even with signing, they just need to replay an
existing good token/cookie pair, which they can get directly from
the server any time they want.

- This sort of couples CSRF with sessions, a contrib app. Currently
nothing except some of the other contrib apps are tied up with
sessions. It will break if sessions were to be removed in future or
the API changed. Also, this means that if one website is using
sessions CSRF, all of the other must be too. It would actually kind
of be a step because of the coupling.

- If this were successfully implemented, is this exposing any
critical security flaws otherwise? Will it cause compatibility
issues?

- Encryption itself comes with its own issues. It will need high
considerations.

As Paul McMillan said "This is a hard problem", I'll delegate figuring
this to future me. I will look into [The Tangled Web][tangled-web]
and [Google's Browser Security Handbook][gobrowsersec] for ideas,
again suggested by Paul on the IRC.

##Centralized tokenization
There are multiple places in django which use some or other kinds of
tokens:

- contirb.auth (random password, password reset)
- formtools
- session (backends)
- cache
- csrf
- etags

Token generation is pretty common around the framework. So, instead
of each application having its own token system, and hence needs to be
maintained separately. There should be centralized token system, which
provides an abstract API for everyone to loose. In fact, I have seen
that some apps use `User.objects.make_random_password` from
contrib.auth, which they can be sure of being maintained in the future
for random generation. To me this looks kind of weird.
In last djangocon, a lot of work regarding this was done over [Yarko's
Fork][yarko-fork].

I had a discussion with Yarko Tymciurak regarding this. The work is
nearly ready for a merge, only some tasks left. I can work over these
to insure that the already done significant work gets in django and is
updated for 1.5.

- Porting more stuff to the new system (README.sec in
[yarko's fork][yarko-fork])
- Testing - See if the current coverage of the tests is enough, write
them if not.
- Compatibility issues
- API Documentation

I will study the changes done at djangocon and then attempt the tasks
mentioned above.


##Integrating django-secure
A really useful app for catching security configuration related
mistakes is [carljm's django-secure][djang-secure]. It is specially
useful to find out issues that might have been introduced while quick
changes to settings for development. This project is popular and
useful enough that it can be shipped with django. I haven't been able
give this enough time yet. I can think of two ways of integrating
this:

- Dropping it as a contrib app
This seems pretty straight forward would require minimal amount of
changes.

- Distribute around the framework:
Like CSRF, this can also be distributed framework wide and hence it
won't be optional to have. Apps can still define custom checks in
the same way when `django-secure` was installed as a pluggable
application.

The app might also need some changes whilst being integrated:

- More security checks, if required
- Adjust according to the changes introduced above.

#Plan
I think that the tasks CSRF enhancements and centralized tokenization
will be enough to span through the SoC period. If after a thorough
implementation and testing of these, I still have time, django-secure
integration can be looked into.


Roughly this proposal can span over a maximum of 5 tasks. Each task
will generally have the following steps:

a. Initial Research. Design decisions
b. Implementation with minor parallel tests.
c. Thorough and regression testing to to achieve security quality.
d. Configuration/Settings changes and handle compatibility issues.
e. Documentation.

Tasks (with most effort requiring steps in parenthesis):

1. Origin Checking (b, c)
2. Multiple Allowed Domains (b, c)
3. Less restrictive CSRF checking over HTTPS / CORS for HTTPS (a, b)
4. Unified Tokenization (a,c,e)
5. Integration of django-secure (d,e)

I'll be using [my fork of django][gh-fork] over github. I'll probably
use the following branch names:
csrf-enhancements (origin checking, multiple request domains etc)
centralized-tokenization (djangocon2011-sec)

##Timeline
Week 1: Task 1.a, 1.b.
Week 2: Task 1.c, 1.d
Week 3: Task 2.a, 2.b. Start task 3.a
Week 4: Task 2.c, 2.d
Week 5: Task 1.e, 2.e (Doing these together might be beneficial)
Week 6-7: Complete 3.a. Task 3.b
Week 7-9: Task 3.c, 3.d
Week 10: Task 3.e
Week 11-12: Tasks 4.abcde (max possible)
Week 13: Complete Task 4 and maybe Max of Task 5

*I am sorry for writing these as if written by a bot, the deadline was
so close so had to adopt this method*.

[yarko-fork]: https://github.com/yarko/django
[w3c-cors-draft]: http://www.w3.org/TR/access-control/
[orig-check-ticket]: https://code.djangoproject.com/ticket/16859
[tangled-web]: http://www.amazon.com/The-Tangled-Web-Securing-Applications/dp/1593273886/
[gobrowsersec]: http://code.google.com/p/browsersec/wiki/Main
[django-secure]: https://github.com/carljm/django-secure
[gh-fork]: https://github.com/crodjer/django

[Rohan Jain]

Hi,

Last week I looked into the Ticket [#18194][0]:

- Trivial attempts to handle the issue.
- Wrote a minor initial patch.
- The test fails for Cache and Cookie backend.

Also, I looked at the talks from Paul regarding advanced security
topics at py/django cons. Realised that why I should not attempt
anything related to encryption in my project.

There is high academic pressure currently, so I am not able to give
enough time to these. I think the situation will be better this
weekend onwards.

I'll try to work on:

- Write tests which emulate the problem in #18194 well, and then work
on the final fix.
- Start looking into resources useful for my project, like [The
Tangled Web][1].

Rohan Jain


[0]: https://code.djangoproject.com/ticket/18194
[1]: http://www.amazon.com/The-Tangled-Web-Securing-Applications/dp/1593273886

[Rohan Jain]

Hi,

Since my last check in I worked on improvements to
contrib.sessions:

- Introduction of signing framework
- Session expiry checks (for Ticket [#18194][0]
- And some other trivial patches.

The tests (existing and the one which I added) are passing.

These changes are in my [Pull Request #78][1] over github.
Paul, could you please review it to see if the patches are usable.

Next, I'll make the changes which may be required in documentation
because of the above.
Today is official start date of the GSoC project, so I'll now start
concentrating more on the project now.

Rohan Jain

[0]: https://code.djangoproject.com/ticket/18194

[1]: https://github.com/django/django/pull/78

On Mon, May 7, 2012 at 12:21 PM, Rohan Jain <cro...@gmail.com> wrote:
> Hi,
>

[Rohan Jain]

I have done some work on CSRF revolving around origin header checking.
The origin header is fairly new and is not yet implemented in a
uniform fashion in the major browsers, so it cannot be solely relied
upon for CSRF checks. Instead we check if the header exists and use it
only for rejection of requests.

If an `HTTP_ORIGIN` header is sent by the browser, it checks if it
matches to be from a valid origin. In case it is not the request is
rejected right away. Otherwise it proceeds for further CSRF checks

Because of the way browser cookies behave, I had to do some unpleasant
tweaks to origin header checking such that `COOKIE_DOMAIN_SETTING` is
followed. From the commit:

if not ((good_origin.startswith('.')
and good_origin.count('.') is origin.count('.')
and origin.endswith(good_origin))
or origin[origin.find('://')+3:] == good_origin):

All this is because using cookies open the possibility for allowing
cross subdomain requests (a side effect?). I had a chat with Paul for
breaking (fixing) this behaviour. The plan is to:

- Make the strict referer checking, currently implemented only for
HTTPS, default for http requests too. The http scheme which is more
popular definitely deserves better checking.
- Make the origin checking work similar to the way referer checking
is done. Also, will it be safe to bypass referer checking in case
of origin header being present?
- Implement a PERMITTTED_DOMAINS setting which lets administrators
explicitly mention the domains which are permitted to make the
otherwise restricted requests. This gives more control to the
administrators.
- The PERMITTED_DOMAINS setting, a list, accepts patterns. Regex
cannot be used here because characters like `.`, `-` will need
escaping. So I decided to settle on the simpler unix style globs.
The pattern matching is done through the `fnmatch` library
function, documented [here][fnmatch-docs]. One can use this setting
like this:

PERMITTED_DOMAINS = [
'www.example.com', # Exactly the domain `www.example.com`
'*.supertrusteddomain.com', # All subdomains to `supertrusteddomain.com`
'?.services.example.com', # Single letter subdomains to `.services.example.com`
]

I have done an initial implementation of these, changes in [pull
request #95][pull-95]. I'll now proceed to clean these up, writing
better tests and documentation for these. Also with these, we can
completely get rid of the cookie based CSRF check system.

--
Thanks
Rohan Jain

[fnmatch-docs]: http://docs.python.org/library/fnmatch.html
[pull-95]: https://github.com/django/django/pull/95

[Rohan Jain]

Hi,

This check-in is a little delayed. Meanwhile, I continued working on
CSRF and started a on centralized tokenization.

As I had posted, I implemented the permitted domains system, as
`CSRF_PERMITTED_DOMAINS`. I did some minor cleanups and tests, the
behaviour is mostly the same. Since with the origin header peresent,
referer checking can be neglected, now the system bypasses referer
checking when origin header is not none.

The new permitted domains system is practically invalidated because of
the existing csrf cookie and token functionality. So for this, I
created a separate branch on my fork, which goes by
[purge-cookies][0]. In this I'll try to remove the CSRF cookie
functionality without affecting the interface which CSRF checking
system provides here. This interface could generate deprecation
warnings about CSRF cookie setting and csrf middleware token. Later we
can remove this interface completely too.

I would also like to get some suggestions if completely shifting
towards header checking will be a good idea. From all I could find out
from other frameworks out there, tokens and cookies are used to save a
state for CSRF checks generally. Header checks are hardly used
anywhere. Is it because of some flaw with how the headers system work?
Given that referer checking is already being used for request which
use the https scheme, I am assuming this is okay to do and will only
decrease the vulnerabilites which exists from cookies system.

If there are some reasons for using a cookie, we could at least start
siging them. Probably have the post form send a signed (and timed?)
hash apart from the csrf token for verification. Because otherwise it
is easy to break the system if the attacker has access to some other
subdomain, maybe on a shared hosting.

Awaiting some feedback on CSRF, I have now started looking into
integrating a centeralized tokenization system with django. Work for
this already exists, done in djangocon 2011 and exists at
[yarko/django][1] on github. The commits were on django-old, so it
can't be merged directly. I cherry picked commits by commits and ran
the merge conflicts with my best guesses. The result of the merge
(broken) exists at [centralized-tokenization][2] branch of my fork.
I'll have to look into this one more and see if I get it to work, as
it does at yarko/django.

Also, my first change sets regarding contrib.sessions are still
pending some feedback, at [pull-78].[3].

--
Thanks
Rohan Jain

[0]: https://github.com/crodjer/django/tree/purge-cookies
[1]: https://github.com/yarko/django
[2]: https://github.com/crodjer/django/commits/centralized-tokenization
[3]: https://github.com/django/django/pull/78

[Rohan Jain]

Hi,

Centralized Tokenization:
I have merged the work already done for centralized-tokenization in
last djangocon at [yarko/django][0]. Since it there have been
significant amount of changes since then, merging and resolving
conflicts took a little more time then expected. As of now the tests
are passing. I am yet to do some minor work to move remaining
components to the token system. I'll post it up by tonight or tomorrow
and then document how the system works.

CSRF:
Clubbing the information regarding CSRF, I have started writing down
an overview about it in one place, listing the pros and cons of the
methods for CSRF checking. Its in my same soc proposal gist:
<https://gist.github.com/2203174#file_csrf.mkd>
Also added the content of the gist in the end of this mail.


--
Thanks
Rohan

[0]: https://github.com/yarko/django
[1]: https://github.com/crodjer/django/commits/centralized-tokenization

## Current state

Django currently uses two methods for protection against CSRF:

- In case of http non SSL requests, a conventional token system is used.
- In case of https requests, we do referer checking.

Both these system introduce a difference in behaviour. Using cookie-token
system provides a side effect feature which lets you access the CSRF token
across multiple subdomains. But the other, referer checking looks for the
request to come from the same host and hence, doesn't allow for cookie like
cross subdomain behaviour.

The fact that there is a `CSRF_COOKIE_DOMAIN` setting shows that django supports
the cross domain capabilities in the CSRF system. So users might utilize this
while building multiple sites. When they change to https scheme, they will see
a new surprising behaviour.

## The modified system
The working tree is at [crodjer/django](https://github.com/crodjer/django/tree/csrf-enhancements)
over github.

###Origin and Referer Checking, Permitted domains.

- Origin header, a fairly new introduction, has CSRF as one of the target.
Unlike the referer header, it sends back only the domain and the path
information, so the people don't need to disable these through plugins like
the referer header.
- With origin header absent, referer header can be used as a fallback.
- Strict referer checking is already there in https scheme which is used in all
major deployments. We can bring it to the non secure scheme also.
- With this method a permitted domains functionality is possible, which lets
explicit specification of domains which are trusted and can bypass CSRF. A
prototype implementation is there at crodjer/django, csrf-enhancements
branch.

- Cons:

- Since origin checking is still a new concept, it is not implemented yet in
major browsers.
- Referer headers spoofing is also heard of in some old versions of flash.
- Plugins which disable referer header are popular. A client which doesn't
provide origin header and with one such plugin, will fail against this
kind of checking.

###CSRF Cookies (Time signed):

- A random token generated by the server stored in the browser cookies. For
verification, every non get request will need to provide a signed version of
the same token. This can then be verified on the browser side.
- This can be implemented by adding signing to the existing csrf token system,
using the signing framework.
- A conventional method of CSRF checks, all the major frameworks have similar
systems.
- Signing takes care of the side effects due to cross domain behaviour of
cookies.
- Cons:

- Relies on the browser cookies system, which introduces insecurities.
- Can be broken easily by having a parallel legitimate session, which gives
a valid token, signature pair. This generator can then be used in MITM
attacks.

Given these two methods, I am slightly inclined at Origin/Referer checking,
which is already being completely relied upon in case of the secure requests.

Or we could loose the possibility of permitted domains functionality and
implement both kinds of checks.

[Luke Plant]

On 23/07/12 08:07, Rohan Jain wrote:
> ###CSRF Cookies (Time signed):
>
> - A random token generated by the server stored in the browser cookies. For
> verification, every non get request will need to provide a signed version of
> the same token. This can then be verified on the browser side.
> - This can be implemented by adding signing to the existing csrf token system,
> using the signing framework.
> - A conventional method of CSRF checks, all the major frameworks have similar
> systems.
> - Signing takes care of the side effects due to cross domain behaviour of
> cookies.
> - Cons:
>
> - Relies on the browser cookies system, which introduces insecurities.
> - Can be broken easily by having a parallel legitimate session, which gives
> a valid token, signature pair. This generator can then be used in MITM
> attacks.

I don't understand the motivation for this part. Could you describe the
attack scenarios where this method improves our security with respect to
the current system?

Regards,

Luke

--
"Pretension: The downside of being better than everyone else is
that people tend to assume you're pretentious." (despair.com)

Luke Plant || http://lukeplant.me.uk/

[Rohan Jain]

On 11:06 +0100 / 23 Jul, Luke Plant wrote:
> On 23/07/12 08:07, Rohan Jain wrote:
> > ###CSRF Cookies (Time signed):
> >
> > - A random token generated by the server stored in the browser cookies. For
> > verification, every non get request will need to provide a signed version of
> > the same token. This can then be verified on the browser side.
> > - This can be implemented by adding signing to the existing csrf token system,
> > using the signing framework.
> > - A conventional method of CSRF checks, all the major frameworks have similar
> > systems.
> > - Signing takes care of the side effects due to cross domain behaviour of
> > cookies.
> > - Cons:
> >
> > - Relies on the browser cookies system, which introduces insecurities.
> > - Can be broken easily by having a parallel legitimate session, which gives
> > a valid token, signature pair. This generator can then be used in MITM
> > attacks.
>
> I don't understand the motivation for this part. Could you describe the
> attack scenarios where this method improves our security with respect to
> the current system?

With this, attacker won't be able to directly set arbitrary tokens on
other sub domains through cookies, they will need a signature of the
token with the form which is to be verified against the cookie.
Plus it also puts a limit on the duration a token stays valid on the
server side.

Yes, still with this, someone can spoof the whole pair using a separate
legitimate session. So really it doesn't make it completely secure,
just makes it little difficult for the attacker.

If it weren't for the possibility of attacker injecting cookies from
other subdomains, I think CSRF token should be a fine check for
CSRF.

That is why I am siding on adding referer checking in case of non
https scheme requests too.

--
Thanks
Rohan

[Luke Plant]

On 23/07/12 14:24, Rohan Jain wrote:
> With this, attacker won't be able to directly set arbitrary tokens on
> other sub domains through cookies, they will need a signature of the
> token with the form which is to be verified against the cookie.
> Plus it also puts a limit on the duration a token stays valid on the
> server side.
>
> Yes, still with this, someone can spoof the whole pair using a separate
> legitimate session. So really it doesn't make it completely secure,
> just makes it little difficult for the attacker.

So, to make it clear:

Attacker controls evil.example.com, and wants to attack example.com. By
appropriate setting of a cookie, and by providing a matching token in
the form, they can forge a request to example.com

With your proposed change, they are in no way hampered. Using HMAC to do
sign the value, and a timestamp, as you mention, they can simply
regularly directly contact example.com and pull in a valid token/cookie
value pair, which they can use since there is no correlation to the
session of the person being attacked. I'm pretty sure this can be done
entirely in Javascript too.

Yes, this change would make it "a little difficult". But the value of
"little" is very small. It's like suggesting we add ROT 13 encryption to
one of the values - sure it makes it a "little" more difficult, but it
does not *materially* affect the feasibility of the attack. The
resources they need are identical: the ability to read the Django source
code, and the subdomain control that they have in both cases.

On the other hand, this adds significant complication to our code, which
is the last thing you need for security related code. I'm -1 on this
change. I did highlight all these things before.

> If it weren't for the possibility of attacker injecting cookies from
> other subdomains, I think CSRF token should be a fine check for
> CSRF.
>
> That is why I am siding on adding referer checking in case of non
> https scheme requests too.

I really don't think we can consider this - for HTTP, proxies can and do
strip the referer header. Quoting from Barth, Jackon and Mitchell:

<<<
Over HTTP, sites cannot afford
to block requests that lack a Referer header because
they would cease to be compatible with the sizable
percentage (roughly 3–11%) of users
>>

This makes strict referer checking a non-started, and lax referer
checking (only check it if it is present) has known flaws.

[Rohan Jain]

Thanks for this quote. I didn't know about this paper and the fact
that the web behaves differentially with respect to referer checking
in http from https earlier. So, yes relying on it isn't the best idea.

I had one more idea, "Pluggable CSRF checkers".

Currently, the CSRF middleware has two kinds of checks, referer (for
https) and secret validation token (common). These with origin header
based checker (if we add it) come in conditional blocks, making
switching them difficult. So what I propose to do is decouple their
logic from CSRF middleware and each of them provide a checker. It goes
like this:

A setting for configuring global CSRF checkers:

CSRF_CHECKERS = {
'django.middleware.csrf.checkers.OriginChecker',
# This one can be strict for https and lax for http
'django.middleware.csrf.checkers.RefererChecker',
# contrib.sessions could provide a csrf checker maintained
# with sessions. This stores the token in session data.
'django.contrib.sessions.csrf_checkers.SessionChecker'
}

Views can have their own custom checkers:

from django.view.decorators.csrf import csrf_checkers

@csrf_checkers(
'django.middleware.csrf.checkers.OriginChecker',
'django.middleware.csrf.checkers.TokenChecker',
)
def foo(request):
'''This view needs token and origin for CSRF checks. For some
reason session can't be used here.'''

@csrf_checkers(
'django.middleware.csrf.checkers.OriginChecker',
'django.middleware.csrf.checkers.SignedTokenChecker',
)
def bar(request):
'''This view for some reason needs time signed token and
origin for CSRF checks.'''


A base checker class:

class BaseCSRFChecker(object):

def __init__(request, *args, **kwargs):
self.request = request

def _accept(self, request):
request.csrf_processing_done = True
return None

def _reject(self, request, reason):
return _get_failure_view()(request, reason=reason)

def _check(strict=True):
raise NotImplementedError ("Checkers must implement a checking mechanism.")

def check(strict=True):
# Checkers will implement the method in their
self._check(strict=strict)

With this I can see some benefits:

- More explicit about the presence of CSRF checking.
- Easier to modify CSRF behaviour in the future.
- Possibility of using contrib.sessions for CSRF, while not
completely tying CSRF to the app.
- Custom checkers can make people prevent using csrf_exempt for some
views (Maybe something which isn't supposed to be accessed with
cookies).

This might look like too much overkill or complication but I think if
not making it faster, this at least won't add any overhead to the
middleware.

--
Thanks
Rohan

[Alex Ogier]

On Tue, Jul 24, 2012 at 11:37 PM, Rohan Jain <cro...@gmail.com> wrote:
>
> I had one more idea, "Pluggable CSRF checkers".
>
> Currently, the CSRF middleware has two kinds of checks, referer (for
> https) and secret validation token (common). These with origin header
> based checker (if we add it) come in conditional blocks, making
> switching them difficult. So what I propose to do is decouple their
> logic from CSRF middleware and each of them provide a checker. It goes
> like this:
>
> A setting for configuring global CSRF checkers:
>
> CSRF_CHECKERS = {
> 'django.middleware.csrf.checkers.OriginChecker',
> # This one can be strict for https and lax for http
> 'django.middleware.csrf.checkers.RefererChecker',
> # contrib.sessions could provide a csrf checker maintained
> # with sessions. This stores the token in session data.
> 'django.contrib.sessions.csrf_checkers.SessionChecker'
> }
>

I don't think this is a good idea. If you enumerate security features
in settings.py, then later additions won't be picked up by default. If
Django add a new CSRF checking mechanism, we want everybody to take
advantage of it with no modifications.

Ordinarily I agree with you, explicit is better than implicit.
However, in the case of security features, I think this is inverted:
Django sites should be implicitly enrolled in all security mechanisms
if possible, and should be able to explicitly opt out if necessary.
Almost everyone should be using every single protection Django offers
on all their requests, and therefore it should be verbose and
discouraged to turn off these protections.

Best,
Alex Ogier

[Rohan Jain]

Hi,

Sorry for the delay in getting back. I was meanwhile working on
centralized tokenization for few days, while still trying to figure
something better for CSRF.

Hmm, that is a valid point. I can drop the configurable CSRF settings.
But still a modular CSRF checkers might be useful, in which the
checkers are selected dynamically. When sessions app is present, use
sessions checker instead of cookies based CSRF token store. Also we
can have switches which incorporate existing behaviour based on
https/http connection and also origin header checking based on its
presence. I will do a prototype implementation for this to polish and
clarify the idea.


## Centralized Tokenization:

A functioning implementation is up at the [centralize-tokenization][0]
branch on my fork. I have written an [initial documentation][1] for
this too. This is basically a merge with my cleanups and
customizations over the work done in period of [djangocon 2011][2].

[0]: https://github.com/crodjer/django/commits/centralized-tokenization
[1]: https://gist.github.com/2203174#file_tokenization.mkd
[2]: https://github.com/yarko/django/commits/djangocon2011-sec

[Rohan Jain]

Hi,

Today is the 'pencils down' date for this GSoC project. Past 4 months
have been a great learning experience from me. Just being in the
context of security side of the web has been really beneficial. Moving
around in a very well written code base is also delightful.
Meanwhile, I did get to work on multiple sections of the code base.
But compared to the expectations at the initial phase of the program,
I myself am a little disappointed about the output I was able to
generate out of the time - specially regarding CSRF.
This can be attributed to my being unable to give consistent attention
to the project through the duration and to solicit feedback. Also, the
project comprised of multiple discreet tasks, so not enough
consideration was there for each task.

This is a overview of various complete/incomplete topics I worked on.

Centralized Tokenization:
This was about integrating a central tokenization api throughout the
django project. I tracked this at [centralized-tokenization][0] branch
of my fork on github. This is near completion, and also has a
documentation[1] of the API with basic usage examples.

Improvements to sessions:
My first task, related to some improvements to the sessions framework.
Includes signing and session cleanup based on the engine. Tracked at
[sessions-improvements][2] branch.

CSRF Enhancements:
It occupied most of my time through the project. I tried multiple ways
to solve the issues under this. Tracked at [csrf-enhancements][3].
First up, I started with using the origin header for doing a CSRF
check. One implementation with some tests is at
[csrf-origin-checking][4]. Recently I started on the idea of
implementing moduler checkers for each kind of CSRF check, but
haven't got anything useful out of it yet. While progressing, it
seemed like I was virtually writing a middleware per checker, so now I
have moved on to attempt on CSRF cookie store. Basically something
which should seamlessly switch Session/Cookie for storage based on the
availability of session.

I'll try to hang around the django bug tracker and submit some
patches.

--
Thanks
Rohan Jain

[0]: https://github.com/crodjer/django/tree/centralized-tokenization
[1]: https://github.com/crodjer/django/blob/centralized-tokenization/docs/topics/tokenization.txt
[2]: https://github.com/crodjer/django/tree/sessions-improvements
[3]: https://github.com/crodjer/django/tree/csrf-enhancements
[4]: https://github.com/crodjer/django/tree/csrf-origin-checking
[5]: https://github.com/crodjer/django/tree/csrf-moduler-checkers

[Andrew Godwin]

Thanks for your work during the GSOC, Rohan - don't worry about not achieving everything, it looks like there's still some useful code there!

Hopefully we can get some of the code merged, especially centralised tokenisation if it's so near completion, as it looks like a nice bit of cleanup code!

Andrew

--
You received this message because you are subscribed to the Google Groups "Django developers" group.
To post to this group, send email to django-d...@googlegroups.com.
To unsubscribe from this group, send email to django-develop...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.

[Rohan Jain]

Hi,

Thanks for the reply.

On 10:50 +0100 / 21 Aug, Andrew Godwin wrote:
> Thanks for your work during the GSOC, Rohan - don't worry about not
> achieving everything, it looks like there's still some useful code there!
>
> Hopefully we can get some of the code merged, especially centralised
> tokenisation if it's so near completion, as it looks like a nice bit of
> cleanup code!

Hoping that we can merge and ship it by 1.5. There is a possibility of
running into issues still, I'll try to fix those as they come up in
future.

--
Rohan

[Rohan Jain]

Hi,

Ref Merging Centralized Tokenization.
I found out that some tests for sessions are failing with it. Stupidly,
I didn't have sessions in my test subset for tokenization. Anyway,
I'll try to get it fixed and write some tests for tokens in next few
days.

--
Thanks
Rohan

On 10:50 +0100 / 21 Aug, Andrew Godwin wrote: