Revisiting proxied SSL headers

[Paul McMillan]

About a year ago Luke Plant wontfixed ticket #14597, which requested
that Django add support setting for request.is_secure based on
proxy-added headers (like X-Forwarded-Protocol) when Django is served
by an HTTPS proxy [1].

Luke's reasons for closing were analogous to the
SetRemoteAddrFromForwardedFor header [2], which Django removed because
it was unreliable, really easy to get wrong, and a security issue when
configured incorrectly.

More recently, we changed the way we support the X-Forwarded-Host
header for security reasons [3]. Support defaults to disabled, and
users who need it can turn it back on.

I bring these two examples up, because they are conflicting examples
of how we might handle the SSL header issue. One complicating factor
for the ssl header is that there are many (4 or so) ways of specifying
the same meaning (that the proxy received the request on a secure
connection).

CarlJM's django-secure package [4] solves this problem by requiring
the user to specify which header they want, if they need support for
this.

Luke's concerns about the security of this setting are extremely well
founded. Enabling it when it is not needed is a very serious security
problem, and negates many of the benefits of using SSL. In contrast,
since Django doesn't support this configuration out of the box, we
have users who are losing other benefits that SSL could provide when
Django assumes all requests are insecure. The most striking example of
this is our CSRF protection, where we enforce much more rigorous
requirements on secured connections.

In light of our handling of X-Forwarded-Host (which has known security
issues), I propose that we reconsider support for X-Forwarded-Protocol
(and relatives). The setting should require the user to find out which
header their server is using, and configure it appropriately. This is
enough of a barrier to entry that users won't simply set it to "on"
without thinking. Additionally, if the conversation about including a
django-secure style configuration check command comes to fruition, it
gives us a chance to warn users again about the dangers of needlessly
enabling the setting.

-Paul

[1] https://code.djangoproject.com/ticket/14597
[2] https://docs.djangoproject.com/en/dev/releases/1.1/#removed-setremoteaddrfromforwardedfor-middleware
[3] https://www.djangoproject.com/weblog/2011/sep/09/security-releases-issued/
[4] https://github.com/carljm/django-secure

[Luke Plant]

On 24/09/11 01:06, Paul McMillan wrote:

> CarlJM's django-secure package [4] solves this problem by requiring
> the user to specify which header they want, if they need support for
> this.
>
> Luke's concerns about the security of this setting are extremely well
> founded. Enabling it when it is not needed is a very serious security
> problem, and negates many of the benefits of using SSL. In contrast,
> since Django doesn't support this configuration out of the box, we
> have users who are losing other benefits that SSL could provide when
> Django assumes all requests are insecure. The most striking example of
> this is our CSRF protection, where we enforce much more rigorous
> requirements on secured connections.

Hmm, I hadn't thought of the security implications of 'is_secure'
returning false negatives as well as false positives. We do need it to
be reliable in both ways. In the comments on that ticket I had an attack
scenario where getting is_secure() == True when it should have been
False opened up vulnerabilities, but you have one where the reverse also
opens up vulnerabilities, so we definitely need to reconsider this.
Further, the one I had thought of applies to code that theoretically
exists, or exists outside of Django, but your is code that is definitely
within Django, which swings it the other way.

It is a tricky problem, because I don't know of any perfect solution. My
concern is not only that it is possible to configure incorrectly, it
appears to be virtually impossible to configure correctly, as it appears
to be very hard to get web servers to filter incoming headers, and so
filter a X-Forwarded-Protocol=SSL header that is set by a MITM.

My current thinking is that we go with your suggestion, and to cover the
problem of a faked HTTPS connection that I was concerned about we just
make it clear exactly which way is_secure() is unreliable, and that the
main Apache instance must be configured to do the redirection from HTTP
to HTTPS if needed. (This is already indicated here:
https://docs.djangoproject.com/en/dev/topics/security/#ssl-https but
that would need re-working in light of your proposed changes).

Luke

--
I never hated a man enough to give him his diamonds back. (Zsa Zsa
Gabor)

Luke Plant || http://lukeplant.me.uk/

[Carl Meyer]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 09/24/2011 09:02 AM, Luke Plant wrote:
> It is a tricky problem, because I don't know of any perfect solution. My
> concern is not only that it is possible to configure incorrectly, it
> appears to be virtually impossible to configure correctly, as it appears
> to be very hard to get web servers to filter incoming headers, and so
> filter a X-Forwarded-Protocol=SSL header that is set by a MITM.

Is this actually the case? I know you mention in the ticket that
Webfaction's front-end proxy doesn't filter the header they use
correctly, but do you have any other evidence that it is "very hard to
get web servers to filter incoming headers"? I haven't dug into it
deeply, but I've tested with ep.io and I know they do filter their
proxy-SSL header correctly, and I've also tested my own nginx
reverse-proxy setup, and it enforces the header correctly in all cases,
and the configuration to make it do so was trivial (use proxy_set_header
X-Forwarded-Protocol in both the HTTP and HTTPS cases).

I certainly think the documentation for this feature needs to be
extremely clear that setting it is a security risk unless you are
absolutely sure that your front-end proxy enforces the header correctly.
Perhaps we could also supply some advice on how to check that?

Carl
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk5+IrAACgkQ8W4rlRKtE2fXFQCfXoI8mW3QWLpNGZb+Tj2UFP/0
MHsAnjUybjgm4YqaJIxj1b2sUEDYqI4T
=LPYT
-----END PGP SIGNATURE-----

[Luke Plant]

On 24/09/11 19:34, Carl Meyer wrote:
> On 09/24/2011 09:02 AM, Luke Plant wrote:
>> It is a tricky problem, because I don't know of any perfect solution. My
>> concern is not only that it is possible to configure incorrectly, it
>> appears to be virtually impossible to configure correctly, as it appears
>> to be very hard to get web servers to filter incoming headers, and so
>> filter a X-Forwarded-Protocol=SSL header that is set by a MITM.
>
> Is this actually the case? I know you mention in the ticket that
> Webfaction's front-end proxy doesn't filter the header they use
> correctly, but do you have any other evidence that it is "very hard to
> get web servers to filter incoming headers"? I haven't dug into it
> deeply, but I've tested with ep.io and I know they do filter their
> proxy-SSL header correctly, and I've also tested my own nginx
> reverse-proxy setup, and it enforces the header correctly in all cases,
> and the configuration to make it do so was trivial (use proxy_set_header
> X-Forwarded-Protocol in both the HTTP and HTTPS cases).

OK, maybe I was generalising from Apache. I do remember searching for a
way to get Apache to do it, and there didn't seem to be any obvious
solution. I've got a feeling I looked into at least one other popular
web server. I may also have been basing my information on how hard it
seems to have been to get WebFaction to fix this - 2 years and waiting,
and usually are very responsive if there is an easy fix. However, it may
just have been overlooked.

I'm happy to be proved wrong, of course. Apache is very popular, though,
so if its hard in Apache, it could be said to be hard full stop.

Regards,

[Tom Evans]

On Sat, Sep 24, 2011 at 9:28 PM, Luke Plant <L.Pla...@cantab.net> wrote:
>
> I'm happy to be proved wrong, of course. Apache is very popular, though,
> so if its hard in Apache, it could be said to be hard full stop.
>

RequestHeader unset X-Forwarded-Protocol

Not precisely what I'd call hard.

From a-business-that-uses-this-sort-of-stuff POV, we don't really mind
if this is fixed in Django at all. A lot of the enterprise bits that
our architecture needs (SetRemoteAddrFromForwardedFor, SSL middleware)
was either missing, or crippled with no intention to fix it
(SetRemoteAddrFromForwardedFor couldn't handle being behind multiple
proxies and was finally dumped).

It was pointed out to us that what we do is not particularly like the
majority of users, so we have been maintaining our own middleware
classes that updates or monkey patches the request object with the
appropriate info.

I suppose it is analogous to DB routers. Django doesn't provide
routers to handle the common ways to scale a database, but they are
simple enough to write for your specific setup. There is a simple way
to add your own fixups to requests, and it works, so we don't need to
burden the core or contrib with it.

Cheers

Tom

[Luke Plant]

On 26/09/11 12:45, Tom Evans wrote:
> On Sat, Sep 24, 2011 at 9:28 PM, Luke Plant <L.Pla...@cantab.net> wrote:
>>
>> I'm happy to be proved wrong, of course. Apache is very popular, though,
>> so if its hard in Apache, it could be said to be hard full stop.
>>
>
> RequestHeader unset X-Forwarded-Protocol
>
> Not precisely what I'd call hard.

I am indeed happy to have been proved wrong :-) ... if slightly
embarrassed...

I suppose we should check that this definitely works in conjunction with
mod_proxy and whichever module it is that sets X-Forwarded-Protocol/Ssl.

> I suppose it is analogous to DB routers. Django doesn't provide
> routers to handle the common ways to scale a database, but they are
> simple enough to write for your specific setup. There is a simple way
> to add your own fixups to requests, and it works, so we don't need to
> burden the core or contrib with it.

Given the security problems of getting HttpRequest.is_secure() wrong
either way, and the common solution to this particular problem, I think
it is better to have support in the core for this.

Luke

--
"I regret I wasn't born with opposable toes." (Calvin and Hobbes)

Luke Plant || http://lukeplant.me.uk/

[Cal Leeming [Simplicity Media Ltd]]

Just my two cents worth, but I think something like this is such a 'per case basis', that it probably shouldn't be included in the core.

Unless you can guarantee that all web application servers/load balancers are going to correctly handle the header out of the box (i.e. inject/strip where necessary), then there's no way this could be "securely" introduced. 

The reason I say per case basis, is because we've had to implement this same middleware ourselves into multiple clients, all of which had to be slightly different due to the handling of the SSL header at the load balancer.

Cal

Luke Plant || http://lukeplant.me.uk/

--

You received this message because you are subscribed to the Google Groups "Django developers" group.
To post to this group, send email to django-d...@googlegroups.com.
To unsubscribe from this group, send email to django-develop...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.

[Carl Meyer]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 09/26/2011 06:16 AM, Cal Leeming [Simplicity Media Ltd] wrote:
> Unless you can guarantee that all web application servers/load balancers
> are going to correctly handle the header out of the box (i.e.
> inject/strip where necessary), then there's no way this could be
> "securely" introduced.

The proposal is not to do anything automatically or by default, but to
require the user to explicitly set what header to look for, with
documentation warning them that they should only do this if they know
that the header is always set by their proxy.

I agree with Luke and Paul that support for this should be in core - as
they've discussed, the status quo is in itself a security problem. And
while fixing it with a custom middleware is certainly possible, it
requires monkey-patching the request.is_secure() method. Anytime we have
to tell a sizable percentage of our users "you really should add this
custom code to your project that monkeypatches Django core" it's a
pretty strong signal that core should provide better hooks instead.

> The reason I say per case basis, is because we've had to implement this
> same middleware ourselves into multiple clients, all of which had to be
> slightly different due to the handling of the SSL header at the load
> balancer.

Can you be more specific here? The only implementation differences that
I can imagine would be which header is checked, and which value is
expected to indicate SSL vs non-SSL. The proposed hook (currently
implemented in django-secure) already accounts for this variation by
requiring the user to explicitly specify the header and value that
should indicate an SSL request.

Carl
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk6AqrMACgkQ8W4rlRKtE2dxmwCfRf+gXWLvm/Bup6r0ySJ0qVVU
luMAmwd5XT98WU/kIWLhbUy8NA3aPy7K
=AS+A
-----END PGP SIGNATURE-----

[Cal Leeming [Simplicity Media Ltd]]

On Mon, Sep 26, 2011 at 5:39 PM, Carl Meyer <ca...@oddbird.net> wrote:

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 09/26/2011 06:16 AM, Cal Leeming [Simplicity Media Ltd] wrote:
> Unless you can guarantee that all web application servers/load balancers
> are going to correctly handle the header out of the box (i.e.
> inject/strip where necessary), then there's no way this could be
> "securely" introduced.

The proposal is not to do anything automatically or by default, but to
require the user to explicitly set what header to look for, with
documentation warning them that they should only do this if they know
that the header is always set by their proxy.

Ah - in that case (is it has to be configured by the user first, and doesn't take any assumptions) then that would be much better). I thought you meant a patch to make this "just work - out of the box".

 

I agree with Luke and Paul that support for this should be in core - as
they've discussed, the status quo is in itself a security problem. And
while fixing it with a custom middleware is certainly possible, it
requires monkey-patching the request.is_secure() method. Anytime we have
to tell a sizable percentage of our users "you really should add this
custom code to your project that monkeypatches Django core" it's a
pretty strong signal that core should provide better hooks instead.

> The reason I say per case basis, is because we've had to implement this
> same middleware ourselves into multiple clients, all of which had to be
> slightly different due to the handling of the SSL header at the load
> balancer.

Can you be more specific here? The only implementation differences that
I can imagine would be which header is checked, and which value is
expected to indicate SSL vs non-SSL. The proposed hook (currently
implemented in django-secure) already accounts for this variation by
requiring the user to explicitly specify the header and value that
should indicate an SSL request.

Yeah - the thing you mentioned above about forcing the user to configure the exact header tackles this point.

The problem I had previously, was that some load balancers don't forcibly send or strip certain SSL offloading headers (such as X-Forwarded-Proto), which could result in the user being able to trick the web application into thinking it's in HTTPS, when it's not. But this isn't an issue if it's "configured on demand".

 

Carl
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk6AqrMACgkQ8W4rlRKtE2dxmwCfRf+gXWLvm/Bup6r0ySJ0qVVU
luMAmwd5XT98WU/kIWLhbUy8NA3aPy7K
=AS+A
-----END PGP SIGNATURE-----

--