[Django] #29800: Django hangs when Content-Length hangs
Django
-----------------------------------------+------------------------
Reporter: monester | Owner: nobody
Type: Bug | Status: new
Component: Uncategorized | Version: 2.1
Severity: Normal | Keywords:
Triage Stage: Unreviewed | Has patch: 0
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
UI/UX: 0 |
-----------------------------------------+------------------------
When sending incorrect content-length header to application, django hangs
on reading request.body.
Steps to reproduce:
curl -v -X POST -H "Accept: */*" -H "Content-Length: 22" -d "1"
http://127.0.0.1:8000
with following handler:
{{{#!python
def index(request):
return HttpResponse(content=request.body)
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/29800>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
-------------------------------+--------------------------------------
Reporter: monester | Owner: nobody
Type: Bug | Status: new
Component: Uncategorized | Version: 2.1
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
--
Ticket URL: <https://code.djangoproject.com/ticket/29800#comment:1>
Django
-------------------------------+------------------------------------
Reporter: monester | Owner: nobody
Type: Bug | Status: new
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Tim Graham):
* component: Uncategorized => HTTP handling
* stage: Unreviewed => Accepted
Comment:
I can reproduce this, although I had to add `@csrf_exempt` to the view to
bypass the CSRF error page.
--
Ticket URL: <https://code.djangoproject.com/ticket/29800#comment:2>
Django
-------------------------------+------------------------------------
Reporter: monester | Owner: nobody
Type: Bug | Status: new
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Herbert Fortes):
* cc: Herbert Fortes (added)
--
Ticket URL: <https://code.djangoproject.com/ticket/29800#comment:3>
Django
-------------------------------------+-------------------------------------
Reporter: Alexander Charykov | Owner:
| patriksletmo
Type: Bug | Status: assigned
Component: HTTP handling | Version: 2.1
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by patriksletmo):
* owner: nobody => patriksletmo
* status: new => assigned
--
Ticket URL: <https://code.djangoproject.com/ticket/29800#comment:4>
Django
-------------------------------------+-------------------------------------
Reporter: Alexander Charykov | Owner:
| patriksletmo
Type: Bug | Status: assigned
Component: HTTP handling | Version: 2.1
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Tim Graham):
The security team discussed this issue to confirm that we shouldn't treat
it as a security issue.
From Markus Holtermann:
The issue can also be reproduced with:
$ gunicorn foo.wsgi --access-logfile - --worker-connections 5
From Simon Charette:
I cannot reproduce when Django is behind a reverse proxy such as NGINX
with proxy_request_buffering on which is the default.
Given the development server should never be used in production and
gunicorn documentation strongly recommends to use it behind a proxy server
this doesn't sound severe to me. We should add a timeout on body read to
the development server but it's unlikely to be an attack vector if you
already follow best practice.
I mean, both Django's development server and gunicorn have been known to
be vulnerable to slow loris attacks and friends for a while at this point
at it feels to me that serving Django apps behind a reverse proxy is the
norm nowadays.
From Collin Anderson:
I this has come up before (or maybe it's a related issue):
https://groups.google.com/d/topic/django-developers/bYzHczKNoqM/discussion
From Claude Paroz:
Might be worth reaching to Graham Dumpleton about this issue. The WSGI
specs talk about a CONTENT_LENGTH shorter than the payload, but I'm not
sure it addresses the reverse issue.
--
Ticket URL: <https://code.djangoproject.com/ticket/29800#comment:5>
Django
-------------------------------------+-------------------------------------
| Sletmo
Type: Bug | Status: assigned
Component: HTTP handling | Version: 2.1
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Patrik Sletmo):
I skimmed through the WSGI specification and it contains a description on
how to handle Content-Length headers shorter than the payload, but as
Claude Paroz guessed it does not give any instructions for handling the
opposite case.
As the recommended way of running Django in production is behind a reverse
proxy I don't think that there should be any fix implemented that attempts
to mitigate the effect of a Slowloris attack. I do however think that from
a usability standpoint it could make sense not to hang the entire
application in case the request given above is received.
Would there be any practical implications that could motivate against
using a timeout period for receiving the data? If nginx is used as a
reverse proxy with the default option proxy_request_buffering set, the
timeout would limit the time required for transmitting the request from
the reverese proxy to Django, and this really shouldn't take any
substantial amount of time. I suppose that implementing a timeout option
in Django in some situations could create the need to configure a timeout
in both the reverse proxy and in Django, but I think that these cases are
rather slim.
--
Ticket URL: <https://code.djangoproject.com/ticket/29800#comment:6>
Django
-------------------------------------+-------------------------------------
Reporter: Alexander Charykov | Owner: Patrik
| Sletmo
Type: Bug | Status: assigned
Component: HTTP handling | Version: 2.1
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Patrik Sletmo):
I've been thinking some more on how to handle this issue and right now I'm
considering the possibility of not fixing it at all. The two alternatives
would in that case be the following:
1. Mark the issue as "wontfix".
or
2. Document the issue so that users are aware of its existence, and refer
to the use of a reverse proxy. My suggestion is that it could be added
here somehow https://docs.djangoproject.com/en/2.1/topics/security
/#additional-security-topics.
The reasoning behind not providing any fix beyond what has been suggested
above is simple. The types of HTTP requests triggering the hang are
typically only crafted by a malicious party with the intention to trigger
this sort of bug. I can not imagine any case in which the behaviour
documented in this issue would come as a surprise to anyone performing
this request on their development server. As this bug is nearly never
triggered unintentionally when developing, and also protected against when
deployed properly in production, I think that it seems strange to
introduce additional functionality that must be maintained. I would say
that an introduced timeout comes with more complications than it solves.
Do you have any opinion on the above mentioned proposals, Alexander
Charykov and Tim Graham?
--
Ticket URL: <https://code.djangoproject.com/ticket/29800#comment:7>
Django
------------------------------------+------------------------------------
Reporter: Alexander Charykov | Owner: (none)
Type: Bug | Status: new
Component: HTTP handling | Version: 2.1
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Patrik Sletmo):
* owner: Patrik Sletmo => (none)
* status: assigned => new
Comment:
Since I lack the required insight in the Django project and thus cannot
decide on what solution would be the best one on my own, I've chosen to
deassign myself from this issue. I hope that my reasoning above may become
useful in the future.
--
Ticket URL: <https://code.djangoproject.com/ticket/29800#comment:8>
Django
------------------------------------+------------------------------------
Reporter: Alexander Charykov | Owner: (none)
Component: HTTP handling | Version: 2.1
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Jannik Schürg):
I think this is what it boils down to:
{{{#!python
from wsgiref.simple_server import make_server
def app(environ, start_response):
# does not timeout for malformd 'Content-Length'
environ['wsgi.input'].read()
make_server('', 8000, app).serve_forever()
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/29800#comment:9>
Django
------------------------------------+------------------------------------
Reporter: Alexander Charykov | Owner: (none)
Component: HTTP handling | Version: 2.1
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Zach Garwood):
Replying to [comment:7 Patrik Sletmo]:
> 2. Document the issue so that users are aware of its existence, and
refer to the use of a reverse proxy. My suggestion is that it could be
added here somehow https://docs.djangoproject.com/en/2.1/topics/security
/#additional-security-topics.
I think this is the best course of action.
--
Ticket URL: <https://code.djangoproject.com/ticket/29800#comment:10>
Django
------------------------------------+------------------------------------
Reporter: Alexander Charykov | Owner: (none)
Component: HTTP handling | Version: 2.1
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Old description:
> When sending incorrect content-length header to application, django hangs
> on reading request.body.
>
> Steps to reproduce:
>
> curl -v -X POST -H "Accept: */*" -H "Content-Length: 22" -d "1"
> http://127.0.0.1:8000
>
> with following handler:
> {{{#!python
> def index(request):
> return HttpResponse(content=request.body)
> }}}
New description:
When sending incorrect content-length header to application, django hangs
on reading request.body.
Steps to reproduce:
curl -v -X POST -H "Accept: */*" -H "Content-Length: 22" -d "1"
http://127.0.0.1:8000
with following handler:
{{{#!python
from django.http import HttpResponse
from django.views.decorators.csrf import csrf_exempt
@csrf_exempt
def index(request):
return HttpResponse(content=request.body)
}}}
--
Comment (by Manan):
Added a csrf_exempt to bypass the default 403 CSRF error page
--
Ticket URL: <https://code.djangoproject.com/ticket/29800#comment:11>
Django
------------------------------------+------------------------------------
Reporter: Alexander Charykov | Owner: (none)
Component: HTTP handling | Version: 2.1
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Sam Kuffer):
* Attachment "doc_update.diff" added.
Django
------------------------------------+------------------------------------
Reporter: Alexander Charykov | Owner: (none)
Component: HTTP handling | Version: 2.1
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Sam Kuffer):
* cc: Sam Kuffer (added)
--
Ticket URL: <https://code.djangoproject.com/ticket/29800#comment:12>
Django
------------------------------------+------------------------------------
Reporter: Alexander Charykov | Owner: (none)
Component: HTTP handling | Version: 2.1
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Adam Zapletal):
I confirmed that this is still an issue and would be happy to open a pull
request with the proposed docs patch above if there's interest in it.
--
Ticket URL: <https://code.djangoproject.com/ticket/29800#comment:13>
Django
------------------------------------+------------------------------------
Reporter: Alexander Charykov | Owner: (none)
Component: HTTP handling | Version: 2.1
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Mariusz Felisiak):
* cc: Natalia Bidart (added)
--
Ticket URL: <https://code.djangoproject.com/ticket/29800#comment:14>
Django
------------------------------------+------------------------------------
Reporter: Alexander Charykov | Owner: (none)
Component: HTTP handling | Version: 2.1
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by thenomadlad):
Hi folks, I encountered a similar bug and since this thread was the only
place I could see information on this bug I thought I'll share what I
found after working around this issue
TL;DR I think this is because of the way gunicorn uses the content-length
header
[here](https://github.com/benoitc/gunicorn/blob/ab9c8301cb9ae573ba597154ddeea16f0326fc15/gunicorn/http/body.py#L128-L129)
My service is a flask+gunicorn service in a kubernetes cluster. we have
distributed tracing enabled at the flask layer and when I include a
content-length header, I saw traces start and end at the load-balancer
layer with a 400 status code after a 1s timeout, but no traces start for
the flask application. Between the load-balancer and my flask app, I had a
few places to look - the k8s-networking, docker-runtime, python 3.10 and
gunicorn. Naturally my biggest suspect was gunicorn
Gunicorn chooses different "readers" based on whether a `content-length`
header is set, the `transfer-encoding` header has value `chunked`, or
neither:
https://github.com/benoitc/gunicorn/blob/ab9c8301cb9ae573ba597154ddeea16f0326fc15/gunicorn/http/message.py#L125-L154
if content-length is set, it uses a `LengthReader` is used, which loops
infinitely until a certain length is reached:
https://github.com/benoitc/gunicorn/blob/ab9c8301cb9ae573ba597154ddeea16f0326fc15/gunicorn/http/body.py#L126-L130
This is a feature of gunicorn, I think it relies on timing out when the
reader doesn't have enough bytes
> I cannot reproduce when Django is behind a reverse proxy such as NGINX
with proxy_request_buffering on which is the default.
I believe this might be happening because proxy_request_buffering adds a
correct content-length header after buffering the request on the nginx
side? not sure I have the tools to verify this tbh so I would appreciate
it if someone can confirm this
--
Ticket URL: <https://code.djangoproject.com/ticket/29800#comment:15>