[Django] #23815: CsrfViewMiddleware UnicodeDecodeError
Django
-------------------------+-------------------------------------------------
Reporter: | Owner: nobody
codeitloadit | Status: new
Type: | Version: 1.6
Uncategorized | Keywords: middleware CsrfViewMiddleware
Component: HTTP | UnicodeDecodeError
handling | Has patch: 0
Severity: Normal | UI/UX: 0
Triage Stage: |
Unreviewed |
Easy pickings: 0 |
-------------------------+-------------------------------------------------
The csrf middleware is raising an exception when the HTTP_REFERER contains
non-ascii characters. Since this code is attempting to validate the
referer, I would expect these request to just be rejected and return 403.
Here is the code in `django/middleware/csrf.py`:
{{{
referer = request.META.get('HTTP_REFERER')
if referer is None:
return self._reject(request, REASON_NO_REFERER)
# Note that request.get_host() includes the port.
good_referer = 'https://%s/' % request.get_host()
if not same_origin(referer, good_referer):
reason = REASON_BAD_REFERER % (referer, good_referer)
return self._reject(request, reason)
}}}
This issue is very similar to
[https://code.djangoproject.com/ticket/20356] which was patched by
[https://github.com/django/django/commit/8fd44b2551b9cca765b216a31306f9c6935f1492]
which just encodes the referer like so:
{{{
referer = force_text(request.META.get('HTTP_REFERER', ''),
errors='replace')
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/23815>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
-------------------------------------+-------------------------------------
Reporter: codeitloadit | Owner: nobody
Type: Bug | Status: new
Component: HTTP handling | Version: master
Severity: Normal | Resolution:
Keywords: middleware | Triage Stage: Accepted
CsrfViewMiddleware | Needs documentation: 0
UnicodeDecodeError | Patch needs improvement: 0
Has patch: 0 | UI/UX: 0
Needs tests: 0 |
Easy pickings: 0 |
-------------------------------------+-------------------------------------
Changes (by charettes):
* needs_better_patch: => 0
* needs_tests: => 0
* version: 1.6 => master
* needs_docs: => 0
* type: Uncategorized => Bug
* stage: Unreviewed => Accepted
--
Ticket URL: <https://code.djangoproject.com/ticket/23815#comment:1>
Django
-------------------------------------+-------------------------------------
Reporter: codeitloadit | Owner: nobody
Component: HTTP handling | Version: master
Keywords: middleware | Triage Stage: Accepted
CsrfViewMiddleware | Needs documentation: 0
UnicodeDecodeError | Patch needs improvement: 0
Has patch: 0 | UI/UX: 0
Needs tests: 0 |
Easy pickings: 0 |
-------------------------------------+-------------------------------------
* status: new => closed
* resolution: => needsinfo
Comment:
Cannot repeat that error, I've tested it with several different unicode
chars in REFERER and it works correctly.
--
Ticket URL: <https://code.djangoproject.com/ticket/23815#comment:2>
Django
-------------------------------------+-------------------------------------
Reporter: codeitloadit | Owner: nobody
Component: HTTP handling | Version: master
Severity: Normal | Resolution:
CsrfViewMiddleware |
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Changes (by living180):
* status: closed => new
* resolution: needsinfo =>
Comment:
I've seen this in production: Django 1.6.5, Python 2.7, and here are some
examples of `REFERER` causing the issue:
`'\xd8B\xf6I\xdf'`
`'|\xcaH'`
Let me know if I can be of any help resolving this issue.
--
Ticket URL: <https://code.djangoproject.com/ticket/23815#comment:3>
Django
-------------------------------------+-------------------------------------
Reporter: codeitloadit | Owner: nobody
Type: Bug | Status: new
Component: HTTP handling | Version: master
Severity: Normal | Resolution:
Keywords: middleware | Triage Stage: Accepted
CsrfViewMiddleware |
UnicodeDecodeError |
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by timgraham):
Can you test with Django 1.7 and/or master? 1.6 is only receiving security
fixes so if this issue has been fixed since then we can close the ticket.
--
Ticket URL: <https://code.djangoproject.com/ticket/23815#comment:4>
Django
-------------------------------------+-------------------------------------
Reporter: codeitloadit | Owner: nobody
Type: Bug | Status: new
Component: HTTP handling | Version: master
Severity: Normal | Resolution:
Keywords: middleware | Triage Stage: Accepted
CsrfViewMiddleware |
UnicodeDecodeError |
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by living180):
I was able to reproduce with Django 1.7.2/Python 2.7.9. Reproducing
requires accessing Django using HTTPS, because the affected code is behind
`if request.is_secure():`. To achieve this, I used the `django-sslserver`
application (https://github.com/teddziuba/django-sslserver) in conjunction
with a simple project with the Django admin enabled. Using the `requests`
module to supply a bad `REFERER` header when POST-ing to the admin login
page:
{{{#!python
import requests
requests.post('https://localhost:8000/admin/login/',
headers={'referer': '\xd8B\xf6I\xdf'},
verify=False).text
}}}
I get the `UnicodeDecodeError`.
--
Ticket URL: <https://code.djangoproject.com/ticket/23815#comment:5>
Django
-------------------------------------+-------------------------------------
Reporter: codeitloadit | Owner: nobody
Type: Bug | Status: new
Component: HTTP handling | Version: master
Severity: Normal | Resolution:
Keywords: middleware | Triage Stage: Accepted
CsrfViewMiddleware |
UnicodeDecodeError |
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by claudep):
Reproducible with:
{{{
#!diff
diff --git a/tests/csrf_tests/tests.py b/tests/csrf_tests/tests.py
index 9c6e2e5..f22cddb 100644
--- a/tests/csrf_tests/tests.py
+++ b/tests/csrf_tests/tests.py
@@ -300,6 +300,11 @@ class CsrfViewMiddlewareTest(TestCase):
req2 = CsrfViewMiddleware().process_view(req, post_form_view, (),
{})
self.assertNotEqual(None, req2)
self.assertEqual(403, req2.status_code)
+ # Non-ASCII
+ req.META['HTTP_REFERER'] = b'\xd8B\xf6I\xdf'
+ req2 = CsrfViewMiddleware().process_view(req, post_form_view, (),
{})
+ self.assertNotEqual(None, req2)
+ self.assertEqual(403, req2.status_code)
@override_settings(ALLOWED_HOSTS=['www.example.com'])
def test_https_good_referer(self):
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/23815#comment:6>
Django
-------------------------------------+-------------------------------------
Reporter: codeitloadit | Owner: nobody
Type: Bug | Status: new
Component: HTTP handling | Version: master
Severity: Normal | Resolution:
Keywords: middleware | Triage Stage: Accepted
CsrfViewMiddleware |
UnicodeDecodeError |
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* has_patch: 0 => 1
Comment:
https://github.com/django/django/pull/3843
--
Ticket URL: <https://code.djangoproject.com/ticket/23815#comment:7>
Django
-------------------------------------+-------------------------------------
Reporter: codeitloadit | Owner: nobody
Type: Bug | Status: new
Component: HTTP handling | Version: master
Severity: Normal | Resolution:
CsrfViewMiddleware | checkin
UnicodeDecodeError |
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* stage: Accepted => Ready for checkin
--
Ticket URL: <https://code.djangoproject.com/ticket/23815#comment:8>
Django
-------------------------------------+-------------------------------------
Reporter: codeitloadit | Owner: nobody
Component: HTTP handling | Version: master
Keywords: middleware | Triage Stage: Ready for
CsrfViewMiddleware | checkin
UnicodeDecodeError |
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* status: new => closed
* resolution: => fixed
Comment:
In [changeset:"27dd7e727153cbf12632a2161217340123687c44"]:
{{{
#!CommitTicketReference repository=""
revision="27dd7e727153cbf12632a2161217340123687c44"
Fixed #23815 -- Prevented UnicodeDecodeError in CSRF middleware
Thanks codeitloadit for the report, living180 for investigations
and Tim Graham for the review.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/23815#comment:9>
Django
-------------------------------------+-------------------------------------
Reporter: codeitloadit | Owner: nobody
Type: Bug | Status: closed
Component: HTTP handling | Version: master
Severity: Normal | Resolution: fixed
Keywords: middleware | Triage Stage: Ready for
CsrfViewMiddleware | checkin
UnicodeDecodeError |
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Claude Paroz <claude@…>):
In [changeset:"d8fb557a519a419a53d648ce1ef12dad8673151f"]:
{{{
#!CommitTicketReference repository=""
revision="d8fb557a519a419a53d648ce1ef12dad8673151f"
[1.7.x] Fixed #23815 -- Prevented UnicodeDecodeError in CSRF middleware
Thanks codeitloadit for the report, living180 for investigations
and Tim Graham for the review.
Backport of 27dd7e7271 from master.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/23815#comment:10>