[Django] #23593: UnicodeDecodeError when handling uncaught exception with UTF-8 in URL
Django
-------------------------------+--------------------------------
Reporter: edevil | Owner: nobody
Type: Bug | Status: new
Component: HTTP handling | Version: master
Severity: Normal | Keywords: UnicodeDecodeError
Triage Stage: Unreviewed | Has patch: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------+--------------------------------
When DEBUG==False, there is UTF-8 in URL, and an uncaught exception
occurs:
{{{
Traceback (most recent call last):
File "/Users/andre/work/penv/discosite/lib/python2.7/site-
packages/django/core/handlers/wsgi.py", line 187, in __call__
response = self.get_response(request)
File "/Users/andre/work/penv/discosite/lib/python2.7/site-
packages/django/core/handlers/base.py", line 199, in get_response
response = self.handle_uncaught_exception(request, resolver,
sys.exc_info())
File "/Users/andre/work/penv/discosite/lib/python2.7/site-
packages/django/core/handlers/base.py", line 231, in
handle_uncaught_exception
'request': request
File
"/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/logging/__init__.py",
line 1175, in error
self._log(ERROR, msg, args, **kwargs)
File
"/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/logging/__init__.py",
line 1268, in _log
self.handle(record)
File
"/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/logging/__init__.py",
line 1278, in handle
self.callHandlers(record)
File
"/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/logging/__init__.py",
line 1318, in callHandlers
hdlr.handle(record)
File
"/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/logging/__init__.py",
line 749, in handle
self.emit(record)
File "/Users/andre/work/penv/discosite/lib/python2.7/site-
packages/django/utils/log.py", line 124, in emit
message = "%s\n\nRequest repr(): %s" % (self.format(record),
request_repr)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 30:
ordinal not in range(128)
}}}
Example request:
{{{
$ curl "http://localhost:8000/%5BSC%5D%20TIC%207%C2%BAC"
}}}
This did not happen < 1.7.
--
Ticket URL: <https://code.djangoproject.com/ticket/23593>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
Reporter: edevil | Owner: nobody
Type: Bug | Status: new
Component: HTTP handling | Version: master
Keywords: UnicodeDecodeError | Triage Stage: Unreviewed
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by timgraham):
* needs_better_patch: => 0
* needs_tests: => 0
* needs_docs: => 0
Comment:
I couldn't reproduce this using the code from the tutorial:
{{{
$ curl "http://localhost:8000/%5BSC%5D%20TIC%207%C2%BAC"
<h1>Not Found</h1><p>The requested URL /[SC] TIC 7ºC was not found on this
server.</p>
}}}
Could you provide more details? Ideally, a test case for Django's test
suite if possible.
--
Ticket URL: <https://code.djangoproject.com/ticket/23593#comment:1>
Django
Reporter: edevil | Owner: nobody
Type: Bug | Status: new
Component: HTTP handling | Version: master
Keywords: UnicodeDecodeError | Triage Stage: Unreviewed
Easy pickings: 0 | UI/UX: 0
Comment (by edevil):
As I mentioned, this happens when DEBUG=False, an exception is thrown
while servicing the request, and there is UTF-8 in the URL. It seems you
only reproduced the "UTF-8 in the URL part"...
You can put the following url pattern:
{{{
urlpatterns = patterns('',
url(r'^(*)$', 'utils.cross_domain.cross_domain_xml'),
)
}}}
and this will cause an exception. Then self.format(record) will end up:
(notice it's unicode)
{{{
u'Internal Server Error: /[SC] TIC 7\xbaC\nTraceback (most recent call
last):\n File "/Users/andre/work/penv/discosite/lib/python2.7/site-
packages/django/core/handlers/base.py", line 87, in get_response\n
response = middleware_method(request)\n File
"/Users/andre/work/penv/discosite/lib/python2.7/site-
packages/django/middleware/common.py", line 72, in process_request\n if
(not urlresolvers.is_valid_path(request.path_info, urlconf) and\n File
"/Users/andre/work/penv/discosite/lib/python2.7/site-
packages/django/core/urlresolvers.py", line 614, in is_valid_path\n
resolve(path, urlconf)\n File
"/Users/andre/work/penv/discosite/lib/python2.7/site-
packages/django/core/urlresolvers.py", line 489, in resolve\n return
get_resolver(urlconf).resolve(path)\n File
"/Users/andre/work/penv/discosite/lib/python2.7/site-
packages/django/core/urlresolvers.py", line 340, in resolve\n sub_match
= pattern.resolve(new_path)\n File
"/Users/andre/work/penv/discosite/lib/python2.7/site-
packages/django/core/urlresolvers.py", line 211, in resolve\n match =
self.regex.search(path)\n File
"/Users/andre/work/penv/discosite/lib/python2.7/site-
packages/django/core/urlresolvers.py", line 179, in regex\n (regex,
six.text_type(e)))\nImproperlyConfigured: "^(*)$" is not a valid regular
expression: nothing to repeat'
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/23593#comment:2>
Django
Reporter: edevil | Owner: nobody
Type: Bug | Status: new
Severity: Release blocker | Resolution:
Keywords: UnicodeDecodeError | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 1 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by timgraham):
* has_patch: 0 => 1
* version: master => 1.7
* severity: Normal => Release blocker
* needs_tests: 0 => 1
* stage: Unreviewed => Accepted
Comment:
Adding `from __future__ import unicode_literals` to `django/utils/log.py`
seems to fix the issue. We also need a regression test and a mention in
the 1.7.1 release notes.
--
Ticket URL: <https://code.djangoproject.com/ticket/23593#comment:3>
Django
Reporter: edevil | Owner: timgraham
Type: Bug | Status: assigned
Component: HTTP handling | Version: 1.7
Severity: Release blocker | Resolution:
Keywords: UnicodeDecodeError | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by timgraham):
* owner: nobody => timgraham
* status: new => assigned
--
Ticket URL: <https://code.djangoproject.com/ticket/23593#comment:4>
Django
Reporter: edevil | Owner: timgraham
Type: Bug | Status: assigned
Component: HTTP handling | Version: 1.7
Severity: Release blocker | Resolution:
Keywords: UnicodeDecodeError | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by timgraham):
* needs_tests: 1 => 0
Comment:
Bisected to ad6fcdb8d24d84a130f847f45b51d7e892ccca08.
[https://github.com/django/django/pull/3318 PR]
--
Ticket URL: <https://code.djangoproject.com/ticket/23593#comment:5>
Django
Reporter: edevil | Owner: timgraham
Component: HTTP handling | Version: 1.7
Keywords: UnicodeDecodeError | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Tim Graham <timograham@…>):
* status: assigned => closed
* resolution: => fixed
Comment:
In [changeset:"9dff5ce7c70780f4ee91567d98dba227093b49e2"]:
{{{
#!CommitTicketReference repository=""
revision="9dff5ce7c70780f4ee91567d98dba227093b49e2"
Fixed #23593 -- Fixed crash in AdminEmailHandler with non-ASCII characters
in request.
Thanks edevil for the report and Simon Charette for review.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/23593#comment:6>
Django
Reporter: edevil | Owner: timgraham
Type: Bug | Status: closed
Component: HTTP handling | Version: 1.7
Severity: Release blocker | Resolution: fixed
Keywords: UnicodeDecodeError | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Tim Graham <timograham@…>):
In [changeset:"96f022203a552fef19403c01b12fa6d30da560a0"]:
{{{
#!CommitTicketReference repository=""
revision="96f022203a552fef19403c01b12fa6d30da560a0"
[1.7.x] Fixed #23593 -- Fixed crash in AdminEmailHandler with non-ASCII
characters in request.
Thanks edevil for the report and Simon Charette for review.
Backport of 9dff5ce7c7 from master
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/23593#comment:7>