[Django] #20368: MemoryError can block rendering of traceback page
Django
-------------------------------+--------------------
Reporter: ironfroggy | Owner: nobody
Type: Uncategorized | Status: new
Component: Uncategorized | Version: 1.3
Severity: Normal | Keywords:
Triage Stage: Unreviewed | Has patch: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------+--------------------
When preparing frame variables for the traceback page on a low memory VM,
it is possible to hit a MemoryError during the html escaping. For the
purposes of a traceback page, there should be some fallback for this.
In django/views/debug.py:117 the call to force_escape() could be wrapped
in a try/except and catch the MemoryError, representing the variable in
the frame as something like "could not display, representation too large".
As it stands, this just hides the original error by never producing the
traceback page.
{{{
Traceback (most recent call last):
File "/home/vagrant/devel/cms_dev/local/lib/python2.7/site-
packages/django/core/servers/basehttp.py", line 283, in run
self.result = application(self.environ, self.start_response)
File "/home/vagrant/devel/cms_dev/local/lib/python2.7/site-
packages/django/contrib/staticfiles/handlers.py", line 68, in __call__
return self.application(environ, start_response)
File "/home/vagrant/devel/cms_dev/local/lib/python2.7/site-
packages/django/core/handlers/wsgi.py", line 273, in __call__
response = self.get_response(request)
File "/home/vagrant/devel/cms_dev/local/lib/python2.7/site-
packages/django/core/handlers/base.py", line 182, in get_response
response = self.handle_uncaught_exception(request, resolver,
sys.exc_info())
File "/home/vagrant/devel/cms_dev/local/lib/python2.7/site-
packages/django/core/handlers/base.py", line 203, in
handle_uncaught_exception
return debug.technical_500_response(request, *exc_info)
File "/home/vagrant/devel/cms_dev/local/lib/python2.7/site-
packages/django/views/debug.py", line 59, in technical_500_response
html = reporter.get_traceback_html()
File "/home/vagrant/devel/cms_dev/local/lib/python2.7/site-
packages/django/views/debug.py", line 117, in get_traceback_html
frame['vars'] = [(k, force_escape(pprint(v))) for k, v in
frame['vars']]
File "/home/vagrant/devel/cms_dev/local/lib/python2.7/site-
packages/django/template/defaultfilters.py", line 37, in _dec
return func(*args, **kwargs)
File "/home/vagrant/devel/cms_dev/local/lib/python2.7/site-
packages/django/template/defaultfilters.py", line 398, in force_escape
return mark_safe(escape(value))
File "/home/vagrant/devel/cms_dev/local/lib/python2.7/site-
packages/django/utils/functional.py", line 259, in wrapper
return func(*args, **kwargs)
File "/home/vagrant/devel/cms_dev/local/lib/python2.7/site-
packages/django/utils/html.py", line 34, in escape
return mark_safe(force_unicode(html).replace('&',
'&').replace('<', '<').replace('>', '>').replace('"',
'"').replace("'", '''))
MemoryError
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/20368>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
Reporter: ironfroggy | Owner: nobody
Type: Uncategorized | Status: new
Component: Uncategorized | Version: 1.3
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by jgeskens):
* needs_better_patch: => 0
* needs_docs: => 0
* needs_tests: => 0
* stage: Unreviewed => Accepted
Comment:
I also get this a lot. Especially when dealing with big variables (for
example megabytes of binary data) in your stacktrace...
--
Ticket URL: <https://code.djangoproject.com/ticket/20368#comment:1>
Django
Reporter: ironfroggy | Owner: nobody
Type: Uncategorized | Status: new
Component: Uncategorized | Version: 1.3
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
-------------------------------+------------------------------------
Changes (by jgeskens):
* easy: 0 => 1
--
Ticket URL: <https://code.djangoproject.com/ticket/20368#comment:2>
Django
Reporter: ironfroggy | Owner: nobody
Type: Uncategorized | Status: new
Component: Uncategorized | Version: 1.3
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 1 | UI/UX: 0
-------------------------------+------------------------------------
* has_patch: 0 => 1
--
Ticket URL: <https://code.djangoproject.com/ticket/20368#comment:3>
Django
Reporter: ironfroggy | Owner: nobody
Type: Cleanup/optimization | Status: new
Component: Template system | Version: 1.3
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by timo):
* component: Uncategorized => Template system
* type: Uncategorized => Cleanup/optimization
* easy: 1 => 0
--
Ticket URL: <https://code.djangoproject.com/ticket/20368#comment:4>
Django
Reporter: ironfroggy | Owner: nobody
Type: Cleanup/optimization | Status: new
Component: Template system | Version: 1.3
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by timo):
Similar issue in #20534 which I closed as a duplicate of this one. The
patch there looks to be less invasive although I haven't reviewed either
in detail.
--
Ticket URL: <https://code.djangoproject.com/ticket/20368#comment:5>
Django
Reporter: ironfroggy | Owner: nobody
Type: Cleanup/optimization | Status: new
Component: Template system | Version: 1.3
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by wdoekes):
* cc: walter+django@… (added)
--
Ticket URL: <https://code.djangoproject.com/ticket/20368#comment:6>
Django
Reporter: ironfroggy | Owner: nobody
Type: Cleanup/optimization | Status: new
Component: Template system | Version: 1.3
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by timgraham):
* needs_better_patch: 0 => 1
Comment:
A GitHub PR would make review of the patch easier. I spotted a `u''`
prefix which isn't valid on Python 3.2 and the exception format needs to
be `Exception as e` for Python 3 compatibility.
--
Ticket URL: <https://code.djangoproject.com/ticket/20368#comment:7>
Django
Reporter: ironfroggy | Owner: nobody
Type: Cleanup/optimization | Status: new
Component: Template system | Version: 1.3
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1
Comment (by wdoekes):
Here, my fix from #20534:
{{{
--- django/views/debug.py.orig 2013-05-31 13:02:36.807798172 +0200
+++ django/views/debug.py 2013-05-31 13:01:52.555274822 +0200
@@ -244,7 +244,17 @@ class ExceptionReporter(object):
frames = self.get_traceback_frames()
for i, frame in enumerate(frames):
if 'vars' in frame:
- frame['vars'] = [(k, force_escape(pprint(v))) for k, v in
frame['vars']]
+ vars = []
+ for k, v in frame['vars']:
+ v = pprint(v)
+ # The force_escape filter assume unicode, make sure
that works
+ if isinstance(v, str):
+ v = v.decode('utf-8', 'replace') # don't choke
on non-utf-8 input
+ # You may be looking at large blobs of data, trim it
+ if len(v) > 4096:
+ v = u'%s... <trimmed %d bytes string>' %
(v[0:4096], len(v))
+ vars.append((k, force_escape(v)))
+ frame['vars'] = vars
frames[i] = frame
unicode_hint = ''
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/20368#comment:8>
Django
Reporter: ironfroggy | Owner: nobody
Type: Cleanup/optimization | Status: new
Component: Template system | Version: 1.3
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1
Comment (by timgraham):
Looks reasonable, we'd just need some tests.
--
Ticket URL: <https://code.djangoproject.com/ticket/20368#comment:9>
Django
Reporter: ironfroggy | Owner: nobody
Type: Cleanup/optimization | Status: new
Component: Template system | Version: 1.3
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1
Comment (by wdoekes):
Now filed as PR: https://github.com/django/django/pull/3054
(replacing the `u''` and `isinstance(..., str)`)
--
Ticket URL: <https://code.djangoproject.com/ticket/20368#comment:10>
Django
Reporter: ironfroggy | Owner: nobody
Type: Cleanup/optimization | Status: closed
Component: Template system | Version: 1.3
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1
Changes (by Tim Graham <timograham@…>):
* status: new => closed
* resolution: => fixed
Comment:
In [changeset:"e0e28bfe715b3f7d4e6cc7ab7bf4000b22c0cf79"]:
{{{
#!CommitTicketReference repository=""
revision="e0e28bfe715b3f7d4e6cc7ab7bf4000b22c0cf79"
Fixed #20368 -- Made TECHNICAL_500 more robust against bad input.
This limits large variables and avoids non-utf-8 in the TECHNICAL_500
output.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/20368#comment:11>
Django
Reporter: ironfroggy | Owner: nobody
Type: Cleanup/optimization | Status: closed
Component: Template system | Version: 1.3
Severity: Normal | Resolution: fixed
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1
Comment (by GitHub <noreply@…>):
In [changeset:"21f13ff5b3d5a42d62f38398c010efcdce30dad7" 21f13ff5]:
{{{
#!CommitTicketReference repository=""
revision="21f13ff5b3d5a42d62f38398c010efcdce30dad7"
Refs #23919 -- Removed an used block in
ExceptionReporter.get_traceback_data().
The test from refs #20368 only runs this block on Python 2.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/20368#comment:12>