[Django] #29343: An exception AttributeError during HEAD request on media object
Django
-------------------------------------------+------------------------
Reporter: Yehor Smoliakov | Owner: nobody
Type: Bug | Status: new
Component: HTTP handling | Version: 2.0
Severity: Normal | Keywords:
Triage Stage: Unreviewed | Has patch: 0
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
UI/UX: 0 |
-------------------------------------------+------------------------
In developer mode I ran this command in the console:
{{{
curl --head
http://127.0.0.1:8000/media/blog/author/UqdQXbTuSjW_UbGCNMhFpA.jpg
}}}
And got this exception:
{{{
[20/Apr/2018 10:36:43] "HEAD /media/blog/author/UqdQXbTuSjW_UbGCNMhFpA.jpg
HTTP/1.1" 500 59
----------------------------------------
Exception happened during processing of request from ('127.0.0.1', 49750)
Traceback (most recent call last):
File "/usr/lib/python3.6/wsgiref/handlers.py", line 138, in run
self.finish_response()
File "/usr/lib/python3.6/wsgiref/handlers.py", line 180, in
finish_response
self.write(data)
File "/usr/lib/python3.6/wsgiref/handlers.py", line 279, in write
self._write(data)
File "/usr/lib/python3.6/wsgiref/handlers.py", line 453, in _write
result = self.stdout.write(data)
File "/usr/lib/python3.6/socketserver.py", line 775, in write
self._sock.sendall(b)
ConnectionResetError: [Errno 104] Connection reset by peer
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.6/wsgiref/handlers.py", line 141, in run
self.handle_error()
File "/home/yehor/.local/share/virtualenvs/webapp-Dlno2ZDu/lib/python3.6
/site-packages/django/core/servers/basehttp.py", line 86, in handle_error
super().handle_error()
File "/usr/lib/python3.6/wsgiref/handlers.py", line 368, in handle_error
self.finish_response()
File "/usr/lib/python3.6/wsgiref/handlers.py", line 180, in
finish_response
self.write(data)
File "/usr/lib/python3.6/wsgiref/handlers.py", line 274, in write
self.send_headers()
File "/usr/lib/python3.6/wsgiref/handlers.py", line 331, in send_headers
if not self.origin_server or self.client_is_modern():
File "/usr/lib/python3.6/wsgiref/handlers.py", line 344, in
client_is_modern
return self.environ['SERVER_PROTOCOL'].upper() != 'HTTP/0.9'
TypeError: 'NoneType' object is not subscriptable
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.6/socketserver.py", line 639, in
process_request_thread
self.finish_request(request, client_address)
File "/usr/lib/python3.6/socketserver.py", line 361, in finish_request
self.RequestHandlerClass(request, client_address, self)
File "/usr/lib/python3.6/socketserver.py", line 696, in __init__
self.handle()
File "/home/yehor/.local/share/virtualenvs/webapp-Dlno2ZDu/lib/python3.6
/site-packages/django/core/servers/basehttp.py", line 154, in handle
handler.run(self.server.get_app())
File "/usr/lib/python3.6/wsgiref/handlers.py", line 144, in run
self.close()
File "/usr/lib/python3.6/wsgiref/simple_server.py", line 35, in close
self.status.split(' ',1)[0], self.bytes_sent
AttributeError: 'NoneType' object has no attribute 'split'
----------------------------------------
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/29343>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
---------------------------------+--------------------------------------
Reporter: Yehor Smoliakov | Owner: nobody
Component: HTTP handling | Version: 2.0
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Tim Graham):
* status: new => closed
* resolution: => needsinfo
Comment:
I don't think this is a bug in Django -- the traceback comes from Python's
wsgiref. Feel free to reopen if you can explain why Django is at fault.
--
Ticket URL: <https://code.djangoproject.com/ticket/29343#comment:1>
Django
---------------------------------+--------------------------------------
Type: Bug | Status: new
Component: HTTP handling | Version: 2.0
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by mandm):
* status: closed => new
* resolution: needsinfo =>
Comment:
Similar problem with HEAD requests after upgrading to Django `1.10.8`
(`1.8.x` & `1.9.x` work)
{{{
curl -I http://localhost:8000/
}}}
gives this error
{{{
Traceback (most recent call last):
File "/usr/lib/python2.7/wsgiref/handlers.py", line 86, in run
self.finish_response()
File "/usr/lib/python2.7/wsgiref/handlers.py", line 128, in
finish_response
self.write(data)
File "/usr/lib/python2.7/wsgiref/handlers.py", line 217, in write
self._write(data)
File "/usr/lib/python2.7/socket.py", line 328, in write
self.flush()
File "/usr/lib/python2.7/socket.py", line 307, in flush
self._sock.sendall(view[write_offset:write_offset+buffer_size])
error: [Errno 104] Connection reset by peer
ERROR: 2018-06-16 11:53:28,639 - "HEAD / HTTP/1.1" 500 59
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/29343#comment:2>
Django
---------------------------------+--------------------------------------
Type: Bug | Status: new
Component: HTTP handling | Version: 2.0
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Claude Paroz):
This happens when streaming a `FileResponse` through a
`wsgiref.FileWrapper` instance, when the file is bigger than the streaming
block size. `curl` is closing the connection when it receives the first
chunk, and the error then happens when `FileWrapper` is writing the second
chunk to the socket.
I cannot yet state about a possible culprit in that scenario.
--
Ticket URL: <https://code.djangoproject.com/ticket/29343#comment:3>
Django
---------------------------------+------------------------------------
Type: Bug | Status: new
Component: HTTP handling | Version: 2.0
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Tim Graham):
* stage: Unreviewed => Accepted
Comment:
Tentatively accept for further investigation.
--
Ticket URL: <https://code.djangoproject.com/ticket/29343#comment:4>
Django
---------------------------------+------------------------------------
Type: Bug | Status: new
Component: HTTP handling | Version: 2.0
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Jannik Schürg):
I can not reproduce this. I tried Python 3.6.4 and 2.7.6 with Django
1.10.8 and 2.0 on Mac OS. The setup was a new project with media serving
and a 50MB jpg file.
Could you provide an example project or more information in order to
reproduce the error?
I got
{{{
[2018-11-07 05:07:39,626] - Broken pipe from ('127.0.0.1', 63603)
}}}
but no exception.
--
Ticket URL: <https://code.djangoproject.com/ticket/29343#comment:5>
Django
---------------------------------+------------------------------------
Type: Bug | Status: new
Component: HTTP handling | Version: 2.0
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Jannik Schürg):
* cc: Jannik Schürg (added)
--
Ticket URL: <https://code.djangoproject.com/ticket/29343#comment:6>
Django
---------------------------------+------------------------------------
Type: Bug | Status: new
Component: HTTP handling | Version: 2.0
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Jannik Schürg):
I could reproduce it now a few times but not consistently, it is a timing
issue maybe. Or my version of curl is more hesitant with TCP RST (leading
to ConnectionResetError).
Here is what happens, I think:
1. Curl closes the connection by resetting it instead of closing it (RST
vs. FIN).
2. An exception is thrown in wsgiref/socket.
3. `wsgiref.handlers.BaseHandler.close()` is called, which reset the state
of the handler (in particular `self.environ` is set to `None`).
4. The error handler `handle_error()` is called. Here Django has
overwritten the method and checks if `is_broken_pipe_error()` in which
case nothing would happen.
5. But curl did reset instead of close, so we have not "broken pipe," but
"connection reset". The default handler is called, which tries to use
`self.environ`, exception.
A proper fix probably would be to modify
`core.servers.basehttp.ServerHandler` by adding the method
{{{#!python
def finish_response(self):
try:
if self.environ['REQUEST_METHOD'] == 'HEAD':
self.finish_content()
else:
if not self.result_is_file() or not self.sendfile():
for data in self.result:
self.write(data)
self.finish_content()
finally:
self.close()
}}}
With this the server does no longer try to send the body for a HEAD
request (as it should be).
I am sure there are cases where this does produce an incorrect HEAD
response according to the HTTP standard, but at least no body is sent
anymore.
Alternatively/additionally one might want to change the check with
`is_broken_pipe()`. For example one could also check for connection reset,
or if `self.environ` is not `None`, or if the connection was closed
(overwrite close() method).
--
Ticket URL: <https://code.djangoproject.com/ticket/29343#comment:7>
Django
Reporter: Yehor Smoliakov | Owner: Jannik Schürg
Type: Bug | Status: assigned
Component: HTTP handling | Version: 2.0
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Jacob Walls):
* owner: nobody => Jannik Schürg
* status: new => assigned
* has_patch: 0 => 1
Comment:
[https://github.com/django/django/pull/10676 PR]
--
Ticket URL: <https://code.djangoproject.com/ticket/29343#comment:8>
Django
---------------------------------+-----------------------------------------
Reporter: Yehor Smoliakov | Owner: Jannik Schürg
Type: Bug | Status: assigned
Keywords: | Triage Stage: Accepted
Needs tests: 1 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Claude Paroz):
* needs_tests: 0 => 1
--
Ticket URL: <https://code.djangoproject.com/ticket/29343#comment:9>
Django
---------------------------------+-----------------------------------------
Reporter: Yehor Smoliakov | Owner: Jannik Schürg
Type: Bug | Status: assigned
Keywords: | Triage Stage: Accepted
Needs tests: 1 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Dave Johansen):
* cc: Dave Johansen (added)
--
Ticket URL: <https://code.djangoproject.com/ticket/29343#comment:10>
Django
---------------------------------+-----------------------------------------
Reporter: Yehor Smoliakov | Owner: Jannik Schürg
Type: Bug | Status: assigned
Keywords: | Triage Stage: Accepted
Needs tests: 1 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Dave Johansen):
This might already be known, but the issue appears to be basically all
`HEAD` requests in general and not just those related to serving static
files. To demonstrate that, I added a simpler reproducer in #33115 and I
tested the patch from the PR and it resolved the issue for me.
--
Ticket URL: <https://code.djangoproject.com/ticket/29343#comment:11>
Django
---------------------------------+-----------------------------------------
Reporter: Yehor Smoliakov | Owner: Jannik Schürg
Component: HTTP handling | Version: 2.0
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 1 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Mariusz Felisiak):
* status: assigned => closed
* resolution: => duplicate
Comment:
Duplicate of #28054.
--
Ticket URL: <https://code.djangoproject.com/ticket/29343#comment:12>