Intermittent IOError exception raised and emailed to admins during file upload.

[Tai Lee]

There are many questions about this on django-users. The usual answer
is to ignore the errors, which are probably generated as a result of
the client prematurely terminating a connection, usually during an
upload.

Nobody seems to be able to reliably reproduce the error, and it seems
to happen with large and small uploads alike, but always
intermittently.

It has been suggested that a client dropping the connection should be
within what one can expect from a networked client, and thus Django
should not email the exception and traceback to the admins in this
case. From my preliminary investigation with this issue, I tend to
agree.

Is it feasible to handle this exception in Django, or are there other
issues that would prevent us from isolating this case from other
legitimate exceptions that may be raised at this point?

Here's a recent traceback:

Traceback (most recent call last):

File "/home/tailee/django/core/handlers/base.py", line 105, in
get_response
response = middleware_method(request, callback, callback_args,
callback_kwargs)

File "/home/tailee/django/middleware/csrf.py", line 224, in
process_view
request_csrf_token = request.POST.get('csrfmiddlewaretoken', '')

File "/home/tailee/django/core/handlers/wsgi.py", line 209, in
_get_post
self._load_post_and_files()

File "/home/tailee/django/http/__init__.py", line 207, in
_load_post_and_files
self._post, self._files = self.parse_file_upload(self.META, self)

File "/home/tailee/django/http/__init__.py", line 169, in
parse_file_upload
return parser.parse()

File "/home/tailee/django/http/multipartparser.py", line 192, in
parse
for chunk in field_stream:

File "/home/tailee/django/http/multipartparser.py", line 314, in next
output = self._producer.next()

File "/home/tailee/django/http/multipartparser.py", line 468, in next
for bytes in stream:

File "/home/tailee/django/http/multipartparser.py", line 314, in next
output = self._producer.next()

File "/home/tailee/django/http/multipartparser.py", line 375, in next
data = self.flo.read(self.chunk_size)

File "/home/tailee/django/http/multipartparser.py", line 405, in read
return self._file.read(num_bytes)

File "/home/tailee/django/http/__init__.py", line 231, in read
return self._stream.read(*args, **kwargs)

IOError: request data read error

[Graham Dumpleton]

The problem is you can't just catch IOError, because the presence of WSGI middleware may mean that IOError could be generated for other reasons when reading wsgi.input. To base detection on the description in the IOError, ie., 'request data read error', would mean having to know what that description could be on all WSGI hosting mechanisms, and then just hope that there aren't other reasons why it could arise from WSGI middleware.

So, no simple solution. This unfortunately is one of the short comings of the WSGI specification. That is, there is no standardised exception type to represent actual real problems with reading original wsgi.input. Part of the reason is that there is no standard module associated with the WSGI specification to hold the exception type, so nothing to compare against. The only solution would be for WSGI specification to be enhanced to require a unique exception type to be used for wsgi.input exceptions such as aborted connection, with the type of that exception being passed through in the WSGI environ dictionary so it could be compare against. For example something like:

  try:

    data = environ['wsgi.input'].read()

  except wsgi['wsgi.input_exception], e:

    # ignore it

In summary, no real practical solution at this time for this.

Graham

[Tai Lee]

Thanks for your detailed explanation, Graham. I'll try to approach
this from another angle and see if I can determine what is actually
causing the connections to be dropped, as I'm seeing this error a few
times a day on a low traffic site using Apache at the front for static
media, with proxy pass to Django under Apache/mod_wsgi.

Cheers.
Tai.


On Feb 4, 6:17 pm, Graham Dumpleton <graham.dumple...@gmail.com>
wrote:

[-RAX-]

Have you considered the possibility that it would be a client/browser
side error?

Both windows7 and chrome have a known open bug with large file
uploads. Many of my customers have reported this bug, all of them
using windows7 and chrome.

Has anyone seen this happening from a Mac or from Linux?

[Tai Lee]

It seems to happen with very small file uploads as well. I've seen it reported with 30KB uploads, while at the same time 60MB uploads work. If I can't find a problem with the server causing disconnects, and it is actually on the client side, I'll just have to continue to ignore the exception reports Django emails as a result :)

Cheers.
Tai.

[Jacob Kaplan-Moss]

On Fri, Feb 4, 2011 at 2:45 AM, Tai Lee <real....@mrmachine.net> wrote:
> It seems to happen with very small file uploads as well. I've seen it reported with 30KB uploads, while at the same time 60MB uploads work. If I can't find a problem with the server causing disconnects, and it is actually on the client side, I'll just have to continue to ignore the exception reports Django emails as a result :)

Based on my experience with this error (which, sadly, is extensive)
it's almost certainly client-side. When there are server-side
disconnects it's not intermittent: every server-side disconnect issue
I've seen has been reproducible and thus fixable.

But there's a whole bunch of client-side issues that could be causing
this, and you probably can't do anything about it. Some examples of
where the problem can be:

* Browser: you can get this when a user just clicks the "stop" button
or navigates away. Some browsers close the connection cleanly; others
don't.
* Browser: some versions of IE, especially on Vista, randomly
terminate uploads. Most of the time it's seemed to be related to the
windows firewall thing.
* Browser: various development versions of Firefox, Chrome, and Safari
have all had upload/disconnect issues. To my knowledge none of these
issues have made it into shipping versions, but many users of these
browsers use the cutting edge.
* Dodgy 'net connections, especially public ones. For example, I can
pretty consistently trigger an IOError by uploading a file from Kansas
City International Airport.
* Badly-configured intermediary caches, like those run by ISPs. AOL
dialup users (yes, there are still some of these!) seem to throw lots
of IOErrors. (I traced this town by noting that certain /24 and /32 IP
spaces seemed more prevalent in the error tracebacks and tracked down
those block owners.)

So yeah, all of those are out of your control, which is why I (and
many others) usually just say to ignore the error.

However, I think it's a bad idea to automatically suppress these
errors (which you could do with a piece of middleware or a custom log
handler in 1.3). That's because this error could *also* be caused by
something misconfigured on your end (a messed up HTTP load balancer,
for example). So if you just suppress the error you could end up
missing something that *is* fixable.

So we're back to "read and then ignore," sadly. I would *love* a
better way, but I can't think of one.

Jacob

[tK]

Well, if this is a problem with WSGI, why not enhance the
specification in the way Graham proposed? I'm sure that whoever is
responsible for writing the spec is open for suggestions like this one
(while WSGI attempts to be as lean as possible, I'd say that not being
able to reliably know when a connection to the server broke
client-side is a major shortcoming). And while that is an ugly, ugly
idea, this could be implemented in common wsgi implementations now.
Thomas

> --
> You received this message because you are subscribed to the Google Groups "Django developers" group.
> To post to this group, send email to django-d...@googlegroups.com.
> To unsubscribe from this group, send email to django-develop...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
>
>

[Tai Lee]

As a datapoint, I've had 9 of these errors from 5 different people (2
people retried 3 times each) and 30 successful uploads from 14
different people in the past 7 days on a low traffic site (300
visits).

That's a pretty low success rate. Seven of the errors had a user agent
indicating MSIE 8, one was MSIE 7, and one was Chrome on OS X. If
almost 25% of upload attempts are failing, and 33% of users are
experiencing difficulty, that seems to indicate that web browsers
(especially MSIE) are extremely unreliable at performing uploads.

I'd love to see some data on the reliability of uploads on other
higher traffic sites across different browsers and using different
server side technologies, to see if this really is typical.

Cheers.
Tai.


On Feb 5, 2:23 am, Jacob Kaplan-Moss <ja...@jacobian.org> wrote:

[Drew Volpe]

Did you ever track down what was happening ?

Was any more information discovered on this problem ?

I'm seeing a similar issue in a eb service written in Django which
receives data from mobile apps as a POST. Not infrequently, I'm
seeing the same "IOError: request data read error" when I try to read
request.raw_post_data. It happens more than spotty connections would
explain.

[Nick Phillips]

On Mon, 2011-02-14 at 11:10 -0800, Drew Volpe wrote:
> Did you ever track down what was happening ?
>
> Was any more information discovered on this problem ?
>
> I'm seeing a similar issue in a eb service written in Django which
> receives data from mobile apps as a POST. Not infrequently, I'm
> seeing the same "IOError: request data read error" when I try to read
> request.raw_post_data. It happens more than spotty connections would
> explain.

User double-clicks on submit button, second click killing the connection
created by the first press and starting another one?

Cheers,

Nick
--
Nick Phillips / +64 3 479 4195 / nick.p...@otago.ac.nz
# these statements are my own, not those of the University of Otago

[Drew Volpe]

On Feb 14, 3:15 pm, Nick Phillips <nick.phill...@otago.ac.nz> wrote:


> > I'm seeing a similar issue in a eb service written in Django which
> > receives data from mobile apps as a POST.  Not infrequently, I'm
> > seeing the same "IOError: request data read error" when I try to read
> > request.raw_post_data.  It happens more than spotty connections would
> > explain.
>
> User double-clicks on submit button, second click killing the connection
> created by the first press and starting another one?

In my case, the request is made in a background thread within a mobile
app (there's no form or webbrowser), so it's not that. And I'm pretty
sure the mobile app is not killing the connection as it's able to push
data into another, similar API without any problems.

If Tai or anyone else found more information or data on this, I'd be
interested.

[Graham Dumpleton]

Presuming Apace/mod_wsgi is being used, are you using https or just http? Are you using daemon mode of mod_wsgi? If yes to latter, how have you configured daemon mode? Also what Apache MPM are you using and what MPM configuration settings? Finally, what Apache and mod_wsgi versions are you using?

Graham

On Tuesday, February 15, 2011 6:50:12 AM UTC+8, Drew Volpe wrote:

[Drew Volpe]

https only
Apache/2.2.16 on Ubuntu 10.10 (32 bit)
Python 2.6.6
mod_wsgi 3.2 embedded mode
mpm worker:
<IfModule mpm_worker_module>
StartServers 2
MinSpareThreads 25
MaxSpareThreads 75
ThreadLimit 64
ThreadsPerChild 25
MaxClients 150
MaxRequestsPerChild 0
</IfModule>

Right now, there's only 1-3 clients each making requests a few times
an hour. Each request is a POST with typically only a few k of data.

The exception happens in the view as soon as request.raw_post_data is
accessed:

File "/usr/lib/pymodules/python2.6/django/core/handlers/wsgi.py",
line 205, in _get_raw_post_data
size=content_length)
File "/usr/lib/pymodules/python2.6/django/core/handlers/wsgi.py",
line 69, in safe_copyfileobj
buf = fsrc.read(min(length, size))


In my case, it's not a show-stopper as we now just catch the exception
and have the client retry (and the user never knows) but it sounds
similar to Tai's problem.


Dv

On Feb 14, 7:06 pm, Graham Dumpleton <graham.dumple...@gmail.com>
wrote:

[Graham Dumpleton]

On Wednesday, February 16, 2011 5:16:01 AM UTC+11, Drew Volpe wrote:

https only

Any chance you can do some tests, if you yourself can replicate it, of whether it fails when using straight http. Most times I have heard about this, it is somehow specifically related to use of https connection. There were some bugs in Apache SSL implementation but you seem to have reasonably new Apache and think problems were in slightly older versions of Apache.

Graham

 

On Feb 14, 7:06 pm, Graham Dumpleton <graham.d...@gmail.com>

wrote:
> Presuming Apace/mod_wsgi is being used, are you using https or just http? Are you using daemon mode of mod_wsgi? If yes to latter, how have you configured daemon mode? Also what Apache MPM are you using and what MPM configuration settings? Finally, what Apache and mod_wsgi versions are you using?
>
> Graham
>
> On Tuesday, February 15, 2011 6:50:12 AM UTC+8, Drew Volpe wrote:

[Graham Dumpleton]

BTW, you didn't say whether you were using mod_wsgi embedded mode or daemon mode and what latter configuration was if using it.

Graham

On Feb 14, 7:06 pm, Graham Dumpleton <graha...@gmail.com>

wrote:
> Presuming Apace/mod_wsgi is being used, are you using https or just http? Are you using daemon mode of mod_wsgi? If yes to latter, how have you configured daemon mode? Also what Apache MPM are you using and what MPM configuration settings? Finally, what Apache and mod_wsgi versions are you using?
>
> Graham
>
> On Tuesday, February 15, 2011 6:50:12 AM UTC+8, Drew Volpe wrote:

[Drew Volpe]

It oddly seems to have stopped happening when we moved to staging.
It's a different server though the apache config is exactly the same
(the settings I sent in my previous message: mod_wsgi 3.2, embedded
mode, mpm worker). We also rolled out a few fixes in our mobile app so
it's possible it was being caused by a bug in the client. If I have a
chance after launch, I'll go back and run the old version and see I
can reproduce.


Drew


On Feb 15, 6:15 pm, Graham Dumpleton <graham.dumple...@gmail.com>
wrote: