IOError: Client read error (Timeout?)
Trey
time ago. I run a small to medium sized web application that takes
profile pictures. By far my largest customer service issue is people
not being able to upload their photos.
For the most part I have played it down as their connection sucking or
perhaps doing something stupid with the browser, but there are a
couple of things that I am running into that are causing an issue.
1. I can't replicate this, no matter what I do with my browser in the
middle of an upload.
2. Judging by the django code near the problem, this is working on
information that has already been received.
3. I get this a few times a day at least, different people every time.
If someone with some knowledge of the core or http areas of django
could speak up on this I would be very grateful.
Stack trace for reference....
Traceback (most recent call last):
File "/usr/lib/python2.5/site-packages/django/core/handlers/
base.py", line 77, in get_response
response = callback(request, *callback_args, **callback_kwargs)
File "/var/www/yttp/user/views.py", line 78, in profile_edit
if 'photo' in request.FILES:
File "/usr/lib/python2.5/site-packages/django/core/handlers/
modpython.py", line 86, in _get_files
self._load_post_and_files()
File "/usr/lib/python2.5/site-packages/django/core/handlers/
modpython.py", line 51, in _load_post_and_files
self._post, self._files =
http.parse_file_upload(self._req.headers_in, self.raw_post_data)
File "/usr/lib/python2.5/site-packages/django/core/handlers/
modpython.py", line 120, in _get_raw_post_data
self._raw_post_data = self._req.read()
IOError: Client read error (Timeout?)
Malcolm Tredinnick
> There are other people that have brought this up a little bit some
> time ago. I run a small to medium sized web application that takes
> profile pictures. By far my largest customer service issue is people
> not being able to upload their photos.
>
> For the most part I have played it down as their connection sucking or
> perhaps doing something stupid with the browser, but there are a
> couple of things that I am running into that are causing an issue.
I internally wince a bit every time this problem comes up. There's
something going on that makes it occur regularly for certain
combinations of browsers and servers (there's a report in Edgewall's own
Trac installation about a repeatable upload problem for one group of
people that is the same problem -- see [1]). However, it's not easily
reproducible for other people.
[1] http://trac.edgewall.org/ticket/5226
>
> 1. I can't replicate this, no matter what I do with my browser in the
> middle of an upload.
> 2. Judging by the django code near the problem, this is working on
> information that has already been received.
> 3. I get this a few times a day at least, different people every time.
>
> If someone with some knowledge of the core or http areas of django
> could speak up on this I would be very grateful.
The error itself appears to be coming from inside mod_python. The
self._req attribute is mod_python's request object and we're in the
process of reading data from there. Unfortunately, that doesn't help
very much with the problem diagnosis, but if you Google around for the
exact error message, you'll see it appearing in other places as well.
Just not commonly enough to help work out what's really going on. I
don't think there's a lot we can do on the Django side to mitigate this.
If it was me running a site that had this problem and I was really
dedicated to solving it, I'd try to put a rolling tcpdump on the
connection to try and capture the problem live. It will be a real bear
to try and interpret the results, but if you have accurate
(synchronised) clocks on all the machines in question and are capturing
the exception when it happens (with a timestamp), you should be able to
get close. Tcpdump has the ability to roll the logs after a certain size
and to create no more than a certain number of log files before reusing
the first ones (so it can use a ring of files, essentially). And
remember that it doesn't capture all the packets, only the first few
bytes -- enough to determine the protocol and see the start of the
message body. You can tweak this, of course, but the point is you won't
be capturing *all* of the traffic to your site, so storage requirements
won't be unmanageably huge (but they will be large). If you can set
aside a few gigabytes of disk space to capture logs and are able to
respond reasonably fast when the error occurs (so you can grab the log
that will contain the error before it gets overwritten next time
around), this is a viable technique.
Unfortunately, it may not tell you a lot (it's always a good idea to
know what you hope to get out of a major debugging thing like this). You
might see that either there is a slight hold-up in the client sending
data and maybe you'll be able to see the browser type in the HTTP
headers. Or you might see no break in the packet timings at all, which
says there was a hold up inside application space (Apache, I guess). I'm
not sure how either piece of information will help much, although
knowing where to look further will always be a benefit.
I guess I'd also try to ensure I was running the latest mod_python (and
Apache) in cases like this. The code is getting better all the time --
you just don't see any huge "I upgrade mod_python and the wheels
completely fell off my site" problems -- so taking advantage of all the
bug fixes you can can't hurt here, just in case it is an
application-layer problem.
Please realise that everything I've written here is along the lines of
how I would go about trying to gather more information. I have no
special insight into what is causing the problem, although from reading
a bunch of problem reports on the web, it sounds like it is repeatable
for certain client setups, so might not be just a sporadic connection
issue.
Regards,
Malcolm
Graham Dumpleton
up in various ways, albeit rarely. This may be another example to add
to the existing ones. The other known examples are:
http://issues.apache.org/jira/browse/MODPYTHON-212
http://issues.apache.org/jira/browse/MODPYTHON-234
I haven't sat down to try and work out how to fix the code in
mod_python, but being mindful that it was doing the wrong thing, I
completely wrote from scratch in mod_wsgi the input reading code and
didn't rely on the mod_python code in any way. Thus, as far as I know
these problems shouldn't occur with mod_wsgi.
Graham
On Oct 7, 2:06 am, Malcolm Tredinnick <malc...@pointy-stick.com>
wrote:
Trey
see if I can catch the error in the act. I have been doing something
similar with strace and some sort of segfault that is very hard to
catch.
Graham, I haven't researched mod_wsgi much, is it an alternative to
mod_python or does it just relate to the development server (as that's
the only place I've seen it mentioned)
trey
On Oct 6, 7:01 pm, Graham Dumpleton <Graham.Dumple...@gmail.com>
wrote:
Graham Dumpleton
> Interesting approach Malcom, I will read up on a few of the tools to
> see if I can catch the error in the act. I have been doing something
> similar with strace and some sort of segfault that is very hard to
> catch.
>
> Graham, I haven't researched mod_wsgi much, is it an alternative to
> mod_python or does it just relate to the development server (as that's
> the only place I've seen it mentioned)
For the purposes of Python web applications which support the WSGI
interface, such as Django, then mod_wsgi is a viable replacement for
mod_python. In some respects, mod_wsgi is also a viable replacement
for FASTCGI solutions in many cases as mod_wsgi supports both an
embedded mode (like mod_python) and a daemon mode (like FASTCGI).
The only feature I know of missing in mod_wsgi 1.X that some Django
people like to use from mod_python is the ability to define an Apache
authentication handler in mod_python so for example they can use
Django user authentication for authenticating users accessing a Trac
instance. This ability to define an Apache authentication handler, as
well as group authorisation has already though been implemented for
mod_wsgi 2.0 and is available from Subversion repository trunk for
mod_wsgi.
In respect to mod_wsgi relating to development server, the only thing
I can think of that you are talking about here is how I might have
previously indicated how mod_wsgi could easily be used in situations
where needing to support multiple developers working on the same
Django application but where each has its own workspace. Yes mod_wsgi
can help in this situation, but this is just an outcome from its
flexible configuration options in providing both embedded and daemon
modes. For production sites mod_wsgi is more than capable and it
shouldn't be seen as only useful for development environments.
Graham
Trey
to be a very good alternative solution to mod_python. One which we'll
probably try.
Is there a reason Django doesn't represent it in it's documentation? I
know everyone is busy and it could be that it just isn't added yet. I
would just like to know if mod_python is the official choice over
mod_wsgi for a reason.
Graham, if you know of any comparisons feature or performance wise let
me know. I am still searching around but I can see your an advocate so
you may know already.
Trey
PS: I was confused, I saw wsgi on the dev server, not mod_wsgi.
On Oct 8, 6:37 pm, Graham Dumpleton <Graham.Dumple...@gmail.com>
Malcolm Tredinnick
> I took the liberty of doing some research on mod_wsgi and it appears
> to be a very good alternative solution to mod_python. One which we'll
> probably try.
>
> Is there a reason Django doesn't represent it in it's documentation? I
> know everyone is busy and it could be that it just isn't added yet. I
> would just like to know if mod_python is the official choice over
> mod_wsgi for a reason.
Because mod_wsgi is basically brand new, whereas mod_python has been
around for years and years and was used since day 1 in Django. There are
lots of other solutions besides mod_python and mod_wsgi and we don't
document them, either. At some point, we have to pick a few options and
go with it. Quite possibly, in the future, we'll add documentation for
mod_wsgi -- chances go up once somebody submits a patch to Trac, for
example.
Django isn't 100% WSGI-compliant at the moment, but we're close and the
missing bits (mainly ensuring SCRIPT_NAME is always available) will be
filled in very soon (I think there are some patches in Trac awaiting
final review).
Regards,
Malcolm
Bjørn Stabell
> There are other people that have brought this up a little bit some
> time ago. I run a small to medium sized web application that takes
> profile pictures. By far my largest customer service issue is people
> not being able to upload their photos.
>
> For the most part I have played it down as their connection sucking or
> perhaps doing something stupid with the browser, but there are a
> couple of things that I am running into that are causing an issue.
>
> 1. I can't replicate this, no matter what I do with my browser in the
> middle of an upload.
> 2. Judging by the django code near the problem, this is working on
> information that has already been received.
> 3. I get this a few times a day at least, different people every time.
> File "/usr/lib/python2.5/site-packages/django/core/handlers/
> modpython.py", line 120, in _get_raw_post_data
> self._raw_post_data = self._req.read()
>
> IOError: Client read error (Timeout?)
We're seeing this too, on several Django sites. Annoyingly, as well,
the IOError exception itself is broken; if you look at the docs,
IOErrors (which are a form of EnvironmentErrors) should have a two- or
three-tuple .args, one of which would be the errno, but the ones
thrown by mod_python/Django seems to have only one item in the tuple;
the string you see above.
It's particularly annoying since we'd like to treat this as an info/
debug-level error, not an error-level error, and we could if we just
had access to the errno. (We're trying to silence non-errors so we
can do proper monitoring.)
Rgds,
Bjorn
Graham Dumpleton
directly isn't good, although for slightly other reasons.
https://issues.apache.org/jira/browse/MODPYTHON-92
That it isn't using IOError correctly in as much as no errno is
present adds to why it shouldn't do as it is. It isn't just a simple
matter of supplying errno though as in certain situations there
wouldn't actually be any.
As far as ignoring these errors, which occur when remote client closes
connection, it isn't necessarily a good idea for the lower level
layers to do so, as there are possible implications in doing so. For
example, it can hide problems in your own application and make it
harder to debug. A similar issue was brought up in relation to
mod_wsgi:
http://code.google.com/p/modwsgi/issues/detail?id=29
Your observation about errno means that mod_wsgi should also change
what error exception type it is using as well.
Graham
On Nov 2, 8:31 pm, Bjørn Stabell <bjo...@gmail.com> wrote:
> On Oct 6, 11:20 pm, Trey <jameslon...@gmail.com> wrote:
>
>
>
> > There are other people that have brought this up a little bit some
> > time ago. I run a small to medium sized web application that takes
> > profile pictures. By far my largest customer service issue is people
> > not being able to upload their photos.
>
> > For the most part I have played it down as their connection sucking or
> > perhaps doing something stupid with the browser, but there are a
> > couple of things that I am running into that are causing an issue.
>
> > 1. I can't replicate this, no matter what I do with my browser in the
> > middle of an upload.
> > 2. Judging by the django code near the problem, this is working on
> > information that has already been received.
> > 3. I get this a few times a day at least, different people every time.
> [...]
> > File "/usr/lib/python2.5/site-packages/django/core/handlers/
> > modpython.py", line 120, in _get_raw_post_data
> > self._raw_post_data = self._req.read()
>
> > IOError: Client read error (Timeout?)
>
> We're seeing this too, on several Django sites. Annoyingly, as well,
> the IOError exception itself is broken; if you look at the docs,
> IOErrors (which are a form of EnvironmentErrors) should have a two- or
> three-tuple .args, one of which would be the errno, but the ones
> thrown bymod_python/Django seems to have only one item in the tuple;
Graham Dumpleton
this is in some way meant to define a standard. That is, in zip and
bz2 modules, they raise IOError's with only a string value and no
errno.
Graham
On Nov 4, 9:32 am, Graham Dumpleton <Graham.Dumple...@gmail.com>
wrote: