"client connection closed" issue
hads
connection closed" I'd like to ask a question if I may.
In an application I have a few requests that process for a number of
seconds, if a client closes the connection (browses away, closes the
browser etc.) then the apache log will show something like the
following;
[Fri Sep 07 21:27:18 2007] [error] [client 192.168.0.5] mod_wsgi
(pid=6913): Exception occurred within WSGI script '/mnt/share/www/buds/
buds.wsgi'., referer: http://buds.nice.net.nz/group/
[Fri Sep 07 21:27:18 2007] [error] [client 192.168.0.5] IOError:
client connection closed, referer: http://buds.nice.net.nz/group/
If the client then tries to make another request they won't get a
response for a number of seconds (60ish), some sort of timeout. Making
a request from another computer on the network isn't affected.
Is this a known/common issue and if so is it something that can be
worked around easily?
This may be an issue with my apache configuration or code that I'm
using, and nothing at all to do with mod_wsgi. If it is and this is
the wrong list to ask on then please feel free to let me know.
Cheers,
hads
Graham Dumpleton
I can't see that this would be an issue with mod_wsgi itself or even
the Apache configuration.
What package are you using on top of mod_wsgi? How are responses
returned by the application? Do you return a complete response in one
go from the application or use yield to return the response in bits?
Initially the only reason I could see for this would be an application
timeout related to some data for that specific client/user. If yield
was used to return data in bits and when mod_wsgi is writing back the
data, that error with the client connection occurs, then mod_wsgi
would simply stop that request at that point and finish up with an
error. The mod_wsgi code would not go on to keep calling the generator
so that your handler could return all the data it was producing. A
consequence of this might be that if after yielding all the data the
application would normally release some resource, such as a lock, then
that would never occur.
The WSGI specification requires that whether a response is successful
or not, that the WSGI adapter for the web server calls close() on any
generator. The purpose of this is so that any generator could release
any resources that it may have acquired, or close off files etc. If
you are using yield directly though, you cant add stuff to be called
in the close() method of the generator. In that case the code would
need to be restructured to a degree to somehow use an intermediary
generator that you provide which allows you to define a close() method
which will always release the resources.
In summary, if you have code which looks like:
lock.acquire()
yield "1"
yield "2"
lock.release()
and when "2" was being written to the client, closure of the
connection was detected and an exception raised, then the subsequent
line which releases the lock would never be called.
Tricky issue actually and means that using 'yield' not as simple as it looks.
What you might do to debug the issue is at the start of your WSGI
application, log some output. Ie.,
print >> environ['wsgi.errors'], 'start request'
If you cant easily add this into the application, use a little wrapper
around it which does it.
This will at least confirm whether the application is being entered on
the subsequent request or if it isn't even getting that far.
BTW, if using yield, also read:
http://code.google.com/p/modwsgi/wiki/RegisteringCleanupCode
as this talks about how to ensure that code is executed at end of request.
Please let me know if this sounds at all relevant. At least indicate
whether debug shows whether the application is entered for subsequent
request. If it doesn't, then Apache/mod_wsgi or something else could
be the problem, although cant think what or how at the moment.
Graham
hads
wrote:
> What package are you using on top of mod_wsgi? How are responses
> returned by the application? Do you return a complete response in one
> go from the application or use yield to return the response in bits?
I'm using Werkzeug with Jinja templating on this particular
application. I've just been doing a bit of debugging as you suggested
(and I should have done earlier). I've come to the (quick, quite
untested) conclusion that it may be an issue with some of the flup
middleware I'm using. The SessionMiddleware, ErrorMiddleware and
GzipMiddleware were all being used.
I need to do some proper testing as you suggested although it's
getting late here and it's Friday night :) It will also help when my
devel server is upgraded next week as it's a bit CPU bound at the
moment which is making the debugging process more complex.
I'll drop a line back when I have more info or hopefully a solution.
Cheers for your helpful input.
hads
Graham Dumpleton
> wrote:
> > What package are you using on top of mod_wsgi? How are responses
> > returned by the application? Do you return a complete response in one
> > go from the application or use yield to return the response in bits?
>
> I'm using Werkzeug with Jinja templating on this particular
> application. I've just been doing a bit of debugging as you suggested
> (and I should have done earlier). I've come to the (quick, quite
> untested) conclusion that it may be an issue with some of the flup
> middleware I'm using. The SessionMiddleware, ErrorMiddleware and
> GzipMiddleware were all being used.
>
> I need to do some proper testing as you suggested although it's
> getting late here and it's Friday night :)
Hmmm, didn't twig until later that that must mean you are in in NZ or
Australia, as it was late on Friday where I am at the same time. :-)
> It will also help when my
> devel server is upgraded next week as it's a bit CPU bound at the
> moment which is making the debugging process more complex.
>
> I'll drop a line back when I have more info or hopefully a solution.
Looking at Werkzeug the yield issue could indeed be the issue as their
Response object doesn't do anything to support the concept of the
close() method. In particular their __call__() method simply uses
yield directly.
def __call__(self, environ, start_response):
headers = self.headers.to_list(self.charset)
if self._cookies is not None:
for morsel in self._cookies.values():
headers.append(('Set-Cookie', morsel.output(header='')))
status = '%d %s' % (self.status, HTTP_STATUS_CODES[self.status])
charset = self.charset or 'ascii'
start_response(status, headers)
for item in self.response:
if isinstance(item, unicode):
yield item.encode(charset)
else:
yield str(item)
This means the recipe they give of:
response
Can be an iterator or a string or None. If it's an iterator, it
should return strings (unicode/str), and those strings will be joined
together to form the content of the response.
If you pass it an iterable it's used for direct data submission (streaming):
def my_view(...):
def wrapped():
for item in range(100):
yield str(item) + ' '
time.sleep(0.1)
return Response(wrapped, mimetype='text/html')
This will then take 10 seconds to submit all data but the browser
will render after each iteration.
could incur problems if after the final yield there was some cleanup
required but it never got their because of attempt to write one of the
yield values failing because of closed client connection.
Note that this isn't a mod_wsgi specific issue and would arise with
any WSGI adapter where closure of client connection before complete
response is returned could occur and would be detected by the adapter.
Graham
hads
wrote:
> Hmmm, didn't twig until later that that must mean you are in in NZ or
> Australia, as it was late on Friday where I am at the same time. :-)
Yeah, South Island, NZ.
> Looking at Werkzeug the yield issue could indeed be the issue as their
> Response object doesn't do anything to support the concept of the
> close() method. In particular their __call__() method simply uses
> yield directly.
[snip]
> This means the recipe they give of:
>
> response
>
> Can be an iterator or a string or None. If it's an iterator, it
> should return strings (unicode/str), and those strings will be joined
> together to form the content of the response.
[snip]
> could incur problems if after the final yield there was some cleanup
> required but it never got their because of attempt to write one of the
> yield values failing because of closed client connection.
I'm returning a generated template so it's a single string, no
iterating.
I've narrowed it down to the flup session middleware which is causing
the long delay, which at first glance seems to fit into your original
thought of "an application timeout related to some data for that
specific client/user" quite well.
If I remove the session middleware from the application then the long
timeout goes away. I'll do some looking through the different pieces
of code to find out where and what I need to do to fix the issue.
> Note that this isn't a mod_wsgi specific issue and would arise with
> any WSGI adapter where closure of client connection before complete
> response is returned could occur and would be detected by the adapter.
Yes, definitely not a mod_wsgi specific thing. Thanks much for your
help with it though.
Cheers,
Hadley