How should connections closed by remote peer be handled?

[Jacob Kristhammar]

Hi,

I'm trying to figure out how to properly handle connections closed by
the client in Tornado apps. There's mainly two things that I'm
concerned with.

* It's possible to handle closed connection by using IOStream's
set_close_callback but the iostream object is buried deep inside of
Tornado making it feel kind of awkward
(self.request.connection.stream).
(Also, Tornado doesn't detect connections closed by the remote peer
when using select until a read/write operation is performed. see
http://github.com/facebook/tornado/issues/#issue/37)

* Because of the asynchronous nature of Tornado, e.g. execution might
have been passed of to a AsyncHTTPClient callback after which the
connection is closed and the callback will be invoked in a state when
the connection is no longer alive.

1) Trying to write to the connection using self.write() and
self.finish() will trigger an IOError (when epoll is used) which will
be caught when the callback is properly wrapped with
web.async_callback.
Again, it would be possible to check if the connection is still alive
before writing but this requires another stretch far into tornado.
2) The real pickle occurs in the scenario just described but there's
some error-condition and web.tornado.HTTPError is used in the request
handler to send an error to the waiting client.
In this situation the same IOError will be raised but this time from
the code inside of _handle_request_exception and not from the
callback. This will cause a unhandled exception that will propagate
all the way up. (the except in async_callback catched the HTTPError
which in turn generated the IOError).

These situations can be reproduced and tested by running this server
http://gist.github.com/238741 (epoll must be used)

Step 1.
in one terminal:
python server.py --logging=debug

Step 2.
in another terminal:
- situation 1
curl 'localhost:8888?delay=3'& \sleep 1 && killall curl

- situation 2
curl 'localhost:8888?error=yes&delay=3'& \sleep 1 && killall curl

In situation 1 the error will be caught and the "application flow"
won't be broken.
In situation 2 an unhandled exception will be raised and the
"application flow" will be broken, i.e. "doing some application clean
up..." won't be logged.


With all of this in mind, what do you think is a good strategy to
handle connections closed by the client?

* Always check connection status before write?
* Wrap potential IOError sources with try/except-error handling?
* Wrap the async_callback with code that handles the situation in
situation 2 above and ignore the IOErros?
* Patch async_callback to handle it?
* ... ?

I'd love to keep on using the tools and abstractions provided by
tornado e.g. signal/send errors by raising HTTPErrors etc, but maybe
they should only be used in more simple situation when the risk of
problems mentioned above are minimal and obvious.

BR,
Jacob


## server.py
import time
import logging
import functools

import tornado.httpserver
import tornado.ioloop
import tornado.options
import tornado.web

from tornado.options import define, options

define("port", default=8888, help="run on the given port", type=int)

class Longrunner(object):
def do_later(self, evilbit, on_success, on_error):
logging.info("doing application stuff...")
if evilbit:
on_error("oops!")
else:
on_success("ok!")
logging.info("doing some application clean up...")


class MainHandler(tornado.web.RequestHandler):
@tornado.web.asynchronous
def get(self):
delay = int(self.get_argument("delay", 0))
error = self.get_argument("error", None)
longrunner = functools.partial(Longrunner().do_later,
error,
self.async_callback
(self.on_success),
self.async_callback
(self.on_error))
tornado.ioloop.IOLoop.instance().add_timeout(time.time() +
delay,
longrunner)

def on_success(self, data):
self.write(data)
self.finish()

def on_error(self, data):
raise tornado.web.HTTPError(400)


def main():
tornado.options.parse_command_line()
application = tornado.web.Application([
(r"/", MainHandler),
])
http_server = tornado.httpserver.HTTPServer(application)
http_server.listen(options.port)
tornado.ioloop.IOLoop.instance().start()


if __name__ == "__main__":
main()