deleting tornado's sockets on shutdown

[Ethan Collins]

I have a custom shutdown routine and one of the last thing I do is calling IOLoop.stop().

- Can I immediately delete tornado's socket at this point?

-Ethan

[Ben Darnell]

Tornado won't do anything on its own after IOLoop.stop, so in general it should be fine to do whatever you want at this point. Simply deleting the socket object is unlikely to do anything because of other references, but HTTPServer.stop() may be called before or after IOLoop.stop() to close the socket (which is the main thing that's worth doing before shutting down).

-Ben

--
You received this message because you are subscribed to the Google Groups "Tornado Web Server" group.
To unsubscribe from this group and stop receiving emails from it, send an email to python-tornad...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Ethan Collins]

Yes, I am calling HTTPServer.stop(). Below is my code, that's called from the appropriate signal handler:

def _shutdown(http_server, loop, app):
    log.info("Stopping HTTPServer...")
    http_server.stop()

    log.info("Waiting max 10.0 secs for IOLoops to finish it's work...")
    deadline = loop.time() + 10.0
    _stop_all(loop, app, deadline)

def _stop_all(loop, app, deadline):
    now = loop.time()

    if now < deadline and (loop._callbacks or loop._timeouts):
        loop.add_timeout(now + 1, \
                         functools.partial(_stop_all, loop, app, deadline))
    else:
        log.info("Closing database connection...")
        app.db.close()

        _flush_py_stdout()

        log.info("Stopping IOLoop...")
        loop.stop()
        log.info("Shutdown.")

        # TODO: delete the unix sockets here.

        # This shutdown will be on the logging module itself (and not on 'log').
        logging.shutdown()

-Ethan

[Ben Darnell]

Ah, I see, you're talking about unix sockets.  I think it should be safe to delete the socket "file" immediately after HTTPServer.stop; existing connections will continue to work.  And even if I'm wrong about that, you can definitely delete it after the IOLoop has stopped.  However, there's a subtle issue here (noted in http://www.tornadoweb.org/en/stable/ioloop.html#tornado.ioloop.IOLoop.stop):  The IOLoop is not completely stopped until start() has returned, so your final cleanup should go after the call to IOLoop.start(), not the call to stop().

-Ben

[Ethan Collins]

Hi Ben,

Thanks for the pointer. Makes sense. So the updated code will look something like this:

def _stop_all(loop, app, deadline):
    now = loop.time()

    if now < deadline and (loop._callbacks or loop._timeouts):
        loop.add_timeout(now + 1, \
                         functools.partial(_stop_all, loop, app, deadline))
    else:

        log.info("Stopping IOLoop...")
        loop.stop()
        return


And in the app's main(), it looks like this:

    <snip>
    loop.start()

   
    log.info("Closing database connection...")
    app.db.close()
    _flush_py_stdout()

    log.info("Shutdown.")

    # Delete the unix sockets here:
    try:
        st = os.stat(file)
    except OSError as err:
        log.debug('Err: tornado app socket not found. code: %s', err.errno)
    else:
        os.remove(file)

   
    # This shutdown will be on the logging module itself (and not on 'log').
    logging.shutdown()
  

    # Shutdown complete
    return

Best,
Ethan

[Ethan Collins]

Hi Ben,

I was just wondering if it's ok to call lOLoop.stop immediately after HTTPServer.stop. If you look at my code posted 3 posts back, I was polling for ~10secs (an absurdly large number) and checking for (IOLoop._callbacks or IOLoop._timeouts) before calling IOLoop.stop. If it's possible to IOLoop.stop immediately after httpserver.stop, then the code becomes much simplier.

-Ethan

[Ben Darnell]

On Fri, Dec 20, 2013 at 5:12 PM, Ethan Collins <collins...@gmail.com> wrote:

Hi Ben,

I was just wondering if it's ok to call lOLoop.stop immediately after HTTPServer.stop. If you look at my code posted 3 posts back, I was polling for ~10secs (an absurdly large number) and checking for (IOLoop._callbacks or IOLoop._timeouts) before calling IOLoop.stop. If it's possible to IOLoop.stop immediately after httpserver.stop, then the code becomes much simplier.

This will work fine from the server's perspective, but it may cause some client connections to be cut off in flight.  The point of the "http_server.stop(); io_loop.add_timeout(N, io_loop.stop)" pattern is to stop accepting new connections but allow any existing connections to finish what they're doing.  If you just want to flush the logs it's fine to stop everything at once, but if you want the restart to be invisible from the client's perspective you need to wait.  

It's not possible to detect when the server is idle from inspecting IOLoop internals (A user downloading a large file will show up in _handlers, not _callbacks or _timeouts, for example.  If you use curl_httpclient or autoreload then you'll always have at least one entry in _timeouts).  I recommend using a fixed timeout for the shutdown period.  If you want to detect idleness, you'll need to do it at a higher level (an in-flight request counter could be a good thing to add to HTTPServer, although even then you'll probably need special handling for things like long polling and websockets)

-Ben

[Ethan Collins]

Hi Ben,

Thanks for the info. I was trying to debug a bit. I notice that the _handlers always contain the key 6. I tried this without any connections to tornado. What is this default handler?

-Ethan

[Ben Darnell]

On Sat, Dec 21, 2013 at 11:30 AM, Ethan Collins <collins...@gmail.com> wrote:

Hi Ben,

Thanks for the info. I was trying to debug a bit. I notice that the _handlers always contain the key 6. I tried this without any connections to tornado. What is this default handler?

It's the waker pipe: 

https://github.com/facebook/tornado/blob/master/tornado/ioloop.py#L506

We use this to wake up the IOLoop when add_callback is called from another thread; it's a variant of the "self-pipe trick" for signal handlers (http://cr.yp.to/docs/selfpipe.html)

-Ben