Feature Request: Graceful Shutdown

[Didip Kerabat]

I've seen this problem several times in this mailing list, and it's actually my problem as well.

When server receives SIGTERM or SIGINT, can it wait until ioloop._timeouts is empty?

Right now I have a duct-tapey solution that involves using PeriodicCallback -> every second, check if _timeouts is empty, then shutdown.

Is this something that can be built-in feature?

- Didip -

[Didip Kerabat]

Whoops, I'm a derp.

I meant IOLoop()._callbacks, not IOLoop()._timeouts.

- Didip -

[Ben Darnell]

I think some sort of graceful shutdown support is a good idea,
although the details are tricky and fairly application specific, so we
may not be able to do something that just works out of the box.

The first sticking point is how you ensure that you're not taking on
new work while you're shutting down. Stopping the HTTPServer is one
way to do this (and this is one thing that we should probably make
easier or more discoverable than it currently is), but depending on
your deployment strategy and load balancing stack it may be somewhat
disruptive, and you'd be better off signaling your load balancer in
another way. You probably also wouldn't want infrequent
PeriodicCallbacks to start up in this period, but curl_httpclient
depends on a PeriodicCallback that runs every second.

The second problem is deciding when it makes sense to shut down.
Waiting for the callback queue to be empty is simple, but I'm not sure
it's graceful enough, since async handlers will be cut off in the
middle. In general I'm of the opinion that it's not worth trying to
identify when the server is "idle" and it's best to just kill the
server N seconds after it's no longer receiving new traffic.

-Ben