Implementing graceful shutdown

[Tatu Saloranta]

One challenge I have noticed in shutting down nodes of distributed
systems is that of graceful shutdown, by first starting to decline new
requests, and stopping actively contacting other nodes, completing
processing of existing requests and so on.
DropWizard exposes at least configuration setting for this
(http.shutdownGracePeriod), but while this helps from Jetty side, I
don't yet know how to plug in my own handlers; and my own shutdown
hooks (for Servlets etc) only seem to get called after this period has
ended. I would want to get notified of initial shutdown initiation, to
stop active parts, and then later on could handle full cleanup.

I was wondering if anyone has found useful additional hooks to connect
to, perhaps some kind of additional lifecycle events?

Apologies if above sounds vague; I can share more details as necessary.

-+ Tatu +-

[Ted M. Young [@jitterted]]

How are you (or others on this list!) initiating the shutdown of the node(s), Ctrl+C, or JMX, or an API call? I'm thinking about how I'd manage (monitor, shutdown, restart) multiple nodes from a web-based dashboard, and I haven't looked into how to remotely do this.

;ted
--

http//about.me/tedmyoung

-+ Tatu +-

--
You received this message because you are subscribed to the Google Groups "dropwizard-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dropwizard-us...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Nick Telford]

Unless I've misunderstood you, this is what the Managed interface is for.

The start() method of each Managed object is called in the order that it is added to the Environment through Environment#manage(...), with the web-server always being started last. The stop() method is called in the reverse order, with the web-server always being stopped first.

In your stop() method, you can do blocking graceful shutdowns with configurable timeouts.

Regards,
--
Nick Telford

[Coda Hale]

The ideal shutdown sequence is:

1. The server socket is closed.

2. The server thread pool quiesces, handling all in-flight requests.

3. All managed objects are notified that the system is shutting down.

This is what Dropwizard does.

--
You received this message because you are subscribed to the Google Groups "dropwizard-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dropwizard-us...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Coda Hale
http://codahale.com

[Tatu Saloranta]

On Sat, Feb 23, 2013 at 3:12 AM, Nick Telford <nick.t...@gmail.com> wrote:
> Unless I've misunderstood you, this is what the Managed interface is for.
>
> The start() method of each Managed object is called in the order that it is
> added to the Environment through Environment#manage(...), with the
> web-server always being started last. The stop() method is called in the
> reverse order, with the web-server always being stopped first.
>
> In your stop() method, you can do blocking graceful shutdowns with
> configurable timeouts.

I am familiar with these (and underlying destroy() for Servlets), but
my use is bit more involved, and simple linear shutdown won't quite
work (or at least not as well). I will elaborate on this in another
answer.

-+ Tatu +-

[Tatu Saloranta]

On Fri, Feb 22, 2013 at 7:01 PM, Ted M. Young [@jitterted]
<tedy...@gmail.com> wrote:
> How are you (or others on this list!) initiating the shutdown of the
> node(s), Ctrl+C, or JMX, or an API call? I'm thinking about how I'd manage
> (monitor, shutdown, restart) multiple nodes from a web-based dashboard, and
> I haven't looked into how to remotely do this.

For me simple Ctrl+C, signal from which is caught by Jetty and
percolates as expected through DropWizard. JMX may work better for
remote access case.

-+ Tatu +-

[Tatu Saloranta]

On Sat, Feb 23, 2013 at 11:15 AM, Coda Hale <coda...@gmail.com> wrote:
> The ideal shutdown sequence is:
>
> 1. The server socket is closed.
> 2. The server thread pool quiesces, handling all in-flight requests.
> 3. All managed objects are notified that the system is shutting down.
>
> This is what Dropwizard does.

Yes, and this works well for services that are completely driven by
client requests (which is pretty much any service I have written
before one I am currently working on).
I assume that shutdown period limits maximum time between steps 1 and
3, to limit maximum time it takes for system to shut down, regardless
of possible slow clients and such.

In my case services also collaborate with each other, mostly to
synchronize data. Minimally I just need notification from step (1), to
start shutdown active parts of services (stop sending synchronization
requests; receivers are already stopped by DW). Otherwise it is likely
to have in-flight out-going requests, as they are not drained, causing
spurious errors when incoming requests are exhausted.

An additional useful thing with this is that service itself can also
be aware of the shutdown process, so that logging of problems can be
changed to suppress problems that are likely caused by shutdown
process.

I hope this makes more sense.

I can look at the code, and perhaps one possibility is to just
register alternate shutdown hooks; these should be called (AFAIK) in
parallel, in arbitrary order. If so there is no need for specific
support. But I wanted to check if something might already exists for
this.

-+ Tatu +-