JVM question: SIGTERM

[Nils Kilden-Pedersen]

From what I've read, the normal way to deal with a SIGTERM is to add a shutdown hook, which makes sense.

However I'm not sure if SIGTERM also leads to interruption of all running threads, or if each thread have to add their own shutdown hook to properly shut down (or any shutdown hook be responsible for interrupting running threads)?

Thanks.

[√iktor Ҡlang]

I'm not sure I understand what you mean, when you add a shutdown hook, you add an unstarted Thread.

Cheers,

√

--

Viktor Klang
Director of Engineering

Typesafe - The software stack for applications that scale
Twitter: @viktorklang

[Nils Kilden-Pedersen]

On Sun, Jan 6, 2013 at 4:57 PM, √iktor Ҡlang <viktor...@gmail.com> wrote:

I'm not sure I understand what you mean, when you add a shutdown hook, you add an unstarted Thread.

Yes. But what I don't know yet is if all running threads are interrupted, and if so, does that happen before or after the shutdown hook(s) is run?
 

[AGYNAMIX Torsten Uhlmann]

Hi Nils,

from the JavaDoc at Runtime.addShutdownHook:

A shutdown hook is simply an initialized but unstarted thread. When the virtual machine begins its shutdown sequence it will start all registered shutdown hooks in some unspecified order and let them run concurrently. When all the hooks have finished it will then run all uninvoked finalizers if finalization-on-exit has been enabled. Finally, the virtual machine will halt. Note that daemon threads will continue to run during the shutdown sequence, as will non-daemon threads if shutdown was initiated by invoking the exit method.

So from the description I would answer your question on whether threads are stopped prior or after the shutdown hook with a definitive "maybe" :) 

And you need only one shutdown hook, not one per running thread. However you can register multiple...

Does that help you a bit?

Torsten.

[Nils Kilden-Pedersen]

On Mon, Jan 7, 2013 at 2:51 AM, AGYNAMIX Torsten Uhlmann <T.Uh...@agynamix.de> wrote:

Hi Nils,

from the JavaDoc at Runtime.addShutdownHook:

A shutdown hook is simply an initialized but unstarted thread. When the virtual machine begins its shutdown sequence it will start all registered shutdown hooks in some unspecified order and let them run concurrently. When all the hooks have finished it will then run all uninvoked finalizers if finalization-on-exit has been enabled. Finally, the virtual machine will halt. Note that daemon threads will continue to run during the shutdown sequence, as will non-daemon threads if shutdown was initiated by invoking the exit method.

So from the description I would answer your question on whether threads are stopped prior or after the shutdown hook with a definitive "maybe" :) 

And you need only one shutdown hook, not one per running thread. However you can register multiple...

Does that help you a bit?

Not really, because I had already read that :-)

To boil it down: What happens with the running threads? Are they interrupted, which allows for proper resource deallocation, or are they effectively stopped? I have a hard time believing the latter, since the Thread.stop() method itself has been deprecated (as unsafe) since forever, but I haven't found a definite answer. I guess I have to test it and hope that it's representative across implementations.

 

[√iktor Ҡlang]

On Mon, Jan 7, 2013 at 3:23 PM, Nils Kilden-Pedersen <nil...@gmail.com> wrote:

On Mon, Jan 7, 2013 at 2:51 AM, AGYNAMIX Torsten Uhlmann <T.Uh...@agynamix.de> wrote:

Hi Nils,

from the JavaDoc at Runtime.addShutdownHook:

A shutdown hook is simply an initialized but unstarted thread. When the virtual machine begins its shutdown sequence it will start all registered shutdown hooks in some unspecified order and let them run concurrently. When all the hooks have finished it will then run all uninvoked finalizers if finalization-on-exit has been enabled. Finally, the virtual machine will halt. Note that daemon threads will continue to run during the shutdown sequence, as will non-daemon threads if shutdown was initiated by invoking the exit method.

So from the description I would answer your question on whether threads are stopped prior or after the shutdown hook with a definitive "maybe" :) 

And you need only one shutdown hook, not one per running thread. However you can register multiple...

Does that help you a bit?

Not really, because I had already read that :-)

To boil it down: What happens with the running threads? Are they interrupted, which allows for proper resource deallocation,

What type of resource deallocation are we talking about here, the process will be terminated...

 

or are they effectively stopped? I have a hard time believing the latter, since the Thread.stop() method itself has been deprecated (as unsafe) since forever, but I haven't found a definite answer. I guess I have to test it and hope that it's representative across implementations.

 

Torsten.

Am 07.01.2013 um 03:05 schrieb Nils Kilden-Pedersen <nil...@gmail.com>:

On Sun, Jan 6, 2013 at 4:57 PM, √iktor Ҡlang <viktor...@gmail.com> wrote:

I'm not sure I understand what you mean, when you add a shutdown hook, you add an unstarted Thread.

Yes. But what I don't know yet is if all running threads are interrupted, and if so, does that happen before or after the shutdown hook(s) is run?
 

Cheers,

√

On Sun, Jan 6, 2013 at 11:13 PM, Nils Kilden-Pedersen <nil...@gmail.com> wrote:

From what I've read, the normal way to deal with a SIGTERM is to add a shutdown hook, which makes sense.

However I'm not sure if SIGTERM also leads to interruption of all running threads, or if each thread have to add their own shutdown hook to properly shut down (or any shutdown hook be responsible for interrupting running threads)?

Thanks.

--

Viktor Klang
Director of Engineering

Typesafe - The software stack for applications that scale
Twitter: @viktorklang

[Nils Kilden-Pedersen]

On Mon, Jan 7, 2013 at 8:28 AM, √iktor Ҡlang <viktor...@gmail.com> wrote:

On Mon, Jan 7, 2013 at 3:23 PM, Nils Kilden-Pedersen <nil...@gmail.com> wrote:

On Mon, Jan 7, 2013 at 2:51 AM, AGYNAMIX Torsten Uhlmann <T.Uh...@agynamix.de> wrote:

Hi Nils,

from the JavaDoc at Runtime.addShutdownHook:

A shutdown hook is simply an initialized but unstarted thread. When the virtual machine begins its shutdown sequence it will start all registered shutdown hooks in some unspecified order and let them run concurrently. When all the hooks have finished it will then run all uninvoked finalizers if finalization-on-exit has been enabled. Finally, the virtual machine will halt. Note that daemon threads will continue to run during the shutdown sequence, as will non-daemon threads if shutdown was initiated by invoking the exit method.

So from the description I would answer your question on whether threads are stopped prior or after the shutdown hook with a definitive "maybe" :) 

And you need only one shutdown hook, not one per running thread. However you can register multiple...

Does that help you a bit?

Not really, because I had already read that :-)

To boil it down: What happens with the running threads? Are they interrupted, which allows for proper resource deallocation,

What type of resource deallocation are we talking about here, the process will be terminated...

Does it matter? This is relevant in the generic case, just like the difference between calling stop() vs. interrupt() on a thread.

[√iktor Ҡlang]

Of course it matters. Are we talking external resources? (I guess we are, since the process will exit and all local resources be reclaimed by the OS).

If we're talking about external resources we cannot rely on addShutdownHook, as someone could forcibly terminate the process anyway.

Cheers,

√ 

[Nils Kilden-Pedersen]

This matters for the same reason Thread.interrupt() method was introduced, replacing stop(). Enough said.

[Nils Kilden-Pedersen]

Which incidentally is also why not all threads are daemon threads.

[Brian Maso]

I have no idea if it matters to anyone paying attention to this thread... But it is somewhat interesting to note that Thread.stop()  is implemented by the VM causing a Throwable to be thrown at the targeted Thread -- specifically an instance of the ominously-sounding ThreadDeath class. 

--
Best regards,
Brian Maso
(949) 395-8551
Follow me: @bmaso
br...@blumenfeld-maso.com

[√iktor Ҡlang]

On Mon, Jan 7, 2013 at 5:05 PM, Nils Kilden-Pedersen <nil...@gmail.com> wrote:

This matters for the same reason Thread.interrupt() method was introduced, replacing stop(). Enough said.

If you can come up with an actual use-case we can start discussing potential solutions to that problem,

right now it's unsure to me whether there is a problem or not.

Are you looking for a way to interrupt all running threads, triggered from the outside, or something else?

Cheers,

√

[Nils Kilden-Pedersen]

It probably doesn't. I suspect that was just the mechanism, just as InterruptedException is thrown when calling interrupt(). Difference being that with stop() the state of locks are different than with interrupt().

To conclude:
From what I've read since, it appears that the threads are indeed stopped, not interrupted, which means that the right procedure, if this matters to the application, is to add one (or more) shutdown hooks that interrupts the threads that needs interruption, to ensure proper predictable thread termination.

[√iktor Ҡlang]

On Mon, Jan 7, 2013 at 5:52 PM, Nils Kilden-Pedersen <nil...@gmail.com> wrote:

On Mon, Jan 7, 2013 at 10:37 AM, Brian Maso <br...@blumenfeld-maso.com> wrote:

On Monday, January 7, 2013, Nils Kilden-Pedersen wrote:

On Mon, Jan 7, 2013 at 8:28 AM, √iktor Ҡlang <viktor...@gmail.com> wrote:

On Mon, Jan 7, 2013 at 3:23 PM, Nils Kilden-Pedersen <nil...@gmail.com> wrote:

On Mon, Jan 7, 2013 at 2:51 AM, AGYNAMIX Torsten Uhlmann <T.Uh...@agynamix.de> wrote:

Hi Nils,

from the JavaDoc at Runtime.addShutdownHook:

A shutdown hook is simply an initialized but unstarted thread. When the virtual machine begins its shutdown sequence it will start all registered shutdown hooks in some unspecified order and let them run concurrently. When all the hooks have finished it will then run all uninvoked finalizers if finalization-on-exit has been enabled. Finally, the virtual machine will halt. Note that daemon threads will continue to run during the shutdown sequence, as will non-daemon threads if shutdown was initiated by invoking the exit method.

So from the description I would answer your question on whether threads are stopped prior or after the shutdown hook with a definitive "maybe" :) 

And you need only one shutdown hook, not one per running thread. However you can register multiple...

Does that help you a bit?

Not really, because I had already read that :-)

To boil it down: What happens with the running threads? Are they interrupted, which allows for proper resource deallocation,

What type of resource deallocation are we talking about here, the process will be terminated...

Does it matter? This is relevant in the generic case, just like the difference between calling stop() vs. interrupt() on a thread.

I have no idea if it matters to anyone paying attention to this thread... But it is somewhat interesting to note that Thread.stop()  is implemented by the VM causing a Throwable to be thrown at the targeted Thread -- specifically an instance of the ominously-sounding ThreadDeath class. 

It probably doesn't. I suspect that was just the mechanism, just as InterruptedException is thrown when calling interrupt().

"interrupt" does not throw an InterruptedException, it sets an interrupted flag, which you can inspect and decide t throw an InterruptedException (this is done by normal JDK thinks like IO, monitors etc)

Evidence:

scala> val t = new Thread(new Runnable { def run = try { while(true) {} } catch { case i: InterruptedException => i.printStackTrace } })

t: java.lang.Thread = Thread[Thread-8,5,main]

scala> t.start

scala> t.interrupt

scala> t.isInterrupted

res5: Boolean = true

 

Difference being that with stop() the state of locks are different than with interrupt(). 

It's more like it is not really specified what will happen on Thread.stop, normally I assume it is implemented by issuing a thrown ThreadDeath at a safepoint, however, you never know with loaded native libs etc.

 

To conclude:
From what I've read since, it appears that the threads are indeed stopped, not interrupted, which means that the right procedure, if this matters to the application, is to add one (or more) shutdown hooks that interrupts the threads that needs interruption, to ensure proper predictable thread termination.

This only works if all those Threads are executing code that respects Thread.isInterrupted, which I assume it does in your case, but just pointing that out.

Cheers,

√

[Som Snytt]

On Mon, Jan 7, 2013 at 6:23 AM, Nils Kilden-Pedersen <nil...@gmail.com> wrote:

On Mon, Jan 7, 2013 at 2:51 AM, AGYNAMIX Torsten Uhlmann <T.Uh...@agynamix.de> wrote:

from the JavaDoc at Runtime.addShutdownHook:

 Note that daemon threads will continue to run during the shutdown sequence, as will non-daemon threads if shutdown was initiated by invoking the exit method.

So from the description I would answer your question on whether threads are stopped prior or after the shutdown hook with a definitive "maybe" :) 

Not really, because I had already read that :-)

To boil it down: What happens with the running threads? Are they interrupted, which allows for proper resource deallocation, or are they effectively stopped?

The confusing language in the "Note that" seems to mean simply: Usually we start the shutdown sequence when non-daemon threads are dead; the remaining daemon threads keep running.  If someone calls exit, all threads continue running.

However, threads are not interrupted.

Apologies in advance if I'm over-simplifying the question or construing it narrowly. (I think the original question was something like, does sigterm invoke exit while sigkill aborts like halt.)  Viktor responded to the larger question about resources.

Simple example, similar for shutting down thread pools, for example.
https://gist.github.com/4477288

[Nils Kilden-Pedersen]

On Mon, Jan 7, 2013 at 11:15 AM, √iktor Ҡlang <viktor...@gmail.com> wrote:

On Mon, Jan 7, 2013 at 5:52 PM, Nils Kilden-Pedersen <nil...@gmail.com> wrote:

On Mon, Jan 7, 2013 at 10:37 AM, Brian Maso <br...@blumenfeld-maso.com> wrote:

On Monday, January 7, 2013, Nils Kilden-Pedersen wrote:

On Mon, Jan 7, 2013 at 8:28 AM, √iktor Ҡlang <viktor...@gmail.com> wrote:

On Mon, Jan 7, 2013 at 3:23 PM, Nils Kilden-Pedersen <nil...@gmail.com> wrote:

On Mon, Jan 7, 2013 at 2:51 AM, AGYNAMIX Torsten Uhlmann <T.Uh...@agynamix.de> wrote:

Hi Nils,

from the JavaDoc at Runtime.addShutdownHook:

A shutdown hook is simply an initialized but unstarted thread. When the virtual machine begins its shutdown sequence it will start all registered shutdown hooks in some unspecified order and let them run concurrently. When all the hooks have finished it will then run all uninvoked finalizers if finalization-on-exit has been enabled. Finally, the virtual machine will halt. Note that daemon threads will continue to run during the shutdown sequence, as will non-daemon threads if shutdown was initiated by invoking the exit method.

So from the description I would answer your question on whether threads are stopped prior or after the shutdown hook with a definitive "maybe" :) 

And you need only one shutdown hook, not one per running thread. However you can register multiple...

Does that help you a bit?

Not really, because I had already read that :-)

To boil it down: What happens with the running threads? Are they interrupted, which allows for proper resource deallocation,

What type of resource deallocation are we talking about here, the process will be terminated...

Does it matter? This is relevant in the generic case, just like the difference between calling stop() vs. interrupt() on a thread.

I have no idea if it matters to anyone paying attention to this thread... But it is somewhat interesting to note that Thread.stop()  is implemented by the VM causing a Throwable to be thrown at the targeted Thread -- specifically an instance of the ominously-sounding ThreadDeath class. 

It probably doesn't. I suspect that was just the mechanism, just as InterruptedException is thrown when calling interrupt().

"interrupt" does not throw an InterruptedException, it sets an interrupted flag, which you can inspect and decide t throw an InterruptedException (this is done by normal JDK thinks like IO, monitors etc)

Sure, poorly worded on my part.
 

Evidence:

scala> val t = new Thread(new Runnable { def run = try { while(true) {} } catch { case i: InterruptedException => i.printStackTrace } })

t: java.lang.Thread = Thread[Thread-8,5,main]

scala> t.start

scala> t.interrupt

scala> t.isInterrupted

res5: Boolean = true

 

Difference being that with stop() the state of locks are different than with interrupt(). 

It's more like it is not really specified what will happen on Thread.stop, normally I assume it is implemented by issuing a thrown ThreadDeath at a safepoint, however, you never know with loaded native libs etc.

 

To conclude:
From what I've read since, it appears that the threads are indeed stopped, not interrupted, which means that the right procedure, if this matters to the application, is to add one (or more) shutdown hooks that interrupts the threads that needs interruption, to ensure proper predictable thread termination.

This only works if all those Threads are executing code that respects Thread.isInterrupted, which I assume it does in your case, but just pointing that out.

Right.
 

[Nils Kilden-Pedersen]

On Mon, Jan 7, 2013 at 12:44 PM, Som Snytt <som....@gmail.com> wrote:

On Mon, Jan 7, 2013 at 6:23 AM, Nils Kilden-Pedersen <nil...@gmail.com> wrote:

On Mon, Jan 7, 2013 at 2:51 AM, AGYNAMIX Torsten Uhlmann <T.Uh...@agynamix.de> wrote:

from the JavaDoc at Runtime.addShutdownHook:

 Note that daemon threads will continue to run during the shutdown sequence, as will non-daemon threads if shutdown was initiated by invoking the exit method.

So from the description I would answer your question on whether threads are stopped prior or after the shutdown hook with a definitive "maybe" :) 

Not really, because I had already read that :-)

To boil it down: What happens with the running threads? Are they interrupted, which allows for proper resource deallocation, or are they effectively stopped?

The confusing language in the "Note that" seems to mean simply: Usually we start the shutdown sequence when non-daemon threads are dead; the remaining daemon threads keep running.  If someone calls exit, all threads continue running.

I don't know how to interpret that, because the SIGTERM case appears undefined (in the Javadocs at least).
 

However, threads are not interrupted.

Right, this seems to be the case.
 

[ScottC]

Fir example you may want to flush some data to disk or network before termination. We are talking the difference between kill 9 and kill 15. Pure process termination can be bad for many applications.

[ScottC]

All threads will have their interrupt flag set. Sleeping or waiting threads will throw InterupteedException. Shutdown hooks are not necessary if all non-daemon threads deal with this correctly. Non daemon threads that catch interuptedException and keep runing without setting the interupted flag will prevent vm shutdown. Shut down hooks that do not exit will also prevent shutdown.

[ScottC]

There are thousands of use case. Most applications behave differently between SIGTERM and SIGKILL for very good reasons. Safe vs unsafe shutdown matters. Unsafe shutdown may require messy or expensive cleanup upon restart.

The attitude demanding a use case rather than answering the question is unecessary.

[ScottC]

My experience differs, several of my applications manage shurdown via interupted exceptions. However this is only thrown when your thread hits something that throws it after checking the interupted flag. A thread doing IO, waiting for a lock, or sleeping will be woken up on sigterm. If your threads are daemons or not will affect the behavior. If they are managed or not by a thread pool or framework it will as well as those may have their own shutdown hooks.

Shutdown hooks are the most flexible and safe tools here, but not the only way.

[Nils Kilden-Pedersen]

On Mon, Jan 7, 2013 at 2:08 PM, ScottC <scott...@gmail.com> wrote:

All threads will have their interrupt flag set.   Sleeping or waiting threads will throw InterupteedException.  

Have you tested this or do you have a reference? I'd love for that to be true.
 

[Scott Carey]

On Jan 7, 2013 10:38 AM, "Nils Kilden-Pedersen" <nil...@gmail.com> wrote:
>
> On Mon, Jan 7, 2013 at 2:08 PM, ScottC <scott...@gmail.com> wrote:
>>
>> All threads will have their interrupt flag set.   Sleeping or waiting threads will throw InterupteedException.  
>
>
> Have you tested this or do you have a reference? I'd love for that to be true.
>  

Not recently and I only have my smartphone available today.  I have several applications that rely on this and have fixed several "aplication fails to stop on SIGTERM" bugs by properly catching InteruptedException, cleaning up, and exiting the work loop; or by making sure some code that caught it and did not initiate shutdown reset the interrupted flag properly; or by setting the daemon flag on threads that do not require clean shutdown.

[Scott Carey]

The important detail that Victor referenced is that calling interrupted () merely sets the interrupt flag.  Various JDK and system level methods that declare InterruptedExceptions check this flag and if it is true throw the exception. If your code never touches such a method, it wil not get that exception.

[Nils Kilden-Pedersen]

On Mon, Jan 7, 2013 at 2:47 PM, Scott Carey <scott...@gmail.com> wrote:

On Jan 7, 2013 10:38 AM, "Nils Kilden-Pedersen" <nil...@gmail.com> wrote:
>
> On Mon, Jan 7, 2013 at 2:08 PM, ScottC <scott...@gmail.com> wrote:
>>
>> All threads will have their interrupt flag set.   Sleeping or waiting threads will throw InterupteedException.  
>
>
> Have you tested this or do you have a reference? I'd love for that to be true.
>  

Not recently and I only have my smartphone available today.  I have several applications that rely on this and have fixed several "aplication fails to stop on SIGTERM" bugs by properly catching InteruptedException, cleaning up, and exiting the work loop; or by making sure some code that caught it and did not initiate shutdown reset the interrupted flag properly; or by setting the daemon flag on threads that do not require clean shutdown.

Ok, that's great to hear. I'm going to rely on this for now and verify later.

Thanks.


 

[√iktor Ҡlang]

On Mon, Jan 7, 2013 at 9:23 PM, ScottC <scott...@gmail.com> wrote:

There are thousands of use case. Most applications behave differently between SIGTERM and SIGKILL for very good reasons.  Safe vs unsafe shutdown matters.  Unsafe shutdown may require messy or expensive cleanup upon restart.

The attitude demanding a use case rather than answering the question is unecessary.

I firmly believe that to solve a problem you first need to understand it. The problem == the use-case, I have spent all too much time in my life helping people with the wrong solution, so the very least I ask is to understand the use-case before I suggest appropriate solutions.

Hope that makes sense.

Cheers,

√