JVM hanging after -main

[Drew Raines]

I have a command line utility that calls (exit 0) at the end of
(-main). It looks like this:

(defn exit [status]
(shutdown-agents)
(flush)
(System/exit status))

Yet, despite this, the JVM never exits. Here is a snippet of jstack
output:

--8<---------------cut here---------------start------------->8---
"DestroyJavaVM" prio=10 tid=0x00000000406c7c00 nid=0x445d waiting on condition [0x0000000000000000..0x0000000041e48d10]
java.lang.Thread.State: RUNNABLE

Locked ownable synchronizers:
- None

"pool-1-thread-10" prio=10 tid=0x00007fd9744fac00 nid=0x447c waiting on condition [0x0000000042e58000..0x0000000042e58da0]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00007fd981140198> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
--8<---------------cut here---------------end--------------->8---

What else can I try to force everything to shut down?

Thanks!

-Drew

[Chas Emerick]

I believe this is an issue related to how the threadpools used by
executors are populated by default, i.e. they use non-daemon threads.

clojure.lang.Agent uses instances of these default threadpool
configurations, which is likely the cause of the delayed shutdown of
the JVM (I'll bet that if you let the process linger for a couple
minutes, the threadpools' timeouts will trip, the non-daemon threads
will die, and the process will exit gracefully).

The solution would be for clojure.lang.Agent to create
ExecutorServices whose threadpools use a ThreadFactory that only
creates daemon threads. I don't really use agents though, so I'm not
at all in a position to write or test that patch (or be aware of what
other consequences might arise from having agents executing on daemon
threads -- I presume there aren't any, but...).

- Chas

>  jvm-hang.txt
> 11KViewDownload

[Mark Volkmann]

Does it shutdown if you do this before the exit?

(shutdown-agents)

--
R. Mark Volkmann
Object Computing, Inc.

[Drew Raines]

Chas Emerick wrote:

> I believe this is an issue related to how the threadpools used by
> executors are populated by default, i.e. they use non-daemon
> threads.
>
> clojure.lang.Agent uses instances of these default threadpool
> configurations, which is likely the cause of the delayed shutdown
> of the JVM (I'll bet that if you let the process linger for a
> couple minutes, the threadpools' timeouts will trip, the non-daemon
> threads will die, and the process will exit gracefully).

Oddly enough, this is the behavior I generally see interactively.
When spawned via cron(8), however, the java processes linger for
weeks and I have to manually kill(1) them.

Is there something in that execution environment that would suppress
the threadpool timeout?

-Drew

[billh04]

On Apr 16, 11:25 am, Drew Raines <aarai...@gmail.com> wrote:
> I have a command line utility that calls (exit 0) at the end of
> (-main).  It looks like this:
>
>   (defn exit [status]
>     (shutdown-agents)
>     (flush)
>     (System/exit status))
>
> Yet, despite this, the JVM never exits.

The documentation for "shutdown-agents" says "Initiates a shutdown of
the thread pools that back the agent system. Running actions will
complete, but no new actions will be accepted".

I think you are responsible for ending the currently running agents.
The usual method is to set up some field which the agents monitor
looking for some value indicating the application is ending.

You have a command-line utility, so the following doesn't apply to
your case. I have a gui application with multiple windows for each
"document". When the document window closes, I need to stop the
associated agents for that window. Rather than set up some field as I
suggested above, I do the following.

===========================
(defn bossAgentAction [_ achiGameHandle #^JFrame achiFrame]
(when (.isDisplayable achiFrame)
....
(send-off *agent* #'bossAgentAction achiGameHandle achiFrame)))
nil)
===========================

That is, I use the JFrame itself to indicate when the agent thread
should terminate.

[Drew Raines]

billh04 wrote:

> I think you are responsible for ending the currently running
> agents. The usual method is to set up some field which the agents
> monitor looking for some value indicating the application is
> ending.

By "ending currently running agents" do you mean the action run on
the agent? My problem doesn't seem to be action-related because when
I take out the agents and execute the same code synchronously, the
JVM exits as I expect.

For the record, I was able to consistently hang the JVM indefinitely
from the command line in addition to invocation from cron, contrary
to my claim in my other message. I suspect that others are not
seeing any issues because most Java environments are persistent.
Anyone else scripting with Clojure using agents?

Thanks.

-Drew

[Timothy Pratley]

Hi Drew,

I've been trying to recreate your issue....
(defn forever[x] (while true (Thread/sleep 100) (print \-) (flush)))
(let [a (agent 0)] (send (agent 0) forever))
(println (Thread/activeCount) "threads active")
(shutdown-agents)
(println "I'm here")
;(System/exit 0)

Notably shutdown-agents is a non-blocking call, and without the exit
the program runs forever.
With the exit, for me the program terminates - but I believe this is
where the behavior which may vary...
something different occurring in your environment to mine - shutodown
hooks? JVM?
My only other stab in the dark is could your (flush) actually block
before System/exit is called?

Can you try my example and see if it hangs for you, and/or if there
some code you can paste that hangs?

Regards,
Tim.

[Eric Schoen]

Ha!  Almost nine years later, a Google search surfaced this thread in response to  "java jvm hang exit DestroyJavaVM thread." I have a Clojure based web crawler built around pegasus (and thus, both threadpool agents and clojure core async agents) that uses a command line interface to crawl in a subprocess for a few minutes at a time.  The subprocess was randomly stubbornly not exiting after I shut down pegasus, and now I understand why.  Adding a System/exit call made all the difference.  Thanks!