not quitting, (1537/3056/1.6.0_13-b03/Linux 2.6.18-4-amd64/amd64 x2)
Martin Dorey
javax.swing.TimerQueue.firstTimer == null, so I don't think there are any queued timers
terminator.TerminatorFrame has one instance, whose peer == null, which I think means it's not displayable
I see no deadlocks, indeed, no anything, in the backtraces.
There are no interesting-looking exceptions in the log. I haven't been running Terminator with this sort of command line before today. Exiting with failure's not something that often happens to me.
Why isn' t the vm quitting?
martind@whitewater:~$ cat /home/martind/.terminator/logs/terminator-32423.log
2009-05-29T11:24:47.455-0700 Terminator: Java 1.6.0_13 (VM 11.3-b02, runtime 1.6.0_13-b03)
2009-05-29T11:24:47.455-0700 Terminator: Linux 2.6.18-4-amd64/amd64 x2
2009-05-29T11:24:47.456-0700 Terminator: Revision 1537 (3056)
2009-05-29T11:24:47.456-0700 Terminator: Built 2009-05-29T09:26:36-07:00
2009-05-29T11:24:47.563-0700 Terminator: echo localhost:45476 > /home/martind/.terminator/terminator-server-port_0.0
2009-05-29T11:24:47.737-0700 Terminator: Created PtyProcess[pid=32460,fd=27,pty="/dev/pts/2"] and logging to /home/martind/.terminator/logs/2009-05-29T112447.701-0700-%2Fbin%2Fbash+-c+ssh+-t+-o+StrictHostKeyChecking%3Dno+-o+BatchMode%3Dyes+mercury2.us.dev.bluearc.com+%2Fhome%2Fmartind%2Ftmp%2Frunmercury-martind-mercury2-connect-script.txt
2009-05-29T12:52:22.303-0700 Terminator: Problem reading output from PtyProcess[pid=32460,fd=27,pty="/dev/pts/2"]
Associated exception:
java.io.IOException: read(27, buffer, 0, 8192) failed: Input/output error
at terminator.terminal.PtyProcess$PtyInputStream.read(PtyProcess.java:31)
at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:264)
at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:306)
at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:158)
at java.io.InputStreamReader.read(InputStreamReader.java:167)
at terminator.terminal.TerminalControl$ReaderRunnable.run(TerminalControl.java:150)
at java.lang.Thread.run(Thread.java:619)
2009-05-29T12:52:22.304-0700 Terminator: calling waitFor on PtyProcess[pid=32460,fd=27,pty="/dev/pts/2"]
2009-05-29T12:52:22.305-0700 Terminator: waitFor returned on PtyProcess[pid=32460,fd=-1,pty="/dev/pts/2",didExitNormally,exitValue=129]
martind@whitewater:~$
2009-05-29 13:38:28
Full thread dump Java HotSpot(TM) 64-Bit Server VM (11.3-b02 mixed mode):
"TimerQueue" daemon prio=10 tid=0x00002aaaf55ef400 nid=0x7eca in Object.wait() [0x000000004133b000..0x000000004133be40]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at javax.swing.TimerQueue.run(TimerQueue.java:236)
- locked <0x00002aaab3c1c6b0> (a javax.swing.TimerQueue)
at java.lang.Thread.run(Thread.java:619)
"TerminatorServer" daemon prio=10 tid=0x00002aaaf5000400 nid=0x7ec9 runnable [0x000000004123a000..0x000000004123aac0]
java.lang.Thread.State: RUNNABLE
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
- locked <0x00002aaab3947260> (a java.net.SocksSocketImpl)
at java.net.ServerSocket.implAccept(ServerSocket.java:453)
at java.net.ServerSocket.accept(ServerSocket.java:421)
at e.util.InAppServer$ConnectionAccepter.acceptConnections(InAppServer.java:145)
at e.util.InAppServer$ConnectionAccepter.run(InAppServer.java:138)
at java.lang.Thread.run(Thread.java:619)
"DestroyJavaVM" prio=10 tid=0x00002aaaf573c400 nid=0x7eb9 waiting on condition [0x0000000000000000..0x000000004022adb0]
java.lang.Thread.State: RUNNABLE
"AWT-EventQueue-0" prio=10 tid=0x00002aaaf4398800 nid=0x7ec8 in Object.wait() [0x0000000041139000..0x0000000041139b40]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at java.awt.EventQueue.getNextEvent(EventQueue.java:479)
- locked <0x00002aaab3c1c2b0> (a e.debug.EventDispatchThreadHangMonitor)
at java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:236)
at java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:184)
at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:174)
at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:169)
at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:161)
at java.awt.EventDispatchThread.run(EventDispatchThread.java:122)
"AWT-XAWT" daemon prio=10 tid=0x00002aaaf5707c00 nid=0x7ec6 runnable [0x0000000040f37000..0x0000000040f37c40]
java.lang.Thread.State: RUNNABLE
at sun.awt.X11.XToolkit.waitForEvents(Native Method)
at sun.awt.X11.XToolkit.run(XToolkit.java:548)
at sun.awt.X11.XToolkit.run(XToolkit.java:523)
at java.lang.Thread.run(Thread.java:619)
"Java2D Disposer" daemon prio=10 tid=0x00002aaaf4137c00 nid=0x7ec5 in Object.wait() [0x0000000040e36000..0x0000000040e36cc0]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00002aaab3c04758> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
- locked <0x00002aaab3c04758> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
at sun.java2d.Disposer.run(Disposer.java:125)
at java.lang.Thread.run(Thread.java:619)
"EventDispatchThreadHangMonitor" daemon prio=10 tid=0x00002aaaf4317800 nid=0x7ec4 in Object.wait() [0x0000000040d35000..0x0000000040d35d40]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.util.TimerThread.mainLoop(Timer.java:509)
- locked <0x00002aaab3c1be30> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:462)
"Low Memory Detector" daemon prio=10 tid=0x00002aaaf4122400 nid=0x7ec2 runnable [0x0000000000000000..0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"CompilerThread1" daemon prio=10 tid=0x00002aaaf411fc00 nid=0x7ec1 waiting on condition [0x0000000000000000..0x0000000040a312d0]
java.lang.Thread.State: RUNNABLE
"CompilerThread0" daemon prio=10 tid=0x00002aaaf4118c00 nid=0x7ec0 waiting on condition [0x0000000000000000..0x00000000409303d0]
java.lang.Thread.State: RUNNABLE
"Signal Dispatcher" daemon prio=10 tid=0x00002aaaf4117000 nid=0x7ebf waiting on condition [0x0000000000000000..0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Finalizer" daemon prio=10 tid=0x00002aaaf3ce5800 nid=0x7ebe in Object.wait() [0x000000004072f000..0x000000004072fc40]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00002aaab393e228> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
- locked <0x00002aaab393e228> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
"Reference Handler" daemon prio=10 tid=0x00002aaaf3ce3c00 nid=0x7ebd in Object.wait() [0x000000004062e000..0x000000004062ecc0]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00002aaab393e1a0> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:485)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
- locked <0x00002aaab393e1a0> (a java.lang.ref.Reference$Lock)
"VM Thread" prio=10 tid=0x00002aaaf3cde400 nid=0x7ebc runnable
"GC task thread#0 (ParallelGC)" prio=10 tid=0x000000004011e000 nid=0x7eba runnable
"GC task thread#1 (ParallelGC)" prio=10 tid=0x000000004011f800 nid=0x7ebb runnable
"VM Periodic Task Thread" prio=10 tid=0x00002aaaf4124800 nid=0x7ec3 waiting on condition
JNI global references: 877
Heap
PSYoungGen total 7872K, used 7675K [0x00002aaade2e0000, 0x00002aaadefd0000, 0x00002aaaf3830000)
eden space 7424K, 97% used [0x00002aaade2e0000,0x00002aaade9f5550,0x00002aaadea20000)
from space 448K, 94% used [0x00002aaadef60000,0x00002aaadefc9978,0x00002aaadefd0000)
to space 512K, 0% used [0x00002aaadeed0000,0x00002aaadeed0000,0x00002aaadef50000)
PSOldGen total 14656K, used 4334K [0x00002aaab3830000, 0x00002aaab4680000, 0x00002aaade2e0000)
object space 14656K, 29% used [0x00002aaab3830000,0x00002aaab3c6b838,0x00002aaab4680000)
PSPermGen total 35072K, used 18004K [0x00002aaaae430000, 0x00002aaab0670000, 0x00002aaab3830000)
object space 35072K, 51% used [0x00002aaaae430000,0x00002aaaaf5c5208,0x00002aaab0670000)
Elliott Hughes
On Fri, May 29, 2009 14:20, Martin Dorey wrote:
> Using jvisualvm (please do tell if there's a better way of manually
> simulating HungAwtExit's calls):
think only jconsole would make use of that. (if jvisualvm does, that would
be more worthwhile.)
> javax.swing.TimerQueue.firstTimer == null, so I don't think there are any
> queued timers terminator.TerminatorFrame has one instance, whose peer ==
> null, which I think means it's not displayable
>
> I see no deadlocks, indeed, no anything, in the backtraces.
>
>
> There are no interesting-looking exceptions in the log. I haven't been
> running Terminator with this sort of command line before today. Exiting
> with failure's not something that often happens to me.
>
> Why isn' t the vm quitting?
--elliott
Elliott Hughes, http://www.jessies.org/~enh/
Martin Dorey
> we could make it available via the JMX stuff, but i
> think only jconsole would make use of that
I've poked about in this process with jconsole without finding any way of invoking arbitrary methods, so I'm reading that sentence as meaning that source changes would be required. Either that or invocation changes to allow the debugger to attach, right? What a weird pain in the ass. jvisualvm seems to be using some kind of remote method invocation. This is my process and I am r00t anyway - why doesn't it give me access to the code? Perhaps it wouldn't be so disappointing but for having such good access to the heap.
> does jvisualvm say there are still non-daemon threads?
As opposed to the pre-jvisualvm backtraces below, which listed six, all of which I think are fine:
"DestroyJavaVM" prio=10 tid=0x00002aaaf573c400 nid=0x7eb9 waiting on condition [0x0000000000000000..0x000000004022adb0]
java.lang.Thread.State: RUNNABLE
"AWT-EventQueue-0" prio=10 tid=0x00002aaaf4398800 nid=0x7ec8 in Object.wait() [0x0000000041139000..0x0000000041139b40]
java.lang.Thread.State: WAITING (on object monitor)
"VM Thread" prio=10 tid=0x00002aaaf3cde400 nid=0x7ebc runnable
"GC task thread#0 (ParallelGC)" prio=10 tid=0x000000004011e000 nid=0x7eba runnable
"GC task thread#1 (ParallelGC)" prio=10 tid=0x000000004011f800 nid=0x7ebb runnable
"VM Periodic Task Thread" prio=10 tid=0x00002aaaf4124800 nid=0x7ec3 waiting on condition
jvisualvm says "15 live threads, 13 daemons"
I count 16 daemons in its thread dump, 4 or 5 of which I think jvisualvm or jconsole caused to be started. I'm not sure what it means by a "live" thread but the lack of any java.lang.Thread.State for 4 of the above non-daemons makes me think that DestroyJavaVM and AWT-EventQueue-0 are probably the 2 counted as "live" but not "daemons".
I was also interested to see jvisualvm mention this:
JNI global references: 2685
So far I've successfully fought the tidy-minded impulse to kill off the window in question, so the process is still available for poking about it, if you have any further ideas.
Elliott Hughes
>> we could make it available via the JMX stuff, but i think only jconsole
>> would make use of that
>
> I've poked about in this process with jconsole without finding any way
> of invoking arbitrary methods, so I'm reading that sentence as meaning that
> source changes would be required. Either that or invocation changes to
> allow the debugger to attach, right? What a weird pain in the ass.
> jvisualvm seems to be using some kind of remote method invocation. This
> is my process and I am r00t anyway - why doesn't it give me access to the
> code? Perhaps it wouldn't be so disappointing but for having such good
> access to the heap.
time i used it), but jdb can use the same (?) serviceability agent
mechanism to attach to a running JVM without prior arrangement:
jdb -connect sun.jvm.hotspot.jdi.SAPIDAttachingConnector:pid=<pid>
>> does jvisualvm say there are still non-daemon threads?
>
> As opposed to the pre-jvisualvm backtraces below, which listed six, all
> of which I think are fine:
>
> "DestroyJavaVM" prio=10 tid=0x00002aaaf573c400 nid=0x7eb9 waiting on
> condition [0x0000000000000000..0x000000004022adb0] java.lang.Thread.State:
> RUNNABLE
>
>
> "AWT-EventQueue-0" prio=10 tid=0x00002aaaf4398800 nid=0x7ec8 in
> Object.wait() [0x0000000041139000..0x0000000041139b40]
> java.lang.Thread.State: WAITING (on object monitor)
>
>
> "VM Thread" prio=10 tid=0x00002aaaf3cde400 nid=0x7ebc runnable
>
>
> "GC task thread#0 (ParallelGC)" prio=10 tid=0x000000004011e000
> nid=0x7eba runnable
>
> "GC task thread#1 (ParallelGC)" prio=10 tid=0x000000004011f800
> nid=0x7ebb runnable
>
> "VM Periodic Task Thread" prio=10 tid=0x00002aaaf4124800 nid=0x7ec3
> waiting on condition
>
> jvisualvm says "15 live threads, 13 daemons"
>
> I count 16 daemons in its thread dump, 4 or 5 of which I think jvisualvm
> or jconsole caused to be started. I'm not sure what it means by a "live"
> thread but the lack of any java.lang.Thread.State for 4 of the above
> non-daemons makes me think that DestroyJavaVM and AWT-EventQueue-0 are
> probably the 2 counted as "live" but not "daemons".
>
> I was also interested to see jvisualvm mention this:
>
> JNI global references: 2685
get collected. i don't think we leak them. (i don't think we manipulate
them at all.)
> So far I've successfully fought the tidy-minded impulse to kill off the
> window in question, so the process is still available for poking about it,
> if you have any further ideas.
exit while there's a window on screen.
--elliott
Martin Dorey
> wasn't?
I'll try the jdb incantation in a bit, ta.
Martin Dorey
Perhaps the reason SAPIDAttachingConnector isn't mentioned on what I think is the current page on attaching jdb for Linux (http://java.sun.com/javase/6/docs/technotes/tools/solaris/jdb.html, despite the "solaris" in the url) is because, sadly, it doesn't seem to let you do anything useful.
EventDispatchThreadHangMonitor[1] suspend
Command 'suspend' is not supported on a read-only VM connection
EventDispatchThreadHangMonitor[1]
http://java.sun.com/javase/6/docs/technotes/tools/solaris/jdb.html suggested that I'd have to do that in order to be able to print anything:
> If the current thread is suspended (either through an event such as a breakpoint or through the suspend command),
> local variables and fields can be displayed with the print and dump commands.
And indeed I can't print anything (this is essentially the same message I get for "print badger"):
EventDispatchThreadHangMonitor[1] print e.debug.EventDispatchThreadHangMonitor.hangCount
com.sun.tools.example.debug.expr.ParseException: Name unknown: e.debug.EventDispatchThreadHangMonitor.hangCount
e.debug.EventDispatchThreadHangMonitor.hangCount = null
Requested stack frame is no longer active: 0
EventDispatchThreadHangMonitor[1]
On experimenting with:
args << "-agentlib:jdwp=server=y,transport=dt_socket,suspend=n,address=#{10000 + Process.pid()}"
I find that HungAwtExit isn't there to invoke. I've added a hack to TerminatorServer to let me ask for it to be invoked, but I foresee having to grovel with jvisualvm for the secret.
-----Original Message-----
From: terminat...@googlegroups.com [mailto:terminat...@googlegroups.com] On Behalf Of Martin Dorey
Sent: Sunday, May 31, 2009 11:42