Ouch! find-objects trashes the server

81 views

Skip to first unread message

Anthony Shortland

unread,

Oct 14, 2010, 2:51:47 PM10/14/10

to ControlTier Accounting

I've run into this problem repeatedly since the latter releases of 3.4.

Running an innocuous find-objects command such as this:

[dnbiadm@gplnxv8 objects]$ ctl -p Dnbgp -t ProjectBuilder -o dnbgp -c find-objects -- -type DnbgpV3IntegratedBuilder -name '.*' -purge
Deleting: /dnbusr1/dnbiadm/ctier/ctl/depots/Dnbgp/deployments/ProjectBuilder/dnbgp/var/sessions/find-objects/find.session

... goes away for minutes and results in this:

2010-10-14 11:27:26.584::WARN: Problem scavenging sessions
java.lang.OutOfMemoryError: Java heap space
   at java.util.HashMap.newValueIterator(HashMap.java:892)
   at java.util.HashMap$Values.iterator(HashMap.java:955)
   at org.mortbay.jetty.servlet.HashSessionManager.scavenge(HashSessionManager.java:249)
   at org.mortbay.jetty.servlet.HashSessionManager.access$000(HashSessionManager.java:44)
   at org.mortbay.jetty.servlet.HashSessionManager$2.run(HashSessionManager.java:213)
   at java.util.TimerThread.mainLoop(Timer.java:512)
   at java.util.TimerThread.run(Timer.java:462)
Exception in thread "PatternStage-40434" java.lang.OutOfMemoryError: Java heap space
   at java.util.HashMap.addEntry(HashMap.java:797)
   at java.util.HashMap.put(HashMap.java:431)
   at java.util.HashSet.add(HashSet.java:194)
   at com.hp.hpl.jena.reasoner.rulesys.impl.Generator.notifyBlockedOn(Generator.java:206)
   at com.hp.hpl.jena.reasoner.rulesys.impl.LPInterpreter.run(LPInterpreter.java:358)
   at com.hp.hpl.jena.reasoner.rulesys.impl.LPInterpreter.next(LPInterpreter.java:187)
   at com.hp.hpl.jena.reasoner.rulesys.impl.Generator.pump(Generator.java:239)
   at com.hp.hpl.jena.reasoner.rulesys.impl.Generator.pump(Generator.java:226)
   at com.hp.hpl.jena.reasoner.rulesys.impl.LPBRuleEngine.pump(LPBRuleEngine.java:308)
   at com.hp.hpl.jena.reasoner.rulesys.impl.LPTopGoalIterator.moveForward(LPTopGoalIterator.java:79)
   at com.hp.hpl.jena.reasoner.rulesys.impl.LPTopGoalIterator.next(LPTopGoalIterator.java:186)
   at com.hp.hpl.jena.util.iterator.WrappedIterator.next(WrappedIterator.java:68)
   at com.hp.hpl.jena.util.iterator.WrappedIterator.next(WrappedIterator.java:68)
   at com.hp.hpl.jena.util.iterator.UniqueExtendedIterator.nextIfNew(UniqueExtendedIterator.java:61)
   at com.hp.hpl.jena.util.iterator.UniqueExtendedIterator.hasNext(UniqueExtendedIterator.java:69)
   at com.hp.hpl.jena.util.iterator.WrappedIterator.hasNext(WrappedIterator.java:64)
   at com.hp.hpl.jena.util.iterator.FilterIterator.hasNext(FilterIterator.java:43)
   at com.hp.hpl.jena.graph.query.QueryTriple$SimpleApplyer.applyToTriples(QueryTriple.java:146)
   at com.hp.hpl.jena.graph.query.StageElement$FindTriples.run(StageElement.java:50)
   at com.hp.hpl.jena.graph.query.StageElement$RunValuatorSet.run(StageElement.java:67)
   at com.hp.hpl.jena.graph.query.QueryTriple$SimpleApplyer.applyToTriples(QueryTriple.java:148)
   at com.hp.hpl.jena.graph.query.StageElement$FindTriples.run(StageElement.java:50)
   at com.hp.hpl.jena.graph.query.PatternStageBase.run(PatternStageBase.java:43)
   at com.hp.hpl.jena.graph.query.PatternStageBase$1.run(PatternStageBase.java:53)

... after which the server needs to be re-started.

The standard Jetty JAVA_OPTIONS are:

[dnbiadm@gplnxv8 jetty-6.1.14]$ echo $JAVA_OPTIONS
-XX:MaxPermSize=256m -Xmx1024m -Xms256m -Djobcenter.config.location=/dnbusr1/dnbiadm/ctier/jobcenter/jobcenter-config.properties -Dreportcenter.config.location=/dnbusr1/dnbiadm/ctier/reportcenter/reportcenter-config.properties

... which never seem to cause a problem otherwise, so I'm inclined to believe that something in Jena is spinning out of control.

Has anyone else seen this? Any tips on avoiding the problem?

Anthony.

Noah Campbell

unread,

Oct 14, 2010, 3:00:58 PM10/14/10

to contr...@googlegroups.com

Does increasing the -Xmx to something greater then 1gb fix the problem?

--
You received this message because you are subscribed to the Google Groups "ControlTier" group.
To post to this group, send email to contr...@googlegroups.com
To unsubscribe from this group, send email to controltier...@googlegroups.com
For more options, visit this group at http://groups.google.com/group/controltier?hl=en
http://wiki.controltier.org

Anthony Shortland

unread,

Oct 14, 2010, 3:14:20 PM10/14/10

to ControlTier Accounting

I should note that after restarting the server, the command (without the -purge) runs cleanly a a number of times:

[dnbiadm@gplnxv8 objects]$ ctl -p Dnbgp -t ProjectBuilder -o dnbgp -c find-objects -- -type DnbgpV3IntegratedBuilder -name '.*'

Deleting: /dnbusr1/dnbiadm/ctier/ctl/depots/Dnbgp/deployments/ProjectBuilder/dnbgp/var/sessions/find-objects/find.session
.
.
.
session data saved: /dnbusr1/dnbiadm/ctier/ctl/depots/Dnbgp/deployments/ProjectBuilder/dnbgp/var/sessions/find-objects/find.session

... and then trashes the server again (on the fourth go)!

Sometimes the server just sits and spins:

[dnbiadm@gplnxv8 ~]$ ps -fp 28135
UID PID PPID C STIME TTY TIME CMD
dnbiadm 28135 1 94 11:52 pts/6 00:16:57 /usr/java/jdk1.5.0_16/bin/java -XX:MaxPermSize=256m -Xmx1024m -Xms256m -Djobcenter.config.location=/dnbusr1/dnbi

Here's a thread dump:

Full thread dump Java HotSpot(TM) 64-Bit Server VM (1.5.0_16-b02 mixed mode):
"Thread-37" prio=1 tid=0x0000002af8d8f1a0 nid=0x491 runnable [0x0000000041263000..0x0000000041263b30]
   at java.net.SocketInputStream.socketRead0(Native Method)
   at java.net.SocketInputStream.read(SocketInputStream.java:129)
   at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
   at java.io.BufferedInputStream.read(BufferedInputStream.java:235)
   - locked <0x0000002ad727b560> (a java.io.BufferedInputStream)
   at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2196)
   at java.io.ObjectInputStream$BlockDataInputStream.peek(ObjectInputStream.java:2486)
   at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2496)
   at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1273)
   at java.io.ObjectInputStream.readObject(ObjectInputStream.java:348)
   at org.apache.log4j.net.SocketNode.run(SocketNode.java:77)
   at java.lang.Thread.run(Thread.java:595)
"PatternStage-2253" prio=1 tid=0x0000002af31d6ff0 nid=0x7729 runnable [0x0000000044a9b000..0x0000000044a9bdb0]
   at com.hp.hpl.jena.reasoner.rulesys.impl.LPBRuleEngine.generatorFor(LPBRuleEngine.java:267)
   - waiting to lock <0x0000002ab8ae52c8> (a com.hp.hpl.jena.reasoner.rulesys.impl.LPBRuleEngine)
   at com.hp.hpl.jena.reasoner.rulesys.impl.ConsumerChoicePointFrame.init(ConsumerChoicePointFrame.java:71)
   at com.hp.hpl.jena.reasoner.rulesys.impl.ConsumerChoicePointFrame.<init>(ConsumerChoicePointFrame.java:60)
   at com.hp.hpl.jena.reasoner.rulesys.impl.LPInterpreter.setupTabledCall(LPInterpreter.java:671)
   at com.hp.hpl.jena.reasoner.rulesys.impl.LPInterpreter.run(LPInterpreter.java:565)

   at com.hp.hpl.jena.reasoner.rulesys.impl.LPInterpreter.next(LPInterpreter.java:187)
   at com.hp.hpl.jena.reasoner.rulesys.impl.Generator.pump(Generator.java:239)
   at com.hp.hpl.jena.reasoner.rulesys.impl.Generator.pump(Generator.java:226)
   at com.hp.hpl.jena.reasoner.rulesys.impl.LPBRuleEngine.pump(LPBRuleEngine.java:308)

   - locked <0x0000002ab8ae52c8> (a com.hp.hpl.jena.reasoner.rulesys.impl.LPBRuleEngine)
   at com.hp.hpl.jena.reasoner.rulesys.impl.LPTopGoalIterator.moveForward(LPTopGoalIterator.java:79)
   - locked <0x0000002ab8ae52c8> (a com.hp.hpl.jena.reasoner.rulesys.impl.LPBRuleEngine)
   - locked <0x0000002addbbc818> (a com.hp.hpl.jena.reasoner.rulesys.impl.LPTopGoalIterator)

   at com.hp.hpl.jena.reasoner.rulesys.impl.LPTopGoalIterator.next(LPTopGoalIterator.java:186)
   at com.hp.hpl.jena.util.iterator.WrappedIterator.next(WrappedIterator.java:68)
   at com.hp.hpl.jena.util.iterator.WrappedIterator.next(WrappedIterator.java:68)
   at com.hp.hpl.jena.util.iterator.UniqueExtendedIterator.nextIfNew(UniqueExtendedIterator.java:61)
   at com.hp.hpl.jena.util.iterator.UniqueExtendedIterator.hasNext(UniqueExtendedIterator.java:69)
   at com.hp.hpl.jena.util.iterator.WrappedIterator.hasNext(WrappedIterator.java:64)
   at com.hp.hpl.jena.util.iterator.FilterIterator.hasNext(FilterIterator.java:43)

- locked <0x0000002addbbca68> (a com.hp.hpl.jena.util.iterator.FilterDropIterator)

   at com.hp.hpl.jena.graph.query.QueryTriple$SimpleApplyer.applyToTriples(QueryTriple.java:146)
   at com.hp.hpl.jena.graph.query.StageElement$FindTriples.run(StageElement.java:50)
   at com.hp.hpl.jena.graph.query.StageElement$RunValuatorSet.run(StageElement.java:67)
   at com.hp.hpl.jena.graph.query.QueryTriple$SimpleApplyer.applyToTriples(QueryTriple.java:148)
   at com.hp.hpl.jena.graph.query.StageElement$FindTriples.run(StageElement.java:50)
   at com.hp.hpl.jena.graph.query.PatternStageBase.run(PatternStageBase.java:43)
   at com.hp.hpl.jena.graph.query.PatternStageBase$1.run(PatternStageBase.java:53)

"btpool0-12" prio=1 tid=0x0000002af5b3fe70 nid=0x7657 in Object.wait() [0x0000000043f8f000..0x0000000043f90e30]
   at java.lang.Object.wait(Native Method)
   - waiting on <0x0000002ab8ae49f0> (a EDU.oswego.cs.dl.util.concurrent.BoundedBuffer)
   at java.lang.Object.wait(Object.java:474)
   at EDU.oswego.cs.dl.util.concurrent.BoundedBuffer.take(Unknown Source)
   - locked <0x0000002ab8ae49f0> (a EDU.oswego.cs.dl.util.concurrent.BoundedBuffer)
   at com.hp.hpl.jena.graph.query.BufferPipe.fetch(BufferPipe.java:43)
   at com.hp.hpl.jena.graph.query.BufferPipe.hasNext(BufferPipe.java:69)
   at com.hp.hpl.jena.graph.query.SimpleQueryEngine$1.hasNext(SimpleQueryEngine.java:59)
   at com.hp.hpl.jena.rdql.QueryEngine$ResultsIterator.hasNext(QueryEngine.java:167)
   at com.hp.hpl.jena.rdql.QueryResultsStream.hasNext(QueryResultsStream.java:58)
   at org.opendepo.services.query.QueryImpl.processResult(QueryImpl.java:197)
   at org.opendepo.services.query.QueryEngineImpl.processQuery(QueryEngineImpl.java:131)
   at com.networkgps.itnav.query.Access.getQueryResults(Access.java:111)
   at com.networkgps.itnav.query.Access.processQuery(Access.java:92)
   at com.networkgps.itnav.query.ColonyQuery.go(ColonyQuery.java:71)
   at com.networkgps.itnav.context.ContextProcessBeanBase.execute(ContextProcessBeanBase.java:101)
   at org.apache.commons.scaffold.util.ProcessBeanBase.execute(Unknown Source)
   at org.apache.struts.scaffold.ProcessAction.executeLogic(Unknown Source)
   at com.networkgps.itnav.context.BaseContextProcessAction.executeLogic(BaseContextProcessAction.java:283)
   at org.apache.struts.scaffold.BaseHelperAction.executeLogic(Unknown Source)
   at org.apache.struts.scaffold.BaseAction.execute(Unknown Source)
   at com.networkgps.itnav.context.BaseContextProcessAction.execute(BaseContextProcessAction.java:229)
   at com.networkgps.itnav.context.BaseContextProcessAction.execute(BaseContextProcessAction.java:267)
   at org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:484)
   at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:274)
   at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1482)
   at org.apache.struts.action.ActionServlet.doPost(ActionServlet.java:525)
   at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
   at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
   at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502)
   at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:363)
   at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
   at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
   at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
   at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
   at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
   at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
   at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
   at org.mortbay.jetty.Server.handle(Server.java:324)
   at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
   at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:879)
   at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:741)
   at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:213)
   at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403)
   at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
   at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:451)
"btpool0-11" prio=1 tid=0x0000002af87a8f80 nid=0x75ee in Object.wait() [0x0000000044091000..0x0000000044091db0]
   at java.lang.Object.wait(Native Method)
   - waiting on <0x0000002ab87850f0> (a org.mortbay.thread.BoundedThreadPool$PoolThread)
   at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:491)
   - locked <0x0000002ab87850f0> (a org.mortbay.thread.BoundedThreadPool$PoolThread)
"btpool0-10" prio=1 tid=0x0000002af57b20d0 nid=0x75ed in Object.wait() [0x0000000044798000..0x0000000044798d30]
   at java.lang.Object.wait(Native Method)
   - waiting on <0x0000002ab8785050> (a org.mortbay.thread.BoundedThreadPool$PoolThread)
   at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:491)
   - locked <0x0000002ab8785050> (a org.mortbay.thread.BoundedThreadPool$PoolThread)
"DestroyJavaVM" prio=1 tid=0x0000002af2c62400 nid=0x6de7 waiting on condition [0x0000000000000000..0x0000007fbfffcd20]
"ContinuationBayeux-0" daemon prio=1 tid=0x0000002af12c5c90 nid=0x6e27 in Object.wait() [0x0000000043e8f000..0x0000000043e8fd30]
   at java.lang.Object.wait(Native Method)
   - waiting on <0x0000002ab162cc08> (a java.util.TaskQueue)
   at java.util.TimerThread.mainLoop(Timer.java:509)
   - locked <0x0000002ab162cc08> (a java.util.TaskQueue)
   at java.util.TimerThread.run(Timer.java:462)
"Timer-11" daemon prio=1 tid=0x0000002af365f760 nid=0x6e26 in Object.wait() [0x0000000043d8e000..0x0000000043d8ecb0]
   at java.lang.Object.wait(Native Method)
   - waiting on <0x0000002ab1ae94e8> (a java.util.TaskQueue)
   at java.util.TimerThread.mainLoop(Timer.java:509)
   - locked <0x0000002ab1ae94e8> (a java.util.TaskQueue)
   at java.util.TimerThread.run(Timer.java:462)
"Timer-10" daemon prio=1 tid=0x0000002af71b43b0 nid=0x6e25 in Object.wait() [0x0000000043c8d000..0x0000000043c8dc30]
   at java.lang.Object.wait(Native Method)
   - waiting on <0x0000002ab1c85f40> (a java.util.TaskQueue)
   at java.util.TimerThread.mainLoop(Timer.java:509)
   - locked <0x0000002ab1c85f40> (a java.util.TaskQueue)
   at java.util.TimerThread.run(Timer.java:462)
"Timer-9" daemon prio=1 tid=0x0000002af74e3a60 nid=0x6e24 in Object.wait() [0x0000000043b8c000..0x0000000043b8cbb0]
   at java.lang.Object.wait(Native Method)
   - waiting on <0x0000002ab1ae8cc8> (a java.util.TaskQueue)
   at java.util.TimerThread.mainLoop(Timer.java:509)
   - locked <0x0000002ab1ae8cc8> (a java.util.TaskQueue)
   at java.util.TimerThread.run(Timer.java:462)
"Timer-8" daemon prio=1 tid=0x0000002af2dff5a0 nid=0x6e23 in Object.wait() [0x0000000043a8b000..0x0000000043a8bb30]
   at java.lang.Object.wait(Native Method)
   - waiting on <0x0000002ab1b09358> (a java.util.TaskQueue)
   at java.util.TimerThread.mainLoop(Timer.java:509)
   - locked <0x0000002ab1b09358> (a java.util.TaskQueue)
   at java.util.TimerThread.run(Timer.java:462)
"Timer-7" daemon prio=1 tid=0x0000002af31f3ee0 nid=0x6e21 in Object.wait() [0x000000004398a000..0x000000004398ae30]
   at java.lang.Object.wait(Native Method)
   - waiting on <0x0000002ab09aff38> (a java.util.TaskQueue)
   at java.util.TimerThread.mainLoop(Timer.java:509)
   - locked <0x0000002ab09aff38> (a java.util.TaskQueue)
   at java.util.TimerThread.run(Timer.java:462)
"Thread-23" prio=1 tid=0x0000002af59cc760 nid=0x6e20 runnable [0x0000000043889000..0x0000000043889db0]
   at java.net.SocketInputStream.socketRead0(Native Method)
   at java.net.SocketInputStream.read(SocketInputStream.java:129)
   at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
   at java.io.BufferedInputStream.read(BufferedInputStream.java:235)
   - locked <0x0000002ab0a1dec8> (a java.io.BufferedInputStream)
   at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2196)
   at java.io.ObjectInputStream$BlockDataInputStream.peek(ObjectInputStream.java:2486)
   at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2496)
   at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1273)
   at java.io.ObjectInputStream.readObject(ObjectInputStream.java:348)
   at org.apache.log4j.net.SocketNode.run(SocketNode.java:77)
   at java.lang.Thread.run(Thread.java:595)
"quartzScheduler_QuartzSchedulerThread" prio=1 tid=0x0000002af5fb2a00 nid=0x6e1f in Object.wait() [0x0000000043788000..0x0000000043788d30]
   at java.lang.Object.wait(Native Method)
   - waiting on <0x0000002ab1685770> (a java.lang.Object)
   at org.quartz.core.QuartzSchedulerThread.run(QuartzSchedulerThread.java:433)
   - locked <0x0000002ab1685770> (a java.lang.Object)
"quartzScheduler_Worker-10" prio=1 tid=0x0000002afcc0ec80 nid=0x6e1e in Object.wait() [0x0000000043687000..0x0000000043687cb0]
   at java.lang.Object.wait(Native Method)
   - waiting on <0x0000002ab1686ce8> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)
   at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:519)
   - locked <0x0000002ab1686ce8> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)
"quartzScheduler_Worker-9" prio=1 tid=0x0000002af3977c20 nid=0x6e1d in Object.wait() [0x0000000043586000..0x0000000043586c30]
   at java.lang.Object.wait(Native Method)
   - waiting on <0x0000002ab16ab9d8> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)
   at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:519)
   - locked <0x0000002ab16ab9d8> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)
"quartzScheduler_Worker-8" prio=1 tid=0x0000002af8dfb050 nid=0x6e1c in Object.wait() [0x0000000043485000..0x0000000043485bb0]
   at java.lang.Object.wait(Native Method)
   - waiting on <0x0000002ab13b1e80> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)
   at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:519)
   - locked <0x0000002ab13b1e80> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)
"quartzScheduler_Worker-7" prio=1 tid=0x0000002af3076120 nid=0x6e1b in Object.wait() [0x0000000043384000..0x0000000043384b30]
   at java.lang.Object.wait(Native Method)
   - waiting on <0x0000002ab13b1dd0> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)
   at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:519)
   - locked <0x0000002ab13b1dd0> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)
"quartzScheduler_Worker-6" prio=1 tid=0x0000002af59ecaf0 nid=0x6e1a in Object.wait() [0x0000000043283000..0x0000000043283ab0]
   at java.lang.Object.wait(Native Method)
   - waiting on <0x0000002ab13b1d20> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)
   at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:519)
   - locked <0x0000002ab13b1d20> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)
"quartzScheduler_Worker-5" prio=1 tid=0x0000002af68b1f30 nid=0x6e19 in Object.wait() [0x0000000043182000..0x0000000043182e30]
   at java.lang.Object.wait(Native Method)
   - waiting on <0x0000002ab13b1c70> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)
   at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:519)
   - locked <0x0000002ab13b1c70> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)
"quartzScheduler_Worker-4" prio=1 tid=0x0000002af86f7420 nid=0x6e18 in Object.wait() [0x0000000043081000..0x0000000043081db0]
   at java.lang.Object.wait(Native Method)
   - waiting on <0x0000002ab13b1bc0> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)
   at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:519)
   - locked <0x0000002ab13b1bc0> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)
"quartzScheduler_Worker-3" prio=1 tid=0x0000002af300e8b0 nid=0x6e17 in Object.wait() [0x0000000042f80000..0x0000000042f80d30]
   at java.lang.Object.wait(Native Method)
   - waiting on <0x0000002ab13b1b10> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)
   at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:519)
   - locked <0x0000002ab13b1b10> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)
"quartzScheduler_Worker-2" prio=1 tid=0x0000002af3977310 nid=0x6e16 in Object.wait() [0x0000000042e7f000..0x0000000042e7fcb0]
   at java.lang.Object.wait(Native Method)
   - waiting on <0x0000002ab13b1a60> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)
   at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:519)
   - locked <0x0000002ab13b1a60> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)
"quartzScheduler_Worker-1" prio=1 tid=0x0000002af3977060 nid=0x6e15 in Object.wait() [0x0000000042d7e000..0x0000000042d7ec30]
   at java.lang.Object.wait(Native Method)
   - waiting on <0x0000002ab13b19b0> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)
   at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:519)
   - locked <0x0000002ab13b19b0> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)
"HSQLDB Timer @6de0e650" daemon prio=1 tid=0x0000002af5f21730 nid=0x6e14 in Object.wait() [0x0000000042c7d000..0x0000000042c7dbb0]
   at java.lang.Object.wait(Native Method)
   - waiting on <0x0000002aadd03510> (a org.hsqldb.lib.HsqlTimer)
   at org.hsqldb.lib.HsqlTimer.nextTask(Unknown Source)
   - locked <0x0000002aadd03510> (a org.hsqldb.lib.HsqlTimer)
   at org.hsqldb.lib.HsqlTimer$TaskRunner.run(Unknown Source)
   at java.lang.Thread.run(Thread.java:595)
"Timer-6" daemon prio=1 tid=0x0000002af71332f0 nid=0x6e13 in Object.wait() [0x0000000042b7c000..0x0000000042b7cb30]
   at java.lang.Object.wait(Native Method)
   - waiting on <0x0000002aadd036f8> (a java.util.TaskQueue)
   at java.util.TimerThread.mainLoop(Timer.java:509)
   - locked <0x0000002aadd036f8> (a java.util.TaskQueue)
   at java.util.TimerThread.run(Timer.java:462)
"Thread-20" prio=1 tid=0x0000002af2d3b380 nid=0x6e12 runnable [0x0000000042a7b000..0x0000000042a7bab0]
   at java.net.PlainSocketImpl.socketAccept(Native Method)
   at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
   - locked <0x0000002aad6e0b70> (a java.net.SocksSocketImpl)
   at java.net.ServerSocket.implAccept(ServerSocket.java:450)
   at java.net.ServerSocket.accept(ServerSocket.java:421)
   at com.controltier.reporter.services.InputServer$worker.run(InputServer.java:81)
   at java.lang.Thread.run(Thread.java:595)
"Store org.hibernate.cache.StandardQueryCache Spool Thread" daemon prio=1 tid=0x0000002af1838420 nid=0x6e11 waiting on condition [0x000000004297a000..0x000000004297ae30]
   at java.lang.Thread.sleep(Native Method)
   at net.sf.ehcache.store.DiskStore.spoolAndExpiryThreadMain(DiskStore.java:589)
   at net.sf.ehcache.store.DiskStore.access$800(DiskStore.java:65)
   at net.sf.ehcache.store.DiskStore$SpoolAndExpiryThread.run(DiskStore.java:1075)
"Store org.hibernate.cache.UpdateTimestampsCache Spool Thread" daemon prio=1 tid=0x0000002af1838b50 nid=0x6e10 waiting on condition [0x0000000042879000..0x0000000042879db0]
   at java.lang.Thread.sleep(Native Method)
   at net.sf.ehcache.store.DiskStore.spoolAndExpiryThreadMain(DiskStore.java:589)
   at net.sf.ehcache.store.DiskStore.access$800(DiskStore.java:65)
   at net.sf.ehcache.store.DiskStore$SpoolAndExpiryThread.run(DiskStore.java:1075)
"HSQLDB Timer @5058431c" daemon prio=1 tid=0x0000002af5772fa0 nid=0x6e0e in Object.wait() [0x0000000042778000..0x0000000042778d30]
   at java.lang.Object.wait(Native Method)
   - waiting on <0x0000002aad2b0bb8> (a org.hsqldb.lib.HsqlTimer)
   at java.lang.Object.wait(Object.java:474)
   at org.hsqldb.lib.HsqlTimer.nextTask(Unknown Source)
   - locked <0x0000002aad2b0bb8> (a org.hsqldb.lib.HsqlTimer)
   at org.hsqldb.lib.HsqlTimer$TaskRunner.run(Unknown Source)
   at java.lang.Thread.run(Thread.java:595)
"Timer-5" daemon prio=1 tid=0x0000002af5775770 nid=0x6e0d in Object.wait() [0x0000000042677000..0x0000000042677cb0]
   at java.lang.Object.wait(Native Method)
   - waiting on <0x0000002aad6e0ca0> (a java.util.TaskQueue)
   at java.util.TimerThread.mainLoop(Timer.java:509)
   - locked <0x0000002aad6e0ca0> (a java.util.TaskQueue)
   at java.util.TimerThread.run(Timer.java:462)
"RMI TCP Accept-1099" daemon prio=1 tid=0x0000002af3716730 nid=0x6e0c runnable [0x0000000042576000..0x0000000042576c30]
   at java.net.PlainSocketImpl.socketAccept(Native Method)
   at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
   - locked <0x0000002aad6e0110> (a java.net.SocksSocketImpl)
   at java.net.ServerSocket.implAccept(ServerSocket.java:450)
   at java.net.ServerSocket.accept(ServerSocket.java:421)
   at sun.rmi.transport.tcp.TCPTransport.run(TCPTransport.java:340)
   at java.lang.Thread.run(Thread.java:595)
"GC Daemon" daemon prio=1 tid=0x0000002af2c21860 nid=0x6e0b in Object.wait() [0x0000000042475000..0x0000000042475bb0]
   at java.lang.Object.wait(Native Method)
   - waiting on <0x0000002aac538528> (a sun.misc.GC$LatencyLock)
   at sun.misc.GC$Daemon.run(GC.java:100)
   - locked <0x0000002aac538528> (a sun.misc.GC$LatencyLock)
"RMI Reaper" prio=1 tid=0x0000002af2c210f0 nid=0x6e0a in Object.wait() [0x0000000042374000..0x0000000042374b30]
   at java.lang.Object.wait(Native Method)
   - waiting on <0x0000002aac538728> (a java.lang.ref.ReferenceQueue$Lock)
   at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:120)
   - locked <0x0000002aac538728> (a java.lang.ref.ReferenceQueue$Lock)
   at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:136)
   at sun.rmi.transport.ObjectTable$Reaper.run(ObjectTable.java:336)
   at java.lang.Thread.run(Thread.java:595)
"Timer-4" daemon prio=1 tid=0x0000002af2c21e90 nid=0x6e09 in Object.wait() [0x0000000042273000..0x0000000042273ab0]
   at java.lang.Object.wait(Native Method)
   - waiting on <0x0000002aac4a0cf0> (a java.util.TaskQueue)
   at java.lang.Object.wait(Object.java:474)
   at java.util.TimerThread.mainLoop(Timer.java:483)
   - locked <0x0000002aac4a0cf0> (a java.util.TaskQueue)
   at java.util.TimerThread.run(Timer.java:462)
"RMI TCP Accept-0" daemon prio=1 tid=0x0000002af398b960 nid=0x6e08 runnable [0x0000000042172000..0x0000000042172e30]
   at java.net.PlainSocketImpl.socketAccept(Native Method)
   at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
   - locked <0x0000002aac538980> (a java.net.SocksSocketImpl)
   at java.net.ServerSocket.implAccept(ServerSocket.java:450)
   at java.net.ServerSocket.accept(ServerSocket.java:421)
   at sun.rmi.transport.tcp.TCPTransport.run(TCPTransport.java:340)
   at java.lang.Thread.run(Thread.java:595)
"ObservationManager" daemon prio=1 tid=0x0000002af340af30 nid=0x6e07 in Object.wait() [0x0000000042071000..0x0000000042071db0]
   at java.lang.Object.wait(Native Method)
   - waiting on <0x0000002aac4da508> (a org.apache.commons.collections.buffer.BlockingBuffer)
   at java.lang.Object.wait(Object.java:474)
   at org.apache.commons.collections.buffer.BlockingBuffer.remove(BlockingBuffer.java:107)
   - locked <0x0000002aac4da508> (a org.apache.commons.collections.buffer.BlockingBuffer)
   at org.apache.jackrabbit.core.observation.ObservationDispatcher.run(ObservationDispatcher.java:138)
   at java.lang.Thread.run(Thread.java:595)
"derby.rawStoreDaemon" daemon prio=1 tid=0x0000002af340c110 nid=0x6e06 in Object.wait() [0x0000000041f70000..0x0000000041f70d30]
   at java.lang.Object.wait(Native Method)
   - waiting on <0x0000002aac4e6a90> (a org.apache.derby.impl.services.daemon.BasicDaemon)
   at org.apache.derby.impl.services.daemon.BasicDaemon.rest(Unknown Source)
   - locked <0x0000002aac4e6a90> (a org.apache.derby.impl.services.daemon.BasicDaemon)
   at org.apache.derby.impl.services.daemon.BasicDaemon.run(Unknown Source)
   at java.lang.Thread.run(Thread.java:595)
"derby.rawStoreDaemon" daemon prio=1 tid=0x0000002af398c770 nid=0x6e05 in Object.wait() [0x0000000041e6f000..0x0000000041e6fcb0]
   at java.lang.Object.wait(Native Method)
   - waiting on <0x0000002aac539990> (a org.apache.derby.impl.services.daemon.BasicDaemon)
   at org.apache.derby.impl.services.daemon.BasicDaemon.rest(Unknown Source)
   - locked <0x0000002aac539990> (a org.apache.derby.impl.services.daemon.BasicDaemon)
   at org.apache.derby.impl.services.daemon.BasicDaemon.run(Unknown Source)
   at java.lang.Thread.run(Thread.java:595)
"Timer-3" daemon prio=1 tid=0x0000002af2cb88c0 nid=0x6e04 in Object.wait() [0x0000000041d6e000..0x0000000041d6ec30]
   at java.lang.Object.wait(Native Method)
   - waiting on <0x0000002aac741e38> (a java.util.TaskQueue)
   at java.lang.Object.wait(Object.java:474)
   at java.util.TimerThread.mainLoop(Timer.java:483)
   - locked <0x0000002aac741e38> (a java.util.TaskQueue)
   at java.util.TimerThread.run(Timer.java:462)
"derby.antiGC" daemon prio=1 tid=0x0000002af398c040 nid=0x6e03 in Object.wait() [0x0000000041c6d000..0x0000000041c6dbb0]
   at java.lang.Object.wait(Native Method)
   - waiting on <0x0000002aac742300> (a org.apache.derby.impl.services.monitor.AntiGC)
   at java.lang.Object.wait(Object.java:474)
   at org.apache.derby.impl.services.monitor.AntiGC.run(Unknown Source)
   - locked <0x0000002aac742300> (a org.apache.derby.impl.services.monitor.AntiGC)
   at java.lang.Thread.run(Thread.java:595)
"Timer-2" daemon prio=1 tid=0x0000002af1791c90 nid=0x6e02 in Object.wait() [0x0000000041b6c000..0x0000000041b6cb30]
   at java.lang.Object.wait(Native Method)
   - waiting on <0x0000002aac7424e0> (a java.util.TaskQueue)
   at java.util.TimerThread.mainLoop(Timer.java:509)
   - locked <0x0000002aac7424e0> (a java.util.TaskQueue)
   at java.util.TimerThread.run(Timer.java:462)
"btpool0-9" prio=1 tid=0x0000002af1439ce0 nid=0x6e01 in Object.wait() [0x0000000041a6b000..0x0000000041a6bab0]
   at java.lang.Object.wait(Native Method)
   - waiting on <0x0000002aac4a07a8> (a org.mortbay.thread.BoundedThreadPool$PoolThread)
   at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:491)
   - locked <0x0000002aac4a07a8> (a org.mortbay.thread.BoundedThreadPool$PoolThread)
"btpool0-7" prio=1 tid=0x0000002aef9adbb0 nid=0x6dff in Object.wait() [0x0000000041869000..0x0000000041869db0]
   at java.lang.Object.wait(Native Method)
   - waiting on <0x0000002aac4a0708> (a org.mortbay.thread.BoundedThreadPool$PoolThread)
   at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:491)
   - locked <0x0000002aac4a0708> (a org.mortbay.thread.BoundedThreadPool$PoolThread)
"btpool0-5" prio=1 tid=0x0000002af103cd00 nid=0x6dfd in Object.wait() [0x0000000041667000..0x0000000041667cb0]
   at java.lang.Object.wait(Native Method)
   - waiting on <0x0000002aac4a0668> (a org.mortbay.thread.BoundedThreadPool$PoolThread)
   at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:491)
   - locked <0x0000002aac4a0668> (a org.mortbay.thread.BoundedThreadPool$PoolThread)
"btpool0-4" prio=1 tid=0x0000002aef94c4b0 nid=0x6dfc runnable [0x0000000041565000..0x0000000041566c30]
   at java.lang.AbstractStringBuilder.<init>(AbstractStringBuilder.java:44)
   at java.lang.StringBuilder.<init>(StringBuilder.java:69)
   at org.opendepo.depo.map.SingleTransactionLockingStrategyImpl.readlock(SingleTransactionLockingStrategyImpl.java:92)
   at org.opendepo.services.metadata.Workbench.readlock(Workbench.java:1070)
   at org.opendepo.depo.map.ExtKnowledgeMap.readlockTypes(ExtKnowledgeMap.java:1248)
   at org.opendepo.depo.map.DefaultClsMapStrategy.getSuperclassesList(DefaultClsMapStrategy.java:1627)
   at org.opendepo.depo.map.ExtKnowledgeMap.getSuperclassesList(ExtKnowledgeMap.java:2040)
   at org.opendepo.depo.map.EntityCls.getSuperclassList(EntityCls.java:257)
   at com.networkgps.itnav.instances.ClassEntity.populatePropertiesMinimal(ClassEntity.java:292)
   at com.networkgps.itnav.instances.ClassEntity.loadMinimal(ClassEntity.java:403)
   at com.networkgps.itnav.instances.PropertyEntity.getMinimalClassEntity(PropertyEntity.java:138)
   at com.networkgps.itnav.instances.PropertyEntity.setAllowedClasses(PropertyEntity.java:214)
   at com.networkgps.itnav.instances.PropertyEntity.populateProperties(PropertyEntity.java:834)
   at com.networkgps.itnav.instances.PropertyEntity.load(PropertyEntity.java:876)
   at com.networkgps.itnav.instances.ClassEntity.setPropertiesMap(ClassEntity.java:223)
   at com.networkgps.itnav.instances.DeploymentEntity.initializeFromDefaults(DeploymentEntity.java:294)
   at com.networkgps.itnav.instances.Access.createResourceEntity(Access.java:516)
   at com.networkgps.itnav.instances.Access.getInstance(Access.java:670)
   at com.networkgps.itnav.instances.Access.getInstance(Access.java:607)
   at com.networkgps.itnav.query.Access.formatQueryResults(Access.java:246)
   at com.networkgps.itnav.query.Access.processQuery(Access.java:93)
   at com.networkgps.itnav.query.ColonyQuery.go(ColonyQuery.java:71)
   at com.networkgps.itnav.context.ContextProcessBeanBase.execute(ContextProcessBeanBase.java:101)
   at org.apache.commons.scaffold.util.ProcessBeanBase.execute(Unknown Source)
   at org.apache.struts.scaffold.ProcessAction.executeLogic(Unknown Source)
   at com.networkgps.itnav.context.BaseContextProcessAction.executeLogic(BaseContextProcessAction.java:283)
   at org.apache.struts.scaffold.BaseHelperAction.executeLogic(Unknown Source)
   at org.apache.struts.scaffold.BaseAction.execute(Unknown Source)
   at com.networkgps.itnav.context.BaseContextProcessAction.execute(BaseContextProcessAction.java:229)
   at com.networkgps.itnav.context.BaseContextProcessAction.execute(BaseContextProcessAction.java:267)
   at org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:484)
   at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:274)
   at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1482)
   at org.apache.struts.action.ActionServlet.doPost(ActionServlet.java:525)
   at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
   at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
   at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502)
   at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:363)
   at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
   at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
   at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
   at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
   at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
   at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
   at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
   at org.mortbay.jetty.Server.handle(Server.java:324)
   at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
   at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:879)
   at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:741)
   at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:213)
   at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403)
   at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
   at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:451)
"btpool0-3" prio=1 tid=0x0000002aef94b980 nid=0x6dfb in Object.wait() [0x0000000041465000..0x0000000041465bb0]
   at java.lang.Object.wait(Native Method)
   - waiting on <0x0000002aac4a0528> (a org.mortbay.thread.BoundedThreadPool$PoolThread)
   at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:491)
   - locked <0x0000002aac4a0528> (a org.mortbay.thread.BoundedThreadPool$PoolThread)
"btpool0-2 - Acceptor1 SelectChann...@0.0.0.0:8080" prio=1 tid=0x0000002af10419d0 nid=0x6dfa runnable [0x0000000041364000..0x0000000041364b30]
   at sun.nio.ch.PollArrayWrapper.poll0(Native Method)
   at sun.nio.ch.PollArrayWrapper.poll(PollArrayWrapper.java:100)
   at sun.nio.ch.PollSelectorImpl.doSelect(PollSelectorImpl.java:56)
   at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
   - locked <0x0000002ab16464b0> (a sun.nio.ch.Util$1)
   - locked <0x0000002ab1646498> (a java.util.Collections$UnmodifiableSet)
   - locked <0x0000002ab1631b80> (a sun.nio.ch.PollSelectorImpl)
   at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
   at org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:429)
   at org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:185)
   at org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
   at org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:707)
   at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:451)
"btpool0-0 - Acceptor0 SelectChann...@0.0.0.0:8080" prio=1 tid=0x0000002af0447000 nid=0x6df8 runnable [0x0000000041162000..0x0000000041162e30]
   at sun.nio.ch.PollArrayWrapper.poll0(Native Method)
   at sun.nio.ch.PollArrayWrapper.poll(PollArrayWrapper.java:100)
   at sun.nio.ch.PollSelectorImpl.doSelect(PollSelectorImpl.java:56)
   at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
   - locked <0x0000002ab1631830> (a sun.nio.ch.Util$1)
   - locked <0x0000002ab1631818> (a java.util.Collections$UnmodifiableSet)
   - locked <0x0000002ab1631600> (a sun.nio.ch.PollSelectorImpl)
   at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
   at org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:429)
   at org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:185)
   at org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
   at org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:707)
   at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:451)
"Timer-1" daemon prio=1 tid=0x0000002aef94ce60 nid=0x6df7 runnable [0x0000000041061000..0x0000000041061db0]
   at java.io.UnixFileSystem.getBooleanAttributes0(Native Method)
   at java.io.UnixFileSystem.getBooleanAttributes(UnixFileSystem.java:228)
   at java.io.File.exists(File.java:702)
   at org.mortbay.util.Scanner.scanFile(Scanner.java:366)
   at org.mortbay.util.Scanner.scanFiles(Scanner.java:301)
   at org.mortbay.util.Scanner.scan(Scanner.java:279)
   at org.mortbay.util.Scanner$1.run(Scanner.java:232)

at java.util.TimerThread.mainLoop(Timer.java:512)
at java.util.TimerThread.run(Timer.java:462)

"Timer-0" daemon prio=1 tid=0x0000002af0ddba00 nid=0x6df6 in Object.wait() [0x0000000040f60000..0x0000000040f60d30]
   at java.lang.Object.wait(Native Method)
   - waiting on <0x0000002aac77f210> (a java.util.TaskQueue)
   at java.util.TimerThread.mainLoop(Timer.java:509)
   - locked <0x0000002aac77f210> (a java.util.TaskQueue)
   at java.util.TimerThread.run(Timer.java:462)
"Low Memory Detector" daemon prio=1 tid=0x0000002aef912030 nid=0x6df4 runnable [0x0000000000000000..0x0000000000000000]
"CompilerThread1" daemon prio=1 tid=0x0000002aef9105d0 nid=0x6df3 waiting on condition [0x0000000000000000..0x0000000040c5c500]
"CompilerThread0" daemon prio=1 tid=0x0000002aef90f230 nid=0x6df2 waiting on condition [0x0000000000000000..0x0000000040b5b460]
"AdapterThread" daemon prio=1 tid=0x0000002aef90de20 nid=0x6df1 waiting on condition [0x0000000000000000..0x0000000000000000]
"Signal Dispatcher" daemon prio=1 tid=0x0000002aef90cb80 nid=0x6df0 waiting on condition [0x0000000000000000..0x0000000000000000]
"Finalizer" daemon prio=1 tid=0x0000002aec9d3150 nid=0x6def in Object.wait() [0x0000000040859000..0x0000000040859db0]
   at java.lang.Object.wait(Native Method)
   - waiting on <0x0000002aac7c1490> (a java.lang.ref.ReferenceQueue$Lock)
   at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:120)
   - locked <0x0000002aac7c1490> (a java.lang.ref.ReferenceQueue$Lock)
   at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:136)
   at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
"Reference Handler" daemon prio=1 tid=0x0000002aec9d29b0 nid=0x6dee in Object.wait() [0x0000000040758000..0x0000000040758d30]
   at java.lang.Object.wait(Native Method)
   - waiting on <0x0000002aac4a7aa0> (a java.lang.ref.Reference$Lock)
   at java.lang.Object.wait(Object.java:474)
   at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
   - locked <0x0000002aac4a7aa0> (a java.lang.ref.Reference$Lock)
"VM Thread" prio=1 tid=0x0000002aec9ce4c0 nid=0x6ded runnable
"GC task thread#0 (ParallelGC)" prio=1 tid=0x0000000040134010 nid=0x6de9 runnable
"GC task thread#1 (ParallelGC)" prio=1 tid=0x0000000040135660 nid=0x6dea runnable
"GC task thread#2 (ParallelGC)" prio=1 tid=0x00000000401364b0 nid=0x6deb runnable
"GC task thread#3 (ParallelGC)" prio=1 tid=0x0000000040137330 nid=0x6dec runnable
"VM Periodic Task Thread" prio=1 tid=0x0000002aef914a10 nid=0x6df5 waiting on condition

If I were to guess I'd say the the problem originates with the implementation of the "-out" option from September 2009:

------------------------------------------------------------------------
r1726 | gschueler | 2009-09-22 16:22:53 -0700 (Tue, 22 Sep 2009) | 1 line

req #2859720: add -out option to find-objects command, which stores query results in project.xml format in the specified file.
------------------------------------------------------------------------

Anthony.

On Oct 14, 2010, at 11:51 AM, Anthony Shortland wrote:

Anthony Shortland

unread,

Oct 14, 2010, 3:51:49 PM10/14/10

to contr...@googlegroups.com

Note that this is happening under 3.4.12.

After doubling the maximum heap size to 2048 MB I was able to run the command quite a number of times:

[dnbiadm@gplnxv8 objects]$ history 20
1031
1032 ps
1033 ctl -p Dnbgp -t ProjectBuilder -o dnbgp -c find-objects -- -type DnbgpV3IntegratedBuilder -name '.*'
1034 ctl -p Dnbgp -t ProjectBuilder -o dnbgp -c find-objects -- -type DnbgpV3IntegratedBuilder -name '.*'
1035 ctl -p Dnbgp -t ProjectBuilder -o dnbgp -c find-objects -- -type DnbgpV3IntegratedBuilder -name '.*'
1036 ctl -p Dnbgp -t ProjectBuilder -o dnbgp -c find-objects -- -type DnbgpV3IntegratedBuilder -name '.*'
1037 ctl -p Dnbgp -t ProjectBuilder -o dnbgp -c find-objects -- -type DnbgpV3IntegratedBuilder -name '.*'
1038 ctl -p Dnbgp -t ProjectBuilder -o dnbgp -c find-objects -- -type DnbgpV3IntegratedBuilder -name '.*'
1039 ctl -p Dnbgp -t ProjectBuilder -o dnbgp -c find-objects -- -type DnbgpV3IntegratedBuilder -name '.*'
1040 ctl -p Dnbgp -t ProjectBuilder -o dnbgp -c find-objects -- -type DnbgpV3IntegratedBuilder -name '.*'
1041 ctl -p Dnbgp -t ProjectBuilder -o dnbgp -c find-objects -- -type DnbgpV3IntegratedBuilder -name '.*'
1042 ctl -p Dnbgp -t ProjectBuilder -o dnbgp -c find-objects -- -type DnbgpV3IntegratedBuilder -name '.*'
1043 ctl -p Dnbgp -t ProjectBuilder -o dnbgp -c find-objects -- -type DnbgpV3IntegratedBuilder -name '.*'
1044 history
1045 ctl -p Dnbgp -t ProjectBuilder -o dnbgp -c find-objects -- -type DnbgpV3IntegratedBuilder -name '.*'
1046 ctl -p Dnbgp -t ProjectBuilder -o dnbgp -c find-objects -- -type DnbgpV3IntegratedBuilder -name '.*'
1047 ctl -p Dnbgp -t ProjectBuilder -o dnbgp -c find-objects -- -type DnbgpV3IntegratedBuilder -name '.*'

...without seeing heap space problems, however the server JVM continues to "spin" even after the command has completed:

[dnbiadm@gplnxv8 jetty-6.1.14]$ ps -fp 2851

UID PID PPID C STIME TTY TIME CMD

dnbiadm 2851 1 82 12:16 pts/6 00:15:10 /usr/java/jdk1.5.0_16/bin/java -XX:MaxPermSize=256m -Xmx2048m -Xms256m -Djobcenter.config.location=/dnbusr1/dnbi

Here's another thread dump:

Full thread dump Java HotSpot(TM) 64-Bit Server VM (1.5.0_16-b02 mixed mode):

"DestroyJavaVM" prio=1 tid=0x0000002b3b544b10 nid=0xb23 waiting on condition [0x0000000000000000..0x0000007fbfffcc60]

"ContinuationBayeux-0" daemon prio=1 tid=0x0000002b3b571df0 nid=0xb68 in Object.wait() [0x0000000043e8f000..0x0000000043e8fb30]
   at java.lang.Object.wait(Native Method)
   - waiting on <0x0000002ab34a36d8> (a java.util.TaskQueue)
   at java.util.TimerThread.mainLoop(Timer.java:509)
   - locked <0x0000002ab34a36d8> (a java.util.TaskQueue)
   at java.util.TimerThread.run(Timer.java:462)

"Timer-11" daemon prio=1 tid=0x0000002b3b571300 nid=0xb67 in Object.wait() [0x0000000043d8e000..0x0000000043d8eab0]
   at java.lang.Object.wait(Native Method)
   - waiting on <0x0000002ab347ddc8> (a java.util.TaskQueue)
   at java.util.TimerThread.mainLoop(Timer.java:509)
   - locked <0x0000002ab347ddc8> (a java.util.TaskQueue)
   at java.util.TimerThread.run(Timer.java:462)

"Timer-10" daemon prio=1 tid=0x0000002b3ebfea20 nid=0xb66 in Object.wait() [0x0000000043c8d000..0x0000000043c8de30]
   at java.lang.Object.wait(Native Method)
   - waiting on <0x0000002ab333eff0> (a java.util.TaskQueue)
   at java.util.TimerThread.mainLoop(Timer.java:509)
   - locked <0x0000002ab333eff0> (a java.util.TaskQueue)
   at java.util.TimerThread.run(Timer.java:462)

"Timer-9" daemon prio=1 tid=0x0000002b3ebfb570 nid=0xb65 in Object.wait() [0x0000000043b8c000..0x0000000043b8cdb0]
   at java.lang.Object.wait(Native Method)
   - waiting on <0x0000002ab330cd00> (a java.util.TaskQueue)
   at java.util.TimerThread.mainLoop(Timer.java:509)
   - locked <0x0000002ab330cd00> (a java.util.TaskQueue)
   at java.util.TimerThread.run(Timer.java:462)

"Timer-8" daemon prio=1 tid=0x0000002b3eb503e0 nid=0xb64 in Object.wait() [0x0000000043a8b000..0x0000000043a8bd30]
   at java.lang.Object.wait(Native Method)
   - waiting on <0x0000002ab3523908> (a java.util.TaskQueue)
   at java.util.TimerThread.mainLoop(Timer.java:509)
   - locked <0x0000002ab3523908> (a java.util.TaskQueue)
   at java.util.TimerThread.run(Timer.java:462)

"Timer-7" daemon prio=1 tid=0x0000002b369964c0 nid=0xb61 in Object.wait() [0x000000004398a000..0x000000004398ac30]
   at java.lang.Object.wait(Native Method)
   - waiting on <0x0000002ab32ffb08> (a java.util.TaskQueue)
   at java.util.TimerThread.mainLoop(Timer.java:509)
   - locked <0x0000002ab32ffb08> (a java.util.TaskQueue)
   at java.util.TimerThread.run(Timer.java:462)

"Thread-23" prio=1 tid=0x0000002b33f77c60 nid=0xb60 runnable [0x0000000043889000..0x0000000043889bb0]

   at java.net.SocketInputStream.socketRead0(Native Method)
   at java.net.SocketInputStream.read(SocketInputStream.java:129)
   at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
   at java.io.BufferedInputStream.read(BufferedInputStream.java:235)

- locked <0x0000002ab2917e98> (a java.io.BufferedInputStream)

   at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2196)
   at java.io.ObjectInputStream$BlockDataInputStream.peek(ObjectInputStream.java:2486)
   at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2496)
   at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1273)
   at java.io.ObjectInputStream.readObject(ObjectInputStream.java:348)
   at org.apache.log4j.net.SocketNode.run(SocketNode.java:77)
   at java.lang.Thread.run(Thread.java:595)

"quartzScheduler_QuartzSchedulerThread" prio=1 tid=0x0000002b36707380 nid=0xb5f in Object.wait() [0x0000000043788000..0x0000000043788b30]
   at java.lang.Object.wait(Native Method)
   - waiting on <0x0000002ab278dce8> (a java.lang.Object)
   at org.quartz.core.QuartzSchedulerThread.run(QuartzSchedulerThread.java:433)
   - locked <0x0000002ab278dce8> (a java.lang.Object)

"quartzScheduler_Worker-10" prio=1 tid=0x0000002b367062b0 nid=0xb5e in Object.wait() [0x0000000043687000..0x0000000043687ab0]
   at java.lang.Object.wait(Native Method)
   - waiting on <0x0000002ab278e5f8> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)
   at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:519)
   - locked <0x0000002ab278e5f8> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)

"quartzScheduler_Worker-9" prio=1 tid=0x0000002b366b1af0 nid=0xb5d in Object.wait() [0x0000000043586000..0x0000000043586e30]
   at java.lang.Object.wait(Native Method)
   - waiting on <0x0000002ab278e8e0> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)
   at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:519)
   - locked <0x0000002ab278e8e0> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)

"quartzScheduler_Worker-8" prio=1 tid=0x0000002b366b0f00 nid=0xb5c in Object.wait() [0x0000000043485000..0x0000000043485db0]
   at java.lang.Object.wait(Native Method)
   - waiting on <0x0000002ab0f2eea0> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)
   at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:519)
   - locked <0x0000002ab0f2eea0> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)

"quartzScheduler_Worker-7" prio=1 tid=0x0000002b31613d30 nid=0xb5b in Object.wait() [0x0000000043384000..0x0000000043384d30]
   at java.lang.Object.wait(Native Method)
   - waiting on <0x0000002ab0f2edf0> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)
   at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:519)
   - locked <0x0000002ab0f2edf0> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)

"quartzScheduler_Worker-6" prio=1 tid=0x0000002b32e97050 nid=0xb5a in Object.wait() [0x0000000043283000..0x0000000043283cb0]
   at java.lang.Object.wait(Native Method)
   - waiting on <0x0000002ab0f2ed40> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)
   at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:519)
   - locked <0x0000002ab0f2ed40> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)

"quartzScheduler_Worker-5" prio=1 tid=0x0000002b32e964e0 nid=0xb59 in Object.wait() [0x0000000043182000..0x0000000043182c30]
   at java.lang.Object.wait(Native Method)
   - waiting on <0x0000002ab0f2ec90> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)
   at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:519)
   - locked <0x0000002ab0f2ec90> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)

"quartzScheduler_Worker-4" prio=1 tid=0x0000002b36206750 nid=0xb58 in Object.wait() [0x0000000043081000..0x0000000043081bb0]
   at java.lang.Object.wait(Native Method)
   - waiting on <0x0000002ab0f2ebe0> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)
   at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:519)
   - locked <0x0000002ab0f2ebe0> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)

"quartzScheduler_Worker-3" prio=1 tid=0x0000002b3624ef50 nid=0xb57 in Object.wait() [0x0000000042f80000..0x0000000042f80b30]
   at java.lang.Object.wait(Native Method)
   - waiting on <0x0000002ab0f2eb30> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)
   at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:519)
   - locked <0x0000002ab0f2eb30> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)

"quartzScheduler_Worker-2" prio=1 tid=0x0000002b3a1fb950 nid=0xb56 in Object.wait() [0x0000000042e7f000..0x0000000042e7fab0]
   at java.lang.Object.wait(Native Method)
   - waiting on <0x0000002ab0f2ea80> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)
   at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:519)
   - locked <0x0000002ab0f2ea80> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)

"quartzScheduler_Worker-1" prio=1 tid=0x0000002b3a1fb6a0 nid=0xb55 in Object.wait() [0x0000000042d7e000..0x0000000042d7ee30]
   at java.lang.Object.wait(Native Method)
   - waiting on <0x0000002ab0f2e9d0> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)
   at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:519)
   - locked <0x0000002ab0f2e9d0> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)

"HSQLDB Timer @496af7bf" daemon prio=1 tid=0x0000002b3691d750 nid=0xb54 in Object.wait() [0x0000000042c7d000..0x0000000042c7ddb0]
   at java.lang.Object.wait(Native Method)
   - waiting on <0x0000002ab2089bd8> (a org.hsqldb.lib.HsqlTimer)
   at org.hsqldb.lib.HsqlTimer.nextTask(Unknown Source)
   - locked <0x0000002ab2089bd8> (a org.hsqldb.lib.HsqlTimer)

at org.hsqldb.lib.HsqlTimer$TaskRunner.run(Unknown Source)
at java.lang.Thread.run(Thread.java:595)

"Timer-6" daemon prio=1 tid=0x0000002b33907c40 nid=0xb51 in Object.wait() [0x0000000042b7c000..0x0000000042b7cd30]
   at java.lang.Object.wait(Native Method)
   - waiting on <0x0000002ab0cfb458> (a java.util.TaskQueue)
   at java.util.TimerThread.mainLoop(Timer.java:509)
   - locked <0x0000002ab0cfb458> (a java.util.TaskQueue)
   at java.util.TimerThread.run(Timer.java:462)

"Thread-20" prio=1 tid=0x0000002b33327610 nid=0xb4f runnable [0x0000000042a7b000..0x0000000042a7bcb0]

at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)

- locked <0x0000002aae0adb88> (a java.net.SocksSocketImpl)

   at java.net.ServerSocket.implAccept(ServerSocket.java:450)
   at java.net.ServerSocket.accept(ServerSocket.java:421)
   at com.controltier.reporter.services.InputServer$worker.run(InputServer.java:81)
   at java.lang.Thread.run(Thread.java:595)

"Store org.hibernate.cache.StandardQueryCache Spool Thread" daemon prio=1 tid=0x0000002b3647c500 nid=0xb4d waiting on condition [0x000000004297a000..0x000000004297ac30]

   at java.lang.Thread.sleep(Native Method)
   at net.sf.ehcache.store.DiskStore.spoolAndExpiryThreadMain(DiskStore.java:589)
   at net.sf.ehcache.store.DiskStore.access$800(DiskStore.java:65)
   at net.sf.ehcache.store.DiskStore$SpoolAndExpiryThread.run(DiskStore.java:1075)

"Store org.hibernate.cache.UpdateTimestampsCache Spool Thread" daemon prio=1 tid=0x0000002b3648a640 nid=0xb4c waiting on condition [0x0000000042879000..0x0000000042879bb0]

   at java.lang.Thread.sleep(Native Method)
   at net.sf.ehcache.store.DiskStore.spoolAndExpiryThreadMain(DiskStore.java:589)
   at net.sf.ehcache.store.DiskStore.access$800(DiskStore.java:65)
   at net.sf.ehcache.store.DiskStore$SpoolAndExpiryThread.run(DiskStore.java:1075)

"HSQLDB Timer @1bfa1ba1" daemon prio=1 tid=0x0000002b2cd69130 nid=0xb4b in Object.wait() [0x0000000042778000..0x0000000042778b30]
at java.lang.Object.wait(Native Method)
- waiting on <0x0000002aaf60e048> (a org.hsqldb.lib.HsqlTimer)

at java.lang.Object.wait(Object.java:474)
at org.hsqldb.lib.HsqlTimer.nextTask(Unknown Source)

- locked <0x0000002aaf60e048> (a org.hsqldb.lib.HsqlTimer)

at org.hsqldb.lib.HsqlTimer$TaskRunner.run(Unknown Source)
at java.lang.Thread.run(Thread.java:595)

"Timer-5" daemon prio=1 tid=0x0000002b33042440 nid=0xb4a in Object.wait() [0x0000000042677000..0x0000000042677ab0]
   at java.lang.Object.wait(Native Method)
   - waiting on <0x0000002aacf0e0f8> (a java.util.TaskQueue)
   at java.util.TimerThread.mainLoop(Timer.java:509)
   - locked <0x0000002aacf0e0f8> (a java.util.TaskQueue)
   at java.util.TimerThread.run(Timer.java:462)

"RMI TCP Accept-1099" daemon prio=1 tid=0x0000002b3172b6e0 nid=0xb48 runnable [0x0000000042576000..0x0000000042576e30]

at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)

- locked <0x0000002aac4ef390> (a java.net.SocksSocketImpl)

   at java.net.ServerSocket.implAccept(ServerSocket.java:450)
   at java.net.ServerSocket.accept(ServerSocket.java:421)
   at sun.rmi.transport.tcp.TCPTransport.run(TCPTransport.java:340)
   at java.lang.Thread.run(Thread.java:595)

"GC Daemon" daemon prio=1 tid=0x0000002b31242aa0 nid=0xb47 in Object.wait() [0x0000000042475000..0x0000000042475db0]
   at java.lang.Object.wait(Native Method)
   - waiting on <0x0000002aac4e0be0> (a sun.misc.GC$LatencyLock)
   at sun.misc.GC$Daemon.run(GC.java:100)
   - locked <0x0000002aac4e0be0> (a sun.misc.GC$LatencyLock)

"RMI Reaper" prio=1 tid=0x0000002b31694cb0 nid=0xb46 in Object.wait() [0x0000000042374000..0x0000000042374d30]
   at java.lang.Object.wait(Native Method)
   - waiting on <0x0000002aac4e0de0> (a java.lang.ref.ReferenceQueue$Lock)
   at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:120)
   - locked <0x0000002aac4e0de0> (a java.lang.ref.ReferenceQueue$Lock)

   at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:136)
   at sun.rmi.transport.ObjectTable$Reaper.run(ObjectTable.java:336)
   at java.lang.Thread.run(Thread.java:595)

"Timer-4" daemon prio=1 tid=0x0000002b316944d0 nid=0xb45 in Object.wait() [0x0000000042273000..0x0000000042273cb0]
at java.lang.Object.wait(Native Method)
- waiting on <0x0000002aac4a8b48> (a java.util.TaskQueue)

at java.lang.Object.wait(Object.java:474)
at java.util.TimerThread.mainLoop(Timer.java:483)

- locked <0x0000002aac4a8b48> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:462)

"RMI TCP Accept-0" daemon prio=1 tid=0x0000002b33052900 nid=0xb44 runnable [0x0000000042172000..0x0000000042172c30]

at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)

- locked <0x0000002aac4e1038> (a java.net.SocksSocketImpl)

   at java.net.ServerSocket.implAccept(ServerSocket.java:450)
   at java.net.ServerSocket.accept(ServerSocket.java:421)
   at sun.rmi.transport.tcp.TCPTransport.run(TCPTransport.java:340)
   at java.lang.Thread.run(Thread.java:595)

"ObservationManager" daemon prio=1 tid=0x0000002b313fd100 nid=0xb43 in Object.wait() [0x0000000042071000..0x0000000042071bb0]
at java.lang.Object.wait(Native Method)
- waiting on <0x0000002aac4e1688> (a org.apache.commons.collections.buffer.BlockingBuffer)

at java.lang.Object.wait(Object.java:474)
at org.apache.commons.collections.buffer.BlockingBuffer.remove(BlockingBuffer.java:107)

- locked <0x0000002aac4e1688> (a org.apache.commons.collections.buffer.BlockingBuffer)

at org.apache.jackrabbit.core.observation.ObservationDispatcher.run(ObservationDispatcher.java:138)
at java.lang.Thread.run(Thread.java:595)

"derby.rawStoreDaemon" daemon prio=1 tid=0x0000002b3300f9d0 nid=0xb42 in Object.wait() [0x0000000041f70000..0x0000000041f70b30]
   at java.lang.Object.wait(Native Method)
   - waiting on <0x0000002aac549578> (a org.apache.derby.impl.services.daemon.BasicDaemon)
   at org.apache.derby.impl.services.daemon.BasicDaemon.rest(Unknown Source)
   - locked <0x0000002aac549578> (a org.apache.derby.impl.services.daemon.BasicDaemon)

at org.apache.derby.impl.services.daemon.BasicDaemon.run(Unknown Source)
at java.lang.Thread.run(Thread.java:595)

"derby.rawStoreDaemon" daemon prio=1 tid=0x0000002b2fc18e10 nid=0xb41 in Object.wait() [0x0000000041e6f000..0x0000000041e6fab0]
   at java.lang.Object.wait(Native Method)
   - waiting on <0x0000002aac500880> (a org.apache.derby.impl.services.daemon.BasicDaemon)
   at org.apache.derby.impl.services.daemon.BasicDaemon.rest(Unknown Source)
   - locked <0x0000002aac500880> (a org.apache.derby.impl.services.daemon.BasicDaemon)

at org.apache.derby.impl.services.daemon.BasicDaemon.run(Unknown Source)
at java.lang.Thread.run(Thread.java:595)

"Timer-3" daemon prio=1 tid=0x0000002b32e0fa00 nid=0xb40 in Object.wait() [0x0000000041d6e000..0x0000000041d6ee30]
at java.lang.Object.wait(Native Method)
- waiting on <0x0000002aac4fc860> (a java.util.TaskQueue)

at java.lang.Object.wait(Object.java:474)
at java.util.TimerThread.mainLoop(Timer.java:483)

   - locked <0x0000002aac4fc860> (a java.util.TaskQueue)
   at java.util.TimerThread.run(Timer.java:462)

"derby.antiGC" daemon prio=1 tid=0x0000002b33022f40 nid=0xb3f in Object.wait() [0x0000000041c6d000..0x0000000041c6ddb0]
   at java.lang.Object.wait(Native Method)
   - waiting on <0x0000002aac712590> (a org.apache.derby.impl.services.monitor.AntiGC)

at java.lang.Object.wait(Object.java:474)
at org.apache.derby.impl.services.monitor.AntiGC.run(Unknown Source)

   - locked <0x0000002aac712590> (a org.apache.derby.impl.services.monitor.AntiGC)
   at java.lang.Thread.run(Thread.java:595)

"Timer-2" daemon prio=1 tid=0x0000002b2fcc8400 nid=0xb3e in Object.wait() [0x0000000041b6c000..0x0000000041b6cd30]
   at java.lang.Object.wait(Native Method)
   - waiting on <0x0000002aac4fc9e8> (a java.util.TaskQueue)
   at java.util.TimerThread.mainLoop(Timer.java:509)
   - locked <0x0000002aac4fc9e8> (a java.util.TaskQueue)
   at java.util.TimerThread.run(Timer.java:462)

"btpool0-9" prio=1 tid=0x0000002b31512af0 nid=0xb3d in Object.wait() [0x0000000041a6b000..0x0000000041a6bcb0]
   at java.lang.Object.wait(Native Method)
   - waiting on <0x0000002aac4a85f8> (a org.mortbay.thread.BoundedThreadPool$PoolThread)
   at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:491)
   - locked <0x0000002aac4a85f8> (a org.mortbay.thread.BoundedThreadPool$PoolThread)

"btpool0-8" prio=1 tid=0x0000002b31511760 nid=0xb3c in Object.wait() [0x000000004196a000..0x000000004196ac30]
   at java.lang.Object.wait(Native Method)
   - waiting on <0x0000002aac4a8558> (a org.mortbay.thread.BoundedThreadPool$PoolThread)
   at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:491)
   - locked <0x0000002aac4a8558> (a org.mortbay.thread.BoundedThreadPool$PoolThread)

"btpool0-7" prio=1 tid=0x0000002b315103d0 nid=0xb3b in Object.wait() [0x0000000041869000..0x0000000041869bb0]
   at java.lang.Object.wait(Native Method)
   - waiting on <0x0000002aac4a84b8> (a org.mortbay.thread.BoundedThreadPool$PoolThread)
   at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:491)
   - locked <0x0000002aac4a84b8> (a org.mortbay.thread.BoundedThreadPool$PoolThread)

"btpool0-6" prio=1 tid=0x0000002b3150f040 nid=0xb3a in Object.wait() [0x0000000041768000..0x0000000041768b30]
   at java.lang.Object.wait(Native Method)
   - waiting on <0x0000002aac4a8418> (a org.mortbay.thread.BoundedThreadPool$PoolThread)
   at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:491)
   - locked <0x0000002aac4a8418> (a org.mortbay.thread.BoundedThreadPool$PoolThread)

"btpool0-5" prio=1 tid=0x0000002b3150dcb0 nid=0xb39 in Object.wait() [0x0000000041667000..0x0000000041667ab0]
   at java.lang.Object.wait(Native Method)
   - waiting on <0x0000002aac4a8378> (a org.mortbay.thread.BoundedThreadPool$PoolThread)
   at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:491)
   - locked <0x0000002aac4a8378> (a org.mortbay.thread.BoundedThreadPool$PoolThread)

"btpool0-4" prio=1 tid=0x0000002b30fb4a70 nid=0xb38 in Object.wait() [0x0000000041566000..0x0000000041566e30]
   at java.lang.Object.wait(Native Method)
   - waiting on <0x0000002aac4a82d8> (a org.mortbay.thread.BoundedThreadPool$PoolThread)
   at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:491)
   - locked <0x0000002aac4a82d8> (a org.mortbay.thread.BoundedThreadPool$PoolThread)

"btpool0-3" prio=1 tid=0x0000002b30fb3ad0 nid=0xb37 in Object.wait() [0x0000000041465000..0x0000000041465db0]
   at java.lang.Object.wait(Native Method)
   - waiting on <0x0000002aac4a8238> (a org.mortbay.thread.BoundedThreadPool$PoolThread)
   at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:491)
   - locked <0x0000002aac4a8238> (a org.mortbay.thread.BoundedThreadPool$PoolThread)

"btpool0-2 - Acceptor1 SelectChann...@0.0.0.0:8080" prio=1 tid=0x0000002b31069c10 nid=0xb36 runnable [0x0000000041364000..0x0000000041364d30]

   at sun.nio.ch.PollArrayWrapper.poll0(Native Method)
   at sun.nio.ch.PollArrayWrapper.poll(PollArrayWrapper.java:100)
   at sun.nio.ch.PollSelectorImpl.doSelect(PollSelectorImpl.java:56)
   at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)

   - locked <0x0000002ab34a8f10> (a sun.nio.ch.Util$1)
   - locked <0x0000002ab34a8ef8> (a java.util.Collections$UnmodifiableSet)
   - locked <0x0000002ab34a8e10> (a sun.nio.ch.PollSelectorImpl)

   at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
   at org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:429)
   at org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:185)
   at org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
   at org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:707)
   at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:451)

"btpool0-1" prio=1 tid=0x0000002b31069130 nid=0xb35 in Object.wait() [0x0000000041263000..0x0000000041263cb0]
   at java.lang.Object.wait(Native Method)
   - waiting on <0x0000002aac4a80f8> (a org.mortbay.thread.BoundedThreadPool$PoolThread)
   at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:491)
   - locked <0x0000002aac4a80f8> (a org.mortbay.thread.BoundedThreadPool$PoolThread)

"btpool0-0 - Acceptor0 SelectChann...@0.0.0.0:8080" prio=1 tid=0x0000002b30fe6ee0 nid=0xb34 runnable [0x0000000041162000..0x0000000041162c30]

   at sun.nio.ch.PollArrayWrapper.poll0(Native Method)
   at sun.nio.ch.PollArrayWrapper.poll(PollArrayWrapper.java:100)
   at sun.nio.ch.PollSelectorImpl.doSelect(PollSelectorImpl.java:56)
   at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)

   - locked <0x0000002ab35df458> (a sun.nio.ch.Util$1)
   - locked <0x0000002ab35df440> (a java.util.Collections$UnmodifiableSet)
   - locked <0x0000002ab35df2e0> (a sun.nio.ch.PollSelectorImpl)

   at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
   at org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:429)
   at org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:185)
   at org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
   at org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:707)
   at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:451)

"Timer-1" daemon prio=1 tid=0x0000002b30fe76f0 nid=0xb33 in Object.wait() [0x0000000041061000..0x0000000041061bb0]
   at java.lang.Object.wait(Native Method)
   - waiting on <0x0000002aac6c4238> (a java.util.TaskQueue)
   at java.util.TimerThread.mainLoop(Timer.java:509)
   - locked <0x0000002aac6c4238> (a java.util.TaskQueue)
   at java.util.TimerThread.run(Timer.java:462)

"Timer-0" daemon prio=1 tid=0x0000002b31274010 nid=0xb32 in Object.wait() [0x0000000040f60000..0x0000000040f60b30]
   at java.lang.Object.wait(Native Method)
   - waiting on <0x0000002aac6c64c0> (a java.util.TaskQueue)
   at java.util.TimerThread.mainLoop(Timer.java:509)
   - locked <0x0000002aac6c64c0> (a java.util.TaskQueue)
   at java.util.TimerThread.run(Timer.java:462)

"Low Memory Detector" daemon prio=1 tid=0x0000002b2fc12070 nid=0xb30 runnable [0x0000000000000000..0x0000000000000000]

"CompilerThread1" daemon prio=1 tid=0x0000002b2fc10610 nid=0xb2f waiting on condition [0x0000000000000000..0x0000000040c5c6d0]

"CompilerThread0" daemon prio=1 tid=0x0000002b2fc0f270 nid=0xb2e waiting on condition [0x0000000000000000..0x0000000040b5b660]

"AdapterThread" daemon prio=1 tid=0x0000002b2fc0de60 nid=0xb2d waiting on condition [0x0000000000000000..0x0000000000000000]

"Signal Dispatcher" daemon prio=1 tid=0x0000002b2fc0cbc0 nid=0xb2c runnable [0x0000000000000000..0x0000000000000000]

"Finalizer" daemon prio=1 tid=0x0000002b2cd28150 nid=0xb2b in Object.wait() [0x0000000040859000..0x0000000040859bb0]
   at java.lang.Object.wait(Native Method)
   - waiting on <0x0000002aac729b30> (a java.lang.ref.ReferenceQueue$Lock)
   at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:120)
   - locked <0x0000002aac729b30> (a java.lang.ref.ReferenceQueue$Lock)

at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:136)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)

"Reference Handler" daemon prio=1 tid=0x0000002b2cd279b0 nid=0xb2a in Object.wait() [0x0000000040758000..0x0000000040758b30]
at java.lang.Object.wait(Native Method)
- waiting on <0x0000002aac4a05b0> (a java.lang.ref.Reference$Lock)

at java.lang.Object.wait(Object.java:474)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)

- locked <0x0000002aac4a05b0> (a java.lang.ref.Reference$Lock)

"VM Thread" prio=1 tid=0x0000002b2cd234c0 nid=0xb29 runnable

"GC task thread#0 (ParallelGC)" prio=1 tid=0x0000000040134010 nid=0xb25 runnable

"GC task thread#1 (ParallelGC)" prio=1 tid=0x0000000040135660 nid=0xb26 runnable

"GC task thread#2 (ParallelGC)" prio=1 tid=0x00000000401364b0 nid=0xb27 runnable

"GC task thread#3 (ParallelGC)" prio=1 tid=0x0000000040137330 nid=0xb28 runnable

"VM Periodic Task Thread" prio=1 tid=0x0000002b2fc14a50 nid=0xb31 waiting on condition

I suspect its only a matter of time before the server turns turtle!

As a point of reference, I started the server and waited a few minutes without running any commands until it stabilized:

[dnbiadm@gplnxv8 jetty-6.1.14]$ date
Thu Oct 14 12:48:57 PDT 2010
[dnbiadm@gplnxv8 jetty-6.1.14]$ ps -fp 24607

UID PID PPID C STIME TTY TIME CMD

dnbiadm 24607 1 11 12:38 pts/6 00:01:14 /usr/java/jdk1.5.0_16/bin/java -XX:MaxPermSize=256m -Xmx2048m -Xms256m -Djobcenter.config.location=/dnbusr1/dnbi

... here's the thread dump at that time:

Full thread dump Java HotSpot(TM) 64-Bit Server VM (1.5.0_16-b02 mixed mode):

"DestroyJavaVM" prio=1 tid=0x0000002b318a0df0 nid=0x601f waiting on condition [0x0000000000000000..0x0000007fbfffcc60]

"ContinuationBayeux-0" daemon prio=1 tid=0x0000002b36587470 nid=0x605f in Object.wait() [0x0000000043e8f000..0x0000000043e8fd30]
   at java.lang.Object.wait(Native Method)
   - waiting on <0x0000002ab3649478> (a java.util.TaskQueue)
   at java.util.TimerThread.mainLoop(Timer.java:509)
   - locked <0x0000002ab3649478> (a java.util.TaskQueue)
   at java.util.TimerThread.run(Timer.java:462)

"Timer-11" daemon prio=1 tid=0x0000002b3834b900 nid=0x605e in Object.wait() [0x0000000043d8e000..0x0000000043d8ecb0]
   at java.lang.Object.wait(Native Method)
   - waiting on <0x0000002ab3649520> (a java.util.TaskQueue)
   at java.util.TimerThread.mainLoop(Timer.java:509)
   - locked <0x0000002ab3649520> (a java.util.TaskQueue)
   at java.util.TimerThread.run(Timer.java:462)

"Timer-10" daemon prio=1 tid=0x0000002b37d69850 nid=0x605d in Object.wait() [0x0000000043c8d000..0x0000000043c8dc30]
   at java.lang.Object.wait(Native Method)
   - waiting on <0x0000002ab36c82e0> (a java.util.TaskQueue)
   at java.util.TimerThread.mainLoop(Timer.java:509)
   - locked <0x0000002ab36c82e0> (a java.util.TaskQueue)
   at java.util.TimerThread.run(Timer.java:462)

"Timer-9" daemon prio=1 tid=0x0000002b3bdfbc10 nid=0x605c in Object.wait() [0x0000000043b8c000..0x0000000043b8cbb0]
   at java.lang.Object.wait(Native Method)
   - waiting on <0x0000002ab3701ac0> (a java.util.TaskQueue)
   at java.util.TimerThread.mainLoop(Timer.java:509)
   - locked <0x0000002ab3701ac0> (a java.util.TaskQueue)
   at java.util.TimerThread.run(Timer.java:462)

"Timer-8" daemon prio=1 tid=0x0000002b38a6b7d0 nid=0x605b in Object.wait() [0x0000000043a8b000..0x0000000043a8bb30]
   at java.lang.Object.wait(Native Method)
   - waiting on <0x0000002ab36f0e08> (a java.util.TaskQueue)
   at java.util.TimerThread.mainLoop(Timer.java:509)
   - locked <0x0000002ab36f0e08> (a java.util.TaskQueue)
   at java.util.TimerThread.run(Timer.java:462)

"Timer-7" daemon prio=1 tid=0x0000002b389c9b90 nid=0x6059 in Object.wait() [0x000000004398a000..0x000000004398ae30]
   at java.lang.Object.wait(Native Method)
   - waiting on <0x0000002ab1fb47e8> (a java.util.TaskQueue)
   at java.util.TimerThread.mainLoop(Timer.java:509)
   - locked <0x0000002ab1fb47e8> (a java.util.TaskQueue)
   at java.util.TimerThread.run(Timer.java:462)

"Thread-23" prio=1 tid=0x0000002b36fc00a0 nid=0x6058 runnable [0x0000000043889000..0x0000000043889db0]

   at java.net.SocketInputStream.socketRead0(Native Method)
   at java.net.SocketInputStream.read(SocketInputStream.java:129)
   at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
   at java.io.BufferedInputStream.read(BufferedInputStream.java:235)

- locked <0x0000002ab1f76928> (a java.io.BufferedInputStream)

   at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2196)
   at java.io.ObjectInputStream$BlockDataInputStream.peek(ObjectInputStream.java:2486)
   at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2496)
   at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1273)
   at java.io.ObjectInputStream.readObject(ObjectInputStream.java:348)
   at org.apache.log4j.net.SocketNode.run(SocketNode.java:77)
   at java.lang.Thread.run(Thread.java:595)

"quartzScheduler_QuartzSchedulerThread" prio=1 tid=0x0000002b2fc2fca0 nid=0x6057 in Object.wait() [0x0000000043788000..0x0000000043788d30]
   at java.lang.Object.wait(Native Method)
   - waiting on <0x0000002ab1843548> (a java.lang.Object)
   at org.quartz.core.QuartzSchedulerThread.run(QuartzSchedulerThread.java:433)
   - locked <0x0000002ab1843548> (a java.lang.Object)

"quartzScheduler_Worker-10" prio=1 tid=0x0000002b3449eb60 nid=0x6056 in Object.wait() [0x0000000043687000..0x0000000043687cb0]
   at java.lang.Object.wait(Native Method)
   - waiting on <0x0000002ab184d670> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)
   at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:519)
   - locked <0x0000002ab184d670> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)

"quartzScheduler_Worker-9" prio=1 tid=0x0000002b3448ae70 nid=0x6055 in Object.wait() [0x0000000043586000..0x0000000043586c30]
   at java.lang.Object.wait(Native Method)
   - waiting on <0x0000002ab1863d90> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)
   at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:519)
   - locked <0x0000002ab1863d90> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)

"quartzScheduler_Worker-8" prio=1 tid=0x0000002b2fc16ae0 nid=0x6054 in Object.wait() [0x0000000043485000..0x0000000043485bb0]
   at java.lang.Object.wait(Native Method)
   - waiting on <0x0000002ab2c383f0> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)
   at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:519)
   - locked <0x0000002ab2c383f0> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)

"quartzScheduler_Worker-7" prio=1 tid=0x0000002b3634abd0 nid=0x6053 in Object.wait() [0x0000000043384000..0x0000000043384b30]
   at java.lang.Object.wait(Native Method)
   - waiting on <0x0000002ab2c38340> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)
   at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:519)
   - locked <0x0000002ab2c38340> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)

"quartzScheduler_Worker-6" prio=1 tid=0x0000002b3158bf60 nid=0x6052 in Object.wait() [0x0000000043283000..0x0000000043283ab0]
   at java.lang.Object.wait(Native Method)
   - waiting on <0x0000002ab2c38290> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)
   at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:519)
   - locked <0x0000002ab2c38290> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)

"quartzScheduler_Worker-5" prio=1 tid=0x0000002b32dd6410 nid=0x6051 in Object.wait() [0x0000000043182000..0x0000000043182e30]
   at java.lang.Object.wait(Native Method)
   - waiting on <0x0000002ab2c381e0> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)
   at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:519)
   - locked <0x0000002ab2c381e0> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)

"quartzScheduler_Worker-4" prio=1 tid=0x0000002b3158d570 nid=0x6050 in Object.wait() [0x0000000043081000..0x0000000043081db0]
   at java.lang.Object.wait(Native Method)
   - waiting on <0x0000002ab2c38130> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)
   at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:519)
   - locked <0x0000002ab2c38130> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)

"quartzScheduler_Worker-3" prio=1 tid=0x0000002b33fcc340 nid=0x604f in Object.wait() [0x0000000042f80000..0x0000000042f80d30]
   at java.lang.Object.wait(Native Method)
   - waiting on <0x0000002ab2c38080> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)
   at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:519)
   - locked <0x0000002ab2c38080> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)

"quartzScheduler_Worker-2" prio=1 tid=0x0000002b38223190 nid=0x604e in Object.wait() [0x0000000042e7f000..0x0000000042e7fcb0]
   at java.lang.Object.wait(Native Method)
   - waiting on <0x0000002ab2c37fd0> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)
   at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:519)
   - locked <0x0000002ab2c37fd0> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)

"quartzScheduler_Worker-1" prio=1 tid=0x0000002b38222ee0 nid=0x604d in Object.wait() [0x0000000042d7e000..0x0000000042d7ec30]
   at java.lang.Object.wait(Native Method)
   - waiting on <0x0000002ab2c37f20> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)
   at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:519)
   - locked <0x0000002ab2c37f20> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)

"HSQLDB Timer @753059df" daemon prio=1 tid=0x0000002b32dd57c0 nid=0x604c in Object.wait() [0x0000000042c7d000..0x0000000042c7dbb0]
   at java.lang.Object.wait(Native Method)
   - waiting on <0x0000002ab1f6b688> (a org.hsqldb.lib.HsqlTimer)
   at org.hsqldb.lib.HsqlTimer.nextTask(Unknown Source)
   - locked <0x0000002ab1f6b688> (a org.hsqldb.lib.HsqlTimer)

at org.hsqldb.lib.HsqlTimer$TaskRunner.run(Unknown Source)
at java.lang.Thread.run(Thread.java:595)

"Timer-6" daemon prio=1 tid=0x0000002b384ff790 nid=0x604b in Object.wait() [0x0000000042b7c000..0x0000000042b7cb30]
   at java.lang.Object.wait(Native Method)
   - waiting on <0x0000002aae687cf0> (a java.util.TaskQueue)
   at java.util.TimerThread.mainLoop(Timer.java:509)
   - locked <0x0000002aae687cf0> (a java.util.TaskQueue)
   at java.util.TimerThread.run(Timer.java:462)

"Thread-20" prio=1 tid=0x0000002b33540de0 nid=0x604a runnable [0x0000000042a7b000..0x0000000042a7bab0]

at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)

- locked <0x0000002aad86b6f0> (a java.net.SocksSocketImpl)

   at java.net.ServerSocket.implAccept(ServerSocket.java:450)
   at java.net.ServerSocket.accept(ServerSocket.java:421)
   at com.controltier.reporter.services.InputServer$worker.run(InputServer.java:81)
   at java.lang.Thread.run(Thread.java:595)

"Store org.hibernate.cache.StandardQueryCache Spool Thread" daemon prio=1 tid=0x0000002b386b84e0 nid=0x6049 waiting on condition [0x000000004297a000..0x000000004297ae30]

   at java.lang.Thread.sleep(Native Method)
   at net.sf.ehcache.store.DiskStore.spoolAndExpiryThreadMain(DiskStore.java:589)
   at net.sf.ehcache.store.DiskStore.access$800(DiskStore.java:65)
   at net.sf.ehcache.store.DiskStore$SpoolAndExpiryThread.run(DiskStore.java:1075)

"Store org.hibernate.cache.UpdateTimestampsCache Spool Thread" daemon prio=1 tid=0x0000002b33abc310 nid=0x6048 waiting on condition [0x0000000042879000..0x0000000042879db0]

   at java.lang.Thread.sleep(Native Method)
   at net.sf.ehcache.store.DiskStore.spoolAndExpiryThreadMain(DiskStore.java:589)
   at net.sf.ehcache.store.DiskStore.access$800(DiskStore.java:65)
   at net.sf.ehcache.store.DiskStore$SpoolAndExpiryThread.run(DiskStore.java:1075)

"HSQLDB Timer @2cfa930d" daemon prio=1 tid=0x0000002b33ac76b0 nid=0x6047 in Object.wait() [0x0000000042778000..0x0000000042778d30]
at java.lang.Object.wait(Native Method)
- waiting on <0x0000002aad86c128> (a org.hsqldb.lib.HsqlTimer)

at java.lang.Object.wait(Object.java:474)
at org.hsqldb.lib.HsqlTimer.nextTask(Unknown Source)

- locked <0x0000002aad86c128> (a org.hsqldb.lib.HsqlTimer)

at org.hsqldb.lib.HsqlTimer$TaskRunner.run(Unknown Source)
at java.lang.Thread.run(Thread.java:595)

"Timer-5" daemon prio=1 tid=0x0000002b36340570 nid=0x6045 in Object.wait() [0x0000000042677000..0x0000000042677cb0]
   at java.lang.Object.wait(Native Method)
   - waiting on <0x0000002aad8bbcb8> (a java.util.TaskQueue)
   at java.util.TimerThread.mainLoop(Timer.java:509)
   - locked <0x0000002aad8bbcb8> (a java.util.TaskQueue)
   at java.util.TimerThread.run(Timer.java:462)

"RMI TCP Accept-1099" daemon prio=1 tid=0x0000002b30c68dd0 nid=0x6044 runnable [0x0000000042576000..0x0000000042576c30]

at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)

- locked <0x0000002aad8bc638> (a java.net.SocksSocketImpl)

   at java.net.ServerSocket.implAccept(ServerSocket.java:450)
   at java.net.ServerSocket.accept(ServerSocket.java:421)
   at sun.rmi.transport.tcp.TCPTransport.run(TCPTransport.java:340)
   at java.lang.Thread.run(Thread.java:595)

"GC Daemon" daemon prio=1 tid=0x0000002b332fc7c0 nid=0x6043 in Object.wait() [0x0000000042475000..0x0000000042475bb0]
   at java.lang.Object.wait(Native Method)
   - waiting on <0x0000002aac506f30> (a sun.misc.GC$LatencyLock)
   at sun.misc.GC$Daemon.run(GC.java:100)
   - locked <0x0000002aac506f30> (a sun.misc.GC$LatencyLock)

"RMI Reaper" prio=1 tid=0x0000002b342fab20 nid=0x6042 in Object.wait() [0x0000000042374000..0x0000000042374b30]
   at java.lang.Object.wait(Native Method)
   - waiting on <0x0000002aac507130> (a java.lang.ref.ReferenceQueue$Lock)
   at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:120)
   - locked <0x0000002aac507130> (a java.lang.ref.ReferenceQueue$Lock)

   at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:136)
   at sun.rmi.transport.ObjectTable$Reaper.run(ObjectTable.java:336)
   at java.lang.Thread.run(Thread.java:595)

"Timer-4" daemon prio=1 tid=0x0000002b332cf5d0 nid=0x6041 in Object.wait() [0x0000000042273000..0x0000000042273ab0]
at java.lang.Object.wait(Native Method)
- waiting on <0x0000002aac4d03b0> (a java.util.TaskQueue)

at java.lang.Object.wait(Object.java:474)
at java.util.TimerThread.mainLoop(Timer.java:483)

- locked <0x0000002aac4d03b0> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:462)

"RMI TCP Accept-0" daemon prio=1 tid=0x0000002b323eec10 nid=0x6040 runnable [0x0000000042172000..0x0000000042172e30]

at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)

- locked <0x0000002aac507388> (a java.net.SocksSocketImpl)

   at java.net.ServerSocket.implAccept(ServerSocket.java:450)
   at java.net.ServerSocket.accept(ServerSocket.java:421)
   at sun.rmi.transport.tcp.TCPTransport.run(TCPTransport.java:340)
   at java.lang.Thread.run(Thread.java:595)

"ObservationManager" daemon prio=1 tid=0x0000002b323fe750 nid=0x603f in Object.wait() [0x0000000042071000..0x0000000042071db0]
at java.lang.Object.wait(Native Method)
- waiting on <0x0000002aac4d0cc0> (a org.apache.commons.collections.buffer.BlockingBuffer)

at java.lang.Object.wait(Object.java:474)
at org.apache.commons.collections.buffer.BlockingBuffer.remove(BlockingBuffer.java:107)

- locked <0x0000002aac4d0cc0> (a org.apache.commons.collections.buffer.BlockingBuffer)

at org.apache.jackrabbit.core.observation.ObservationDispatcher.run(ObservationDispatcher.java:138)
at java.lang.Thread.run(Thread.java:595)

"derby.rawStoreDaemon" daemon prio=1 tid=0x0000002b316ff560 nid=0x603e in Object.wait() [0x0000000041f70000..0x0000000041f70d30]
   at java.lang.Object.wait(Native Method)
   - waiting on <0x0000002aac507a68> (a org.apache.derby.impl.services.daemon.BasicDaemon)
   at org.apache.derby.impl.services.daemon.BasicDaemon.rest(Unknown Source)
   - locked <0x0000002aac507a68> (a org.apache.derby.impl.services.daemon.BasicDaemon)

at org.apache.derby.impl.services.daemon.BasicDaemon.run(Unknown Source)
at java.lang.Thread.run(Thread.java:595)

"derby.rawStoreDaemon" daemon prio=1 tid=0x0000002b314483e0 nid=0x603d in Object.wait() [0x0000000041e6f000..0x0000000041e6fcb0]
   at java.lang.Object.wait(Native Method)
   - waiting on <0x0000002aac58bef0> (a org.apache.derby.impl.services.daemon.BasicDaemon)
   at org.apache.derby.impl.services.daemon.BasicDaemon.rest(Unknown Source)
   - locked <0x0000002aac58bef0> (a org.apache.derby.impl.services.daemon.BasicDaemon)

at org.apache.derby.impl.services.daemon.BasicDaemon.run(Unknown Source)
at java.lang.Thread.run(Thread.java:595)

"Timer-3" daemon prio=1 tid=0x0000002b33699d00 nid=0x603c in Object.wait() [0x0000000041d6e000..0x0000000041d6ec30]
at java.lang.Object.wait(Native Method)
- waiting on <0x0000002aac633a78> (a java.util.TaskQueue)

at java.lang.Object.wait(Object.java:474)
at java.util.TimerThread.mainLoop(Timer.java:483)

   - locked <0x0000002aac633a78> (a java.util.TaskQueue)
   at java.util.TimerThread.run(Timer.java:462)

"derby.antiGC" daemon prio=1 tid=0x0000002b307375e0 nid=0x603b in Object.wait() [0x0000000041c6d000..0x0000000041c6dbb0]
   at java.lang.Object.wait(Native Method)
   - waiting on <0x0000002aac630a88> (a org.apache.derby.impl.services.monitor.AntiGC)

at java.lang.Object.wait(Object.java:474)
at org.apache.derby.impl.services.monitor.AntiGC.run(Unknown Source)

   - locked <0x0000002aac630a88> (a org.apache.derby.impl.services.monitor.AntiGC)
   at java.lang.Thread.run(Thread.java:595)

"Timer-2" daemon prio=1 tid=0x0000002b323ed660 nid=0x603a in Object.wait() [0x0000000041b6c000..0x0000000041b6cb30]
   at java.lang.Object.wait(Native Method)
   - waiting on <0x0000002aac6d3b28> (a java.util.TaskQueue)
   at java.util.TimerThread.mainLoop(Timer.java:509)
   - locked <0x0000002aac6d3b28> (a java.util.TaskQueue)
   at java.util.TimerThread.run(Timer.java:462)

"btpool0-9" prio=1 tid=0x0000002b31710970 nid=0x6039 in Object.wait() [0x0000000041a6b000..0x0000000041a6bab0]
   at java.lang.Object.wait(Native Method)
   - waiting on <0x0000002aac4cfef8> (a org.mortbay.thread.BoundedThreadPool$PoolThread)
   at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:491)
   - locked <0x0000002aac4cfef8> (a org.mortbay.thread.BoundedThreadPool$PoolThread)

"btpool0-8" prio=1 tid=0x0000002b3124de00 nid=0x6038 in Object.wait() [0x000000004196a000..0x000000004196ae30]
   at java.lang.Object.wait(Native Method)
   - waiting on <0x0000002aac4cfe58> (a org.mortbay.thread.BoundedThreadPool$PoolThread)
   at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:491)
   - locked <0x0000002aac4cfe58> (a org.mortbay.thread.BoundedThreadPool$PoolThread)

"btpool0-7" prio=1 tid=0x0000002b3124cdd0 nid=0x6037 in Object.wait() [0x0000000041869000..0x0000000041869db0]
   at java.lang.Object.wait(Native Method)
   - waiting on <0x0000002aac4cfdb8> (a org.mortbay.thread.BoundedThreadPool$PoolThread)
   at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:491)
   - locked <0x0000002aac4cfdb8> (a org.mortbay.thread.BoundedThreadPool$PoolThread)

"btpool0-6" prio=1 tid=0x0000002b3120d2a0 nid=0x6036 in Object.wait() [0x0000000041768000..0x0000000041768d30]
   at java.lang.Object.wait(Native Method)
   - waiting on <0x0000002aac4cfd18> (a org.mortbay.thread.BoundedThreadPool$PoolThread)
   at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:491)
   - locked <0x0000002aac4cfd18> (a org.mortbay.thread.BoundedThreadPool$PoolThread)

"btpool0-5" prio=1 tid=0x0000002b315d8710 nid=0x6035 in Object.wait() [0x0000000041667000..0x0000000041667cb0]
   at java.lang.Object.wait(Native Method)
   - waiting on <0x0000002aac4cfc78> (a org.mortbay.thread.BoundedThreadPool$PoolThread)
   at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:491)
   - locked <0x0000002aac4cfc78> (a org.mortbay.thread.BoundedThreadPool$PoolThread)

"btpool0-4" prio=1 tid=0x0000002b312090b0 nid=0x6034 in Object.wait() [0x0000000041566000..0x0000000041566c30]
   at java.lang.Object.wait(Native Method)
   - waiting on <0x0000002aac4cfbd8> (a org.mortbay.thread.BoundedThreadPool$PoolThread)
   at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:491)
   - locked <0x0000002aac4cfbd8> (a org.mortbay.thread.BoundedThreadPool$PoolThread)

"btpool0-3" prio=1 tid=0x0000002b312087f0 nid=0x6033 in Object.wait() [0x0000000041465000..0x0000000041465bb0]
   at java.lang.Object.wait(Native Method)
   - waiting on <0x0000002aac4cfb38> (a org.mortbay.thread.BoundedThreadPool$PoolThread)
   at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:491)
   - locked <0x0000002aac4cfb38> (a org.mortbay.thread.BoundedThreadPool$PoolThread)

"btpool0-2 - Acceptor1 SelectChann...@0.0.0.0:8080" prio=1 tid=0x0000002b31208020 nid=0x6032 runnable [0x0000000041364000..0x0000000041364b30]

   at sun.nio.ch.PollArrayWrapper.poll0(Native Method)
   at sun.nio.ch.PollArrayWrapper.poll(PollArrayWrapper.java:100)
   at sun.nio.ch.PollSelectorImpl.doSelect(PollSelectorImpl.java:56)
   at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)

   - locked <0x0000002ab3648798> (a sun.nio.ch.Util$1)
   - locked <0x0000002ab3648780> (a java.util.Collections$UnmodifiableSet)
   - locked <0x0000002ab36485a0> (a sun.nio.ch.PollSelectorImpl)

   at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
   at org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:429)
   at org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:185)
   at org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
   at org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:707)
   at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:451)

"btpool0-1" prio=1 tid=0x0000002b31224580 nid=0x6031 in Object.wait() [0x0000000041263000..0x0000000041263ab0]
   at java.lang.Object.wait(Native Method)
   - waiting on <0x0000002aac4cf9f8> (a org.mortbay.thread.BoundedThreadPool$PoolThread)
   at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:491)
   - locked <0x0000002aac4cf9f8> (a org.mortbay.thread.BoundedThreadPool$PoolThread)

"btpool0-0 - Acceptor0 SelectChann...@0.0.0.0:8080" prio=1 tid=0x0000002b31223e90 nid=0x6030 runnable [0x0000000041162000..0x0000000041162e30]

   at sun.nio.ch.PollArrayWrapper.poll0(Native Method)
   at sun.nio.ch.PollArrayWrapper.poll(PollArrayWrapper.java:100)
   at sun.nio.ch.PollSelectorImpl.doSelect(PollSelectorImpl.java:56)
   at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)

   - locked <0x0000002ab36489b0> (a sun.nio.ch.Util$1)
   - locked <0x0000002ab3648998> (a java.util.Collections$UnmodifiableSet)
   - locked <0x0000002ab3648628> (a sun.nio.ch.PollSelectorImpl)

   at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
   at org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:429)
   at org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:185)
   at org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
   at org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:707)
   at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:451)

"Timer-1" daemon prio=1 tid=0x0000002b31225060 nid=0x602f in Object.wait() [0x0000000041061000..0x0000000041061db0]
   at java.lang.Object.wait(Native Method)
   - waiting on <0x0000002aac7b6850> (a java.util.TaskQueue)
   at java.util.TimerThread.mainLoop(Timer.java:509)
   - locked <0x0000002aac7b6850> (a java.util.TaskQueue)
   at java.util.TimerThread.run(Timer.java:462)

"Timer-0" daemon prio=1 tid=0x0000002b30cfd1c0 nid=0x602e in Object.wait() [0x0000000040f60000..0x0000000040f60d30]
   at java.lang.Object.wait(Native Method)
   - waiting on <0x0000002aac7bc4a8> (a java.util.TaskQueue)
   at java.util.TimerThread.mainLoop(Timer.java:509)
   - locked <0x0000002aac7bc4a8> (a java.util.TaskQueue)
   at java.util.TimerThread.run(Timer.java:462)

"Low Memory Detector" daemon prio=1 tid=0x0000002b2cd41c80 nid=0x602c runnable [0x0000000000000000..0x0000000000000000]

"CompilerThread1" daemon prio=1 tid=0x0000002b2cd400e0 nid=0x602b waiting on condition [0x0000000000000000..0x0000000040c5c4e0]

"CompilerThread0" daemon prio=1 tid=0x0000002b2cd3ebc0 nid=0x602a waiting on condition [0x0000000000000000..0x0000000040b5b450]

"AdapterThread" daemon prio=1 tid=0x0000002b2cd3d4f0 nid=0x6029 waiting on condition [0x0000000000000000..0x0000000000000000]

"Signal Dispatcher" daemon prio=1 tid=0x0000002b2cd3c280 nid=0x6028 waiting on condition [0x0000000000000000..0x0000000000000000]

"Finalizer" daemon prio=1 tid=0x0000002b2cd28150 nid=0x6027 in Object.wait() [0x0000000040859000..0x0000000040859db0]
   at java.lang.Object.wait(Native Method)
   - waiting on <0x0000002aac7bdba8> (a java.lang.ref.ReferenceQueue$Lock)
   at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:120)
   - locked <0x0000002aac7bdba8> (a java.lang.ref.ReferenceQueue$Lock)

at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:136)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)

"Reference Handler" daemon prio=1 tid=0x0000002b2cd279b0 nid=0x6026 in Object.wait() [0x0000000040758000..0x0000000040758d30]
at java.lang.Object.wait(Native Method)
- waiting on <0x0000002aac4a05b0> (a java.lang.ref.Reference$Lock)

at java.lang.Object.wait(Object.java:474)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)

- locked <0x0000002aac4a05b0> (a java.lang.ref.Reference$Lock)

"VM Thread" prio=1 tid=0x0000002b2cd234c0 nid=0x6025 runnable

"GC task thread#0 (ParallelGC)" prio=1 tid=0x0000000040134010 nid=0x6021 runnable

"GC task thread#1 (ParallelGC)" prio=1 tid=0x0000000040135660 nid=0x6022 runnable

"GC task thread#2 (ParallelGC)" prio=1 tid=0x00000000401364b0 nid=0x6023 runnable

"GC task thread#3 (ParallelGC)" prio=1 tid=0x0000000040137330 nid=0x6024 runnable

"VM Periodic Task Thread" prio=1 tid=0x000000004012af60 nid=0x602d waiting on condition

62 threads in each case.

Thanks,

Anthony.

Anthony Shortland

unread,

Oct 14, 2010, 3:57:09 PM10/14/10

to ControlTier Accounting

Ran jstat while running an iteration of the command:

[dnbiadm@gplnxv8 jetty-6.1.14]$ jstat -gccause 24607 5s

S0 S1 E O P YGC YGCT FGC FGCT GCT LGCC GCC

0.00 0.00 0.67 33.77 84.53 27 1.710 16 13.293 15.003 System.gc() No GC

0.00 0.00 0.69 33.77 84.53 27 1.710 16 13.293 15.003 System.gc() No GC

0.00 0.00 0.20 33.77 86.27 28 1.719 17 14.361 16.080 System.gc() No GC

0.00 0.00 0.50 33.77 86.27 28 1.719 17 14.361 16.080 System.gc() No GC

0.00 0.00 0.52 33.77 86.27 28 1.719 17 14.361 16.080 System.gc() No GC

0.00 0.00 0.54 33.77 86.27 28 1.719 17 14.361 16.080 System.gc() No GC

0.00 0.00 6.55 33.77 88.80 28 1.719 17 14.361 16.080 System.gc() No GC

0.00 0.00 37.00 33.77 88.98 28 1.719 17 14.361 16.080 System.gc() No GC

0.00 99.93 64.87 33.77 88.98 29 1.839 17 14.361 16.200 unknown GCCause No GC

0.00 99.99 11.09 39.36 88.98 31 2.374 17 14.361 16.735 unknown GCCause No GC

0.00 48.21 16.53 56.95 88.98 33 3.070 17 14.361 17.431 unknown GCCause No GC

0.00 22.36 60.23 75.24 88.98 35 3.573 17 14.361 17.934 unknown GCCause No GC

0.00 28.06 0.00 88.23 88.98 37 3.981 18 14.361 18.342 Allocation Failure unknown GCCause

32.28 0.00 93.84 51.53 88.05 38 4.067 18 16.524 20.591 unknown GCCause No GC

37.62 20.19 100.00 60.13 88.05 41 4.501 18 16.524 21.025 unknown GCCause Allocation Failure

0.00 43.79 4.02 68.64 88.05 43 5.187 18 16.524 21.711 unknown GCCause No GC

64.94 37.27 100.00 77.29 88.05 45 5.442 18 16.524 21.966 unknown GCCause Allocation Failure

49.62 23.79 100.00 83.32 88.05 47 5.955 18 16.524 22.479 unknown GCCause Allocation Failure

54.78 0.00 94.45 89.12 88.05 48 6.451 18 16.524 22.975 unknown GCCause No GC

59.91 0.00 0.00 96.86 88.05 50 6.936 19 16.524 23.460 Allocation Failure unknown GCCause

0.00 73.54 16.08 63.38 89.35 51 7.327 19 20.211 27.538 unknown GCCause No GC

65.32 48.24 100.00 68.25 89.36 53 7.607 19 20.211 27.818 unknown GCCause Allocation Failure

71.39 0.00 91.20 71.39 89.40 54 8.143 19 20.211 28.354 unknown GCCause No GC

53.10 0.00 81.13 76.33 89.41 56 8.602 19 20.211 28.812 unknown GCCause No GC

0.00 100.00 66.12 81.15 90.06 59 8.998 19 20.211 29.209 unknown GCCause No GC

0.00 0.60 55.25 81.17 90.14 65 9.270 19 20.211 29.480 unknown GCCause No GC

0.00 0.80 68.33 81.17 90.38 67 9.366 19 20.211 29.576 unknown GCCause No GC

0.00 0.80 68.35 81.17 90.38 67 9.366 19 20.211 29.576 unknown GCCause No GC

0.00 0.80 68.37 81.17 90.38 67 9.366 19 20.211 29.576 unknown GCCause No GC

Anthony.

Reply all

Reply to author

Forward

0 new messages