what is the case of so many overlord threads?

[qm...@sohu.com]

Dear ALL:

      

       I have a druid test cluster that have four servers. A transqulity-firehose  ingest real-time data to them every time.

      

       Today, one server does not work well, i use the ps -eLf to  list the thred ,i find many overlord threads .

     

       tServer-4:~ # ps -ef|grep druid      ----show the druid process

20035    13809     1 28 Apr21 ?        1-21:54:05 java -server -Xms16g -Xmx16g -XX:MaxDirectMemorySize=24g -Duser.timezone=UTC+8 -Dfile.encoding=UTF-8 -Djava.io.tmpdir=var/tmp -Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager -cp conf/druid/_common:conf/druid/historical:lib/* io.druid.cli.Main server historical

20035    13815     1  0 Apr21 ?        00:01:46 java -server -Xms2g -Xmx2g -Duser.timezone=UTC+8 -Dfile.encoding=UTF-8 -Djava.io.tmpdir=var/tmp -Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager -cp conf/druid/_common:conf/druid/middleManager:lib/* io.druid.cli.Main server middleManager

20035    13821     1  0 Apr21 ?        00:25:22 java -server -Xms2g -Xmx2g -Duser.timezone=UTC+8 -Dfile.encoding=UTF-8 -Djava.io.tmpdir=var/tmp -Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager -Dderby.stream.error.file=var/druid/derby.log -cp conf/druid/_common:conf/druid/coordinator:lib/* io.druid.cli.Main server coordinator

20035    13829     1  0 Apr21 ?        00:22:18 java -server -Xms2g -Xmx2g -Duser.timezone=UTC+8 -Dfile.encoding=UTF-8 -Djava.io.tmpdir=var/tmp -Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager -cp conf/druid/_common:conf/druid/overlord:lib/* io.druid.cli.Main server overlord

Server-4:~ # ps -eLf    -----show the thread, thera are many overlord threads

20035    13829     1  6269  0 47376 Apr25 ?       00:00:00 java -server -Xms2g -Xmx2g -Duser.timezone=UTC+8 -Dfile.encoding=UTF-8 -D

20035    13829     1  6270  0 47376 Apr25 ?       00:00:00 java -server -Xms2g -Xmx2g -Duser.timezone=UTC+8 -Dfile.encoding=UTF-8 -D

20035    13829     1  6271  0 47376 Apr25 ?       00:00:00 java -server -Xms2g -Xmx2g -Duser.timezone=UTC+8 -Dfile.encoding=UTF-8 -D

20035    13829     1  6272  0 47376 Apr25 ?       00:00:00 java -server -Xms2g -Xmx2g -Duser.timezone=UTC+8 -Dfile.encoding=UTF-8 -D

20035    13829     1  6273  0 47376 Apr25 ?       00:00:00 java -server -Xms2g -Xmx2g -Duser.timezone=UTC+8 -Dfile.encoding=UTF-8 -D

20035    13829     1  6274  0 47376 Apr25 ?       00:00:00 java -server -Xms2g -Xmx2g -Duser.timezone=UTC+8 -Dfile.encoding=UTF-8 -D

20035    13829     1  6275  0 47376 Apr25 ?       00:00:00 java -server -Xms2g -Xmx2g -Duser.timezone=UTC+8 -Dfile.encoding=UTF-8 -D

20035    13829     1  6276  0 47376 Apr25 ?       00:00:00 java -server -Xms2g -Xmx2g -Duser.timezone=UTC+8 -Dfile.encoding=UTF-8 -D

20035    13829     1  6277  0 47376 Apr25 ?       00:00:00 java -server -Xms2g -Xmx2g -Duser.timezone=UTC+8 -Dfile.encoding=UTF-8 -D

20035    13829     1  6278  0 47376 Apr25 ?       00:00:00 java -server -Xms2g -Xmx2g -Duser.timezone=UTC+8 -Dfile.encoding=UTF-8 -D

20035    13829     1  6279  0 47376 Apr25 ?       00:00:00 java -server -Xms2g -Xmx2g -Duser.timezone=UTC+8 -Dfile.encoding=UTF-8 -D

20035    13829     1  6281  0 47376 Apr25 ?       00:00:00 java -server -Xms2g -Xmx2g -Duser.timezone=UTC+8 -Dfile.encoding=UTF-8 -D

20035    13829     1  6282  0 47376 Apr25 ?       00:00:00 java -server -Xms2g -Xmx2g -Duser.timezone=UTC+8 -Dfile.encoding=UTF-8 -D

20035    13829     1  6283  0 47376 Apr25 ?       00:00:00 java -server -Xms2g -Xmx2g -Duser.timezone=UTC+8 -Dfile.encoding=UTF-8 -D

20035    13829     1  6284  0 47376 Apr25 ?       00:00:00 java -server -Xms2g -Xmx2g -Duser.timezone=UTC+8 -Dfile.encoding=UTF-8 -D

Server-4:~ # ps -eLf |grep 13829 |wc -l   ------there are 47377 overlord threads,my god,it is too much!!!

47377

the overlord config are list:

    druid.service=druid/overlord

    druid.port=8090

    druid.indexer.queue.startDelay=PT30S

    druid.indexer.runner.type=remote

    druid.indexer.storage.type=metadata

    #timeout to kill the task when the task state is dead

    druid.indexer.autoscale.workerIdleTimeout = PT15M

    druid.indexer.autoscale.doAutoscale = true

I have no sense of the question, who/which application make so many overlord?   Is it the transqulity-firehose? 

Pls help me,thanks 

[Gian Merlino]

That does look like a lot of threads… try running "jstack -l 13829" which should tell us what they're doing.

Gian

--
You received this message because you are subscribed to the Google Groups "Druid Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to druid-development+unsubscribe@googlegroups.com.
To post to this group, send email to druid-development@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/druid-development/c700536c-6fa8-4a60-b69b-17aba8ac6cc9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[qm...@sohu.com]

the overload 13829 which have 47373 threads was dead by itself...

but there are same case in other  three servers. I use "jstack -l xxxxxxx" and get the information logs.  Typical items are listed( many information are same,so I just pick up some different in every log)

server1 :   

there are  368 thread in overolrd : the jstack  informations are:

"qtp1692092775-300" #300 daemon prio=5 os_prio=0 tid=0x00007fa6f8363000 nid=0x9a4c waiting on condition [0x00007fa6da81e000]

   java.lang.Thread.State: TIMED_WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000805d77e8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)

        at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)

        at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:516)

        at org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)

        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:575)

        at java.lang.Thread.run(Thread.java:745)

   Locked ownable synchronizers:

        - None

"qtp1692092775-206-acceptor-3@689faf79-ServerConnector@2100d047{HTTP/1.1}{0.0.0.0:8090}" #206 daemon prio=4 os_prio=0 tid=0x00007fa6

f04e3000 nid=0x99ed waiting for monitor entry [0x00007fa6e067c000]

   java.lang.Thread.State: BLOCKED (on object monitor)

        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:234)

        - waiting to lock <0x00000000fab9d5e0> (a java.lang.Object)

        at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:377)

        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:500)

        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:620)

        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:540)

        at java.lang.Thread.run(Thread.java:745)

   Locked ownable synchronizers:

        - None

"qtp1692092775-203-acceptor-0@3b11620a-ServerConnector@2100d047{HTTP/1.1}{0.0.0.0:8090}" #203 daemon prio=4 os_prio=0 tid=0x00007fa6

f08d8000 nid=0x99ea runnable [0x00007fa6e097f000]

   java.lang.Thread.State: RUNNABLE

        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)

        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)

        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)

        - locked <0x00000000fab9d5e0> (a java.lang.Object)

        at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:377)

        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:500)

        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:620)

        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:540)

        at java.lang.Thread.run(Thread.java:745)

   Locked ownable synchronizers:

        - None

"qtp1692092775-201-selector-ServerConnectorManager@47be0f9b/2" #201 daemon prio=5 os_prio=0 tid=0x00007fa6f08d5000 nid=0x99e8 runnab

le [0x00007fa6e0b81000]

   java.lang.Thread.State: RUNNABLE

        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)

        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)

        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)

        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

        - locked <0x00000000fab9fe18> (a sun.nio.ch.Util$2)

        - locked <0x00000000fab9fe08> (a java.util.Collections$UnmodifiableSet)

        - locked <0x00000000fab9fce0> (a sun.nio.ch.EPollSelectorImpl)

        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)

        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)

        at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:596)

        at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:545)

        at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)

        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:620)

        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:540)

        at java.lang.Thread.run(Thread.java:745)

   Locked ownable synchronizers:

        - None

"Curator-LeaderSelector-0" #197 daemon prio=5 os_prio=0 tid=0x00007fa6f0001000 nid=0x99e4 in Object.wait() [0x00007fa6e0e83000]

   java.lang.Thread.State: WAITING (on object monitor)

        at java.lang.Object.wait(Native Method)

        - waiting on <0x00000000fab9a830> (a org.apache.curator.framework.recipes.locks.LockInternals)

        at java.lang.Object.wait(Object.java:502)

        at org.apache.curator.framework.recipes.locks.LockInternals.internalLockLoop(LockInternals.java:308)

        - locked <0x00000000fab9a830> (a org.apache.curator.framework.recipes.locks.LockInternals)

        at org.apache.curator.framework.recipes.locks.LockInternals.attemptLock(LockInternals.java:218)

        at org.apache.curator.framework.recipes.locks.InterProcessMutex.internalLock(InterProcessMutex.java:232)

        at org.apache.curator.framework.recipes.locks.InterProcessMutex.acquire(InterProcessMutex.java:89)

        at org.apache.curator.framework.recipes.leader.LeaderSelector.doWork(LeaderSelector.java:386)

        at org.apache.curator.framework.recipes.leader.LeaderSelector.doWorkLoop(LeaderSelector.java:441)

        at org.apache.curator.framework.recipes.leader.LeaderSelector.access$100(LeaderSelector.java:64)

        at org.apache.curator.framework.recipes.leader.LeaderSelector$2.call(LeaderSelector.java:245)

        at org.apache.curator.framework.recipes.leader.LeaderSelector$2.call(LeaderSelector.java:239)

        at java.util.concurrent.FutureTask.run(FutureTask.java:266)

        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

        at java.util.concurrent.FutureTask.run(FutureTask.java:266)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

        at java.lang.Thread.run(Thread.java:745)

   Locked ownable synchronizers:

        - <0x00000000fab9a260> (a java.util.concurrent.ThreadPoolExecutor$Worker)

"HttpClient-Timer-0" #28 daemon prio=5 os_prio=0 tid=0x00007fa6f459a000 nid=0x99e3 sleeping[0x00007fa6e0f85000]

   java.lang.Thread.State: TIMED_WAITING (sleeping)

        at java.lang.Thread.sleep(Native Method)

        at org.jboss.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:445)

        at org.jboss.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:364)

        at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)

        at java.lang.Thread.run(Thread.java:745)

   Locked ownable synchronizers:

        - None

"Curator-Framework-0" #196 daemon prio=5 os_prio=0 tid=0x00007fa6f459d000 nid=0x99e2 waiting on condition [0x00007fa6e1086000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x0000000080641168> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at java.util.concurrent.DelayQueue.take(DelayQueue.java:211)

        at java.util.concurrent.DelayQueue.take(DelayQueue.java:70)

        at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:804)

        at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:64)

        at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:267)

        at java.util.concurrent.FutureTask.run(FutureTask.java:266)

        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)

        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

        at java.lang.Thread.run(Thread.java:745)

   Locked ownable synchronizers:

        - <0x00000000fab2e1c8> (a java.util.concurrent.ThreadPoolExecutor$Worker)

"main-EventThread" #195 daemon prio=5 os_prio=0 tid=0x00007fa6f4891800 nid=0x99e1 waiting on condition [0x00007fa6e1187000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000fac92598> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501)

   Locked ownable synchronizers:

        - None

"main-SendThread(10.137.66.85:24002)" #194 daemon prio=5 os_prio=0 tid=0x00007fa6f4597000 nid=0x99e0 runnable [0x00007fa6e1288000]

   java.lang.Thread.State: RUNNABLE

        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)

        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)

        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)

        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

        - locked <0x00000000fac91550> (a sun.nio.ch.Util$2)

        - locked <0x00000000fac91540> (a java.util.Collections$UnmodifiableSet)

        - locked <0x00000000fac910d8> (a sun.nio.ch.EPollSelectorImpl)

        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)

        at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)

        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)

   Locked ownable synchronizers:

        - None

"HttpClient-Netty-Worker-154" #184 daemon prio=5 os_prio=0 tid=0x00007fa6fa476800 nid=0x9926 runnable [0x00007fa6e2092000]

   java.lang.Thread.State: RUNNABLE

        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)

        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)

        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)

        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

        - locked <0x00000000fad865b8> (a sun.nio.ch.Util$2)

        - locked <0x00000000fad865a8> (a java.util.Collections$UnmodifiableSet)

        - locked <0x00000000fad865c8> (a sun.nio.ch.EPollSelectorImpl)

        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)

        at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68)

        at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:434)

        at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)

        at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)

        at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)

        at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)

        at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

        at java.lang.Thread.run(Thread.java:745)

   Locked ownable synchronizers:

        - <0x00000000fad86698> (a java.util.concurrent.ThreadPoolExecutor$Worker)

"HttpClient-Netty-Boss-0" #29 daemon prio=5 os_prio=0 tid=0x00007fa6f8a71800 nid=0x985b runnable [0x00007fa6eca4d000]

   java.lang.Thread.State: RUNNABLE

        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)

        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)

        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)

        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

        - locked <0x00000000fab062c8> (a sun.nio.ch.Util$2)

        - locked <0x00000000fab062b8> (a java.util.Collections$UnmodifiableSet)

        - locked <0x00000000fab06190> (a sun.nio.ch.EPollSelectorImpl)

        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)

        at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68)

        at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:434)

        at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)

        at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)

        at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)

        at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

        at java.lang.Thread.run(Thread.java:745)

   Locked ownable synchronizers:

        - <0x00000000fab05e50> (a java.util.concurrent.ThreadPoolExecutor$Worker)

"C2 CompilerThread6" #11 daemon prio=9 os_prio=0 tid=0x00007fa6f8011800 nid=0x9496 waiting on condition [0x0000000000000000]

   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:

        - None

"GC task thread#50 (ParallelGC)" os_prio=0 tid=0x000000000067f800 nid=0x93ce runnable

"GC task thread#51 (ParallelGC)" os_prio=0 tid=0x0000000000681000 nid=0x93d1 runnable

"GC task thread#52 (ParallelGC)" os_prio=0 tid=0x0000000000683000 nid=0x93d2 runnable

"VM Periodic Task Thread" os_prio=0 tid=0x00007fa6f0009800 nid=0x94af waiting on condition

JNI global references: 283

server2

there are  613 thread in overolrd : the jstack  informations are: 

Thread 117643: (state = IN_NATIVE)

 - sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0 (Compiled frame; information may be imprecise)

 - sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=269 (Compiled frame)

 - sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=93 (Compiled frame)

 - sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=86 (Compiled frame)

 - sun.nio.ch.SelectorImpl.select(long) @bci=30, line=97 (Compiled frame)

 - org.jboss.netty.channel.socket.nio.SelectorUtil.select(java.nio.channels.Selector) @bci=4, line=68 (Compiled frame)

 - org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(java.nio.channels.Selector) @bci=1, line=434 (Compiled frame)

 - org.jboss.netty.channel.socket.nio.AbstractNioSelector.run() @bci=56, line=212 (Compiled frame)

 - org.jboss.netty.channel.socket.nio.AbstractNioWorker.run() @bci=1, line=89 (Interpreted frame)

 - org.jboss.netty.channel.socket.nio.NioWorker.run() @bci=1, line=178 (Interpreted frame)

 - org.jboss.netty.util.ThreadRenamingRunnable.run() @bci=55, line=108 (Interpreted frame)

 - org.jboss.netty.util.internal.DeadLockProofWorker$1.run() @bci=14, line=42 (Interpreted frame)

 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=95, line=1142 (Interpreted

 frame)

 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame)

 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)

 

 

 Thread 116090: (state = BLOCKED)

 - java.lang.Object.wait(long) @bci=0 (Interpreted frame)

 - java.lang.Object.wait() @bci=2, line=502 (Interpreted frame)

 - java.lang.ref.Reference.tryHandlePending(boolean) @bci=54, line=191 (Interpreted frame)

 - java.lang.ref.Reference$ReferenceHandler.run() @bci=1, line=153 (Interpreted frame)

 

server3

there are  309 thread in overolrd : the jstack  informations are:

sun.jvm.hotspot.debugger.DebuggerException: sun.jvm.hotspot.runtime.VMVersionMismatchException: Supported versions are 25.112-b15. Target VM is 25.72-b15

        at sun.jvm.hotspot.HotSpotAgent.setupVM(HotSpotAgent.java:435)

        at sun.jvm.hotspot.HotSpotAgent.go(HotSpotAgent.java:305)

        at sun.jvm.hotspot.HotSpotAgent.attach(HotSpotAgent.java:140)

        at sun.jvm.hotspot.tools.Tool.start(Tool.java:185)

        at sun.jvm.hotspot.tools.Tool.execute(Tool.java:118)

        at sun.jvm.hotspot.tools.JStack.main(JStack.java:92)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        at java.lang.reflect.Method.invoke(Method.java:498)

        at sun.tools.jstack.JStack.runJStackTool(JStack.java:140)

        at sun.tools.jstack.JStack.main(JStack.java:106)

Caused by: sun.jvm.hotspot.runtime.VMVersionMismatchException: Supported versions are 25.112-b15. Target VM is 25.72-b15

        at sun.jvm.hotspot.runtime.VM.checkVMVersion(VM.java:227)

        at sun.jvm.hotspot.runtime.VM.<init>(VM.java:294)

        at sun.jvm.hotspot.runtime.VM.initialize(VM.java:370)

        at sun.jvm.hotspot.HotSpotAgent.setupVM(HotSpotAgent.java:431)

        ... 11 more

 

在 2017年4月28日星期五 UTC+8上午11:36:32，qm...@sohu.com写道：