Thanks for the response Darra, when we had dumped the jstack, we see many following threads(almost 400+):
"Thread-3020 (ActiveMQ-remoting-threads-ActiveMQServerImpl::serverUUID=49a160ae-59ec-11ed-bf2c-1e82d8e109d1-201407772)" #6789 prio=5 os_prio=0 tid=0x00007fb4b6f99800 nid=0x152dc waiting on condition [0x00007fb4971c1000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x000000070611c6e0> (a java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
And as indicated earlier, we see lot of "anon" output in pmap output:
00007f6c22301000 1016 24 24 rw--- [ anon ]
00007f6c223ff000 12 0 0 ----- [ anon ]
00007f6c22402000 1016 124 124 rw--- [ anon ]
00007f6c22500000 12 0 0 ----- [ anon ]
00007f6c22503000 1016 88 88 rw--- [ anon ]
00007f6c22601000 12 0 0 ----- [ anon ]
00007f6c22604000 3064 1908 1908 rw--- [ anon ]
00007f6c22902000 12 0 0 ----- [ anon ]
00007f6c22905000 1016 92 92 rw--- [ anon ]
00007f6c22a03000 12 0 0 ----- [ anon ]
00007f6c22a06000 1016 92 92 rw--- [ anon ]
00007f6c22b04000 12 0 0 ----- [ anon ]
00007f6c22b07000 1016 92 92 rw--- [ anon ]
00007f6c22c05000 12 0 0 ----- [ anon ]
00007f6c22c08000 1016 100 100 rw--- [ anon ]
00007f6c22d06000 12 0 0 ----- [ anon ]
00007f6c22d09000 1016 92 92 rw--- [ anon ]
00007f6c22e07000 12 0 0 ----- [ anon ]
00007f6c22e0a000 1016 92 92 rw--- [ anon ]
00007f6c22f08000 12 0 0 ----- [ anon ]
00007f6c22f0b000 1016 96 96 rw--- [ anon ]
00007f6c23009000 12 0 0 ----- [ anon ]
00007f6c2300c000 1016 96 96 rw--- [ anon ]
00007f6c2310a000 12 0 0 ----- [ anon ]
00007f6c2310d000 3064 2084 2084 rw--- [ anon ]
00007f6c2340b000 12 0 0 ----- [ anon ]
00007f6c2340e000 1016 92 92 rw--- [ anon ]
We had checked and Heap Dump is remaining constant. GC is also happening. Any suggestions or help.