CPU Loading High - Lock find in elastic search

195 views

Skip to first unread message

JOHN...@yahoo.com

unread,

Jan 30, 2015, 12:54:33 AM1/30/15

to dot...@googlegroups.com

Hi dotCMS Team

We found that the CPU loading keeping 100%. The thread dumps shows there are 71 threads meet a lock. The following shows one of them (Full thread dump attached).

"elasticsearch[d44eacf4-95e3-4a9b-a8da-4ce9b0ccc5e1][transport_server_worker][T#15]{New I/O worker #66}" Id=176 RUNNABLE (in native)
      at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
      at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
      at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
      at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) - locked sun.nio.ch.Util$2@56e3c4a1 - locked java.util.Collections$UnmodifiableSet@60fc7af4 - locked sun.nio.ch.EPollSelectorImpl@1644e9d9
      at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
      at com.dotcms.repackage.org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
      at com.dotcms.repackage.org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
      at com.dotcms.repackage.org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206) ... Number of locked synchronizers = 1 - java.util.concurrent.ThreadPoolExecutor$Worker@51f6249c

Please advise how to fix this.

We are running dotCMS 3.0.

Thanks in advance.

ThreadDump_IncreasedMaxThread.txt

drdgr...@gmail.com

unread,

Feb 6, 2015, 5:01:26 AM2/6/15

to dot...@googlegroups.com, JOHN...@yahoo.com

I have this also ..

Tipping VMWare warnings all the time..

On Friday, 30 January 2015 05:54:33 UTC, JOHN...@yahoo.com wrote:

Hi dotCMS Team

We found that the CPU loading keeping 100%. The thread dumps shows there are 71 threads meet a lock. The following shows one of them (Full thread dump attached).

"elasticsearch[d44eacf4-95e3-4a9b-a8da-4ce9b0ccc5e1][transport_server_worker][T#15]{New I/O worker #66}" Id=176 RUNNABLE (in native)
      at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
      at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
      at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)

at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) - locked sun.nio....@56e3c4a1 - locked java.util.Collections$UnmodifiableSet@60fc7af4 - locked sun.nio.ch.EPo...@1644e9d9

      at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
      at com.dotcms.repackage.org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
      at com.dotcms.repackage.org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
      at com.dotcms.repackage.org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206) ... Number of locked synchronizers = 1 - java.util.concurrent.ThreadPoolExecutor$Worker@51f6249c

Jason Tesser

unread,

Feb 6, 2015, 8:01:21 AM2/6/15

to dot...@googlegroups.com, JOHN...@yahoo.com

this looks like slow disk access to me. In the upcoming 3.1 we do have an upgrade to a much later version of elasticsearch which does have some improvements but again this looks like slow disk I/O

--
You received this message because you are subscribed to the Google Groups "dotCMS User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dotcms+un...@googlegroups.com.
To post to this group, send email to dot...@googlegroups.com.
Visit this group at http://groups.google.com/group/dotcms.
For more options, visit https://groups.google.com/d/optout.

Daniel Graham

unread,

Feb 6, 2015, 8:51:33 AM2/6/15

to dot...@googlegroups.com, JOHN...@yahoo.com

So can we turn it off or throttle it.. ?

You received this message because you are subscribed to a topic in the Google Groups "dotCMS User Group" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/dotcms/Bw4UnP8tN98/unsubscribe.
To unsubscribe from this group and all its topics, send an email to dotcms+un...@googlegroups.com.

Jason Tesser

unread,

Feb 6, 2015, 9:08:29 AM2/6/15

to dot...@googlegroups.com, JOHN...@yahoo.com

no this is what we use to index. I would recommend solving it at the hardware level. The problem is the I/O is probably being throttled already by VM ware

Daniel Graham

unread,

Feb 6, 2015, 9:41:11 AM2/6/15

to dot...@googlegroups.com, JOHN...@yahoo.com

Pretty standard VMWare installation - large, expensive, SAN .. here's dotCMS using both cores so 200% CPU.

Affecting general site performance.

Jason Tesser

unread,

Feb 6, 2015, 9:49:37 AM2/6/15

to dot...@googlegroups.com, JOHN...@yahoo.com

looks like maybe you are only running with a Gig of memory for dotCMS I would recommend more. Try to goto 4 and see what happens. We ship with 1 for testing but this is not for any kind of production running

see this http://dotcms.com/docs/latest/MemoryConfiguration

Daniel Graham

unread,

Feb 6, 2015, 10:12:13 AM2/6/15

to dot...@googlegroups.com, JOHN...@yahoo.com

Thanks,

will give that a try.

Regards,

Dan

Daniel Graham

unread,

Feb 6, 2015, 10:55:40 AM2/6/15

to dot...@googlegroups.com, JOHN...@yahoo.com

With 3Gb of memory ..

Lasted about 10 mins then up to 100%

Nathan Keiter

unread,

Feb 6, 2015, 11:19:09 AM2/6/15

to dot...@googlegroups.com

Your memory is still way low IMO.

We allocate 12G to the JVM on our production VM’s which are allocated 16G total memory for OS and such.

Nathan

Nathan I. Keiter

Lead Network Applications Programmer

DataSystems

Gettysburg College

Campus Box 2453

717-337-6993

Measure your success by the level of service you provide.

Si hoc legere scis nimium eruditionis habes.

Daniel Graham

unread,

Feb 9, 2015, 4:46:08 AM2/9/15

to dot...@googlegroups.com

Note,

this is still an issue with 3Gb RAM. I'm going to try higher - but the RAM isn't being used - only 1.3Gb is being used of 3Gb - this is a very small site. Maybe 200 visitors per day.

Kind of making the site unusable now - pages served very slowly - I can chuck resource at it, but doesn't seem quite right.

Will Ezell

unread,

Feb 9, 2015, 7:18:34 AM2/9/15

to dot...@googlegroups.com

A couple of thoughts - first, can you take/share a thread dump? It makes it easier to see what's going on.

Second, try turning off the ESAPI validation in the system.properties file (just to see)

Set this and see if it makes a difference. We have seen the ESAPI library eat up your processor when hit with a strange pattern.

com.liferay.util.Xss.ESAPI.validation=false

Oh, and third, what DB are you using?

3250 Mary Street
Suite 405
Miami, FL 33133
Main: 305-900-2001 | Direct: 978.294.9429

wi...@dotcms.com | dotcms.com

Jason Tesser

unread,

Feb 9, 2015, 7:22:21 AM2/9/15

to dot...@googlegroups.com

as I said before that looks like file I/O

Daniel Graham

unread,

Feb 9, 2015, 7:27:33 AM2/9/15

to dot...@googlegroups.com

Thread Dump below, database is MySQL.

Will apply setting and report back.

Regards,

Dan

org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:501)
at org.apache.coyote.http11.Http11Processor.setRequestLineReadTimeout(Http11Processor.java:167)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:946)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607) ... Number of locked synchronizers = 1 - java.util.concurrent.ThreadPoolExecutor$Worker@4b20fab9
"http-bio-80-exec-108" Id=309 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7ba2d794
at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7ba2d794
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:86)
at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:32)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) ...
"http-bio-80-exec-107" Id=308 RUNNABLE (in native)
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:516)
at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:501)
at org.apache.coyote.http11.Http11Processor.setRequestLineReadTimeout(Http11Processor.java:167)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:946)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607) ... Number of locked synchronizers = 1 - java.util.concurrent.ThreadPoolExecutor$Worker@6e492fb2
"http-bio-80-exec-106" Id=307 RUNNABLE (in native)
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:516)
at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:501)
at org.apache.coyote.http11.Http11Processor.setRequestLineReadTimeout(Http11Processor.java:167)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:946)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607) ... Number of locked synchronizers = 1 - java.util.concurrent.ThreadPoolExecutor$Worker@70805f70
"http-bio-80-exec-105" Id=304 RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:516)
at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:501)
at org.apache.coyote.http11.Http11Processor.setRequestLineReadTimeout(Http11Processor.java:167)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:946)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607) ... Number of locked synchronizers = 1 - java.util.concurrent.ThreadPoolExecutor$Worker@5e0b6311
"http-bio-80-exec-104" Id=303 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7ba2d794
at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7ba2d794
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:86)
at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:32)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) ...
"http-bio-80-exec-102" Id=299 RUNNABLE (in native)
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:516)
at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:501)
at org.apache.coyote.http11.Http11Processor.setRequestLineReadTimeout(Http11Processor.java:167)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:946)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607) ... Number of locked synchronizers = 1 - java.util.concurrent.ThreadPoolExecutor$Worker@ca11631
"http-bio-80-exec-100" Id=297 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7ba2d794
at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7ba2d794
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:86)
at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:32)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) ...
"http-bio-80-exec-99" Id=296 RUNNABLE
at sun.management.ThreadImpl.dumpThreads0(Native Method)
at sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:440)
at com.dotcms.cmsmaintenance.ajax.ThreadMonitorTool.getAllThreadInfos(ThreadMonitorTool.java:75)
at com.dotcms.cmsmaintenance.ajax.ThreadMonitorTool.getThreads(ThreadMonitorTool.java:50)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606) ... Number of locked synchronizers = 1 - java.util.concurrent.ThreadPoolExecutor$Worker@71258cdc
"http-bio-80-exec-98" Id=295 RUNNABLE (in native)
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:516)
at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:501)
at org.apache.coyote.http11.Http11Processor.setRequestLineReadTimeout(Http11Processor.java:167)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:946)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607) ... Number of locked synchronizers = 1 - java.util.concurrent.ThreadPoolExecutor$Worker@d70ca21
"Timer-3" Id=292 TIMED_WAITING on java.util.TaskQueue@46edd212
at java.lang.Object.wait(Native Method) - waiting on java.util.TaskQueue@46edd212
at java.util.TimerThread.mainLoop(Timer.java:552)
at java.util.TimerThread.run(Timer.java:505)
"http-bio-80-exec-89" Id=285 RUNNABLE (in native)
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:516)
at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:501)
at org.apache.coyote.http11.Http11Processor.setRequestLineReadTimeout(Http11Processor.java:167)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:946)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607) ... Number of locked synchronizers = 1 - java.util.concurrent.ThreadPoolExecutor$Worker@47a5b9cf
"http-bio-80-exec-75" Id=270 RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:516)
at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:501)
at org.apache.coyote.http11.Http11Processor.setRequestLineReadTimeout(Http11Processor.java:167)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:946)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607) ... Number of locked synchronizers = 1 - java.util.concurrent.ThreadPoolExecutor$Worker@4c6397ef
"http-bio-80-exec-71" Id=266 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7ba2d794
at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7ba2d794
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:86)
at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:32)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) ...
"http-bio-80-exec-64" Id=256 RUNNABLE (in native)
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:516)
at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:501)
at org.apache.coyote.http11.Http11Processor.setRequestLineReadTimeout(Http11Processor.java:167)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:946)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607) ... Number of locked synchronizers = 1 - java.util.concurrent.ThreadPoolExecutor$Worker@11a3dbb9
"elasticsearch[715fb6e5-b6cb-4148-a5b9-b7ff9705514c][flush][T#1]" Id=248 WAITING on com.dotcms.repackage.org.elasticsearch.common.util.concurrent.EsExecutors$ExecutorScalingQueue@292b6f78
at sun.misc.Unsafe.park(Native Method) - waiting on com.dotcms.repackage.org.elasticsearch.common.util.concurrent.EsExecutors$ExecutorScalingQueue@292b6f78
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at com.dotcms.repackage.org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:706)
at com.dotcms.repackage.org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.xfer(LinkedTransferQueue.java:615)
at com.dotcms.repackage.org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.take(LinkedTransferQueue.java:1109)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) ...
"elasticsearch[715fb6e5-b6cb-4148-a5b9-b7ff9705514c][generic][T#8]" Id=234 TIMED_WAITING on java.util.concurrent.SynchronousQueue$TransferStack@2b8aacff
at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.SynchronousQueue$TransferStack@2b8aacff
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) ...
"elasticsearch[715fb6e5-b6cb-4148-a5b9-b7ff9705514c][management][T#1]" Id=233 WAITING on com.dotcms.repackage.org.elasticsearch.common.util.concurrent.EsExecutors$ExecutorScalingQueue@1d63afae
at sun.misc.Unsafe.park(Native Method) - waiting on com.dotcms.repackage.org.elasticsearch.common.util.concurrent.EsExecutors$ExecutorScalingQueue@1d63afae
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at com.dotcms.repackage.org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:706)
at com.dotcms.repackage.org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.xfer(LinkedTransferQueue.java:615)
at com.dotcms.repackage.org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.take(LinkedTransferQueue.java:1109)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) ...
"Java2D Disposer" Id=204 WAITING on java.lang.ref.ReferenceQueue$Lock@2e17b7e8
at java.lang.Object.wait(Native Method) - waiting on java.lang.ref.ReferenceQueue$Lock@2e17b7e8
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151)
at sun.java2d.Disposer.run(Disposer.java:145)
at java.lang.Thread.run(Thread.java:745)
"Thread-38" Id=197 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@172af7fc
at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@172af7fc
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) ...
"Stxx cache expiration thread" Id=194 TIMED_WAITING
at java.lang.Thread.sleep(Native Method)
at com.dotcms.repackage.com.oroad.stxx.cache.CacheManagerImpl$1.run(CacheManagerImpl.java:133)
at java.lang.Thread.run(Thread.java:745)
"elasticsearch[715fb6e5-b6cb-4148-a5b9-b7ff9705514c][search][T#4]" Id=191 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5fecd46
at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5fecd46
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
"elasticsearch[715fb6e5-b6cb-4148-a5b9-b7ff9705514c][search][T#2]" Id=193 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5fecd46
at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5fecd46
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
"elasticsearch[715fb6e5-b6cb-4148-a5b9-b7ff9705514c][search][T#3]" Id=192 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5fecd46
at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5fecd46
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
"elasticsearch[715fb6e5-b6cb-4148-a5b9-b7ff9705514c][search][T#5]" Id=190 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5fecd46
at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5fecd46
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
"elasticsearch[715fb6e5-b6cb-4148-a5b9-b7ff9705514c][search][T#6]" Id=189 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5fecd46
at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5fecd46
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
"elasticsearch[715fb6e5-b6cb-4148-a5b9-b7ff9705514c][search][T#1]" Id=188 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5fecd46
at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5fecd46
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
"http-bio-80-exec-8" Id=177 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7ba2d794
at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7ba2d794
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:86)
at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:32)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) ...
"http-bio-80-exec-3" Id=172 RUNNABLE
at java.util.regex.Pattern$5.isSatisfiedBy(Pattern.java:5151)
at java.util.regex.Pattern$5.isSatisfiedBy(Pattern.java:5151)
at java.util.regex.Pattern$5.isSatisfiedBy(Pattern.java:5151)
at java.util.regex.Pattern$5.isSatisfiedBy(Pattern.java:5151)
at java.util.regex.Pattern$5.isSatisfiedBy(Pattern.java:5151)
at java.util.regex.Pattern$5.isSatisfiedBy(Pattern.java:5151)
at java.util.regex.Pattern$5.isSatisfiedBy(Pattern.java:5151)
at java.util.regex.Pattern$5.isSatisfiedBy(Pattern.java:5151) ... Number of locked synchronizers = 1 - java.util.concurrent.ThreadPoolExecutor$Worker@67f9af69
"http-bio-80-AsyncTimeout" Id=168 TIMED_WAITING
at java.lang.Thread.sleep(Native Method)
at org.apache.tomcat.util.net.JIoEndpoint$AsyncTimeout.run(JIoEndpoint.java:152)
at java.lang.Thread.run(Thread.java:745)
"http-bio-80-Acceptor-0" Id=167 RUNNABLE (in native)
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
at java.net.ServerSocket.implAccept(ServerSocket.java:530)
at java.net.ServerSocket.accept(ServerSocket.java:498)
at org.apache.tomcat.util.net.DefaultServerSocketFactory.acceptSocket(DefaultServerSocketFactory.java:60)
at org.apache.tomcat.util.net.JIoEndpoint$Acceptor.run(JIoEndpoint.java:220)
at java.lang.Thread.run(Thread.java:745)
"ContainerBackgroundProcessor[StandardEngine[Catalina]]" Id=166 TIMED_WAITING
at java.lang.Thread.sleep(Native Method)
at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1508)
at java.lang.Thread.run(Thread.java:745)
"pool-4-thread-1" Id=165 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@15b8f8bc
at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@15b8f8bc
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
"pool-3-thread-1" Id=164 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@22d209c8
at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@22d209c8
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
"Thread-35" Id=163 TIMED_WAITING
at java.lang.Thread.sleep(Native Method)
at com.dotcms.workflow.EscalationThread.run(EscalationThread.java:86)
"QuartzScheduler_DotMarketingQuartzExclusiveScheduler-715fb6e5-b6cb-4148-a5b9-b7ff9705514c_MisfireHandler" Id=160 TIMED_WAITING
at java.lang.Thread.sleep(Native Method)
at org.quartz.impl.jdbcjobstore.JobStoreSupport$MisfireHandler.run(JobStoreSupport.java:3963)
"QuartzScheduler_DotMarketingQuartzExclusiveScheduler-715fb6e5-b6cb-4148-a5b9-b7ff9705514c_ClusterManager" Id=159 TIMED_WAITING
at java.lang.Thread.sleep(Native Method)
at org.quartz.impl.jdbcjobstore.JobStoreSupport$ClusterManager.run(JobStoreSupport.java:3878)
"QuartzScheduler_DotMarketingQuartzScheduler-715fb6e5-b6cb-4148-a5b9-b7ff9705514c_MisfireHandler" Id=158 TIMED_WAITING
at java.lang.Thread.sleep(Native Method)
at org.quartz.impl.jdbcjobstore.JobStoreSupport$MisfireHandler.run(JobStoreSupport.java:3963)
"QuartzScheduler_DotMarketingQuartzScheduler-715fb6e5-b6cb-4148-a5b9-b7ff9705514c_ClusterManager" Id=157 TIMED_WAITING
at java.lang.Thread.sleep(Native Method)
at org.quartz.impl.jdbcjobstore.JobStoreSupport$ClusterManager.run(JobStoreSupport.java:3878)
"Timer-2" Id=156 TIMED_WAITING on java.util.TaskQueue@b803b54
at java.lang.Object.wait(Native Method) - waiting on java.util.TaskQueue@b803b54
at java.util.TimerThread.mainLoop(Timer.java:552)
at java.util.TimerThread.run(Timer.java:505)
"DotMarketingQuartzLocalScheduler_QuartzSchedulerThread" Id=155 TIMED_WAITING on java.lang.Object@fb70db2
at java.lang.Object.wait(Native Method) - waiting on java.lang.Object@fb70db2
at org.quartz.core.QuartzSchedulerThread.run(QuartzSchedulerThread.java:410)
"DotMarketingQuartzLocalScheduler_Worker-1" Id=154 TIMED_WAITING on org.quartz.simpl.SimpleThreadPool$WorkerThread@9dd3700
at java.lang.Object.wait(Native Method) - waiting on org.quartz.simpl.SimpleThreadPool$WorkerThread@9dd3700
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:543)
"Timer-1" Id=153 TIMED_WAITING on java.util.TaskQueue@3ae4292d
at java.lang.Object.wait(Native Method) - waiting on java.util.TaskQueue@3ae4292d
at java.util.TimerThread.mainLoop(Timer.java:552)
at java.util.TimerThread.run(Timer.java:505)
"DotMarketingQuartzScheduler_QuartzSchedulerThread" Id=152 TIMED_WAITING on java.lang.Object@594b678f
at java.lang.Object.wait(Native Method) - waiting on java.lang.Object@594b678f
at org.quartz.core.QuartzSchedulerThread.run(QuartzSchedulerThread.java:410)
"DotMarketingQuartzScheduler_Worker-10" Id=151 TIMED_WAITING on org.quartz.simpl.SimpleThreadPool$WorkerThread@657c0b15
at java.lang.Object.wait(Native Method) - waiting on org.quartz.simpl.SimpleThreadPool$WorkerThread@657c0b15
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:543)
"DotMarketingQuartzScheduler_Worker-9" Id=150 TIMED_WAITING on org.quartz.simpl.SimpleThreadPool$WorkerThread@32bbc3c0
at java.lang.Object.wait(Native Method) - waiting on org.quartz.simpl.SimpleThreadPool$WorkerThread@32bbc3c0
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:543)
"DotMarketingQuartzScheduler_Worker-8" Id=149 TIMED_WAITING on org.quartz.simpl.SimpleThreadPool$WorkerThread@45308c45
at java.lang.Object.wait(Native Method) - waiting on org.quartz.simpl.SimpleThreadPool$WorkerThread@45308c45
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:543)
"DotMarketingQuartzScheduler_Worker-7" Id=148 TIMED_WAITING on org.quartz.simpl.SimpleThreadPool$WorkerThread@7649297f
at java.lang.Object.wait(Native Method) - waiting on org.quartz.simpl.SimpleThreadPool$WorkerThread@7649297f
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:543)
"DotMarketingQuartzScheduler_Worker-6" Id=147 TIMED_WAITING on org.quartz.simpl.SimpleThreadPool$WorkerThread@3d438d84
at java.lang.Object.wait(Native Method) - waiting on org.quartz.simpl.SimpleThreadPool$WorkerThread@3d438d84
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:543)
"DotMarketingQuartzScheduler_Worker-5" Id=146 TIMED_WAITING on org.quartz.simpl.SimpleThreadPool$WorkerThread@2c819b2d
at java.lang.Object.wait(Native Method) - waiting on org.quartz.simpl.SimpleThreadPool$WorkerThread@2c819b2d
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:543)
"DotMarketingQuartzScheduler_Worker-4" Id=145 TIMED_WAITING on org.quartz.simpl.SimpleThreadPool$WorkerThread@57a0fc55
at java.lang.Object.wait(Native Method) - waiting on org.quartz.simpl.SimpleThreadPool$WorkerThread@57a0fc55
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:543)
"DotMarketingQuartzScheduler_Worker-3" Id=144 TIMED_WAITING on org.quartz.simpl.SimpleThreadPool$WorkerThread@3bca9fc0
at java.lang.Object.wait(Native Method) - waiting on org.quartz.simpl.SimpleThreadPool$WorkerThread@3bca9fc0
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:543)
"DotMarketingQuartzScheduler_Worker-2" Id=143 TIMED_WAITING on org.quartz.simpl.SimpleThreadPool$WorkerThread@23c8d0fa
at java.lang.Object.wait(Native Method) - waiting on org.quartz.simpl.SimpleThreadPool$WorkerThread@23c8d0fa
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:543)
"DotMarketingQuartzScheduler_Worker-1" Id=142 TIMED_WAITING on org.quartz.simpl.SimpleThreadPool$WorkerThread@1256f23b
at java.lang.Object.wait(Native Method) - waiting on org.quartz.simpl.SimpleThreadPool$WorkerThread@1256f23b
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:543)
"Timer-0" Id=141 TIMED_WAITING on java.util.TaskQueue@1dfff488
at java.lang.Object.wait(Native Method) - waiting on java.util.TaskQueue@1dfff488
at java.util.TimerThread.mainLoop(Timer.java:552)
at java.util.TimerThread.run(Timer.java:505)
"DotMarketingQuartzExclusiveScheduler_QuartzSchedulerThread" Id=140 TIMED_WAITING on java.lang.Object@f0f1c1b
at java.lang.Object.wait(Native Method) - waiting on java.lang.Object@f0f1c1b
at org.quartz.core.QuartzSchedulerThread.run(QuartzSchedulerThread.java:410)
"DotMarketingQuartzExclusiveScheduler_Worker-1" Id=139 TIMED_WAITING on org.quartz.simpl.SimpleThreadPool$WorkerThread@7fcfa270
at java.lang.Object.wait(Native Method) - waiting on org.quartz.simpl.SimpleThreadPool$WorkerThread@7fcfa270
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:543)
"elasticsearch[715fb6e5-b6cb-4148-a5b9-b7ff9705514c][warmer][T#1]" Id=138 WAITING on com.dotcms.repackage.org.elasticsearch.common.util.concurrent.EsExecutors$ExecutorScalingQueue@28fe3856
at sun.misc.Unsafe.park(Native Method) - waiting on com.dotcms.repackage.org.elasticsearch.common.util.concurrent.EsExecutors$ExecutorScalingQueue@28fe3856
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at com.dotcms.repackage.org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:706)
at com.dotcms.repackage.org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.xfer(LinkedTransferQueue.java:615)
at com.dotcms.repackage.org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.take(LinkedTransferQueue.java:1109)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) ...
"elasticsearch[715fb6e5-b6cb-4148-a5b9-b7ff9705514c][riverClusterService#updateTask][T#1]" Id=132 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4a24ab20
at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4a24ab20
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
"elasticsearch[715fb6e5-b6cb-4148-a5b9-b7ff9705514c][transport_client_timer][T#1]{Hashed wheel timer #2}" Id=122 TIMED_WAITING
at java.lang.Thread.sleep(Native Method)
at com.dotcms.repackage.org.elasticsearch.common.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:503)
at com.dotcms.repackage.org.elasticsearch.common.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:401)
at com.dotcms.repackage.org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at java.lang.Thread.run(Thread.java:745)
"elasticsearch[715fb6e5-b6cb-4148-a5b9-b7ff9705514c][clusterService#updateTask][T#1]" Id=131 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2d5ee7e7
at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2d5ee7e7
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:539)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
"elasticsearch[715fb6e5-b6cb-4148-a5b9-b7ff9705514c][transport_server_boss][T#1]{New I/O server boss #20}" Id=128 RUNNABLE (in native)

at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)

at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) - locked sun.nio.ch.Util$2@7f47304 - locked java.util.Collections$UnmodifiableSet@45930d7e - locked sun.nio.ch.EPollSelectorImpl@f91332
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
at com.dotcms.repackage.org.elasticsearch.common.netty.channel.socket.nio.NioServerBoss.select(NioServerBoss.java:163)
at com.dotcms.repackage.org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206) ... Number of locked synchronizers = 1 - java.util.concurrent.ThreadPoolExecutor$Worker@746d2cfa
"elasticsearch[715fb6e5-b6cb-4148-a5b9-b7ff9705514c][transport_server_worker][T#4]{New I/O worker #19}" Id=127 RUNNABLE

at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)

at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) - locked sun.nio.ch.Util$2@4c12957 - locked java.util.Collections$UnmodifiableSet@3c8fbb33 - locked sun.nio.ch.EPollSelectorImpl@48521c99

at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at com.dotcms.repackage.org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
at com.dotcms.repackage.org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)

at com.dotcms.repackage.org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206) ... Number of locked synchronizers = 1 - java.util.concurrent.ThreadPoolExecutor$Worker@41fd0e04
"elasticsearch[715fb6e5-b6cb-4148-a5b9-b7ff9705514c][transport_server_worker][T#3]{New I/O worker #18}" Id=126 RUNNABLE

at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)

at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) - locked sun.nio.ch.Util$2@3130e431 - locked java.util.Collections$UnmodifiableSet@3a285d0b - locked sun.nio.ch.EPollSelectorImpl@63ec773f

at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at com.dotcms.repackage.org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
at com.dotcms.repackage.org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)

at com.dotcms.repackage.org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206) ... Number of locked synchronizers = 1 - java.util.concurrent.ThreadPoolExecutor$Worker@503d1d2f
"elasticsearch[715fb6e5-b6cb-4148-a5b9-b7ff9705514c][transport_server_worker][T#2]{New I/O worker #17}" Id=125 RUNNABLE

at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)

at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) - locked sun.nio.ch.Util$2@506f5491 - locked java.util.Collections$UnmodifiableSet@3dc2003e - locked sun.nio.ch.EPollSelectorImpl@33a3e32e

at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at com.dotcms.repackage.org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
at com.dotcms.repackage.org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)

at com.dotcms.repackage.org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206) ... Number of locked synchronizers = 1 - java.util.concurrent.ThreadPoolExecutor$Worker@f3f07a
"elasticsearch[715fb6e5-b6cb-4148-a5b9-b7ff9705514c][transport_server_worker][T#1]{New I/O worker #16}" Id=124 RUNNABLE

at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)

at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) - locked sun.nio.ch.Util$2@e00347b - locked java.util.Collections$UnmodifiableSet@4abebb24 - locked sun.nio.ch.EPollSelectorImpl@4de14594

at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at com.dotcms.repackage.org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
at com.dotcms.repackage.org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)

at com.dotcms.repackage.org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206) ... Number of locked synchronizers = 1 - java.util.concurrent.ThreadPoolExecutor$Worker@2b8fdebb
"elasticsearch[715fb6e5-b6cb-4148-a5b9-b7ff9705514c][transport_client_boss][T#1]{New I/O boss #15}" Id=123 RUNNABLE

at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)

at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) - locked sun.nio.ch.Util$2@4ecc8983 - locked java.util.Collections$UnmodifiableSet@339495c2 - locked sun.nio.ch.EPollSelectorImpl@fc3d685

at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at com.dotcms.repackage.org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
at com.dotcms.repackage.org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)

at com.dotcms.repackage.org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206) ... Number of locked synchronizers = 1 - java.util.concurrent.ThreadPoolExecutor$Worker@5a8da784
"elasticsearch[715fb6e5-b6cb-4148-a5b9-b7ff9705514c][transport_client_worker][T#4]{New I/O worker #14}" Id=121 RUNNABLE

at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)

at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) - locked sun.nio.ch.Util$2@5b37f2b2 - locked java.util.Collections$UnmodifiableSet@19737773 - locked sun.nio.ch.EPollSelectorImpl@2c3e88f5

at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at com.dotcms.repackage.org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
at com.dotcms.repackage.org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)

at com.dotcms.repackage.org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206) ... Number of locked synchronizers = 1 - java.util.concurrent.ThreadPoolExecutor$Worker@77bd86f5
"elasticsearch[715fb6e5-b6cb-4148-a5b9-b7ff9705514c][transport_client_worker][T#3]{New I/O worker #13}" Id=120 RUNNABLE

at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)

at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) - locked sun.nio.ch.Util$2@24864eb0 - locked java.util.Collections$UnmodifiableSet@1c31dee3 - locked sun.nio.ch.EPollSelectorImpl@5ce9adeb

at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at com.dotcms.repackage.org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
at com.dotcms.repackage.org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)

at com.dotcms.repackage.org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206) ... Number of locked synchronizers = 1 - java.util.concurrent.ThreadPoolExecutor$Worker@5c61c98d
"elasticsearch[715fb6e5-b6cb-4148-a5b9-b7ff9705514c][transport_client_worker][T#2]{New I/O worker #12}" Id=119 RUNNABLE

at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)

at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) - locked sun.nio.ch.Util$2@80eaf70 - locked java.util.Collections$UnmodifiableSet@1241daa1 - locked sun.nio.ch.EPollSelectorImpl@70841566

at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at com.dotcms.repackage.org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
at com.dotcms.repackage.org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)

at com.dotcms.repackage.org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206) ... Number of locked synchronizers = 1 - java.util.concurrent.ThreadPoolExecutor$Worker@e7f9109
"elasticsearch[715fb6e5-b6cb-4148-a5b9-b7ff9705514c][transport_client_worker][T#1]{New I/O worker #11}" Id=118 RUNNABLE

at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)

at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) - locked sun.nio.ch.Util$2@2b421456 - locked java.util.Collections$UnmodifiableSet@176549e6 - locked sun.nio.ch.EPollSelectorImpl@22707ee0

at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at com.dotcms.repackage.org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
at com.dotcms.repackage.org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)

at com.dotcms.repackage.org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206) ... Number of locked synchronizers = 1 - java.util.concurrent.ThreadPoolExecutor$Worker@27fe8236
"elasticsearch[715fb6e5-b6cb-4148-a5b9-b7ff9705514c][[ttl_expire]]" Id=117 TIMED_WAITING
at java.lang.Thread.sleep(Native Method)
at com.dotcms.repackage.org.elasticsearch.indices.ttl.IndicesTTLService$PurgerThread.run(IndicesTTLService.java:129)
"elasticsearch[715fb6e5-b6cb-4148-a5b9-b7ff9705514c][scheduler][T#1]" Id=116 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@799b5ca
at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@799b5ca
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) ...
"elasticsearch[715fb6e5-b6cb-4148-a5b9-b7ff9705514c][[timer]]" Id=115 TIMED_WAITING
at java.lang.Thread.sleep(Native Method)
at com.dotcms.repackage.org.elasticsearch.threadpool.ThreadPool$EstimatedTimeThread.run(ThreadPool.java:611)
"Thread-25" Id=109 TIMED_WAITING
at java.lang.Thread.sleep(Native Method)
at com.dotcms.cluster.common.ClusterServerActionThread.run(ClusterServerActionThread.java:87)
"Thread-24" Id=103 TIMED_WAITING
at java.lang.Thread.sleep(Native Method)
at com.dotmarketing.common.reindex.ReindexThread.run(ReindexThread.java:218)
"H2 Log Writer CACHE_DB1" Id=53 TIMED_WAITING on org.h2.store.WriterThread@7c6de4ab
at java.lang.Object.wait(Native Method) - waiting on org.h2.store.WriterThread@7c6de4ab
at org.h2.store.WriterThread.run(WriterThread.java:103)
at java.lang.Thread.run(Thread.java:745)
"H2 Log Writer CACHE_DB9" Id=51 TIMED_WAITING on org.h2.store.WriterThread@12bfd65f
at java.lang.Object.wait(Native Method) - waiting on org.h2.store.WriterThread@12bfd65f
at org.h2.store.WriterThread.run(WriterThread.java:103)
at java.lang.Thread.run(Thread.java:745)
"H2 Log Writer CACHE_DB4" Id=49 TIMED_WAITING on org.h2.store.WriterThread@70930096
at java.lang.Object.wait(Native Method) - waiting on org.h2.store.WriterThread@70930096
at org.h2.store.WriterThread.run(WriterThread.java:103)
at java.lang.Thread.run(Thread.java:745)
"H2 Log Writer CACHE_DB8" Id=47 TIMED_WAITING on org.h2.store.WriterThread@430bb197
at java.lang.Object.wait(Native Method) - waiting on org.h2.store.WriterThread@430bb197
at org.h2.store.WriterThread.run(WriterThread.java:103)
at java.lang.Thread.run(Thread.java:745)
"H2 Log Writer CACHE_DB5" Id=45 TIMED_WAITING on org.h2.store.WriterThread@34b852e4
at java.lang.Object.wait(Native Method) - waiting on org.h2.store.WriterThread@34b852e4
at org.h2.store.WriterThread.run(WriterThread.java:103)
at java.lang.Thread.run(Thread.java:745)
"H2 Log Writer CACHE_DB6" Id=43 TIMED_WAITING on org.h2.store.WriterThread@314a11c6
at java.lang.Object.wait(Native Method) - waiting on org.h2.store.WriterThread@314a11c6
at org.h2.store.WriterThread.run(WriterThread.java:103)
at java.lang.Thread.run(Thread.java:745)
"H2 Log Writer CACHE_DB7" Id=41 TIMED_WAITING on org.h2.store.WriterThread@75d4f771
at java.lang.Object.wait(Native Method) - waiting on org.h2.store.WriterThread@75d4f771
at org.h2.store.WriterThread.run(WriterThread.java:103)
at java.lang.Thread.run(Thread.java:745)
"H2 Log Writer CACHE_DB3" Id=39 TIMED_WAITING on org.h2.store.WriterThread@70c15826
at java.lang.Object.wait(Native Method) - waiting on org.h2.store.WriterThread@70c15826
at org.h2.store.WriterThread.run(WriterThread.java:103)
at java.lang.Thread.run(Thread.java:745)
"H2 Log Writer CACHE_DB2" Id=37 TIMED_WAITING on org.h2.store.WriterThread@2582609f
at java.lang.Object.wait(Native Method) - waiting on org.h2.store.WriterThread@2582609f
at org.h2.store.WriterThread.run(WriterThread.java:103)
at java.lang.Thread.run(Thread.java:745)
"Abandoned connection cleanup thread" Id=27 TIMED_WAITING on java.lang.ref.ReferenceQueue$Lock@129579f6
at java.lang.Object.wait(Native Method) - waiting on java.lang.ref.ReferenceQueue$Lock@129579f6
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
at com.mysql.jdbc.AbandonedConnectionCleanupThread.run(AbandonedConnectionCleanupThread.java:43)
"fileinstall-/usr/local/SIS/applications/dotserver/tomcat-7.0.54/webapps/ROOT/WEB-INF/felix/load" Id=24 TIMED_WAITING on com.dotcms.repackage.org.apache.felix.fileinstall.internal.DirectoryWatcher@137a0f02
at java.lang.Object.wait(Native Method) - waiting on com.dotcms.repackage.org.apache.felix.fileinstall.internal.DirectoryWatcher@137a0f02
at com.dotcms.repackage.org.apache.felix.fileinstall.internal.DirectoryWatcher.run(DirectoryWatcher.java:268)
"FelixStartLevel" Id=23 WAITING on java.util.ArrayList@326f564b
at java.lang.Object.wait(Native Method) - waiting on java.util.ArrayList@326f564b
at java.lang.Object.wait(Object.java:503)
at com.dotcms.repackage.org.apache.felix.framework.FrameworkStartLevelImpl.run(FrameworkStartLevelImpl.java:279)
at java.lang.Thread.run(Thread.java:745)
"FelixDispatchQueue" Id=22 WAITING on java.util.ArrayList@2b8a6fcb
at java.lang.Object.wait(Native Method) - waiting on java.util.ArrayList@2b8a6fcb
at java.lang.Object.wait(Object.java:503)
at com.dotcms.repackage.org.apache.felix.framework.util.EventDispatcher.run(EventDispatcher.java:1063)
at com.dotcms.repackage.org.apache.felix.framework.util.EventDispatcher.access$000(EventDispatcher.java:54)
at com.dotcms.repackage.org.apache.felix.framework.util.EventDispatcher$1.run(EventDispatcher.java:101)
at java.lang.Thread.run(Thread.java:745)
"Log4J Active Asynchronous Appender" Id=21 WAITING on java.util.LinkedList@db18ec2
at java.lang.Object.wait(Native Method) - waiting on java.util.LinkedList@db18ec2
at java.lang.Object.wait(Object.java:503)
at com.dotcms.repackage.org.apache.log4j.appender.LoggingEventLinkedList.takeLast(LoggingEventLinkedList.java:91)
at com.dotcms.repackage.org.apache.log4j.appender.LoggingEventDispatcher.dispatchAndWait(LoggingEventDispatcher.java:232)
at com.dotcms.repackage.org.apache.log4j.appender.LoggingEventDispatcher.run(LoggingEventDispatcher.java:81)
at java.lang.Thread.run(Thread.java:745)
"Log4J Active Asynchronous Appender" Id=20 WAITING on java.util.LinkedList@4136994
at java.lang.Object.wait(Native Method) - waiting on java.util.LinkedList@4136994
at java.lang.Object.wait(Object.java:503)
at com.dotcms.repackage.org.apache.log4j.appender.LoggingEventLinkedList.takeLast(LoggingEventLinkedList.java:91)
at com.dotcms.repackage.org.apache.log4j.appender.LoggingEventDispatcher.dispatchAndWait(LoggingEventDispatcher.java:232)
at com.dotcms.repackage.org.apache.log4j.appender.LoggingEventDispatcher.run(LoggingEventDispatcher.java:81)
at java.lang.Thread.run(Thread.java:745)
"Log4J Active Asynchronous Appender" Id=19 WAITING on java.util.LinkedList@167875a3
at java.lang.Object.wait(Native Method) - waiting on java.util.LinkedList@167875a3
at java.lang.Object.wait(Object.java:503)
at com.dotcms.repackage.org.apache.log4j.appender.LoggingEventLinkedList.takeLast(LoggingEventLinkedList.java:91)
at com.dotcms.repackage.org.apache.log4j.appender.LoggingEventDispatcher.dispatchAndWait(LoggingEventDispatcher.java:232)
at com.dotcms.repackage.org.apache.log4j.appender.LoggingEventDispatcher.run(LoggingEventDispatcher.java:81)
at java.lang.Thread.run(Thread.java:745)
"Log4J Active Asynchronous Appender" Id=18 WAITING on java.util.LinkedList@3e732bdb
at java.lang.Object.wait(Native Method) - waiting on java.util.LinkedList@3e732bdb
at java.lang.Object.wait(Object.java:503)
at com.dotcms.repackage.org.apache.log4j.appender.LoggingEventLinkedList.takeLast(LoggingEventLinkedList.java:91)
at com.dotcms.repackage.org.apache.log4j.appender.LoggingEventDispatcher.dispatchAndWait(LoggingEventDispatcher.java:232)
at com.dotcms.repackage.org.apache.log4j.appender.LoggingEventDispatcher.run(LoggingEventDispatcher.java:81)
at java.lang.Thread.run(Thread.java:745)
"Log4J Active Asynchronous Appender" Id=17 WAITING on java.util.LinkedList@7b4456e4
at java.lang.Object.wait(Native Method) - waiting on java.util.LinkedList@7b4456e4
at java.lang.Object.wait(Object.java:503)
at com.dotcms.repackage.org.apache.log4j.appender.LoggingEventLinkedList.takeLast(LoggingEventLinkedList.java:91)
at com.dotcms.repackage.org.apache.log4j.appender.LoggingEventDispatcher.dispatchAndWait(LoggingEventDispatcher.java:232)
at com.dotcms.repackage.org.apache.log4j.appender.LoggingEventDispatcher.run(LoggingEventDispatcher.java:81)
at java.lang.Thread.run(Thread.java:745)
"Log4J Active Asynchronous Appender" Id=16 WAITING on java.util.LinkedList@43a4d1f5
at java.lang.Object.wait(Native Method) - waiting on java.util.LinkedList@43a4d1f5
at java.lang.Object.wait(Object.java:503)
at com.dotcms.repackage.org.apache.log4j.appender.LoggingEventLinkedList.takeLast(LoggingEventLinkedList.java:91)
at com.dotcms.repackage.org.apache.log4j.appender.LoggingEventDispatcher.dispatchAndWait(LoggingEventDispatcher.java:232)
at com.dotcms.repackage.org.apache.log4j.appender.LoggingEventDispatcher.run(LoggingEventDispatcher.java:81)
at java.lang.Thread.run(Thread.java:745)
"Log4J Active Asynchronous Appender" Id=15 WAITING on java.util.LinkedList@79d44e84
at java.lang.Object.wait(Native Method) - waiting on java.util.LinkedList@79d44e84
at java.lang.Object.wait(Object.java:503)
at com.dotcms.repackage.org.apache.log4j.appender.LoggingEventLinkedList.takeLast(LoggingEventLinkedList.java:91)
at com.dotcms.repackage.org.apache.log4j.appender.LoggingEventDispatcher.dispatchAndWait(LoggingEventDispatcher.java:232)
at com.dotcms.repackage.org.apache.log4j.appender.LoggingEventDispatcher.run(LoggingEventDispatcher.java:81)
at java.lang.Thread.run(Thread.java:745)
"Log4J Active Asynchronous Appender" Id=14 WAITING on java.util.LinkedList@616efa98
at java.lang.Object.wait(Native Method) - waiting on java.util.LinkedList@616efa98
at java.lang.Object.wait(Object.java:503)
at com.dotcms.repackage.org.apache.log4j.appender.LoggingEventLinkedList.takeLast(LoggingEventLinkedList.java:91)
at com.dotcms.repackage.org.apache.log4j.appender.LoggingEventDispatcher.dispatchAndWait(LoggingEventDispatcher.java:232)
at com.dotcms.repackage.org.apache.log4j.appender.LoggingEventDispatcher.run(LoggingEventDispatcher.java:81)
at java.lang.Thread.run(Thread.java:745)
"GC Daemon" Id=11 TIMED_WAITING on sun.misc.GC$LatencyLock@3d064321
at java.lang.Object.wait(Native Method) - waiting on sun.misc.GC$LatencyLock@3d064321
at sun.misc.GC$Daemon.run(GC.java:117)
"Signal Dispatcher" Id=5 RUNNABLE
"Finalizer" Id=3 WAITING on java.lang.ref.ReferenceQueue$Lock@66214ad3
at java.lang.Object.wait(Native Method) - waiting on java.lang.ref.ReferenceQueue$Lock@66214ad3
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
"Reference Handler" Id=2 WAITING on java.lang.ref.Reference$Lock@13b79707
at java.lang.Object.wait(Native Method) - waiting on java.lang.ref.Reference$Lock@13b79707
at java.lang.Object.wait(Object.java:503)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133)
"main" Id=1 RUNNABLE (in native)
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
at java.net.ServerSocket.implAccept(ServerSocket.java:530)
at java.net.ServerSocket.accept(ServerSocket.java:498)
at org.apache.catalina.core.StandardServer.await(StandardServer.java:451)
at org.apache.catalina.startup.Catalina.await(Catalina.java:777)
at org.apache.catalina.startup.Catalina.start(Catalina.java:723)
at sun.reflect.NativeMethodAccessorImpl.invoke0(

Falzone, Chris

unread,

Feb 9, 2015, 7:59:10 AM2/9/15

to dot...@googlegroups.com

The thread dumps from the maintenance portlet only show like the first 10 lines of each thread's stack. We need a proper thread dump to look. Ideally we need to see to see 2-5 dumps in a row taken every 5-10 seconds during the time when your CPU is pegged. Use kill -3 <PID> on linux systems and on Windows jstack should get us what we need.

http://docs.oracle.com/javase/7/docs/technotes/tools/share/jstack.html

Thanks!

Christopher Falzone
Interactive Developer

A Q U E N T
Digital, Creative, and Marketing Talent

aquent.com
cfal...@aquent.com

Daniel Graham

unread,

Feb 9, 2015, 8:37:31 AM2/9/15

to dot...@googlegroups.com

Here's the full stack trace from jstack ..

2015-02-09 13:36:12

Full thread dump Java HotSpot(TM) 64-Bit Server VM (24.65-b04 mixed mode):

"http-bio-80-exec-168" daemon prio=10 tid=0x00007f86d8ed2000 nid=0x438c waiting on condition [0x00007f86c2288000]

java.lang.Thread.State: TIMED_WAITING (parking)

at sun.misc.Unsafe.park(Native Method)

- parking to wait for <0x0000000744ee04b8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)

at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)

at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)

at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:86)

at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:32)

at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)

at java.lang.Thread.run(Thread.java:745)

"http-bio-80-exec-167" daemon prio=10 tid=0x00007f86d9182800 nid=0x438b waiting on condition [0x00007f86c278d000]