Thread count keeps growing at tomcat 8.5 with CAS 5.1

[Song, Doe-Hyun]

Good Morning All,

 

Since we go to production with CAS 5.1, we keep having growing Thread counts.

 

We use ehcache for HA implementation and ldap for authentication.

Two types of threads – Timer and pool-3-thread – keep growing as time goes.

 

Any suggestion to debug this issue?

 

Thread dump :

 

 

Full thread dump OpenJDK 64-Bit Server VM (25.101-b13 mixed mode):

 

"Timer-73" #512 daemon prio=5 os_prio=0 tid=0x00007f0520033000 nid=0x8041 in Object.wait() [0x00007f0509b5c000]

   java.lang.Thread.State: TIMED_WAITING (on object monitor)

        at java.lang.Object.wait(Native Method)

        at java.util.TimerThread.mainLoop(Timer.java:552)

        - locked <0x00000000afb48f10> (a java.util.TaskQueue)

        at java.util.TimerThread.run(Timer.java:505)

 

"Timer-72" #509 daemon prio=5 os_prio=0 tid=0x00007f0520019800 nid=0x7fbc in Object.wait() [0x00007f050a469000]

   java.lang.Thread.State: TIMED_WAITING (on object monitor)

        at java.lang.Object.wait(Native Method)

        at java.util.TimerThread.mainLoop(Timer.java:552)

        - locked <0x00000000afa333a0> (a java.util.TaskQueue)

        at java.util.TimerThread.run(Timer.java:505)

 

"pool-3-thread-23" #508 prio=5 os_prio=0 tid=0x00007f05180b3000 nid=0x7f36 waiting on condition [0x00007f0509d60000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000a7f17930> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

        at java.lang.Thread.run(Thread.java:745)

 

"Timer-71" #507 daemon prio=5 os_prio=0 tid=0x00007f05180b8800 nid=0x7f35 in Object.wait() [0x00007f0509f62000]

   java.lang.Thread.State: TIMED_WAITING (on object monitor)

        at java.lang.Object.wait(Native Method)

        at java.util.TimerThread.mainLoop(Timer.java:552)

        - locked <0x00000000af974c00> (a java.util.TaskQueue)

        at java.util.TimerThread.run(Timer.java:505)

 

"Timer-70" #506 daemon prio=5 os_prio=0 tid=0x00007f05506ba000 nid=0x7eb4 in Object.wait() [0x00007f0509e61000]

   java.lang.Thread.State: TIMED_WAITING (on object monitor)

        at java.lang.Object.wait(Native Method)

        at java.util.TimerThread.mainLoop(Timer.java:552)

        - locked <0x00000000af8cf808> (a java.util.TaskQueue)

        at java.util.TimerThread.run(Timer.java:505)

 

"pool-3-thread-22" #496 prio=5 os_prio=0 tid=0x00007f05180b7800 nid=0x79dd waiting on condition [0x00007f050a063000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000a7f17930> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)

 

Thanks,

Doe

 

The information contained in this e-mail and any attachments is confidential and
intended only for the recipient. If you are not the intended recipient, the
information contained in this message may not be used, copied, or forwarded to
third parties or otherwise distributed for any other purpose. Please notify the
sender if you received this e-mail in error and delete the e-mail and its
attachments promptly.  Nothing in this e-mail may be used or deemed to form the
basis of a contractual or any other legally binding obligation unless separately
confirmed in writing by an authorized representative of ARMADA.

[Emilio Junior Francischetti]

Hi,

we are experiencing  your same problem. Did you find a solution?

May thanks in advance

Emilio

[Dan S]

We are seeing this same issue. Did you ever find a solution for it?

Dan

[Bruno ELIE]

Up, someone's has found something about this ?

[Dan S]

I haven't found a solution to this issue. It seems to be related to SAML2 services. Any time one of them creates a service ticket, the thread count increases and never decreases.

I checked the 5.3 RC2 and it doesn't seem to have the same issue. Once that release is a stable version, I'll be testing that out for our deployment.

On Monday, August 28, 2017 at 9:02:48 AM UTC-5, Song, Doe-Hyun wrote:

[Bergner, Arnold]

We’re facing the same issue on 5.2.2, tomcat 8.0.

 

I’ve also related the number of Timers to the SAML logins. Logging org.opensaml in debug mode, I can see a metadata resolver refresh files every ~10 seconds. Given the refresh interval of 2 hours and the number of Timers, this seems to be the reason.

 

The issue was also part of this pull request: https://github.com/apereo/cas/pull/3130

The result is as you say; it’s probably fixed in 5.3. I couldn’t find a starting point for a backport right away, so we’re hoping for it.

 

Regards,

Arnold

--
- Website: https://apereo.github.io/cas
- Gitter Chatroom: https://gitter.im/apereo/cas
- List Guidelines: https://goo.gl/1VRrw7
- Contributions: https://goo.gl/mh7qDG
---
You received this message because you are subscribed to the Google Groups "CAS Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cas-user+u...@apereo.org.
To view this discussion on the web visit https://groups.google.com/a/apereo.org/d/msgid/cas-user/44df4b59-b9c1-4b15-bf00-3d96f97b9ef4%40apereo.org.

[JF Poulin]

Running into this issue in v5.1.9. Can anybody confirm that it was fixed in v5.3.x? If I wanted to patch v5.1.x, where should I look in the code?

Thanks!

[Bruno ELIE]

Hi,
I can confirm this issue is not present on version 5.3.x