JVM Heap Kept Growing every day

232 views
Skip to first unread message

RJ

unread,
Aug 16, 2017, 12:54:08 PM8/16/17
to cas-...@apereo.org
We have seen that system is heavily swapping after a few days of CAS uptime.

JVM Heap args are Xms 4096m & Xmx4096m, but it kept every day.

Is there a way to find out whats taking all this space? We are talking about 3000 logins a day. This is pretty much common every day. I was hoping that old tickets get expired and new tickets get created and stay active for 8 hours. If thats working, memory requirement should not keep increasing every day.

jvm.memory.heap.usage in stats file kept growing from 3% to 50% within 3 days of uptime.

I don't understand why the tickets are not cleaned up/expired. Is there a way to know the no# of tickets..etc ? We disabled /status URI though.

What parameters do you guys use for TGT lifetime and ST lifetime?

We just use LDAP and hazelcast ticket registry with 5.0.5.

Thanks
RJ

Uxío Prego

unread,
Aug 16, 2017, 1:54:57 PM8/16/17
to CAS Community
Never used hazelcast ticket registry nor 5.0.x, still those figures seem normal to me.

If you just can't kill the curiosity, I recommend you saving a snapshot of your Debian GNU/Linux CAS server, then installing xorg, xserver, lightdm, and a lightweight desktop environment of your choice, and visualvm; then fetching your CAS process from visualvm and profit from great stats and insights. When you are satisfied you restore the saved snapshot so finally everything keeps as clean and performant as before the intervention.

Cheers and regards,

--
- CAS gitter chatroom: https://gitter.im/apereo/cas
- CAS mailing list guidelines: https://apereo.github.io/cas/Mailing-Lists.html
- CAS documentation website: https://apereo.github.io/cas
- CAS project website: https://github.com/apereo/cas
---
You received this message because you are subscribed to the Google Groups "CAS Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cas-user+u...@apereo.org.
To view this discussion on the web visit https://groups.google.com/a/apereo.org/d/msgid/cas-user/CACNfiMJErDCzexjenLo-2tA2JNmbYHLx3JJUNaw_pE6KJvPsSA%40mail.gmail.com.

Uxío Prego

unread,
Aug 16, 2017, 2:01:40 PM8/16/17
to CAS Community
I would reboot CAS servers nightly, if you can afford that.

Try a TGT lifetime of several hours and an ST lifetime of some minutes. But I can't tell the rationale.

David Malia

unread,
Aug 17, 2017, 2:04:58 AM8/17/17
to cas-...@apereo.org
Is your ticket registry running on the same container as the application?

Thanks,
David Malia

To unsubscribe from this group and stop receiving emails from it, send an email to cas-user+unsubscribe@apereo.org.

--
- CAS gitter chatroom: https://gitter.im/apereo/cas
- CAS mailing list guidelines: https://apereo.github.io/cas/Mailing-Lists.html
- CAS documentation website: https://apereo.github.io/cas
- CAS project website: https://github.com/apereo/cas
---
You received this message because you are subscribed to the Google Groups "CAS Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cas-user+unsubscribe@apereo.org.
To view this discussion on the web visit https://groups.google.com/a/apereo.org/d/msgid/cas-user/6DDC6273-136B-41E0-99EF-AA663DAEE428%40madiva.com.

RJ

unread,
Aug 17, 2017, 10:27:24 PM8/17/17
to cas-...@apereo.org
Yes, we use hazelcast .. which is a kind of in memory ticket registry across two nodes. Is there an option to run ticket registry in another JVM?

To unsubscribe from this group and stop receiving emails from it, send an email to cas-user+u...@apereo.org.

--
- CAS gitter chatroom: https://gitter.im/apereo/cas
- CAS mailing list guidelines: https://apereo.github.io/cas/Mailing-Lists.html
- CAS documentation website: https://apereo.github.io/cas
- CAS project website: https://github.com/apereo/cas
---
You received this message because you are subscribed to the Google Groups "CAS Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cas-user+u...@apereo.org.

--
- CAS gitter chatroom: https://gitter.im/apereo/cas
- CAS mailing list guidelines: https://apereo.github.io/cas/Mailing-Lists.html
- CAS documentation website: https://apereo.github.io/cas
- CAS project website: https://github.com/apereo/cas
---
You received this message because you are subscribed to the Google Groups "CAS Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cas-user+unsubscribe@apereo.org.
To view this discussion on the web visit https://groups.google.com/a/apereo.org/d/msgid/cas-user/CAL3JkEJfY6xXwkNgwx8PxQykZubWSY_FpxhJZxuXi4QAPNCdcA%40mail.gmail.com.

Tom O'Neill

unread,
Aug 18, 2017, 9:12:13 AM8/18/17
to cas-...@apereo.org

Are tickets ever expiring or are the tables just consistently growing?

If you have a problem with your ticket registry cleaner your data set will grow and your heap will never fully recover without recycling services.

You shouldn’t have to restart nightly if the ticket registry cleaner is removing tickets.

 

Thanks,

 

Tom

--
- CAS gitter chatroom: https://gitter.im/apereo/cas
- CAS mailing list guidelines: https://apereo.github.io/cas/Mailing-Lists.html
- CAS documentation website: https://apereo.github.io/cas
- CAS project website: https://github.com/apereo/cas
---
You received this message because you are subscribed to the Google Groups "CAS Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cas-user+u...@apereo.org.

RJ

unread,
Aug 18, 2017, 9:52:24 AM8/18/17
to cas-...@apereo.org
Thanks Tom.

We are using cache based registry, hazelcast. the problem is that we dont know the statistics about no# of tickets in the registry...etc. Any clues ?

To unsubscribe from this group and stop receiving emails from it, send an email to cas-user+unsubscribe@apereo.org.

--
- CAS gitter chatroom: https://gitter.im/apereo/cas
- CAS mailing list guidelines: https://apereo.github.io/cas/Mailing-Lists.html
- CAS documentation website: https://apereo.github.io/cas
- CAS project website: https://github.com/apereo/cas
---
You received this message because you are subscribed to the Google Groups "CAS Community" group.

To unsubscribe from this group and stop receiving emails from it, send an email to cas-user+unsubscribe@apereo.org.

--
- CAS gitter chatroom: https://gitter.im/apereo/cas
- CAS mailing list guidelines: https://apereo.github.io/cas/Mailing-Lists.html
- CAS documentation website: https://apereo.github.io/cas
- CAS project website: https://github.com/apereo/cas
---
You received this message because you are subscribed to the Google Groups "CAS Community" group.

To unsubscribe from this group and stop receiving emails from it, send an email to cas-user+unsubscribe@apereo.org.

--
- CAS gitter chatroom: https://gitter.im/apereo/cas
- CAS mailing list guidelines: https://apereo.github.io/cas/Mailing-Lists.html
- CAS documentation website: https://apereo.github.io/cas
- CAS project website: https://github.com/apereo/cas
---
You received this message because you are subscribed to the Google Groups "CAS Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cas-user+unsubscribe@apereo.org.

--
- CAS gitter chatroom: https://gitter.im/apereo/cas
- CAS mailing list guidelines: https://apereo.github.io/cas/Mailing-Lists.html
- CAS documentation website: https://apereo.github.io/cas
- CAS project website: https://github.com/apereo/cas
---
You received this message because you are subscribed to the Google Groups "CAS Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cas-user+unsubscribe@apereo.org.
 To view this discussion on the web visit https://groups.google.com/a/apereo.org/d/msgid/cas-user/CY4PR02MB2773841CEDAC3FB24DE3BC9FCB800%40CY4PR02MB2773.namprd02.prod.outlook.com.

Tom O'Neill

unread,
Aug 18, 2017, 10:43:35 AM8/18/17
to cas-...@apereo.org

RJ,

 

I’ve only worked with the JPA registry so I can’t offer much insight with hazelcast, unfortunately.

I would suggest using a tool like VisualVM to try to collect some more information from the VM:

https://visualvm.github.io/

 

I’ve had luck reviewing dumps in the past to help with narrowing down the root cause.

To unsubscribe from this group and stop receiving emails from it, send an email to cas-user+u...@apereo.org.

--
- CAS gitter chatroom: https://gitter.im/apereo/cas
- CAS mailing list guidelines: https://apereo.github.io/cas/Mailing-Lists.html
- CAS documentation website: https://apereo.github.io/cas
- CAS project website: https://github.com/apereo/cas
---
You received this message because you are subscribed to the Google Groups "CAS Community" group.

To unsubscribe from this group and stop receiving emails from it, send an email to cas-user+u...@apereo.org.

--
- CAS gitter chatroom: https://gitter.im/apereo/cas
- CAS mailing list guidelines: https://apereo.github.io/cas/Mailing-Lists.html
- CAS documentation website: https://apereo.github.io/cas
- CAS project website: https://github.com/apereo/cas
---
You received this message because you are subscribed to the Google Groups "CAS Community" group.

To unsubscribe from this group and stop receiving emails from it, send an email to cas-user+u...@apereo.org.

--
- CAS gitter chatroom: https://gitter.im/apereo/cas
- CAS mailing list guidelines: https://apereo.github.io/cas/Mailing-Lists.html
- CAS documentation website: https://apereo.github.io/cas
- CAS project website: https://github.com/apereo/cas
---
You received this message because you are subscribed to the Google Groups "CAS Community" group.

To unsubscribe from this group and stop receiving emails from it, send an email to cas-user+u...@apereo.org.

--
- CAS gitter chatroom: https://gitter.im/apereo/cas
- CAS mailing list guidelines: https://apereo.github.io/cas/Mailing-Lists.html
- CAS documentation website: https://apereo.github.io/cas
- CAS project website: https://github.com/apereo/cas
---
You received this message because you are subscribed to the Google Groups "CAS Community" group.

To unsubscribe from this group and stop receiving emails from it, send an email to cas-user+u...@apereo.org.

 

--

- CAS gitter chatroom: https://gitter.im/apereo/cas
- CAS mailing list guidelines: https://apereo.github.io/cas/Mailing-Lists.html
- CAS documentation website: https://apereo.github.io/cas
- CAS project website: https://github.com/apereo/cas
---
You received this message because you are subscribed to the Google Groups "CAS Community" group.

To unsubscribe from this group and stop receiving emails from it, send an email to cas-user+u...@apereo.org.
To view this discussion on the web visit https://groups.google.com/a/apereo.org/d/msgid/cas-user/CACNfiM%2BVejuuHJijBHkRVuFVLzrYrphBFzNrJFMSC6oXnju%2BTw%40mail.gmail.com.

Oschwald Robert

unread,
Aug 18, 2017, 11:51:07 AM8/18/17
to cas-...@apereo.org
You can get the amount of objects in Hazelcast using Hazelcast Management Center or VM tools like visualvm.
I’m not sure if Hazelcast Management Center is available for Open Source License usage, or in Hazelcast Enterprise, only.

I strongly recommend to NOT use the Hazelcast embedded mode, as it is not optimal for production.
Better to setup a hazelcast Cluster, so run a Hazelcast “Client Plus Member” Topology.

See https://hazelcast.com/resources/hazelcast-deployment-operations-guide/

Robert

David Malia

unread,
Aug 19, 2017, 4:53:38 AM8/19/17
to cas-...@apereo.org
It probably is a good idea to move Hazelcast to its own jvm at a minimum.  It looks like the default behavior is to fill the cache until its at 85% of heap space.

--
- CAS gitter chatroom: https://gitter.im/apereo/cas
- CAS mailing list guidelines: https://apereo.github.io/cas/Mailing-Lists.html
- CAS documentation website: https://apereo.github.io/cas
- CAS project website: https://github.com/apereo/cas
---
You received this message because you are subscribed to the Google Groups "CAS Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cas-user+unsubscribe@apereo.org.
To view this discussion on the web visit https://groups.google.com/a/apereo.org/d/msgid/cas-user/E1910550-B266-448A-A09B-3C32FFB0A59A%40gmail.com.

dkopy...@unicon.net

unread,
Aug 19, 2017, 5:20:36 AM8/19/17
to cas-...@apereo.org
OK,

there is a significant memory leak in the version of Thymeleaf layout dialect used by versions of CAS prior to 5.0.7. Since 5.0.7 it's been updated to the layout dialect version containing the memory leak fix.

This is most likely what you are experiencing.

Cheers,
D.




To unsubscribe from this group and stop receiving emails from it, send an email to cas-user+u...@apereo.org.
To view this discussion on the web visit https://groups.google.com/a/apereo.org/d/msgid/cas-user/CAL3JkELjWgjBjsVsCru9FRSZGRdWgp%3D%2B5M%3DRrcxBo%3D8tVjEdSw%40mail.gmail.com.

RJ

unread,
Aug 28, 2017, 7:25:17 PM8/28/17
to cas-...@apereo.org
Here is another take at this:

Started CAS with  -Xms2048m -Xmx4096m about 12 days ago.

Interesting things from stats file is that jvm.memory.heap.committed & jvm.memory.heap.init haven't changed since beginning. 2147483648 is the same value from the beginning.  jvm.memory.heap.usage has varied from 5% to 40%.

jvm.memory.heap.committed, value=2147483648
jvm.memory.heap.init, value=2147483648
jvm.memory.heap.max, value=4294967296
jvm.memory.heap.usage, value=0.33496101573109627
jvm.memory.heap.used, value=1438646608

Weirdest observation (free -m)

12 days ago (when started)
Mem:           6472        2257         749          48        2161        3851
Swap:          4095           0        4095

now:
Mem:           6472        3561         749          48        2161        2547
Swap:          4095           0        4095

Used memory started from 2257 to 3561m.

top -> m
  694 javauser    20   0 7381748 3.263g  17792 S  0.8 51.6 288:34.34 java

So, top command tells that jvm takes 3.263g.  I was thinking that JVM takes only 2G {2147483648 of jvm.memory.heap.committed}.

Does this make any sense ? Thoughts?

To unsubscribe from this group and stop receiving emails from it, send an email to cas-user+u...@apereo.org.

--
- CAS gitter chatroom: https://gitter.im/apereo/cas
- CAS mailing list guidelines: https://apereo.github.io/cas/Mailing-Lists.html
- CAS documentation website: https://apereo.github.io/cas
- CAS project website: https://github.com/apereo/cas
---
You received this message because you are subscribed to the Google Groups "CAS Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cas-user+unsubscribe@apereo.org.

--
- CAS gitter chatroom: https://gitter.im/apereo/cas
- CAS mailing list guidelines: https://apereo.github.io/cas/Mailing-Lists.html
- CAS documentation website: https://apereo.github.io/cas
- CAS project website: https://github.com/apereo/cas
---
You received this message because you are subscribed to the Google Groups "CAS Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cas-user+unsubscribe@apereo.org.
To view this discussion on the web visit https://groups.google.com/a/apereo.org/d/msgid/cas-user/D5BBD223FBD5BCE6.C72A3F6A-13CE-41A8-9121-A06158620A74%40mail.outlook.com.

Tom O'Neill

unread,
Aug 28, 2017, 10:44:12 PM8/28/17
to cas-...@apereo.org

RJ,

 

I could be missing something but your most recent summary of the heap behavior sounds pretty normal.

Your arguments have the JVM heap initializing at 2 GB and maxing out at 4 GB.

 

When garbage collection occurs, some of the memory used by the heap should be freed up.

Sometimes this doesn’t work properly when there is a memory leak or an issue with the cleanup processes within the application.

 

When garbage collection runs how much of the memory is typically getting recovered?

 

Thanks,

 

Tom O’Neill

 

From: cas-...@apereo.org [mailto:cas-...@apereo.org] On Behalf Of RJ
Sent: Monday, August 28, 2017 7:25 PM
To: cas-...@apereo.org
Subject: Re: [cas-user] JVM Heap Kept Growing every day

 

Here is another take at this:

--
- CAS gitter chatroom: https://gitter.im/apereo/cas
- CAS mailing list guidelines: https://apereo.github.io/cas/Mailing-Lists.html
- CAS documentation website: https://apereo.github.io/cas
- CAS project website: https://github.com/apereo/cas
---
You received this message because you are subscribed to the Google Groups "CAS Community" group.

To unsubscribe from this group and stop receiving emails from it, send an email to cas-user+u...@apereo.org.


---
You received this message because you are subscribed to the Google Groups "CAS Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cas-user+u...@apereo.org.

Reply all
Reply to author
Forward
0 new messages