URGENT : Memory leak issue in PROD

[Morning Star]

Hi Team,

CAS Server version :  6.3.7.4

We are facing a serious memory leak issue in production.
After analyzing the dump file, we could see below observation.

Java Static com.hazelcast.instance.impl.OutOfMemoryErrorDispatcher.SERVER_INSTANCES_REF
<<Object might be causing memory leak.>>

Can someone help me with the fix or way to address this issue? Will be really thankful if someone help.

Regards,
Morning Star.

[Ray Bon]

Morning Star,

Other than our site specific hazelcast settings, these are the only other settings. We have not experienced a memory issue.

cas.ticket.registry.hazelcast.cluster.multicastEnabled=false

cas.ticket.registry.hazelcast.cluster.tcpipEnabled=false

Ray

On Sun, 2022-02-27 at 07:57 -0800, Morning Star wrote:

Notice: This message was sent from outside the University of Victoria email system. Please be cautious with links and sensitive information.

-- 

Ray Bon

Programmer Analyst

Development Services, University Systems

2507218831 | CLE 019 | rb...@uvic.ca

I acknowledge and respect the lək̓ʷəŋən peoples on whose traditional territory the university stands, and the Songhees, Esquimalt and WSÁNEĆ peoples whose historical relationships with the land continue to this day.

[Morning Star]

Thanks Ray for your response. Let me try this option & get back.

Regards,

Morning.

[Morning Star]

Hi Ray,

Your help really means a lot to me.

Adding more info: As of now, we have these hazelcast properties in production.

Please take a look & let me know if any of these configuration leads to memory leak.

cas.ticket.registry.hazelcast.page-size=500
cas.ticket.registry.hazelcast.cluster.members=10.xx...,10.xx..,10.xx..
cas.ticket.registry.hazelcast.cluster.instance-name=10.xx..
cas.ticket.registry.hazelcast.cluster.port=5701
cas.ticket.registry.hazelcast.enable-compression=false
cas.ticket.registry.hazelcast.enable-management-center-scripting=true
cas.ticket.registry.hazelcast.crypto.enabled=true
cas.ticket.registry.hazelcast.cluster.asyncBackupCount=1
cas.ticket.registry.hazelcast.cluster.backupCount=0
cas.ticket.registry.hazelcast.crypto.signing.key=keyyyyyyyyyyyyyyyyyyyyyyy
cas.ticket.registry.hazelcast.crypto.signing.keySize=512
cas.ticket.registry.hazelcast.crypto.encryption.key=keyyyyyyyyyyyyyyyyyyyyyyy
cas.ticket.registry.hazelcast.crypto.encryption.keySize=16
cas.ticket.registry.hazelcast.crypto.alg=AES

cas.ticket.registry.cleaner.schedule.start-delay=PT20S
cas.ticket.registry.cleaner.schedule.repeat-interval=PT3600000S
cas.ticket.registry.cleaner.schedule.enabled=true

Regards,

Morning.

[Ray Bon]

Morning,

The default for repeat-interval is 30s. You have it set for about 20d. 

There are a few other properties that differ from the default values (async-backup-count, backup-count, start-delay).

Are you sure you want those property values changed?

Try the defaults and see if the memory problems go away. Then add the properties one at a time.

Ray