CAS 6.6.9 Hazelcast and Ticket Registry errors

[Sathish Sekar]

Hi Team,

I'm upgrading CAS 6.3.7.4 to 6.6.9. The below properties I have added for Hazelcast and Ticket Registry in upgraded version. I'm seeing this error which is printed in logs often. Kindly check below properties and guide me if any modification/addition/deletion of properties.

ERROR [org.apereo.cas.ticket.registry.HazelcastTicketRegistry] - Unable to locate ticket map for ticket definition [DefaultTicketDefinition(implementationClass=class org.apereo.cas.ticket.TicketGrantingTicketImpl, apiClass=interface org.apereo.cas.ticket.TicketGrantingTicket, prefix=TGT, properties=DefaultTicketDefinitionProperties(cascadeRemovals=false, storageName=ticketGrantingTicketsCache, storageTimeout=9223372036854775807, storagePassword=null, excludeFromCascade=false), order=2147483647)]
2023-10-26 21:14:08,694 WARN [org.apereo.cas.ticket.registry.HazelcastTicketRegistry] - No ticket definition could be found in the catalog to match [TGT-16318-********]
2023-10-26 21:14:08,694 ERROR [org.apereo.cas.ticket.registry.HazelcastTicketRegistry] - HazelcastInstance[[101.34.196.41]:5701] is not active!
ProxyRegistry.java:getService:91
ProxyRegistry.java:<init>:73
ProxyServiceImpl.java:lambda$new$0:84

#hazelcast properties

cas.ticket.registry.hazelcast.page-size=500

cas.ticket.registry.hazelcast.cluster.network.members=101.34.194.57,101.34.194.58,101.34.194.59

cas.ticket.registry.hazelcast.cluster.core.instance-name=101.34.194.57

cas.ticket.registry.hazelcast.cluster.network.port=5701

cas.ticket.registry.hazelcast.core.enable-compression=false

cas.ticket.registry.hazelcast.core.enable-management-center-scripting=true

cas.ticket.registry.hazelcast.crypto.enabled=true

cas.ticket.registry.hazelcast.cluster.core.async-backup-count=1

cas.ticket.registry.hazelcast.cluster.core.backup-count=0

cas.ticket.registry.hazelcast.crypto.signing.key=**************

cas.ticket.registry.hazelcast.crypto.signing.keySize=512

cas.ticket.registry.hazelcast.crypto.encryption.key=***********

cas.ticket.registry.hazelcast.crypto.encryption.keySize=16

cas.ticket.registry.hazelcast.crypto.alg=AES

cas.ticket.tgt.remember-me.enabled=true

cas.ticket.tgt.remember-me.time-to-kill-in-seconds=28800

cas.ticket.tgt.timeout.max-time-to-live-in-seconds=28800

cas.ticket.tgt.hard-timeout.time-to-kill-in-seconds=28800

cas.tgc.pin-to-session=true

cas.tgc.remember-me-max-age=P30D

cas.tgc.auto-configure-cookie-path=true

cas.tgc.max-age=-1

cas.tgc.domain=sathishsekar.com

cas.tgc.name=CASTGC

cas.tgc.secure=true

cas.tgc.http-only=true

cas.tgc.crypto.enabled=false

[Jonathon Taylor]

Not sure why it's complaining about an IP address that's not in your configuration.  I wonder if it's picking up a second IP that's configured?  You could try setting this:

cas.ticket.registry.hazelcast.cluster.core.instance-name=localhost

cas.ticket.registry.hazelcast.cluster.network.public-address=101.34.194.57:5701

--
- Website: https://apereo.github.io/cas
- Gitter Chatroom: https://gitter.im/apereo/cas
- List Guidelines: https://goo.gl/1VRrw7
- Contributions: https://goo.gl/mh7qDG
---
You received this message because you are subscribed to the Google Groups "CAS Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cas-user+u...@apereo.org.
To view this discussion on the web visit https://groups.google.com/a/apereo.org/d/msgid/cas-user/bfd02539-1e72-4e46-98d2-e98969ae8538n%40apereo.org.

--

Jonathon Taylor (he/him)

Information Security Office

jona...@berkeley.edu

[Sathish Sekar]

Hi Team,

Prod server stopped due to following issues. Kindly help. I'm facing heap memory issue for long time :(

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "hz.10.34.196.43.MetricsRegistry.thread-2"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "NioReceiver[Catalina-Channel]"
cpa-cas.tomcat is down code=28 .
cpa-cas.tomcat is down code=28 .
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "hz.10.34.196.43.MetricsRegistry.thread-1"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Catalina-utility-1"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Catalina-utility-2"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Tribes-MembershipReceiver[Catalina-Channel]"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "idle-timeout-task"
java.lang.OutOfMemoryError: Java heap space
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Tribes-MembershipReceiver[Catalina-Channel]"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "hz.10.34.196.41.MetricsRegistry.thread-2"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "PooledConnectionFactory@1754546700"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Tribes-MembershipSender[Catalina-Channel]"

Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.hazelcast.core.HazelcastInstance]: Factory method 'casTicketRegistryHazelcastInstance' threw exception; nested exception is java.lang.IllegalStateException: Node failed to start!
                at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:185)
                at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:653)
                ... 131 more
        Caused by: java.lang.IllegalStateException: Node failed to start!
                at com.hazelcast.instance.impl.HazelcastInstanceImpl.<init>(HazelcastInstanceImpl.java:126)
                at com.hazelcast.instance.impl.HazelcastInstanceFactory.constructHazelcastInstance(HazelcastInstanceFactory.java:217)
                at com.hazelcast.instance.impl.HazelcastInstanceFactory.getOrCreateHazelcastInstance(HazelcastInstanceFactory.java:114)
                at org.apereo.cas.config.HazelcastTicketRegistryConfiguration.casTicketRegistryHazelcastInstance(HazelcastTicketRegistryConfiguration.java:73)
                at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
                at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
                at java.base/java.lang.reflect.Method.invoke(Method.java:566)
                at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:154)

[Sathish Sekar]

Hi Team,

Hazelcast is causing problem. Since we have 3 servers clustered. Earlier we have tomcat memory of 512mb min and max. I posted in CAS community and they suggested to increase memory by max 2gb. It is running fine for 12 days and stopped showing heap memory issue. The following are capturing during heap error

1. Catalina.out.log size is more than 500 mb

2. facing above hazelcast errors in logs

KINDLY DO NEEDFUL

[Meysam Shirazi]

Hi Sathish

Did you check the port(5701) access and connectivity of hazelcast members?

you can set the hazelcast.log.level to debug for more details in log.

[Ray Bon]

Sathish,

Your async-backup-count and backup-count have non default values. Is it possible these values are causing hazelcast to consume memory? 

Try using default values to see if memory use improves.

You can monitor the JVM with JDK Misson Control or jConsole.

Ray

P.S. To keep your config file as simple as possible, only add properties that are set to non default values.

P.P.S. The value for instance-name is usually something human readable and applies to all hazelcast members, such as 'cas-preprod' rather than an ip address.

On Tue, 2023-11-14 at 09:54 -0800, Sathish Sekar wrote:

Notice: This message was sent from outside the University of Victoria email system. Please be cautious with links and sensitive information.