CAS 6.6.9 Heap memory issue

[Sathish Sekar]

Hi Team,

I have upgraded to CAS 6.3.7.4 to 6.6.9 recently and I'm facing java heap memory error. In CAS 6.3.7.4 I didn't face such issues in prod environment. When I upgraded to CAS 6.6.9 I had faced this issue thrice and I have posted this in community and as per their feedback and as per the CAS documentation I have upgraded memory maximum 2GB as earlier I have maximum of 512mb of memory in prod. After upgrade application is running for 10 days and suddenly it stopped showing java heap memory error in catalina.out.log. Kindly suggest any other possible solutions for heap memory errors so that I can fix in prod environment.

Note:

Since I have vulnerabilities in CAS 6.6.9 I have excluded following jars and included non vulnerable version which is highlighted in bold.

spring-security-config-5.7.3.jar
spring-security-config-5.7.10.jar
spring-security-web-5.7.3.jar
spring-security-web-5.7.10.jar
spring-web-5.3.22.jar
spring-web-5.3.30.jar
hazelcast-5.1.3.jar
hazelcast-5.1.7.jar
hazelcast-sql-5.1.3.jar
hazelcast-sql-5.1.7.jar
spring-boot-actuator-autoconfigure-2.7.3.jar
spring-boot-actuator-autoconfigure-2.7.11.jar
json-20160810.jar
json-20230227.jar
json-smart-2.4.8.jar
json-smart-2.4.9.jar
spring-webmvc-5.3.22.jar
spring-webmvc-5.3.30.jar
spring-boot-autoconfigure-2.7.3.jar
spring-boot-autoconfigure-2.7.11.jar
hjson-3.0.0.jar
hjson-3.0.1.jar
guava-30.1.1-jre.jar
guava-32.0.1-jre.jar
tomcat-catalina-9.0.75.jar
tomcat-catalina-9.0.76.jar
tomcat-embed-core-9.0.75.jar
tomcat-embed-core-9.0.76.jar
jose4j-0.8.0.jar
jose4j-0.9.3.jar
jersey-common-2.25.1.jar
jersey-common-2.34.jar
bcprov-jdk18on-1.71.jar
bcprov-jdk18on-1.74.jar
spring-expression-5.3.22.jar
spring-expression-5.3.30.jar
spring-security-crypto-5.7.3.jar
spring-security-crypto-5.7.10.jar
woodstox-core-6.3.1.jar
woodstox-core-6.4.0.jar
jackson-databind-2.13.4.jar
jackson-databind-2.13.5.jar
jackson-core-2.13.4.jar
jackson-core-2.13.5.jar

[Mohamed Amdouni]

Hi,

Please post the stack trace 

Did you configurer the Jvm to generate a heap dump when out of memory exception is encouraged ?

Is it about meta space ?

Best regards 

--
- Website: https://apereo.github.io/cas
- Gitter Chatroom: https://gitter.im/apereo/cas
- List Guidelines: https://goo.gl/1VRrw7
- Contributions: https://goo.gl/mh7qDG
---
You received this message because you are subscribed to the Google Groups "CAS Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cas-user+u...@apereo.org.
To view this discussion on the web visit https://groups.google.com/a/apereo.org/d/msgid/cas-user/0880771f-aff5-4009-a892-0bdcdacc4ecbn%40apereo.org.

[Sathish Sekar]

Hi Team,

I don't have any specific error printed in logs. In catalina.out.log in middle java heap space error occurred. Kindly check the below error.

[185818.716s][info][gc] GC(48064) Pause Full (G1 Evacuation Pause) 511M->496M(512M) 448.791ms
[185818.716s][info][gc] GC(48065) Concurrent Cycle 449.268ms
Heap dump file created [621232465 bytes in 9.160 secs]
[185822.515s][info][gc] GC(48066) Pause Young (Concurrent Start) (G1 Humongous Allocation) 503M->500M(512M) 16.183ms
[185822.516s][info][gc] GC(48067) Concurrent Cycle
26-Oct-2023 21:05:38.023 INFO [Membership-MemberExpired] org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.memberDisappeared Received memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 34, 196, 42}:23022,{10, 34, 196, 42},23022, alive=185626428, securePort=-1, UDP Port=-1, id={41 -111 -19 13 -41 -24 67 -48 -83 40 -125 119 -16 -128 -60 24 }, payload={}, command={}, domain={}]] message. Will verify.
java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
26-Oct-2023 21:05:38.080 INFO [Membership-MemberExpired] org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.memberDisappeared Verification complete. Member still alive[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 34, 196, 42}:23022,{10, 34, 196, 42},23022, alive=185626428, securePort=-1, UDP Port=-1, id={41 -111 -19 13 -41 -24 67 -48 -83 40 -125 119 -16 -128 -60 24 }, payload={}, command={}, domain={}]]
[185822.682s][info][gc] GC(48068) To-space exhausted
[185822.682s][info][gc] GC(48068) Pause Young (Normal) (G1 Evacuation Pause) 509M->509M(512M) 23.104ms
[185822.716s][info][gc] GC(48069) To-space exhausted
[185822.716s][info][gc] GC(48069) Pause Young (Normal) (G1 Evacuation Pause) 510M->510M(512M) 7.140ms
[185823.269s][info][gc] GC(48070) Pause Full (G1 Evacuation Pause) 510M->502M(512M) 552.151ms
[185823.281s][info][gc] GC(48067) Concurrent Cycle 765.269ms

[Sathish Sekar]

Hi Team,

Kindly help on the below exception for memory issue.

java.lang.OutOfMemoryError: Java heap space at java.lang.Integer.valueOf(Integer.java:1059) ~[?:?] at sun.nio.ch.EPollSelectorImpl.processEvents(EPollSelectorImpl.java:194) ~[?:?] at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:137) ~[?:?] at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124) ~[?:?] at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:136) ~[?:?] at com.hazelcast.internal.networking.nio.NioThread.selectLoop(NioThread.java:292) ~[hazelcast-5.1.7.jar:5.1.7] at com.hazelcast.internal.networking.nio.NioThread.executeRun(NioThread.java:249) ~[hazelcast-5.1.7.jar:5.1.7] at com.hazelcast.internal.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:102) ~[hazelcast-5.1.7.jar:5.1.7]

On Thursday, November 16, 2023 at 1:01:30 PM UTC+5:30 Mohamed Amdouni wrote:

[Sathish Sekar]

Hi Team,

Additionally I can find exceptions for java heap memory error in prod. Kindly do needful please

AsyncLogger error handling event seq=11325854, value='Logger=com.hazelcast.spi.impl.operationexecutor.impl.OperationExecutorImpl Level=ERROR Message=[101.34.192.43]:5701 [dev] [5.1.7] Java heap space': org.apache.logging.log4j.core.appender.AppenderLoggingException: java.lang.NoClassDefFoundError: Could not initialize class org.apache.logging.log4j.util.PrivateSecurityManagerStackTraceUtil

AsyncLogger error handling event seq=11325852, value='Logger=com.hazelcast.internal.partition.operation.FinalizePromotionOperation Level=ERROR Message=[101.34.192.43]:5701 [dev] [5.1.7] Java heap space': org.apache.logging.log4j.core.appender.AppenderLoggingException: java.lang.NoClassDefFoundError: Could not initialize class org.apache.logging.log4j.util.PrivateSecurityManagerStackTraceUtil  

org.apache.logging.log4j.core.appender.AppenderLoggingException: java.lang.OutOfMemoryError: Java heap space at org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(AppenderControl.java:165) at org.apache.logging.log4j.core.config.AppenderControl.callAppender0(AppenderControl.java:134) at org.apache.logging.log4j.core.config.AppenderControl.callAppenderPreventRecursion(AppenderControl.java:125) at org.apache.logging.log4j.core.config.AppenderControl.callAppender(AppenderControl.java:89) at org.apache.logging.log4j.core.config.LoggerConfig.callAppenders(LoggerConfig.java:675) at org.apache.logging.log4j.core.async.AsyncLoggerConfig.callAppenders(AsyncLoggerConfig.java:149) at org.apache.logging.log4j.core.config.LoggerConfig.processLogEvent(LoggerConfig.java:633) at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:616) at org.apache.logging.log4j.core.async.AsyncLoggerConfig.log(AsyncLoggerConfig.java:143) at org.apache.logging.log4j.core.async.AsyncLoggerConfig.logToAsyncLoggerConfigsOnCurrentThread(AsyncLoggerConfig.java:191) at org.apache.logging.log4j.core.async.AsyncLoggerConfigDisruptor$Log4jEventWrapperHandler.onEvent(AsyncLoggerConfigDisruptor.java:112) at org.apache.logging.log4j.core.async.AsyncLoggerConfigDisruptor$Log4jEventWrapperHandler.onEvent(AsyncLoggerConfigDisruptor.java:98) at com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:168) at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125) at java.base/java.lang.Thread.run(Thread.java:829)

AsyncLogger error handling event seq=11325849, value='Logger=com.hazelcast.spi.impl.operationexecutor.impl.OperationExecutorImpl Level=ERROR Message=[101.34.192.43]:5701 [dev] [5.1.7] Failed to process: com.hazelcast.internal.partition.operation.FinalizePromotionOperation{serviceName='null', identityHash=1381425466, partitionId=44, replicaIndex=0, callId=0, invocationTime=-1 (1969-12-31 15:59:59.999), waitTimeout=-1, callTimeout=9223372036854775807, tenantControl=com.hazelcast.spi.impl.tenantcontrol.NoopTenantControl@0} on: hz.10.34.196.43.partition-operation.thread-0': org.apache.logging.log4j.core.appender.AppenderLoggingException: java.lang.OutOfMemoryError: Java heap space

Log4j2-TF-15-AsyncLoggerConfig-11 ERROR An exception occurred processing Appender casFile org.apache.logging.log4j.core.appender.AppenderLoggingException: java.lang.OutOfMemoryError: Java heap space at org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(AppenderControl.java:165) at org.apache.logging.log4j.core.config.AppenderControl.callAppender0(AppenderControl.java:134) at org.apache.logging.log4j.core.config.AppenderControl.callAppenderPreventRecursion(AppenderControl.java:125) at org.apache.logging.log4j.core.config.AppenderControl.callAppender(AppenderControl.java:89) at org.apache.logging.log4j.core.config.LoggerConfig.callAppenders(LoggerConfig.java:675) at org.apache.logging.log4j.core.async.AsyncLoggerConfig.callAppenders(AsyncLoggerConfig.java:149) at org.apache.logging.log4j.core.config.LoggerConfig.processLogEvent(LoggerConfig.java:633) at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:616) at org.apache.logging.log4j.core.async.AsyncLoggerConfig.log(AsyncLoggerConfig.java:143) at org.apache.logging.log4j.core.async.AsyncLoggerConfig.logToAsyncLoggerConfigsOnCurrentThread(AsyncLoggerConfig.java:191) at org.apache.logging.log4j.core.async.AsyncLoggerConfigDisruptor$Log4jEventWrapperHandler.onEvent(AsyncLoggerConfigDisruptor.java:112) at org.apache.logging.log4j.core.async.AsyncLoggerConfigDisruptor$Log4jEventWrapperHandler.onEvent(AsyncLoggerConfigDisruptor.java:98) at com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:168) at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125) at java.base/java.lang.Thread.run(Thread.java:829)

Exception in thread "HTTP-Dispatcher" java.lang.OutOfMemoryError: Java heap space at java.base/java.util.HashMap$KeySet.iterator(HashMap.java:913) at java.base/java.util.HashSet.iterator(HashSet.java:173) at java.base/sun.nio.ch.Util$2.iterator(Util.java:352)

[Mohamed Amdouni]

These exceptions will not help you. They are consequences. Check instead the server startup log files. Your memory settings as sayed before. 

[Doug C]

I am wondering if this is related to https://github.com/apereo/cas/pull/5652.  We have been discussing this issue because myself and some others have experienced a slowdown due to the merged pull request but the fix provided by the pull request is related to fixing out-of-memory issues with the embedded Tomcat.

[Winson Chen]

Hello,

Are you hazelcast registry?

if so, we encountered a similar problem in our prod environment. Especially after we converted one of our popular app from cas to Oauth. Here is what our config that made out cas stable, without restarting on a schedule

cas.ticket.registry.hazelcast.cluster.core.eviction-policy=LRU
cas.ticket.registry.hazelcast.cluster.core.max-size-policy=USED_HEAP_PERCENTAGE
cas.ticket.registry.hazelcast.cluster.core.max-size=50

VM has 9G of ram

OpenJDK 11

 -Xms6144m -Xmx6144m

Hopefully this helps

Winson