CAS v7.0.0 Performance issue.

[Shavi Teotia]

I have recently updated the cas version on my application from 6.6.2 to 7.0.0.

There is some performance issue, that usually occurs when there is no load on the server.

My CPU utilization graph goes up till 98% and application goes down, start giving 503, we have to restart it or redeploy it.

We checked through the heap dump we found, the issue is related to the ConcurrentIndexedCollection, which is used for registered service indexing.

So my question is can we anyhow disable this or is there any other way to optimize, any specific property that needs to be declared.

Please let me know if any other information is also required.

[Ray Bon]

Shavi,

Could this be related to the storage mechanism you use for services?

Are you able to try a different back end?

Ray

On Fri, 2024-02-23 at 00:09 -0800, Shavi Teotia wrote:

Notice: This message was sent from outside the University of Victoria email system. Please be cautious with links and sensitive information.

[Shavi Teotia]

Hi Ray and Team,

We have an enterprise application, cannot change the backend. But Could you please suggest what would be the impact if the indexing is stopped from this piece of code.

Another point we are using hazelcast registry, is there any specific setting  or property that needs to be done in that case.

[Łukasz Woźniak]

Same happend to Us. We have CAS on AWS in kubernetes. We have git for services and redis for tickets and mfa. We have 2 pod running with Horizontal Pod Autoscaling enabled. Autoscale never grow higher than 3 pods.

When we deploy CAS from version 6.5 to 7.0, CPU is always almost 100%. HPA scaled the app to 15 pods (max).

When I debug the App, I think CAS on 7.0 uses Virtual Thread from Java 21 and not all library is ready for that. 

 Is IT possibile to turn this off ?

Lukas

--
- Website: https://apereo.github.io/cas
- Gitter Chatroom: https://gitter.im/apereo/cas
- List Guidelines: https://goo.gl/1VRrw7
- Contributions: https://goo.gl/mh7qDG
---
You received this message because you are subscribed to the Google Groups "CAS Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cas-user+u...@apereo.org.
To view this discussion on the web visit https://groups.google.com/a/apereo.org/d/msgid/cas-user/6a7653e9-ca89-4e20-84e2-8fa5476e6765n%40apereo.org.

[Fatih Deniz]

Please check the indexes, we have had similar issues with CAS 6.6.10 , find operation was with ticket id and no index was created for it causing full collection scan in mongo db. No idea if it was fixed or not with the newer version.

To view this discussion on the web visit https://groups.google.com/a/apereo.org/d/msgid/cas-user/CAD1CM_ghK9F-GXVXPqG2PEpQmSb4EZUHU%2BNstA9%2B_TN4rYMCsA%40mail.gmail.com.

[Shavi Teotia]

Hi Ray and team,

 Could you please help we are stuck at this point. Its been long we are trying to resolve this.

[Ray Bon]

Shavi,

Hazelcast is not listed as a storage option for services; ConcurrentIndexedCollection is related to service storage.

What do you use for storing services?

Have you tried increasing memory used by the application container?

Ray

[Łukasz Woźniak]

What protocol do you use ? Do You use OAuth? In my situation, I found that on the OAuth there is a bug in 7.0.0. CAS filter out scopes not mentioned in properties for OIDC( it shouldn't be like that). So application connecting with OAuth was doing Ddos, because It get empty scopes.

I know It is not related, but maybe It help.

Prometheus metrics help a lot.

--
- Website: https://apereo.github.io/cas
- Gitter Chatroom: https://gitter.im/apereo/cas
- List Guidelines: https://goo.gl/1VRrw7
- Contributions: https://goo.gl/mh7qDG
---
You received this message because you are subscribed to the Google Groups "CAS Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cas-user+u...@apereo.org.

To view this discussion on the web visit https://groups.google.com/a/apereo.org/d/msgid/cas-user/bcc288ede48052bb1a71aee72047a99789062c80.camel%40uvic.ca.

[Fatih Deniz]

This we just experienced that uses OAuth protocol during a relatively high load scenario, would you be able to give more insights to this issue? What are some quick fix options available if you know any?

To view this discussion on the web visit https://groups.google.com/a/apereo.org/d/msgid/cas-user/CAD1CM_jyyJqzdfrFUFnsneQJLY55C1c5bvW47ywa6afC8QbnRA%40mail.gmail.com.

[Łukasz Woźniak]

When You hit for accessToken CAS filterout claims that are diffrent than declared for OIDC. This happened here org/apereo/cas/support/oauth/web/DefaultOAuth20RequestParameterResolver.java:169 It shouldn't be like that for OAuth

For now, We override this class, and when someone ask for accessToken with own claims we do not remove them.

To view this discussion on the web visit https://groups.google.com/a/apereo.org/d/msgid/cas-user/CAF-bL413ALkMjYn6kjx%2BBVu5SW1k_ACjFd%3Dhfo_FojkS96_%2B4Q%40mail.gmail.com.

[Pablo Vidaurri]

Interesting that you bring up VT with JDK21. There are many articles out there about issues with VT JDK21with pinning issues:

https://www.infoworld.com/article/3713220/java-virtual-threads-hit-with-pinning-issue.html

[Shavi Teotia]

Hi Misagh and team,

I have recently updated the cas version on my application from 6.6.2 to 7.0.0.

There is some performance issue, that usually occurs when there is no load on the server.

My CPU utilization graph goes up till 98% and application goes down, start giving 503, we have to restart it or redeploy it.

We checked through the heap dump we found, the issue is related to the ConcurrentIndexedCollection, which is used for registered service indexing.

Now as per the further analysis we saw a commit where use of ConcurrentIndexedCollection is being restricted with a use of property, which is part of this commit: 

https://github.com/apereo/cas/commit/5faca0050da630404e634922eec5f7a99898b24e

Could you please let us know if this commit could be released in the v7.0.3 or any other interim release.

Currently we are stuck as this is regularly throwing outOfMemory error.

Thanks and Regards

Shavi Teotia