Excessive memory usage for keycloak 11.0.3 and incorrect last refresh date

[Ryan Liu]

Hi,

We have a keycloak cluster on GKE with version 11.0.3 deployed using the codecentric/keycloak helm chart version of 9.5.0. In the cluster there are 3 replicas of keycloak and 1 PostgreSQL DB. 

Based on the observation, we found some issues that it couldn't explain itself:

1. The traffic comes into our cluster will increase in a specific time during the date. Each time when the traffic comes, the memory usage of keycloak will increase by around 2GB and it never goes down. Is this an expected behaviour of keycloak?

2. We are expecting GC to do the job to stabilize memory usage; however we see that memory usage has been increasing during days until the jboss server received a kill signal. 

3. We are using offline tokens with default settings ( refresh token revoke disabled, 30 days idle timeout and 5 mins lifespan for access token, etc). In the cluster, there are only around 50 offline sessions and 50 active sessions. Will the offline sessions ( sticky) keep  consuming more and more memory?

4. We also notice that the Last Refresh date/time always shown in year 1970. Is this related to any of our settings?

Below figure shows our keycloak instances would crash due to out of resources once a while:

Below figure shows whenever the traffic is growing in the cluster during a day, the memory usage by keycloak would increase by around 2GB each time and then remain relatively stable for the rest of the day. 

Below is the configuration for our deployment:

Below is the last refresh date of the offline sessions:

Any help and information is appreciated. Thanks in advance!

[Phil Fleischer]

Are you quite sure you only have 50 offline sessions?  Can you do a count on the backing table in the database?? The ui might have a limit.

In my experience the offline session is a major contributor to cache usage which would never get cleaned up by garbage collector from old gen space and will immediately return even after a restart because the system will refill the cache from the database.

We had a particular client that was requesting offline access but was never reusing the token and logging in with credentials every time. Later on as well someone increased the expiration way higher than default and the memory grew to infinity.

---

From: keyclo...@googlegroups.com <keyclo...@googlegroups.com> on behalf of Ryan Liu <wyan...@gmail.com>
Sent: Thursday, December 24, 2020 3:47:48 PM
To: Keycloak User <keyclo...@googlegroups.com>
Subject: [keycloak-user] Excessive memory usage for keycloak 11.0.3 and incorrect last refresh date

 

--
You received this message because you are subscribed to the Google Groups "Keycloak User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to keycloak-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/keycloak-user/01bd3848-8fa7-4d5b-b5fa-ac11c55be11dn%40googlegroups.com.

[Ryan Liu]

Hi Phil, 

Thanks for the reply. I totally agree that it seems like an issue related to sticky sessions. 

I just did following queries:

keycloak=> SELECT COUNT(*) FROM public.offline_user_session; 

 count ------- 34 (1 row) 

 keycloak=> SELECT COUNT(*) FROM public.offline_client_session; 

 count ------- 34 (1 row)

It seems like the number of offline session is not more than what we saw in the UI. Is there another table(s) should I verify?  

[Ryan Liu]

The memory usage has been growing during the holiday.... I know that we set MaxRamPercentage to 80 and it has passed the threshold. 

There was no obvious GC happened. 

Below is a mem usage of  each pod:

Any ideas?

[Thomas Darimont]

Hello Ryan,

that's quite odd... you could try to exec into a Keycloak container and print the current class histogram to get an idea for what might consume so much memory. 

You can use 

    $JAVA_HOME/bin/jps -lmv 

or

  $JAVA_HOME/bin/jcmd 

or 

  ps auxw | grep java

to find the pid of the keycloak process, then run:

$JAVA_HOME/bin/jcmd $KEYCLOAK_PID  GC.class_histogram 

See also:

https://dzone.com/articles/jvm-tuning-using-jcmd

Cheers,

Thomas

To view this discussion on the web visit https://groups.google.com/d/msgid/keycloak-user/36e598b7-2413-40ff-8760-7b65fe2a964cn%40googlegroups.com.

[Ryan Liu]

Hi Thomas, 

Thanks for the tips. Below's what I got from the container:

It seems like Infinispan is not actually taking too much memory. I can't really tell what's in the other caches though. 

[Ryan Liu]

From previous memory usage histogram we can see that AuthenticatedClientSessionEntity & ActionTokenValueEntity contributed to most of the memory..

Also we did a test without offline_tokens, it seems like even with regular refresh token the usage of memory was still growing fast...

Maybe we did something incorrectly from the client side? We are using spring boot adapter. 

[Ryan Liu]

Below's the test config we are using to verify the problem, which is the same as what we use in production.