Offline access tokens failing after restart of Keycloak

[Hylton Peimer]

I have Keycloak 9 running under Kubernetes with MySQL.  We have offline_access scoped tokens being used by clients.  Sometimes when the Keycloak pod is restarted the tokens become invalid:

[0m [33m20:44:32,577 WARN  [org.keycloak.events] (default task-419) type=REFRESH_TOKEN_ERROR, realmId=datos, clientId=web, userId=6109f4d3-16f4-45d2-a735-48254f5d5f72, ipAddress=xx.xxx.234.114, error=invalid_token, grant_type=refresh_token, refresh_token_type=Offline, refresh_token_id=8cbc00c0-7708-4cdb-9d06-5a53a3162ac4, client_auth_method=client-secret

The HTTP status code is 400.  What could be causing such a failure to maintain offline sessions after restart of Keycloak?

[Hylton Peimer]

It seems the problem is that we only have one instance of Keycloak running in the Kubernetes cluster. After replicating the instance and setting up with JGroups (KUBE_PING), the problem doesn't seem reproduce.  Does this make sense?

[Phil Fleischer]

That would make sense to me.  I’m working with a rather old version but that behavior sounds right.

As far as I understood in standalone/domain mode, it should rebuild the offline sessions from the database when restarted but this might take some time to complete so perhaps if you force killed or accessed the instance before it completes this cache reload task, it might appear sessions are missing.

If you’re running in HA, then the offline session cache will use the sync settings to determine the number of owner nodes that are used to replicate the cache (default to 2 i believe).  If you kill all the instances owning the cache you’ll lose sessions.  

— Phil

-- 
You received this message because you are subscribed to the Google Groups "Keycloak User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to keycloak-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/keycloak-user/b407cda5-1c57-4a68-a1a1-48eccd072971n%40googlegroups.com.

[Andy To]

Hi, new to this form, and was looking at a similar issue running Keycloak 12.

So, if I am understanding this correctly, when running in HA mode, the offline session cache will be used instead?  

Our setup is running the keycloak 12 container with the standalone-ha.xml mode, and a couple of jboss-cli scripts to modify the configurations accordingly.  We scale up to 10 instances at certain times of the day, and scale back down to 1 instance afterwards.  We currently have cache replication set at 2.  When I had switched to using offline tokens instead of normal refresh tokens, i was hoping that if the token didn't exist in the cache, it would go to the database, but if the above is true, then are my only choices to either increase the cache replication to 10, or to setup an external infinispan cache via hotrod?

Thanks,

Andy

[Andy To]

In case anyone else was wondering about this, it appears that you can force a rebuild of the offline session cache if you bring down ALL keycloak instances, the first instance you bring up will attempt to rebuild the offline sessions in its cache, and subsequent instances will then replicate the sessions between themselves as normal.

As we only upscale/downscale once a day, our interim solution is to downsize to 0 instances, then bring 1 instance up afterwards to rebuild from the DB.  Longer term solution is to setup an external infinispan cache.