Keycloak Cross Datacenter Replication with external Infinispan server

[Yogesh Dragon]

Hi, hopefully someone has experience of how to setup keycloak with

infinispan as a remote cache.

My goal is to run keycloak across multiple data centres with replicated

database and cache, however i am struggling to get anything working.

I am running keycloak 12.0.4 and infinispan 9.4.20

I have 2 unclustered keycloak instances (A and B), sharing a remote cache

and H2 backend.

When i log in to the admin console on one of the instances (lets say A),

through the shared remote store i'm expecting to see the session on the

other keycloak instance (B).  Using the infinispan manager console i can

see that the sessions are being cached as the cache content entries value

increments, from both directions (A + B), however, the values never appear

within the other instance.

Here are snippets of my clustered.xml and standalone-ha.xml

clustered.xml

\<replicated-cache-configuration name="sessions-cfg" mode="SYNC"

start="EAGER" batching="false">

\<transaction mode="NON_XA" locking="PESSIMISTIC"/>

\</replicated-cache-configuration>

                    \<transaction mode="NON_XA" locking="PESSIMISTIC"/>

                    \<backups>

                       \<backup site="site1" failure-policy="FAIL" strategy="SYNC" enabled="true"/>

                    \</backups>

\<replicated-cache name="work" configuration="sessions-cfg" />

\<replicated-cache name="sessions" configuration="sessions-cfg" />

\<replicated-cache name="clientSessions" configuration="sessions-cfg"/>

\<replicated-cache name="offlineSessions" configuration="sessions-cfg" />

\<replicated-cache name="offlineClientSessions"

configuration="sessions-cfg"/>

\<replicated-cache name="actionTokens" configuration="sessions-cfg" />

\<replicated-cache name="loginFailures" configuration="sessions-cfg" />

standalone-ha.xml

\<replicated-cache name="work" mode="SYNC">

\<remote-store cache="work" remote-servers="remote-cache"

passivation="false" fetch-state="false" purge="false" preload="false"

shared="true">

\<property name="rawValues">true</property>

\<property

name="marshaller">org.keycloak.cluster.infinispan.KeycloakHotRodMarshallerFactory</property>

\</remote-store>

\</replicated-cache>

\<distributed-cache name="sessions" mode="SYNC" owners="1">

\<remote-store cache="sessions" remote-servers="remote-cache"

passivation="false" fetch-state="false" purge="false" preload="false"

shared="true">

\<property name="rawValues">true</property>

\<property

name="marshaller">org.keycloak.cluster.infinispan.KeycloakHotRodMarshallerFactory</property>

\</remote-store>

\</distributed-cache>

\<distributed-cache name="offlineSessions" mode="SYNC" owners="1">

\<remote-store cache="offlineSessions" remote-servers="remote-cache"

passivation="false" fetch-state="false" purge="false" preload="false"

shared="true">

\<property name="rawValues">true</property>

\<property

name="marshaller">org.keycloak.cluster.infinispan.KeycloakHotRodMarshallerFactory</property>

\</remote-store>

\</distributed-cache>

\<distributed-cache name="clientSessions" mode="SYNC" owners="1">

\<remote-store cache="clientSessions" remote-servers="remote-cache"

passivation="false" fetch-state="false" purge="false" preload="false"

shared="true">

\<property name="rawValues">true</property>

\<property

name="marshaller">org.keycloak.cluster.infinispan.KeycloakHotRodMarshallerFactory</property>

\</remote-store>

\</distributed-cache>

\<distributed-cache name="offlineClientSessions" mode="SYNC" owners="1">

\<remote-store cache="offlineClientSessions" remote-servers="remote-cache"

passivation="false" fetch-state="false" purge="false" preload="false"

shared="true">

\<property name="rawValues">true</property>

\<property

name="marshaller">org.keycloak.cluster.infinispan.KeycloakHotRodMarshallerFactory</property>

\</remote-store>

\</distributed-cache>

\<distributed-cache name="loginFailures" mode="SYNC" owners="1">

\<remote-store cache="loginFailures" remote-servers="remote-cache"

passivation="false" fetch-state="false" purge="false" preload="false"

shared="true">

\<property name="rawValues">true</property>

\<property

name="marshaller">org.keycloak.cluster.infinispan.KeycloakHotRodMarshallerFactory</property>

\</remote-store>

\</distributed-cache>

\<distributed-cache name="actionTokens" mode="SYNC" owners="2">

\<eviction max-entries="-1" strategy="NONE"/>

\<expiration max-idle="-1" interval="300000"/>

\<remote-store cache="actionTokens" remote-servers="remote-cache"

passivation="false" fetch-state="false" purge="false" preload="true"

shared="true">

\<property name="rawValues">true</property>

\<property

name="marshaller">org.keycloak.cluster.infinispan.KeycloakHotRodMarshallerFactory</property>

\</remote-store>

\</distributed-cache>

\<distributed-cache name="authenticationSessions" mode="SYNC" owners="1"/>

Backend log from keycloak server 

2021-04-19 20:24:46,516 DEBUG [org.keycloak.models.sessions.infinispan.remotestore.RemoteCacheSessionListener] (Thread-0) Received event from remote store. Event 'CLIENT_CACHE_ENTRY_MODIFIED', key '30abf6ae-cdc6-4691-a6e7-2dff27198178', skip 'true'

I guess the "skip" parameter should be false according to official keycloak documentation .

any ideas on what i might be missing?

Thanks

Yogi