Cross-datacenter replication - Remote cache authorization

Maxim Efimov

unread,

Sep 14, 2020, 6:57:06 AM9/14/20

to Keycloak User

Hi!

I am trying to set up cross-datacenter replication mode for Keycloak 10.0.2 and get the following error:

10:34:56,791 FATAL [org.keycloak.services] (ServerService Thread Pool -- 64) org.infinispan.client.hotrod.exceptions.HotRodClientException:Request for messageId=64 returned server error (status=0x85): java.lang.SecurityException: ISPN006017: Unauthorized 'GET' operation

All configuration is made in accordance with the documentation https://www.keycloak.org/docs/10.0/server_installation/#serversetup

An external Infinispan cluster is based on infinispan/server:11.0.3.Final-1 docker image with the following resulting configuration:

infinispan.xml:

<infinispan

xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

xsi:schemaLocation="urn:infinispan:config:11.0 https://infinispan.org/schemas/infinispan-config-11.0.xsd

urn:infinispan:server:11.0 https://infinispan.org/schemas/infinispan-server-11.0.xsd

urn:org:jgroups https://www.jgroups.org/schema/jgroups-4.2.xsd"

xmlns="urn:infinispan:config:11.0"

xmlns:server="urn:infinispan:server:11.0">

<stack-file name="image-tcp" path="jgroups-tcp.xml"/>

<stack-file name="relay-global" path="jgroups-relay.xml"/>

<relay.RELAY2 xmlns="urn:org:jgroups" site="test1" max_site_masters="1"

can_become_site_master="true"/>

<remote-sites default-stack="relay-global">

<remote-site name="test1"/>

<remote-site name="test2"/>

</remote-sites>

</stack>

</jgroups>

<cache-container name="default" statistics="true">

<transport cluster="infinispan"

stack="xsite"/>

</cache-container>

<inet-address value="${infinispan.bind.address:172.21.1.26}"/>

</interface>

</interfaces>

<socket-bindings default-interface="public" port-offset="0">

<socket-binding name="default" port="11222"/>

</socket-bindings>

<security-realms>

<security-realm name="default">

<properties-realm groups-attribute="Roles">

<user-properties path="users.properties" relative-to="infinispan.server.config.path"

plain-text="true"/>

<group-properties path="groups.properties" relative-to="infinispan.server.config.path"/>

</properties-realm>

</security-realm>

</security-realms>

</security>

<hotrod-connector name="hotrod">

<sasl mechanisms="SCRAM-SHA-512 SCRAM-SHA-384 SCRAM-SHA-256 SCRAM-SHA-1 DIGEST-SHA-512 DIGEST-SHA-384 DIGEST-SHA-256 DIGEST-SHA DIGEST-MD5 PLAIN"

qop="auth" server-name="infinispan"/>

</authentication>

</hotrod-connector>

<rest-connector name="rest">

</rest-connector>

</endpoints>

</server>

</infinispan>

Keycloak starts with the following startup script:

cross_cluster_replication.cli: |

embed-server --server-config=standalone-ha.xml --std-out=echo

batch

echo https://www.keycloak.org/docs/10.0/server_installation/index.html#serversetup

echo ** 3.a Add site name to the UDP stack **

/subsystem=jgroups/stack=udp/transport=UDP:write-attribute(name=site,value=test1)

echo ** 3.b Add module org.keycloak.keycloak-model-infinispan to the keycloak cache-container **

/subsystem=infinispan/cache-container=keycloak:write-attribute(name=module,value=org.keycloak.keycloak-model-infinispan)

echo ** 3.c Update replicated-cache work element **

/subsystem=infinispan/cache-container=keycloak/replicated-cache=work/store=remote:add( \

remote-servers=[remote-cache], \

cache=work, \

passivation=false, \

fetch-state=false, \

purge=false, \

preload=false, \

shared=true, \

properties={ \

rawValues=true, \

marshaller=org.keycloak.cluster.infinispan.KeycloakHotRodMarshallerFactory, \

protocolVersion=2.9 \

})

echo ** 3.d Update distributed-cache sessions element **

/subsystem=infinispan/cache-container=keycloak/distributed-cache=sessions/store=remote:add( \

remote-servers=[remote-cache], \

cache=sessions, \

passivation=false, \

fetch-state=false, \

purge=false, \

preload=false, \

shared=true, \

properties={ \

rawValues=true, \

marshaller=org.keycloak.cluster.infinispan.KeycloakHotRodMarshallerFactory, \

protocolVersion=2.9 \

})

echo ** 3.e Update distributed-cache offlineSessions element **

/subsystem=infinispan/cache-container=keycloak/distributed-cache=offlineSessions/store=remote:add( \

remote-servers=["remote-cache"], \

cache=offlineSessions, \

passivation=false, \

fetch-state=false, \

purge=false, \

preload=false, \

shared=true, \

properties={ \

rawValues=true, \

marshaller=org.keycloak.cluster.infinispan.KeycloakHotRodMarshallerFactory, \

protocolVersion=2.9 \

})

echo ** 3.e Update distributed-cache clientSessions element **

/subsystem=infinispan/cache-container=keycloak/distributed-cache=clientSessions/store=remote:add( \

remote-servers=["remote-cache"], \

cache=clientSessions, \

passivation=false, \

fetch-state=false, \

purge=false, \

preload=false, \

shared=true, \

properties={ \

rawValues=true, \

marshaller=org.keycloak.cluster.infinispan.KeycloakHotRodMarshallerFactory, \

protocolVersion=2.9 \

})

echo ** 3.e Update distributed-cache offlineClientSessions element **

/subsystem=infinispan/cache-container=keycloak/distributed-cache=offlineClientSessions/store=remote:add( \

remote-servers=["remote-cache"], \

cache=offlineClientSessions, \

passivation=false, \

fetch-state=false, \

purge=false, \

preload=false, \

shared=true, \

properties={ \

rawValues=true, \

marshaller=org.keycloak.cluster.infinispan.KeycloakHotRodMarshallerFactory, \

protocolVersion=2.9 \

})

echo ** 3.e Update distributed-cache loginFailures element **

/subsystem=infinispan/cache-container=keycloak/distributed-cache=loginFailures/store=remote:add( \

remote-servers=["remote-cache"], \

cache=loginFailures, \

passivation=false, \

fetch-state=false, \

purge=false, \

preload=false, \

shared=true, \

properties={ \

rawValues=true, \

marshaller=org.keycloak.cluster.infinispan.KeycloakHotRodMarshallerFactory, \

protocolVersion=2.9 \

})

echo ** 3.e Update distributed-cache actionTokens element **

/subsystem=infinispan/cache-container=keycloak/distributed-cache=actionTokens/store=remote:add( \

cache=actionTokens, \

remote-servers=["remote-cache"], \

passivation=false, \

fetch-state=false, \

purge=false, \

preload=false, \

shared=true, \

properties={ \

rawValues=true, \

marshaller=org.keycloak.cluster.infinispan.KeycloakHotRodMarshallerFactory, \

protocolVersion=2.9 \

})

echo ** 3.f Add remote socket binding to infinispan server **

/socket-binding-group=standard-sockets/remote-destination-outbound-socket-binding=remote-cache:add(host=infinispan-test, port=11222)

echo ** 3.h Enable debug logging for Infinispan **

/subsystem=logging/logger=org.keycloak.cluster.infinispan:add(level=DEBUG)

/subsystem=logging/logger=org.keycloak.connections.infinispan:add(level=DEBUG)

/subsystem=logging/logger=org.keycloak.models.cache.infinispan:add(level=DEBUG)

/subsystem=logging/logger=org.keycloak.models.sessions.infinispan:add(level=DEBUG)

/subsystem=logging/logger=org.infinispan.client.hotrod:add(level=DEBUG)

echo ** 3.g Enable remoteStoreSecurity **

/subsystem=keycloak-server/spi=connectionsInfinispan/provider=default:write-attribute(name=properties, \

value={ \

cacheContainer=java:jboss/infinispan/container/keycloak, \

remoteStoreSecurityEnabled=true, \

remoteStoreSecurityHotRodEndpoint="infinispan-test:11222", \

remoteStoreSecurityServerName=infinispan, \

remoteStoreSecurityRealm=default, \

remoteStoreSecurityUsername=keycloak_user, \

remoteStoreSecurityPassword=keycloak_password \

} \

)

run-batch

stop-embedded-server

As far as I understand, Keycloak do not enter this method for some reason https://github.com/keycloak/keycloak/blob/969b09f530b3fe6b6a8c3f0d25ad52c1d8b6c970/model/infinispan/src/main/java/org/keycloak/connections/infinispan/RemoteCacheProvider.java#L130 because there are no log records from this method in my Keycloak log, despite the module=org.keycloak.keycloak-model-infinispan is been added to the keycloak cache-container.

I have tried to connect to the external infinispan cluster with a HotRod client 9.4.18 version. Credentials are absolutely correct and I can get data from the cluster.

What could be wrong with this setup?

gilles.etc...@gmail.com

unread,

Sep 25, 2020, 12:14:29 PM9/25/20

to Keycloak User

Hello,

We are facing the same problems when trying to connect a keycloak 10.0.2 instance to an infinispan 11.0.3.FINAL image.

We tried many things with no success, always some Security exception.

==> We did not manage to configure this security as said in the keycloak documentation :

We tried this with no success :

<cache-container name="default" statistics="true">

But we are less lucky than you as I did not manage to configure properly keycloak 10.0.2 to to an 9.4.X infinispan instance...

The server is added .. and excluded...

15:50:57,563 INFO [org.infinispan.client.hotrod.impl.transport.netty.ChannelFactory] (Thread-0) ISPN004014: New server added(172.24.0.2:11222), adding to the pool.
15:50:57,563 INFO [org.infinispan.client.hotrod.impl.transport.netty.ChannelFactory] (Thread-0) ISPN004016: Server not in cluster anymore(infinispan:11222), removing from the pool.

Would you mind sharing a test cli for the kc configuration and the clustered.xml files of the 9.5.X infinispan servers ?

Regards,

Gilles

Maxim Efimov

unread,

Sep 29, 2020, 8:56:07 PM9/29/20

to Keycloak User

Hi Gilles!

I did not try to set up 9.4.X Infinispan. I am still trying to set up 11.0.3.Final and observed the following.

Unfortunately, I did not find the way how to properly configure infinispan/server:11.0.3.Final image using standard configuration options provided by the image creators, so I was forced to make a custom configuration and provide it to the container.

Keycloak is able to make some remote cache operations with anonymous authentication on the ISPN side, but after some time it stucks on required ADMIN permissions.

Error: FATAL [org.keycloak.services] (ServerService Thread Pool -- 62) org.infinispan.client.hotrod.exceptions.HotRodClientException:Request for messageId=136 returned server error (status=0x85): java.lang.SecurityException: ISPN000287: Unauthorized access: subject 'null' lacks 'ADMIN' permission

infinispan.xml config part:

<hotrod-connector name="hotrod">

<policy> <no-anonymous value="true" /> </policy>

</sasl>

</authentication>

</hotrod-connector>

With totally disabled authentication, the Keycloak server starts correctly and writes all needed caches, but only in one region. All caches are replicated correctly to the remote region, but the remote region's Keycloak tries to read sessions cache and has no success with this operation. It just retries until my k8s liveness probe restarts the container.

Remote Keycloak's logs:

00:00:03,808 DEBUG [org.keycloak.models.sessions.infinispan.InfinispanAuthenticationSessionProviderFactory] (ServerService Thread Pool -- 67) Registered cluster listeners

00:00:03,810 DEBUG [org.keycloak.models.sessions.infinispan.InfinispanUserSessionProviderFactory] (ServerService Thread Pool -- 67) Will preload sessions with transaction timeout 300 seconds

00:00:03,819 INFO [org.keycloak.models.sessions.infinispan.InfinispanUserSessionProviderFactory] (ServerService Thread Pool -- 67) Remote store configured for cache 'sessions'

00:00:03,832 INFO [org.keycloak.models.sessions.infinispan.InfinispanUserSessionProviderFactory] (ServerService Thread Pool -- 67) Remote store configured for cache 'clientSessions'

00:00:03,836 INFO [org.keycloak.models.sessions.infinispan.InfinispanUserSessionProviderFactory] (ServerService Thread Pool -- 67) Remote store configured for cache 'offlineSessions'

00:00:03,838 INFO [org.keycloak.models.sessions.infinispan.InfinispanUserSessionProviderFactory] (ServerService Thread Pool -- 67) Remote store configured for cache 'offlineClientSessions'

00:00:03,849 INFO [org.keycloak.models.sessions.infinispan.InfinispanUserSessionProviderFactory] (ServerService Thread Pool -- 67) Remote store configured for cache 'loginFailures'

00:00:03,853 DEBUG [org.keycloak.models.sessions.infinispan.InfinispanUserSessionProviderFactory] (ServerService Thread Pool -- 67) Start pre-loading userSessions from persistent storage

00:00:03,862 DEBUG [org.keycloak.models.sessions.infinispan.initializer.OfflinePersistentUserSessionLoader] (ServerService Thread Pool -- 67) Persistent sessions loaded already.

00:00:03,867 DEBUG [org.keycloak.models.sessions.infinispan.InfinispanUserSessionProviderFactory] (ServerService Thread Pool -- 67) Pre-loading userSessions from persistent storage finished

00:00:03,876 DEBUG [org.keycloak.models.sessions.infinispan.InfinispanUserSessionProviderFactory] (ServerService Thread Pool -- 67) Registered cluster listeners

00:00:03,879 DEBUG [org.keycloak.models.sessions.infinispan.InfinispanUserSessionProviderFactory] (ServerService Thread Pool -- 67) Check pre-loading sessions from remote cache 'offlineClientSessions'

00:00:03,883 DEBUG [org.keycloak.models.sessions.infinispan.remotestore.RemoteCacheSessionsLoader] (ServerService Thread Pool -- 67) Sessions already loaded in current DC. Skip sessions loading from remote cache 'offlineClientSessions'

00:00:03,884 DEBUG [org.keycloak.models.sessions.infinispan.InfinispanUserSessionProviderFactory] (ServerService Thread Pool -- 67) Pre-loading sessions from remote cache 'offlineClientSessions' finished

00:00:03,884 DEBUG [org.keycloak.models.sessions.infinispan.InfinispanUserSessionProviderFactory] (ServerService Thread Pool -- 67) Check pre-loading sessions from remote cache 'sessions'

00:00:03,885 DEBUG [org.keycloak.models.sessions.infinispan.remotestore.RemoteCacheSessionsLoader] (ServerService Thread Pool -- 67) Sessions maybe not yet loaded in current DC. Will load them from remote cache 'sessions'

00:00:04,886 DEBUG [org.keycloak.models.sessions.infinispan.remotestore.RemoteCacheSessionsLoader] (ServerService Thread Pool -- 67) Sessions maybe not yet loaded in current DC. Will load them from remote cache 'sessions'

00:00:05,887 DEBUG [org.keycloak.models.sessions.infinispan.remotestore.RemoteCacheSessionsLoader] (ServerService Thread Pool -- 67) Sessions maybe not yet loaded in current DC. Will load them from remote cache 'sessions'

.... The same log records every 1second ...

00:00:13,765 DEBUG [org.keycloak.models.sessions.infinispan.changes.sessions.CrossDCLastSessionRefreshListener] (EE-ManagedExecutorService-default-Thread-3) Received refreshes. Offline true, refreshes: {}.

00:01:03,835 DEBUG [org.keycloak.models.sessions.infinispan.changes.sessions.CrossDCLastSessionRefreshStore] (Timer-2) Sending lastSessionRefreshes for key 'lastSessionRefreshes'. Refreshes: {}

00:01:03,917 DEBUG [org.keycloak.models.sessions.infinispan.changes.sessions.CrossDCLastSessionRefreshStore] (Timer-2) Sending lastSessionRefreshes for key 'lastSessionRefreshes-offline'. Refreshes: {}

00:01:03,944 DEBUG [org.keycloak.models.sessions.infinispan.remotestore.RemoteCacheSessionsLoader] (ServerService Thread Pool -- 67) Sessions maybe not yet loaded in current DC. Will load them from remote cache 'sessions'

00:01:03,993 DEBUG [org.keycloak.models.sessions.infinispan.changes.sessions.PersisterLastSessionRefreshStore] (Timer-2) Updating 0 userSessions with lastSessionRefresh: 1601424003

00:01:04,945 DEBUG [org.keycloak.models.sessions.infinispan.remotestore.RemoteCacheSessionsLoader] (ServerService Thread Pool -- 67) Sessions maybe not yet loaded in current DC. Will load them from remote cache 'sessions'

.... And then again 'Will load them from remote cache 'sessions' until SIGTERM received ...

I found a similar issue https://issues.redhat.com/browse/KEYCLOAK-6783?attachmentOrder=asc. Marek Posolda mentioned: "On JDG side, there needs to be secured HotRod endpoint deployed in addition to default "unsecured" HotRod endpoint", but it seems that Infinispan 11 cannot be configured with 2 different socket-bindings using the same HotRod protocol, the official Keycloak documentation does not mention this nuance and there are no two different endpoints in the Keycloak repository https://github.com/keycloak/keycloak/tree/10.0.2/testsuite/integration-arquillian/servers/cache-server/jboss/common

пятница, 25 сентября 2020 г. в 19:14:29 UTC+3, gilles.etc...@gmail.com:

Maxim Efimov

unread,

Nov 4, 2020, 10:24:02 AM11/4/20

to Keycloak User

Unfortunately, my post on https://keycloak.discourse.group/ has been blocked by the automatic anti-spam plugin.
All attempts to fix the post or contact the forum administrators were unsuccessful. It is very disappointing.