Infinispan user session cache with jdbc-store write-through persistence

[Thomas Darimont]

Hello Keycloak Developers,

Infinispan supports storing cache contents in a database via the jdbc-store module, however as of Keycloak 15.0.2 a write-through is not possible 

with Keycloak because the code in `org.keycloak.models.sessions.infinispan.changes.InfinispanChangelogBasedTransaction` is configured to explicitly 

skip the write to any configured cache-store via `CacheDecorators.skipCacheStore(cache)`.

If one could configure `InfinispanChangelogBasedTransaction` to propagate cache changes to a cache store, if a cache store is present for a cache, 

then one could store cache data in an external store, which survives restarts without the need for an external infinispan cluster. 

As a PoC I did a small patch to the `org.keycloak.models.sessions.infinispan.CacheDecorators` and configured a jdbc-store for the user `sessions` cache:

```

public class CacheDecorators {

    private static final boolean IGNORE_SKIP_CACHE_STORE=Boolean.getBoolean("keycloak.infinispan.ignoreSkipCacheStore");

...

    public static <K, V> AdvancedCache<K, V> skipCacheStore(Cache<K, V> cache) {

        if (IGNORE_SKIP_CACHE_STORE) {

            return cache.getAdvancedCache();

        }

        return             cache.getAdvancedCache().withFlags(Flag.SKIP_CACHE_STORE);

    }

}

```

If I set the system property `-Dkeycloak.infinispan.ignoreSkipCacheStore=true` when I start a Keycloak instance, then cache writes are propagated to 

the configured cache-store.

```

<distributed-cache name="sessions" owners="1">

    <expiration lifespan="900000000000000000"/>

    <jdbc-store data-source="KeycloakDS" max-batch-size="1000" fetch-state="true" passivation="false" preload="false" purge="false" shared="true">

        <property name="databaseType">POSTGRES</property>

        <table fetch-size="5000" drop-on-stop="false" prefix="ispn">

            <data-column type="bytea"/>

        </table>

    </jdbc-store>

</distributed-cache>

```

The following example .cli script shows how to configure a JDBC cache store for the user sessions cache that is backed by a postgres database.

Note that we use the KeycloakDS datasource here.

0300-onstart-setup-ispn-jdbc-store.cli:

```

embed-server --server-config=${env.KEYCLOAK_CONFIG_FILE:standalone-ha.xml} --std-out=echo

echo Using server configuration file:

:resolve-expression(expression=${env.JBOSS_HOME}/standalone/configuration/${env.KEYCLOAK_CONFIG_FILE:standalone-ha.xml})

echo SETUP: Begin Infinispan jdbc-store configuration.

/subsystem=infinispan/cache-container=keycloak/distributed-cache=sessions:remove()

batch

/subsystem=infinispan/cache-container=keycloak/distributed-cache=sessions:add(owners=1)

/subsystem=infinispan/cache-container=keycloak/distributed-cache=sessions/component=expiration:add(lifespan=900000000000000000)

/subsystem=infinispan/cache-container=keycloak/distributed-cache=sessions/component=eviction:add(max-entries=1)

/subsystem=infinispan/cache-container=keycloak/distributed-cache=sessions/store=jdbc:add( \

  datasource="java:jboss/datasources/KeycloakDS", \

  passivation=false, \

  fetch-state=true, \

  preload=false, \

  purge=false, \

  shared=true, \

  max-batch-size=1000 \

)

/subsystem=infinispan/cache-container=keycloak/distributed-cache=sessions/store=jdbc:write-attribute( \

  name=properties.databaseType, \

  value=POSTGRES \

)

/subsystem=infinispan/cache-container=keycloak/distributed-cache=sessions/store=jdbc/table=string:add( \

  data-column={type=bytea}, \

  drop-on-stop=false, \

  fetch-size=5000, \

  prefix=ispn \

)

/subsystem=infinispan/cache-container=keycloak/distributed-cache=sessions/store=jdbc/write=behind:add( \

  modification-queue-size=1024 \

)

run-batch

echo SETUP: Finished Infinispan jdbc-store configuration.

stop-embedded-server

```

With this in place, Keycloak will create a table called: ispn_sessions if missing where the session contents are stored.

When a user signs in, an entry is added to the table. When the user signs out again, then the entry is removed from the table. 

Multiple Keycloak instances can access the session information. If I stop all Keycloak instances and restart the cluster, all users are still correctly logged in.

The usual way to achieve this is to use an external infinispan cluster. However, this introduces a new level of complexity. 

The configuration shown above is IMHO much more straightforward and is probably enough for most use cases.

Is there anything against making the above "skipCacheStore" functionality configurable via a configuration setting on the 

`org.keycloak.models.sessions.infinispan.InfinispanUserSessionProviderFactory` like `allowCacheStore=true`. This can then be passed down 

to `org.keycloak.models.sessions.infinispan.InfinispanUserSessionProvider` to the `InfinispanChangelogBasedTransaction`.

There we could wrap all calls to `CacheDecorators.skipCacheStore(cache)` with a method like `getDecoratedAdvancedCache()` which does:

```

protected AdvancedCache<K, SessionEntityWrapper<V>> getDecoratedAdvancedCache() {

    if (allowCacheStore) {

        return cache.getAdvancedCache();

    }

    return CacheDecorators.skipCacheStore(cache);

}

```

I published a small example for this here: https://github.com/thomasdarimont/keycloak-project-example/tree/main/deployments/local/cluster#run-with-infinispan-cache-content-stored-in-jdbc-store

What do you guys think?

Cheers,

Thomas

[Thomas Darimont]

Note that the configuration:

```

/subsystem=infinispan/cache-container=keycloak/distributed-cache=sessions/component=eviction:add(max-entries=1)

```

is just an example - I wanted to see if it is possible to keep the number of in-memory cache items low with the other sessions still stored in the database.

This works like a charm :)

This effectively allows us to have only a smaller number of sessions in memory, e.g. 100k but keep the bulk-load of sessions e.g. 20mio in the database.

Cheers,

Thomas

[Thomas Darimont]

Additional note:

an entry in the ispn_session table looks like this:

id,datum,version,segment

"da04b231-3097-42d2-82c7-7fbdab87e914","E'\\x9801EA078A019C040AF8039....B62F209EDF8DEAB62F2A00'","1629627721214","203"

The columns are:

id - the session uuid 

datum - the serialized? session data (~540 bytes in a plain keycloak installation, I used bytea type for postgres)

version - the timestamp of the last session change (milliseconds, could be used to detect and clean up dangling sessions if necessary)

segment - the logical cache segment where the data is stored, could be used to partition large tables

Cheers,

Thomas

[Zhandos Zhylkaidar]

Hello,

Any updates on this ? Will we be able to use the write-through (or behind) feature of Infinispan in future versions of Keycloak ?

I want to add some persistent storage to our Keycloak cluster. I am not considering a remote Infinispan cache, because it is still an in-memory storage which is quite expensive to scale.

I wanted to use jdbc-store to implement this, but it seems to be disabled in latest Keycloak (as Thomas mentioned). Therefore, I am opting out to using offline sessions, but offline sessions serve a bit different purpose by design, so I am still hesitant to go for it.

Thanks,

Zhandos.

[Zhandos Zhylkaidar]

I just tested Keycloak 13 and 15 with the file-store enabled.

In both cases sessions were not written to file unless I enable the passivation and eviction. After enabling passivation, sessions are saved to file either during an eviction or on node shut down.

[Sven-Torben Janus]

Hello,

for reference: ignoring the SKIP_CACHE_STORE flag has been discussed before, see https://lists.jboss.org/pipermail/keycloak-dev/2018-August/011089.html

See also:

- https://issues.redhat.com/browse/KEYCLOAK-5656

- https://issues.redhat.com/browse/KEYCLOAK-8814

- https://issues.redhat.com/browse/KEYCLOAK-9877

@Thomas Do you know what the current state for Store.X is? Shouldn't this make the whole infinispan configuration obsolete once it is implemented?

Regards

Sven-Torben

[Thomas Darimont]

Hello Sven-Torben,

I think the SKIP_CACHE_STORE flag should be made configurable OR dynamically inferred, e.g. disabled if a remote store for a particular cache is configured.

The dynamic inference is probably more elegant but I haven't tried it yet.

In my experiments I just used a patched version of the `CacheDecorators` as mentioned earlier, which works quite well: I can limit the number of sessions in memory and off-load all other sessions into the database. User sessions are still functional even after server / cluster restarts. I think this approach could allow current Keycloak versions (even without the new store) to support large numbers of sessions backed by a database without an additional external infinispan store.

I'll try to infer the remote cache configuration and let you guys know if that worked.

For a discussion about the current state of the new store take a look at this keycloak-dev thread: https://groups.google.com/g/keycloak-dev/c/TVV9ctVeIpM/m/q4Ir9fGrBQAJ

Cheers,

Thomas

--
You received this message because you are subscribed to a topic in the Google Groups "Keycloak Dev" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/keycloak-dev/sOBzG76f2FE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to keycloak-dev...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/keycloak-dev/854fa909-9617-4ef6-bff6-33c72f58b8d7n%40googlegroups.com.

[Anton Bessonov]

Hello devs, hello Thomas,

from my point of view, this is a very valuable addition. I've no possibility to test it at scale yet, but in dev environment it worked as expected (thanks Thomas for patched jars, this is great!). Especially because the change is very small (only CacheDecorators) and behind a setting/cli-config, is there any chance to include it?

Best regards,

Anton

[Simon Schürg]

Hello Thomas,

First of all, thanks for your efforts on cache persistence!

Does this procedure also work with Keycloak 17 and the Quarkus distribution?

And are there any plans to officially support Infinispan cache persistence in future Keycloak releases?

My use case is a single node keycloak setup with cache persistence to not lose sessions on server restarts.

Best regards,

Simon

[Olivier Boudet]

Hello,

Thank you Thomas for your efforts on persisting sessions :) We are thinking about using a similar configuration as you described in your first email.

Is it discouraged to use this kind of persistence and -Dkeycloak.infinispan.ignoreSkipCacheStore=true ? It is much simpler as using a remote infinispan cluster and avoid this issue : https://github.com/keycloak/keycloak/discussions/10577

Thanks

[Anand Kulkarni]

Thomas, thanks for the efforts, I am also needing this. We're on 16 and thinking if I we should first upgrade to 19 before attempting this.

Olivier, were you able to achieve persistence with ignoreSkipCacheStore flag? Also, you are on which version of keycloak? 

[Olivier Boudet]

Hello,

Yes we are able to use a modified jar to add ignoreSkipCacheStore on keycloak 17.0.1.

It is working fine in production since several months, we have around 2 millions sessions in database. 

--
You received this message because you are subscribed to a topic in the Google Groups "Keycloak Dev" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/keycloak-dev/sOBzG76f2FE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to keycloak-dev...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/keycloak-dev/d18712aa-1634-4103-b176-9d73370ff8b2n%40googlegroups.com.

[Anand Kulkarni]

Thanks Olivier!

[Björn Eickvonder]

Are there any news on this topic.

I don't feel well patching Keycloak (though it works). Any chance to get this into Keycloak?

[Erick Moreno]

My current production setup with about 4M sessions on Keycloak 20 would also be highly benefited by this feature.

You received this message because you are subscribed to the Google Groups "Keycloak Dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to keycloak-dev...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/keycloak-dev/97e52327-6731-44b2-ad52-5d1acdbefd1an%40googlegroups.com.

--

Erick Moreno

Science is like magic, but real

[Anand Kulkarni]

Below worked for me - we upgraded to Keycloak 19 and used a separate infinispan remote caching cluster that uses JDBC(postgres) as a persistent storage.

It's out in prod for a few months and had no issues so far.

To view this discussion on the web visit https://groups.google.com/d/msgid/keycloak-dev/97e52327-6731-44b2-ad52-5d1acdbefd1an%40googlegroups.com.