Offline Session Lazy Loading

[田畑義之 / TABATA，YOSHIYUKI]

Hello,

 

Relating to KEYCLOAK-11019 "Offline sessions preloading still slow in some cases"[1], we'd like to start a new discussion in the keycloak-community about "Offline Session Lazy Loading"[2].

 

We can shorten the startup time by tuning some parameters or adding indexes, but there is a limit we can shorten.

In the PR, we'd like to consider resolving the root cause, that is, consider providing offline session loading methods other than the existing method preloading all offline sessions in the startup.

Keycloak source code seems to be rooted in the offline session preloading, so it's not easy to change this, but kind of lazy loading is requested by many users, so even in the long run, we'd like to start the discussion.

 

What do you think about this? I would be very happy if you give any kind of comment on that.

 

[1] https://issues.redhat.com/browse/KEYCLOAK-11019

[2] https://github.com/keycloak/keycloak-community/pull/190

 

Regards,

Yoshiyuki Tabata

Hitachi, Ltd.

 

[Schuster Sebastian (IOC/PDL22)]

Hi everybody,

 

we are VERY interested in this is really causing us a lot of headache currently. We are definitely willing to contribute here!

 

Best regards,
Sebastian

 

Mit freundlichen Grüßen / Best regards

Dr.-Ing. Sebastian Schuster

Project Delivery Berlin 22 (IOC/PDL22)
Bosch.IO GmbH | Ullsteinstr. 128 | 12109 Berlin | GERMANY | www.bosch.io
Tel. +49 30 726112-485 | Mobil +49 152 02177668 | Telefax +49 30 726112-100 | Threema / Threema Work: MF9VMEAE | Sebastian...@bosch.io

Sitz: Berlin, Registergericht: Amtsgericht Charlottenburg; HRB 148411 B
Aufsichtsratsvorsitzender: Dr.-Ing. Thorsten Lücke; Geschäftsführung: Dr. Stefan Ferber, Dr. Aleksandar Mitrovic, Yvonne Reckling

--
You received this message because you are subscribed to the Google Groups "Keycloak Dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to keycloak-dev...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/keycloak-dev/TY2PR01MB2284EB222C5D49B89B4E31CEB9E00%40TY2PR01MB2284.jpnprd01.prod.outlook.com.

[Stian Thorgersen]

I'd say offline sessions should really rather "live" in the database, rather than in the cache, where the cache should act as an invalidation cache only. In that regards I can think of at least a handful of issues that we'd need to overcome:

* Session idle is expensive in its current state as it updates the last used time for a session whenever it is touched. This may require a re-think on how session idle works. 

* Listing sessions - in admin console and account console we need the ability to list sessions. When all sessions live one place this is easier, but perhaps we simple should fetch regular sessions from Infinispan, then query for offline sessions from the database

Another thing is that I don't think it's only offline sessions that should be persisted and durable. Sessions are usually long lived entities. People don't want to regularly re-authenticate on their workstation, and especially not on mobile. Does that mean actually most sessions should be persisted to the database? This relates to step-up/re-auth as it may be generally better to think of sessions as long-lived entities, that are refreshed by re-authentication, rather than new sessions being created.

Session idle is really trying to achieve two things:

* Garbage collection of unused sessions - this doesn't require a regular update on last session use as we do today, but rather can be much less aggressive. Say a session is garbage collected if not used for 7 days, then updating the session "keep-alive" every day is perhaps  sufficient

* Logout automatically on idle - there's two parts to this. One is solved by client sessions, where a client session has its own independent max/idle capabilities, so sensitive applications have the ability to disconnect users when not active. Second part is solved better by requiring re-authentication after a period of time for a session

--

[Till Markus (IOC/PDL22)]

Would it be an starting point to use the “current” semantics of offline sessions for all sessions and rename that thing to durable sessions where offline sessions just is a durable session which is lazy loaded always?

 

On the negative side this will effect startup time for standard sessions as it would get all problems of offline sessions.

 

While writing I notice this is a no go, so may it should be considered to have some internal state to actively recache “active” sessions e.g.

last used time > something.

Or it could be sufficient to just pick the top (x1000) sessions which are no offline sessions on startup ordered by last update time.

 

This also implies that any kind of session can be lazy loaded.

 

On the plus side it could be easy to just persist all sessions in a new entity and also “slow” migrate old offline sessions (on lazy load/ or creation).

Also this would merge both approaches in a configurable way. The administrator could optimize for startup time or for pre warmed caches.

 

Btw if lazy loading is an option but an issue for some setups it could also be considered to run a background worker on the master node which consistently adds all sessions to the cache in a throttled operation mode. This way the startup can be quick + reasonable amount of sessions loaded on startup and all other sessions added on the fly. (the last part for sure needs a lot of tinkering to resolve the conflicts between users which show up and fill the cache lazy and the background loader)

 

 

 

Mit freundlichen Grüßen / Best regards

Markus Till

Project Delivery Berlin 22 (IOC/PDL22)

Bosch.IO GmbH | Ziegelei 7 | 88090 Immenstaad | GERMANY | www.bosch.io
Tel. +49 30 726112-354 | Mobil +49 172 5782078 | Telefax +49 30 726112-100 | Threema / Threema Work: FHKUKVDR | Marku...@bosch.io

Sitz: Berlin, Registergericht: Amtsgericht Charlottenburg; HRB 148411 B
Aufsichtsratsvorsitzender: Dr.-Ing. Thorsten Lücke; Geschäftsführung: Dr. Stefan Ferber, Dr. Aleksandar Mitrovic, Yvonne Reckling

​

To view this discussion on the web visit https://groups.google.com/d/msgid/keycloak-dev/CAJgngAf8Aio0uwfKSBeAMDZX0BzMrE3VRBY9SKyjdoAANoYRZg%40mail.gmail.com.

[Pedro Igor Craveiro e Silva]

Glad to see this coming. Indeed, it is a critical issue for large-scale deployments.

I've started with an initial comment on your proposal.

--

[Pedro Igor Craveiro e Silva]

I agree with your points.

However, I think the pre-load issue is something we could fix without necessarily a re-design of sessions (at least for now). This is a quite common issue (quite critical?) that maybe we could quickly fix while still discussing how we should store sessions.

To view this discussion on the web visit https://groups.google.com/d/msgid/keycloak-dev/CAJgngAf8Aio0uwfKSBeAMDZX0BzMrE3VRBY9SKyjdoAANoYRZg%40mail.gmail.com.

[Stian Thorgersen]

If a session is persisted/durable, why would you want to preload it at all? I'd rather argue that it should just always be loaded on demand. There is probably the need to configure if sessions are durable (persisted to DB), or non-durable (only in Infinispan).

[Thomas Darimont]

Having support for lazy-loading of offline sessions would IMHO also solve another problem: "perceived" offline session loss.

In many projects that use a lot of offline sessions (> x*100k) I observe the problem that Keycloak seems to "lose" offline sessions from time to time.
This becomes noticeable by the fact that for some offline-token no token refresh requests are possible anymore, because Keycloak can't find them in-memory. 

However, the corresponding offline session entries are still present and marked as valid sessions in the database.

This problem is often observed in the context of Kubernet based deployments where rolling upgrade is used and Keycloak is used in standalone HA mode without an external Infinispan store.

So far we have not found a 100% reliable way to prevent the loss of offline session information with Keycloak in standalone HA clustered mode in Kubernetes.
Maybe the solution is to store the sessions in an external Infinispan cluster that is then accessed by the Keycloaks in standalone-HA mode, but this is barely documented [1] and increases complexity by a lot.

One way to detect offline session loss is to compare the result of `/auth/admin/realms/${realmId}/clients/${clientUuid}/offline-session-count`, which is computed from cache and the number 

of actual offline_user_sessions and offline_client_sessions entries in the database.
If these values differ (by large) you have an indicator for session loss, which means that from Keycloak's point of view some offline sessions are no longer there and some users (mobile apps) need to login again.

This problem becomes worse the more users you have that use clients with offline-access.

Cheers,

Thomas

[1] https://www.keycloak.org/docs/latest/server_installation/index.html#crossdc-mode

To view this discussion on the web visit https://groups.google.com/d/msgid/keycloak-dev/CAJgngAeCRoNUC2GsEy0qiWG%3Dbqixn7Lm1R3xdV1n1URD-9d7ZA%40mail.gmail.com.

[Thomas Darimont]

Forgot to mention: If you run just a single-node Keycloak standalone-ha "cluster" in Kubernetes, we don't observe the loss of offline-sessions.

[田畑義之 / TABATA，YOSHIYUKI]

Hi,

 

Thank you for your comments.

 

In the long run, Stian's idea is the best idea to eliminate the slow offline session loading problem, I think.

However, many users "now" pay attention to this problem, so in the keycloak-community, first of all, we'd like to consider Pedro's idea which is without a re-design of sessions.

[Christian Becker]

Hello,

I’m a bit confused, that this topic still needs discussion, as based on our information (from January), a redesign of the storage and cache architecture was well underway. Unfortunately RedHat support didn’t provide a reference ticket for this, but our assumption was, that this will likely be included in the next major release.

To the actual proposal: We would love to have this implemented as soon as possible. Especially when the database is PostgreSQL, the previous optimisations didn’t improve the loading time much.

Afaik we also suggested this ~2 years ago, but there were some issues with Infinispan, that made it impossible to implement a “load on cache miss”. Is this still the case?

The lazy loading would be a very good improvement, that likely requires a minimum of implementation effort and also won’t complicate operations. With an optional background loading task, it would also allow for a flag, that disables this “read through” behaviour, once all sessions are back in Infinispan. This prevents load spikes on the database, should someone maliciously request massive amounts of invalid/expired sessions.

Apart from that, one could also imagine a setup, where Infinispan is not big enough to keep all sessions in memory, but only a subset, in order to keep the setup small and not waste massive amounts of memory.

All of this should only be needed for offline sessions, as they are likely required over months and are mostly used by people, so lost sessions immediately cause an inconvenience. Non-offline sessions can just remain in Infinispan and do not need to be loaded from the database. In most cases it doesn’t matter so much, when they are lost.

The session counts, that Pedro mentions in the GitHub comment are likely negligible as he mentioned, but i also see another issue with these counts: In our case, they cause quite some load spikes on Infinispan, as the sessions are literally being counted through a iterator when the session count is requested. But improving on this is another discussion.

Best Regards,
Christian

> To view this discussion on the web visit https://groups.google.com/d/msgid/keycloak-dev/TY2PR01MB22842C3766A3D35982BED431B9F90%40TY2PR01MB2284.jpnprd01.prod.outlook.com.

[Thomas Darimont]

Dear Keycloak developers,

This week I was bitten by the problem of missing offline user-sessions in infinispan cache after rolling upgrades once again and gave it another spin.

I found a simple (and IMHO robust) way to recover an offline user-session from the database if the respective session could not be found in the infinispan cache.

# Dynamic offline user-session importing

The trick is to enhance the user-session lookup logic in "InfinispanUserSessionProvider#getUserSession", which is used by the Keycloak infrastructure to find user-sessions in the cache.

The idea is to perform an offline user-session lookup with a "PersistenceProvider", if the user-session was not found during cache lookup, and re-importing the session into the infinispan cache if found.

# Branch & Code

Branch: https://github.com/thomasdarimont/keycloak/tree/issue/KEYCLOAK-11019-lazy-offline-session-loading
Changes to Keycloak 11.0.3: https://github.com/keycloak/keycloak/compare/11.0.3...thomasdarimont:issue/KEYCLOAK-11019-lazy-offline-session-loading

## How it works

The PersistenceProvider lookup is performed by a new method InfinispanUserSessionProvider#getUserSessionEntityFromPersistenceProvider which is called by InfinispanUserSessionProvider#getUserSession.

In "InfinispanUserSessionProvider#getUserSessionEntityFromPersistenceProvider" we try to lookup an offline user-session from the 

configured "UserSessionPersisterProvider" via the given session id.

If we find an offline-session that is still valid, we use "org.keycloak.models.UserSessionProvider#importUserSessions" to dynamically re-import

the offline user-session into infinispan. After that, we perform a second infinispan cache lookup to ensure that we have the proper wrapping around our "UserSessionEntity".

With this in place, we can successfully (and transparently) recover offline user-sessions not present in infinispan, even after rolling cluster upgrades! :)

For this to work, I needed to add a way to load a single user-session by it's session id via the "UserSessionPersisterProvider", thus I added a new method "loadUserSession(sessionId,offline)" 

to "org.keycloak.models.session.UserSessionPersisterProvider#loadUserSession" with an appropriate implementation in "org.keycloak.models.jpa.session.JpaUserSessionPersisterProvider#loadUserSession".

# Further work: Lazy-loading for offline user-sessions

The above can serve as the foundation for lazy-loading of offline-session in "org.keycloak.models.sessions.infinispan.InfinispanUserSessionProviderFactory#loadPersistentSessions".

We could make the loading of offline user-sessions optional or load only a subset of the sessions into memory. Additionally, we could also use a bounded cache with a "short" TTL to reduce memory consumption over time.

This would significantly speed up startup times, especially in scenarios where many offline-sessions (> 100k, >1 million) are used.

The current implementation would fix the dreaded "offline user-session not found" problem that plagues many large Keycloak installations.

What do you think about this approach? I'd love to hear your input on the points above. If you think that I'm on the right track, then I'd love to get some guidance for adapting the code 

in "InfinispanUserSessionProviderFactory#loadPersistentSessions" to support lazy-loading.

Some things to consider:

- The implementation performs database lookups for all infinispan cache lookup misses for offline user-sessions; this will increase the database-load. However, this should IMHO not be an issue with proper indices.

   Attackers could try to perform a DoS attack by bombarding Keycloak with requests with offline user-sessions that do not exist anymore.
   To mitigate this, we could store those failed session ids in a bounded "invalid-offline-user-sessions-ids" cache with a short TTL to reject some of those requests.

- The logic might import an offline user-session twice into the infinispan cluster if the data could not be replicated quickly enough among the cluster nodes. 

  I don't know if this is an issue as both user-sessions contain the same data, but this needs more thought.

See the relevant code for the proposed user-session lookup below:

In  org.keycloak.models.sessions.infinispan.InfinispanUserSessionProvider:

```

...

    protected UserSessionAdapter getUserSession(RealmModel realm, String id, boolean offline) {

        UserSessionEntity userSessionEntityFromCache = getUserSessionEntity(realm, id, offline);
        if (userSessionEntityFromCache != null) {
            return wrap(realm, userSessionEntityFromCache, offline);
        }

        if (!offline) {
            return null;
        }

        // Try to recover from potentially lost offline-sessions by attempting to fetch and re-import
        // the offline session information from the PersistenceProvider.
        UserSessionEntity userSessionEntityFromPersistenceProvider = getUserSessionEntityFromPersistenceProvider(realm, id, offline);
        if (userSessionEntityFromPersistenceProvider != null) {
            // we successfully recovered the offline session!
            return wrap(realm, userSessionEntityFromPersistenceProvider, offline);
        }

        // no luck, the session is really not there anymore
        return null;
    }

...

    private UserSessionEntity getUserSessionEntityFromPersistenceProvider(RealmModel realm, String id, boolean offline) {

        log.debugf("Offline user-session not found in infinispan, attempting UserSessionPersisterProvider lookup for sessionId=%s", id);
        UserSessionPersisterProvider persister = session.getProvider(UserSessionPersisterProvider.class);
        UserSessionModel persistentUserSession = persister.loadUserSession(id, offline);

        if (persistentUserSession == null) {
            log.debugf("Offline user-session not found in UserSessionPersisterProvider for sessionId=%s", id);
            return null;
        }

        log.debugf("Offline user-session found in UserSessionPersisterProvider, attempting to reimport user-session for sessionId=%s", id);
        session.sessions().importUserSessions(Collections.singleton(persistentUserSession), offline);
        log.debugf("Offline user-session imported, trying another lookup for sessionId=%s", id);

        UserSessionEntity ispnUserSessionEntity = getUserSessionEntity(realm, id, offline);

        if (ispnUserSessionEntity != null) {
            log.debugf("Offline user-session found after import for sessionId=%s", id);
            return ispnUserSessionEntity;
        }

        log.debugf("Offline user-session could not be found after import for sessionId=%s", id);
        return null;
    }

```

Cheers,

Thomas

[Pedro Igor]

Hi Thomas,

That is an interesting work and also related to discussions we had with Marek about how to avoid loading sessions during startup.

The approach you took is pretty much what we had in mind, but in addition to the drawbacks you pointed out, there is also the impact on the admin console where we query the cache for all sessions for a given user/client.

I would say that we should definitely take this approach and do not do any blocking code during startup.

Another approach though is to load sessions using a background task, maybe as a complement to this approach. It might help to reduce the chance of a cache miss and also help with database trips.

As for having duplicate entries in the cache, it should be possible to solve that with some ISPN trick ?

As a last note, I think Stian is planning a complete review of how we manage sessions based on a more stateless model.

[Thomas Darimont]

Hello Pedro

Thanks for looking into this and your valuable feedback!

I'd rather avoid loading large amounts of data into the cache upfront, be it synchronous or asynchronous. I've seen many large Keycloak installations with several million offline sessions that needed large heaps (>16-32 GB and larger) to cope with the number of offline-sessions that were quite often barely used. Of course, the usage pattern depends on how the application uses tokens, and some scenarios might benefit from pre-loading a set of offline-sessions.

If this were configurable along the lines of e.g.:

- offline-session-pre-loading-enabled: true/false (current default: true)

- offline-session-pre-loading-max: -1/0/ n>0 (current default: -1 )

-1: everything

0: none

n>0: load at most n offline-sessions into memory... (perhaps this limit could also be controlled with the infinispan user-offline-session, client-offline-session cache size).

One could also think of a hybrid model, where you only load the offline-session uuid's into memory without any other session data, to check if the session is present at all and then fetch the actual session later on. This would still require some pre-loading at startup but would solve the problem of the DoS attack that I mentioned earlier while requiring less memory than loading everything into the heap.

The proposed solution seems like the right candidate that would work reliably and provide a good "enough" performance. But this needs to be tested a bit to get a reliable performance baseline.

Regarding the feature "we query the cache for all sessions for a given user/client in the admin-console":
What is the use-case here?
-> To see how many sessions are active / exist?
Here we can do a simple (and quick) database count query to get the same information.
-> To list all active sessions?
Does someone needs to see all 10 million sessions, or does an admin wants to see whether a concrete user has a session?
This list can be paginated and fetched from the database on an as needed basis.

Regarding the duplicate cache entries, I'm currently unsure whether this would be a problem at all in practice, but I'm not that familiar with infinispan. Perhaps someone from Redhat could elaborate more here, e.g., by providing the necessary ISPN tricks ;-).
In my (local) tests, I didn't see a problem with this, but I have to admit that I only tried this with a few million offline sessions and multiple docker containers on the same host. However, things might look different in a real distributed environment with higher latencies (remote keycloaks, remote database etc.).

I know about the redesign of the storage layer for "Keycloak-X" - I support this effort, and I'm looking forward to it. However, IMHO the current Keycloak community would benefit most from a solution that also works with the traditional Keycloak codebase since many companies already use the Wildfly based Keycloak distribution.

I'd be happy to send a PR for this to help to solve the dreaded "missing offline-sessions" once and for all. :)

Cheers,
Thomas

To view this discussion on the web visit https://groups.google.com/d/msgid/keycloak-dev/4f2fc0af-2c74-480c-8deb-abe9d352c855n%40googlegroups.com.

[Pedro Igor Craveiro e Silva]

On Wed, Jan 6, 2021 at 11:50 AM Thomas Darimont <thomas....@googlemail.com> wrote:

Hello Pedro

Thanks for looking into this and your valuable feedback!

I'd rather avoid loading large amounts of data into the cache upfront, be it synchronous or asynchronous. I've seen many large Keycloak installations with several million offline sessions that needed large heaps (>16-32 GB and larger) to cope with the number of offline-sessions that were quite often barely used. Of course, the usage pattern depends on how the application uses tokens, and some scenarios might benefit from pre-loading a set of offline-sessions.

If this were configurable along the lines of e.g.:

- offline-session-pre-loading-enabled: true/false (current default: true)

- offline-session-pre-loading-max: -1/0/ n>0 (current default: -1 )

-1: everything

0: none

n>0: load at most n offline-sessions into memory... (perhaps this limit could also be controlled with the infinispan user-offline-session, client-offline-session cache size).

You are still running blocking code during startup, and IMO we should avoid that. Hence my proposal to load async. Even if that means loading -1, n >0, or 0.

Of course, we are relieving startup, but still pushing too many to ISPN as the offlinesessions cache is distributable.

I agree that a configuration option would allow people to configure the behavior accordingly to their needs.

But in any case, your changes should consider the existing functionalities that rely on querying offlinesessions without necessarily having the ID. Take a look at the client settings and offline access tabs. My point is that we should also consider this when discussing the changes you are proposing.

Also note that when using Keycloak with JDG, the scenario changes a bit. You don't have sessions fetched from DB but from JDG. You also should expect better performance when "pre-loading" using JDG. I have seen customers using Ms of offline sessions and with JDG things got a bit better. Not only in terms of restarts but also in terms of availability as sessions are stored in JDG. Maybe this is another thing your changes should consider, I'm not sure if you would need to query the remote cache to fetch sessions on demand.

At the same time, JDG is not a silver bullet and it might happen you suffer from similar problems (mainly time to boot) as when using a DB.

I would say that another improvement in regards to offline sessions is that today we always try to parallelize work across nodes as well as use the number of available cpus to define the parallelism. IMO, we should make this more configurable as the proper setting really depends on the environment.

 

One could also think of a hybrid model, where you only load the offline-session uuid's into memory without any other session data, to check if the session is present at all and then fetch the actual session later on. This would still require some pre-loading at startup but would solve the problem of the DoS attack that I mentioned earlier while requiring less memory than loading everything into the heap.

Please, let's avoid loading things at startup ... Or at least run code that might block the server. We should either go to lazy load or a hybrid approach async + lazy-load. 

 

The proposed solution seems like the right candidate that would work reliably and provide a good "enough" performance. But this needs to be tested a bit to get a reliable performance baseline.

Regarding the feature "we query the cache for all sessions for a given user/client in the admin-console":
What is the use-case here?
-> To see how many sessions are active / exist?
Here we can do a simple (and quick) database count query to get the same information.
-> To list all active sessions?
Does someone needs to see all 10 million sessions, or does an admin wants to see whether a concrete user has a session?
This list can be paginated and fetched from the database on an as needed basis.

Regarding the duplicate cache entries, I'm currently unsure whether this would be a problem at all in practice, but I'm not that familiar with infinispan. Perhaps someone from Redhat could elaborate more here, e.g., by providing the necessary ISPN tricks ;-).
In my (local) tests, I didn't see a problem with this, but I have to admit that I only tried this with a few million offline sessions and multiple docker containers on the same host. However, things might look different in a real distributed environment with higher latencies (remote keycloaks, remote database etc.).

I also don't see a problem. I'm not an expert in ISPN either, but usually, it provides great tools to solve this kind of problem. We would need to investigate. Maybe Marek is already using something to solve a similar problem ...

 

I know about the redesign of the storage layer for "Keycloak-X" - I support this effort, and I'm looking forward to it. However, IMHO the current Keycloak community would benefit most from a solution that also works with the traditional Keycloak codebase since many companies already use the Wildfly based Keycloak distribution.

Sure, and the dist will be there for a long time. I also want to see this move forward as I've seen not only people from the community but customers complaining about it.

 

I'd be happy to send a PR for this to help to solve the dreaded "missing offline-sessions" once and for all. :)

IMO, if we just add the configuration option to disable/enable your initial changes, I'm good. The default would be to load everything. People can disable it if it does not make sense.

If you want to take into consideration the counting/listing of offline sessions for clients, that would be good. But I'm not 100% if it is necessary but expected if you disable pre-loading.

But again, your changes are still not complete and more things (such as cross-dc, admin console) need to be considered. I hope Marek can chime in to also give his thoughts, I'm also interested in continuing this discussion.

[Pascal Knüppel]

I am currently reading and exploring the client adapters and I got a
simple problem to which I cannot find a solution...

I got a frontend application and several backend applications
- frontent
- backend-1
- backend-2

my frontend now authenticates a user and should access backend-1 on
behalf of this user on certain requests. In some other cases that are
optional I want to access backend-2 on behalf of the user.

I found a pretty good detailed description in how to set this up in the
keycloak administration guide and it all works. I use custom
client-scopes to make it work BUT
how do I tell the client-adapter to access different accessTokens for
backend-1 and backend-2?
All I can do is securing my application (currently running in JBoss EAP)
and require a login for different endpoints. But once the user is
authenticated I do not see any chance to get another access token.
And this is not all. The user is accessing my application by a browser,
so when the user authenticates how am I able to tell the client-adapter
that it should send the additional scope for the audience to backend-1?

Best regards
Pascal Knüppel

[Thomas Darimont]

Hi Pedro,

Thanks again for your input!

I agree that we should not do any loading-during startup. As a start, I added support to disable the pre-loading of offline sessions from the database at startup. 

As you suggested, pre-loading is enabled by default to keep current Keycloak behavior. At a later stage we could also disable the pre-loading from a remote-cache storage.

I also adapted the fetching of offline session counts by using a database query instead of asking the infinispan cache, if pre-loading offline sessions from the database is disabled.

I revised my initial PoC and sent a PR with the minimal changes: https://github.com/keycloak/keycloak/pull/7722

Cheers,

Thomas