Faster offline session cache initialisation on startup

[Peter Flintholm]

Hi,

I have modified the JpaUserSessionPersisterProvider.loadUserSessions for significantly better performance when loading a large number of offline sessions on startup.

Using the modified startup with 1.5mil offline sessions is approx 3 minutes.

I know there has been a lot discussion regarding this issue.

Is there any interest in my modification ?

If so, I will try to make a PR.

-Peter

[Till Markus (IOC/PAU1)]

Hello Peter,

We would really like to see what this is about.

 

Mit freundlichen Grüßen / Best regards

Markus

--
You received this message because you are subscribed to the Google Groups "Keycloak Dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to keycloak-dev...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/keycloak-dev/ba363196-2968-4be1-9dca-9f14684d32b7n%40googlegroups.com.

[Stian Thorgersen]

+1 Could you provide more details?

We do have another related PR in the works, which adds support to lazy load offline sessions on demand:

https://github.com/keycloak/keycloak/pull/7722

Question is are the improvements you mentioned still relevant if we are loading on demand instead of eagerly at startup?

To view this discussion on the web visit https://groups.google.com/d/msgid/keycloak-dev/cbe778ba7d944802b5b082cb34450312%40bosch.io.

[Peter Flintholm]

Hi

Sorry for the long delay.

I have just made a PR with my proposed changes - https://github.com/keycloak/keycloak/pull/7902

I preparation of this PR, I did a lot of performance testing and found that the by far most important factor in offline session loading is adjusting the segment size. Also, for very large number of offline sessions to increase timeouts of course.

For anyone trying to optimise their setup, it might be helpful to look at these performance figures:

500.000 offline sessions, segment size 64 (default) : current 1850 sec, new 62 sec.

500.000 offline sessions, segment size 256 : current 510 sec, new 46 sec.

500.000 offline sessions, segment size 4096: current 69 sec, new 35 sec.

500.000 offline sessions, segment size 10240: current 52 sec, new 30 sec.

500.000 offline sessions, segment size 20480: current 47 sec, new 30 sec.

2.500.000 offline sessions, segment size 10240: current 383 sec, new 205 sec.

5.000.000 offline sessions, segment size 10240: current 1065 sec, new 426 sec.

(the size 500.000 test was done on a Mac - the larger were run on a Azure VM and hosted db. Timing is only for the session cache initialization - not the entire server startup)

Note, the tests were all done on a single Keycloak instance and Postgres db. I think the result might be very different on other databases - also the segment size might have different impact)

Also, the offline session were done by inserting row directly in the tables, but I don't believe it makes any difference. The results are pretty consistent with what i have seen with real production data.

Memory usage is stil exorbitant - approx 25GB ram for 5.000.000 sessions. A colleague is working on an alternative implementation of the session provider, that keeps the offline sessions in database. This is IMO the appropriate long term solution.

-Peter

---

[Thomas Darimont]

Hi Peter,

great analysis! You might want to point your colleague to this PR: https://github.com/keycloak/keycloak/pull/7722

Cheers Thomas

To view this discussion on the web visit https://groups.google.com/d/msgid/keycloak-dev/673b8302-d3cb-4bb8-8297-adbe81bb1353n%40googlegroups.com.

[Peter Flintholm]

I just had time to look at the https://github.com/keycloak/keycloak/pull/7722.

 

This is very promising, and my suggested PR would definitely only make sense if session are loaded eagerly at startup.

 

One of my concerns is also the memory usage. PR7722 could help on that, but only assuming frequent server restarts.

I think PR 7722 with a cache size limit feature would be the ideal solution. Any thoughts on that ?

 

-Peter

To view this discussion on the web visit https://groups.google.com/d/msgid/keycloak-dev/673b8302-d3cb-4bb8-8297-adbe81bb1353n%40googlegroups.com.

[Schuster Sebastian (IOC/PAU1)]

Hi Peter,

 

you could configure expiration on the Infinispan cache directly to have entries automatically be removed after a certain time.

 

Best regards,

Sebastian

 

Mit freundlichen Grüßen / Best regards

Dr.-Ing. Sebastian Schuster

Project Delivery Berlin 22 (IOC/PDL22)
Bosch.IO GmbH | Ullsteinstr. 128 | 12109 Berlin | GERMANY | www.bosch.io
Tel. +49 30 726112-485 | Mobil +49 152 02177668 | Telefax +49 30 726112-100 | Threema / Threema Work: MF9VMEAE | Sebastian...@bosch.io

Sitz: Berlin, Registergericht: Amtsgericht Charlottenburg; HRB 148411 B
Aufsichtsratsvorsitzender: Dr.-Ing. Thorsten Lücke; Geschäftsführung: Dr. Stefan Ferber, Dr. Aleksandar Mitrovic, Yvonne Reckling

To view this discussion on the web visit https://groups.google.com/d/msgid/keycloak-dev/B75E7E50-DEF8-463A-9570-B15D654E373B%40flintholm.net.