Atleast once delivery for admin events

[Baltabayev Artur (IOC/PAU2)]

Hello,

 

during development the need for a 'atleast once delivery' mechanism for admin events came up,

because the current behavior of Keycloak permits the following scenario:

 

    - User creation is triggered via Admin API

    - User is persisted in the database

    - AdminEventBuilder fails when trying to call onEvent()

    - Admin Event doesn't exist although the user is persisted

 

The context is that there are third party applications, which synchronize with the state users in Keycloak

through admin events. The actual event delivery is done through a custom SPI in our case.

 

Our suggestion would be to include a retry mechanism, that could guarantee the event delivery.

Of course users should also be able to turn it on / off in the settings.

 

Before suggesting implementation details, we would like to know if there is any interest in this contribution ?

 

Mit freundlichen Grüßen / Best regards

Artur Baltabayev

Bosch IoT Permissions - Product Area User Management (IOC/PAU-PM)
Bosch.IO GmbH | Ziegelei 7 | 88090 Immenstaad | GERMANY | www.bosch.io
Tel. +49 7545 202-497 | Fax +49 7545 202-301 | Artur.Ba...@bosch.io

Registered Office: Berlin, Registration Court: Amtsgericht Charlottenburg; HRB 148411 B
Chairman of the Supervisory Board: Dr.-Ing. Thorsten Lücke; Managing Directors: Dr. Stefan Ferber, Dr. Aleksandar Mitrovic, Yvonne Reckling

​

[Alec Henninger]

Strictly speaking, the only way to guarantee at least once delivery would be by persisting an event to be dequeued as part of the same transaction as the related state change. Practically speaking, this means within the same database. Sometimes this is called the outbox pattern: https://microservices.io/patterns/data/transactional-outbox.html

Another way this is commonly done is to take advantage of the internal database log which is how debezium works I believe: https://debezium.io/

The downside there is listening to changes is tightly coupled to the database schema.

A retry mechanism as you've described might help increase reliability but it is not a guarantee as the server can still crash before it successfully delivered the message, or may have to give up retrying eventually to avoiding queueing up too many operations in memory. Still, this can be useful if you don't strictly need guaranteed delivery in absolutely every case.

Hope that helps think about the options. Cheers,

Alec

--
You received this message because you are subscribed to the Google Groups "Keycloak Dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to keycloak-dev...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/keycloak-dev/60dc15f0beac4d5dba7dee7c27042e37%40bosch.io.

[Thomas Darimont]

Hello,

+1 to what Alec said. I think debezium might be a good way to solve this.

You might take a look at this:

http://juliuskrah.com/blog/2020/01/06/streaming-changes-from-keycloak-using-debezium-cdc/

Cheers,

Thomas

To view this discussion on the web visit https://groups.google.com/d/msgid/keycloak-dev/CAO1UCizZnt4XVfbAJpmR%2BULF%3DiCuQMpa1_JFQ7VWgH3ChqhNSA%40mail.gmail.com.

[Stian Thorgersen]

The built-in event store does persist the events in the same tx as the updates, so that's guaranteed. A custom event listener would depend on where you are sending the events, and it can also enlist a tx for guarantees. Right now though how to handle this is something you'd choose in your custom event listener.

Debezium is a fairly brittle approach as there's no guarantees the KC database schema doesn't change, and there are also events that don't update the DB like LDAP updates, and in the new store we're also adding more support for additional stores for things.

To view this discussion on the web visit https://groups.google.com/d/msgid/keycloak-dev/CAK-7U1h7Yo-nJV8_YJ7w0sXOVZvya199bRibaX1G1skn3LgM3A%40mail.gmail.com.

[Baltabayev Artur (IOC/PAU2)]

Hello,

Thanks for all the answers. We found out, that the current logic does not trigger a transaction rollback if the event store (for any reason) is not able to persist

the event. This is due to a checked exception in the AdminEventBuilder class. So technically the events are not guaranteed.

 

Mit freundlichen Grüßen / Best regards

Artur Baltabayev

Bosch IoT Permissions - Product Area User Management (IOC/PAU-PM)
Bosch.IO GmbH | Ziegelei 7 | 88090 Immenstaad | GERMANY | www.bosch.io
Tel. +49 7545 202-497 | Fax +49 7545 202-301 | Artur.Ba...@bosch.io

Registered Office: Berlin, Registration Court: Amtsgericht Charlottenburg; HRB 148411 B
Chairman of the Supervisory Board: Dr.-Ing. Thorsten Lücke; Managing Directors: Dr. Stefan Ferber, Dr. Aleksandar Mitrovic, Yvonne Reckling

​