Transaction documentation?
350 views
Skip to first unread message
Garth
Mar 30, 2022, 6:00:25 AM3/30/22
to keyclo...@googlegroups.com
Hi All,
I'm looking to get a better understanding of how transactions are performed in Keycloak. I've dug into the code, and done some tracing in my IDE, and I have a basic understanding of how it works. However, I'd like to see if there is any documentation about how it is built and some of the design decisions that were made. Does anyone know of such a document?
Thanks!
Garth
I'm looking to get a better understanding of how transactions are performed in Keycloak. I've dug into the code, and done some tracing in my IDE, and I have a basic understanding of how it works. However, I'd like to see if there is any documentation about how it is built and some of the design decisions that were made. Does anyone know of such a document?
Thanks!
Garth
Garth
Mar 31, 2022, 4:11:05 AM3/31/22
to keyclo...@googlegroups.com
One concrete question: Where are the JPA/Hibernate transactions set up and run? I assumed they would be wrapped in a `JpaKeycloakTransaction`, but that doesn't seem to be the case. I wanted to experiment with some conditional transaction retry (based on SQLException state), but I couldn't find a consistent place to do it. Ideas?
Alexander Schwartz
Apr 1, 2022, 5:47:22 AM4/1/22
to Garth, keyclo...@googlegroups.com
To understand what is currently implemented, I recommend looking at the DefaultKeycloakTransactionManager where its begin() and commit() methods are being called; usually from a request filter. During a transaction multiple transactions enlist to commit their work at the end of a request.
Best,
Alexander
--
You received this message because you are subscribed to the Google Groups "Keycloak Dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to keycloak-dev...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/keycloak-dev/4ce810fb-d83f-412b-9162-0810b6d2fd98%40www.fastmail.com.
Garth
Apr 1, 2022, 6:13:07 AM4/1/22
to Alexander Schwartz, Till Markus (IOC/PAU1)
Thank you Alexander. This is very helpful. I have been looking at the DefaultKeycloakTransactionManager, and I understand now how it uses the transaction manager and runs multiple KeycloakTransactions.
Specifically on my JPA/Hibernate question, I can't figure out where the those transactions are set up. I'd like to experiment with a transaction retry, but I can't find where those queries are getting added to the transaction manager. I can catch the retryable exception upstream and hack it, but it doesn't seem to be the right way to do it.
A stack trace of the exception (attached) doesn't give any useful information, as it just shows the transaction manager and the JDBC driver calls in the stack.
Any ideas?
Thanks again,
Garth
On Fri, Apr 1, 2022, at 11:47 AM, Alexander Schwartz wrote:
> To understand what is currently implemented, I recommend looking at the
> DefaultKeycloakTransactionManager where its begin() and commit()
> methods are being called; usually from a request filter. During a
> transaction multiple transactions enlist to commit their work at the
> end of a request.
>
> Best,
> Alexander
>
> On Thu, Mar 31, 2022 at 10:11 AM Garth <ga...@tunnel19.com> wrote:
>> One concrete question: Where are the JPA/Hibernate transactions set up and run? I assumed they would be wrapped in a `JpaKeycloakTransaction`, but that doesn't seem to be the case. I wanted to experiment with some conditional transaction retry (based on SQLException state), but I couldn't find a consistent place to do it. Ideas?
>>
>> On Wed, Mar 30, 2022, at 12:00 PM, Garth wrote:
>> > Hi All,
>> >
>> > I'm looking to get a better understanding of how transactions are
>> > performed in Keycloak. I've dug into the code, and done some tracing in
>> > my IDE, and I have a basic understanding of how it works. However, I'd
>> > like to see if there is any documentation about how it is built and
>> > some of the design decisions that were made. Does anyone know of such a
>> > document?
>> >
>> > Thanks!
>> > Garth
>>
>> --
>> You received this message because you are subscribed to the Google Groups "Keycloak Dev" group.
>> To unsubscribe from this group and stop receiving emails from it, send an email to keycloak-dev...@googlegroups.com <mailto:keycloak-dev%2Bunsu...@googlegroups.com>.
>
> asch...@redhat.com
>
> <https://www.redhat.com/>
> <https://groups.google.com/d/msgid/keycloak-dev/CAA6cDELn3noJV2FuxDBgkgGHaq5FE_X91JD7EYd%2BvWfUh%2BCgSQ%40mail.gmail.com?utm_medium=email&utm_source=footer>.
Specifically on my JPA/Hibernate question, I can't figure out where the those transactions are set up. I'd like to experiment with a transaction retry, but I can't find where those queries are getting added to the transaction manager. I can catch the retryable exception upstream and hack it, but it doesn't seem to be the right way to do it.
A stack trace of the exception (attached) doesn't give any useful information, as it just shows the transaction manager and the JDBC driver calls in the stack.
Any ideas?
Thanks again,
Garth
On Fri, Apr 1, 2022, at 11:47 AM, Alexander Schwartz wrote:
> To understand what is currently implemented, I recommend looking at the
> DefaultKeycloakTransactionManager where its begin() and commit()
> methods are being called; usually from a request filter. During a
> transaction multiple transactions enlist to commit their work at the
> end of a request.
>
> Best,
> Alexander
>
> On Thu, Mar 31, 2022 at 10:11 AM Garth <ga...@tunnel19.com> wrote:
>> One concrete question: Where are the JPA/Hibernate transactions set up and run? I assumed they would be wrapped in a `JpaKeycloakTransaction`, but that doesn't seem to be the case. I wanted to experiment with some conditional transaction retry (based on SQLException state), but I couldn't find a consistent place to do it. Ideas?
>>
>> On Wed, Mar 30, 2022, at 12:00 PM, Garth wrote:
>> > Hi All,
>> >
>> > I'm looking to get a better understanding of how transactions are
>> > performed in Keycloak. I've dug into the code, and done some tracing in
>> > my IDE, and I have a basic understanding of how it works. However, I'd
>> > like to see if there is any documentation about how it is built and
>> > some of the design decisions that were made. Does anyone know of such a
>> > document?
>> >
>> > Thanks!
>> > Garth
>>
>> --
>> You received this message because you are subscribed to the Google Groups "Keycloak Dev" group.
>> To view this discussion on the web visit https://groups.google.com/d/msgid/keycloak-dev/4ce810fb-d83f-412b-9162-0810b6d2fd98%40www.fastmail.com.
>>
>
>
> --
> Alexander Schwartz, RHCE
>
> He/Him
>
> Principal Software Engineer, Keycloak
>
> Red Hat <https://www.redhat.com/> - Germany remote
>>
>
>
> --
> Alexander Schwartz, RHCE
>
> He/Him
>
> Principal Software Engineer, Keycloak
>
>
> asch...@redhat.com
>
> <https://www.redhat.com/>
>
> --
> You received this message because you are subscribed to the Google
> Groups "Keycloak Dev" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to keycloak-dev...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/keycloak-dev/CAA6cDELn3noJV2FuxDBgkgGHaq5FE_X91JD7EYd%2BvWfUh%2BCgSQ%40mail.gmail.com
> --
> You received this message because you are subscribed to the Google
> Groups "Keycloak Dev" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to keycloak-dev...@googlegroups.com.
> To view this discussion on the web visit
> <https://groups.google.com/d/msgid/keycloak-dev/CAA6cDELn3noJV2FuxDBgkgGHaq5FE_X91JD7EYd%2BvWfUh%2BCgSQ%40mail.gmail.com?utm_medium=email&utm_source=footer>.
Garth
Apr 4, 2022, 12:37:57 PM4/4/22
to Till Markus (IOC/PAU1)
Some notes on the JPA/Hibernate question:
- using JpaKeycloakTransaction wasn't viable, as it looks like it only wraps the EntityManager transaction when JTA is disabled. As a separate question, I couldn't figure out why that would ever be the case, but I digress.
- JpaKeycloakTransaction was also only used in DefaultJpaConnectionProviderFactory (legacy), while it was imported but not used in AbstractJpaConnectionProviderFactory (quarkus). Another separate question, is that because JTA is always enabled with the new quarkus version?
- I wrapped the EntityManager (piggybacking on the proxy code in PersistenceExceptionConverter), which helped me understand when/how the EntityManager was getting called. But, since the EntityManager.getTransaction()... is not getting used, it was not useful in finding where to catch and deal with Connection exceptions when committing the transaction.
I still am no closer to understanding where the code is that schedules the Connection.commit with the transaction manager. Is that something Hibernate does? Can someone point me to where in the Keycloak code that happens, or is it some automagic JPA/Hibernate thing?
My goal is to catch and retry transactions that use the database and throw a specific SQLException.
> https://groups.google.com/d/msgid/keycloak-dev/4f6d4b74-98d9-4ae0-bc10-875179fc5f3f%40www.fastmail.com.
>
> Attachments:
> * exception.log
- using JpaKeycloakTransaction wasn't viable, as it looks like it only wraps the EntityManager transaction when JTA is disabled. As a separate question, I couldn't figure out why that would ever be the case, but I digress.
- JpaKeycloakTransaction was also only used in DefaultJpaConnectionProviderFactory (legacy), while it was imported but not used in AbstractJpaConnectionProviderFactory (quarkus). Another separate question, is that because JTA is always enabled with the new quarkus version?
- I wrapped the EntityManager (piggybacking on the proxy code in PersistenceExceptionConverter), which helped me understand when/how the EntityManager was getting called. But, since the EntityManager.getTransaction()... is not getting used, it was not useful in finding where to catch and deal with Connection exceptions when committing the transaction.
I still am no closer to understanding where the code is that schedules the Connection.commit with the transaction manager. Is that something Hibernate does? Can someone point me to where in the Keycloak code that happens, or is it some automagic JPA/Hibernate thing?
My goal is to catch and retry transactions that use the database and throw a specific SQLException.
>
> Attachments:
> * exception.log
Alexander Schwartz
Apr 7, 2022, 3:27:17 AM4/7/22
to Garth, Till Markus (IOC/PAU1)
Hi Garth,
sorry for the late reply.
My understanding of a rolled back transaction is that some of the data read during processing is now stale, and the processing needs to start from the beginning, that is: start the transaction from the very beginning.
When processing a request, there might be more than one transaction running; one for the actual request from the caller, sometimes another nested one (an "autonomous transaction" as it is sometimes called in other contexts).
The example you had at hand touches KeycloakModelUtils.runJobInTransaction() that is designed for such a nested autonomous transaction. To retry such a transaction, I'd recommend wrapping the whole body of the method (starting with creating the session) into a retry loop, probably with a configurable retry count. Please note that callers might not expect such a retry and there might be side effects - let's see how it goes and cover for that later.
There are also different wrappers for the WildFly and Quarkus for the actual request. I believe for Quarkus this would be QuarkusRequestFilter.createBlockingHandler(). Here, the same pattern would apply about wrapping the whole promise. Here, a retry must only occur when the header of the response hasn't been written to the caller. Retrying here could actually be safer than doing a retry in the runJobInTransaction() as I would expect less side effects. Still, for startup activities wrapping the runJobInTransaction() might still be necessary.
Please let me know how this goes, and add both the list and my email address in CC.
Best,
Alexander
To view this discussion on the web visit https://groups.google.com/d/msgid/keycloak-dev/c8cb46d9-fbc0-4e16-88ca-c6f6c3c1cfd3%40www.fastmail.com.
Alexander Schwartz, RHCE
He/Him
Principal Software Engineer, Keycloak
Garth
Apr 7, 2022, 6:45:22 AM4/7/22
to Alexander Schwartz, Till Markus (IOC/PAU1)
Hi Alexander,
Thank you for the response and the additional information.
I have been working in AbstractRequestFilter, QuarkusRequestFilter, and KeycloakModelUtils.runJobInTransaction to build a retry loop. I have a working version for KeycloakModelUtils.runJobInTransaction for the startup tasks (e.g. creating master realm), and I'm trying to make it work for the Wildfly and Quarkus request filters.
I still don't really understand when/how JTA is used in Quarkus versus Wildfly. There seems to be differing logic for when XA is enabled in Quarkus and JTA is enabled in Wildfly. Maybe it's just how the new code is written (and uses the dependency injector) that is confusing me. I'd like to be able to use JpaKeycloakTransaction and enlist that as a transaction, as I was able to do in the Wildfly version, but it doesn't seem like that works in Quarkus. Anyway, any more information to help my understanding here would be very helpful.
I'll post my WIP diff to github once I have it cleaned up a bit.
Thanks!
Garth
>> >>> To unsubscribe from this group and stop receiving emails from it, send an email to keycloak-dev...@googlegroups.com <mailto:keycloak-dev%2Bunsu...@googlegroups.com> <mailto:keycloak-dev%2Bunsu...@googlegroups.com <mailto:keycloak-dev%252Buns...@googlegroups.com>>.
Thank you for the response and the additional information.
I have been working in AbstractRequestFilter, QuarkusRequestFilter, and KeycloakModelUtils.runJobInTransaction to build a retry loop. I have a working version for KeycloakModelUtils.runJobInTransaction for the startup tasks (e.g. creating master realm), and I'm trying to make it work for the Wildfly and Quarkus request filters.
I still don't really understand when/how JTA is used in Quarkus versus Wildfly. There seems to be differing logic for when XA is enabled in Quarkus and JTA is enabled in Wildfly. Maybe it's just how the new code is written (and uses the dependency injector) that is confusing me. I'd like to be able to use JpaKeycloakTransaction and enlist that as a transaction, as I was able to do in the Wildfly version, but it doesn't seem like that works in Quarkus. Anyway, any more information to help my understanding here would be very helpful.
I'll post my WIP diff to github once I have it cleaned up a bit.
Thanks!
Garth
>> >>> To view this discussion on the web visit https://groups.google.com/d/msgid/keycloak-dev/4ce810fb-d83f-412b-9162-0810b6d2fd98%40www.fastmail.com.
>> >>>
>> >>
>> >>
>> >> --
>> >> Alexander Schwartz, RHCE
>> >>
>> >> He/Him
>> >>
>> >> Principal Software Engineer, Keycloak
>> >>
>> >> Red Hat <https://www.redhat.com/> - Germany remote
>> >>
>> >> asch...@redhat.com
>> >>
>> >> <https://www.redhat.com/>
>> >>
>> >> --
>> >> You received this message because you are subscribed to the Google
>> >> Groups "Keycloak Dev" group.
>> >> To unsubscribe from this group and stop receiving emails from it, send
>> >> an email to keycloak-dev...@googlegroups.com <mailto:keycloak-dev%2Bunsu...@googlegroups.com>.
>> >> To view this discussion on the web visit
>> >>>
>> >>
>> >>
>> >> --
>> >> Alexander Schwartz, RHCE
>> >>
>> >> He/Him
>> >>
>> >> Principal Software Engineer, Keycloak
>> >>
>> >> Red Hat <https://www.redhat.com/> - Germany remote
>> >>
>> >> asch...@redhat.com
>> >>
>> >> <https://www.redhat.com/>
>> >>
>> >> --
>> >> You received this message because you are subscribed to the Google
>> >> Groups "Keycloak Dev" group.
>> >> To unsubscribe from this group and stop receiving emails from it, send
>> >> an email to keycloak-dev...@googlegroups.com <mailto:keycloak-dev%2Bunsu...@googlegroups.com>.
>> >> To view this discussion on the web visit
>> >> https://groups.google.com/d/msgid/keycloak-dev/CAA6cDELn3noJV2FuxDBgkgGHaq5FE_X91JD7EYd%2BvWfUh%2BCgSQ%40mail.gmail.com
>> >> <https://groups.google.com/d/msgid/keycloak-dev/CAA6cDELn3noJV2FuxDBgkgGHaq5FE_X91JD7EYd%2BvWfUh%2BCgSQ%40mail.gmail.com?utm_medium=email&utm_source=footer>.
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> > Groups "Keycloak Dev" group.
>> > To unsubscribe from this group and stop receiving emails from it, send
>> >> <https://groups.google.com/d/msgid/keycloak-dev/CAA6cDELn3noJV2FuxDBgkgGHaq5FE_X91JD7EYd%2BvWfUh%2BCgSQ%40mail.gmail.com?utm_medium=email&utm_source=footer>.
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> > Groups "Keycloak Dev" group.
>> > To unsubscribe from this group and stop receiving emails from it, send
>> > an email to keycloak-dev...@googlegroups.com <mailto:keycloak-dev%2Bunsu...@googlegroups.com>.
>> > To view this discussion on the web visit
>> > To view this discussion on the web visit
>> > https://groups.google.com/d/msgid/keycloak-dev/4f6d4b74-98d9-4ae0-bc10-875179fc5f3f%40www.fastmail.com.
>> >
>> > Attachments:
>> > * exception.log
>>
>> --
>> You received this message because you are subscribed to the Google Groups "Keycloak Dev" group.
>> >
>> > Attachments:
>> > * exception.log
>>
>> --
>> You received this message because you are subscribed to the Google Groups "Keycloak Dev" group.
>> To unsubscribe from this group and stop receiving emails from it, send an email to keycloak-dev...@googlegroups.com <mailto:keycloak-dev%2Bunsu...@googlegroups.com>.
>> To view this discussion on the web visit https://groups.google.com/d/msgid/keycloak-dev/c8cb46d9-fbc0-4e16-88ca-c6f6c3c1cfd3%40www.fastmail.com.
>>
>
>
> --
> Alexander Schwartz, RHCE
>
> He/Him
>
> Principal Software Engineer, Keycloak
>
>>
>
>
> --
> Alexander Schwartz, RHCE
>
> He/Him
>
> Principal Software Engineer, Keycloak
>
Alexander Schwartz
Apr 8, 2022, 3:57:28 AM4/8/22
to Garth, Till Markus (IOC/PAU1)
Hi Garth,
I don't have experience in that area you're asking about so I have unfortunately nothing to share here.
Good luck with your investigation.
Best,
Alexander
Stefan Guilhen
Apr 8, 2022, 8:03:51 AM4/8/22
to Garth, Alexander Schwartz, Till Markus (IOC/PAU1)
Hi Garth,
On Thu, Apr 7, 2022 at 7:45 AM Garth <ga...@tunnel19.com> wrote:
I still don't really understand when/how JTA is used in Quarkus versus Wildfly. There seems to be differing logic for when XA is enabled in Quarkus and JTA is enabled in Wildfly. Maybe it's just how the new code is written (and uses the dependency injector) that is confusing me. I'd like to be able to use JpaKeycloakTransaction and enlist that as a transaction, as I was able to do in the Wildfly version, but it doesn't seem like that works in Quarkus. Anyway, any more information to help my understanding here would be very helpful.
I'm not a Keycloak tx expert but I think this is handled by the DefaultKeycloakTransactionManager itself: in the begin() method it uses the JtaTransactionManagerLookup provider to check if there's a JTA transaction manager available and if it is the case, wraps into into a KeycloakTransaction and enlists it. The implementations for WildFly and Quarkus are the JBossJtaTransacionManagerLookup, which performs a JNDI lookup, and the QuarkusJtaTransactionManagerLookup.
I'll post my WIP diff to github once I have it cleaned up a bit.
Great, looking forward to that.
To unsubscribe from this group and stop receiving emails from it, send an email to keycloak-dev...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/keycloak-dev/53d375f5-3779-4b56-b7fa-a315b839339f%40www.fastmail.com.
Reply all
Reply to author
Forward
0 new messages