A XAResource hook for announcing the end of the recovery cycle (Agroal integration)

70 views
Skip to first unread message

Ondra Chaloupka

unread,
Jun 3, 2020, 6:38:57 AM6/3/20
to narayana-users
Hi,

I would like to discuss here my proposal[1] of providing an information about ending of the recovery scan for XAResource, see https://issues.redhat.com/browse/JBTM-3325.
This proposal comes from discussion how to help Agroal/Quarkus to integrate with Narayana recovery processing.

The recovery processing uses registered XAResourceRecovery[2] instances to ask the RM for XAResources which will be considered during recovery.
The Agroal needs to provide the XAResource from the JDBC XAConnection. On call of the XAResourceRecovery#getXAResources() it creates a new connection (taken from the pool) and returns it to the recovery processing.
Then there is no easy way for Agroal to find out that the recovery cycle finished and the resource data could be released (the connection could be closed).

The idea of the proposal is to make the XAResource closable (ie. AutoCloseable) and for recovery then calling the XAResource#close() at time when Narayana finishes with the recovery for particular XAResource at the end of each recovery cycle.

Are there some concerns about such feature?

Thanks for your ideas,
Ondra

Mark Little

unread,
Jun 3, 2020, 6:53:19 AM6/3/20
to Ondra Chaloupka, narayana-users
Do you have a flow diagram to illustrate what you expect to happen and the participants? Given that this shouldn't be persistent information it sounds more like a job for a Synchronization but until I see the flow I won't know for sure.

Mark.

--
You received this message because you are subscribed to the Google Groups "narayana-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to narayana-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/narayana-users/5082bc90-a70e-4907-95bc-b6c0f1a30780%40googlegroups.com.

Michael Musgrove

unread,
Jun 3, 2020, 6:54:37 AM6/3/20
to Ondra Chaloupka, narayana-users
My concerns were expressed on the JBTM [1] and Ondra responded to my concerns in the comments section on the JBTM.


On Wed, Jun 3, 2020 at 11:39 AM Ondra Chaloupka <ocha...@redhat.com> wrote:
--
You received this message because you are subscribed to the Google Groups "narayana-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to narayana-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/narayana-users/5082bc90-a70e-4907-95bc-b6c0f1a30780%40googlegroups.com.


--
Michael Musgrove

JBoss, by Red Hat
Registered Address: Red Hat Ltd, 6700 Cork Airport Business Park, Kinsale Road, Co. Cork.
Registered in the Companies Registration Office, Parnell House, 14 Parnell Square, Dublin 1, Ireland, No.304873
Directors:Michael Cunningham (USA), Vicky Wiseman (USA), Michael O'Neill, Keith Phelan, Matt Parson (USA)


Jonathan Halliday

unread,
Jun 3, 2020, 7:25:02 AM6/3/20
to narayana-users

It's harder than it looks.

It's worth noting that many users don't actually want it. We've had complaints when creating new connections every 2 minutes puts additional load on the db servers, especially when app server clusters are running recovery simultaneously. Thus recovery helper implementations normally assume they hold the connection for the lifetime of the server, or until it breaks. Why does the connection pool care? What's agroals's actual requirement here? Why can't it support a user (the recovery system is no different to an app here) from checking out a connection for an arbitrarily long interval?

We don't have a good way to tracking the lifetime of an XAResource. It's NOT limited to the time between TMSTARTSCAN and TMENDSCAN, or indeed to a single recovery cycle. Bottom up recovery mostly uses it in that way, but top down recovery from other modules doesn't.  You could wrap the XAResource in a reference counting object, or rely on finalizers (yuck), but it's generally non-trivial for anything other than the gc to know when we're done with a XAResource.

Jonathan

Ondra Chaloupka

unread,
Jun 4, 2020, 6:01:55 AM6/4/20
to narayana-users
Thanks Jonathan, that's are great points. Especially, I haven't realized the fact about other recovery modules. I'm not an Agroal contributor but my perspective is following.

The connections are not about to be created every 2 minutes. The AutoCloseable for XAResource is up to the implementation to cover. For Agroal it's to return the connection back to the pool. I assume this is a benefit for Agroal design to use the standard manner of handling of connection either for business logic or for the recovery.
As I was studying e.g. the IronJacamar code there is a special separate connection only for recovery which is never closed. It requires the special handling around such connection (e.g. like "is still alive?").

I consider that recovery behaves a little differently to a user. The recovery asks every two minutes for "a new XAResource" (connection). The resource is expected to be working and is never closed/returned.
A user asks for a connection and works with it for arbitrary long time and then it closes it. If he needs a new connection then he asks for a new one. Not closing an opened connection and asking for a new one is considered as a bad dangerous practice and is to blame the user for such a wrong usage.

I see now that the other modules may work with the XAResource loaded by bottom-up recovery in XARecoveryModule. As I understand it, then currently it's the AtomicActionRecoveryModule which reloads the state from the persistent storage but if the XAResourceRecord is not capable to deserialize then it asks the XARecoveryModule for help[1]. Currently it's the only place, what I can see, where this inter recovery module interaction happens.

I'm not sure if the XAResource reference counting would help as the resource is created only once, and I expect to work only with one instance here, while I want to find a place where to close it safely.
As an idea, what about to have some synchronization for XAResourceRecovery[2] (like `public XAResource[] getXAResources(EndRecoverySynchronization sync);`) and which will be called at time when all recovery modules finished its work for particular recovery cycle, aka. somewhere at PeriodicRecovery[3]. What do you think?

Anyway, I will point Luis to this thread if he wants to provide details about Agroal implementation considerations.

Thanks,
Ondra

Ondra Chaloupka

unread,
Jun 22, 2020, 9:52:36 AM6/22/20
to narayana-users
I'm closing this discussion for now as there is no more data coming about possible resolution.
First, I would need to understand the Luis' needs for the integration and for seconds what are limits of my proposal from Jonathan.
As the current integration of Narayana-Quarkus-Agroal works then I do not consider this as high priority issue and I'm leaving it for the future investigation. I'm going to comment with a summary to https://issues.redhat.com/browse/JBTM-3325
Reply all
Reply to author
Forward
0 new messages