Recover and clear transaction heuristic state by mbean management operation

117 views
Skip to first unread message

Ondra Chaloupka

unread,
Jun 11, 2020, 7:38:52 AM6/11/20
to narayana-users
Hi,

I have a question on Narayana management operations exposed via mbeans. I wonder about 'clearHeuristic'[1] in particular. This is the operation which is invoked when WildFly cli executes ':recover'[2][3].

Let me explain the scenario which I consider wrong. I came to it when I've been designing testcase for WFTC-85[4]. There are two WildFly servers where the first one calls EJB remote bean on the second server. There is transaction context propagated over the call.
There happens to be an network intermittent failure at time when commit is called. As the first WildFly server enlisted only the EJB remote call as the XAResource there is used the 1PC. On the RMFAIL error (emitted based on the fact there is connection crash) the XAResourceRecord is assigned with heuristic error[5]. The transaction is marked as heuristics. Then the user comes to the WildFly console and wants to finished it. He invokes the ':recover' but nothing happens. The transaction/participant is left uncommitted.
I found that the reason is that the management operation 'clearHeuristic' is invoked only on the "covering" transaction - BasicAction is modified[6] -> [7] -> [8]. But the XAResourceRecord heuristics state[9] is unchanged. Then during recovery commit retry the XAResource.commit is denied[10].

I would like to understand if this could be considered as a flaw of the heuristics management processing or if that's by design for reasons.

After some experiments I've got to a working PoC[11] which main point is to permits to clear heuristic decision state from the AbstractRecord - instantiated as the XAResourceRecord in this case. When the heuristic state of the record (XAResourceRecord._heuristics) is cleared and saved by the management operation then the recovery cycle is able to take it and replay the processing on it.

Would you have some thoughts here?

Thanks a lot
Ondra


[2] /subsystem=transactions/log-store=log-store/transactions=.../participants=...:recover

Mark Little

unread,
Jun 17, 2020, 4:58:30 AM6/17/20
to Ondra Chaloupka, narayana-users
When you say below that the transaction/participant is left uncommitted which do you mean? Transaction or participant? Heuristic decisions aren’t associated with uncommitted states, they are only associated with (potentially wrong) committed states.

Mark.


--
You received this message because you are subscribed to the Google Groups "narayana-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to narayana-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/narayana-users/9c2b154f-443e-49f1-a3ad-67aa0724e6ffo%40googlegroups.com.

---
Mark Little
mli...@redhat.com

JBoss, by Red Hat
Registered Address: Red Hat Ltd, 6700 Cork Airport Business Park, Kinsale Road, Co. Cork.
Registered in the Companies Registration Office, Parnell House, 14 Parnell Square, Dublin 1, Ireland, No.304873
Directors:Michael Cunningham (USA), Vicky Wiseman (USA), Michael O'Neill, Keith Phelan, Matt Parson (USA)




Ondra Chaloupka

unread,
Jun 17, 2020, 5:54:46 AM6/17/20
to narayana-users
On Wednesday, June 17, 2020 at 10:58:30 AM UTC+2, Mark Little wrote:
When you say below that the transaction/participant is left uncommitted which do you mean? Transaction or participant? Heuristic decisions aren’t associated with uncommitted states, they are only associated with (potentially wrong) committed states.

The transaction which I refer to is the one which is started on one WildFly server and is propagated with the context to the second WildFly server. The transaction manager works with the EJB remoting call as with a standard XA resource. Then there is done a transactional work on the second server. There are enlisted two participants - in case of the test case they are only mock XA resources. At time of the commit (1PC processing) - which is emitted from the first server - the mock XA resources are prepared on the second server but then failure at commit happens (it's simulation of network error which can happen for e.g. database JDBC driver trying to connect to a database to commit the prepared resource). At that time the whole transaction is marked as heuristic. There is not clear if the commit call to the database passed through and (the possible) database has been committed or it's only in a prepared state. As said with this failure the transaction is marked as heuristic and the EJB remoting XA resource participant on the first WildFly server is marked with heuristic state as well.
Does my explanation make sense?
 

Mark.


To unsubscribe from this group and stop receiving emails from it, send an email to narayana-users+unsubscribe@googlegroups.com.

Michael Musgrove

unread,
Jun 17, 2020, 6:32:50 AM6/17/20
to Ondra Chaloupka, narayana-users
The tooling clears heuristics by putting them back onto the prepared list. The linked PoC is marking the resource as finished which hides the heuristic. As far as I can recall the tooling only allows deleting heuristics (with the usual warnings about doing this) or allows marking them as prepared for another attempt at finishing it.


Mark.


To unsubscribe from this group and stop receiving emails from it, send an email to narayana-user...@googlegroups.com.

---
Mark Little
mli...@redhat.com

JBoss, by Red Hat
Registered Address: Red Hat Ltd, 6700 Cork Airport Business Park, Kinsale Road, Co. Cork.
Registered in the Companies Registration Office, Parnell House, 14 Parnell Square, Dublin 1, Ireland, No.304873
Directors:Michael Cunningham (USA), Vicky Wiseman (USA), Michael O'Neill, Keith Phelan, Matt Parson (USA)




--
You received this message because you are subscribed to the Google Groups "narayana-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to narayana-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/narayana-users/7d883b72-2843-415f-935e-caa4258ee698o%40googlegroups.com.


--
Michael Musgrove

Mark Little

unread,
Jun 17, 2020, 7:03:47 AM6/17/20
to Michael Musgrove, Ondra Chaloupka, narayana-users
Yes, heuristic resolution is a manual process because only an administrator, say, can tell the system which way the transaction was meant to complete and hence which participants went the wrong way. An admin could resolve the heuristic entirely "off line" and then use tooling to clear/delete the heuristic information from the participant or transaction log.

Mark.

To view this discussion on the web, visit https://groups.google.com/d/msgid/narayana-users/CACKZbau--xEeW5zdQkyzcVidsrvr2KQ13_L%2B9_i0kMq6rax_ZA%40mail.gmail.com.

Ondra Chaloupka

unread,
Jun 18, 2020, 3:46:34 AM6/18/20
to narayana-users
Thanks Mike and Mark for the feedback.
Agree with all what was said - the tooling is meant for admin to manually resolve the heuristics. As said, the tooling currently provides a way to delete or recover (marking them as prepared).

Now, my question is about the particular scenario and the particular way how code handles it.

I tried to explain the code path of the scenario in my first post (the part where I talk about BasiAction and where I linked the source code). If I do not go that deep to the code path the issue that I observe is that the transaction (BasicAction) as whole is marked as heuristic. At that time the XAResourceRecord is marked as heuristic (by changing the _heuristic flag) as well. But the WildFly recover functionality (or the clearHeuristic action in mbean) reverts only the state of the transaction prepared. But the heuristic state of the resource is not reverted back to "ready to be handled".

This behaviour seems to me wrong and I would like to fix it. I wanted to understand if I haven't missed some aspect. I would like to create a JBTM and propose a fix that would be clearing the heuristic flag from the resource as well.

@Mike: please, can you elaborate a bit more about the statement "The linked PoC is marking the resource as finished which hides the heuristic". My point was opposite. I'm changing only the '_heuristic' flag (https://github.com/ochaloup/narayana/commit/7ef2af5c469f634f82696bc0ad1120b9dc684097#diff-9613cb84dbca3e6a76d760b2b4626befR792) and I intentionally do not touch the participant/resource state. What have I missed?

Thank you
Ondra

Mark.


To unsubscribe from this group and stop receiving emails from it, send an email to narayana-users+unsubscribe@googlegroups.com.

---
Mark Little
mli...@redhat.com

JBoss, by Red Hat
Registered Address: Red Hat Ltd, 6700 Cork Airport Business Park, Kinsale Road, Co. Cork.
Registered in the Companies Registration Office, Parnell House, 14 Parnell Square, Dublin 1, Ireland, No.304873
Directors:Michael Cunningham (USA), Vicky Wiseman (USA), Michael O'Neill, Keith Phelan, Matt Parson (USA)




--
You received this message because you are subscribed to the Google Groups "narayana-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to narayana-users+unsubscribe@googlegroups.com.


--
Michael Musgrove

JBoss, by Red Hat
Registered Address: Red Hat Ltd, 6700 Cork Airport Business Park, Kinsale Road, Co. Cork.
Registered in the Companies Registration Office, Parnell House, 14 Parnell Square, Dublin 1, Ireland, No.304873
Directors:Michael Cunningham (USA), Vicky Wiseman (USA), Michael O'Neill, Keith Phelan, Matt Parson (USA)


--
You received this message because you are subscribed to the Google Groups "narayana-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to narayana-users+unsubscribe@googlegroups.com.

Michael Musgrove

unread,
Jun 18, 2020, 5:32:56 AM6/18/20
to Ondra Chaloupka, narayana-users
For JTA the tooling only supports resource records that are in-lined with the transaction log.
For JTS we support resource records that are located in a separate part of the object store (separate from the actual transaction log).

Does your use case match either of these two strategies?


Mark.


To unsubscribe from this group and stop receiving emails from it, send an email to narayana-user...@googlegroups.com.

---
Mark Little
mli...@redhat.com

JBoss, by Red Hat
Registered Address: Red Hat Ltd, 6700 Cork Airport Business Park, Kinsale Road, Co. Cork.
Registered in the Companies Registration Office, Parnell House, 14 Parnell Square, Dublin 1, Ireland, No.304873
Directors:Michael Cunningham (USA), Vicky Wiseman (USA), Michael O'Neill, Keith Phelan, Matt Parson (USA)




--
You received this message because you are subscribed to the Google Groups "narayana-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to narayana-user...@googlegroups.com.


--
Michael Musgrove

JBoss, by Red Hat
Registered Address: Red Hat Ltd, 6700 Cork Airport Business Park, Kinsale Road, Co. Cork.
Registered in the Companies Registration Office, Parnell House, 14 Parnell Square, Dublin 1, Ireland, No.304873
Directors:Michael Cunningham (USA), Vicky Wiseman (USA), Michael O'Neill, Keith Phelan, Matt Parson (USA)


--
You received this message because you are subscribed to the Google Groups "narayana-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to narayana-user...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "narayana-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to narayana-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/narayana-users/05338914-30bf-4ffd-befb-5348f1f70f65o%40googlegroups.com.

Ondra Chaloupka

unread,
Jun 18, 2020, 6:24:21 AM6/18/20
to narayana-users
@Mike: this is JTA. I'm not sure what is meant by term "in-lined with the transaction log". The resource is the XAResourceRecord which uses the save_state and restore_state[1] methods during the recovery to load the state from the transaction log. From that I assume this is in-lined with the transaction log, right? If so, then the handling matches the first strategy.


Mark.


To unsubscribe from this group and stop receiving emails from it, send an email to narayana-users+unsubscribe@googlegroups.com.

---
Mark Little
mli...@redhat.com

JBoss, by Red Hat
Registered Address: Red Hat Ltd, 6700 Cork Airport Business Park, Kinsale Road, Co. Cork.
Registered in the Companies Registration Office, Parnell House, 14 Parnell Square, Dublin 1, Ireland, No.304873
Directors:Michael Cunningham (USA), Vicky Wiseman (USA), Michael O'Neill, Keith Phelan, Matt Parson (USA)




--
You received this message because you are subscribed to the Google Groups "narayana-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to narayana-users+unsubscribe@googlegroups.com.


--
Michael Musgrove

JBoss, by Red Hat
Registered Address: Red Hat Ltd, 6700 Cork Airport Business Park, Kinsale Road, Co. Cork.
Registered in the Companies Registration Office, Parnell House, 14 Parnell Square, Dublin 1, Ireland, No.304873
Directors:Michael Cunningham (USA), Vicky Wiseman (USA), Michael O'Neill, Keith Phelan, Matt Parson (USA)


--
You received this message because you are subscribed to the Google Groups "narayana-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to narayana-users+unsubscribe@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "narayana-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to narayana-users+unsubscribe@googlegroups.com.

Michael Musgrove

unread,
Jun 18, 2020, 8:12:01 AM6/18/20
to Ondra Chaloupka, narayana-users
So it should work. Maybe it's just a bug rather than a feature enhancement.

NB in-line just means that the resource records are stored in the same log as the transaction record (whereas JTS stores resource records in a log which is separate from the transaction log).



Mark.


To unsubscribe from this group and stop receiving emails from it, send an email to narayana-user...@googlegroups.com.

---
Mark Little
mli...@redhat.com

JBoss, by Red Hat
Registered Address: Red Hat Ltd, 6700 Cork Airport Business Park, Kinsale Road, Co. Cork.
Registered in the Companies Registration Office, Parnell House, 14 Parnell Square, Dublin 1, Ireland, No.304873
Directors:Michael Cunningham (USA), Vicky Wiseman (USA), Michael O'Neill, Keith Phelan, Matt Parson (USA)




--
You received this message because you are subscribed to the Google Groups "narayana-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to narayana-user...@googlegroups.com.


--
Michael Musgrove

JBoss, by Red Hat
Registered Address: Red Hat Ltd, 6700 Cork Airport Business Park, Kinsale Road, Co. Cork.
Registered in the Companies Registration Office, Parnell House, 14 Parnell Square, Dublin 1, Ireland, No.304873
Directors:Michael Cunningham (USA), Vicky Wiseman (USA), Michael O'Neill, Keith Phelan, Matt Parson (USA)


--
You received this message because you are subscribed to the Google Groups "narayana-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to narayana-user...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "narayana-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to narayana-user...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "narayana-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to narayana-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/narayana-users/5b09aeda-edd6-4659-944a-59fbdd98b21do%40googlegroups.com.

Michael Musgrove

unread,
Jun 18, 2020, 9:01:06 AM6/18/20
to Michael Musgrove, Ondra Chaloupka, narayana-users
We have different tooling MBeans to handle the various Arjuna record types. If your resource is not one of the usual ones then maybe we need to create a dedicated MBean for it - it depends on your use case.

To view this discussion on the web, visit https://groups.google.com/d/msgid/narayana-users/CAKJ8-j7uvdwEsuFVHFiAUB%2BexVYRJO-%3DtosVdxOQr61b6kKMNA%40mail.gmail.com.

Mark Little

unread,
Jun 18, 2020, 9:05:15 AM6/18/20
to Michael Musgrove, Michael Musgrove, Ondra Chaloupka, narayana-users
Maybe start by building an example which should work?

Mark.

To view this discussion on the web, visit https://groups.google.com/d/msgid/narayana-users/CACKZbauGVVjbvaaVWRzPv7nUhHgoP8ayQ6rFiVnDm9xEaFispA%40mail.gmail.com.

Ondra Chaloupka

unread,
Jun 18, 2020, 11:11:15 AM6/18/20
to narayana-users
Thanks Mike I haven't realized that the expected behaviour is to have a mbean handling various record.
But as I was examining the code now I can see that the participant log record is the one managed by LogRecordWrapper. I consider it logic as the WildFly :recover call is run against the participant not against the transction. The 'clearHeuristic' is defined at LogRecordWrapper. Then it just pass the call up to the AtomicAction. I mean the :recover is called on LogRecord (which is defined by javadoc that it works with participants) but the call is then redirected to the AtomicAction which works with transaction.

From the discussion I would think it's a bug. I'm going to create a JBTM.

The scenario/example can be checked here


Mark.


To unsubscribe from this group and stop receiving emails from it, send an email to narayana-users+unsubscribe@googlegroups.com.

---
Mark Little
mli...@redhat.com

JBoss, by Red Hat
Registered Address: Red Hat Ltd, 6700 Cork Airport Business Park, Kinsale Road, Co. Cork.
Registered in the Companies Registration Office, Parnell House, 14 Parnell Square, Dublin 1, Ireland, No.304873
Directors:Michael Cunningham (USA), Vicky Wiseman (USA), Michael O'Neill, Keith Phelan, Matt Parson (USA)




--
You received this message because you are subscribed to the Google Groups "narayana-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to narayana-users+unsubscribe@googlegroups.com.


--
Michael Musgrove

JBoss, by Red Hat
Registered Address: Red Hat Ltd, 6700 Cork Airport Business Park, Kinsale Road, Co. Cork.
Registered in the Companies Registration Office, Parnell House, 14 Parnell Square, Dublin 1, Ireland, No.304873
Directors:Michael Cunningham (USA), Vicky Wiseman (USA), Michael O'Neill, Keith Phelan, Matt Parson (USA)


--
You received this message because you are subscribed to the Google Groups "narayana-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to narayana-users+unsubscribe@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "narayana-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to narayana-users+unsubscribe@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "narayana-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to narayana-users+unsubscribe@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "narayana-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to narayana-users+unsubscribe@googlegroups.com.


--
Michael Musgrove

JBoss, by Red Hat
Registered Address: Red Hat Ltd, 6700 Cork Airport Business Park, Kinsale Road, Co. Cork.
Registered in the Companies Registration Office, Parnell House, 14 Parnell Square, Dublin 1, Ireland, No.304873
Directors:Michael Cunningham (USA), Vicky Wiseman (USA), Michael O'Neill, Keith Phelan, Matt Parson (USA)


--
You received this message because you are subscribed to the Google Groups "narayana-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to narayana-users+unsubscribe@googlegroups.com.

Mark Little

unread,
Jun 19, 2020, 4:51:57 AM6/19/20
to Ondra Chaloupka, narayana-users
So you’ve created an example which should work and it doesn’t? I don’t see that answered below.

Mark.


To unsubscribe from this group and stop receiving emails from it, send an email to narayana-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/narayana-users/c23c60b5-7db9-4074-acc8-22183c6a622fo%40googlegroups.com.

Ondra Chaloupka

unread,
Jun 19, 2020, 5:43:01 AM6/19/20
to narayana-users

Michael Musgrove

unread,
Jun 19, 2020, 9:38:19 AM6/19/20
to narayana-users
Ondra, your first post says that you are propagating a transaction. Since narayana only supports JTS for transaction propagation you will be using something external to narayana (WFTC). Since there is no tooling integration between narayana and WFTC I would expect there to be issues for manually resolving these transactions.

Also the javadoc for the test your reference in your last post says that the client commit uses 1PC but the remote server uses 2PC so are these two separate transactions. Furthermore, since there is no logging for 1PC transactions how do you handle the absence of a log in failure scenarios. It sounds like you need some extra tooling support for your use case - that was what I was trying to get at in my first response on this thread, i.e. it sounds like a new use case/feature and not a bug.

Ondra Chaloupka

unread,
Jun 19, 2020, 1:54:53 PM6/19/20
to narayana-users
Ok. Then if it's a new use case then let's discuss how it should behave and if there is an approach how to change the Narayana codebase or not.

This is about JTA transaction and the context is propagate with WFTC. When the EJB remoting is used then it's always the WFTC as the middle layer which processes the propagation. The WFTC "simluates" the behaviour of the standard XA resource. In other scenarios (e.g. when 2PC is used) the tooling works fine. As WFTC works as the usual XAResource the :recover is capable to switch the transaction back to the prepared and later commit.

In case of this scenario there is used the 1PC. Thew 1PC WFTC XAResource returns RMFAIL on commit[1] and changes the _heuristic flag on the XAResourceRecord. This is probably the place and difference that the standard handling does not consider the fact there could be reason to :recover the 1PC.
But it's not accurate that there is no logging for 1PC. In case of failure the 1PC record is saved[2].

What happens here is that 1PC fails, the transaction outcome is logged in the object store with the heuristic failure. The :recover does not touch such record as the heuristic failure updates not only the BasicAction log record but the XAResourceRecord as well (with the mentioned _heuristic flag).

As I'm going though the code again and again I think the possible, and maybe better, approach would be to integrate from the WFTC side in way that the 1PC error code does not mean the _heuristic flag to be filled. If the 1PC record is saved but the _heuristic flag is not tourched then the :recover functionality could work. I think if the WFTC throws a specific XAException.errorCode then the handling goes under default branch[3]. That would bean the WFTC would be informing the TM that "commit retry" could be replied. Which is normally not expected for 1PC. But WFTC saves data at the client side and it does have such ability.

What do you think to integrate the 1PC WFTC with specific error code?


o.

Mark Little

unread,
Jun 22, 2020, 4:38:16 AM6/22/20
to Michael Musgrove, narayana-users
Is this work driven by a customer need?

Mark.


--
You received this message because you are subscribed to the Google Groups "narayana-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to narayana-user...@googlegroups.com.

Mark Little

unread,
Jun 22, 2020, 4:39:20 AM6/22/20
to Ondra Chaloupka, narayana-users
Have you looked at how JTS interposition works when there’s a coordinator, a single remote interposed resource and then many resources registered locally to it?

Mark.


--
You received this message because you are subscribed to the Google Groups "narayana-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to narayana-user...@googlegroups.com.

Ondra Chaloupka

unread,
Jun 22, 2020, 5:40:21 AM6/22/20