How to involve neo4j in XA Transaction / two phases commit

[Martin Feng]

Hi

I asked this question in stackoverflow and get no answer. It seems it needs some knowledge of neo4j kernel code to answer. So maybe it's better to ask here.

I am using neo4j along with other data sources (e.g., MySQL). I checked neo4j kernel source, although inside neo4j, neo4j storage and lucene index are wrapped as XA resource and managed by 2 phases commit, but this XA feature is not exposed to outside of neo4j.

Any idea of how to achieve 2 phases commit when using neo4j along with other data sources? What I am doing now is to modify neo4j source code mainly in

org.neo4j.kernel.TopLevelTransaction.java
org.neo4j.kernel.impl.transaction.TransactionImpl.java
org.neo4j.kernel.impl.transaction.TxManager.java

To break neo4j transaction commit into 2 phases and expose the function through TopLevelTransaction.

The problems of doing this:

1. I need to check neo4j code each time neo4j is updated.
2. I need to lock the TxManager during the 2 phases which might influence neo4j performance.

Any better ideas?

The link on stackoverflow is: http://stackoverflow.com/questions/19486184/how-to-involve-neo4j-in-xa-transaction-two-phases-commit

[Michael Hunger]

Hi Martin,

Does this help you?

src/test/java/org/neo4j/kernel/impl/transaction/JOTMTransactionManager.java

src/test/java/org/neo4j/kernel/impl/transaction/UseJOTMAsTxManagerIT.java

You have to integrate with an external XA-TM like JOTM, Atomikos or one of the AppServer's.

Michael

--
You received this message because you are subscribed to the Google Groups "Neo4j" group.
To unsubscribe from this group and stop receiving emails from it, send an email to neo4j+un...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Martin Feng]

Thanks Michael

Please correct me if I am wrong. 

I assume it means the code managing the XA transactions need to be run in the same process with neo4j. In case I am using neo4j as an independent server, this code need to be inside neo4j unmanaged server extension. In this case I cannot add other computers to share the XA transaction management workload.

More unfortunately, I am using SAP HANA, which doesn't seem to support XA transaction. Luckily there is only one XA incompatible data source in my system.

So for me, it's better to expose the 2 phases commit function through rest API. Then I can

1. Do the first phase commit at neo4j

2. Commit to HANA

3. Rollback or do the second commit at neo4j according to the result of step 2.

Moreover, it means I can add multiple application servers to manager these transactions.

Here is what I have done:

I noticed that transactions are saved in thread local storage, so I use dedicate threads (each mapped to a certain transaction id known to application server code) to run db operations. 

I broke the commit into 2 phases, based on the voting logic in org.neo4j.kernel.impl.transaction.doCommit(). I changed the synchronized in org.neo4j.kernel.impl.transaction.TxManager to a lock so I can hold the lock between the 2 phases. Finally, I add timeout logic to rollback and free that lock. :-)

I wonder have neo4j run into similar requirement before (I mean need to expose the 2 phases commit out of neo4j server)?

[Michael Hunger]

Usually the XA-TM's also work across the wire with appropriate resources, e.g. an XA-enabled jdbc driver.

Am 24.10.2013 um 06:48 schrieb Martin Feng <fen...@gmail.com>:

Thanks Michael

Please correct me if I am wrong. 

I assume it means the code managing the XA transactions need to be run in the same process with neo4j. In case I am using neo4j as an independent server, this code need to be inside neo4j unmanaged server extension. In this case I cannot add other computers to share the XA transaction management workload.

More unfortunately, I am using SAP HANA, which doesn't seem to support XA transaction. Luckily there is only one XA incompatible data source in my system.

So for me, it's better to expose the 2 phases commit function through rest API. Then I can

1. Do the first phase commit at neo4j

2. Commit to HANA

3. Rollback or do the second commit at neo4j according to the result of step 2.

makes sense

the only drawback is, if anything fails in neo it was already committed to HANA and you have to compensate for that.

What is your use-case with this architecture?

[Martin Feng]

Hi Michael,

With some privilege to add extension build by C++ into HANA SQL Script engine, we are using HANA mainly as parallel computing platform processing data stored in memory .

We have neo4j save relationships among kinds of data points, and the time series values of the data points are saved in HANA. So we need to make sure there are corresponding time series values for data points stored in neo4j.

In this case, even if I am using a XA compatible database instead of HANA, as long as neo4j is used as a standalone server (I assume neo4j doesn't expose XA interface when run as stand alone server), I need to expose the 2 phases commit of neo4j through REST API.

[Martin Feng]

Hi Michael

It has been more than 1 year since I modified neo4j 1.9.4 kernel code to expose the 2 phases commit through REST API. It runs very well. I noticed that the transaction related code in 2.x changed quite a bit. So to upgrade to neo4j 2.x, I have to read and modify the code again. So my question is:

1. Will neo4j consider expose the 2 phases commit through REST API?

2. If neo4j won't do it, is there any chance I could submit my changes to neo4j source?

Thanks.

On Thursday, October 24, 2013 12:51:24 PM UTC+8, Michael Hunger wrote: