Using TransactionEventHandler in HA Environment

[Paul Jackson]

I use a TransactionEventHandler to keep track of what is going on in a graph, however, in a high-availability environment, updates are made "under the covers" and so in this way, I lose track of the graph status when other instances receive writes.

It seems my options are:

1) Only write to one instance (should be the master for performance reasons).

2) Aggregate the updates from all the distributed instances and track centrally.

3) Publish the updates to a message bus and let any one interested in the updates subscribe to the bus.

I'm curious what people think of these choices. Are there others? I am leaning away from option 1 because keeping up with which instance is master sounds difficult and brittle. Is this a common problem?

Thanks,

Paul Jackson

[Mattias Persson]

Hi Paul,

it's a known issue at least. Would it be ok if you registered the same transaction event handlers on all instances and transactions pulled from the master would also trigger the event handlers on each slave? Maybe that was your expectation to begin with?

Best,
Mattias

2012/5/8 Paul Jackson <paul_j...@g1.com>

--
Mattias Persson, [mat...@neotechnology.com]
Hacker, Neo Technology
www.neotechnology.com

[Paul Jackson]

Hi Mattias,

I was hoping to see the events fire on all instances (master and slave) regardless of whether the transaction was committed on a master or slave. The problem I see with this, however, is the race condition that I expect would occur when a new (out-of-date) slave is brought on line and is updated before the user can register the transaction handler. Seems such a scenario would require passing the handler in as a constructor argument.

So it seems I'm left with coordinating the handlers to get a complete view of the activity.

Thanks,

-Paul

On Wednesday, May 9, 2012 3:33:00 AM UTC-4, Mattias Persson wrote:

Hi Paul,

it's a known issue at least. Would it be ok if you registered the same transaction event handlers on all instances and transactions pulled from the master would also trigger the event handlers on each slave? Maybe that was your expectation to begin with?

Best,
Mattias

2012/5/8 Paul Jackson

I use a TransactionEventHandler to keep track of what is going on in a graph, however, in a high-availability environment, updates are made "under the covers" and so in this way, I lose track of the graph status when other instances receive writes.

[Mattias Persson]

2012/5/9 Paul Jackson <paul_j...@g1.com>

Hi Mattias,

I was hoping to see the events fire on all instances (master and slave) regardless of whether the transaction was committed on a master or slave. The problem I see with this, however, is the race condition that I expect would occur when a new (out-of-date) slave is brought on line and is updated before the user can register the transaction handler. Seems such a scenario would require passing the handler in as a constructor argument.

True, which of course is possible to add. I'd say the same problem exists in the single instance scenario where a recovery is performed before the constructor returns.

So it seems I'm left with coordinating the handlers to get a complete view of the activity.

Thanks,

-Paul

On Wednesday, May 9, 2012 3:33:00 AM UTC-4, Mattias Persson wrote:

Hi Paul,

it's a known issue at least. Would it be ok if you registered the same transaction event handlers on all instances and transactions pulled from the master would also trigger the event handlers on each slave? Maybe that was your expectation to begin with?

Best,
Mattias

2012/5/8 Paul Jackson

I use a TransactionEventHandler to keep track of what is going on in a graph, however, in a high-availability environment, updates are made "under the covers" and so in this way, I lose track of the graph status when other instances receive writes.

It seems my options are:

1) Only write to one instance (should be the master for performance reasons).

2) Aggregate the updates from all the distributed instances and track centrally.

3) Publish the updates to a message bus and let any one interested in the updates subscribe to the bus.

I'm curious what people think of these choices. Are there others? I am leaning away from option 1 because keeping up with which instance is master sounds difficult and brittle. Is this a common problem?

Thanks,

Paul Jackson

[Paul Jackson]

OK, now I am confused.

My understanding is that for a graph that was improperly shut down, the recovery process recovers data that was committed. It seems that for these transactions, the events should have already been fired in the process that wrote them, so if the system fired them during recovery, the events would be thrown a second time. Perhaps it depends upon the exact timing of when a record is considered committed. My guess would be that it occurs between the beforeCommit and AfterCommit events are raised. If this is correct, are these the two risks:

1) If the process dies between the time that beforeCommit is called and the transaction is committed, then the beforeCommit listeners receive a false event

2)  If the process dies between the time that the transaction is committed and afterCommit is called and, then the afterCommit is never raised

I think you must be referring to a solution to the second risk - an ideal solution would fire the afterCommit events during recovery. Is this correct? In my scenario, it would be adequate to have some way of detecting that a recovery occurred, in which case I could run a consistency check during my initialization process, but the ideal solution would be... ideal.

Thanks,
-Paul

[Mattias Persson]

2012/5/9 Paul Jackson <paul_j...@g1.com>

OK, now I am confused.

My understanding is that for a graph that was improperly shut down, the recovery process recovers data that was committed. It seems that for these transactions, the events should have already been fired in the process that wrote them, so if the system fired them during recovery, the events would be thrown a second time. Perhaps it depends upon the exact timing of when a record is considered committed. My guess would be that it occurs between the beforeCommit and AfterCommit events are raised. If this is correct, are these the two risks:

1) If the process dies between the time that beforeCommit is called and the transaction is committed, then the beforeCommit listeners receive a false event

2)  If the process dies between the time that the transaction is committed and afterCommit is called and, then the afterCommit is never raised

I think you must be referring to a solution to the second risk - an ideal solution would fire the afterCommit events during recovery. Is this correct? In my scenario, it would be adequate to have some way of detecting that a recovery occurred, in which case I could run a consistency check during my initialization process, but the ideal solution would be... ideal.

Yup, you're absolutely correct. During recovery, transactions marked with COMMIT record but not DONE record aren't guaranteed to have reached the handlers with afterCommit and those could be sent to the handlers on recovery, but that isn't happening currently, and would also require handlers to be registered before recovery of course. Registration of handlers shouldn't sit on GraphDatabaseService, but instead on GraphDatabaseFactory imho. This and/or notification about recovery performed and completed could make the transaction event handling more solid.

Thanks,
-Paul

On Wednesday, May 9, 2012 9:12:30 AM UTC-4, Mattias Persson wrote:

True, which of course is possible to add. I'd say the same problem exists in the single instance scenario where a recovery is performed before the constructor returns.