MT3 + RMQ publishing domain event after saving aggregate

[C. J. Russell]

We are experimenting with using MT3 on RMQ for publishing Domain events to other processes.  This has worked well so far, however once we started looking at failure scenarios, we got a little discouraged when trying to assure that an event is only published on a successful commit of data to our data store, and on that the event is always published (at least eventually) upon a successful commit of the data.

The basic scenarios that we are looking at are as follows:

Consume MT Command/Event

   ** Execute Business Logic **

   Start DB Transaction

      Persist Entity to Database

   Commit Database Changes

   Publish Domain Event Fails (no message sent and no way to recover)

Consume MT Command/Event

   ** Execute Business Logic **

   Start DB Transaction

      Persist Entity to Database

      Publish Domain Event

   Commit Database Changes Fails (message sent, but Entity changes not persisted)

Looking through discussions on this group and elsewhere, the best solution outside of DTC that I have seen so far is something similar to the Outbox and DeDup patterns described here:  http://docs.particular.net/nservicebus/outbox/

Consume MT Command/Event

   Check to see if Command/Event has already been processed and events dispatched, if so ignore

   ** Execute Business Logic **

   Start DB Transaction

      Persist Entity to Database

      Persist Domain Events to Database

   Commit Database Changes

   Dispatch Events

   Update Domain Events in Database to acknowledge dispatch

Just curious if we are missing something, and this is already handled somehow in MT3, or if this has been implemented somewhere already?  Are these failure scenarios uncommon enough that the loss of domain events is considered acceptable?  Any response is much appreciated!

Thank you,

C. J.

[Chris Patterson]

So in your example, what happens in the case of:

Consume MT Command/Event

   Check to see if Command/Event has already been processed and events dispatched, if so ignore

   ** Execute Business Logic **

   Start DB Transaction

      Persist Entity to Database

      Persist Domain Events to Database

   Commit Database Changes

   Dispatch Events

   Update Domain Events in Database to acknowledge dispatch -- FAILS

If the last step fails, is the message reprocessed, and the events published again? Exactly once delivery in a system is the hardest problem to solve. It's much easier to either make the event consumers idempotent (so that events published more than once don't cause the system duplicate information) than it is to try and stall the entire system to provide exactly once delivery of a message.

We always use the model of:

Consume MT Command/Event

   ** Execute Business Logic **

   Start DB Transaction

      Persist Entity to Database

      Publish Domain Event

   Commit Database Changes Fails (message sent, but Entity changes not persisted)

And if the commit fails, the message is reprocessed, the database is updated, and the events are published again. The correlation information in the event allows an event consumer to ensure that any action taken is not duplicative.

It's also possible in the database to do a "flush" before a commit, ensuring the proper isolation levels are used, to determine in advance without failing the transaction if the transaction _might_ fail. We've done this in a few places using NHibernate. Try to insert, and if a duplicate key exception occurs, read and compare to the existing record in the event handler.

In fact, that's the default behavior for sagas in the upcoming release of MT3.

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-dis...@googlegroups.com.
To post to this group, send email to masstrans...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/masstransit-discuss/1c0efee2-ee21-440d-8974-9f6a899f5c47%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Brian Weeres]

Here is a simplistic explanation of what we do. We have a consumer base class that Consumes the message, starts transaction and then calls an abstract implementation method passing the message. The consumer base class has its own publish methods so when the derived consumer is publishing it doesn't actually publish to the bus but the event/command is put into a list. When the implementation method returns to the base class the tx is committed and if everything worked then the messages are published to the bus.  

[C. J. Russell]

In the scenario that you suggested, does the domain event that you published get rolled back?  If not, and it gets consumed, then the consumer of this event could be acting on information that was not committed to the database.  I can see where this would be acceptable behavior in a lot of cases, so just trying to make sure that my understanding is correct and once a message is published, there is no turning back.

In the scenario I presented, if the last step fails it would reprocess the message and republish the events (perhaps skipping Entity persistence), so the consumers would still have to be idempotent.  I think the difference between the two is that you have assured that the data changes are always persisted before the event is published, ensuring the consumer is acting on accurate information.

I think if we end up moving forward with MT3, we would likely use one of these approaches.  I appreciate the quick response!

Thank you,

C. J.

[C. J. Russell]

Thanks Brian,

In this case, if you couldn't reach RMQ after you committed the DB transaction, wouldn't you lose your published event, or do you compensate for exceptions on your Publish?  

Thank you,

C. J.

[Udi Dahan]

Ø  Commit Database Changes Fails (message sent, but Entity changes not persisted)

 

Chris – does that mean that you don’t use auto-increment DB identifiers when you’re inserting records (and including those IDs in your messages)?

 

Because if you did, and the transaction failed on persist, when you go to process the message a second time, you may get a different ID, and if you publish that, then it makes it impossible for the recipient of the messages to de-duplicate on their end (as the second message has a different entity ID in it than the first).

 

Thanks,

 

Udi

To view this discussion on the web visit https://groups.google.com/d/msgid/masstransit-discuss/CAHX4WYegdjO6PnTYWfzDn7OVEmSsUTM_X%3D8us9i%2B2suFW2HKTA%40mail.gmail.com.

[Udi Dahan]

Brian,

 

That’s pretty much the way we’ve done it in NServiceBus with our “outbox” feature (http://docs.particular.net/nservicebus/outbox/).

 

Do you use a persistent list for those outgoing messages?

 

Cheers,

 

Udi

--

You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-dis...@googlegroups.com.
To post to this group, send email to masstrans...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/masstransit-discuss/9567432b-46d0-4c55-bebb-ea313e03d42c%40googlegroups.com.

[Chris Patterson]

That's correct, we don't use auto-incrementing identifiers. We create an identifier (using NewId) on the outermost entry point to the system both for traceability and to reduce instances of duplicate transactions. These identifiers are always correlated to subsequent transactions by the tracking services.

In cases where we control the sending application (which may be outside our context boundary), we've ported NewId to the source platform (Java) and injected that into the source system. In fact, we even publish events to RabbitMQ from Java that are consumed by MassTransit to facilitate end-to-end tracking external to the NRT processing of transactions.

When we can't control the input, we've used some other approaches to recover from duplicates (preventing them can cause an intolerable system delay by having up front de-duplication -- storage systems will inevitably block reads), by identifying related messages using business identifiers and then correlating and determining the most recent message and ensuring that is the last observed message by the remainder of the system.

In most cases, the entire transaction state is maintained along the critical path of the message flow through the system (either on a routing slip via Courier, or just in the command messages themselves).

[Brian Weeres]

We persist all incoming and outgoing messages for idempotent and that logic is part of this transaction handling.If it's a simple consumer that does not need idempotent handling we have a class attribute that can be applied to a consumer to bypass that logic.

To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-discuss+unsub...@googlegroups.com.

[Brian Weeres]

In our scenario if the transaction gets rolled back the events are never published as they are only published to an internal list (that is persisted) during the transaction and then published to the bus if the transaction completes successfully. If the tx completed but the publish of the messages fails we still have the messages that were supposed to be published and can retry the publishes or take whatever action we need to. (If the publishes fail it is a big issue.)

[Brian Weeres]

Wow, looked at your link, that is almost exactly what we do. Confirms our approach I guess.

On Friday, August 21, 2015 at 7:17:49 AM UTC-5, Udi Dahan wrote:

To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-discuss+unsub...@googlegroups.com.

[Van Ly]

Hi Chris,

I think that the point here is not about the deduplication. It's about a message fired when the transaction was not committed. The transaction may be reprocessed, or may never be reprocessed anymore (perhaps bug in code which leading to missing validation, transaction cannot be committed, message moved to error queue, user tries another command with valid data). In this case, the system will act on a message that should be never fired.

I'm considering an ESB library and if I choose MT, I think that I have to implement this feature myself.

Regards,

Van Ly

[David Prothero]

This video illustrates exactly how difficult this problem would be to solve so that it had the kind of airtight quality you were looking for:

http://vimeo.com/111998645