CQRS with NServiceBus: How to process published events in independent transactions?

567 views
Skip to first unread message

Manuel Pallier

unread,
Aug 17, 2016, 3:05:15 PM8/17/16
to Particular Software
Helpful information to include
Product name: NServiceBus
Version: 5.2.14
Stacktrace: -
Description:

Hi,

I'm currently modifying an in-development application based on the CQRS and ES patterns to use NServiceBus (based on MSMQ). Right now NServiceBus is used for the communication between clients and the server, to send commands and publish events (e.g. for change notification). Next I'd like to replace the in-memory message bus on the server. This bus is used to publish the domain events (that have been saved in the event store) to the event handlers (e.g. de-normalizers).

Just replacing my in-memory bus with NServiceBus publish/subscribe is easy. But I'd also like to make all event handlers independent so that one can fail and all others still commit their changes. Example: The event 'CustomerCreatedEvent' is saved in the event store. Multiple classes handle the event. If 'CustomerEmailNotifier' throws an exception, 'CustomerReadModelGenerator' should still be executed (and commited). Later, 'CustomerEmailNotifier' should be retried, just like it would happen if it was the only handler of the event.

As far as I understand the documentation, this use case is not how NServiceBus works by default. Instead, all handlers are processed in a single transaction scope. But unless I misunderstand CQRS, domain event handlers should be independent of each other, right? So how do I implement this CQRS domain event publishing with NServiceBus?

Szymon Pobiega

unread,
Aug 17, 2016, 11:45:55 PM8/17/16
to Particular Software
Hi Manuel

You are right. In a single endpoint, all handlers for a given message type are executed in a single transaction context. But with NServiceBus nothing prevents you from using more than one endpoint. In fact, if you were to use a single endpoint/queue for all the denormalizers, you would not need NSB in the first place.

In your case you should probably Publish the domain event via NServiceBus. On the other side you would have a number of endpoints, each subscribed for these domain events. Each endpoint can host a number of handlers, based on their requirements e.g. if on the read side you have a document DB, a SQL DB and a SMTP server, even though you might have 10s of handlers, you might need only three endpoints. This way if SQL is down, handlers that don't use it can continue to work and you don't pay the cost of physically separate deployment for each and every handler.

Szymon

--
You received this message because you are subscribed to the Google Groups "Particular Software" group.
To unsubscribe from this group and stop receiving emails from it, send an email to particularsoftw...@googlegroups.com.
To post to this group, send email to particula...@googlegroups.com.
Visit this group at https://groups.google.com/group/particularsoftware.
To view this discussion on the web visit https://groups.google.com/d/msgid/particularsoftware/056cf5aa-bfa0-4532-a885-5f7bb4d6b856%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Manuel Pallier

unread,
Aug 18, 2016, 8:25:42 AM8/18/16
to Particular Software
Hi,

Thanks for your help. This sounds like a good solution. But how do I then manage my event subscriptions? For one endpoint I'd just subscribe to all implementations of my 'IEvent' interface. But when I then have multiple endpoints, I need to subscribe the correct events to the correct endpoints. Do I need to create different marker interfaces or are there better ways?


Am Donnerstag, 18. August 2016 05:45:55 UTC+2 schrieb Szymon Pobiega:
Hi Manuel

You are right. In a single endpoint, all handlers for a given message type are executed in a single transaction context. But with NServiceBus nothing prevents you from using more than one endpoint. In fact, if you were to use a single endpoint/queue for all the denormalizers, you would not need NSB in the first place.

In your case you should probably Publish the domain event via NServiceBus. On the other side you would have a number of endpoints, each subscribed for these domain events. Each endpoint can host a number of handlers, based on their requirements e.g. if on the read side you have a document DB, a SQL DB and a SMTP server, even though you might have 10s of handlers, you might need only three endpoints. This way if SQL is down, handlers that don't use it can continue to work and you don't pay the cost of physically separate deployment for each and every handler.

Szymon

śr., 17.08.2016 o 21:05 użytkownik Manuel Pallier <manuel....@beko.at> napisał:
Helpful information to include
Product name: NServiceBus
Version: 5.2.14
Stacktrace: -
Description:

Hi,

I'm currently modifying an in-development application based on the CQRS and ES patterns to use NServiceBus (based on MSMQ). Right now NServiceBus is used for the communication between clients and the server, to send commands and publish events (e.g. for change notification). Next I'd like to replace the in-memory message bus on the server. This bus is used to publish the domain events (that have been saved in the event store) to the event handlers (e.g. de-normalizers).

Just replacing my in-memory bus with NServiceBus publish/subscribe is easy. But I'd also like to make all event handlers independent so that one can fail and all others still commit their changes. Example: The event 'CustomerCreatedEvent' is saved in the event store. Multiple classes handle the event. If 'CustomerEmailNotifier' throws an exception, 'CustomerReadModelGenerator' should still be executed (and commited). Later, 'CustomerEmailNotifier' should be retried, just like it would happen if it was the only handler of the event.

As far as I understand the documentation, this use case is not how NServiceBus works by default. Instead, all handlers are processed in a single transaction scope. But unless I misunderstand CQRS, domain event handlers should be independent of each other, right? So how do I implement this CQRS domain event publishing with NServiceBus?

--
You received this message because you are subscribed to the Google Groups "Particular Software" group.
To unsubscribe from this group and stop receiving emails from it, send an email to particularsoftware+unsub...@googlegroups.com.

Szymon Pobiega

unread,
Aug 22, 2016, 2:29:51 AM8/22/16
to Particular Software
Hi

In most cases NServiceBus automatic subscriptions should handle everything. NSB automatically subscribes to event E when it finds a handler for it in the loaded assemblies and (in case of MSMQ) there is proper routing configuration in the MessageEndpointMappings section http://docs.particular.net/nservicebus/messaging/message-owner. So each of your subscriber endpoint should point to the publisher and the subscriptions will be automatically created on startup.

Szymon 

czw., 18.08.2016 o 14:25 użytkownik Manuel Pallier <manuel....@beko.at> napisał:
Hi,

Thanks for your help. This sounds like a good solution. But how do I then manage my event subscriptions? For one endpoint I'd just subscribe to all implementations of my 'IEvent' interface. But when I then have multiple endpoints, I need to subscribe the correct events to the correct endpoints. Do I need to create different marker interfaces or are there better ways?


Am Donnerstag, 18. August 2016 05:45:55 UTC+2 schrieb Szymon Pobiega:
Hi Manuel

You are right. In a single endpoint, all handlers for a given message type are executed in a single transaction context. But with NServiceBus nothing prevents you from using more than one endpoint. In fact, if you were to use a single endpoint/queue for all the denormalizers, you would not need NSB in the first place.

In your case you should probably Publish the domain event via NServiceBus. On the other side you would have a number of endpoints, each subscribed for these domain events. Each endpoint can host a number of handlers, based on their requirements e.g. if on the read side you have a document DB, a SQL DB and a SMTP server, even though you might have 10s of handlers, you might need only three endpoints. This way if SQL is down, handlers that don't use it can continue to work and you don't pay the cost of physically separate deployment for each and every handler.

Szymon

śr., 17.08.2016 o 21:05 użytkownik Manuel Pallier <manuel....@beko.at> napisał:
Helpful information to include
Product name: NServiceBus
Version: 5.2.14
Stacktrace: -
Description:

Hi,

I'm currently modifying an in-development application based on the CQRS and ES patterns to use NServiceBus (based on MSMQ). Right now NServiceBus is used for the communication between clients and the server, to send commands and publish events (e.g. for change notification). Next I'd like to replace the in-memory message bus on the server. This bus is used to publish the domain events (that have been saved in the event store) to the event handlers (e.g. de-normalizers).

Just replacing my in-memory bus with NServiceBus publish/subscribe is easy. But I'd also like to make all event handlers independent so that one can fail and all others still commit their changes. Example: The event 'CustomerCreatedEvent' is saved in the event store. Multiple classes handle the event. If 'CustomerEmailNotifier' throws an exception, 'CustomerReadModelGenerator' should still be executed (and commited). Later, 'CustomerEmailNotifier' should be retried, just like it would happen if it was the only handler of the event.

As far as I understand the documentation, this use case is not how NServiceBus works by default. Instead, all handlers are processed in a single transaction scope. But unless I misunderstand CQRS, domain event handlers should be independent of each other, right? So how do I implement this CQRS domain event publishing with NServiceBus?

--
You received this message because you are subscribed to the Google Groups "Particular Software" group.
To unsubscribe from this group and stop receiving emails from it, send an email to particularsoftw...@googlegroups.com.
--
You received this message because you are subscribed to the Google Groups "Particular Software" group.
To unsubscribe from this group and stop receiving emails from it, send an email to particularsoftw...@googlegroups.com.

To post to this group, send email to particula...@googlegroups.com.
Visit this group at https://groups.google.com/group/particularsoftware.

Manuel Pallier

unread,
Aug 22, 2016, 3:13:13 AM8/22/16
to Particular Software
Hi,

Sorry, but I still don't understand that. Could you please give me some minimal configuration example for the following situation:

A server that handles two things:
1. Receiving commands from clients and publishing events to clients (I assume that can be done with one endpoint?). All that commands and events are defined in an assembly that is shared by client and server.
2. Publishing events to itself to multiple endpoints for independent processing. Those events are defined in a different assembly that only the server uses.


Am Montag, 22. August 2016 08:29:51 UTC+2 schrieb Szymon Pobiega:
Hi

In most cases NServiceBus automatic subscriptions should handle everything. NSB automatically subscribes to event E when it finds a handler for it in the loaded assemblies and (in case of MSMQ) there is proper routing configuration in the MessageEndpointMappings section http://docs.particular.net/nservicebus/messaging/message-owner. So each of your subscriber endpoint should point to the publisher and the subscriptions will be automatically created on startup.

Szymon 

czw., 18.08.2016 o 14:25 użytkownik Manuel Pallier <manuel....@beko.at> napisał:
Hi,

Thanks for your help. This sounds like a good solution. But how do I then manage my event subscriptions? For one endpoint I'd just subscribe to all implementations of my 'IEvent' interface. But when I then have multiple endpoints, I need to subscribe the correct events to the correct endpoints. Do I need to create different marker interfaces or are there better ways?


Am Donnerstag, 18. August 2016 05:45:55 UTC+2 schrieb Szymon Pobiega:
Hi Manuel

You are right. In a single endpoint, all handlers for a given message type are executed in a single transaction context. But with NServiceBus nothing prevents you from using more than one endpoint. In fact, if you were to use a single endpoint/queue for all the denormalizers, you would not need NSB in the first place.

In your case you should probably Publish the domain event via NServiceBus. On the other side you would have a number of endpoints, each subscribed for these domain events. Each endpoint can host a number of handlers, based on their requirements e.g. if on the read side you have a document DB, a SQL DB and a SMTP server, even though you might have 10s of handlers, you might need only three endpoints. This way if SQL is down, handlers that don't use it can continue to work and you don't pay the cost of physically separate deployment for each and every handler.

Szymon

śr., 17.08.2016 o 21:05 użytkownik Manuel Pallier <manuel....@beko.at> napisał:
Helpful information to include
Product name: NServiceBus
Version: 5.2.14
Stacktrace: -
Description:

Hi,

I'm currently modifying an in-development application based on the CQRS and ES patterns to use NServiceBus (based on MSMQ). Right now NServiceBus is used for the communication between clients and the server, to send commands and publish events (e.g. for change notification). Next I'd like to replace the in-memory message bus on the server. This bus is used to publish the domain events (that have been saved in the event store) to the event handlers (e.g. de-normalizers).

Just replacing my in-memory bus with NServiceBus publish/subscribe is easy. But I'd also like to make all event handlers independent so that one can fail and all others still commit their changes. Example: The event 'CustomerCreatedEvent' is saved in the event store. Multiple classes handle the event. If 'CustomerEmailNotifier' throws an exception, 'CustomerReadModelGenerator' should still be executed (and commited). Later, 'CustomerEmailNotifier' should be retried, just like it would happen if it was the only handler of the event.

As far as I understand the documentation, this use case is not how NServiceBus works by default. Instead, all handlers are processed in a single transaction scope. But unless I misunderstand CQRS, domain event handlers should be independent of each other, right? So how do I implement this CQRS domain event publishing with NServiceBus?

--
You received this message because you are subscribed to the Google Groups "Particular Software" group.
To unsubscribe from this group and stop receiving emails from it, send an email to particularsoftware+unsub...@googlegroups.com.
To post to this group, send email to particula...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Particular Software" group.
To unsubscribe from this group and stop receiving emails from it, send an email to particularsoftware+unsub...@googlegroups.com.

Szymon Pobiega

unread,
Aug 22, 2016, 3:21:11 AM8/22/16
to Particular Software
Hi

Here's an example:

Clients:

<MessageEndpointMappings>
<add Assembly="Contracts" Endpoint="server@server_machine" />
</MessageEndpointMappings>

Server:

<MessageEndpointMappings>
<!-- No mappings required -->
</MessageEndpointMappings>

Event processors:

<MessageEndpointMappings>
<add Assembly="EventProcessorContracts" Endpoint="server@server_machine" />
</MessageEndpointMappings>

The server does not need to configure anything because it will be only replying to commands or publishing messages. The clients can safely route all the server API contract assemblies to the server. Each event processing endpoint needs to point to the server for the subscription messages to be routed properly. So although it is server who publishes these events, the routing config goes to the subscriber (the event processor). Each event processor will subscribe only these events for which it has handlers.

Hope that helps,
Szymon

To unsubscribe from this group and stop receiving emails from it, send an email to particularsoftw...@googlegroups.com.
To post to this group, send email to particula...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Particular Software" group.
To unsubscribe from this group and stop receiving emails from it, send an email to particularsoftw...@googlegroups.com.

To post to this group, send email to particula...@googlegroups.com.
Visit this group at https://groups.google.com/group/particularsoftware.
--
You received this message because you are subscribed to the Google Groups "Particular Software" group.
To unsubscribe from this group and stop receiving emails from it, send an email to particularsoftw...@googlegroups.com.

To post to this group, send email to particula...@googlegroups.com.
Visit this group at https://groups.google.com/group/particularsoftware.

Manuel Pallier

unread,
Aug 22, 2016, 3:29:40 AM8/22/16
to Particular Software
Hi,

Until now I planned to have all the server side processing, including the independent event processing, in a single windows service on the server. Isn't that possible?
To unsubscribe from this group and stop receiving emails from it, send an email to particularsoftware+unsub...@googlegroups.com.
To post to this group, send email to particula...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Particular Software" group.
To unsubscribe from this group and stop receiving emails from it, send an email to particularsoftware+unsub...@googlegroups.com.

To post to this group, send email to particula...@googlegroups.com.
Visit this group at https://groups.google.com/group/particularsoftware.

--
You received this message because you are subscribed to the Google Groups "Particular Software" group.
To unsubscribe from this group and stop receiving emails from it, send an email to particularsoftware+unsub...@googlegroups.com.

Szymon Pobiega

unread,
Aug 22, 2016, 3:37:19 AM8/22/16
to Particular Software
Hi

If you want a single host process, you have two options:
 * Go back to the single event processor and, instead of Publishing the internal events, use SendLocal to put it at the end of the queue. That has a disadvantage of using the same queue as clients are using.
 * Host the server endpoint and all the event processor endpoints in a single process

If you go with the second approach, you need to do two things
 * Provide the routing via code. http://docs.particular.net/nservicebus/messaging/message-owner shows how to pass the contents of the routing configuration section in code, for each endpoint
 * Configure the assembly scanning http://docs.particular.net/nservicebus/hosting/assembly-scanning#exclude-a-list-approach so that each even processor only load the handlers it should

Given the added complexity of this approach, I think it would be easier to have separate hosting processes. This also gives you an advantage of separate failure handling i.e. you don't want you server process to go down because SMTP server is not responding. 

Szymon

To unsubscribe from this group and stop receiving emails from it, send an email to particularsoftw...@googlegroups.com.
To post to this group, send email to particula...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Particular Software" group.
To unsubscribe from this group and stop receiving emails from it, send an email to particularsoftw...@googlegroups.com.

To post to this group, send email to particula...@googlegroups.com.
Visit this group at https://groups.google.com/group/particularsoftware.

--
You received this message because you are subscribed to the Google Groups "Particular Software" group.
To unsubscribe from this group and stop receiving emails from it, send an email to particularsoftw...@googlegroups.com.

To post to this group, send email to particula...@googlegroups.com.
Visit this group at https://groups.google.com/group/particularsoftware.
--
You received this message because you are subscribed to the Google Groups "Particular Software" group.
To unsubscribe from this group and stop receiving emails from it, send an email to particularsoftw...@googlegroups.com.

To post to this group, send email to particula...@googlegroups.com.
Visit this group at https://groups.google.com/group/particularsoftware.

Manuel Pallier

unread,
Aug 22, 2016, 4:20:59 AM8/22/16
to Particular Software
Hi,

Do I understand you correctly that you say I have to sperate my multiple endpoints with all their event handlers to their own assemblies? And then in each endpoint configuration I have to exclude all my endpoint assemblies except the current one?
But the domain events itself can stay in a shared assembly, right?

About failure handling: How could a not responding SMTP server cause a problem if all my endpoints are in a single process? Shouldn't all other endpoints continue working fine if one endpoint can't process its messages? Is there some sort of blocking or single threaded processing done?
To unsubscribe from this group and stop receiving emails from it, send an email to particularsoftware+unsub...@googlegroups.com.
To post to this group, send email to particula...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Particular Software" group.
To unsubscribe from this group and stop receiving emails from it, send an email to particularsoftware+unsub...@googlegroups.com.

To post to this group, send email to particula...@googlegroups.com.
Visit this group at https://groups.google.com/group/particularsoftware.

--
You received this message because you are subscribed to the Google Groups "Particular Software" group.
To unsubscribe from this group and stop receiving emails from it, send an email to particularsoftware+unsub...@googlegroups.com.

To post to this group, send email to particula...@googlegroups.com.
Visit this group at https://groups.google.com/group/particularsoftware.

--
You received this message because you are subscribed to the Google Groups "Particular Software" group.
To unsubscribe from this group and stop receiving emails from it, send an email to particularsoftware+unsub...@googlegroups.com.

Szymon Pobiega

unread,
Aug 22, 2016, 9:33:09 AM8/22/16
to Particular Software
Hi

Yes, that was what I meant.

Regarding the failure scenarios, assuming you have a handler that sends e-mail messages based on the events processed. It would be probably easier to make it crash when the SMTP server is not available rather than continue processing events and forward them all to the error queue for re-processing. This is the circuit breaker pattern https://particular.net/blog/protect-your-software-with-the-circuit-breaker-design-pattern we use in multiple places in NServiceBus.

Szymon

To unsubscribe from this group and stop receiving emails from it, send an email to particularsoftw...@googlegroups.com.
To post to this group, send email to particula...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Particular Software" group.
To unsubscribe from this group and stop receiving emails from it, send an email to particularsoftw...@googlegroups.com.

To post to this group, send email to particula...@googlegroups.com.
Visit this group at https://groups.google.com/group/particularsoftware.

--
You received this message because you are subscribed to the Google Groups "Particular Software" group.
To unsubscribe from this group and stop receiving emails from it, send an email to particularsoftw...@googlegroups.com.

To post to this group, send email to particula...@googlegroups.com.
Visit this group at https://groups.google.com/group/particularsoftware.

--
You received this message because you are subscribed to the Google Groups "Particular Software" group.
To unsubscribe from this group and stop receiving emails from it, send an email to particularsoftw...@googlegroups.com.

To post to this group, send email to particula...@googlegroups.com.
Visit this group at https://groups.google.com/group/particularsoftware.
--
You received this message because you are subscribed to the Google Groups "Particular Software" group.
To unsubscribe from this group and stop receiving emails from it, send an email to particularsoftw...@googlegroups.com.

To post to this group, send email to particula...@googlegroups.com.
Visit this group at https://groups.google.com/group/particularsoftware.

Manuel Pallier

unread,
Sep 6, 2016, 6:20:52 AM9/6/16
to Particular Software
Hi,

I didn't have the time to try your suggestion yet, but I've read the documentation you linked. In http://docs.particular.net/nservicebus/hosting/assembly-scanning#exclude-a-list-approach there is sample code that shows that not only assemblies, but also specific types can be excluded. Would it be possible to create my independent endpoints without having to create separate assemblies by using this API? Like excluding all types from specific namespaces for the endpoints?
To unsubscribe from this group and stop receiving emails from it, send an email to particularsoftware+unsub...@googlegroups.com.
To post to this group, send email to particula...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Particular Software" group.
To unsubscribe from this group and stop receiving emails from it, send an email to particularsoftware+unsub...@googlegroups.com.

To post to this group, send email to particula...@googlegroups.com.
Visit this group at https://groups.google.com/group/particularsoftware.

--
You received this message because you are subscribed to the Google Groups "Particular Software" group.
To unsubscribe from this group and stop receiving emails from it, send an email to particularsoftware+unsub...@googlegroups.com.

To post to this group, send email to particula...@googlegroups.com.
Visit this group at https://groups.google.com/group/particularsoftware.

--
You received this message because you are subscribed to the Google Groups "Particular Software" group.
To unsubscribe from this group and stop receiving emails from it, send an email to particularsoftware+unsub...@googlegroups.com.

To post to this group, send email to particula...@googlegroups.com.
Visit this group at https://groups.google.com/group/particularsoftware.

--
You received this message because you are subscribed to the Google Groups "Particular Software" group.
To unsubscribe from this group and stop receiving emails from it, send an email to particularsoftware+unsub...@googlegroups.com.

Justin Alexander

unread,
Sep 7, 2016, 9:18:29 AM9/7/16
to Particular Software
I'm genuinely curious. What is the perceived benefit of having all handlers and messages in the same assembly?

From strictly a modeling perspective, I would expect different namespaces and classes to be used in order to maintain the "single responsibility principle". Those responsibilities can then be spread across any number of different "end points". With that type of separation already in place within the code, spitting out different DLL's for specific namespaces seems simple and natural.

I've been using nServiceBus for nearly 10 years now. One of the things I've really come to appreciate is that its relatively easy to refactor our systems when the need for an additional end point arises. For example, I might start out with a single end point that handles 2 events and 2 commands. Over time, we learn that Command and Event A are much more time sensitive, so we decide to create a new end point that is dedicated to handling those types of "critical SLA" messages. 9 out of 10 times, this is just a matter of moving some code around to create the necessary DLL's and add a new end point project to host them.

Manuel Pallier

unread,
Sep 8, 2016, 3:20:37 AM9/8/16
to Particular Software
I was just thinking about some quick tests to ensure that this solution even works. Of course moving the handlers to separate assemblies is cleaner.
Reply all
Reply to author
Forward
0 new messages