Integration Events in Event Sourced System?

[Ratan Parai]

Hi,

We are developing a ES based system and this is our first one. We are developing our own event store using https://github.com/gregoryyoung/m-r/ as reference. We are using Cosmos DB as event store's persistent storage, partition key of Cosmos DB as stream id and azure function which listen to Cosmos DB change-feed and publish the events to Service Bus topic. The query service listen to events from the topic's subscription and populate the read model. 

Now we have a User aggregate root. when a user create a registration request (UserRegistrationRequested event) we need a separate service that will perform some external validation, for example, validate mobile number and email address. My initial though was the Validation service will listen to UserRegistrationRequested event from ServiceBus (The same topic/exchange we are using for generating read model) and publish UserMobileNumberValidated and UserEmailAddressValidated event. Which will be then picked up by the Registration service's event handler and call the user.ValidateMobileNumber() and user.ValidateEmailAddress() to create aggregate specific event and store them. 

But then I am loosing the capability to regenerate read model by replaying all events from the beginning in the topic/exchange. Because I don't want to trigger all those validation process. 

Also another issue, when the UserActivated event happens I want to send email to the user's email address letting the user know you can now login. But the payload of UserActivated event does not have email address (aggregate Id only). So the Email service can't just listen to the UserActivated event and send email! 

So my current thought is, okay I need use two different kind of event, The event store event (we currently calling it DomainEvent which is still a big question to me, should we simply call it Event or something else?) and Integration Event. If we do that, from where I should publish the integration events? 

We can only perform transaction in single partition insert in the same container (collection). So we can not write into two different container, event store and integration event container (outbox pattern) at the same transaction.  So we can not ensure integration events get published no matter what. 

In my current state I am feeling I am stuck and can't figure out what to do. Can anyone please confirm if I am on the right track or not and help me figure out how to implement integration events in ES based system? Also any reference of how to implement long running process (saga) with external dependency like manual validation and timeout in event sourced system?

[Chris Richardson]

Ratan,

The message broker/Event Store should enable the read model and the validation service to be separate/independent event consumers

You can then rebuild the read model by replaying events for just the read model.

The validation service would not be affected.

For example, the Kafka-flavor of Eventuate has separate Kafka consumer groups for each of those.

You could then replay events by resetting the consumer offsets for that consumer group.

Chris

-- 

Learn microservices - http://learn.microservices.io

Microservices application platform http://eventuate.io

Consulting and training http://chrisrichardson.net

--
You received this message because you are subscribed to the Google Groups "DDD/CQRS" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dddcqrs+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dddcqrs/40b52434-b259-419d-a221-7512378d2b87%40googlegroups.com.

[Alexander Tkacuk]

First of all.

ReadModels should only be rebuilt by events that mutate the application state.

It is very important to distinguish between commands and events.

Well

As I understood it the validation of the email and PhoneNumber is not completed in the UserRegistrationRequested event. And that's just the state stored in the Entity or ValueObject. "EmaleIsNotValidatedYet". That's exactly what can be mapped in the ReadModel as well. Only after the validation "Sync / async" will another event change this state "EmailIsCheckedAndIsNotValid" or "EmailIsCheckedAndIsValid" These are just examples. The names should be based on the ubiquitous language

[Ratan Sunder Parai]

Chris,

Thanks a lot for your response.

if I understand you correctly, I should use the event sourcing events
for both generating read models and integration between services rather
than separate integration events. But use two different kind of
consumers/subscriptions.

So, for example: if another service need to know "email address" for an
aggregate, the service need to listen to and keep it's own storage for
the email address added or modified event and when it receive an
"UserActivatedEvent" with payload { "id" : 123 } , then it have to look
for email address in it's own storage for the userId and send email to
that user's email.

Ratan

P.S. Sorry for late response. I was on vacation.

On 8/8/2019 3:45 PM, Chris Richardson wrote:
> Ratan,
>
> The message broker/Event Store should enable the read model and the
> validation service to be separate/independent event consumers
> You can then rebuild the read model by replaying events for just the
> read model.
> The validation service would not be affected.
>
> For example, the Kafka-flavor of Eventuate has separate Kafka consumer
> groups for each of those.
> You could then replay events by resetting the consumer offsets for that
> consumer group.
>
> Chris
>
> -- 
> Learn microservices - http://learn.microservices.io

> <http://learn.microservices.io/>

> Microservices application platform http://eventuate.io

> <http://eventuate.io/>

> Consulting and training http://chrisrichardson.net

> <http://chrisrichardson.net/>

> <mailto:dddcqrs+u...@googlegroups.com>.

> To view this discussion on the web visit
> https://groups.google.com/d/msgid/dddcqrs/40b52434-b259-419d-a221-7512378d2b87%40googlegroups.com

> <https://groups.google.com/d/msgid/dddcqrs/40b52434-b259-419d-a221-7512378d2b87%40googlegroups.com?utm_medium=email&utm_source=footer>.

>
>
>
> --
> You received this message because you are subscribed to the Google
> Groups "DDD/CQRS" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to dddcqrs+u...@googlegroups.com

> <mailto:dddcqrs+u...@googlegroups.com>.

> To view this discussion on the web visit

> https://groups.google.com/d/msgid/dddcqrs/CAA9exRvyEG15r8imfwD7dUJEYo-5bFdiaEwKC%2BMZqgsBjPb63g%40mail.gmail.com
> <https://groups.google.com/d/msgid/dddcqrs/CAA9exRvyEG15r8imfwD7dUJEYo-5bFdiaEwKC%2BMZqgsBjPb63g%40mail.gmail.com?utm_medium=email&utm_source=footer>.

[Ratan Sunder Parai]

Tkacuk,

There is manual and validation from multiple third parties before the
user's registration can be completed and be called a user in our system.
Also those validation will be triggered if any user change some
information after the registration (depending on field he/she change).

Those validations are long running process. that's why we have designed
three different services for this,

1. Registration/UserApplication service (request for membership)
2. Membership service (users who are already a member of our system)
3. Validation service (validate only manual and third party validation /
long running validation)

Those services need to exchange messages between them to complete
business workflow like registration to the service.


Ratan

> --
> You received this message because you are subscribed to the Google
> Groups "DDD/CQRS" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to dddcqrs+u...@googlegroups.com
> <mailto:dddcqrs+u...@googlegroups.com>.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/dddcqrs/5bc6f60d-4d04-4295-b85b-310cc5de64d9%40googlegroups.com
> <https://groups.google.com/d/msgid/dddcqrs/5bc6f60d-4d04-4295-b85b-310cc5de64d9%40googlegroups.com?utm_medium=email&utm_source=footer>.

[Tomas Ptacnik]

Hi Parai,

regarding read models and what other services can subscribe to:

It's basically the same thing as another service might need to build its own read model from events of another service. 

As for example in the case where you want to send an e-mail and you want to avoid a synchronous call to "user service". You need to listen some user events and build a local user read model.

Watch out that for sending e-mails, you need not only an email, but also a name and a language of the user.

Best,

Tomas

To unsubscribe from this group and stop receiving emails from it, send an email to dddcqrs+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dddcqrs/5842c426-c2f9-1207-7696-379ea205d46c%40gmail.com.