Let's talk about events...

[mynkow]

Consider these 3 ARs. User, Group, Activity. Some of the commands are:

cmd1 = CreateUser(name,email);

cmd2 = AddUserToGroup(userId, groupId);

When

      user is added to group

Then

      create user joined to group activity. Activity is something like notification, but users can interact with it like commenting, liking, etc.

When

      new user is created

Then

      automatically add that user to Group A and Group B

----------------------------------------------------------------------------------------

FLOW_1:

1) We send cmd2 

2) Command Handler handles it, Loads the Group AR and adds the new member. The Group AR raises async event like UserJoinedGroup

The event is not part of the transaction (eventual consistence).

3) Commit the transaction, write the UserJoinedGroup to the event store, whatever.

4) At some point of time UserJoinedGroup is handled in DomainEventHandler and the new Activity is created

Well, now I want to replay the events from the ES. When the UserJoinedGroup is replayed it will create a new activity. May be this is correct

but not in all cases. What if we have a handler which sends and email or does something else that we do not want to be executed when events

are replayed. How do you handle this?

FLOW_2:

1) We send cmd1 

2) Command Handler handles it, Create the new user. The User AR raises async event like UserCreated. 

UserCreatedDomainEventHandler should add the new user to Group A and Group B but without raising UserJoinedGroup.

How? Should I publish a new command to reuse the code in the command handler? Probably not because it will become too heavy and also 

the cmd2 handler will raise the UserJoinedGroup event. So May be I should create a new overload method in Group AR which does the same thing except

raising the event?!?

OK, what do you think? How you actually do such things?

[Dmitry Aleksandrov]

Looks like you should use Sagas.

FLOW1:

Sagas are starting by some event (for example  UserJoinedGroup ) and can send another command (CreateActivityCommand) to the system.

In the Saga you can detect is it new event or just event replayed for event log to handle it correctly.

FLOW2:

Saga should be started by UserCreated and will send 2 commands JoinUserToTheGroupCommand 

Something like that. 

Please correct my if i'm wrong (I'm still new in these things). 

Regards,

Dmitry

2012/2/18 mynkow <myn...@gmail.com>

--
Dmitry

[mynkow]

hmm, I do not think this is a saga. IMO, in most cases saga is a long running process between 2 systems or bounded contexts. The cases in my first post are so simple and there is no need to complicate the things, no?

[Shane C]

I tend to follow the saga view given in this blog post http://blog.jonathanoliver.com/2010/09/cqrs-sagas-with-event-sourcing-part-i-of-ii/ which says that we use a saga when 'the “transaction”, business process, or activity spans more than one message.'  and this example seems to follow that..

[Nuno Lopes]

FLOW_1:

1) We send cmd2 

2) Command Handler handles it, Loads the Group AR and adds the new member. The Group AR raises async event like UserJoinedGroup

The event is not part of the transaction (eventual consistence).

3) Commit the transaction, write the UserJoinedGroup to the event store, whatever.

4) At some point of time UserJoinedGroup is handled in DomainEventHandler and the new Activity is created

Well, now I want to replay the events from the ES. When the UserJoinedGroup is replayed it will create a new activity. May be this is correct

but not in all cases. What if we have a handler which sends and email or does something else that we do not want to be executed when events

are replayed. How do you handle this?

Events are published after the transaction has been completed, not before. The transaction is only completed after the object is successfully added to the repository. In other words, after changes are saved in the repository.

On flow 2 I don't understand what you want to do. You want to perform an operation against an object without publishing / rising an event?

Cheers,

Nuno

[mynkow]

about flow1: How to remove events that should not be replayed, like generating an activity? Is anyone doing ES or we are just talking?

flow2: There is one common operation. The operation is AddUserToGroup. This operation is invoked by different parts of the application. Part 1 should also raise event, but part 2 should not. Kepp this in mind when you read the flow again

Best regards

[Dmitry Aleksandrov]

flow1 - I'm doing event replay in following way:
1) send EventReplaingStarted event
2) all handlers/sagas knows that no external communication should be
performed until EventsReplayed occurs
3) replays events
4) send  EventsReplayed event

Works fine for me.
Is there any thoughts about pros/cons of following way?

Cheers

2012/2/22 mynkow <myn...@gmail.com>

>
> about flow1: How to remove events that should not be replayed, like generating an activity? Is anyone doing ES or we are just talking?
>
> flow2: There is one common operation. The operation is AddUserToGroup. This operation is invoked by different parts of the application. Part 1 should also raise event, but part 2 should not. Kepp this in mind when you read the flow again
>
> Best regards

--
Dmitry

[mynkow]

10x for the reply Aleksandrov. It makes sense. Unfortunately I have no real experience to discuss pros/cons about this workflow. Let's hope someone else will join this discussion.

best regards, mynkow

[João Bragança]

But now your sagas / services have to know about this. IMO it would be
much simpler if the replay code just new()'ed up the handlers it wants
to pump messages through. Use a convention, like a marker interface /
common base class / attribute.

[Greg Young]

Why not just during replay not hook Saga's publish to anything? :) /he publishes into /dev/null then hook him up after replay and he is at right state.

--
Le doute n'est pas une condition agréable, mais la certitude est absurde.

[Rinat Abdullin]

Dmitry,

That's the approach we were initially using for systems. However, there is a problem that shows up as system grows bigger.

Sending events one by one through the system can take quite a bit some time due to latency, plus all this switching between operation modes in handlers adds complexity.

So what we are doing instead now - processing all events as batch during the start-up against in-memory storage (if needed; only changed projections are affected). Then, changes are swapped with the real persistent store.

Best,

Rinat

[Dmitry Aleksandrov]

Thanks all of you guys for the suggestions!

Rinat, just to clarify - as I understand you do following:
1) read events from persistent store
2) switch infrastructure to use temporary in-memory persistent store
3) replay all events readed from persistent store
4) switch infrastructure back to use normal store

Could you clarify that my understanding is correct?

BTW, I have said that you have used start/stop events to identify
replaing events priviously. Do you remeber/know approx. amount of
events when you run into perfomance problems with such events
replaying approach.

Many thanks,
Dmitry

2012/2/24 Rinat Abdullin <rinat.a...@gmail.com>:

--
Dmitry

[Dmitry Aleksandrov]

> BTW, I have said that you have used start/stop events to identify
> replaing events priviously. Do you remeber/know approx. amount of
> events when you run into perfomance problems with such events
> replaying approach.

I mean "BTW, *You* have said that you have used start/stop events to identify
replaing events priviously."

[Greg Young]

It depends on the projections when they get to be too big ...

As an example the inventoryitemcounts projection in Simple.CQRS only uses a single partition (aggregate) within the event stream. As such it can be trivially map/reduced.

[Rinat Abdullin]

A few clarifications. Approaches have evolved a bit since then

1) amount of events is roughly 70k-300k in various systems

2) projections and views are changing quite often (since it is too convenient for evolving UI)

3) manually managing projections is tedious.

So what we are doing right now:

1) when server starts up (before starting to process messages), it checks projections for changes (via Mono.Cecil) if any of the projections have changed - they are selected for replay

2) selected projections are wired against in-memory store and events are streamed through (no message buses, just direct invocations).

3) after the replay is completed - in-memory store contents are written over the persistent store 

4) server continues with normal operation

If needed, this sequence could be improved to have zero downtime for both server and UI (with routing swap), making replay speeds irrelevant. But for us spending 10 minutes on replay (during which old views are served) and then 2 minutes on overwrite is good enough.

Hope this helps.

Best,

Rinat

[Nuno Lopes]

Hi,

about flow1: How to remove events that should not be replayed, like generating an activity? Is anyone doing ES or we are just talking?

Maybe we are just talking about :)

You don't replay events on an Aggregate in Greg's approach. You reapply the events on an Aggregate. Events reapplied don't get published, so published stuff is handled in consequence, hence no new commands are issued in consequence. In other words NO SIDE EFFECTS. Having said this you mainly reapply events when you need to reconstruct something, say a View or something.

Maybe if you explain better what you mean by replaying events I can give ya a more concise response.

Cheers,

Nuno

[Nuno Lopes]

Furthermore,

The only constraints you have when reapplying:

1) The target structure is a projection of a single events stream of a single Aggregate you need it in the same order the events where declared.
2) The target structure is a projection of multiple events streams each of a single Aggregate, the operation needs to be commutative. Meaning reapplying events/facts of Aggregate A followed by Aggregate B needs to be equal to reapplying events/facts of Aggregate B followed by Aggregate A.
3) If the case is not 1) or 2) probably you are missing an Aggregate in between that joins business activities performed by both A and B.

Cheers,

Nuno

[Nuno Lopes]

Hi,

2) selected projections are wired against in-memory store and events are streamed through (no message buses, just direct invocations).

That is nice. You can also go directly to the events stream of the Aggregate in question and iterate over (considering you database is event centric). So there are no messages flying around. Simple straight Data Transformation's.

Cheers,

Nuno