Confusion about bulk commands and sagas

[Torni]

Hi,
I'm currently trying to understand a bit more about CQRS.

Let's assume I have a command A which then leads to 3 events B,C,D. This command can be executed individually or in a bulk operation, e.g. executing let's say 1000 commands in short time on the same aggregate root.
I know that the state of the domain model during this batch command is not necessary to be calculated and I like to reduce the load on the system by skipping the calculation and e.g. the read model updates.

I've seen that a bulk event concept is used here: http://cqrs.nu/Faq

So my idea was:
BulkCommand -> BulkStartEvent, Event B, C, D, ..., B, C, D, BulkEndEvent

I could then use a Saga to handle the Bulk Command, started by BulkStartEvent and ended by BulkEndEvent, and only update my models (both domain and read) after the End Event.

Now my question is: How do I prevent my normal Event Handlers for event B, C, D from updating the models anyway?

I have several ideas and I do not really like any of them:

1. Simply ignore the problem and let the event handlers do their job anyway,
   
2. Let my Event handlers do the updates anyway and have some nifty caching inplace to reduce the impact on the system. I do not really like this as this only moves the problem around a bit
3. Have a fully independent set of Batch Events, just for the batch command (e.g. derive Event B' from Event B and call it BatchEventB), I do not really like this as I then get many more, fully duplicated events
4. Let my Saga swallow the events B,C,D during its lifetime, e.g. setting some flag which tells my other Event handlers to ignore them. I do not really like this as I then have some kind of state connection between independent event handlers
5. Decoupling/disabling the event handlers during the bulk command and reenabling them after it has ended. I do not like this, because I let a command influence my system logic, this might get really ugly in a multi user environment.
   

The best compromise I can think of is:

• Use 1 if the bulk command is rarely used
• Use 2 if the bulk command is used more often, the caching might be useful at other places too
  
• Use 3 if the bulk command has only a few events.
  

Any better/other ideas on how to handle it?

Best Regards,

Torni

[Kijana Woodard]

Why are you in this "bulk command scenario"?

Is it part of normal operations or something special like "a one time job on db rebuild or new deployment"?

--
You received this message because you are subscribed to the Google Groups "DDD/CQRS" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dddcqrs+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Torni]

I have input data which fits the idea of events, as it is output of a legacy application with an audit trail, which could be considered an event stream. Still tbd if it really fits, I'm not there yet.
I know that the legacy audit trail is fairly high traffic, like 10-100 entries per second.  I currently do not know if these entries are event or commands or a combination of both, tendenceny is more to commands.
Current scenarios I know of:

1. New tenant (deployment of a new tenant while the system is still active for other tenants), probably can be scheduled e.g. at night, at weekend. Happens maybe once a week
   
2. Manual data transfer by the customer, e.g. import of legacy data, might happen daily, but probably not that many bulk commands.
   
3. Our system is the slave system, the bulk commands would be used for synchronization with a master system, might happen every hour, day, week or even permanently.
   

I personally think that the third scenario is probably the most common case and I would consider it as normal operations, whereas the other ones are special cases.

[Kijana Woodard]

Can you query the source messages in a stable order?

If so, consider processing them in batches and saving a checkpoint when you complete a batch.

You may be able to do "the real work" or you may just turn them into Commands for your system. Either way, it's easier not to "thrash" your end since you're working in batches. You can process 1000 source messages and decide there's only one Command for your system to handle.

[Torni]

Yes the audit trail is in a stable order.
I will take a look into doing a checkpoint and batch processing for the source messages.

Thanks for your help.

[Greg Young]

If you happen to be using event store this is even a built in feature
(competing consumers supports doing this)

Studying for the Turing test

[Torni]

Thanks Greg. I haven't yet evaluated any stores, but I'll keep that in mind during evaluation.

[Christian Johansen]

What if the bulk operation is a use case. Rolling out Shifts for shift workers for one month. So a RolloutTemplateCommand is issued, which causes delete, update and create of shifts, since existing shifts might be updated or deleted. 

Some consumers are interested in the exact intend the fact that a Template was rolled out other consumers are only interesting in the fact that shifts changed. How is this handled most  elegantly? what if operations are not idempotent? 

Should We raise one TemplateRolledOut event containing data about all events changed and then let a handler create granular events for this event? What if consumer listens to both the granular events and the big event?

Should We Send a initial event having a correlation id and telling the total number of events and then number all events if order of events is important? Or as suggested a start and an end event ?

Thx. in advance :-)