Snapshots containing large sets

[Chris Nicola]

I have an aggregate root which contains a growing list of history it is tracking.  This history is part of it's state and is used to determine when to do certain things in the future.  However it gets very, very large and it is making loading the
snapshot very slow (in performance benchmarks deserialization of the snapshot is 40%+ of the profile time).

The details of the scenario are as follows.  "Participants" can participate in an rewards program and earn rewards.  What they earn is a product of a set of customizable rules and potentially their history of past participation (for example you might only get awarded if you have participated 5 times before).  A dictionary that matches participants to how many times they have participated is created to do this and it's state is updated with each participation event. 

There will be thousands of participants.

I can think of a couple of ways to handle this.  One way might be to cache the aggregate roots so they are not being constantly loaded from the event store/snapshot data.  The would still push their events down into the store but the state would be cached. 

Another would be to create a separate aggregate root type around each participant that has their participation data, and leverage that in some way through the service layer so that the reward program doesn't have to be responsible for so much data.

Thanks,
Chris

[Greg Young]

Most of the rules are aggregations I take it? eg: # of sales etc.

Could you denormalize these things onto the aggregate root instead of loading the history?

--
Les erreurs de grammaire et de syntaxe ont été incluses pour m'assurer de votre attention

[Rinat Abdullin]

Can you keep your large AR cached in memory (and partition them
between multiple machines, if all ARs do not fit in RAM)?

What serialization format do you use? Have you tried something like ProtoBuf?

Best regards,
Rinat

[Andrea]

How about a hybrid solution?
ie: you can still denormalize onto the aggregateRoot the things you
can and then if your AR is still too slow you can cache/ or try a
better deserailization approach
will have to solve a similar problem pretty soon, so would be
interested to hear how you get on
Cheers


Andrea

[Greg Young]

Also have you considered *not* using a snapshot? If you are tracking the full history of the object in the object wouldn't the snapshot essentially be the same thing as the events?

On Thu, Oct 28, 2010 at 7:16 PM, Chris Nicola <chni...@gmail.com> wrote:

[Chris Nicola]

Yes I've considered all of those options, let me give a bit more details.

The participation history is already as normalized as I can make it (I think).  It is a LUT (IDictionary<Tuple,long>) which maps the particpants (and what/how they participated) to a count of how many times they participated.  Snapshotting this is still more efficient than loading from all transactions, but serializing such a large dictionary takes some time.

Interestingly the memory footprint of this information is still fairly small and so caching is definitely one option we are considering currently.

I have tried caching and this and it saves the trouble of deserialization, but serialization of the snapshot is still necessary.  However I had a realization last night that there is no need to write the snapshot synchronously, so I am going to try async snapshots instead which should help significantly (if not almost totally eliminate the problem).

I will let you all know the results when I'm done.  Today I have some Yaks to shave.

Thanks,
Chris

[Tom Janssens]

In the past I have had similar problems. Looking at the problem in
another way gave me an insight:
Why do you need history ? It is not the history itself that is
valuable, but you usually need the history to acquire some sort of
status on the AR; the history itself is usually only necessary to
validate this state.
Instead of keeping a history for all participated programs, you could
have a ProgramParticipationSaga, which consolidates all the required
events and launches an AwardMultipleParticipations command when
certain events occured.

[João Bragança]

My problem was similar.. my AR needed to keep track of large
dictionaries of data. Serializing / De-serializing this monster was
taking too long. So I re-factored it into a Saga. I think you can sum
it up this way: unless you are working with small data sets, don't use
the write side for reading.

[Chris Nicola]

I could use a type of Saga perhaps but I could see that adding complexity that I don't need yet.  I'm also not convinced that Saga's fit my use case that well.

Another option is a seperate aggregate root for Participant, which tracks individual participants.  The participation event itself could carry the total participation and that would then become the command to the Program to determine how to award points.

This is probably not necessary though given the other options I am considering using caching and asyncronous snapshot saving.

Chris

2010/11/1 João Bragança <joao...@braganca.name>

[Tom Janssens]

If you have an instance which consolidates events about multiple AR's,
you either have another AR or a Saga. If your domain expert cares
about the process details of the transition in itself, I would
consider it an AR, if the domain expert is only interested in the end
result I would call it a saga.
Implementing logic in your existing AR would not be good practice
IMHO, remember SRP.

On 2 nov, 01:06, Chris Nicola <chnic...@gmail.com> wrote:
> I could use a type of Saga perhaps but I could see that adding complexity
> that I don't need yet.  I'm also not convinced that Saga's fit my use case
> that well.
>
> Another option is a seperate aggregate root for Participant, which tracks
> individual participants.  The participation event itself could carry the
> total participation and that would then become the command to the Program to
> determine how to award points.
>
> This is probably not necessary though given the other options I am
> considering using caching and asyncronous snapshot saving.
>
> Chris
>

> 2010/11/1 João Bragança <joao.p...@braganca.name>