Event Sourcing and Horizontal Partitioning/Sharding

[Fabian Schmied]

Hi,

How do people implement horizontal partitioning/sharding in
conjunction with event sourcing? Are there best practices available?

The documentation of Greg Young's Event Store says
(http://geteventstore.com/docs/what-is-an-event-store.html):

"One problem when attempting to use Horizontal Partitioning with a
Relational Database it is necessary to define the key with which the
partitioning should operate. This problem goes away when using events.
Aggregate IDs are the only partition point in the system. No matter
how many aggregates exist or how they may change structures, the
Aggregate Id associated with events is the only partition point in the
system. Horizontally Partitioning an Event Store is a very simple
process."

But what if I have a requirement to partition based on a logical key
rather than on the Aggregate ID? Say I have a system where the storage
of event-sourced aggregates should be partitioned according to some
sort of geographical location (e.g., in order to improve performance
of the command side and to allow for network partitioning). I.e.,
aggregates associated with the North should be stored within an event
store located in the North, aggregates associated with the South
should be stored within an event store located in the South.

How would I approach this requirement?

Ideas I have myself:
- Define aggregate IDs to contain the logical ID as well, e.g.,
"North-1234" rather than just "1234". This, however, wouldn't allow me
to move aggregates from North to South.
- Don't partition at all; prefer an asynchronous command side and
cache any commands while the network is partitioned.
- Other options?

Best regards,
Fabian

[Nils Kilden-Pedersen]

Event stores are fundamentally different. They contain all data for all times, so when viewed as a whole, nothing, other than the id, can be considered stable.

In short, I don't think you can do it (other than your first idea), but I too would be interested to hear others' thoughts on this.

--
You received this message because you are subscribed to the Google Groups "DDD/CQRS" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dddcqrs+u...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Gabriel Schenker]

why would you want to move an aggregate from one partition to another? Is this a realistic scenario?

On Thu, Mar 7, 2013 at 3:04 AM, Fabian Schmied <fabian....@gmail.com> wrote:

[Greg Young]

So we have planned partitionas which is separate from key.

--
You received this message because you are subscribed to the Google Groups "DDD/CQRS" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dddcqrs+u...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Le doute n'est pas une condition agréable, mais la certitude est absurde.

[Fabian Schmied]

> why would you want to move an aggregate from one partition to another? Is
> this a realistic scenario?

Yes. Consider the example I gave below: i have a set of aggregate
roots partitioned by a geographical location (North and South). I want
to physically partition the data (event store) as well in order to
allow for network splits between the North and the South (and also
because of better performance if aggregates can be accessed locally).

For example, the aggregates could represent people and the
geographical partition criterion is their home address.

Now, people can move, so the home address - and with it the partition
criterion - can change.

Best regards,
Fabian

[Fabian Schmied]

> So we have planned partitionas which is separate from key.

I'm not sure I'm reading this correctly: you mean that you're planning
this kind of partitioning as a feature in your Event Store? That would
be awesome. Or was it just a statement summarizing my question?

But regarding your Event Store, let's for a moment consider "ordinary"
partitioning by aggregate ID, as mentioned in the documentation I
cited below. How does this actually work? Say my aggregate IDs are
GUIDs, how is the partition chosen? Or would I have to use some
pattern like "category-GUID" IDs, so that I could choose the partition
based on the category?

And then, apart from your own implementation, what are the experiences
with other event stores, most remarkably JOliver's? Any chance of
getting partitioning there?

Thanks, best regards,
Fabian