Datomic: Event Sourcing without the hassle

[Val Waeselynck]

An article about implementing Event Sourcing via the Datomic database. I thought this may be relevant to post here.

https://vvvvalvalval.github.io/posts/2018-11-12-datomic-event-sourcing-without-the-hassle.html

Features:

• an analysis of Event Sourcing, how it's usually implemented, and the limitations of such implementations
• how Datomic addresses these limitations.
  
  

Questions and feedback welcome!

[Vadim Platonov]

That's interesting, thanks for the write-up!

Won't address the whole thing, just a conceptual point that caught my attention:

Designing Event Types and Event Handlers is hard work

...
I think the lesson here is that an enumeration of application-defined Event Types is a weak language for describing change. 

If you find that designing events is hard work - you're doing the right thing! This means you probably are learning a lot about the domain. I'm not sure I understand the point about the events being a weak language for describing change. You literally capture all the changes within events. It's the only language for describing changes that actually matter. To follow your example - if there's a "exceptional reputation change" event missing in the language - it has to be introduced which will force all the consequences on the downstream business to be considered. If this seems like a constraint - event sourcing might not be the proper solution. 

[Val Waeselynck]

Thanks for the feedback Vadim, discussing your points below:

On Monday, 19 November 2018 10:27:38 UTC+1, Vadim Platonov wrote:

I'm not sure I understand the point about the events being a weak language for describing change. You literally capture all the changes within events. It's the only language for describing changes that actually matter.

That's the most common reaction from Event Sourcing practitioners (see for instance this Hacker News thread), and an understandable one too - I think I would have had the same reaction, had I been exposed to the idea without having used Datomic. I think there's a bit of a Blub Paradox to it. 

I don't want to make the discussion more theoretical than the article already did, and would encourage you to try this approach. Still, I'll try to give you an analogy.

Imagine your domain consists of a dot moving on a square 2-dimensional grid. You could imagine 2 events systems (2 "languages") for describing each state transition:

1. {MoveUp, MoveDown, MoveLeft, MoveRight, MoveUpLeft, MoveDownLeft, MoveUpRight, MoveDownRight, DontMove}
2. {MoveBy(x, y)}

Both event systems describe all transitions in the domain, yet most people will probably agree that 2. is superior to 1. - 2. more expressive, and 1. yields a lot of over-specificity. 2. is more "structural", whereas 1. is more "nominal".

Once we've seen that, we're ready to accept that for any given domain, different event languages will yield have different characteristics of power and flexibility. My experience with Datomic tells me that, for a lot of domains, Datoms are a better language than a set of named Event Types.

To follow your example - if there's a "exceptional reputation change" event missing in the language - it has to be introduced which will force all the consequences on the downstream business to be considered. If this seems like a constraint - event sourcing might not be the proper solution. 

Or maybe we need a less strict definition of "Event Sourcing" than the one you have in mind? I think what matters is not what constitutes Event Sourcing in the strict sense - rather how do we get the benefits of Event Sourcing (see this part of the article).

 

If you find that designing events is hard work - you're doing the right thing! This means you probably are learning a lot about the domain.

Maybe, and at the same time I'm wary of this sort of argument. We try to make flexible architectures precisely because we acknowledge the fact that our understanding of the domain will never be enough to anticipate every technical requirement.

[Johan 't Hart]

On Wed, Nov 21, 2018 at 11:03 AM Val Waeselynck <val.v...@gmail.com> wrote:

Imagine your domain consists of a dot moving on a square 2-dimensional grid. You could imagine 2 events systems (2 "languages") for describing each state transition:

1. {MoveUp, MoveDown, MoveLeft, MoveRight, MoveUpLeft, MoveDownLeft, MoveUpRight, MoveDownRight, DontMove}
2. {MoveBy(x, y)}

Both event systems describe all transitions in the domain, yet most people will probably agree that 2. is superior to 1. - 2. more expressive, and 1. yields a lot of over-specificity. 2. is more "structural", whereas 1. is more "nominal".

I'd say that 1 is more expressive. It shows intent and result can be derived from that. Number 2 shows only the result.

 

Once we've seen that, we're ready to accept that for any given domain, different event languages will yield have different characteristics of power and flexibility. My experience with Datomic tells me that, for a lot of domains, Datoms are a better language than a set of named Event Types.

To me Datomic feels like, as I understand correctly, an SQL database with the transaction log exposed. There are probably very good use-cases for this. But it feels very data-centric.

As opposed to Event Sourcing in the original sense, which is more... how to call it.. 'action centered'?

It shows the intent of the change in data. Correct me if I'm wrong, but with Datomic, I think it is hard to express e.g. this event:

CreditWithdrawn {

  amount: 20,

  reason: 'Payback John'

}

Especially, if you want your current state to just be the balance.

To follow your example - if there's a "exceptional reputation change" event missing in the language - it has to be introduced which will force all the consequences on the downstream business to be considered. If this seems like a constraint - event sourcing might not be the proper solution. 

Or maybe we need a less strict definition of "Event Sourcing" than the one you have in mind? I think what matters is not what constitutes Event Sourcing in the strict sense - rather how do we get the benefits of Event Sourcing (see this part of the article).

Benefits of Event Sourcing are tracking change, for which Datomic looks very promising. But also to track intent, or meta-info about change. I think that part is missing in Datomic.

 

 

If you find that designing events is hard work - you're doing the right thing! This means you probably are learning a lot about the domain.

Maybe, and at the same time I'm wary of this sort of argument. We try to make flexible architectures precisely because we acknowledge the fact that our understanding of the domain will never be enough to anticipate every technical requirement.

 

For this reason, I think it is best to record everything you know about the change in state. Maybe most of the info is currently not needed but in the future it could turn out to be very useful.

E.g. in the CreditWithdrawn example, maybe you currently only need to know how much credit someone has. But in a far later time you might wonder what the most common reason for the withdraw was, which you can derive from reinterpreting the events.

[Val Waeselynck]

Valentin Waeselynck

Software Engineering Contractor

Blog - Open Source - LinkedIn - Developer Story - Twitter

On Wed, 21 Nov 2018 at 20:53, Johan 't Hart <johan...@gmail.com> wrote:

On Wed, Nov 21, 2018 at 11:03 AM Val Waeselynck <val.v...@gmail.com> wrote:

Imagine your domain consists of a dot moving on a square 2-dimensional grid. You could imagine 2 events systems (2 "languages") for describing each state transition:

1. {MoveUp, MoveDown, MoveLeft, MoveRight, MoveUpLeft, MoveDownLeft, MoveUpRight, MoveDownRight, DontMove}
2. {MoveBy(x, y)}

Both event systems describe all transitions in the domain, yet most people will probably agree that 2. is superior to 1. - 2. more expressive, and 1. yields a lot of over-specificity. 2. is more "structural", whereas 1. is more "nominal".

I'd say that 1 is more expressive. It shows intent and result can be derived from that. Number 2 shows only the result.

I disagree - "Move by +1 vertically and +1 horizontally" expresses the intent just as well as "Move TopRight", and is more as expressive (as it describes a larger set of transitions).

 

 

Once we've seen that, we're ready to accept that for any given domain, different event languages will yield have different characteristics of power and flexibility. My experience with Datomic tells me that, for a lot of domains, Datoms are a better language than a set of named Event Types.

To me Datomic feels like, as I understand correctly, an SQL database with the transaction log exposed. There are probably very good use-cases for this. But it feels very data-centric.

As opposed to Event Sourcing in the original sense, which is more... how to call it.. 'action centered'?

It shows the intent of the change in data. Correct me if I'm wrong, but with Datomic, I think it is hard to express e.g. this event:

CreditWithdrawn {

  amount: 20,

  reason: 'Payback John'

}

It's not hard:

{:tx-data [#datom [johns-account :account/balance 1000 false 2132131231]
           #datom [johns-account :account/balance 1020 true 2132131231]
           #datom [alices-account :account/balance 500 false 2132131231]
           #datom [alices-account :account/balance 480 true 2132131231]
           #datom [transaction-entity :myapp/action :myapp.actions/credit-withdrawn]
           #datom [transaction-entity :db/txInstant #inst "2018-08-22"]]}

 

Especially, if you want your current state to just be the balance.

Enforced by processing commands, not events. You'd typically do this with a Transaction Function in Datomic.

To follow your example - if there's a "exceptional reputation change" event missing in the language - it has to be introduced which will force all the consequences on the downstream business to be considered. If this seems like a constraint - event sourcing might not be the proper solution. 

Or maybe we need a less strict definition of "Event Sourcing" than the one you have in mind? I think what matters is not what constitutes Event Sourcing in the strict sense - rather how do we get the benefits of Event Sourcing (see this part of the article).

Benefits of Event Sourcing are tracking change, for which Datomic looks very promising. But also to track intent, or meta-info about change. I think that part is missing in Datomic.

It's not (although I may have expressed it poorly) - search for "Reified Transactions" in this section of the article. You can have your cake and eat it too by annotating transactions. That's what we did above with :myapp/action.

 

 

 

If you find that designing events is hard work - you're doing the right thing! This means you probably are learning a lot about the domain.

Maybe, and at the same time I'm wary of this sort of argument. We try to make flexible architectures precisely because we acknowledge the fact that our understanding of the domain will never be enough to anticipate every technical requirement.

 

For this reason, I think it is best to record everything you know about the change in state. Maybe most of the info is currently not needed but in the future it could turn out to be very useful.

E.g. in the CreditWithdrawn example, maybe you currently only need to know how much credit someone has. But in a far later time you might wonder what the most common reason for the withdraw was, which you can derive from reinterpreting the events.

--
You received this message because you are subscribed to a topic in the Google Groups "DDD/CQRS" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/dddcqrs/x-4jHq7nA5g/unsubscribe.
To unsubscribe from this group and all its topics, send an email to dddcqrs+u...@googlegroups.com.
Visit this group at https://groups.google.com/group/dddcqrs.
For more options, visit https://groups.google.com/d/optout.

[Borys Levytskyi]

If I understand correctly this article, events are replaced by a stream of changes of a single entity attribute called datoms which are bundled together by a transaction meaning that every datom has the same transaction id. So how in this case should we determine which event a particular transaction (ios set of datoms) is representing?

[Val Waeselynck]

On Thursday, 22 November 2018 07:54:58 UTC+1, Borys Levytskyi wrote:

If I understand correctly this article, events are replaced by a stream of changes of a single entity attribute

Sorry, I don't understand what you mean by "single entity attribute" so I can't tell if you understand correctly :/. 

Events are "replaced by" (I'd rather say modelled as) a stream of Transactions, each Transaction being a set of Datoms (the Datoms of a Transaction therefore have the same Transaction Id, but affect different Attributes of different Entities). 

 

called datoms which are bundled together by a transaction meaning that every datom has the same transaction id. So how in this case should we determine which event a particular transaction (ios set of datoms) is representing?

Again, one important takeaway from Datomic is that it challenges the assumption that every change has to be described by a named Event Type - so answering the "which event" question may not be necessary at all.

Having, said that, if you do want every change to be labeled by some Event Type (typically for making the intent of the change more explicit), you do it by adding a Datom which annotates the Transaction Entity itself - see the above reply for an example.

Coming from an Event Sourcing background, you will typically want to do that all the time at first, then realize it's not nearly as often necessary as you initially thought :)

[Vadim Platonov]

To give context - I have built a couple systems using Datomic as well as a system using Greg's Eventstore and my own homegrown Postgres-based event store (https://github.com/dm3/everest) so I'm aware of many issues you run into while modelling and maintaining an event-sourced system.

The main difference between a conventional event-store and Datomic is that in Datomic you decide upfront which entity (Datomic aggregate - different meaning to the DDD Aggregate!) the event properties belong to and are able to query the entity immediately after the transaction is applied to the database. You can still treat every transaction as an event the way Val has shown - using a transaction attribute as the event name. The transactions can then be consumed as an event stream from Datomic transaction log (eventstore subscription) which allows to feed other components (e.g. separate projections) with events as you'd do with a conventional event store.

If you don't have extreme performance requirements, the Datomic approach works well. You might have to move some logic into db functions to achieve atomicity of operations which isn't great. On the other hand you can have arbitrary queries over the data without having to write elaborate projections. I think this is the point Val tried to stress with the "weak language for describing change" argument.

My opposition to the "weak language" claim stems from the fact that the events - the way they should be modelled in a business setting - are the only source of truth you have. Whether you assign some of the properties to entities immediately, as in Datomic, or in the Aggregate's `apply` function theoretically doesn't matter. It does make a difference practically though so I don't argue this point :) Datomic makes choices for the developer which in turn makes many things easier - something a lot of projects could benefit from.

Hopefully this makes the discussion a bit more clear.

[pagoda_5b]

After looking at some of the replies and ensuing discussions, I tend to think of Datomic have its merits if the only thing you need is to have access to historical data and snapshots of the system at any given time.

Yet the transaction log is focused, as with traditional non event-driven system, on what changes happened to your stored entities, possibly "tagged" for additional insight.

This is simply not the idea of event-sourcing. It's easier, as the article points out, because it requires much less up-front work of desigining and evolving your domain events.

On the other hand the goal of an event-sourced and event-driven system is to store the events as they are, not by their change-set on other domain entities.

The additional work required to do that have some benefits, the main being:

 - you get to describe and map in the system what are the relevant domain event, which is paramount if you follow a DDD, especially based on event-storming sessions

 - you get an event-driven system, where events can be published to other contexts without no need to extract intent from the datomic transactions

 - you can apply CQRS, which is almost always a good pair with ES, because your read-model representation is totally decoupled from the write side

From my perspective, the latter point is enabled by the first two, and I don't know if it's achievable with ease without storing the events themselves.

As always, it's about tradeoffs, and the article is very good at showing the benefits of Datomic!

I appreciate any feedback

Ivano

On Friday, November 16, 2018 at 7:40:47 PM UTC+1, Val Waeselynck wrote:

[Val Waeselynck]

Thanks Ivano, let me build on that:

On Friday, 23 November 2018 09:31:53 UTC+1, pagoda_5b wrote:

On the other hand the goal of an event-sourced and event-driven system is to store the events as they are, not by their change-set on other domain entities.

Well, that's not my perception. When using Event Sourcing, my goal is not to store the events as they are. Likewise, when using OOP, my goal is not to organise code in classes. But hey, I guess that's more a philosophical question, and so very subjective.

But more importantly, I don't think it's relevant to draw such oppositions in the case of Datomic. Datomic supports both approaches, and all the spectrum in-between; I just don't want people to stay stuck at one end of the spectrum. You absolutely can use Datomic as a raw Event Log holding a sequence of named events, and derive Aggregates the conventional way; but you can also move the cursor to something more structured. Indeed, when viewed as a data structure, Datomic is more general than both an Event Log and a relational database (you can't achieve that with a SQL database because of its mutability, and you can't achieve that with a raw Event Log because of its lack of indexes).

Cheers,