Is Event Sourcing a good fit for a system that needs post-dated/future dated changes?

[Evan William]

The current application I am building started off a a rather standard n-tier CRUD focused application, but as the requirements come to surface - especially for one particular area, it feels like I am trying to force a square peg through a round hole - and the typical considerations for a CRUD focused system with a RDBMS are working against us. 

The first set of requirements is something that makes it seem like a good fit.

- Strong audit history - no previous changes get lost

- Being able to restore a record to 'as it was' at any given point in time - What did this work order look like on a specific date? Replaying the events up to X date seems to do the trick

So event sourcing seems like a good fit - record things as events, when loading the record - replay them up to a given point in time - and you have can see what that record was like then.

The next few requirements is where I start having  a harder time trying to see where this could fit in.

- Ability to make corrections to a record, of which the effective date is in the past

- Ability to make future dated changes to a record, of which the effective date is in the future

- Any of those changes may require an approval workflow - and not get applied until the approval is complete

This is where I'm having a harder time trying to conceptualize how to handle this. If today is March 1st, 2014, and I am opening up a record and changing a field to say "The contact is Bob, but this is a post dated change that should have been there from February 1st".

If I need to reload the record to how it was as of Feb 15th - I can make sense that, but not so much "loading up how it should have been on Feb 15th" - as the date/time of the event came in at a later date, and similar questions come up for future-dated changes. Today is March 1st, 2014 - I am making a change, but shouldn't apply until April 1st, 2014.

Then some other complications come in such as approval processes, draft versions (and draft versions of records that can reference other draft versions of records) and so on. 

One of the things I am thinking is saving things as 'change requests', and then depending on the process/use cases.

- "I need it how it was on X date" - loading events from beginning of time to X date

- "I need it how it should have been on X date" - loading events from beginning of time to X date + loading events/change requests with the appropriate effective date

This is something I have only recently started researching on, but the better I get my head around it - the more it seems like an appropiate fit for the requirements of the system, any input would be appreciated.

[Kijana Woodard]

Hmmmm.

Load all events (as normal)
In your state accumulator, ignore events after desired effective date.

That seems like it would get you whatever state you need regardless of when the event came in.

For normal processing, you have to know whether the particular bit of state needs to be modified based on effective date. Maintaining last effective date per event processed seems like it should work.

--
You received this message because you are subscribed to the Google Groups "DDD/CQRS" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dddcqrs+u...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Greg Young]

Its very good at this. The answer for how to do it is a bit longer than I want to type here so I will put it into a blog post in terms of how to deal with as of vs at querying.

Cheers,

Greg

--
You received this message because you are subscribed to the Google Groups "DDD/CQRS" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dddcqrs+u...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Le doute n'est pas une condition agréable, mais la certitude est absurde.

[Evan William]

Very interested in the blog post if you get around to doing it, thanks. 

[Greg Young]

http://goodenoughsoftware.net/2014/03/02/event-sourcing-and-postpre-dated-transactions/

Here you go

[Phil Sandler]

I have a project coming up that has a similar requirement--view the state of data at any point in time--and I am contemplating whether ES would be helpful in my scenario.

It seems like this would be a natural fit for ES, as it would make it "easy" to project onto a read model that allows for time-based queries as a first option.  This would look something like a data warehouse with slowly changing dimensions or a temporal database.

As a second option, like Evan and Kijana said:

Evan William wrote:

> What did this work order look like on a specific date? Replaying the events up to X date seems to do the trick

[...]

> So event sourcing seems like a good fit - record things as events, when loading the record - replay them up to a given point in time - and you have can see what that record was like then.

Kijana Woodard wrote:

>Load all events (as normal)
>In your state accumulator, ignore events after desired effective date.

My question on this approach is: do you then have a read model that requires re-running the whole event store each time?

So if userA wants to run a query on the system as of 01/01/13 and userB wants to run a query on the system as of 07/01/13, do we re-run the store once for each of them, and each would get their own temporary read model?

It seems to me this second option wouldn't scale well?

[James Nugent]

> My question on this approach is: do you then have a read model that requires re-running the whole event store each time?
>
> So if userA wants to run a query on the system as of 01/01/13 and userB wants to run a query on the system as of 07/01/13, do we re-run the store once for each of them, and each would get their own temporary read model?
>
> It seems to me this second option wouldn't scale well?

This depends on your usage patterns. For example you might find that the majority of queries are about the state now, in which case pre-compute it, and a minority require replaying and they’ll just have a somewhat higher cost.


James

[Alberto Brandolini]

Hi Evan, 

I am focusing on your:

- Ability to make future dated changes to a record, of which the effective date is in the future

- Any of those changes may require an approval workflow - and not get applied until the approval is complete

You might probably want to dig a little more around what those 'future changes' are. As you stated, some of them might require an approval workflow, while some other I guess will simply 'expire'. An incoming payment might be due on a given date, but I wouldn't mark it as "something that happened" before I see the money on the bank account. In that case I might choose to make some concept explicit, like a 'paymentExpectation' os something like that.

More broadly speaking, I tend to keep planning and execution as separate models, and 'future events' fall in the first category. 

Another 'philosophical' warning is that some DE may have a weird concept of planning, that doesn't allow for uncertainty. They might want the people to execute orders, but you'll probably prefer a system that allows to track the fact that somebody didn't. :-)

Cheers

Alberto

--
You received this message because you are subscribed to the Google Groups "DDD/CQRS" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dddcqrs+u...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Alberto Brandolini
Phone: +39-347-6005027

E-mail: alberto.b...@gmail.com
Twitter: @ziobrando

Company: http://www.avanscoperta.it
Blog: http://ziobrando.blogspot.com
Linkedin: http://www.linkedin.com/in/brando

[Evan William]

Hi Alberto,

Right now I'm trying to wrap my head around the requirements, the restrictions I am running into dealing with CRUD and the desire for the product owner to want everything under the sun even if they end up having conflicting requirements. This post goes off rails a bit from just ES, and more general DDD/CQRS.

Right now, we sort of started with more of a data-driven design - and have ended up with a bit of 'one model to rule them all' - data = domain = read = write = the same to handle drafts, pending changes and concrete final versions. One model to rule them all, and it's making my head hurt and would like to make some changes before this area gets too much further along.

On a high level, some of the other requirements.

- Have draft records, and draft records that can reference other draft records - should be almost no constraints on what is saved (mix in with others wanting immediate consistency and strongly enforced FK constraints and not wanting to create draft tables / another data store for draft versions and then dealing with resolving relationships when things get approved ... and this is a mess)

- post dated, and future dated changes

- full audit history, both draft and final versions

- ability to put any subset of changes through an approval workflow, of which do not get applied until final approval

- full audit history on that approval, for example: gets submitted for approval, gets rejected, a field gets changed and submitted again - having a history of that - what fields changed, who did what when, how long it took between submission to rejection to re-submission, etc.

- ability to load a work order for an 'as at' view and an 'as of' view (for a mix of audit reasons, and business process reasons to get data that we need to run a process)

Things started out as largely CRUD for the first few months of the project, but now we are getting into the heart of the application, and this particular module really does not seem to fit with CRUD that well (a large part of the system fits fine with CRUD though).

I'm still pretty new to DDD/CQRS/ES - been sort of aware of it for awhile, but only really looking into it over the last month or two - but does seem to have clicked quite a bit with me in terms of how it could possibly solve some of the challenges I am facing.

Considering the main modules we have right now - Organization, Contact, Work Order, Time Sheet 

- a org can have many contacts, a contact can be associated to many organizations, work orders have relations to organizations and contacts and timesheets belong to a work order. 

- Seems like : Org, Contact, WO should be AR's - and only reference the others by ID

- Org, Contact pretty much just can be a 'draft' during create, but once active - there will only be 1 version 

- Org, Contact - if a change is done, might have an approval flow - although the product owner is fine with just an 'audit log' of changes

- Work Order has many child records (some entity, some value) 

- WO is one of the ones where being able to 'load it as it is/was at X time' is important 

- A WO could have many 'active' details (committed changes that are different versions of those details)

- A WO could also have pending change requests (which may or may not need to be applied depending on if they get approved)

Personally, I think going with 'pending changes of a specific type' is a better way of going - rate change, termination, extension, approval change, payment party change, etc - but getting a lot of push back and others wanting 'one big CRUD request, even though what is changing in that request can imply a number of different intentions and business processes'. Changing an end date if a 'correction' is far different than changing an end date if it's a termination or an extension.

Although, I think the first challenge I need to overcome is getting appropriate domain experts involved. The person who responsible for most of the requirements  more interested in the reporting data he can get out of the system, not the use of the system.

[Alberto Brandolini]

2014-03-05 14:04 GMT+01:00 Evan William <e.schu...@gmail.com>:

Hi Alberto,

Right now I'm trying to wrap my head around the requirements, the restrictions I am running into dealing with CRUD and the desire for the product owner to want everything under the sun even if they end up having conflicting requirements. This post goes off rails a bit from just ES, and more general DDD/CQRS.

conflicting requirements sounds a lot like "multiple models" to me.  

 

Right now, we sort of started with more of a data-driven design - and have ended up with a bit of 'one model to rule them all' - data = domain = read = write = the same to handle drafts, pending changes and concrete final versions. One model to rule them all, and it's making my head hurt and would like to make some changes before this area gets too much further along.

On a high level, some of the other requirements.

- Have draft records, and draft records that can reference other draft records - should be almost no constraints on what is saved (mix in with others wanting immediate consistency and strongly enforced FK constraints and not wanting to create draft tables / another data store for draft versions and then dealing with resolving relationships when things get approved ... and this is a mess)

- post dated, and future dated changes

- full audit history, both draft and final versions

- ability to put any subset of changes through an approval workflow, of which do not get applied until final approval

- full audit history on that approval, for example: gets submitted for approval, gets rejected, a field gets changed and submitted again - having a history of that - what fields changed, who did what when, how long it took between submission to rejection to re-submission, etc.

- ability to load a work order for an 'as at' view and an 'as of' view (for a mix of audit reasons, and business process reasons to get data that we need to run a process)

Things started out as largely CRUD for the first few months of the project, but now we are getting into the heart of the application, and this particular module really does not seem to fit with CRUD that well (a large part of the system fits fine with CRUD though).

Think 'data' you'll get a CRUD system. Think behaviour, you'll get a different type of system. 

 

I'm still pretty new to DDD/CQRS/ES - been sort of aware of it for awhile, but only really looking into it over the last month or two - but does seem to have clicked quite a bit with me in terms of how it could possibly solve some of the challenges I am facing.

Considering the main modules we have right now - Organization, Contact, Work Order, Time Sheet 

- a org can have many contacts, a contact can be associated to many organizations, work orders have relations to organizations and contacts and timesheets belong to a work order. 

- Seems like : Org, Contact, WO should be AR's - and only reference the others by ID

- Org, Contact pretty much just can be a 'draft' during create, but once active - there will only be 1 version 

- Org, Contact - if a change is done, might have an approval flow - although the product owner is fine with just an 'audit log' of changes

- Work Order has many child records (some entity, some value) 

- WO is one of the ones where being able to 'load it as it is/was at X time' is important 

- A WO could have many 'active' details (committed changes that are different versions of those details)

- A WO could also have pending change requests (which may or may not need to be applied depending on if they get approved)

I'd try to see the system in a different way:

- Which tasks/decisions need to be performed by live users? What information do they need? --> this provides guidance about how to design the read model. (I'd also try to substitute the word 'has' with 'needs to bee seen together with' in some cases).

- Which actions are performed by the system? What information does the system need? What's the exposed behaviour? --> this provides good info for aggregate design.

 

Personally, I think going with 'pending changes of a specific type' is a better way of going - rate change, termination, extension, approval change, payment party change, etc - but getting a lot of push back and others wanting 'one big CRUD request, even though what is changing in that request can imply a number of different intentions and business processes'. Changing an end date if a 'correction' is far different than changing an end date if it's a termination or an extension.

That's one of the main points. But my concern was also a more subtle "don't trust your domain expert blindly on that field"... some tended to say "this will never happen" when they really meant "this should not happen (and last time it happened it really drove us crazy)". You don't want to end up that does what the DE specified, but that turns out to be completely useless for most users.

Although, I think the first challenge I need to overcome is getting appropriate domain experts involved. The person who responsible for most of the requirements  more interested in the reporting data he can get out of the system, not the use of the system.

yeah, to me sounds like you need another DE on board, somebody that cares about the other portion of the system. Not saying it's going to be easy though. Quickly iterating, showing demos and doing AB testing will help finding some of them. 

Cheers

Alberto