Autofac decorators look like a perfect fit for CQRS commands

[Daniel Cazzulino]

http://nblumhardt.com/2011/01/decorator-support-in-autofac-2-4/

Even the sample is very much a CQRS one :)

/kzu

--
Daniel Cazzulino | Developer Lead | MS MVP | Clarius Consulting | +1 425.329.3471

[Greg Young]

If you are composing a common interface like that you aren't really
gaining much from having the container do it...

--
Le doute n'est pas une condition agréable, mais la certitude est absurde.

[Daniel Cazzulino]

Each decorator can have its own dependencies  resolved the same way.

I don't really understand your anti-DI thing, in a world where pretty much every project starts with one already... Why not leverage it if you are already using it for other parts  of the system?

/kzu from Android

On Aug 29, 2011 9:00 PM, "Greg Young" <gregor...@gmail.com> wrote:

[Greg Young]

Work with DDD a bit more and you will discover that the only place you
tend to use DI is to Application Services/Command Handlers. The
dependencies for the use case get injected into the service/command
handler and there isn't a container being used in a ton of other
places as you say.

I have no problem with leveraging a container but in this case it
really doesn't buy me much except for having to teach people how to
use something and risk. I can replace it in most systems in < 50 lines
of code... why take such a big dependency to save 50 lines of code?
Keep in mind the dependency isn't just actually linking to you but
also your release schedule, your versioning of APIs, any learning
curve associated with developers not familiar with your container,
cost of making sure licensing fits our model, and of course let's not
forget the risk that maybe there are bugs in your code. This is
assuming that said framework is non-invasive and doesn't require me to
structure my code in certain ways.

Taking on a framework dependency is a very big decision. There is a
lot of risk that needs to get factored in and frankly when dealing
with a situation of take on a dependency that prevents me from writing
40-50 lines of code .... it will take me longer just to research than
to write the code.

[Chris Martin]

I utilize Autofac's decorator support a bunch.

<interface>IRepository

<ReadModelModule>

builder.RegisterType<SqlRepository>.SingleInstance().Named<IRepository>("sql");

builder.RegisterDecorator<IRepository>((c, repo) => new AppFabricCachingRepo(repo, c.Resolve<AppFabricConfig>()), "sql");

IMO, A whole lot nicer than wrapping it at runtime.

[Chris Martin]

Agreed for the most part.

[Shawn Hinsey]

As a data point, I am working on a pretty ambitious system that incorporates a lot of ideas from CQRS and DDD. I built the initial version of it on top of Windsor because I know it best and already had a pretty solid standard methodology for using it in an unobtrusive way, but I intend on replacing it with MEF once we start talking about making the system public.

The reasons for this are basically what Greg outlines in his email. Additionally, it will help by reducing the intellectual burden of understanding the system, which for most developers new to this design philosophy is already pretty high. I realize that I'd just be swapping one burden for another, but since it's not really something you need to think about much in daily use, the fact that it's a "blessed" component will help a lot of people avoid worrying about it.

This is all pretty specific to my situation but I figured I'd throw it out there because I've been doing a lot of thinking about it lately.

[Daniel Cazzulino]

Work with DDD a bit more and you will discover that the only place you
tend to use DI is to Application Services/Command Handlers.

those two alone would make up for quite a lot of classes. That's a ton of places IMHO ;)

 

The
dependencies for the use case get injected into the service/command
handler and there isn't a container being used in a ton of other
places as you say.

I have no problem with leveraging a container but in this case it
really doesn't buy me much except for having to teach people how to
use something

Nowadays, pretty much everyone is proficient using at least one IoC container.

 

and risk. I can replace it in most systems in < 50 lines
of code... why take such a big dependency to save 50 lines of code?

far from it. manually wiring up dozens of services+handlers hardly makes up for 50 or so LoC. 

and it's hard to maintain code at that too IMO. One huge "Global.asax.cs"-style routine vs auto-wiring and convention-based decoration. Pretty easy choice I'd say.

 

Keep in mind the dependency isn't just actually linking to you but
also your release schedule

wrong. I take a dependency now on (say) Autofac 2.4, and I don't need to update ever if it works and satisfies my scenarios. I control the release schedule and update times.

 

, your versioning of APIs

how?

 

, any learning
curve associated with developers not familiar with your container,
cost of making sure licensing fits our model

there's a ton of OS containers out there, with very liberal licensing.

 

, and of course let's not
forget the risk that maybe there are bugs in your code.

those would be my fault anyways, not the container's, right?

 

This is
assuming that said framework is non-invasive

none of them are in the .NET world nowadays (except maybe for MEF and its required attributes). The Autofac example I sent clearly shows that your code has no clue it's being wired by an IoC at all.

 

and doesn't require me to
structure my code in certain ways.

If it does, it would be a very poor choice of IoC container, I'd say.

 

Taking on a framework dependency is a very big decision.

It might have been a few years back. Nowadays, every project I've worked on starts from day one asking "what DI framework are we gonna use? what unit testing framework? what mocking framework?". It's hardly argued anymore. But maybe I'm just lucky ;)

 

There is a lot of risk

I honestly don't see the risk. I may be missing something, obviously, because I've seen and used IoC containers for years now and the framework has never been an area considered a risk for any PM/DevLead I had to work with... Most containers have been well established and stable for years...

 

that needs to get factored in and frankly when dealing
with a situation of take on a dependency that prevents me from writing
40-50 lines of code

Wiring up dozens (far more I'd say on any reasonable complex system) of handlers with 2-3 service dependencies each, with those services construction too (they may also have their own dependencies), plus building manually the chain-of-responsibility for each handler for the two-three layers (say logging, transaction, security), and dealing with changes on any of that, would hardly take 40-50 lines of code.

It may be the case for your SimpleCQRS sample demo, but I don't think it's even a remotely approximate representation of a real-world application scenario...

 

.... it will take me longer just to research than to write the code.

The blog I just sent shows in less than a printed page, how to decorate your entire set of handlers (say hundreds ;)) with a transactional behavior, without modifying your handlers at all, with a generic handler that is hooked up with one line of code:

builder.RegisterGenericDecorator(typeof(TransactedCommandHandler<>), typeof(ICommandHandler<>), fromKey: "implementor")

I think you're exaggerating the supposed complexity a DI container brings to the table (while at the same time over-minimizing the complexity CQRS itself brings ;)).

Arguing about the benefits and simplicity an IoC container brings to the table feels sooo deja-vu... 

i.e., this is from 2004.... http://onestepback.org/index.cgi/Tech/Ruby/DependencyInjectionInRuby.rdoc

Nowadays, using a container is as easy as:

builder.RegisterAssemblyTypes(Assembly.GetExecutingAssembly())

         .AsImplementedInterfaces();

done!

[Greg Young]

> It might have been a few years back. Nowadays, every project I've worked on
> starts from day one asking "what DI framework are we gonna use?

what mocking framework?"

Awesome you have reached the dogma level of SQL. Congrats.

I won't argue with you as your interests are well known. But please
stop saying not wanting to use a container equates to manually writing
all dependencies

> Wiring up dozens (far more I'd say on any reasonable complex system) of
> handlers with 2-3 service dependencies each, with those services
> construction too (they may also have their own dependencies), plus building

Why on earth would your handlers have 2-3 service dependencies each?!
Sounds like a modelling problem. And why would you be having say
hundreds of handlers in a given context?

> I think you're exaggerating the supposed complexity a DI container brings to
> the table (while at the same time over-minimizing the complexity CQRS itself
> brings ;)).

Could you define what CQRS is to you? CQRS is a very simple pattern.

On Mon, Aug 29, 2011 at 9:31 PM, Daniel Cazzulino

[Daniel Cazzulino]

 > It might have been a few years back. Nowadays, every project I've worked on
> starts from day one asking "what DI framework are we gonna use?

 what mocking framework?"

Awesome you have reached the dogma level of SQL. Congrats.

You wish it was just me :P. Most of the time, I don't have such powers over the projects I work on.

 

I won't argue with you as your interests are well known.

Dunno what you mean :|. I am genuinely interested in CQRS, and actually starting to apply it on a green field project. I just don't believe in throwing out the window every tool we've learned to use and trust just for the sake of it.

 

But please
stop saying not wanting to use a container equates to manually writing
all dependencies

Oh, right! You do that fancy reflection/dynamic thing that is slow as a dog and you call it a day! :P (yeah, I measured, and it's just painfully slow, the fancy dynamic dispatching).

That's why DI containers exist. They have taken that "poor man's" approach and made it fly.

 

> Wiring up dozens (far more I'd say on any reasonable complex system) of
> handlers with 2-3 service dependencies each, with those services
> construction too (they may also have their own dependencies), plus building

Why on earth would your handlers have 2-3 service dependencies each?!

aggregate repository, query side (for set validation, etc.), maybe a logger or something? not too far fetched, no?

 

Sounds like a modelling problem. And why would you be having say
hundreds of handlers in a given context?

in a given *system*. A hundred write/state-mutating operations in a moderately complex system doesn't sound far fetched to me either.

 

> I think you're exaggerating the supposed complexity a DI container brings to
> the table (while at the same time over-minimizing the complexity CQRS itself
> brings ;)).

Could you define what CQRS is to you? CQRS is a very simple pattern.

ok, CQRS + ES, which is the way it's most commonly demoed.

[Greg Young]

>>
>> But please
>> stop saying not wanting to use a container equates to manually writing
>> all dependencies
>
> Oh, right! You do that fancy reflection/dynamic thing that is slow as a dog
> and you call it a day! :P (yeah, I measured, and it's just painfully slow,
> the fancy dynamic dispatching).
> That's why DI containers exist. They have taken that "poor man's" approach
> and made it fly.

Never had it take more than a second. I guess if we are comparing 1
second vs .001 yeah way faster but I don't care. Its done once at
start up and I am perfectly happy to have it finish in one second.

>> > Wiring up dozens (far more I'd say on any reasonable complex system) of
>> > handlers with 2-3 service dependencies each, with those services
>> > construction too (they may also have their own dependencies), plus
>> > building
>>
>> Why on earth would your handlers have 2-3 service dependencies each?!
>
> aggregate repository, query side (for set validation, etc.), maybe a logger
> or something? not too far fetched, no?

a logger.. couldn't we do that in the chain?

public Unit Log(Func<Unit, T> next, T message) : where T:Message {
MyLoggingFramework.Log("received " + message);
next(message);
}

This also shows a bit why I don't use the style of what you are doing
anymore ... much nicer to deal with functions instead of classes that
meet an interface. Its even nicer in a language like Scala. We can
even do "dependency injection"

public void BarHandler(IDepend1 d1, IDepend2 d2, BarCommand cmd) {
}

x = > BarHandler(new CD1(), new CD2());

returning a Func<T,Unit> where T:Command

Now if I ended up having nesting levels of 2-3 levels deep, yeah I
would consider using a DI framework but I generally don't run into
this scenario or its very limited.

A repository ... I would make it come by default (after applying cqrs
there is only one Get method after all no?)
So back to one dependency .. and I need a container to handle my one dependency?

so now I operate on a Func<IRepository, T, Unit> ... just a partial
application away.

On Mon, Aug 29, 2011 at 10:11 PM, Daniel Cazzulino

[Jimmy Bogard]

ditto.

We've found that if we're drawing our bounded contexts well, a lot of big frameworks just kinda disappear. ORMs, DI, etc etc. Micro-frameworks are much more apropos in this area, since I'm likely trying to just figure out how few lines of code it can take me to get data out of a read model.

Where I've found big modeling investments to be more necessary are in developing stand-alone, shrink-wrapped products. A lot of the technologies are constrained, so you're forced into a more traditional DDD approach.

[Daniel Cazzulino]

>> But please
>> stop saying not wanting to use a container equates to manually writing
>> all dependencies
>
> Oh, right! You do that fancy reflection/dynamic thing that is slow as a dog
> and you call it a day! :P (yeah, I measured, and it's just painfully slow,
> the fancy dynamic dispatching).
> That's why DI containers exist. They have taken that "poor man's" approach
> and made it fly.

Never had it take more than a second. I guess if we are comparing 1
second vs .001 yeah way faster but I don't care. Its done once at
start up

It happens every time a command has to be instantiated and the dependency graph built to satisfy a request.

 

and I am perfectly happy to have it finish in one second.

>> > Wiring up dozens (far more I'd say on any reasonable complex system) of
>> > handlers with 2-3 service dependencies each, with those services
>> > construction too (they may also have their own dependencies), plus
>> > building
>>
>> Why on earth would your handlers have 2-3 service dependencies each?!
>
> aggregate repository, query side (for set validation, etc.), maybe a logger
> or something? not too far fetched, no?

a logger.. couldn't we do that in the chain?

public Unit Log(Func<Unit, T> next, T message) : where T:Message {
   MyLoggingFramework.Log("received " + message);
   next(message);
}

that's the kind of useless logging that people always bring up when talking about AOP. meaningless, cross-cutting logging that doesn't tell me anything of use for the business or for troubleshooting. Useful logging goes way further, tracing information/verbose messages as domain changes are performed, calculations done, special conditions met, exceptions when certain conditions happen, etc. I have never needed to use a cross-cutting logging facility, ever. 

 

This also shows a bit why I don't use the style of what you are doing
anymore ... much nicer to deal with functions instead of classes that
meet an interface. Its even nicer in a language like Scala. We can
even do "dependency injection"

public void BarHandler(IDepend1 d1, IDepend2 d2, BarCommand cmd) {
}

x = > BarHandler(new CD1(), new CD2());

returning a Func<T,Unit> where T:Command

Now if I ended up having nesting levels of 2-3 levels deep, yeah I
would consider using a DI framework but I generally don't run into
this scenario or its very limited.

A repository ... I would make it come by default (after applying cqrs
there is only one Get method after all no?)
So back to one dependency .. and I need a container to handle my one dependency?

so now I operate on a Func<IRepository, T, Unit> ... just a partial
application away.

Yup, sounds great in theory. I've fallen in love with that too. It doesn't lend itself too easily to loosely coupled composition though. It only works if you know all the funcs ahead of time and have your glorified Main.cs. Most systems need dynamic composition at runtime, which is way easier with interfaces and DI than with funcs, partial applications and what-nots.... I'm talking about .NET only, of course (Func<T> is .NET after all...)

[Greg Young]

"
Yup, sounds great in theory. I've fallen in love with that too. It
doesn't lend itself too easily to loosely coupled composition though.
It only works if you know all the funcs ahead of time and have your
glorified Main.cs. Most systems need dynamic composition at runtime,
which is way easier with interfaces and DI than with funcs, partial
applications and what-nots.... I'm talking about .NET only, of course
(Func<T> is .NET after all...)"

"Most systems need dynamic composition at runtime" ... you have
measured this how exactly? I tend to not need it. An off the cuff
guess would be < 5% of use cases need this (probably closer to 1% but
will estimate high).

On Mon, Aug 29, 2011 at 10:54 PM, Daniel Cazzulino

[Greg Young]

Going along ... *why* do they need dynamic composition at runtime on
their command handlers? What is it they are doing that they need
dynamic composition? Are there other ways of solving that problem?

[Daniel Cazzulino]

> "Most systems need dynamic composition at runtime"

Meaning, loosely coupled development of independent parts of a system via "modules" that are versioned and deployed independency and composed at runtime using some form of IoC.

Measured: my own experience, and the requirements p&p gathers from big MS customers. I worked on that team for years and I heard that requirement over and over and over. 

Even now that I'm no longer working directly with p&p, I still haven't worked on a single project that didn't need loosely coupled and independently developed modules to be composed at runtime. 

For an e-comerce style app, that would be for example deploying a new website feature by just deploying new views (razor, aspx, whatever) and have new MVC controllers and command handlers auto-wired on the next app restart. Zero config.

/kzu

--
Daniel Cazzulino | Developer Lead | MS MVP | Clarius Consulting | +1 425.329.3471

[Greg Young]

Auto-wiring command handlers is trivial and doesn't require to build
on every request. It requires a build when a new assembly is detected.
Let's try going back to actual dynamic pipelines. When do you use
them? When do you build a different pipeline for a FooCommandHandler
than for another FooCommandHandler ..

And if you are suggesting that DDD or CQRS based systems aren't
compositional then I'm not sure where the conversation can continue
from as no one else seems to have problems with this.

On Mon, Aug 29, 2011 at 11:11 PM, Daniel Cazzulino

[Daniel Cazzulino]

here are some of the companies driving those kinds of requirements:

http://msdn.microsoft.com/en-us/library/ff632023.aspx#AuthorsContributors

/kzu

--
Daniel Cazzulino | Developer Lead | MS MVP | Clarius Consulting | +1 425.329.3471

[Daniel Cazzulino]

On Tue, Aug 30, 2011 at 00:16, Greg Young <gregor...@gmail.com> wrote:

Auto-wiring command handlers is trivial and doesn't require to build
on every request. It requires a build when a new assembly is detected.
Let's try going back to actual dynamic pipelines. When do you use
them? When do you build a different pipeline for a FooCommandHandler
than for another FooCommandHandler ..

? I'd never do that. 

Show me your auto-wiring command handlers without slow-as-a-dog Activator.CreateInstance (and that only works if ALL your command handlers have exactly the same required dependencies and nothing more).

 

And if you are suggesting that DDD or CQRS based systems aren't
compositional

? where did I say that? 

 

then I'm not sure where the conversation can continue
from as no one else seems to have problems with this.

no-one (I've seen or read at least) is doing Func<> style composition and partial application as a general-purpose way of composing complex systems in the .NET. That's all I said. Quite the contrary with "plain old" interface-based and IoC-based composition.

[Greg Young]

> Show me your auto-wiring command handlers without slow-as-a-dog
> Activator.CreateInstance (and that only works if ALL your command handlers
> have exactly the same required dependencies and nothing more).

Umm no it doesn't, and you wouldn't have to have any
Activator.CreateInstance dealing with funcs. If using objects I still
*rarely* ever create a new pipeline in the processing of a command.
There are a few use cases out there that would need this but again
this is < 5%. So I might have to write a little code to do that a few
times in the building of a medium sized system. I am willing to accept
that (we are talking about 2-5 lines of code for each one)

DI is cool, it has its place. There are other methods that are cool
too, they have their places as well. You sound very dogmatic,
especially when you make statements like "before we start a project we
figure out which container and mocking framework to use". I have built
many systems with neither a mocking framework nor a container and yet
somehow they still work and are as well factored as if using such
frameworks. It can be done. There are other ways of solving problems.
Containers are not a new concept, have alternatives, and their own
weaknesses associated with them.

In writing CommandHandlers you will in most circumstances if following
bounded contexts find little to no value in using a container as it
just doesn't save you much.

>> And if you are suggesting that DDD or CQRS based systems aren't
>> compositional
>
> ? where did I say that?

On Mon, Aug 29, 2011 at 11:22 PM, Daniel Cazzulino

[Daniel Cazzulino]

> Show me your auto-wiring command handlers without slow-as-a-dog
> Activator.CreateInstance (and that only works if ALL your command handlers
> have exactly the same required dependencies and nothing more).

Umm no it doesn't, and you wouldn't have to have any
Activator.CreateInstance dealing with funcs. If using objects I still
*rarely* ever create a new pipeline in the processing of a command.

Command handlers should be single-call style, no? (as in, every request gets a new instance of the handler, likewise for its decorators/pipeline). Did I miss something? :S

 

There are a few use cases out there that would need this but again
this is < 5%. So I might have to write a little code to do that a few
times in the building of a medium sized system. I am willing to accept
that (we are talking about 2-5 lines of code for each one)

DI is cool, it has its place. There are other methods that are cool
too, they have their places as well.

I'm sure. I'm trying to understand why I would want to manually write code that there's no need to anymore. Saving 500 lines of code (5 times 100 handlers on a medium-sized system) is good enough for me (heck, saving 2 lines for each handler is also good enough for me!). Less code == less bugs == less code to maintain and document. Auto-wiring by conventions via IoC makes that extremely easy.

 

You sound very dogmatic,
especially when you make statements like "before we start a project we
figure out which container and mocking framework to use".

You keep mis-quoting me. That won't do :P. I said:

> Nowadays, every project I've worked on starts from day one asking "what 

> DI framework are we gonna use? what unit testing framework? what 

> mocking framework?". It's hardly argued anymore. But maybe I'm just lucky ;)

It's not *me* who determines that such is the project kick-start. Hence the "I'm just lucky" :P

I'm pragmatic, rather. I'm having a very hard time convincing anyone on this new project that we have to throw away DI, the ORM, consistency via transactions, real-time write operations, etc. 

And that is (IMHO) CQRS+ES biggest roadblock: it's not helping people adopt it if you're not making them feel this is an evolutionary step, rather than a revolutionary one. And your position (quite dogmatic, I'd say too ;)) on DI and ORMs (both of which took quite some time to become mainstream) isn't helping either. 

My point being that if you tell people "of course you can do CRQS+ES with DI and an ORM!" rather than "why on earth would you want to do such an abomination?", you get them interested, rather than scared off.

 

I have built
many systems with neither a mocking framework nor a container and yet
somehow they still work and are as well factored as if using such
frameworks. It can be done.

Absolutely, not arguing that. you can always write all the code manually, all the way down to a test runner which is for the most part 10-20 lines of reflection code anyways too. If it makes you feel more in control, less "risk", less "complexity", fine. 

 

There are other ways of solving problems.
Containers are not a new concept, have alternatives, and their own
weaknesses associated with them.

In writing CommandHandlers you will in most circumstances if following
bounded contexts find little to no value in using a container as it
just doesn't save you much.

Sure, but people already use them and trust them. So why fight them? I don't think you gain much either by manually writing those boring object wiring statements...

[Greg Young]

>> There are a few use cases out there that would need this but again
>> this is < 5%. So I might have to write a little code to do that a few
>> times in the building of a medium sized system. I am willing to accept
>> that (we are talking about 2-5 lines of code for each one)
>>
>> DI is cool, it has its place. There are other methods that are cool
>> too, they have their places as well.
>
> I'm sure. I'm trying to understand why I would want to manually write code
> that there's no need to anymore. Saving 500 lines of code (5 times 100
> handlers on a medium-sized system) is good enough for me (heck, saving 2
> lines for each handler is also good enough for me!). Less code == less bugs
> == less code to maintain and document. Auto-wiring by conventions via IoC
> makes that extremely easy.

<5% of 100 != 100 you're point is meaningless when comparing 10-25
lines of code vs the dependency. Understanding a point before trying
to argue is useful. As well to have 100 command handlers in a bounded
context pretty much tells me your bounded context is wrong.

> Command handlers should be single-call style, no? (as in, every request gets
> a new instance of the handler, likewise for its decorators/pipeline). Did I
> miss something? :S

Exactly why would you want stateful command handlers? If they are not
stateful ...

what dependencies might they have? A repository might be stateful,
easy enough ... what other stateful things are you passing? Why are
you passing those things?

>"My point being that if you tell people "of course you can do CRQS+ES with DI and an >ORM!" rather than "why on earth would you want to do such an abomination?", you get >them interested, rather than scared off."

Why would I tell them to do such a stupid thing? To do ES with an ORM?
I tell people regularly that they can use an ORM with CQRS, its just
kinda doesn't make any sense at all to use an ORM + ES.

I also don't tell people *not* to use a container. I tell them to
evaluate what they get from taking the dependency and to consider
alternatives. I don't say containers are bad, I say there are choices
and they should be looked at.

[Caleb Vear]

Greg,

I am curious about how you are using Funcs for your command handlers.  Where do you declare the initially?  In my little spike I had a class per command handler, but I am interested in a potentially different approach.

Daniel,

I think Greg is suggesting that he can wire up all of his handlers using some convention based reflection and not have to pay the price of reflection each time it is done by creating and caching delegates or something similar.  I use this approach for applying events to my aggregates.  I only pay the reflection cost once and the next time it is just as fast as if I called the method directly.

As for the logging, from my personal experience I can say that it would be more useful to just print out commands and events as they go through the system than to try and log something more verbose than that.  Otherwise the logging code clutters up the rest of you code and is just another thing that you will need to test.  If you need to debug an issue you are having in production and you can't attach a debugger you can just copy the target aggregate's events to your development PC and then try running the command through.

Cheers,

Caleb

[ashic]

"what dependencies might they have? A repository might be stateful,"

If you do need some stateful, per request dependency, you can easily
pass in a Factory (or a factory method Func<IRepository>) to the only-
one-instance-ever command handler. In the executor, you'd be calling
the factory so there's no risk of Activator.CreateInstance.

[Sharas]

bikeshedding :)

[Rinat Abdullin]

I have a deep and passionate affection for Autofac.

Yet, I find recently that avoiding dependencies (even if it comes at the cost of adding a few boring lines of code now and then) helps to be more efficient with the projects, while keeping their overall costs and complexity low.

This includes all things with DI, mocking, ORMs (and relational storage altogether). I'm not even talking about all the amazing and complexity-boosting things one can do with aspect oriented programming and dynamic proxies (esp. when DI/IOC is added into the mix) :)

Just my personal thoughts, backed up by a rather limited experience.

Best,

Rinat

2011/8/30 Sharas <apkr...@gmail.com>

bikeshedding :)

[Colin Yates]

Whilst I understand some people's reaction to Greg's apparently "hard-line" stance, I absolutely appreciate his point.  After 15 years of doing this stuff, I decided to analyse my working patterns and toolsets and asked myself the question "what is the simplest thing that can work".  In terms of accepted-by-default incidental complexity - boy we are drowning in it!  

Familiarity *doesn't* remove incidental complexity, it merely moves it to others that aren't familiar with the tools/patterns/processes.

If nothing else, if this thread causes people to think "is this really KISS" when looking at their standard toolset then it is a win.

(I am a Java dude; you think *you* have it bad in the .NET world - ha!)

:)

[Daniel Cazzulino]

This has been a very instructive thread, thanks everyone involved. I'm conscious there's quite a bit of un-learning to do, I'm down that path myself.

I'm struggling to find a middle ground that can bring traditional OO/DI/ORM architectures more willingly to this style of development. So far I haven't found the messaging of CQRS+ES particularly compelling for the majority of developers. After all, very few are building the next Twitter/Facebook (that's the main counter argument I hear). So, scalability alone just doesn't cut it. Eventual consistency scares the hell out of everyone. "Can't open SQL Management Studio and see what's in there?", another.

I wonder if anyone has merged both ORM and ES in a Frankenstein of a hybrid that leaves biz owners happy (ORM) and alpha geeks too (CQRS+ES). Something like placing ES and Commands on top of an ORM-driven model... (don't laugh, but this would be a MUCH easier sell: "we'll continue to use what you have used for the past few years, we'll just add a very cool and flexible and future-proof system-wide 'audit log' of everything that happens to the system, and it's gonna be CHEAP to add")

PS: > I won't argue with you as your interests are well known.

I just now understood where that came from, Greg :). I couldn't care less about Moq and its future (I think that's what you meant by "interests are well known", correct me if I'm wrong). If it has to die because it's no longer needed, so be it. It's just another tool that might indeed be the wrong one in our toolset. I don't give a damn. I care about architecture and sound design patterns. Tools, languages and frameworks are disposable. I created Moq 'cause I didn't want to write boring repetitive code. I'll continue to use DI to auto-wire the command handlers for the same reason. Ditto for intellisense, powerful IDEs, etc. I've got no personal attachment to any, it's a pragmatic thing. I hate doing machine's work.

/kzu

--
Daniel Cazzulino | Developer Lead | MS MVP | Clarius Consulting | +1 425.329.3471

[Rinat Abdullin]

That's one of the big problems with CQRS+ES -  scalability seems to be the purpose here.

Personally I find it appealing for ability to make my life simpler. By allowing to keep bounded context focused (while still supporting complex integrations), reducing friction upon deploying new versions with minimum downtime and detaching from a lot of dependencies. Better behavior testing and almost-linear scalability are just an icing on top.

Best,
Rinat

On Tuesday, August 30, 2011, Daniel Cazzulino <k...@clariusconsulting.net> wrote:
> This has been a very instructive thread, thanks everyone involved. I'm conscious there's quite a bit of un-learning to do, I'm down that path myself.
> I'm struggling to find a middle ground that can bring traditional OO/DI/ORM architectures more willingly to this style of development. So far I haven't found the messaging of CQRS+ES particularly compelling for the majority of developers. After all, very few are building the next Twitter/Facebook (that's the main counter argument I hear). So, scalability alone just doesn't cut it. Eventual consistency scares the hell out of everyone. "Can't open SQL Management Studio and see what's in there?", another.
>
> I wonder if anyone has merged both ORM and ES in a Frankenstein of a hybrid that leaves biz owners happy (ORM) and alpha geeks too (CQRS+ES). Something like placing ES and Commands on top of an ORM-driven model... (don't laugh, but this would be a MUCH easier sell: "we'll continue to use what you have used for the past few years, we'll just add a very cool and flexible and future-proof system-wide 'audit log' of everything that happens to the system, and it's gonna be CHEAP to add")
>
>
>
> PS: > I won't argue with you as your interests are well known.
> I just now understood where that came from, Greg :). I couldn't care less about Moq and its future (I think that's what you meant by "interests are well known", correct me if I'm wrong). If it has to die because it's no longer needed, so be it. It's just another tool that might indeed be the wrong one in our toolset. I don't give a damn. I care about architecture and sound design patterns. Tools, languages and frameworks are disposable. I created Moq 'cause I didn't want to write boring repetitive code. I'll continue to use DI to auto-wire the command handlers for the same reason. Ditto for intellisense, powerful IDEs, etc. I've got no personal attachment to any, it's a pragmatic thing. I hate doing machine's work.
>
> /kzu
>
> --

> Daniel Cazzulino | Developer Lead | MS MVP | Clarius Consulting | +1 425.329.3471 <tel:%2B1%20425.329.3471>

>
>
> On Tue, Aug 30, 2011 at 08:26, Colin Yates <colin...@gmail.com> wrote:
>>
>> Whilst I understand some people's reaction to Greg's apparently "hard-line" stance, I absolutely appreciate his point.  After 15 years of doing this stuff, I decided to analyse my working patterns and toolsets and asked myself the question "what is the simplest thing that can work".  In terms of accepted-by-default incidental complexity - boy we are drowning in it!  
>> Familiarity *doesn't* remove incidental complexity, it merely moves it to others that aren't familiar with the tools/patterns/processes.
>>
>> If nothing else, if this thread causes people to think "is this really KISS" when looking at their standard toolset then it is a win.
>> (I am a Java dude; you think *you* have it bad in the .NET world - ha!)
>> :)
>>
>> On 30 August 2011 11:06, Rinat Abdullin <rinat.a...@gmail.com> wrote:
>>>
>>> I have a deep and passionate affection for Autofac.
>>> Yet, I find recently that avoiding dependencies (even if it comes at the cost of adding a few boring lines of code now and then) helps to be more efficient with the projects, while keeping their overall costs and complexity low.
>>> This includes all things with DI, mocking, ORMs (and relational storage altogether). I'm not even talking about all the amazing and complexity-boosting things one can do with aspect oriented programming and dynamic proxies (esp. when DI/IOC is added into the mix) :)
>>> Just my personal thoughts, backed up by a rather limited experience.
>>> Best,
>>> Rinat
>>>
>>> 2011/8/30 Sharas <apkr...@gmail.com>
>>>>

>>>> bikeshedding <http://en.wiktionary.org/wiki/bikeshedding> :)
>>>
>>
>
>

--
Rinat Abdullin | Technology Leader at Lokad.com | Writer at Abdullin.com | Contacts

[Greg Young]

Also scalability isn't why people are going for CQRS, its a removal of
complexity that most people benefit from

--

[Greg Young]

CQRS+ES with an ORM makes no sense at all. Its just retarded, really
when the entire domain model supports only a single query and there is
no mapping whats the point of the ORM?

Many people do CQRS with an ORM though.

On Tue, Aug 30, 2011 at 10:50 AM, Daniel Cazzulino
<k...@clariusconsulting.net> wrote:

--

[Sharas]

Just to clarify, that single query is to load all the events/snapshot for an aggregate?

[James Nugent]

> After all, very few are building the next Twitter/Facebook
> (that's the main counter argument I hear). So, scalability alone just
> doesn't cut it.

This assumes you're after scalability, rather than, say, a complete,
correct audit trail, or the ability to generate different (unknown)
structural models later.

> I wonder if anyone has merged both ORM and ES in a Frankenstein of a hybrid
> that leaves biz owners happy (ORM) and alpha geeks too (CQRS+ES).

I've yet to meet a business owner who could care less what's
'underneath'. Much less one who knows what an ORM is...


James

[Daniel Cazzulino]

Call it the CTO, the Architect, the 'IT guy'. Whoever you have to  present the project, plan and intended architecture to get the contact. They do care, I think.

/kzu from Android

On Aug 30, 2011 1:26 PM, "James Nugent" <jen.b...@googlemail.com> wrote:

[Jimmy Bogard]

We've hit this as well. IT policies directing that "any data access must go through SQL Server and must be on our servers and must conform to ISO naming standards" etc.

Sometimes you have to work in the constraints that exist, but I wouldn't try to artificially constrain myself before development gets started. There's something quite liberating with "File -> New Project" and not pulling in a bunch of tools that aren't needed yet (if ever).

[Daniel Cazzulino]

Yup, my point precisely.

/kzu

--
Daniel Cazzulino | Developer Lead | MS MVP | Clarius Consulting | +1 425.329.3471

[Daniel Cazzulino]

If you think about what they want, it's the ORM keeping the authoritative "current" state of the app, not the event stream. 

So, there's hardly a single query. It's "plain old" ORM-driven persistence, ORM-driven query side too, transactional across command and query ("consistency is overrated" doesn't usually fly ;), and the unanswered questions to your post at http://codebetter.com/gregyoung/2010/08/12/eventual-consistency-and-set-validation/ don't help).

So, they feel safer within the boundaries of a database transaction, even if there is a logical separation between the command behavior and the query "views" (no need for scalability there, just better organization). The full audit log is very easy to sell in that setup, and it could even be used to verify the consistency of the ORM model (i.e. to detect no model changes are going outside of the event bus), and eventually become the source of truth, once they see the actual benefits it brings and the totally unneeded duplication with the ORM...

Just an heretic thought, anyway ;)

/kzu

--
Daniel Cazzulino | Developer Lead | MS MVP | Clarius Consulting | +1 425.329.3471

[Greg Young]

Oddly I do tell people that is fine to do (its a first step).

[Daniel Cazzulino]

So it isn't so retarded in the end, ah? ;-)

/kzu

--
Daniel Cazzulino | Developer Lead | MS MVP | Clarius Consulting | +1 425.329.3471

[Greg Young]

Rather obviously what you described was not Event Sourcing and
therefore not CQRS+ES. My statement was that CQRS+ES+ORM was retarded.
Sorry to not let you have your snicker but please do some reading
before making your ridiculous statements.

On Tue, Aug 30, 2011 at 7:52 PM, Daniel Cazzulino

[Greg Young]

btw have you bothered to watch the class online? I talk about this
architecture extensively in it (both good things and bad)

On Tue, Aug 30, 2011 at 1:11 PM, Daniel Cazzulino
<k...@clariusconsulting.net> wrote:

[Caleb Vear]

To be honest I can see how he confused your statement.  What you are saying (if I read correctly) is that it would be stupid to use an ORM to load the events needed to reconstruct an aggregate root.  Which I suspect you both agree on.  That isn't how I initially read your statement either we always have to remember that when we are all throwing acronyms around right left and center it is easy for us to misunderstand exactly what the other person is meaning by them.

[ashic]

Kind of late to the party (damn timezone..) but...
I've been doing CQRS + ES for over a year (and almost a year in pro
projects that have been delivered). My projects have had no need for
scalability. The main benefit I've found has been better design and
enforcing of DDD. Sure, I haven't got it right everytime but whenever
I've cut a corner, it has been clearly visible (and even documented).
These would've probably just disappeared into unwritten conventions.
The smell has always been contained to a small location and it has
been easy to not have it propagate everywhere. We've revisited many of
those areas and improved it later.

We've had new devs join the team with zero knowledge about any of the
tech being used adding complicated SSRS(!) reports where nobody on the
team knew hardly anything about SSRS. They new person was productive
in the first day and completed some complicated tasks within 3 days.
The simplicity of the architecture was what enabled them to do so.

Testing is also simplified where instead of checking internal
structure, we're checking external behaviour. The free audit was
greatly appreciated.

And all this even before we did composition of command executors.

We have used ORMs for the read side (don't anymore). You know what we
used? EF Code First. To handle the db generation. Besides, since the
tables are just flattened, we didn't run into any real issues when we
did use it. Not having to deal with the convoluted idiosyncrasies of a
particular ORM has been a breath of fresh air.

Scalability? Well, one of the apps we replaced took 3+ minutes to
generate a report that now takes less than a quarter of a second. It
didn't need to be that fast. But it was. Scalability is not the coup
de grace of CQRS + ES. It is merely a nice side effect as far as my
projects are concerned. I really don't get why people keep saying "we
don't have the scalability requirements of twitter or facebook...we
don't need CQRS / ES".

On Aug 31, 1:12 am, Greg Young <gregoryyou...@gmail.com> wrote:
> btw have you bothered to watch the class online? I talk about this
> architecture extensively in it (both good things and bad)
>
> On Tue, Aug 30, 2011 at 1:11 PM, Daniel Cazzulino
>
>
>
> >
>
>
>
>
> <k...@clariusconsulting.net> wrote:
> > If you think about what they want, it's the ORM keeping the authoritative
> > "current" state of the app, not the event stream.
> > So, there's hardly a single query. It's "plain old" ORM-driven persistence,
> > ORM-driven query side too, transactional across command and query
> > ("consistency is overrated" doesn't usually fly ;), and the unanswered
> > questions to your post

> > at http://codebetter.com/gregyoung/2010/08/12/eventual-consistency-and-s...don't

> > help).
> > So, they feel safer within the boundaries of a database transaction, even if
> > there is a logical separation between the command behavior and the query
> > "views" (no need for scalability there, just better organization). The full
> > audit log is very easy to sell in that setup, and it could even be used to
> > verify the consistency of the ORM model (i.e. to detect no model changes are
> > going outside of the event bus), and eventually become the source of truth,
> > once they see the actual benefits it brings and the totally unneeded
> > duplication with the ORM...
> > Just an heretic thought, anyway ;)
> > /kzu
>
> > --
> > Daniel Cazzulino | Developer Lead | MS MVP | Clarius Consulting | +1
> > 425.329.3471
>

> >> > On Tue, Aug 30, 2011 at 08:26, Colin Yates <colin.ya...@gmail.com>

> >> > wrote:
>
> >> >> Whilst I understand some people's reaction to Greg's apparently
> >> >> "hard-line" stance, I absolutely appreciate his point.  After 15 years
> >> >> of
> >> >> doing this stuff, I decided to analyse my working patterns and toolsets
> >> >> and
> >> >> asked myself the question "what is the simplest thing that can work".
> >> >>  In
> >> >> terms of accepted-by-default incidental complexity - boy we are
> >> >> drowning in
> >> >> it!
> >> >> Familiarity *doesn't* remove incidental complexity, it merely moves it
> >> >> to
> >> >> others that aren't familiar with the tools/patterns/processes.
>
> >> >> If nothing else, if this thread causes people to think "is this really
> >> >> KISS" when looking at their standard toolset then it is a win.
> >> >> (I am a Java dude; you think *you* have it bad in the .NET world - ha!)
> >> >> :)
>

> >> >> On 30 August 2011 11:06, Rinat Abdullin <rinat.abdul...@gmail.com>

> >> >> wrote:
>
> >> >>> I have a deep and passionate affection for Autofac.
> >> >>> Yet, I find recently that avoiding dependencies (even if it comes at
> >> >>> the
> >> >>> cost of adding a few boring lines of code now and then) helps to be
> >> >>> more
> >> >>> efficient with the projects, while keeping their overall costs and
> >> >>> complexity low.
> >> >>> This includes all things with DI, mocking, ORMs (and relational
> >> >>> storage
> >> >>> altogether). I'm not even talking about all the amazing and
> >> >>> complexity-boosting things one can do with aspect oriented programming
> >> >>> and
> >> >>> dynamic proxies (esp. when DI/IOC is added into the mix) :)
> >> >>> Just my personal thoughts, backed up by a rather limited experience.
> >> >>> Best,
> >> >>> Rinat
>

> >> >>> 2011/8/30 Sharas <apkro...@gmail.com>

[Greg Young]

It was actually his statement not mine. I repeated it ...

<snip>

I wonder if anyone has merged both ORM and ES in a Frankenstein of a
hybrid that leaves biz owners happy (ORM) and alpha geeks too
(CQRS+ES). Something like placing ES and Commands on top of an
ORM-driven model...

</snip>

Not sure how he could misunderstand his own paragraph. My guess is he
is confusing the concept of "Audit Log" and "Event Sourcing".

[Caleb Vear]

I took his statement to mean:

ORM is used to load what are essentially snapshots of his aggregates.  Events are still generated by all command methods and applied to the aggregate to update the state.  New events would be written to the database along with the ORM session commit, but the ORM wouldn't actually know about it.  You could do something like this with NHibernate by creating a session interceptor.  Obviously this is a little less than ideal, but as a first step on the way to moving to working entirely with ES it might not be the worst thing you could do.

[Greg Young]

He clarified in next post.

<snip>

If you think about what they want, it's the ORM keeping the
authoritative "current" state of the app, not the event stream.

So, there's hardly a single query. It's "plain old" ORM-driven
persistence, ORM-driven query side too, transactional across command
and query ("consistency is overrated" doesn't usually fly ;)

</snip>

pretty much once we say the "event log is not the authoritative" the
concept of ES is not there any more.

My guess is its just a complete misunderstanding as to what ES is.

[Daniel Cazzulino]

I say " Something like placing ES and Commands on top of an ORM-driven model..."

You say "That's retarded"

I elaborate a bit

You say "'Oddly I do tell people that is fine to do (its a first step)."

I say "It's not that retarded after all"

Now that you sent me to watch your videos (which I did, all of them, in their entirety), I'll send you to read the thread, all of it :P

And obviously the right interpretation of my paragraph was in the way Caleb understood it. I guess it's just easier to assume people is stupid and move on.

Man, there's no need to get pissed off. I'm just telling you the answers I get when I start explaining ES. I get comments like "yeah, right, an audit log, what's new?". I DO understand the patterns and the concepts, I'm just sayin' that's not how some people see it and that it's WAY easier to sell it AS IF IT WAS A GLORIFIED AUDIT LOG, than to sell it as a replacement for an ORM.

Precisely one of ashic points "The free audit was greatly appreciated.". 

THAT is an easy sell. Now, and here's the "retarded" thought: if you get the project buying 99% of CQRS+ES, where the only remaining 1% is essentially the loading of aggregates from events ('cause the ORM is also keeping the "current snapshot" at all times), then you've made a HUGE progress over just arguing ad-infinitum whether they should be throwing the ORM, SQL Server and their DBA out the window. 

I a few months, when someone new to the team (as ashic mentions) gets up and running fast, gets results, and they are all coming from the CQRS+ES side of the equation rather than the ORM ("legacy-snapshotter" if you will), then it will be even easier to just ditch the ORM and flip a switch on the aggregates to load from a true event store with snapshots and what-nots.

Or you can keep repeating aloud while they are leaving that that's just a retarded, stupid and obviously ignorant thing to do and go miss the next contract. 

I have the feeling that there has to be an easier path from current status quo ORM to CQRS+ES to bridge the gap. Cross the chasm, ya' know :P http://en.wikipedia.org/wiki/Crossing_the_Chasm

/kzu

--
Daniel Cazzulino | Developer Lead | MS MVP | Clarius Consulting | +1 425.329.3471

[Greg Young]

You seriously must have not watched the video. I talk about what you
are talking about for an hour.

Precisely one of ashic points "The free audit was greatly appreciated.".
> THAT is an easy sell. Now, and here's the "retarded" thought: if you get the
> project buying 99% of CQRS+ES, where the only remaining 1% is essentially
> the loading of aggregates from events

THIS IS NOT EVENT SOURCING THIS IS AN EVENT LOG.

Perhaps you should bother watching the video where I discuss your
"magical" ORM + ES architecture and discuss why it will be a massive
failure?

Anyways, no conversation I have had with you nor any conversation I
have seen from you in this group has been constructive so I will stop
talking now.

On Tue, Aug 30, 2011 at 10:23 PM, Daniel Cazzulino

[Daniel Cazzulino]

On Tue, Aug 30, 2011 at 23:28, Greg Young <gregor...@gmail.com> wrote:

You seriously must have not watched the video. I talk about what you
are talking about for an hour.

 Precisely one of ashic points "The free audit was greatly appreciated.".
> THAT is an easy sell. Now, and here's the "retarded" thought: if you get the
> project buying 99% of CQRS+ES, where the only remaining 1% is essentially
> the loading of aggregates from events

THIS IS NOT EVENT SOURCING THIS IS AN EVENT LOG.

What part of ***AS IF IT WAS*** did you miss?

 

Perhaps you should bother watching the video where I discuss your
"magical"

what's magical about it? I labeled it "frankenstein" for a reason. 

 

ORM + ES architecture and discuss why it will be a massive
failure?

can't possibly be more failure than plain old n-tier ORM-style app without any event log of any kind. In this case, at least when doom settles, I can run away with the event store log I've been keeping all along and rebuild from the ashes.

 

Anyways, no conversation I have had with you nor any conversation I
have seen from you in this group has been constructive so I will stop
talking now.

The thread was much more valuable because of other's contributions, certainly not yours (I've head you repeat everything you said here over and over in all the videos, so not much value added to the discussion..., and certainly not mine as I'm trying to apply this stuff and haven't been so lucky to come across a PM who's happy to throw out the established tools and practices just because I watched a video from a dude on the web :P. (don't take it personally, but to most companies PMs/CTOs/'IT Guy making the decision', we're all just random dudes that come and go).

I'm pretty sure you talk in the videos of how much the ORM-driven architectures suck, but I'm pretty sure you haven't discussed at any substantial length why using the ORM as the "stateful snapshotter" for your aggregates would be such a retarded idea (while I'm totally convinced it's NOT pure ES, you would still be able to reap all the benefits of ES, including loading aggregates at certain points in time, debugging and testing, trend analysis, etc., because you'd still HAVE the event log, it's just that you'll not be using it for loading the aggregates for the purposes of command processing).

But it just may be that I need to spend another couple days looking for that gem in your videos.

[Rinat Abdullin]

I've been going the same route some time ago. ES+ORM will bite you badly when you suddenly discover that your data (in relational data store and in event streams used to rebuild the views or do whatever reporting) is diverging because of the impedance mismatch. In a sense, this combination is more risky, than pure ES or pure ORM approach, since it leads to inconsistent data.

http://en.wikipedia.org/wiki/Object-relational_impedance_mismatch

Best,

Rinat

[Greg Young]

"I'm pretty sure you talk in the videos of how much the ORM-driven
architectures suck, but I'm pretty sure you haven't discussed at any
substantial length why using the ORM as the "stateful snapshotter" for
your aggregates would be such a retarded idea (while I'm totally
convinced it's NOT pure ES, you would still be able to reap all the
benefits of ES, including loading aggregates at certain points in
time, debugging and testing, trend analysis, etc., because you'd still
HAVE the event log, it's just that you'll not be using it for loading
the aggregates for the purposes of command processing)."

Because you can't prove the deltas from the ORM are the same as the
deltas you generate as events so your log is useless, your db from ORM
is master ... Your events are meaningless. If you go one step further
and try to denormalize a model off of them your models will go out of
sync. Your event log is useless because you can't do anything with it.

But again this is all covered ad nsm both in this thread and in the video.

btw: LMAO@"it's NOT pure ES" .. yes there is a name for the pattern
thats "not pure es" its called an event log and has been around for
decades. Maybe you should stop calling it event sourcing? I think I
will start calling MVP MVC but then call it "not pure MVC" that should
avoid confusion with others right?

On Tue, Aug 30, 2011 at 10:56 PM, Daniel Cazzulino

[Greg Young]

I remember our conversation in Paris well (I don't think I will ever forget it).

On Tue, Aug 30, 2011 at 11:05 PM, Rinat Abdullin

[Rinat Abdullin]

Greg, This conversations is so old. Ce n'est pas cool))

Kzu, on a different note. If you are using ORM with something SQL-based, then technically you already have delta-based event sourcing in your solution already. SQL Server does this in the transaction logs to support shipping and replication. 

If server handles scalability, data management and consistency, then there is no need to redo it in our code. On the other side, if we already do ES, then why do we need crutch in form of some relational DB, that forces its storage quirks upon our business code?

Rinat

[Daniel Cazzulino]

You could prove the correctness pretty easily: just have a background process that replays the events and verifies the resulting state is the same in both cases. Not terribly hard to do if the benefit is convincing the product owner to adopt ES.

I'm assuming when you say "event log" as a pattern, you mean as equivalent to "audit log". If that's the case, then that's not what I'm saying. I'm saying use ES to track all state changes, use an ORM to keep the current state.

Just like Fowler says: http://martinfowler.com/eaaDev/EventSourcing.html

The official system of record can either be the event logs or the current application state. If the current application state is held in a database, then the event logs may only be there for audit and special processing.

I guess he must be wrong too and must have confused the pattern. Note that in the latter case, he's not saying that by doing that (keeping state in a database) you automatically become a retarded and aren't using the event sourcing pattern anymore. (at least that's how I read that).

Rinat: the impedance mismatch argument doesn't fit here completely, I think. If I've already "solved" it via my ORM (as in, accomodated to it, usually ;)), why would it become a problem if the same state changes are also being tracked with events and replay to the same aggregate roots?

/kzu

--
Daniel Cazzulino | Developer Lead | MS MVP | Clarius Consulting | +1 425.329.3471

[Greg Young]

Actually martin has rewritten that page a few times in the past few
years (its preliminary). Next rewrite is supposed to include
replaying as a core function per discussions a few months ago in order
to distinguish from an event log (note not an audit log as it contains
only events though may have the same intent)

> You could prove the correctness pretty easily: just have a background
> process that replays the events and verifies the resulting state is the same
> in both cases. Not terribly hard to do if the benefit is convincing the
> product owner to adopt ES.

So ok let me get this straight ... you are going to write all of the
transformations from your event log to run asynchronously against your
database to see that your event log is correct? Providing you wrote
all of these what would be the point of having the ORM write at all to
the db it being you could just use them to build the db model in the
first place...

And you would be less error prone as the same code would build it both
times .... oh and yes this is in the video as well as of the solution
to the issue...

On Tue, Aug 30, 2011 at 11:48 PM, Daniel Cazzulino

[Greg Young]

btw this is not like a 3 minute side bar in the video. This is a
minimum of a 30-60 minute discussion in it.

[Caleb Vear]

Don't you essentially have the same problem with any kind of snapshoting?  How do you know if it actually matches what the events say?

[Rinat Abdullin]

Kzu,

Unfortunately ORM does not solve the mismatch, it only helps to hide this fact. By using ORM you are essentially performing a lossy transformation on your data:

1. "Flatten" behaviors into the state by applying them to ORM entities

2. Let ORM reverse engineer diffs, given two versions of the state or entities it knows about (before changes and after changes to the entity graphs), build resulting queries and then apply them to the database.

Intent is lost.

Paris conversation, that Greg was referring to, was essentially about my initial ignorance of "lossy" transformation here (after all, how can we loose any data here, right?). Then once in production this came back to bite me, when I discovered that event streams had rather subtle differences from the persisted data. Streams were correct, but they were not full and vice versa. Fixing this in the production was a rather messy experience.

This is similar to how in digital photography, you can't really work with the images, if they are already compressed as lossy JPEG (you get all sorts of artifacts and complex software to work around them). However if you have original RAW files, then there are more options available for image post processing. Same is with audio recordings and videos.

This lossy/raw difference in data can be rather subtle (both in music, digital photography and business domains). However, if this is your core domain, then eventually you might encounter it and feel. Hopefully now you know a little better where to look and what to expect :)

Best,

Rinat

[Greg Young]

You also missed part of Fowler's definition:

"Application states can be stored either in memory or on disk. Since
an application state is purely derivable from the event log, you can
cache it anywhere you like."

You would not meet this part of the definition. But the narrative can
probably be cleaned up a little bit. Email sent to Martin.

On Tue, Aug 30, 2011 at 11:48 PM, Daniel Cazzulino

[Greg Young]

> Don't you essentially have the same problem with any kind of snapshoting?
>  How do you know if it actually matches what the events say?

I never do ... however I my current state is purely derivable from the
event log which means I can throw them away and rebuild them if I find
a problem. Events are the source of truth and everything is based off
of them.

This is not the case when you are using an ORM / event log as which
one is right? You can say that you will write handlers off the event
log to double check that the ORM did it right too but now you just did
double the work as you could just use those handlers in the first
place as opposed to having all the handlers and all the ORM mappings.

Also when you get problems more likely than not its going to be some
difference between the handlers and the ORM as opposed to a bug in
generation.

[Werner Clausen]

@Greg

 

Forgive my ignorance (rather new to these concepts as you might have discovered), but what public videos are you referring to when talking about your "class online"? Is it these on Skillsmatters?

 

--
Werner

[Rinat Abdullin]

Part I:1 video on cqrsinfo.com

Part II: 2 videos available via torrent download (was shared by Adam earlier in this group).

Rinat

[Werner Clausen]

Alright, thanks Rinat.

[Anders Ljusberg]

What if you use an event handler to update the ORM model
(synchronously in the same transaction that writes to the event
store)? This is what I'm currently doing when refactoring a brownfield
project. I'm doing it in a way that will allow me to drop the database
and replay events to get it back, but I'm not using the events for
authoritative state. I.e. everything is based off the events, but they
are not used as the source of truth in day-to-day operations. Is this
a really bad idea? The main reason I'm not doing true ES is that we
don't have a proper DDD model. The ORM is used basically as a data
access layer, not as a domain model. And rebuilding it with DDD is not
feasible for now.

/Anders

2011/8/31 Greg Young <gregor...@gmail.com>:

[Greg Young]

This is troublesome as well. If you want to use the ORM you would have
to have a data vs domain model (eg the event maps to a data model
which then gets saved through ORM). This is a pretty common pattern
that people have been successful with.

If working on a brownfield app though and if its as a transient
stepping stone then its not too worrisome.

" I'm doing it in a way that will allow me to drop the database
> and replay events to get it back, but I'm not using the events for
> authoritative state."

If you can do that then your events are authoritative.

[Anders Ljusberg]

2011/8/31 Greg Young <gregor...@gmail.com>:

> This is troublesome as well. If you want to use the ORM you would have
> to have a data vs domain model (eg the event maps to a data model
> which then gets saved through ORM). This is a pretty common pattern
> that people have been successful with.

The ORM model is pretty much just Entity Framework "Generate model
from database". For the events I'm trying to do it more DDD (but they
are created by the command handlers instead of a domain).

> If working on a brownfield app though and if its as a transient
> stepping stone then its not too worrisome.

The plan is to move to proper DDD eventually, but with the skill level
of most developers I'm working with currently it may take some time.

> " I'm doing it in a way that will allow me to drop the database
>> and replay events to get it back, but I'm not using the events for
>> authoritative state."
>
> If you can do that then your events are authoritative.

I'm hoping that I'll be able to use ES eventually without having to
rebuild the entire system; that would be the ultimate proof.. The
thing I'm worried about the most is that the events I'm creating now
may be spanning more than one Aggregate in a future possible domain
model. And that seems like a bad thing when using ES.

/Anders

[Daniel Cazzulino]

I didn't miss anything. The narrative includes a whole bunch of options and variations (like most patterns, I'd say). Not "complying" to a paragraph doesn't mean you're not using the pattern, given the variation you're using is even explicitly called out in the same document!

The pattern definition itself in Fowler's page is:

Event Sourcing ensures that all changes to application state are stored as a sequence of events. Not just can we query these events, we can also use the event log to reconstruct past states, and as a foundation to automatically adjust the state to cope with retroactive changes.

It doesn't say anywhere that you MUST use that sequence of events to reconstruct the current state of the application. 

So, the following clearly (IMO) fits the definition of ES:

1. Use an ORM to load Customer
2. Invoke Customer.ChangeEmail(string newEmail);
3. Customer does a base.ApplyChange(new EmailChangedEvent(newEmail));
4. Customer handles the event, and updates its own Email property
5. On save:
1. Retrieve the EmailChangedEvent taken from the Customer object's PendingChanges/GetChanges/GetUncommitedChanges and save it to the event store
2. Event bus publishes the events
3. ORM context/session is saved, persisting the current state of Customer and its Email (a "snapshot")

Customer is a domain object, not a data object. It just so happens that ORMs have gotten pretty good at hiding the impedance mismatch and you can just be for the most part ignorant that there is a database altogether.

If you need to drop the database, then you can, as everything is going to the event store anyways, and you can even have a default DB initializer that just detects a dropped DB and replays everything back and saves at the end.

I still don't see the "retarded" part here. It even looks simpler than the replay, snapshots, background process to do it, eventual consistency (if they stress it, the event handlers can run in the same transaction than the main domain object save), etc. While still retaining most (all?) benefits from ES.

Something like this is way easier to sell to a customer. They get to keep their DBA and infrastructure and licenses and what-not, and we get them to buy into a better and more future-proof pattern.

Telling them that the "dude from the video on the web" has Martin's email won't help much, I think :P

/kzu

--
Daniel Cazzulino | Developer Lead | MS MVP | Clarius Consulting | +1 425.329.3471

[Greg Young]

"I still don't see the "retarded" part here. It even looks simpler
than the replay, snapshots, background process to do it, eventual
consistency (if they stress it, the event handlers can run in the same
transaction than the main domain object save), etc. While still
retaining most (all?) benefits from ES."

I said doing CQRS + ES + ORM is retarded. This is not ES.

"If you need to drop the database, then you can, as everything is
going to the event store anyways, and you can even have a default DB
initializer that just detects a dropped DB and replays everything back
and saves at the end."

And you only did twice the work (especially when you consider that you
need to be able to determine if what the ORM did is the same as what
the event log says ... eg reconciliation)

<snip from Martin email>

>I agree with you 100% on this. I'll think about whether I should add a clarification.
>
>
>Martin Fowler
>http://martinfowler.com
>
>
>
>
>On Aug 31, 2011, at 24:31 , Greg Young wrote:
>
> Was just dealing with someone who was getting confused over the paragraph of:

>
> "The official system of record can either be the event logs or the
> current application state. If the current application state is held in
> a database, then the event logs may only be there for audit and

> special processing. Alternatively the event logs can be the official
> record and databases can be built from them whenever needed."
>
> He took it to mean just keeping an audit log = event sourcing (eg just
> writing events to a log then saving with an ORM back to a database).
> I am pretty sure we are in agreement that this would not be event
> sourcing as the current state is no longer derivable from the log but
> wanted to drop you a note.

>
> --
> Le doute n'est pas une condition agréable, mais la certitude est absurde.

On Wed, Aug 31, 2011 at 10:52 AM, Daniel Cazzulino

[Daniel Cazzulino]

From your email to Martin: 

>  as the current state is no longer derivable from the log

That is not correct and far from what I'm saying. The progression of steps I showed you clearly shows that ALL state changes go to the event log, and full replay capabilities would still be available if needed (i.e. for debugging, etc.). 

You keep saying that I'm talking about an audit log when I'm clearly NOT ("He took it to mean just keeping an audit log = event sourcing", no wonder he agreed 100%, I would too if some "retarded" guy was saying that). So, you got Martin to say what you wanted to hear (either through genuine misunderstanding of what I'm saying, or conscious distort, don't care) WRT to the difference between and Audit Log (have you seen his example of that??) and Event Sourcing. Congrats! It is completely irrelevant to what we're talking here.

AND, you are contradicting yourself from what you said just a moment ago:

" I'm doing it in a way that will allow me to drop the database
> and replay events to get it back, but I'm not using the events for
> authoritative state."

If you can do that then your events are authoritative.

Which is *exactly* what I'm saying. The DB/ORM is keeping the current state just like an ever-current snapshot would, nothing more. 

I understand that you don't like it, but if done this way, I fail to see how CQRS + ES + ORM wouldn't fit the respective patterns descriptions and benefits... 

/kzu

--
Daniel Cazzulino | Developer Lead | MS MVP | Clarius Consulting | +1 425.329.3471

[Greg Young]

Try actually implementing your ideas.

Enjoy refactoring your domain model.

On Wed, Aug 31, 2011 at 11:16 AM, Daniel Cazzulino

--

[Daniel Cazzulino]

I will. I feel this is a pretty easy sell.

Thanks.

/kzu

--
Daniel Cazzulino | Developer Lead | MS MVP | Clarius Consulting | +1 425.329.3471

[stacy]

Alternatively, something like this might work:

1) First create a CustomerAR event store. A one-time operation:

if(!Exist(custEvtStore)

Load cust using orm. Create initial events from current state and save to new evt store using ado.net(not orm) - a very simple insert(CustomerARid, evt, version).

And henceforth, do this:

2) Load cust using evt store history.

3) CustomerAR does a ApplyChange(new EmailChangedEvent(newEmail)); 

4) on Repo.Save(CustomerAR - Save to evt store(ado.net) and publish evts.

5) CustomerAR handles evt:

      a) Load Customer using orm

      b) use evts to update Customer obj

      c) ORM context/session is saved, persisting the current state of Customer and its Email (a "read model")

So their read model remains in 3NF and they still use the orm to read and write. The Customer object now becomes the read model only. Steps 2-5 is just Greg's SimpleCQRS. The evt store is your ONE authoritative source. 

It's more work moving(copying) the behaviors to AR's, but they get to keep the orm. Later it will be easy to prove the benefits of denormalization and ditch the orm. Hopefully using ado.net for a simple event store read/write won't be a problem.

Just thinking out loud here, trying to be helpful. 

I know to get organizations to change that sometimes you must ride the horse in the direction it's going. Once you blow away all customer objects and re-create them from the event store, they'll be convinced! 

Another trick - include headers in your events/commands that contain - elapse time in AR or from UI request to final evt. Now you have instant and continuous performance stats that show which operations are slow. Another quick side benefit that they might not currently exist.

Hope this helps!

[Daniel Cazzulino]

Great feedback Stacy!

In that approach, I need to maintain two models: the CustomerAR and the Customer (ORM). I understand that the latter would be more of a readonly model that could even be created in an event handler completely outside of the AR, but it seems like there would be a lot of duplication.

And I would still need to get into the snapshots, eventual consistency, etc., so I haven't gained much in that case, other than telling the customer "you wanted a populated normalized database, here it is!"... I'm actually trying to get something simpler too if possible.

Good idea about the headers. Was planning to do that on the command side (having a "created" date and "executed" date). I didn't think it was going to be needed for events though. Not sure an event has a concept of an "elapsed" time... It would definitely have a time (when it happened, of course), and I was thinking about correlating the commands to the events via the simple .NET ActivityId that can be set for the current call context via Trace.CorrelationManager. That would allow to build some useful visualizations also of the effects of certain commands on the system...

How would you know when an event has finished being "handled" by all handlers? (especially since they might be async?)

Thanks

/kzu

--
Daniel Cazzulino | Developer Lead | MS MVP | Clarius Consulting | +1 425.329.3471

[stacy]

Yes, there will be duplication but hopefully only temporarily until you get buy-in by 1) not immediately taking away their orm, and 2) proving some of the benefits.

How would you know when an event has finished being "handled" by all handlers? (especially since they might be async?)

My setup is a little different in that I use Sql Service Broker for transactional messaging and running commands on the AR. So my entry point looks something like this:

public XmlDocument ProcessMsg(XmlDocument cStream, XmlDocument eStreamHistory )

        {

            

            if(eStreamHistory != null)

                _storage.PopulateEventStoreWithHistory(eStreamHistory);

            

            return cStream == null ? null : ProcessMsg(cStream);

        }

               

        private XmlDocument ProcessMsg(XmlDocument cStream)

        {

            var cmds = _storage.GetCommandsFromStream(cStream);

            Send(cmds);

            return GetUncommittedEvents();

        }

So I hit the eventstore before hitting the command handler. In the eventstore, in the ctor I do this:

_processingDuration = Stopwatch.StartNew();

Then in SaveEvents(Id,...),  I add headers to the uncommitted stream of events like:

var emHeaders = new Dictionary<string, object>

                                {

                                    {"ProcessingStart_utc", _processingStartDT},

                                    {"ProcessingDuration", _processingDuration.ElapsedMilliseconds}

                                                                      

                                };

Partial of my event stream shows 377 ms of processing time:

<a:KeyValueOfstringanyType>

                    <a:Key>ProcessingDuration</a:Key>

                    <a:Value i:type="b:long" xmlns:b="http://www.w3.org/2001/XMLSchema">377</a:Value>

</a:KeyValueOfstringanyType>

Now I've captured how long processing that command has taken to the event store with very little work, thanks to this pattern. I just did this on the fly in 5 min when you mentioned above how time consuming reflection was in SimpleCQRS. The larger point though, is the flexibility and ease of use your staff will soon discover once you get them over the hump of trying CQRS-ES - any way you can!

So you might be able to help sell this by showing real-time data - for instance, a chart of where all the processing time is going. Now you can refactor to save on server resources by automatically, and continuously, discover hot spots in "production." And it comes for free with ES. 

That's a good money argument for CTO's who want to stay ahead of the game, and become ultra responsive to customers and other departments like marketing. Only ES lets you go back in time, and do it again, by answering the question you "wish" you had asked 6 month's ago. It's like the CTO will now have eyes on the back of his head.

Not sure if you viewed the correct Greg online course video. But there's a torrent put on this board, within the last 2-3 months,  where he goes into much of the "business" aspects that I'm sure you'll find helpful. I know it's not easy to sort thru all those hours to find the sugar spots. But he certainly makes a strong case, IMHO.

Hope this helps!