[ANN] Scritchy - CQRS without the plumbing

[Tom Janssens]

I would like your opinions on yet another attempt for a CQRS lib/framework. I know most of you are opposed to this, but IMO not being able to have a simple bootstrapper makes building a CQRS app hard for beginner/intermediate developers.

The main motivation for this lib/fw is making the CQRS bootstrapping as effortless as possible (similar effort that RavenDB requires to get started: include a dll, add some config and you are ready to go).

The current implementation only uses an InMemoryEvenstore and no snapshots, but you should get the idea.

The source can be found at  can be found over at https://github.com/ToJans/Scritchy and a demo with a small explanation at http://scritchyexample.apphb.com/ 

I have also attached a screenshot of the app running over at apphb.

Your opinion is appreciated !

Thanks !

Tom

[Tom Janssens]

I have created a small (completely unprepared) screencast where I develop a working CQRS app in exactly 15 minutes (if well prepared, I could probably do it in 5) with this framework.

The video is here: http://www.youtube.com/watch?v=5DKTFZD3hu8

[jason turim]

the volume is very low on the video, i can't hear anything being said.  i had to give up after a minute.

[Tom Janssens]

Hey Jason,

Fair point; I will create a proper one. (This was more a quick & dirty one, and the idea was that the code should be enough to get you going.) 

Thank you for your time !

[Paco Wensveen]

This can be useful as another example, but cannot be used in production because too many things are missing (the plumbing?).

[Tom Janssens]

Hey Paco, 

Thank you for your insight. Apparently this demo does not transfer the message I wanted to transfer.

The plumbing is not there, because everything is wired up automatically for you.

FYI, that's the whole idea behind this framework; nothing is missing in this example; when you press F5 at the end of the demo everything just works...

The plumbing is done for you by the framework based on some conventions/assumptions.

When I call the constructor "var bus = new ScritchyBus()" - typically upon starting your app - this is what happens in the HandlerRegistry:

I scan all the inmem assemblies for classes & properties:

- First I find all ARs (classes derived from Scritchy.Domain.AR) (class BankAccount)

- Second I find all the commands (public void methods in the AR's where there exists a class with the same name as the method that contain a property with the same name as the AR+"Id" (RegisterBankAccount {BankAccountId,OwnerName}

- Then I find all public void methods on all public classes which start with "On" (OnBankAccountRegistered)

- Then I look if I can find a class named like the method, but without the "On" prefix (class BankAccountRegistered), which would be a possible event.

- Now I have a registry of all AR's, events, commands and handlers in a registry.

Note: event handlers should be stateless in this framework, but support IOC through the bus construction (except for the AR ones of course).

Note 2: there might be some false positives on the events & event handlers, but these event handlers will never be called as the commands will only generate domain events.

When you now call bus.RunCommand(new RegisterBankAccount {BankAccountId="account/1",OwnerName="blah"})

- An instance of BankAccount is being loaded with ID = "account/1"

- All previous events from the eventstore for the BankAccount with this Id are being replayed on this instance, so we have an up to date snapshot.

- The command is passed in the command method

- The generated events get stored in the eventstore (which is inmem by default, or using a database if you set the connectionstring "eventstore" in the web/app.config)

- If you did not add the DoNotAppyEvents = true parameter in the bus constructor, the different handler types are loaded one by one, and the On"Eventname" methods will be executed for every single handler type and event

- If you did add the DoNotAppyEvents = true in the bus constructor, you can call bus.ApplyNewEventsToAllHandlers() whenever you feel like updating your views

- There is both a synchronous and an asynchronous method available for both RunCommand and ApplyNewEventsToAllHandlers with a callback (might not be production-ready ;) )

There is a lot more to it (like system wide locks etc), and I assume there is still a lot more todo, but it works...

Thank you for your remark; I now know what I should definitely include in the demo.

Tom

[Shawn Hinsey]

I took a brief look at the code. I think it's an interesting approach, and the use of signalr is definitely interesting, but the locking really concerns me. 

In my view, one of the more powerful aspects of CQRS is the ease with which you can scale it across multiple logical and physical deployment nodes due to a fundamentally shared-nothing encouraged by the notion of eventually consistent read models. 

Of course, this varies with implementations, but locking the world to process an event seems indicative of a larger conceptual problem to me, rather than something that can just be fixed later. Can you elaborate on why you went in this direction?

[Tom Janssens]

Hey Shawn,

Thanks for the compliment !

I use locking when:

- I run a single command: I create a system-wide lock for that specific AR-ID so that only one command can be executed for the same AR simultaneously.

- I process an event: I lock that specific handler type so you can not have threading issues in your stateless handler classes

- I save an Event on the InMemoryEventstore

- When I add en enumeratorcontext to the dictionary that remembers the eventstore index that have not been processed yet by the specific handlers/AR's (so snapshotting should be trivial to add)

- When I increment the index for the enumeratorcontext

FYI the whole framework is properly designed, so it should also be rather trivial to replace the eventstore for example with Jonathan Oliver's eventstore, or dispatch the eventhandling etc; the bus is there for easy bootstrapping, but creating your own should be trivial; this is the constructor of the bus:

        public ScritchyBus(Func<Type, object> LoadHandler = null, IEventstoreAdapter adapter = null, bool DoNotApplyEvents = false)

        {

            if (LoadHandler == null)

                LoadHandler = x => Activator.CreateInstance(x);

            Registry = new ConventionBasedRegistry();

            EventStore = new Scritchy.Infrastructure.Implementations.EventStorage.EventStore(adapter: adapter);

            resolver = new HandlerInstanceResolver(EventStore, Registry, LoadHandler);

            Bus = new CommandBus(EventStore, Registry, resolver);

            applier = new EventApplier(EventStore, Registry, resolver);

            this.DoNotApplyEvents = DoNotApplyEvents;

        }

The idea is for any dev (architect or not) to get started with this simple framework, and as you go along you can replace parts of the backend with more mature/performant/better scalable, completely different infrastructure, as your infrastructure code is completely decoupled from your domain code...

If you have any suggestions or tips, do not hesitate to share them... I am glad you guys are helping me out.

Tom

[Kelly Sommers]

The one thing I've learned through the LMAX Disruptor RingBuffer awareness is that I've more often than not when writing concurrent code let a thread go from top to bottom of the code base introducing a lot of complexity for no good reason.

Taking a commands thread straight through the domain may be introducing far more complexity in my domain than I need. This could introduce a lot of lock contention in high throughput domains.

After performance analysis learning that my domain can execute several millions of operations faster than the input or ouput I decided to try to follow LMAX's concept. A single threaded domain and concurrency exists at the input and output boundaries.

Later,

Kell

[Tom Janssens]

Hey Kell,

That's true; that is why I offer (and always will offer) both a synchronous and an asynchronous method to execute both commands and apply events to event handlers.

Regarding the locking: it's pretty straightforward in this framework; the locks I mentioned in the previous post are the only ones...

Start with a synchronous approach and only switch if necessary... (n=1)

A simple thing to do is IMO split it up in 3 threads:

- UI/App

- Command handling

- Event handling

Tom

[Sharas]

Anybody can explain how disruptors work here? I'm so lazy I cant even read those papers published... crap...

[Tom Janssens]

Sharas,

Attempt #2 ;D

It is a bit off-topic, but the architecture is based on a ring buffer to avoid locks, by having multiple handlers which progress in the same direction on the ring buffer; the only thing they have to be careful with is that certain handlers can not pass other ones (which would cause a lock); Fowler has a pretty good description here: http://martinfowler.com/articles/lmax.html#InputAndOutputDisruptors .

Tom.

[Shawn Hinsey]

I think Fowler's explanation is really thorough and straightforward, and I don't think I could describe them in a more straightforward way, except to say that you can think of a disruptor as a variation of a queue, but that doesn't really tell the story. You can see some interesting discussion here http://news.ycombinator.com/item?id=3173993. It includes comments from LMAX.

[Sarunas Mazylis]

Thanks @Shawn, @Tom. Will have to read it... :D

[√iktor Ҡlang]

It's possible to create a lockless parallel domain using Actors. As an alternative to LMAX.

Cheers,
v

[kell.s...@gmail.com]

Very true! I think Ring Buffers and actors solve two different purposes though and depends where your bottleneck is. 

I'm finding the bottleneck more on the receiving or sending from a communications boundary than my actual domain logic. My domain logic tested without any network involved is extremely fast which means I just need a queue mechanism of some sort when retrieving commands. So far my domain far exceeds the messaging capabilities so 1 thread on my entire domain is working similar to LMAX (so far). 

I think Actors may make sense when your domain itself needs concurrency. 

That's my thoughts anyways :)

Later,

Kell

[Tom Janssens]

For your amusement:

"Muahaha, I've written an LMAX disruptor. I shall ask 1 million dollars" - Dr. Evil.

https://github.com/ToJans/Scritchy/blob/master/Scritchy.Infrastructure/Implementations/Disruptor.cs

[Sławek Sobótka]

So in this approach we do not need CommandHandlers...
Simply AggregateRoots handle commands.

I consider this as shooting int the foot, because in this approach we
get rid of Application Layer. What are consequences of trivializing
layers?
1. It can not be seen in this "hello world" sample, but in real cases
ARs will be overloaded with Application (not Domain) logic:P
But I can imagine systems where there is no App logic, only Domain
logic, so this approach is potentially useful somewhere...
2. What about scenarios where more than one AR (same BC) need to play
to fulfill command? I know about scalability, but what if requirements
demands this?


Cheers,
Sławek Sobótka

[Colin Yates]

Hi :)

Sent from my iPad

On 12 Nov 2011, at 18:54, Tom Janssens <d4s...@gmail.com> wrote:

> Hello

[Sarunas Mazylis]

good day

[Tom Janssens]

Hello Sławek,

Thank you for your comment; I have replied inline below:

Op zaterdag 12 november 2011 11:37:30 UTC+1 schreef Sławek Sobótka het volgende:

So in this approach we do not need CommandHandlers...
Simply AggregateRoots handle commands.

That is a very good  description ! Thank you  !

I consider this as shooting int the foot, because in this approach we
get rid of Application Layer. What are consequences of trivializing
layers?

I am not sure that I understand what you are saying here... In the examples you still have the controllers etc ?

 

1. It can not be seen in this "hello world" sample, but in real cases
ARs will be overloaded with Application (not Domain) logic:P
But I can imagine systems where there is no App logic, only Domain
logic, so this approach is potentially useful somewhere...

Could you elaborate with an example on this ? I assume the logic would still be somewhere in the controller (in case of MVC)

 

2. What about scenarios where more than one AR (same BC) need to play
to fulfill command? I know about scalability, but what if requirements
demands this?

Good remark; On the technical side, there is currently nothing that holds me back to have a command handled by multiple AR types. However; I was not aware this was a valid use case, but maybe it might be a good idea to make this possible...

Can others give me their opinion on this as well ?

Tom

 

Cheers,
Sławek Sobótka

[Tom Janssens]

Howdy !!!

Op zaterdag 12 november 2011 20:00:46 UTC+1 schreef Sharas het volgende:

[Sławek Sobótka]

> > So in this approach we do not need CommandHandlers...
> > Simply AggregateRoots handle commands.
> That is a very good  description ! Thank you  !

Thanks, I just love to name things:)

> > I consider this as shooting int the foot, because in this approach we
> > get rid of Application Layer. What are consequences of trivializing
> > layers?
>
> I am not sure that I understand what you are saying here... In the examples
> you still have the controllers etc ?

Layers is one thing, MVC is another thing.
For example we have layers:

1. Presentation - few totally different presentation technologies)
//Write Stack
2. Application Logic - Command Handlers or Services with AOP (if You
are Java guy emotionally connected with Spring/EJB/whatever;), that
simply publish API to the outer world
3. Domain Logic - DDD Building Blocks
4. Infrastructure

In Presentation Layer You can have changeable:
- web stuff, based on some MVC framework, where View is some
templating/renderable components/etc, Controller is about handling
POST/GET, calling model and redirecting to relevant View, Model - in
general lower layer
- web services
- some kind of remoting (for Mobile apps for example)
- some kind of integration with some ESB
- ...

So from my point of view (Java guy) controller is an artifact which is
very specific for given presentation technology.
So switching Presentation we loose some common Application Logic that
resides in controllers
I know that in Smalltalk in '70 it was different understanding of MVC,
so maybe common understanding is our problem.

Sample of App Layer in further lines...

> > 1. It can not be seen in this "hello world" sample, but in real cases
> > ARs will be overloaded with Application (not Domain) logic:P
> > But I can imagine systems where there is no App logic, only Domain
> > logic, so this approach is potentially useful somewhere...
>
> Could you elaborate with an example on this ? I assume the logic would
> still be somewhere in the controller (in case of MVC)

Briefly described above. App Layer is still something between
Presentation Layer (C) and Domain logic.

> > 2. What about scenarios where more than one AR (same BC) need to play
> > to fulfill command? I know about scalability, but what if requirements
> > demands this?
>
> Good remark; On the technical side, there is currently nothing that holds
> me back to have a command handled by multiple AR types. However; I was not
> aware this was a valid use case, but maybe it might be a good idea to make
> this possible...

What about such a UC:
When confirming Order we must generate an Invoice?
Let's assume that Order and Invoice are both in "Sales" Bounded
Context and let's assume they must be calculated in one transaction.
Of course in real ERP Aggregates would be modeled in totally different
shape and boundary, but lets abstract from domain and focus on
technical issues (because we are discussing a kind of framework here).


Sample code:
http://code.google.com/p/ddd-cqrs-sample/source/browse/trunk/ddd_cqrs-sample/src/main/java/pl/com/bottega/erp/sales/application/commands/handlers/SubmitOrderCommandHandler.java

Some visualization:
http://prezi.com/hi2dmhfej9zu/ddd-cqrs-sample/

Some reading introducing to mentioned code:
http://code.google.com/p/ddd-cqrs-sample/wiki/DomainDrivenDesignBusinessDeveloperMentalModel#Application_Architecture
http://code.google.com/p/ddd-cqrs-sample/wiki/DomainDrivenDesignBusinessDeveloperMentalModel#Application_Layer


Cheers,
Sławek Sobótka

[Tom Janssens]

Op zaterdag 12 november 2011 22:33:52 UTC+1 schreef Sławek Sobótka het volgende:

Not sure whether this is the preference, but I usually do not add an extra layer between the controller and the command bus, except when I might have some reusable code. 

In case I do have some reusable code, I write something that you might name an application service. In most of the cases I do not need an extra abstraction ?

 

> > 1. It can not be seen in this "hello world" sample, but in real cases
> > ARs will be overloaded with Application (not Domain) logic:P
> > But I can imagine systems where there is no App logic, only Domain
> > logic, so this approach is potentially useful somewhere...
>
> Could you elaborate with an example on this ? I assume the logic would
> still be somewhere in the controller (in case of MVC)

Briefly described above. App Layer is still something between
Presentation Layer (C) and Domain logic.

True, but as stated, why add another layer which is usually just a wrapper around bus.RunCommand(x); ?

 

> > 2. What about scenarios where more than one AR (same BC) need to play
> > to fulfill command? I know about scalability, but what if requirements
> > demands this?
>
> Good remark; On the technical side, there is currently nothing that holds
> me back to have a command handled by multiple AR types. However; I was not
> aware this was a valid use case, but maybe it might be a good idea to make
> this possible...

What about such a UC:
When confirming Order we must generate an Invoice?
Let's assume that Order and Invoice are both in "Sales" Bounded
Context and let's assume they must be calculated in one transaction.
Of course in real ERP Aggregates would be modeled in totally different
shape and boundary, but lets abstract from domain and focus on
technical issues (because we are discussing a kind of framework here).

I usually assume some kind of interaction between ARs, so handle this with a saga ? 

But I understand what you are saying, and since it was only a small implementation detail on my part and I have changed the handler, so it now allows you to execute a single command on multiple types of AR's. Here's an example that will exec for both AR's:

class ConfirmOrder

{

  string OrderId;

  string InvoiceId;

  DateTime OrderDate;

}

class OrderId: AR

{

   void ConfirmOrder(...)

}

class InvoiceId: AR

{

   void ConfirmOrder(...)

}

 

This will now execute both. Please note that I have currently not added a transactional boundary, but it is on the todo list.

 

Sample code:
http://code.google.com/p/ddd-cqrs-sample/source/browse/trunk/ddd_cqrs-sample/src/main/java/pl/com/bottega/erp/sales/application/commands/handlers/SubmitOrderCommandHandler.java

Seeing this code made me realize that I might want to extend my framework with 2 extra conventions:

One for command validation before execution (prefixed by "Can") and one for decoration of commands (named "Decorate") and an extra method on the infrastructure: "IsValid"

public class Blah

{

  ISessionContext sContext; // hungarian FTW ! ;)

  public Blah(ISessionContext sContext)

  {

    this.sContext = sContext;

  }

  IEnumerable<string> Validate(ConfirmOrder cmd)

  {

    if (!cmd.Cow) yield return "mooooooh";

  }

  void Decorate(ConfirmOrder cmd)

  {

    cmd.ConfirmedBy=sContext.UserName;

  }

}

Some visualization:
http://prezi.com/hi2dmhfej9zu/ddd-cqrs-sample/

Some reading introducing to mentioned code:
http://code.google.com/p/ddd-cqrs-sample/wiki/DomainDrivenDesignBusinessDeveloperMentalModel#Application_Architecture
http://code.google.com/p/ddd-cqrs-sample/wiki/DomainDrivenDesignBusinessDeveloperMentalModel#Application_Layer


Cheers,
Sławek Sobótka

Thanks for the tips ! 

[Tom Janssens]

Doh! The first examples should say:

class ConfirmOrder

{

  string OrderId;

  string InvoiceId;

  DateTime OrderDate;

}

class Order: AR

{

   void ConfirmOrder(...)

}

class Invoice: AR

{

   void ConfirmOrder(...)

} 

I think I might need some sleep :D

[@yreynhout]

I'd say having multiple collaborating aggregates is quite common,
eventhough only one aggregate gets affected/changed. You can't assume
full consistency for the other aggregates, only eventual consistency
(i.e. they can be changed in the meantime). You could move some of the
internal state of the other aggregates into the command (i.e. surfaced
in the readmodel, displayed to an end-user, copied into the command.
Note this is a matter of trust/security), but that depends on the use
case, and is somewhat counterintuitive to encapsulation.

It's just easier to have an explicit command handler, IMO. The command
contracts and types are not part of the domain type system (the former
are about wire representation, not behavior). As such, it's a slippery
slope towards high coupling/versioning problems. But that's not to say
this is going to affect you/everyone. Really depends on the size of
the system/BC (the problem with simplistic examples is they offer poor
guidance for larger systems).

As for logic, it should be in neither the controller, nor the command
handler. Belongs in domain services or objects.

[Sławek Sobótka]

I need to explain, what I meant by "common logic that resides in
controllers".
You mentioned about:
> bus.RunCommand(x);
ok, that's place for technical stuff.

But my point that we need Application Layer to model Use Cases/Stories
(steps of UCs/Stories)

For example, we have one use case, where we need to do only:

order.confirm();

and other use case where we need to do:

order.confirm();
//automatic invoicing
Invoice invoice = accounting.issueInvoice(order);


So having ApplicationLayer I can:
- model different use cases that orchestrate domain actor
- i can add add additional (application specific) logic like: depends
on who is looged on, depends on server mode (demo, full version),
etc... lots of logic which has nothing to do with domain
All this can be called by different presentation technologies - not
only controllers of some web framework.

I would like to express a general idea here: "inversion of coupling".
My aggregates are free of dependencies to Application stuff. Why?
Because Application orchestrates them instead of Aggregates asking
some Application Context.


In approach presented in Scritchy examples:
Client actually shoot straight to the Aggregates. Therefore all (bad)
coupling to Application stuff must be introduced somewhere in the
domain (Aggregates, Listeners, Sagas). Because some domain actor must
ask a question: ok, do we issue a Invoice at this case or not?
Lets assume that it is not always acceptable scenario: Order fires
OrderCOnfirmedEvent and we always react on this by issuing an Invoice.
No, business is not so trivial. There are may different cases, so we
need some place to model this cases.
Application Serivces are or course procedural artifacts. I'm not a big
fan of procedural approach, but in most cases we simply need them to
model something what is above Domain.

On the other hand, as I said before: I can imagine systems which has
no App Logic and shoothing straight to the Aggregates is perfectly ok.


Of course, We mas say: command can have additional boolean parameter:
issueInvoice. Or we can have 2 commands: ConfirmOrder,
ConfirmOrderWithAutomaticInvoicing
Sometimes it's ok (if user for example checks checkbox) bout sometimes
this is even worse:P
We put application logic to the client, which should be for example
just a s stupid html rendered with only presentation logic (how to
draw fancy DIVs;).


Cheers,
Sławek Sobótka

[Piotr Buda]

What determines that an invoice has to be automatically generated? I
can see this as invoking a chain command - ConfirmOrderCommand and
GenerateInvoiceCommand - if your use case is only about confirming an
order, simply send the ConfirmOrderCommand and it will be handled by:
- ConfirmOrderCommandHandler,
- confirmOrder(ConfirmOrderCommand) in AR

BUT when you need to generate an invoice AFTER confirming an order,
why not having a chain:
- ConfirmOrderAndGenerateInvoiceCommandHandler (simple chain of
ConfirmOrderCommand that in handle() method executes next.handle(),
where next is GenerateInvoiceCommandHandler),
- confirmAndGenerateInvoice that actually calls the confirmOrder
method and then generates an invoice

(note: I very much dislike class names and method names with 'and' in
them, this was just an example)

[Piotr Buda]

Oh, and about stupid presentation layer - just put two buttons there -
one that confirms order and one that confirms order with invoice... or
use checkbox... or whatever! How do you want to distinguish such use
cases if you don't know what client actually wanted??? If you don't
want -any- logic there, provide simple way of executing use case that
user wants (additional buttons, confirmation dialogs, emails,
anything)

On 13 Lis, 13:32, Sławek Sobótka <sso...@gmail.com> wrote:

[Sławek Sobótka]

Piotr, You are right...
...but only if intention of which use case should be performed is
expressed by User.

So user clicks one or another button, than presentation tier prepares
proper command/commad_structure.
Thats obvious. And have described this class of the problem at the end
of the last post.

But I have described also another class of the problem in the middle
of the last post: let's assume that detailed story scenario it depends
on some application context: ex: logged user (premium users have more
automatic stuff or maybe less in order to give the more freedom), some
configuration stuff etc.

Believe me, I'm not so smart to figure out need of Application Logic
Layer;P
I just red about it from people I respect (Evans, Fowler) and it
brings value - in problem classes that I have faced.

btw: During dinner another though came to my mind: DDD blue book
emphasis "make explicit what is implicit" mantra. So Application Logic
layer is just an artifacts that can be used to model UseCase/Story
steps explicit (assuming that there are some steps). If we don't do
that explicit we may need to do that in implicit way hiding logic in
not proper places, volatilizing SRP, encapsulation, increasing
complexity and decreasing testability. That's what it's all about. But
if some technique does not bring value in Your case - just leave it.

Cheers,
Sławek Sobótka

[Tom Janssens]

Op zondag 13 november 2011 11:46:24 UTC+1 schreef @yreynhout het volgende:

I'd say having multiple collaborating aggregates is quite common,
eventhough only one aggregate gets affected/changed. You can't assume
full consistency for the other aggregates, only eventual consistency
(i.e. they can be changed in the meantime). You could move some of the
internal state of the other aggregates into the command (i.e. surfaced
in the readmodel, displayed to an end-user, copied into the command.
Note this is a matter of trust/security), but that depends on the use
case, and is somewhat counterintuitive to encapsulation.

In the past I just launched two commands, one for each BC, so this makes my code a little bit more obvious, and my naming a little bit better (before I had to use 2 different names, so I usually force-fed them to the domain experts ;) )

Passing internal state from the command is not possible with the current framework, unless you would do some unorthodox things. In the current framework, persistence is completely handled by the infrastructure. However, as you already stated, passing in state in commands is a somewhat unconventional approach that probably should be avoided for devs beginning with CQRS (which is my main target audience here).

 

It's just easier to have an explicit command handler, IMO. The command
contracts and types are not part of the domain type system (the former
are about wire representation, not behavior). As such, it's a slippery
slope towards high coupling/versioning problems. But that's not to say
this is going to affect you/everyone. Really depends on the size of
the system/BC (the problem with simplistic examples is they offer poor
guidance for larger systems).

True. But, then again, I do think this is probably not an issue in Scritchy's target audience. 

Let me tell you the main motivation behind this framework:

About a year ago, I joined one of my mates in a new web platform start-up. He asked me to help him out with the proper software architecture, so I suggested CQRS. 

To cut a long story short, this choice was IMO one of the reasons the start-up never got past the initial phase, here's why:

I offered a few interfaces they needed to implement, wire up all the handlers etc. The devs were more used to CRUD-like apps. They considered it way to counterproductive, so they hacked their way around it, forcing stuff in ways it wasn't supposed to be forced. They did not care about "proper arcxhitecture, or things like testablility, or even TDD"; they just wanted a to get a working protoype in as few lines of code as possible...
Ever since then I have been searching for a way to make CQRS "for the masses/without the plumbing/[add your own blah here]".

Here is why I think Scritchy might be the perfect way to get started with CQRS:

- Drop dead stupid conventions: Inherit AR's from a root class/Class names must match method names/method parameter names must match with class properties.

- Very high business value/LOC ratio: all the code you write has direct business value, no need in writing code to do the plumbing.

- Persistence Ignorance: because the ARs and eventhandlers are being instantiated by the infrastructure, you can not have dirty hacks (unless you start getting really ugly). AR's must have a parameter-less constructor, and stateless event handlers use proper IOC.

- Initial speed: getting started with CQRS is as simple as : install-package Scritchy 

- Not having to explain why everything is wired up the way it is: this might seem a stupid reason, but IMO this is one of the main reasons of our failure. As they were already in the project before I got involved, they did not understand why they had to write all this boilerplate code, just to update a single field/execute a simple command, as the only thing they did before was calling an update method in the controller. With this whole new architecture they had to write like umphteen lines just to increase an amount... The result was they simple started updating the viewmodels from the controllers, which I had to fix again, which slowed us down, which....

By having all the wiring done by the framework, you gain 2 advantages: less boilerplate code (as mentioned before), but also making it harder to update viewmodels in a way they are not supposed to be updated. 

The whole idea behind Scritchy is to allow you to get started with CQRS without understanding why everything is wired up the way it is. In my experience, I started understanding the beauty of CQRS and the why's once I started using it. The more I used it, the more my understanding evolved. (And I have to admit I had produced some rather fruitless & ugly attempts before I truly  understood how to build something acceptable, in fact, my understanding is forever evolving, as it should be IMO.)

If Scritchy is starting to hold you back, replacing the parts that hold you back should be trivial. For example, the convention-based auto-wiring is setup here, and could be easily replaced:

https://github.com/ToJans/Scritchy/blob/master/Scritchy.Infrastructure/Configuration/ConventionBasedRegistry.cs

https://github.com/ToJans/Scritchy/blob/master/Scritchy.Infrastructure/Implementations/HandlerRegistry.cs

 

As for logic, it should be in neither the controller, nor the command
handler. Belongs in domain services or objects.

I think this part is simply irrelevant to Scritchy in itself... I think everyone can do this in the way they think it's a best fit.

Thank you for your contribution Yves, it has been really helpful !

Tom

PS: Spring gerust eens binnen als je in de buurt zit en tijd hebt !

[@yreynhout]

Just wanted to add that 2 collaborating aggregates does not mean
multiple bounded contexts. Only one can be changed, but you sure can
use multiple aggregates get that change to happen.

> setup here, and could be easily replaced:https://github.com/ToJans/Scritchy/blob/master/Scritchy.Infrastructur...https://github.com/ToJans/Scritchy/blob/master/Scritchy.Infrastructur...

[Tom Janssens]

FYI, based on everyone's feedback I am creating a wiki/FAQ over at github: https://github.com/ToJans/Scritchy/wiki , so keep them questions/remarks/suggestions coming please!

[Sławek Sobótka]

> - Not having to explain why everything is wired up the way it is: this
> might seem a stupid reason, but IMO this is one of the main reasons of our
> failure. As they were already in the project before I got involved, they
> did not understand why they had to write all this boilerplate code, just to
> update a single field/execute a simple command, as the only thing they did
> before was calling an update method in the controller.

Tom, thanks for presenting wider context of the whole idea.

1. CqRS is great, but if You try to use it for solving CRUD class
problems... faceplant:)
In DDD You have two strategic concepts: Bounded Contexts and Domain
Destination. Each of them may be solved by different application
architecture approach. One which introduces value and is most
productive for example. Therefore one part (Core Domain/Non trivial
context) of the system can be designed using CqRS, another using some
quick and dirty CRUD framework/scaffolding generator, based on one
layer and naive assumptions - thats ok.

2. CRUD does not need to involve expensive expert developers who need
to understand all new sexi ideas.
Look from this point of view: when parent gives 2 years old child
scissors as a toy than bleeding is parent's fault, not child's.

[Nuno Lopes]

Is there an online forum where the Disruptor is being actively explained / discussed?

Cheers,

Nuno

[Daniel Yokomizo]

On Tue, Nov 15, 2011 at 3:22 PM, Nuno Lopes <nbpl...@gmail.com> wrote:
> Is there an online forum where the Disruptor is being actively explained / discussed?

http://groups.google.com/group/lmax-disruptor

> Cheers,
>
> Nuno

[Kelly Sommers]

Here's the .NET port of the LMAX Disruptor which also has LMAX contributors committing code. This is the one I'm using in my .NET projects.

http://code.google.com/p/disruptor-net/

[Nuno Lopes]

Hi, sorry to post here the question but here it goes.

From a quick read on the articles, It seams that disruptor was created to eliminate the latency in exchanging data between concurrent threads, yet the biggest success case is when we use it with a single thread, in which case I would assume that wouldn't be such a problem. I'm sure I'm missing something, can anyone clarify this for me?

Nuno

[Daniel Yokomizo]

On Tue, Nov 15, 2011 at 4:26 PM, Nuno Lopes <nbpl...@gmail.com> wrote:
> Hi, sorry to post here the question but here it goes.
>
> From a quick read on the articles, It seams that disruptor was created to eliminate the latency in exchanging data between concurrent threads, yet the biggest success case is when we use it with a single thread, in which case I would assume that wouldn't be such a problem. I'm sure I'm missing something, can anyone clarify this for me?

There's only a single thread processing [nonblocking] domain logic but
many threads doing [blocking] IO. With another architecture (e.g.
actors) you don't bother with possibly blocking code because you have
many threads, which become a bottleneck on runtimes not designed for
concurrency. If they used Erlang, for example, with very cheap
processes they would not bother with nonblocking code.

> Nuno

[Tom Janssens]

Op dinsdag 15 november 2011 19:07:05 UTC+1 schreef Kelly Sommers het volgende:

Here's the .NET port of the LMAX Disruptor which also has LMAX contributors committing code. This is the one I'm using in my .NET projects.

http://code.google.com/p/disruptor-net/

LOL ! I had no idea that existed... From the looks of it (http://code.google.com/p/disruptor-net/wiki/CodeExamples) I am pretty close with my approach, event though in my case I assume writers and readers are on a at least one distinct thread (hence the Thread.Sleep(0) to switch context). 

Then again, mine's only about 50 LOC, while theirs is probably A LOT more....

Mine also does not do the initialization by itself, but it does support as many readers/writers as you wish (in this case up to 32 using a const).

Has anybody tried using mine ? I am just being curious if it would work, especially since the LOC is so much smaller. (FYI the code is here: https://github.com/ToJans/Scritchy/blob/master/Scritchy.Infrastructure/Implementations/Disruptor.cs )

[Nuno Lopes]

But does the 6 million figure accounts for all consumers in the chain or just the BLP?

If it just the BLP then we need also to assume that a single producer is able to dispatch far mor the 6 million request to the Disruptor to keep the BPL busy. So it must multithreaded.

Is this a correct assumption?

Sent from my iPhone

[Kelly Sommers]

Also keep in mind Disruptor implements what LMAX calls "machine sympathy". If I remember right they pre-allocate objects to optimize for less GC hit with the trade off of memory. They also store things in block sizes to utilize how modern CPU's fetch from the on-die cache.

That's what they describe for their Java implementation. I saw some in the .NET port but I'm not sure how these optimization compare on the CLR versus the JVM.

[Sarunas Mazylis]

I'm guessing, producer can read of the network fast enough so bandwidth doesn't get saturated. Just a guess...

[Tim Gebhardt]

They compare favorably.  In fact, the CLR has reified generics which for certain scenarios allows certain types of data in the Disruptor ring buffer to avoid the extra reference-following memory jump.  Java has no such equivalent functionality.

But the high-level concepts of pre-allocating everything on startup are the same between both systems: The Disruptor code pre-allocates everything it'll need so the only memory allocations should only come from your domain logic, not the scaffolding (the Disruptor) that's running it.  The #1 goal of the Disruptor pattern is reduce the latency of processing events and random garbage collections introduce jitter that ruins your latency numbers.  The fact that reducing latency also helps total system throughput is just a happy side effect :).


Tim Gebhardt
t...@gebhardtcomputing.com

[Tim Gebhardt]

That's generally the idea.  Most of the systems that the Disruptor is ideal for are systems that require low latency, but not a lot of processing per event.  The original LMAX Disruptor was designed for a trade matching engine.  For those not familiar with what those do, it's to match up a buyer and seller of some security on an exchange.  Latency is the highest concern, but the actual work per message isn't really much -- typically just looking up a handful of values in binary trees.

But it's certainly a domain that benefits GREATLY from DDD: matching engines are highly stateful and have a good amount of different state transitions: e.g. The order book for X product is in this state, and trader Y sends in Z message, it's at this time of day so send message A, B, C, and D, and on and on and on...

Some of the original blog posts and presentations by the LMAX guys talk about how their high performance work maps really well to proper object-oriented programming and DDD, which at the time (at least when I discovered it) was weird to people doing high-performance multithreaded programming.  It looked like functional programming would win the day.



Tim Gebhardt
t...@gebhardtcomputing.com

[Tim Gebhardt]

It's the overall system overhead.  The goal of the Disruptor is to deserialize your messages off your network/bus and get them into your BLP as quickly as possible.  Then you want to tie your BLP to a specific processor (thread affinity) so it gets the whole core (or cores) to itself. 

You don't want your framework to take up a lot of overhead, and that's what that number is quantifying: depending on what your BLP is doing you can expect the framework won't get in your way at 6 million messages or less.  6 million messages a second is a LOT: the OPRA feed (the consolidated feed of all traded stock options) is something like 2 million messages per second.

So the 6 million messages is just the benchmark of sending basically a null message through a certain system configuration -- I think that figure was for one producer and one consumer.

But that 6m figure is WAYYYY out of date.  The current trunk builds are at somewhere between 50 and 100 million messages a second :).


Tim Gebhardt
t...@gebhardtcomputing.com

[kell.s...@gmail.com]

Really? I wonder if those design changes are in the .NET port yet. I'm running off the original version still. 

Thanks!

Kell

[Nuno Lopes]

Hi Tim,

You don't want your framework to take up a lot of overhead, and that's what that number is quantifying: depending on what your BLP is doing you can expect the framework won't get in your way at 6 million messages or less.

Humm. Yet martin writes: 

"But the thing that got people's attention at QCon was that this wasn't where they ended up. In fact they ended up by doing all the business logic for their platform: all trades, from all customers, in all markets - on a single thread. A thread that will process 6 million orders per second using commodity hardware."

http://martinfowler.com/articles/images/lmax/arch-full.png

I'm just trying to understand this. Is it the Input Disruptor, the BPL or the all thing? Or s just the framework with a null test?

Nuno.

[Tim Gebhardt]

Hi Nuno,

I can't really speak for the LMAX guys as to what their own system can process, but the original open source release of the code chugged through it's benchmarks at about 6 million messages a second.  There are several benchmarks, but the one that was published to really highlight the low-overhead was the test with 2 threads: 1 producer and 1 consumer sending/receiving a message with a single integer.  For comparison, a fixed array queue processes about 2 million messages a second with a lot more jitter, and a linked list blocking queue is even worse.

Since then the Disruptor code has only gotten significantly faster.

Now the virtue of the Disruptor pattern is to get out of the way of the domain/business logic thread, so it's kind of your upper bound of what you CAN process.  If you're doing something CPU intensive, aren't getting receiving 40 million messages a second, or you mistakenly put some IO on your BLP then you may not see those kind of numbers. 


Tim Gebhardt
t...@gebhardtcomputing.com

[Nuno Lopes]

Thanks for the feed back Tim.

I'll now have a look at it with further detail, hoping I'll manage to understand it :)

Cheers,

Nuno

[Tom Janssens]

Hey Tim,

IMO you explain it pretty good; have you ever considered writing a blog post (series) about it ?

Tom

Typed on a mobile

[jdn]

+1

jdn

[Tom Janssens]

All this white-coat mumbo-jumbo is fun to do, but is anybody who actually tried using it willing to post his/her opinion on the usability/lack of protocol on Scritchy?

BTW, the loadhandler parameter in the scritchybus constructor is supposed to be calling your IOC container, but apparently that is something that is not obvious to some devs. I will try to find a way to make it a little bit more obvious.

I call activator.createinstance as a sensible default, but I would advise anybody to use a proper container.

Thanks!

Typed on a mobile

[Greg Young]

I have some worries with a lack of command handlers/app service layer.

One of the responsibilities of this layer is to make explicit the
dependencies that are used for a use case. As example ...

class FooHandler : Handes<FooCommand> {
public FooHandler(IFooRepository, IDoSomethingService, IBarRepository) {
}
void Handle(FooCommand msg) {
var item = fooRepo.GetById(msg.FooId);
var itemtoJoin = barRepository.GetById(msg.BarId);
itemtoJoin.JoinTo(item, doSomethingService);
barRepository.Save(itemToJoin);
}
}

It is quite common that there are many objects at play in the app service.

--
Le doute n'est pas une condition agréable, mais la certitude est absurde.

[Tom Janssens]

Thanks Greg, that is a good remark; I had no idea including service references was a common practice, and I was doing all my inter-AR-communication through sagas...
What would you think about the following approach:

class AddFooToBar

{

   string FooId;

   string BarId;

   string AnotherFooId;

}

class Bar

{

   void AddFooToBar(Foo Foo,Foo AnotherFoo, IDoSomethingService service)

  {

     // Inject 2 Foo instances based On the convention that the command contains a Foo "name"+Id, 

     // IDoSomethingService could be resolved through IOC)

  }

}

Would this be explicit enough ?

[Tom Janssens]

I added the feature to Scritchy, but I have 2 questions regarding your remark, as I am trying to create up a proper stock list example where these features are used:

class AddFooToBar

{

   string FooId;

   string BarId;

   string AnotherFooId;

}

class Bar

{

  int someblah;

  void AddFooToBar(Foo Foo,Foo AnotherFoo, IDoSomethingService service)

  {

    var someresult = service(someblah);

    if (AnotherFoo.IsForBar(someresult,Foo))

      changes += new FooAddedToBar(this.Id,someresult,Foo.Id );

  }

}

First of all: I assume only the Bar instance will generate events (i.e. the public functions you call on in between AR's should NEVER generate events, cfr. CQS) ?

Second question: I am wondering what would be a good example of a domain service for the StockList example I created; would a rule engine be a good example ?

[@yreynhout]

+1. This is what I was alluding to earlier. I never understood this
"shortcut" in NCQRS either. It's just hiding a responsibility.

On 17 nov, 13:23, Greg Young <gregoryyou...@gmail.com> wrote:
> I have some worries with a lack of command handlers/app service layer.
>
> One of the responsibilities of this layer is to make explicit the
> dependencies that are used for a use case. As example ...
>
> class FooHandler : Handes<FooCommand> {
>     public FooHandler(IFooRepository, IDoSomethingService, IBarRepository) {
>     }
>     void Handle(FooCommand msg) {
>         var item = fooRepo.GetById(msg.FooId);
>         var itemtoJoin = barRepository.GetById(msg.BarId);
>         itemtoJoin.JoinTo(item, doSomethingService);
>         barRepository.Save(itemToJoin);
>     }
>
> }
>
> It is quite common that there are many objects at play in the app service.
>

[Tom Janssens]

FYI I just found out the latest version on NuGet was failing miserably. One could not even execute a single command.
My tests did not notice it as they were referencing an older version of the NuGet package instead of the source, and I never implemented tests for the new functionality. That will teach me adding new features without tests!!! EPIC FAIL !

Anyway, the issue is resolved now, and I have added validators for the commands as well (which can be included in both the AR and non-AR classes); you can see an example in this Gist: https://gist.github.com/1501392

If anybody ever tried using this, received exceptions and just gave up, I ask/beg you to upgrade the NuGet package and try again. My apologies!