Interacting with external services
2,385 views
Skip to first unread message
developmentalmadness
Oct 5, 2010, 1:55:07 PM10/5/10
to DDD/CQRS
I am working on a brownfield project where we are migrating to use
CQRS architecture and we have been debating how we want to interact
with existing WCF services. As I am following discussions and reading
as much as I can my thinking is as follows:
1) If my aggregate needs access to/has a dependency on an external
system then I would use double-dispatch to pass the dependency from
the command handler to the aggregate operation. I assume that I can in
some cases (like if my aggregate just needs data from the external
system) decide to access the external system from my command handler
and pass the response from the external system into my aggregate. So
like this:
MyAR
myAROperation(ISomeService svc)
var result = svc.GetSomething();
doSomething(result);
or this
MyCmdHandler
Handle(cmd)
var result = svc.GetSomething();
ar.myAROperation(result);
2) If I need to send a "command" to an external system I should create
a "domain service" object that wraps the external system. My aggregate
receives a command, raises an event, a saga responds and sends the
command which then the command handler will call the "domain service"
which will call the external system and then raise an event that
indicates if the operation is successful. Then my saga will respond to
that event and send the aggregate the appropriate command (continue or
correct). This sounds right, but a problem with this is what is the
event raised by the "domain service" tied to and how do I persist it?
Also, how do I make this "command" idempotent?
There is concern on my team that #2 will result in too many command/
event classes which translates into too much code to maintain. The
worry is that this means we need to create a command and an event to
wrap every call to the external service in addition to a saga to
coordinate the process. I concede that they may be right, but is it a
valid concern? The benefits I see are that we can compose commands
into processes, and if for each external call we have a command/
failevent/successevent then if the command fails we can correct things
and retry (or auto correct). But if I can make my commands idempotent
then maybe some of this could be reduced. However, I'm not sure how to
make the commands idempotent when the external system is not designed
that way.
The other concern is that of confusing buisiness logic (AR) and
process logic (saga). I am having trouble explaining what the
difference is and that when it is ok for logic to reside outside the
domain (read: AR). The worry is we will get it wrong and end up with
more of the same spahgetti we currently have. This is where I must
tread lightly because this migration is another attempt in a long line
of attempts to create a cohesive architecture across the enterprise.
CQRS architecture and we have been debating how we want to interact
with existing WCF services. As I am following discussions and reading
as much as I can my thinking is as follows:
1) If my aggregate needs access to/has a dependency on an external
system then I would use double-dispatch to pass the dependency from
the command handler to the aggregate operation. I assume that I can in
some cases (like if my aggregate just needs data from the external
system) decide to access the external system from my command handler
and pass the response from the external system into my aggregate. So
like this:
MyAR
myAROperation(ISomeService svc)
var result = svc.GetSomething();
doSomething(result);
or this
MyCmdHandler
Handle(cmd)
var result = svc.GetSomething();
ar.myAROperation(result);
2) If I need to send a "command" to an external system I should create
a "domain service" object that wraps the external system. My aggregate
receives a command, raises an event, a saga responds and sends the
command which then the command handler will call the "domain service"
which will call the external system and then raise an event that
indicates if the operation is successful. Then my saga will respond to
that event and send the aggregate the appropriate command (continue or
correct). This sounds right, but a problem with this is what is the
event raised by the "domain service" tied to and how do I persist it?
Also, how do I make this "command" idempotent?
There is concern on my team that #2 will result in too many command/
event classes which translates into too much code to maintain. The
worry is that this means we need to create a command and an event to
wrap every call to the external service in addition to a saga to
coordinate the process. I concede that they may be right, but is it a
valid concern? The benefits I see are that we can compose commands
into processes, and if for each external call we have a command/
failevent/successevent then if the command fails we can correct things
and retry (or auto correct). But if I can make my commands idempotent
then maybe some of this could be reduced. However, I'm not sure how to
make the commands idempotent when the external system is not designed
that way.
The other concern is that of confusing buisiness logic (AR) and
process logic (saga). I am having trouble explaining what the
difference is and that when it is ok for logic to reside outside the
domain (read: AR). The worry is we will get it wrong and end up with
more of the same spahgetti we currently have. This is where I must
tread lightly because this migration is another attempt in a long line
of attempts to create a cohesive architecture across the enterprise.
Tom Janssens
Oct 6, 2010, 1:51:00 AM10/6/10
to DDD/CQRS
I use #1 all the time, but I'm not sure whteher that is the way to go
On 5 okt, 19:55, developmentalmadness <developmentalmadn...@gmail.com>
wrote:
On 5 okt, 19:55, developmentalmadness <developmentalmadn...@gmail.com>
wrote:
Think Before Coding
Oct 6, 2010, 5:56:03 AM10/6/10
to DDD/CQRS
IMHO, Services should not raise events...
There is no problem with querying data from other services from the
command handler, or inside the aggregate using double dispatch as
proposed in #1...
jeremie / thinkbeforecoding
On 5 oct, 19:55, developmentalmadness <developmentalmadn...@gmail.com>
wrote:
There is no problem with querying data from other services from the
command handler, or inside the aggregate using double dispatch as
proposed in #1...
jeremie / thinkbeforecoding
On 5 oct, 19:55, developmentalmadness <developmentalmadn...@gmail.com>
wrote:
Jonathan Oliver
Oct 6, 2010, 7:44:23 AM10/6/10
to DDD/CQRS
@Mark,
If you're just querying data, you could potentially do that outside of
the domain first and then pass it into the domain. But if the
external service is doing something, then there are a few more things
to consider. Let me project from your original question and modify
the intent somewhat to that of an external service call that does
something, not just querying data...
If you're using event sourcing for #1, it will get the job done, but
you might run into problems during failure conditions. Specifically,
what happens if the command is related to charging someone's credit
card. Let's suppose that you process the command, charge their card,
and before you commit the associated event(s), the machine dies for
whatever reason. When it comes back online you'll process the command
again and charge their card. Bad.
With #2, you can be a little bit more flexible. The aggregate makes
the decision whether the card should be charged and/or how much to
charge. The saga tracks the state of the entire process (which might
be charging a card and then shipping an item). Then, the handler that
receives the message to charge the card has a few options. Prior to
charging a card, it can write a local value saying, "I'm about to
charge card X $123 for order Y. If at any point the process fails,
you'll know when you resume. And if you detect a failure when you
resume, you can query the WCF service to determine if the card was
charged or not.
Granted, all of this could be done inside of ISomeService using double
dispatch off the domain, but #2 breaks up the transactional boundaries
and makes each step more explicit thus giving you fine-grained control
over the entire process. #2 makes the error conditions more explicit
and forces you to deal with them in the form of events back to the
saga.
As far as "more code to maintain" is concerned, you might have a few
more command and event classes, but they are well-named classes that
express intent and have *no* behavior. They're simple DTOs that can
be serialized and transferred across the wire. You may also have one
additional message handler that receives the message and passes it to
your ISomeService, but that's about it.
Also, in #1, how do you deal with timeouts? What about optimistic
concurrency for when an external WCF call takes 10-20 seconds but
another message has been received by the aggregate in the meantime?
When you go to save the results of the WCF call, you'll get an
optimistic concurrency exception and unless you're doing "event
merging" which is an advanced event sourcing concept, you'll throw
away the results of your WCF call and retry the command.
On Oct 6, 3:56 am, Think Before Coding <jeremie.chassa...@gmail.com>
wrote:
If you're just querying data, you could potentially do that outside of
the domain first and then pass it into the domain. But if the
external service is doing something, then there are a few more things
to consider. Let me project from your original question and modify
the intent somewhat to that of an external service call that does
something, not just querying data...
If you're using event sourcing for #1, it will get the job done, but
you might run into problems during failure conditions. Specifically,
what happens if the command is related to charging someone's credit
card. Let's suppose that you process the command, charge their card,
and before you commit the associated event(s), the machine dies for
whatever reason. When it comes back online you'll process the command
again and charge their card. Bad.
With #2, you can be a little bit more flexible. The aggregate makes
the decision whether the card should be charged and/or how much to
charge. The saga tracks the state of the entire process (which might
be charging a card and then shipping an item). Then, the handler that
receives the message to charge the card has a few options. Prior to
charging a card, it can write a local value saying, "I'm about to
charge card X $123 for order Y. If at any point the process fails,
you'll know when you resume. And if you detect a failure when you
resume, you can query the WCF service to determine if the card was
charged or not.
Granted, all of this could be done inside of ISomeService using double
dispatch off the domain, but #2 breaks up the transactional boundaries
and makes each step more explicit thus giving you fine-grained control
over the entire process. #2 makes the error conditions more explicit
and forces you to deal with them in the form of events back to the
saga.
As far as "more code to maintain" is concerned, you might have a few
more command and event classes, but they are well-named classes that
express intent and have *no* behavior. They're simple DTOs that can
be serialized and transferred across the wire. You may also have one
additional message handler that receives the message and passes it to
your ISomeService, but that's about it.
Also, in #1, how do you deal with timeouts? What about optimistic
concurrency for when an external WCF call takes 10-20 seconds but
another message has been received by the aggregate in the meantime?
When you go to save the results of the WCF call, you'll get an
optimistic concurrency exception and unless you're doing "event
merging" which is an advanced event sourcing concept, you'll throw
away the results of your WCF call and retry the command.
On Oct 6, 3:56 am, Think Before Coding <jeremie.chassa...@gmail.com>
wrote:
developmentalmadness
Oct 6, 2010, 1:38:18 PM10/6/10
to DDD/CQRS
@Jeremie, thanks, the idea of raising events from the service didn't
feel right.
@Jonathan
As for #1 it sounds like you're recommending against double dispatch?
Instead you're saying use #2 in both cases, which means for a query to
the external service I need a command, then an event to say "I got the
result", then I can pass the result to my AR?
As for #2 you're saying that breaking up the steps is good and if I
need to do something about controlling impotency on the domain side I
can "write a local value". This tells me the command was attempted,
but since I'm trying again I need to verify whether or not the card
got charged?
Wouldn't it be enough to have a command which attempts to call an
operation on my AR to charge the card using the external service. Then
if it fails I trap the exception and raise a failed event, my saga can
maintain a counter and stop trying over a given threshold then place
the message in an error queue. If successful, my saga then continues
on with the next step in the process. Or how about this:
MyCmdHandler
// query operations
ar.myAROperation(result);
repo.Save(ar);
// command operations
Handle(cmd)
try
var result = svc.DoSomething();
var ar = repo.GetById(cmd.arid);
ar.DoSomething(cmd, result);
repo.Save(ar);
catch
bus.Publish(new CommandFailed());
This way timeouts are handled because we don't get the aggregate until
we've successfully called the service (cmd or query) and we don't mess
with double-dispatch because we just pass in the result of the service
call within the same message handler.
feel right.
@Jonathan
As for #1 it sounds like you're recommending against double dispatch?
Instead you're saying use #2 in both cases, which means for a query to
the external service I need a command, then an event to say "I got the
result", then I can pass the result to my AR?
As for #2 you're saying that breaking up the steps is good and if I
need to do something about controlling impotency on the domain side I
can "write a local value". This tells me the command was attempted,
but since I'm trying again I need to verify whether or not the card
got charged?
Wouldn't it be enough to have a command which attempts to call an
operation on my AR to charge the card using the external service. Then
if it fails I trap the exception and raise a failed event, my saga can
maintain a counter and stop trying over a given threshold then place
the message in an error queue. If successful, my saga then continues
on with the next step in the process. Or how about this:
MyCmdHandler
// query operations
Handle(cmd)
var result = svc.GetSomething();
var ar = repo.GetById(cmd.arid);
var result = svc.GetSomething();
ar.myAROperation(result);
repo.Save(ar);
// command operations
Handle(cmd)
try
var result = svc.DoSomething();
var ar = repo.GetById(cmd.arid);
ar.DoSomething(cmd, result);
repo.Save(ar);
catch
bus.Publish(new CommandFailed());
This way timeouts are handled because we don't get the aggregate until
we've successfully called the service (cmd or query) and we don't mess
with double-dispatch because we just pass in the result of the service
call within the same message handler.
João Bragança
Oct 6, 2010, 2:34:00 PM10/6/10
to DDD/CQRS
What about sending commands from the 'external' service instead? For
example, I need to check that a 3rd party has received a customer's
request for a reservation and verified that they are who they say they
are, and that this reservation has not expired yet. Unfortunately this
web service returns the entire customer reservation history in one
call.
def ExternalService:
def Lookup(correlationId):
// actual webservice call
for result in results:
bus.Send(ChangeCustomerEnrollmentCommand(...)) *
def CustomerEnrollmentSaga:
def Execute(ChangeCustomerEnrollmentCommand c):
unless SagaState[c.AccountNumber].Status == c.Status: // if
they are equal it means we processed this already
if (c.Status == Cancelled):
bus.Send(CancelCustomerEnrollmentCommand()) **
else:
bus.Send(EnrollCustomerCommand()) **
* I should probably have a different command for each of the different
customer enrollment statuses here
** EnrollCustomerCommand / CancelCustomerEnrollmentCommand then go to
command handlers that load the AR and do something
On Oct 6, 10:38 am, developmentalmadness
example, I need to check that a 3rd party has received a customer's
request for a reservation and verified that they are who they say they
are, and that this reservation has not expired yet. Unfortunately this
web service returns the entire customer reservation history in one
call.
def ExternalService:
def Lookup(correlationId):
// actual webservice call
for result in results:
bus.Send(ChangeCustomerEnrollmentCommand(...)) *
def CustomerEnrollmentSaga:
def Execute(ChangeCustomerEnrollmentCommand c):
unless SagaState[c.AccountNumber].Status == c.Status: // if
they are equal it means we processed this already
if (c.Status == Cancelled):
bus.Send(CancelCustomerEnrollmentCommand()) **
else:
bus.Send(EnrollCustomerCommand()) **
* I should probably have a different command for each of the different
customer enrollment statuses here
** EnrollCustomerCommand / CancelCustomerEnrollmentCommand then go to
command handlers that load the AR and do something
On Oct 6, 10:38 am, developmentalmadness
Jonathan Oliver
Oct 6, 2010, 2:53:26 PM10/6/10
to DDD/CQRS
When calling a remote function (such as a web service), we definitely
want to be considerate of mutating invocations (charging a card) vs
queries (checking a balance).
If you're doing a pure read-based call, #1 works fine. You could
potentially avoid the call in your domain altogether and have the
handler perform the lookup and then pass the result into the domain,
as you have done in your example code.
In a mutating call, we have to be extremely careful. In your code you
call "var result = svc.DoSomething()" and then you pass the result
into the aggregate. This works fine except certain failure
conditions. What happens if the process dies? In that situation you
don't even know that you've called "DoSomething()" and now when the
process spins up and retries the message, you've just charged a card
twice. And because you have no record of calling "DoSomething()"
twice, you won't believe your customer until there are a number of
customers that experience this same, intermittent problem.
There's one other issue with this code:
It may not be that the command failed. The command may have
succeeded. It may just be that you hit some kind of optimistic
concurrency exception with your aggregate when you tried to save. It
may be some storage failure, etc. It may be some kind of exception in
your code when processing the result. The command succeeded and
shouldn't be retried. The part that failed was feeding the results
back into the aggregate, rather than the command itself.
By breaking things apart, you can make the process more explicit and
you're forced to think about failure conditions which you can then
handle on a case-by-case basis. Furthermore, you're not cluttering up
your domain and associated handlers with tons of code related to
calling external services.
These concepts are known as "ports" in Alistair Cockburn's "hexagonal
architecture". Essentially we dispatch to worker nodes to perform
some activity and they inform us of the result via async messaging.
Based upon the result, our domain makes business decisions about how
to respond.
Jonathan
On Oct 6, 11:38 am, developmentalmadness
want to be considerate of mutating invocations (charging a card) vs
queries (checking a balance).
If you're doing a pure read-based call, #1 works fine. You could
potentially avoid the call in your domain altogether and have the
handler perform the lookup and then pass the result into the domain,
as you have done in your example code.
In a mutating call, we have to be extremely careful. In your code you
call "var result = svc.DoSomething()" and then you pass the result
into the aggregate. This works fine except certain failure
conditions. What happens if the process dies? In that situation you
don't even know that you've called "DoSomething()" and now when the
process spins up and retries the message, you've just charged a card
twice. And because you have no record of calling "DoSomething()"
twice, you won't believe your customer until there are a number of
customers that experience this same, intermittent problem.
There's one other issue with this code:
// command operations
Handle(cmd)
try
var result = svc.DoSomething();
var ar = repo.GetById(cmd.arid);
ar.DoSomething(cmd, result);
repo.Save(ar);
catch
***bus.Publish(new CommandFailed()); ***
Handle(cmd)
try
var result = svc.DoSomething();
var ar = repo.GetById(cmd.arid);
ar.DoSomething(cmd, result);
repo.Save(ar);
catch
It may not be that the command failed. The command may have
succeeded. It may just be that you hit some kind of optimistic
concurrency exception with your aggregate when you tried to save. It
may be some storage failure, etc. It may be some kind of exception in
your code when processing the result. The command succeeded and
shouldn't be retried. The part that failed was feeding the results
back into the aggregate, rather than the command itself.
By breaking things apart, you can make the process more explicit and
you're forced to think about failure conditions which you can then
handle on a case-by-case basis. Furthermore, you're not cluttering up
your domain and associated handlers with tons of code related to
calling external services.
These concepts are known as "ports" in Alistair Cockburn's "hexagonal
architecture". Essentially we dispatch to worker nodes to perform
some activity and they inform us of the result via async messaging.
Based upon the result, our domain makes business decisions about how
to respond.
Jonathan
On Oct 6, 11:38 am, developmentalmadness
developmentalmadness
Oct 6, 2010, 3:40:31 PM10/6/10
to DDD/CQRS
Ok, then should it be something more like this?:
MyCmdHandler
// command operations
Handle(SomeCmd cmd)
bus.Publish(new DidSomething(result));
else
bus.Publish(new SomethingFailed(result));
catch(Exception ex)
bus.Publish(new BlowedUp<SomeCmd>(ex, cmd));
MySaga
int attempts = 0;
Handle(DidSomething evt)
bus.Send(new DoSomethingNext(evt.AgID, ..));
Handle(SomethingFailed evt)
bus.Send(new Compensate(evt.Result));
Handle(BlowedUp<SomeCmd> evt)
attempts++;
if(attempts <= 5)
bus.Send(evt.Command); //retry
else
errors.Enqueue(new FailedCommand(evt));
In the above case we have our service wrapped in a command handler
which just manages calling the service and then publishing an event
based on the result. Then we have our saga which handles the next step
in our process, including failure and exception cases. I could use a
generic Exception event (BlowedUp<T>(Exception, T)) to prevent the
need to create an event type for each possible command that could fail
and then handle success as well as specific failure cases
(Publish(SomethingFailed) -> Send(Compensate)).
MyCmdHandler
// command operations
Handle(SomeCmd cmd)
try
var result = svc.DoSomething();
if(result.Sucess)
var result = svc.DoSomething();
bus.Publish(new DidSomething(result));
else
bus.Publish(new SomethingFailed(result));
catch(Exception ex)
bus.Publish(new BlowedUp<SomeCmd>(ex, cmd));
MySaga
int attempts = 0;
Handle(DidSomething evt)
bus.Send(new DoSomethingNext(evt.AgID, ..));
Handle(SomethingFailed evt)
bus.Send(new Compensate(evt.Result));
Handle(BlowedUp<SomeCmd> evt)
attempts++;
if(attempts <= 5)
bus.Send(evt.Command); //retry
else
errors.Enqueue(new FailedCommand(evt));
In the above case we have our service wrapped in a command handler
which just manages calling the service and then publishing an event
based on the result. Then we have our saga which handles the next step
in our process, including failure and exception cases. I could use a
generic Exception event (BlowedUp<T>(Exception, T)) to prevent the
need to create an event type for each possible command that could fail
and then handle success as well as specific failure cases
(Publish(SomethingFailed) -> Send(Compensate)).
Jonathan Oliver
Oct 6, 2010, 8:22:41 PM10/6/10
to DDD/CQRS
Typically your messaging infrastructure, such as NServiceBus, will
handle the retry within MyCmdHandler, that way you don't need to build
it into your sagas or into your handlers.
Depending upon the external service that you're calling, you may want
to persist some type of information that says "I'm about to call
DoSomething()". Then, when the call completes successfully, you clean
it up as you publish of the result. The purpose of this is so that,
during failure conditions and retries, you know that you may have
called out to the external service. Knowing that you may have done
some work will allow you to query the external service to determine
the work that was done, if any, such that you can take appropriate
action.
I've outline a few of these "patterns" in a few posts on my blog:
http://jonathan-oliver.blogspot.com/2010/04/messaging-at-least-once-delivery.html
http://jonathan-oliver.blogspot.com/2010/04/idempotency-patterns.html
http://jonathan-oliver.blogspot.com/2010/04/message-idempotency-patterns-and-disk.html
Jonathan
On Oct 6, 1:40 pm, developmentalmadness
handle the retry within MyCmdHandler, that way you don't need to build
it into your sagas or into your handlers.
Depending upon the external service that you're calling, you may want
to persist some type of information that says "I'm about to call
DoSomething()". Then, when the call completes successfully, you clean
it up as you publish of the result. The purpose of this is so that,
during failure conditions and retries, you know that you may have
called out to the external service. Knowing that you may have done
some work will allow you to query the external service to determine
the work that was done, if any, such that you can take appropriate
action.
I've outline a few of these "patterns" in a few posts on my blog:
http://jonathan-oliver.blogspot.com/2010/04/messaging-at-least-once-delivery.html
http://jonathan-oliver.blogspot.com/2010/04/idempotency-patterns.html
http://jonathan-oliver.blogspot.com/2010/04/message-idempotency-patterns-and-disk.html
Jonathan
On Oct 6, 1:40 pm, developmentalmadness
developmentalmadness
Oct 7, 2010, 12:30:09 PM10/7/10
to DDD/CQRS
Ok, one last question on the implementation of persisting the "I'm
about to call...". I read the links you mentioned and it led me to
think of your event store implementation, which includes a Commands
table. Would that be a suitable solution if we were to insert the
command into that table as an indication of "I'm about to call..."?
about to call...". I read the links you mentioned and it led me to
think of your event store implementation, which includes a Commands
table. Would that be a suitable solution if we were to insert the
command into that table as an indication of "I'm about to call..."?
Jonathan Oliver
Oct 7, 2010, 11:13:59 PM10/7/10
to DDD/CQRS
You've got a couple of options:
The easiest is to simply query the external service before you perform
the call. But that's at the expense of higher latency. In most
cases, this isn't a problem.
If performing the extra query prior to the mutating command for the
external service has undesirable effects (such as increased cost for
increased usage of some 3rd-party API), you can persist locally.
This could literally be as simple as writing to a text file. It could
even be a simple DB write that says, "for this message ID, I'm about
to invoke the 3rd-party API." Then, when the call completes, you can
clean it up on or after you dispatch the results.
Again, this only matters for external calls that cause mutating
behavior that is not idempotent, e.g. charging a credit card, etc.
On Oct 7, 10:30 am, developmentalmadness
The easiest is to simply query the external service before you perform
the call. But that's at the expense of higher latency. In most
cases, this isn't a problem.
If performing the extra query prior to the mutating command for the
external service has undesirable effects (such as increased cost for
increased usage of some 3rd-party API), you can persist locally.
This could literally be as simple as writing to a text file. It could
even be a simple DB write that says, "for this message ID, I'm about
to invoke the 3rd-party API." Then, when the call completes, you can
clean it up on or after you dispatch the results.
Again, this only matters for external calls that cause mutating
behavior that is not idempotent, e.g. charging a credit card, etc.
On Oct 7, 10:30 am, developmentalmadness
Udi Dahan
Oct 23, 2010, 9:24:05 AM10/23/10
to ddd...@googlegroups.com
The reason to send messages to gateways to external services and manage the
process using sagas has to do with consistency - you probably can't roll
back a call to an external service when your database rolls back due to a
deadlock.
process using sagas has to do with consistency - you probably can't roll
back a call to an external service when your database rolls back due to a
deadlock.
Cheers,
-- Udi Dahan
or this
No virus found in this incoming message.
Checked by AVG - www.avg.com
Version: 9.0.862 / Virus Database: 271.1.1/3170 - Release Date: 10/05/10
08:34:00
Reply all
Reply to author
Forward
0 new messages