Pattern for handling potentially long-running synchronous requests during commands execution

[Oleg Iavorskyi]

At times during processing of command aggregate would need to call Service to get some information necessary to continue flow (e.g. make decisions). Such Service might be implemented as a synchronous call to external application using something like REST WS. I know from experience that despite all the hopes that external application is available all the time and responds quickly it is simply not true in all cases. Simply blocking the thread in which command is executed is a sure way to get scalability issues very quickly (especially when single-thread command bus is used).

Ideally, at the point of execution of call to Service the command has to be "hibernated" until result is received (with potential cases of timeouts and multiple retrials) freeing up thread for processing of other commands. After result is received the command has to become "active" again and execution should continue. 

What is the good approach to achieve such functionality using Axon?

-------------------
Oleg Iavorskyi

[Allard Buijze]

Hi Oleg.

this type of activity is generally best done using Sagas. They are built for long(er) running processes and have access to scheduling mechanisms. The latter is useful if you need to do retries in case of failure.

You generally don't want to do this kind of processing inside an aggregate, since it will lock any access by other threads (commands) to that aggregate until the service call has completed.

Cheers,

Allard

--
You received this message because you are subscribed to the Google Groups "Axon Framework Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to axonframewor...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Oleg Iavorskyi]

Allard,

Thank you for answer. Using Sagas makes sense but I wonder how this could be practically implemented. As far as I understand Sagas are also executed through the CommandBus which means they use the same thread (in case of Disruptor implementation) as other command handlers. So if I want to make synchronous call from Saga how could I ensure that it won't block the thread?

One of the possibility that I see is to call method on Service domain object in a separate Executor (in different thread) and then raise event from Saga (e.g. RemoteRequestSentEvent). After Service complete the blocking call it could raise event RemoteRequestResponseReceivedEvent (or RemoteRequestResponseTimedOutEvent) and then Saga will listen to this event and issue appropriate command on the aggregate to continue long-running process. I have couple of questions on this approach, though. First, where would such events be stored? I assume they would not be stored to EventStore as there is no aggregate to associate them with except for Saga itself. I want to make sure that there is no case possible to lost the events and block the Saga from progress as a result. Second, is there a more elegant way to achieve execution of remote service call in separate thread (i.e. whether Axon provides this out of the box)?

Another approach might be to raise event from Saga stating that it's ready to get additional data from remote endpoint and then have dedicated EventHandler to make blocking call. In this case it's not clear where to raise the request completion event. Can it be raised from EventHandler itself? In the Axon trader example there is remote service invocation from handler but it's done in fire-and-forget style which doesn't work in my case.

After thinking more about requirements of clients of my APIs I figured out that some of them would require synchronous request-response semantic for execution of some commands. I know that this is very bad idea to do this for potentially long-running processes so my idea is to "cheat" a bit. My REST controller will accept the request, create temporary response queue in my middleware, put request into the input queue for component (specifying output queue for response) and do the blocking read from output queue (with timeout, of course). This way the client would be waiting until timeout occurs (or response is prepared) and the thread will be blocked but at the same time component itself would still be processing all messages in asynchronous manner. I would then have dedicated EventHandler for ConversationCompletedEvent that would put the result into the temporary output queue specified in message and thus will unblock waiting thread in my REST façade. The benefit of this approach is that I could do additional arbitrage of incoming requests, so, for example, if another client wants to execute the same command while it's already in progress I could just alter configuration for EventHandler dynamically to send response upon completion to another temporary output queue instead of initiating same command over again. And some very impatient clients that send lot of requests for a same command could be cut off completely when façade will figure out that they would use more threads than allowed. 

Do you think this approach for handling synchronous-to-asynchronous translation has any particular flaws or could be improved using specific techniques in Axon that I'm not aware about? Unfortunately I don't have option in many case to convert clients to the proper publish/polling model of interaction. 

[Allard Buijze]

Hi Oleg,

did you consider using the AsyncSagaManager? You could also use the AsynchronousCluster or use an event bus that uses a message queue (like AMQP) to handle events asnchronously. The last option ensures that messages aren't lost when the machine crashes.

Cheers,

Allard

[Oleg Iavorskyi]

Allard,

Thank you. This makes sense especially after I realized that Saga does not use Command Bus at all for processing but it's rather special set of Event Handlers - something I should have done earlier as it's stated in documentation directly. The only last thing to figure out is where to emit events associated with remote command execution. First that comes to my mind is to have it done in Saga Event Handler method:

[Call synchronous operation]

[Check result]

[Emit corresponding event] 

This way another Saga Event Handler will continue with flow execution after getting event.

But what if the operation is not synchronous but rather request-reply using message queues? In this case response will come to dedicated queue after processing is completed so I could write the regular component that would create command upon receiving message from queue (e.g. RespondToLongOperationResults) and dedicated command handler that will emit the corresponding Event which would then be processed by Saga Event Handler. Could I use Saga itself as Command Handler in this case or it's more appropriate to have a separate stateless component to avoid any interference with Saga instance? It's probably more question on a style rather than absolute requirement (I would prefer to keep all related logic in same class).

-------------------
Oleg Iavorskyi

[Allard Buijze]

Hi Oleg,

perhaps a Saga is a bit overkill for this type of solution. You could also consider using a simple Event Handler that emits commands based on interaction with an external service. The asynchronous communication with external parties is implemented the same way as you would "normally" do it. Simply use a callback or listening mechanism that sends a command when a reply comes in.

The modeling assignments are very hard to do over email, because the model depends a lot on the little intricacies of the domain. Over email, these details are typically lost. The most important task is to clearly define the actions you can take on aggregates, and events that are emitted as result of that. Then, you probably want to look at what you do based on those events in terms of communication with external parties and what commands are sent as result. If this involves a long running or stateful process, Sagas are the way to go, otherwise a simple Event Handler will typically do.

Finally, you want to look at what information your application needs to query. You design the query model based on those requirements. Finally, you make the Event Handlers to update the query model using the Events emitted by the aggregates.

You might know that AxonFramwork is backed by Trifork, which provides support and consultancy services. One of the things you could think of is a modeling workshop, where we find out how to best implement the model and infrastructure to achieve your requirements. If you need any more information, let me know.

Cheers,

[Oleg Iavorskyi]

Allard,

Thank you. I think this is good advice to do the check first on whether I'm modeling a long-running process or just notification to external parties and then decide whether to use Saga or simple event handler. I think in most of my cases it's long-running process but in some cases I just need to notify other systems so I would use plain Event Handler for this.

My model is quite simple, actually, and sometimes I even think that CQRS/DDD/Axon is a bit overkill but I think it's still good basis for building Event-Driven application so I will give it a shot :)

-------------------
Oleg Iavorskyi