Sagas, routing slips or something else?

[Francesc Castells]

Helpful information to include

Product name: nservicebus

Version: 5

In my app I have operations that I need to split in a couple of steps. For example, the user uploads a file to a temp storage, fills in some information and Saves. This sends a CreateDocumentCommand, which needs to create a Document in the db, move the document to a permanent storage and create a thumbnail. 

Now, I assume the suggestion would be to use a saga for this, but I'm looking at a lighter solution. 

The second option would be routing slips. My understanding is that what this does is sending the same command to several endpoints one after the other so that each one performs a part of the global task. But I don´t like this approach because then it means that several endpoints need to be aware of the CreateDocumentCommand, but to me the CreateThumnailCommand is an independent step which could be called from different places, so why bother the endpoint with details from the caller?

What I'd like to have is a chain of commands. So the original caller could say: I want A, B and C to be done. 

1- Does this make sense?

2- Is there any existing solution for this?

3- If not, would it make sense to build something similar to routing slips but instead of passing an array of routes in the headers, pass an ordered array of commands and endpoints to send them to? Then a behaviour could handle sending the next command of the chain or notify the original sender of the completion. 

Thanks

Francesc

[Szymon Pobiega]

Hi

I understand the reasons for having separate commands. Can you, however, elaborate why are you looking for something "lighter" than Sagas?

Szymon

--
You received this message because you are subscribed to the Google Groups "Particular Software" group.
To unsubscribe from this group and stop receiving emails from it, send an email to particularsoftw...@googlegroups.com.
To post to this group, send email to particula...@googlegroups.com.
Visit this group at https://groups.google.com/group/particularsoftware.
To view this discussion on the web visit https://groups.google.com/d/msgid/particularsoftware/77afcc64-9442-4f07-a0ef-668ed7a02abf%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Francesc Castells]

A couple of things bother me about sagas:

- First you have to code them. It means an extra class for the saga and the data. To have a chain of 3 commands you have to handle the commands and also somehow handle the responses or events that notify that each command has been completed, to transition to the next step of the saga. 

- The second thing that bothers me is saga data persistance. It means a new table in the db to worry about and also versioning the saga data. 

At first glance it seems that it'd be good to have a multistep process without external state persistance.

On Monday, 7 March 2016 14:05:51 UTC+1, Szymon Pobiega wrote:

Hi

I understand the reasons for having separate commands. Can you, however, elaborate why are you looking for something "lighter" than Sagas?

Szymon

2016-03-07 13:00 GMT+01:00 Francesc Castells <fcas...@dgtexperts.com>:

Helpful information to include

Product name: nservicebus

Version: 5

In my app I have operations that I need to split in a couple of steps. For example, the user uploads a file to a temp storage, fills in some information and Saves. This sends a CreateDocumentCommand, which needs to create a Document in the db, move the document to a permanent storage and create a thumbnail. 

Now, I assume the suggestion would be to use a saga for this, but I'm looking at a lighter solution. 

The second option would be routing slips. My understanding is that what this does is sending the same command to several endpoints one after the other so that each one performs a part of the global task. But I don´t like this approach because then it means that several endpoints need to be aware of the CreateDocumentCommand, but to me the CreateThumnailCommand is an independent step which could be called from different places, so why bother the endpoint with details from the caller?

What I'd like to have is a chain of commands. So the original caller could say: I want A, B and C to be done. 

1- Does this make sense?

2- Is there any existing solution for this?

3- If not, would it make sense to build something similar to routing slips but instead of passing an array of routes in the headers, pass an ordered array of commands and endpoints to send them to? Then a behaviour could handle sending the next command of the chain or notify the original sender of the completion. 

Thanks

Francesc

--
You received this message because you are subscribed to the Google Groups "Particular Software" group.

To unsubscribe from this group and stop receiving emails from it, send an email to particularsoftware+unsub...@googlegroups.com.

[Tim]

Hi Francesc,

If you don't need to await for different messages or track a certain state, it should be possible to solve your problem only using message handlers. From your description I can't really tell whether that would be possible in your case or not. Maybe you provide some more details for us.

This sends a CreateDocumentCommand, which needs to create a Document in the db, move the document to a permanent storage and create a thumbnail

From what I understand, would it be possible to create the document in the DB and then publish a "DocumentCreated" event which is handled at a handler which creates the thumbnail and a second handler which moves that document (those handlers can be on different endpoints of course)?  

[Francesc Castells]

Hi Tim,

yes, the example I gave could be solved by giving each handler the knowledge of the next command to send, but this is not always the case. Let me give another example (not from my real domain, but pretty similar):

Let's imagine my application can create clients, projects, milestones and assign developers to projects and I have the commands to create a client, create a project and so on. Now the product owner wants a Quick Create screen which allows to create a client, create a project, create a few milestones and assign a few users. 

I have a QuickCreateProjectCommand with all the info coming into my web api, but my "real" operations would be:

- CreateClientCommand

- CreateProjectCommand

- CreateMilestonCommand (or maybe a single CreateMilestonesCommand)

- AssignDevelopersToProjectCommand

With the idea of the chained commands, I could have my api to send the chain of commands (so, send the CreateClientCommand with an attached chain of commands) or send the QuickCreateProjectCommand to an endpoint that would take care of the decomposition. 

[Tim]

Hi Francesc,

Here are some potential approaches to that problem. There are a lot of other solutions of course, but just to give you some ideas:

1. Saga: While you don't seem to like this approach, it's the obvious one. Have a Saga which is started by QuickCreateProjectCommand and does the orchestration. It sends the CreateClientCommand and awaits a Response or an Event from the handler. Then it continues with the next step.

2. Upfront Ids: Define the Id's of the entities to create upfront (e.g. Guids). This way you just send out all Create commands immediately and pass them the guid of the entity they reference which you already know although they don't exist yet. While all commands other than CreateClientCommand may fail initially (depending on your implementation) because the reference doesn't exist, due to the built-in retry mechanism they will succeed at some point. This approach does work especially well with NoSQL databases or designs without Foreign Keys. This design allows you to decouple your entities much easier which will simplify your endpoints a lot more.

3. Specialized Commands: Sometimes, the pragmatic approach is the best. While there is no built-in way to "piggyback" messages, you can basically implement this in a non-generic way tailored to your scenario. This of course depends whether your described scenario is a rare use case or not, but adding 3 new messages and handlers (e.g. "CreateClientWithProjectCommand" and so on) isn't expensive and would do exactly what you want to do. If you encounter scenarios like this all the time, it's probably good situation to verify if the given commands are really suitable. We're often tempted to create commands around entities because it seems to be the obvious approach. But if you constantly end up with issues like this one, this may not be the ideal solution. 

4. Callbacks: If you're sending these commands from a Web API, you can use the Callbacks package which allows you to register callbacks which are triggered by responses to a message (e.g. CreateClientCommand sends a response with the created id, which is then used by the callback to send out a CreateProjectCommand). While this approach is quite simple it's also less reliable than the above mentioned methods because you can easily get "interrupted" and callbacks aren't persisted. Therefore if the Web API crashes, not the full set of commands is executed.

[Francesc Castells]

Hi Tim,

thanks for the ideas. We'll consider which is the best opion for each scenario we encounter. 

On the other hand, I suppose I should look better into sagas. I've already used them, but I have some concerns for simple scenarios. My concerns are basically having another db table to worry about and maintaining its schema with saga data changes, plus having to know about NH requirements and mapping), would Azure Table persistence help here? Being schema less, these issues shouldn't exist right? Is there documentation on saga data versioning? Also, how does Azure persistence work in terms of transactionability? With outbox I know I can persist my data (SQL Azure) and messages to be sent transactionally, but what about saga data with Azure table storage? Could it happen that messages are sent out but saga data is not persisted or the other way around?

Francesc

[Sean Feldman]

Hi Francesc,

Agree with Tim, saga in this case does sound like a good approach.

As for the Azure Storage Tables - you will not have transactions since azure services do not support inter services transactions coordination. This document could be helpful: http://docs.particular.net/nservicebus/azure/transactions

It also depends on what transport you're using, what storage you're using to store your business data. I would not recommend to jump to Azure Storage Tables just because it's schemaless.

As for SQL Azure and transactions, you can have Elastic Database Transactions, but that requires .NET 4.6.1,  which we don't support it yet.

[Francesc Castells]

Hi Sean,

Yes, I can see that we'll end up using Sagas most of the time, unless we can use a simpler solution in some scenarios. 

We are using SQL Azure for persistence and Azure Service Bus as transport, with Outbox. Both our data and nsb data are stored in the same SQL Azure sharing the DbConnection, therefore we don't need distributed transactions (or the new Elastic Database Transactions). But looking at the link you sent, what we do is the shared local transaction pattern and I see that with Azure Tables I would have to implement one of the other patterns.

--
You received this message because you are subscribed to a topic in the Google Groups "Particular Software" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/particularsoftware/CTDikJSCYf0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to particularsoftw...@googlegroups.com.

To post to this group, send email to particula...@googlegroups.com.
Visit this group at https://groups.google.com/group/particularsoftware.

To view this discussion on the web visit https://groups.google.com/d/msgid/particularsoftware/e9a18bf8-7c7f-40e6-ab9f-8bc8e412ce6c%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Francesc Castells

Software Architect 

LinkedIn

This email (including any attachment) is a corporate message and may contain confidential and/or privileged and/or proprietary information. If you have received this email in error, please notify the sender immediately, do not use or share it and destroy this email. Any unauthorised use, copying or disclosure of the material in this email or of parts hereof (including reliance thereon) is strictly forbidden.

We have taken precautions to minimize the risk of transmitting software viruses but nevertheless advise you to carry out your own virus checks on any attachment of this message. We accept no liability for loss or damage caused by software viruses.

[Johannes Gustafsson]

I would do the opposite.

Sagas are good for one thing, giving order to events that come in parallell. However, they also add mutable state, which should be avoided if you can. In your case, you instead have a chain of events happening one after the other.

A more simple solution (IMHO) could be to store what's needed in each message (like a routing slip) and then use that to move the process forward to the next handler and so on. You do get coupling though between the handlers. On the other hand you can go from commands to events (DocumentCreated, DocumentStoredInDb etc) and have them subscribe to each other in the other direction. That would be a more functional reactive approach.

That's my 2 cents anyway.

Regards,

Johannes

You received this message because you are subscribed to the Google Groups "Particular Software" group.

To unsubscribe from this group and stop receiving emails from it, send an email to particularsoftw...@googlegroups.com.

To post to this group, send email to particula...@googlegroups.com.
Visit this group at https://groups.google.com/group/particularsoftware.

To view this discussion on the web visit https://groups.google.com/d/msgid/particularsoftware/CA%2B0DSjdW0TwdnCi2-%2BR19cY_yDYNL6J12FJkFG4DrcfMog9dmg%40mail.gmail.com.

[Francesc Castells]

Hi Johannes,

what you suggest is similar to my original idea of implementing some sort of Chained Messages infrastructure, but my idea doesn't require adding anything to messages and avoids coupling between handlers. Basically any message could be chained. What I had in mind was something like the following:

// Initialize messages to be chained

var clientId = Guid.NewGuid();

var createClientCommand = new CreateClientCommand(clientId, "Client Name");

var projectId = Guid.NewGuid();

var createProjectCommand = new CreateProjectCommand(projectId, clientId, "New Project");

[...]

var chainCompletedMessage = new ChainCompletedMessage("Project created");

var messageChain = new MessageChain();

messageChain.AddMessage("clientEndpoint", createClientCommand);

messageChain.AddMessage("projectEndpoint", createProjectCommand);

[...]

messageChain.AddMessage("myself", chainCompletedMessage);

bus.SendChain(messageChain); // Extension method

What SendChain would do would be send the first message in the chain to the given endpoint and serialize the rest into a header. Then a behaviour would intercept this header and after the handler was executed, would remove the handled message and send the rest of the chain. This would execute the whole thing in order until the optional last message that would notify the completion.

It's just an idea. I haven't really thought of the implications of this, but after the previous comments and thinking about the business scenarios we have (note that the scenario discussed above is just an example), what I think is that there will be other considerations and potentially decisions to be taken during the process, so most of the cases would benefit from a coordinator like a saga. But I haven't fully discarded routing slips or giving a try to this message chain idea.

To unsubscribe from this group and all its topics, send an email to particularsoftware+unsub...@googlegroups.com.

To post to this group, send email to particula...@googlegroups.com.
Visit this group at https://groups.google.com/group/particularsoftware.

To view this discussion on the web visit https://groups.google.com/d/msgid/particularsoftware/e9a18bf8-7c7f-40e6-ab9f-8bc8e412ce6c%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Francesc Castells

Software Architect 

LinkedIn

This email (including any attachment) is a corporate message and may contain confidential and/or privileged and/or proprietary information. If you have received this email in error, please notify the sender immediately, do not use or share it and destroy this email. Any unauthorised use, copying or disclosure of the material in this email or of parts hereof (including reliance thereon) is strictly forbidden.

We have taken precautions to minimize the risk of transmitting software viruses but nevertheless advise you to carry out your own virus checks on any attachment of this message. We accept no liability for loss or damage caused by software viruses.

--
You received this message because you are subscribed to the Google Groups "Particular Software" group.

To unsubscribe from this group and stop receiving emails from it, send an email to particularsoftware+unsub...@googlegroups.com.

[Tim]

Johannes idea very much goes the same direction as my third suggestion. I'd rather not use events in case. Those events have to carry dedicated information related to that "chained operation" and events shouldn't be designed for a specific handler, I'd recommend to use a command instead. But the general idea of carrying the state within the messages should work as Johannes stated (as long as it's a sequential list of operations). But compared to sagas it requires dedicated messages. Sagas for example also allow you to send out multiple commands which will be processed in parallel and then synchronize them again. In case all the CreateXyz messages are handled on the same endpoint, I still recommend to not use them at all and just create the whole set of entities within a single handler.

Francesc, I'd recommend to not build an infrastructure for this. That's the most complex solution for a problem you can solve with simple messages.

Cheers,

Tim

[Francesc Castells]

Thanks, Tim. I'm convinced on your last sentence. Creating new infrastructure would introduce new maintenance hassles. 

Regarding creating eveything in the same handler, it is an option in some scenarios, although it goes against the rule of not altering multiple aggregates within the same transaction (a rule we break quite often precisely because we didn't have a good way to split complex operations). So, in some cases it will be technically possible to do it, but maybe not desirable. In others, we might just break the rule for the sake of simplicity.

Thanks again all for your advices,

Francesc