Long running operations in a message bus handler

[Brett Bailey]

We use the message bus for long running operations.  In fact it is our standard practice to put anything likely to take more that ~15 seconds onto the message bus so that we don't have to keep the users web request open.

We've hit some problems with this in the past and all have been resolved so far, but I was wondering what you considered "best practice" for dealing with long running operations.

You have mentioned on a couple of posts to us and others the idea of starting the operation from a handler and then having the operation send a "finished" message.  

I was wondering if we could expand this idea further - are you talking about starting up a new process / app domain?  should the the initial handler a saga, so that success / retry / fail  etc can be handled.  Do we need any special considerations for managing the message bus host application - eg preventing shutdown when these operations are still running?

~Brett

[Rafal Gwizdala]

[VacationMode=on] - no time to reply

If you have an operation that fits inside a transaction you don't need to do anything - the operation is atomic and one message handler can do everything, the framework will take care of managing the transaction and retrying in case of failure. By adjusting the transaction timeout (30 sec by default) you can handle quite long tasks, but it's not desirable method if you need to improve concurrency or avoid holding db locks for too long.

So the alternative for really long running tasks is to handle them in async way - you initiate the task with one message (StartTask) and run it in the background. When the job completes you publish an OperationComplete message that infroms other components about work being finished. But you also need to provide additional guard logic for handling failures asynchronously when the OperationComplete message is never published because the operation died in the middle or got stuck - and this can be done by implementing a timeout logic with a scheduled message (you publish an OperationTimeout message with handling date set to your desired operation timeout). When you receive the timeout message you either ignore it (when the job has already completed) or do some error handling when the job isn't done. The timeout handler can report an error to the application or start the failed job again.

The other option is to use the "saga" pattern which is a state machine with persistence for handling long-running processes. The nginn-messagebus implementation is described here: http://lafar6502.github.io/nginn-messagebus-pages//sagas.html. This option, however, doesn't differ too much from the first one, its rather just the matter of different packaging of the same logic.

R

 

--
You received this message because you are subscribed to the Google Groups "nginn-messagebus" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nginn-messageb...@googlegroups.com.
To post to this group, send email to nginn-me...@googlegroups.com.
Visit this group at http://groups.google.com/group/nginn-messagebus.

[Rafal Gwizdala]

It looks like a clarification would be useful: the whole point is to split one long transaction into three short ones (job start, job finish, job check/timeout) and moving the 'job' outside of message bus transaction. Of course the 'job' can do its own transactions too, it would just not be a part of message bus transaction.