Timeout Exception for Transaction with RabbitMQ - Architectural Question

[Brad Schafbuch]

What is the best way to work with a handler that may take a long period of time to process?  The handler is experiencing a transaction timeout with RabbitMQ as it completes, resulting in the message being returned to the queue in RabbitMQ even though the handler completed successfully.

We're looking for best practices for this sort of situation.

Background:

Our system is publishing security (financial instrument) changes. The consumer in question connects to an old 3rd party system which can only import data via flat file using a severely constrained import executable that can only be run once across the entire firm concurrently. To integrate with older legacy imports we are leveraging our enterprise scheduling system to run the import executable (the scheduler uses an execution queue to ensure that only one instance of the import executable is being run).

The consuming endpoint writes files out to be imported including a trigger for the scheduler.  If a second message comes in before the files have been processed the endpoint currently waits for the files to be removed before the handler continues (writing out the new set of files). The files need to have the same names each time to be able to be processed by the scheduler/importer combination (so we cannot just write out the files with new names each time).

What we've encountered is that when there are blocking files in place for more than a minute the handler works as expected - it waits for the files to be removed then writes the new set. However when the handler is finished NServiceBus throws a timeout exception because the transaction to RabbitMQ has timed out (full exception at bottom of post).

Alternatives Discussed:

We are considering setting up a 'queue' for this endpoint (likely in a DB) so that the handler can complete in a timely basis by queuing the data to be processed later. This would require another process to poll for data in that queue and write out the files. This is easily achieved, but we are very interested in better suggestions on how to handle this situation and keep the event driven architecture rather than having to poll for changes.

We considered splitting the file writing to another endpoint and sending a command but we believe we would have similar timeout issues for a command just as we do with the subscribed message. We cannot host the calls to the importer within the endpoint due to the concurrency issues.

Can NSB be told to stop checking for messages on the queue for a small period of time? Potentially allowing us to stop message checking after writing a file and restart when the file has been processed.

We haven't fully researched if this is possible but considered changing the transaction timeout length - this seems like a bad workaround subject to unknown potential issues however.

Full Exception:

2015-01-12 16:02:56.417 INFO  NServiceBus.Unicast.Transport.TransportReceiver Failed to process message

System.Transactions.TransactionAbortedException: The transaction has aborted. ---> System.TimeoutException: Transaction

Timeout

   --- End of inner exception stack trace ---

   at System.Transactions.TransactionStateAborted.BeginCommit(InternalTransaction tx, Boolean asyncCommit, AsyncCallback

asyncCallback, Object asyncState)

   at System.Transactions.CommittableTransaction.Commit()

   at System.Transactions.TransactionScope.InternalDispose()

   at System.Transactions.TransactionScope.Dispose()

   at NServiceBus.Unicast.Transport.TransportReceiver.TryProcess(TransportMessage message) in c:\BuildAgent\work\1b05a2f

ea6e4cd32\src\NServiceBus.Core\Unicast\Transport\TransportReceiver.cs:line 230

   at NServiceBus.Transports.RabbitMQ.RabbitMqDequeueStrategy.ConsumeMessages(Object state) in c:\BuildAgent\work\41ea81

d808fdfd62\src\NServiceBus.RabbitMQ\RabbitMqDequeueStrategy.cs:line 196

[Indu Alagarsamy]

Hi,

Short answer: Saga

Here's a useful video cast by Andreas on reliable integrations.

http://andreasohlund.net/2011/06/03/reliable-integrations-with-nservicebus/

The audio for the 1st minute is shaky and then it's ok.

Hope this helps.

Cheers,

Indu Alagarsamy