"The operation is not valid for the state of the transaction" on Bus.Send

[Adam Jones]

Product name: NServiceBus / NServiceBus.Azure

Version: 4.1.3

We're seeing this exception message show up intermittently:

The operation is not valid for the state of the transaction.
   at System.Transactions.TransactionState.EnlistVolatile(InternalTransaction tx, IEnlistmentNotification enlistmentNotification, EnlistmentOptions enlistmentOptions, Transaction atomicTransaction)
   at System.Transactions.Transaction.EnlistVolatile(IEnlistmentNotification enlistmentNotification, EnlistmentOptions enlistmentOptions)
   at NServiceBus.Unicast.Queuing.Azure.ServiceBus.AzureServiceBusMessageQueueSender.Send(TransportMessage message, Address address) in c:\BuildAgent\work\ba77a0c29cee2af1\src\NServiceBus.Azure\Transports\ServiceBus\AzureServiceBusMessageQueueSender.cs:line 53
   at NServiceBus.Unicast.UnicastBus.SendMessage(List`1 addresses, String correlationId, MessageIntentEnum messageIntent, Object[] messages) in :line 0
   at NServiceBus.Unicast.UnicastBus.SendMessage(Address address, String correlationId, MessageIntentEnum messageIntent, Object[] messages) in :line 0
   at NServiceBus.Unicast.UnicastBus.Send(Object[] messages) in :line 0

We can't tell what is causing it, and we typically see that the message succeeds on the next attempt (first level retry).  Still, we'd love to be able to eliminate them if there is some configuration or work around that could prevent them from coming up.

In researching it, I found this post:

https://groups.google.com/forum/#!searchin/particularsoftware/System.Transactions.TransactionState.EnlistVolatile/particularsoftware/17oRvsFHThU/G07BrdFEo8wJ

It's the same exception, but coming from a Bus.Publish.  We do not explicitly call 

Config.Transactions.Disable()

Should we add this call to the initialization of all of our azure deployed endpoints, and if so, do we also need to include the following?

Configure.Transactions.Advanced(t => t.DisableDistributedTransactions().DoNotWrapHandlersExecutionInATransactionScope());

[Yves Goeleven]

How many messages are you sending in this case?

[Adam Jones]

Not many.  This exception is coming from our TEST environment which sees around 100 messages each hour.  We did have a spike in message processing after restarting our hung endpoints (see https://groups.google.com/forum/?#!topic/particularsoftware/bw2h3CgS_QY).  That spike resulted in around 2000 messages to be received at once.  We have the max concurrency for each node set at or below 10.

[Yves Goeleven]

Is this exception occuring during the spike, or during regular operation?

[Yves Goeleven]

I suspect it pulls to many messages from the queue, relative to how many it can send out in the scope of the transaction. Can you try lowering the batchsize property on the azureservicebus config to 10? (equal to your max concurrency)

[Adam Jones]

We see it intermittently during regular operation as well as during the spike.

[Yves Goeleven]

The transaction that your send occurs in has for some reason been aborted or timed out... 

I would not disable the transaction as that will also disable retry mechanism (that makes your handler succeed eventually), but instead figure out what is blocking or taking to long... are you seeing other exceptions in the same timeframe, is an operation taking to long, something like that is probably having this as a side effect

[Adam Jones]

Yes, we will adjust the batch size to see if this improves behavior.

The handler that is experiencing this issue doesn't have any delays that we have seen as it is not performing any database, disk, or network calls.  It is simply parsing a received block of binary and then Bus.Send a message representing the parsed message.  What is interesting is that, at the same time our TEST environment is getting this exception:

"The operation is not valid for the state of the transaction.

at System.Transactions.TransactionState.EnlistVolatile(InternalTransaction tx, IEnlistmentNotification enlistmentNotification, EnlistmentOptions enlistmentOptions, Transaction atomicTransaction)"

our PROD environment is getting this exception on the matching endpoint:

"System.OperationCanceledException: The operation cannot be performed because the entity has been closed or aborted. ---> System.ServiceModel.CommunicationObjectAbortedException: The communication object, System.ServiceModel.Channels.ClientFramingDuplexSessionChannel, cannot be used for communication because it has been Aborted."

[Yves Goeleven]

Interesting, maybe asb is having issues at that point. But I assume prod & test are on different asb namespaces? so it's not likely that the same asb server is involved in the communication (or the entire cluster must have issues)

[Adam Jones]

That's correct, we use different namespaces between DEV, TEST, and PROD.