Azure ServiceBus errors in production log

Tim Cools

unread,

Sep 12, 2018, 8:35:22 AM9/12/18

to masstransit-discuss

Hi all,

We are running MassTransit in production with Azure ServiceBus (ASB) and we frequently get error logs related to the use of the Azure Service bus. Does other people who run MassTransit+Azure Service also experience this? The latest version of MassTransit is always able to recover from what we've seen, (5.0 wasn't) but the amount of errors is worrying.

I'm aware that (some of) the errors are related to errors returned by ASB, which makes me wondering whether this is related to our setup? Or do other users accept this as normal?

And feedback appreciated!

Tim

Messtransit: 5.1.4

Example exceptions: (not full stack-trace)

2018-09-11 16:34:13.429 +00:00 [Error] ReceiveTransport Faulted: sb://xxxx.servicebus.windows.net/calculation-execution-import System.ArgumentException: The remote server returned an error: (400) Bad Request. The namespace tealalculation-f3125addcolumn does not exist. TrackingId:3d88fc82-10b1-437f-bb53-f1349edd7d07_G25, xxx.servicebus.windows
.net:Calculation.Shared.Messages.Execution.Calculations.Commands/ImportBatchCalculateRequest, Timestamp:9/11/2018 4:34:13 PM ---> System.Net.WebException: The remote server returned an error: (400) Bad Request.

at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
at Microsoft.ServiceBus.Messaging.ServiceBusResourceOperations.GetAsyncResult`1.<>c.<GetAsyncSteps>b__22_1(GetAsyncResult`1 thisPtr, IAsyncResult r)
...

at MassTransit.AzureServiceBusTransport.Contexts.ServiceBusNamespaceContext.<CreateTopic>d__7.MoveNext()

2018-09-10 07:19:28.089 +00:00 [Error] Exception received on receiver: sb://xxx.servicebus.windows.net/calculation-execution during Receive The messaging entity ‘Invalid audience: Namespace cannot be resolved ‘amqp://xxxx.servicebus.windows.net/calculation-execution’’ could not be found., referenceId: 70567d26e32943cdba72b1c949fbf66d_G57 2018-09-10 07:19:28.105 +00:00 [Error] ReceiveTransport Faulted: sb://xxx.servicebus.windows.net/calculation-execution System.AggregateException: One or more errors occurred. ---> System.AggregateException: One or more errors occurred. ---> Microsoft.ServiceBus.Messaging.MessagingEntityNotFoundException: The messaging entity ‘Invalid audience: Namespace cannot be resolved ‘amqp://xxxx.servicebus.windows.net/calculation-execution’’ could not be found., referenceId: 70567d26e32943cdba72b1c949fbf66d_G57

at Microsoft.ServiceBus.Common.AsyncResult.End[TAsyncResult](IAsyncResult result)
...
at MassTransit.AzureServiceBusTransport.Pipeline.MessageReceiverFilter.<GreenPipes-IFilter<MassTransit-AzureServiceBusTransport-ClientContext>-Send>d__7.MoveNext()

2018-09-12 09:27:24.020 +00:00 [Error] ReceiveTransport Faulted: sb://xxx.servicebus.windows.net/RD0003FF5FCA6D_w3wp_bus_bdmbtrxqyc?express=true&autodelete=300System.TimeoutException: The request has timed out after 5000 milliseconds. The successful completion of the request cannot be determined. Additional queries should be made to determine whether or not the operation has succeeded. For more information on exception types and proper exception handling, please refer to http://go.microsoft.com/fwlink/?LinkId=761101 TrackingId:af22c7a4-ff53-4274-9ded-4db2de6f8ea7, SystemTracker:xxxxxx.servicebus.windows.net:RD0003FF5FCA6D_w3wp_bus_3jsoyy89mhyygrsfbdmbtrxqyc, Timestamp:9/12/2018 9:27:24 AM ---> System.Net.WebException: The request was aborted: The request was canceled.

2018-09-11 15:34:12.017 +00:00 [Error] ReceiveTransport Faulted: sb://cccc.servicebus.windows.net/calculation-execution-calculations System.InvalidOperationException: The method ‘OnMessage’ or ‘OnMessageAsync’ has already been called.
at Microsoft.ServiceBus.Messaging.MessageReceiver.OnMessage(MessageReceivePump pump)
at Microsoft.ServiceBus.Messaging.QueueClient.OnMessageAsync(Func`2 callback, OnMessageOptions onMessageOptions)
at MassTransit.AzureServiceBusTransport.Contexts.QueueClientContext.OnMessageAsync(Func`2 callback, EventHandler`1 exceptionHandler)

2018-09-03 12:39:22.570 +00:00 [Error] ReceiveTransport Faulted: sb://cccc.servicebus.windows.net/calculation-execution-calculations The messaging entity ‘Calculation.Shared.Messages.Execution.Calculations.Commands/CalculationsRequest’ could not be found. 2018-09-03 12:39:47.477 +00:00 [Error] Exception received on receiver: sb://xxxx.servicebus.windows.net/calculation-execution-import during Receive Microsoft.ServiceBus.Messaging.MessagingCommunicationException: An error occurred during communication with ‘9853af93f47f4c0282b2102d1d911dc1_G49’. Check the connection information, then retry. ---> System.InvalidOperationException: The AMQP object out-connection52 is closing. Operation ‘begin’ cannot be performed.

at Microsoft.ServiceBus.Messaging.Amqp.AmqpConnection.AddSession(AmqpSession session, Nullable`1 channel)
at Microsoft.ServiceBus.Messaging.Amqp.AmqpConnection.CreateSession(AmqpSessionSettings sessionSettings)
at Microsoft.ServiceBus.Messaging.Amqp.AmqpMessagingFactory.OpenLinkAsyncResult.<GetAsyncSteps>d__35.MoveNext()

2018-09-03 12:39:49.355 +00:00 [Error] Exception received on receiver: sb://cccc.servicebus.windows.net/calculation-execution-projection during Receive Microsoft.ServiceBus.Messaging.MessagingCommunicationException: An error occurred during communication with ‘42cd6d7448ce4c60b4e68bb180a32cd6_G12’. Check the connection information, then retry. ---> System.InvalidOperationException: The AMQP object out-connection56 is closing. Operation ‘begin’ cannot be performed.

at Microsoft.ServiceBus.Messaging.Amqp.AmqpConnection.AddSession(AmqpSession session, Nullable`1 channel)
at Microsoft.ServiceBus.Messaging.Amqp.AmqpConnection.CreateSession(AmqpSessionSettings sessionSettings)
at Microsoft.ServiceBus.Messaging.Amqp.AmqpMessagingFactory.OpenLinkAsyncResult.<GetAsyncSteps>d__35.MoveNext()

2018-09-05 15:43:10.676 +00:00 [Error] Exception received on receiver: sb://cccc.servicebus.windows.net/calculation-execution-notifications-calculations during Receive Microsoft.ServiceBus.Messaging.MessagingException: The service was unable to process the request; please retry the operation. For more information on exception types and proper exception handling, please refer to http://go.microsoft.com/fwlink/?LinkId=761101 TrackingId:76cadec1696845c1a653c62969949efe_G28, SystemTracker:gateway7, Timestamp:9/5/2018 3:43:10 PM, referenceId: 76cadec1696845c1a653c62969949efe_G28

at Microsoft.ServiceBus.Common.AsyncResult.End[TAsyncResult](IAsyncResult result)
at Microsoft.ServiceBus.Messaging.Amqp.AmqpMessagingFactory.EndOpenEntity(IAsyncResult result)
at Microsoft.ServiceBus.Messaging.Amqp.AmqpMessageReceiver.EndCreateLink(IAsyncResult result)

Chris Patterson

unread,

Sep 12, 2018, 9:03:27 AM9/12/18

to masstrans...@googlegroups.com

Are you sure that you're on 5.1.4 - the OnMessage() bug was fixed in that version, so I'm curious why you would still see it.

Also, some of those errors I've never seen before - the audience errors are highly suspect.

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-dis...@googlegroups.com.
To post to this group, send email to masstrans...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/masstransit-discuss/16829e09-48ca-418d-9fa3-e4f2d2021dad%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Tim Cools

unread,

Sep 12, 2018, 9:43:40 AM9/12/18

to masstransit-discuss

Yes we recently (last week) switched to 5.1.4. The OnMessage errors are indeed from an older feature branch with 5.0.0 though, so we can ignore them...

I keep you updated about the audience errors to see if I can give you more information on when this happens...

What about the others? The Timeouts for example, do you also see them from time to time in your logs?

To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-discuss+unsub...@googlegroups.com.

Chris Patterson

unread,

Sep 13, 2018, 8:22:43 AM9/13/18

to masstrans...@googlegroups.com

Timeouts happen with cloud services, it's a fact of life. MT should recover from them and continue processing. We spent a lot of time with Microsoft to try and get them right, there are settings on the namespace in Azure, as well as tier, that help get better performance but it's just SQL Server under the hood, so...

To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-dis...@googlegroups.com.

To post to this group, send email to masstrans...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/masstransit-discuss/16829e09-48ca-418d-9fa3-e4f2d2021dad%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-dis...@googlegroups.com.

To post to this group, send email to masstrans...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/masstransit-discuss/d100e071-98f1-4df6-9988-2126bd635b15%40googlegroups.com.

Tim Cools

unread,

Sep 21, 2018, 4:54:04 AM9/21/18

to masstransit-discuss

Yes, agree, timeouts happen once in a while, and MT recovers nicely... Funny it is SQL Server under the hood :-)

And what about the The messaging entity XX could not be found error? We see this pop-up on all our environments once in a while

Level Error

Utc date/time 2018-09-20T23:58:09.7640680Z

Message Exception received on receiver: sb://XXXX?autodelete=300 during Receive

Exception The messaging entity 'sb://XXXX' could not be found. TrackingId:fb6bedba-58d9-4f0f-866c-13dfedce5a7e_G1, XXX, Timestamp:9/20/2018 11:58:09 PM, referenceId: ad376efae0a443e9b84fbc49b3c66851_G1

SourceContext "MassTransit.AzureServiceBusTransport.Transport.Receiver"

ThreadId 36

application "execution"

environment "prd"

Level Error

Utc date/time 2018-09-20T23:58:09.7640680Z

Message ReceiveTransport Faulted: sb://XXX?autodelete=300

Exception System.AggregateException: One or more errors occurred. ---> System.AggregateException: One or more errors occurred. ---> Microsoft.ServiceBus.Messaging.MessagingEntityNotFoundException: The messaging entity 'sb://xxxx.servicebus.windows.net/RD0003FF20AB2B_w3wp_bus_icioyy89ryyy8zwxbdmb6dzrbe' could not be found. TrackingId:fb6bedba-58d9-4f0f-866c-13dfedce5a7e_G1, SystemTracker:XXXX, Timestamp:9/20/2018 11:58:09 PM, referenceId: ad376efae0a443e9b84fbc49b3c66851_G1

at Microsoft.ServiceBus.Common.AsyncResult.End[TAsyncResult](IAsyncResult result)

at Microsoft.ServiceBus.Messaging.MessageReceiver.RetryReceiveAsyncResult.TryReceiveEnd(IAsyncResult r, IEnumerable`1& messages)

at Microsoft.ServiceBus.Messaging.MessageReceiver.EndTryReceive(IAsyncResult result, IEnumerable`1& messages)

at Microsoft.ServiceBus.Messaging.MessageReceiver.EndReceive(IAsyncResult result)

at Microsoft.ServiceBus.Messaging.MessageReceivePump.PumpAsyncResult.<>c.b__7_3(PumpAsyncResult thisPtr, IAsyncResult r)

at Microsoft.ServiceBus.Messaging.IteratorAsyncResult`1.StepCallback(IAsyncResult result)

--- End of inner exception stack trace ---

at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)

at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

at MassTransit.AzureServiceBusTransport.Pipeline.MessageReceiverFilter.-Send>d__7.MoveNext()