Unhandled MT Exception

198 views

Skip to first unread message

Wayne Brantley

unread,

Dec 3, 2015, 3:19:49 PM12/3/15

to masstransit-discuss

In a test environment with low processing power and some very busy consumers, the service crashed.

My analysis:

Mass transit let an exception through to topshelf. Topshelf caught it with unhandle exception which then attempts to stop the service.

Nothing regarding this exception went to my logs. I have logging setup with TopShelf and MassTransit and get log info, but nothing about this.

This information came from the event logs on the server. We are using SeriLog.

Can you think of a way that an exception could get past MT up to topshelf? My service is simply a proxy to IBusControl.Start/Stop.

Exceptions:

In this there were several exceptions logged over 45 seconds...(as topshelf tried to shut the server down I think).

Here is what they look like.

Initial received two of these: There are the exceptions that started the whole thing....

Failed to stop service. System.InvalidOperationException: An unhandled exception was detected ---> QueryExecutionException

An exception was caught during the execution of an action query: Connection Timeout Expired. The timeout period elapsed during the post-login phase. The connection could have timed out while waiting for server to complete the login process and respond; Or it could have timed out while attempting to create multiple active connections. The duration spent while attempting to connect to this server was - [Pre-Login] initialization=0; handshake=5762; [Login] initialization=0; authentication=0; [Post-Login] complete=9174; . Check InnerException, QueryExecuted and Parameters of this exception to examine the cause of this exception. ---> System.Data.SqlClient.SqlException: Connection Timeout Expired. The timeout period elapsed during the post-login phase. The connection could have timed out while waiting for server to complete the login process and respond; Or it could have t...

Application: Consumer.exe

Framework Version: v4.0.30319

Description: The process was terminated due to an unhandled exception.

Exception Info: QueryExecutionException

Stack:

at System.Runtime.CompilerServices.AsyncMethodBuilderCore+<>c__DisplayClass2.<ThrowAsync>b__5(System.Object)

at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)

at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)

at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()

at System.Threading.ThreadPoolWorkQueue.Dispatch()

1 second later...

Failed to stop service. System.InvalidOperationException: An unhandled exception was detected ---> System.Threading.Tasks.TaskCanceledException: The connection is being disconnected: testUser@rabbit:5672/testHost

at MassTransit.RabbitMqTransport.Integration.RabbitMqConnectionCache.<SendUsingNewConnection>d__13.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)

at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

at MassTransit.RabbitMqTransport.Integration.RabbitMqModelCache.<SendUsingNewModel>d__6.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)

at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

at MassTransit.RabbitMqTransport.RabbitMqSendTransport.<MassTransit-Transports-I...

4 seconds later....

Failed to stop service. System.InvalidOperationException: An unhandled exception was detected ---> RabbitMQ.Client.Exceptions.AlreadyClosedException: Already closed: The AMQP operation was interrupted: AMQP close-reason, initiated by Application, code=200, text="Transport Stopped", classId=0, methodId=0, cause=

at RabbitMQ.Client.Impl.SessionBase.Transmit(Command cmd)

at RabbitMQ.Client.Impl.ModelBase.ModelSend(MethodBase method, ContentHeaderBase header, Byte[] body)

at MassTransit.RabbitMqTransport.Pipeline.RabbitMqBasicConsumer.<RabbitMQ-Client-IBasicConsumer-HandleBasicDeliver>d__18.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

at System.Runtime.CompilerServices.AsyncMethodBuilderCore.<>c__DisplayClass2.<ThrowAsync>b__5(Object state)

at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)

at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, Conte...

2 seconds later...

Faulting application name: Consumer.exe, version: 1.0.1.11, time stamp: 0x565f7875

Faulting module name: KERNELBASE.dll, version: 6.3.9600.18007, time stamp: 0x55c4c341

Exception code: 0xe0434352

Fault offset: 0x000000000000871c

Faulting process id: 0x2238

Faulting application start time: 0x01d12d59527b508c

Faulting application path: C:\service\Consumer.exe

Faulting module path: C:\Windows\system32\KERNELBASE.dll

Report Id: c7f9a3c4-9950-11e5-80f1-00155d3c0306

Faulting package full name:

Faulting package-relative application ID:

Finally, 45 seconds from first exception, I got these two and the service was stopped.

Failed to stop service. System.InvalidOperationException: An unhandled exception was detected ---> QueryExecutionException

at RabbitMQ.Client.Impl.SessionBase.Transmit(Command cmd)

at RabbitMQ.Client.Impl.ModelBase.ModelSend(MethodBase method, ContentHeaderBase header, Byte[] body)

at MassTransit.RabbitMqTransport.Pipeline.RabbitMqBasicConsumer.<RabbitMQ-Client-IBasicConsumer-HandleBasicDeliver>d__18.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

at System.Runtime.CompilerServices.AsyncMethodBuilderCore.<>c__DisplayClass2.<ThrowAsync>b__5(Object state)

at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)

at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, Conte...

The windows error report dump contains it finally crashing in the threading when it was killed:

> KERNELBASE.dll!RaiseException () Unknown

[Managed to Native Transition]

mscorlib.dll!System.Runtime.CompilerServices.AsyncMethodBuilderCore.ThrowAsync.AnonymousMethod__5(object state) Unknown

mscorlib.dll!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state, bool preserveSyncCtx) Unknown

mscorlib.dll!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state, bool preserveSyncCtx) Unknown

mscorlib.dll!System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem() Unknown

mscorlib.dll!System.Threading.ThreadPoolWorkQueue.Dispatch() Unknown

[Native to Managed Transition]

[External Code]

Wayne Brantley

unread,

Dec 4, 2015, 9:31:30 PM12/4/15

to masstransit-discuss

Any ideas on where to look or how this could happen?

Andy Slivinschi

unread,

Feb 24, 2016, 7:27:20 AM2/24/16

to masstransit-discuss

Hi Wayne,

I've faced a similar issue and still did not get to the bottom of it. Have you?

As a work around you can enable the service to restart when it crashes.

            HostFactory.Run(c =>
            {
               
                c.EnableServiceRecovery(rc =>
                {
                    rc.RestartService(1);
                    rc.RestartService(1);
                    rc.SetResetPeriod(1);
                });

              ...

             }

Hope this helps!

Andy

Reply all

Reply to author

Forward

0 new messages