ObjectDisposed exception from C# RabbitMQ 3.5.4 client

[Raphael Koch]

Hi ,

Recently upgraded to the C# RabbitMQ 3.5.4 client from  3.2.1  and am seeing for the first time an ObjectDisposed exception which I never got  with the 3.2.1 version which we had running for ~ 2 years.  This exception surfaced within 2 days of our deploy with 3.5.4.  Attached is the stack trace...   Please let me know of how I should handle this exception or of any other possible remediation.

Thanks

Raphael

Framework Version: v4.0.30319

Description: The process was terminated due to an unhandled exception.

Exception Info: System.ObjectDisposedException

Stack:

   at System.Threading.TimerQueueTimer.Change(UInt32, UInt32)

   at RabbitMQ.Client.Framing.Impl.Connection.HeartbeatReadTimerCallback(System.Object)

   at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)

   at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)

   at System.Threading.TimerQueueTimer.CallCallback()

   at System.Threading.TimerQueueTimer.Fire()

   at System.Threading.TimerQueue.FireNextTimers()

[Michael Klishin]

On 7 August 2015 at 20:39:27, Raphael Koch (rapha...@gmail.com) wrote:
> Recently upgraded to the C# RabbitMQ 3.5.4 client from 3.2.1
> and am seeing for the first time an ObjectDisposed exception
> which I never got with the 3.2.1 version which we had running for
> ~ 2 years. This exception surfaced within 2 days of our deploy
> with 3.5.4. Attached is the stack trace... Please let me know
> of how I should handle this exception or of any other possible
> remediation.

3.2.1 cannot have it because it does not use .NET timers for heartbeats.

This simply means that a timer object was disposed right before an operation
it scheduled tried to re-schedule its next run. It should be harmless but
we should try catching those to avoid scary stack traces. In fact, we already do
in some areas.

Thanks for reporting! 
--
MK

Staff Software Engineer, Pivotal/RabbitMQ

[Raphael Koch]

Hi Michael,

Thanks for your quick response.  Just to clarify, looks like you're saying that this internal exception bled through to the application and should have been handled by the client library itself.   Assume this issue will be fixed in the library, in the meanwhile do you have suggestions how I should handle this in my application...should I just catch/ignore this exception? 

Thanks again for your help,
Raphael

[Michael Klishin]

On 8 August 2015 at 00:42:32, Raphael Koch (rapha...@gmail.com) wrote:
> Thanks for your quick response. Just to clarify, looks like
> you're saying that this internal exception bled through to the
> application and should have been handled by the client library
> itself. Assume this issue will be fixed in the library, in the
> meanwhile do you have suggestions how I should handle this in
> my application...should I just catch/ignore this exception?

 I'd be surprising if this bubbled up on an application
thread but if it does, just ignore it.

[Chris Cook]

Michael Klishin <mklishin@...> writes:


I too am seeing this exception bubble up in my production system and no,
it is not innocuous... That uncaught exception is resulting in the IIS
WAS process being ejected and costs us 15 seconds of restart time to
recover... During which time, our server is unavailable and creates other
exceptions due to it being offline...


We desperately need to fix this ASAP! Currently running 2 nodes of Rabbit
MQ and we see this happen at least once a day on one or both servers...



Any idea when this might bubble up as priority?

Thanks

[Michael Klishin]

On 13 October 2015 at 20:45:15, Chris Cook (chris...@ncr.com) wrote:
> We desperately need to fix this ASAP! Currently running 2 nodes
> of Rabbit
> MQ and we see this happen at least once a day on one or both servers...
>
>
>
> Any idea when this might bubble up as priority?

Have you seen Client library: .NET in https://github.com/rabbitmq/rabbitmq-server/releases/tag/rabbitmq_v3_5_5?

Depending on what exact code path you've seen this exception on, it may be
fairly trivial to catch and ignore, too. 

[Michael Klishin]

On 13 October 2015 at 20:55:24, Cook, Christopher (chris...@ncr.com) wrote:
> I'm unclear how I would go about catching this exception from
> any thread I have ownership over but I suspect that upgrading
> the driver is my best bet

Why would an unhandled exception, say, in .NET client's I/O thread bring your app server
down?

It only makes sense to me if an exception is unhandled on the main thread. 

[Michael Klishin]

On 13 Oct 2015 at 21:15:20, Cook, Christopher (chris...@ncr.com) wrote:
> Note: No client code in the stack, just RabbitMQ Heartbeat thread
> inside System code…
>
> Thus, when the exception is raised, there is nothing to catch
> it and it hits top of stack in a system thread… Windows dumps the
> WAS when this occurs

I had no idea that an unhandled even in a timer callback can bring down
the runtime in .NET. Wow.

Well, please try 3.5.6 (same as 3.5.5 as far as .NET client goes but still). 

[gatesvp]

Having had variants on this with other libraries (e.g. a library for Push Notifications, APNS), I can confirm that the unhandled exceptions in .NET callbacks are fatal and will crash the whole process.

I think your fixes for 3.5.5 with the following commit are sufficient.

https://github.com/rabbitmq/rabbitmq-dotnet-client/commit/4b477ff1d99db55f764feeca580518bd3831fbfd#diff-1db9ee493304b29e589eecca6398158aR929

Though it looks like you're catching a very specific exception, so obviously other types of exceptions would crash the process. This may be expected behaviour in those cases.

[gatesvp]

Having had variants on this with other libraries (e.g. a library for Push Notifications, APNS), I can confirm that the unhandled exceptions in .NET callbacks are fatal and will crash the whole process.

I think your fixes for 3.5.5 with the following commit are sufficient.

https://github.com/rabbitmq/rabbitmq-dotnet-client/commit/4b477ff1d99db55f764feeca580518bd3831fbfd#diff-1db9ee493304b29e589eecca6398158aR929

Though it looks like you're catching a very specific exception, so obviously other types of exceptions would crash the process. This may be expected behaviour in those cases.

[Michael Klishin]

On 14 Oct 2015 at 02:17:52, gatesvp (gat...@gmail.com) wrote:
> Having had variants on this with other libraries (e.g. a library
> for Push Notifications, APNS), I can confirm that the unhandled
> exceptions in .NET callbacks are fatal and will crash the whole
> process.

Thanks for confirming.

> I think your fixes for 3.5.5 with the following commit are sufficient.
> https://github.com/rabbitmq/rabbitmq-dotnet-client/commit/4b477ff1d99db55f764feeca580518bd3831fbfd#diff-1db9ee493304b29e589eecca6398158aR929
>
> Though it looks like you're catching a very specific exception,
> so obviously other types of exceptions would crash the process.
> This may be expected behaviour in those cases.

FWIW we haven’t heard from a few folks who encountered this problem
ever since. 

[Pascal Szorath]

We are having the same problem.

 

This is very annoing when debugging the application.

 

The whole debug session will shutdown and we have to compile again... this issue is poping up on every second debug sesstion... so it is very important for us to fix this.

[Michael Klishin]

On 29 October 2015 at 23:06:06, Pascal Szorath (szo...@gmail.com) wrote:
> The whole debug session will shutdown and we have to compile
> again... this issue is poping up on every second debug sesstion...
> so it is very important for us to fix this.

Have you seen an earlier response about the exception being handled and ignored in .NET client  3.5.5? We haven’t
had this reported ever since:
http://www.rabbitmq.com/changelog.html

I should point out that an unhandled exception in a timer callback bringing down the entire runtime
is a severe runtime issue you should report to Microsoft: I see absolutely no justification for that.
No other runtime I’m aware of has this problem.