RabbitMQ DotNet Client -- Hard to handle exceptions properly

[Daniel Hakim]

Notice:  Due to the error with heartbeats + amqps in the 3.5.3 DotNet build, I rolled back to the 3.4.4 client for now.  So my problem may have already been addressed.

I am attempting to implement automatic reconnection on my C# client when I lose connection to the RabbitMQ server.  I toyed around with ConnectionFactory's AutomaticRecoveryEnabled, but I found that I didn't really understand what it was doing well enough to use it.  I'm still fairly convinced that it would be a better solution than the one I am now building, so if anyone has any links on using it properly, I'd happily read through them. 

Anyway, the solution I'm now building has one class Master, which manages the connection.  And two classes, Publisher, and Subscriber, that manage two IModels on that connection.  I see the ConnectionShutdown event trigger, and I immediately tear down my publisher and subscriber.  When I do that, I'm seeing a variety of different exceptions being thrown on functions in RabbitMQ. 

My notes when you kill a connection through the management interface on the server side:

There doesn't appear to be any guaranteed order between the ConnectionShutdown event and the Publisher/Subscriber inner threads throwing exceptions. 
channel.BasicPublish has thrown AlreadyClosedException, IOException, and NotSupportedException
channel.Dispose has thrown NotSupportedException  (This got me good, since I have it in a using block)
channel.BasicCancel has thrown NotSupportedException
QueueingBasicConsumer throws EndOfStreamException

These are all the non-race condition ones I've caught.  I'm afraid to even try killing the connections as they're still being established. 



1.  Am I doing something seriously wrong by trying to build my own automatic recovery?
2.  Can the client library swallow NotSupportedException on IModel.Dispose calls?  It gets thrown by the closing brace of a using statement and it's not at all clear what it means at the application level. 
3.  I really like AlreadyClosedException and EndOfStreamException.  It looks like IOException and NotSupportedException just snuck through due to the way I'm breakpointing through the code.  Is throwing these exceptions a bug?  (Remember, I'm using 3.4.4, may already be fixed) 

[Michael Klishin]

 On 24 June 2015 at 03:36:18, Daniel Hakim (dha...@trxsystems.com) wrote:
> I am attempting to implement automatic reconnection on my C#
> client when I lose connection to the RabbitMQ server. I toyed
> around with ConnectionFactory's AutomaticRecoveryEnabled,
> but I found that I didn't really understand what it was doing well
> enough to use it. I'm still fairly convinced that it would be a
> better solution than the one I am now building, so if anyone has
> any links on using it properly, I'd happily read through them.

http://rabbitmq.com/dotnet-api-guide.html, Automatic Recovery From Network Failures.

> Anyway, the solution I'm now building has one class Master, which
> manages the connection. And two classes, Publisher, and Subscriber,
> that manage two IModels on that connection. I see the ConnectionShutdown
> event trigger, and I immediately tear down my publisher and subscriber.
> When I do that, I'm seeing a variety of different exceptions being
> thrown on functions in RabbitMQ.
>
> My notes when you kill a connection through the management interface
> on the server side:
>
> There doesn't appear to be any guaranteed order between the ConnectionShutdown
> event and the Publisher/Subscriber inner threads throwing
> exceptions.
> channel.BasicPublish has thrown AlreadyClosedException,
> IOException, and NotSupportedException
> channel.Dispose has thrown NotSupportedException (This got
> me good, since I have it in a using block)
> channel.BasicCancel has thrown NotSupportedException
> QueueingBasicConsumer throws EndOfStreamException

Shutdown event handlers should be executed in the order they are added — to be honest,
this is the kind of .NET implementation details I do not possess.

Automatic connection recovery uses a wrapper connection class plus a shutdown hook.
The only thing automatic recovery doesn't do is trying to internally enqueue
outgoing messages when connection is down — those will throw AlreadyClosedException,
as you've observed. The main reason for that is, to do this properly you need
to have an on-disk log in the client, otherwise you risk running out of heap
because connection can be down for minutes or hours.

This is an improvement we'd like to get to some day but currently don't have
the manpower to do right.
--
MK

Staff Software Engineer, Pivotal/RabbitMQ