Automatic Recovery

[Ralph Churchill]

I've had inconsistent results with automatic recovery and am having trouble tracking down where the problem is. I'm using the latest version of the gem and have automatic recovery enabled (the default).

When our rabbit cluster has an outage or a node is restarted, our clients automatically reconnect without issue. I see errors and warnings in our application logs and then see the connections and channels re-established. However, we often find that some (but not all) channels are no longer bound to a queue. In other words, network connectivity is restored but status as a consumer is not. No exceptions are raised in our application code.

I could be missing something, but sometimes it works perfectly. Is there something I should be looking for?

[Michael Klishin]

See server logs for clues. Topology recovery cannot be guaranteed to always work.

For better or worse, AMQP 0-9-1 topology operations must be idempotent,

and with some failure scenarios and systems the topology changes can happen while the connection is down.

One well known example of this is client-named (well known, not server-named) exclusive queues. There's a natural race condition

between a client's post-recovery redeclaration on another node and RabbitMQ's exclusive queue deletion operation. It can result

in a RESOURCE_LOCKED channel exception which will be visible in the logs. There is nothing the client or RabbitMQ can do about it.

There is something you as a developer can: avoid using client-named exclusive queues; use server-named queues if they must be exclusive,

that means the server will generate a unique [within the vhost] name each time.

[Ralph Churchill]

Thanks! Interesting.

Are these issues that should be surfacing in the client code? Right now, I have no other way to tell that my application is no longer a consumer. My application just sort of blissfully continues on not realizing its "disconnected".

[Michael Klishin]

You can always check if a channel is open and see how many consumers it has.

[Ralph Churchill]

Sure, sure. I just meant that the long-running process that's blocking while waiting for messages, per

http://rubybunny.info/articles/queues.html#blocking_or_nonblocking_behavior

will continue to run without realizing it "disconnected" and will no longer receive messages.

Right? Or is my understanding wrong?

I really appreciate the help and the quick responses, thank you.

[Ralph Churchill]

Are channel.status and channel.any_consumers? things that I could/should be checking in my blocking loop? And, if they "fail" then I could reestablish?

[Michael Klishin]

Such checks should be metric-driven and not performed for every delivery (in fact, if you don't use polling consumers,
It can't be done for every delivery).

I'd inspect server logs for clues before making any changes, or take a traffic capture in case you can reasonably reliably reproduce.
We do not guess in the RabbitMQ community.

> --
> Bunny: http://rubybunny.info
> March Hare: http://rubymarchhare.info
>
> IRC: #rabbitmq on irc.freenode.net
>
> Post to the group: ruby...@googlegroups.com | unsubscribe: ruby-amqp+...@googlegroups.com
> ---
> You received this message because you are subscribed to the Google Groups "Ruby RabbitMQ libraries" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to ruby-amqp+...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/ruby-amqp/451ea908-063c-4e2b-a66b-09ec853d274dn%40googlegroups.com.

[Michael Klishin]

There is Bunny::Channel#open?

> --
> Bunny: http://rubybunny.info
> March Hare: http://rubymarchhare.info
>
> IRC: #rabbitmq on irc.freenode.net
>
> Post to the group: ruby...@googlegroups.com | unsubscribe: ruby-amqp+...@googlegroups.com
> ---
> You received this message because you are subscribed to the Google Groups "Ruby RabbitMQ libraries" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to ruby-amqp+...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/ruby-amqp/4a4ec85d-44c8-4583-8a54-04b5084ab791n%40googlegroups.com.