Reconnect to RabbitMQ broker

[Matthew Ward]

I am running some tests for an async process that consumes messages from rabbitmq. We are trying to figure out how to gracefully handle rabbitmq going away and coming back. The goal is to have the async process reconnect and resume operations. To simulate the situation I start my consumer, then I turn rabbitmq off, wait then turn it back on.

As expected, Connection Refused exceptions are thrown (and caught) while rabbitmq is off. As soon as I bring the server back online, it throws ShutDownSignalExceptions:

com.rabbitmq.client.ShutdownSignalException: connection error; reason: {#method<connection.close>(reply-code=320, reply-text=CONNECTION_FORCED - broker forced connection closure with reason 'shutdown', class-id=0, method-id=0), null, "[B@275e538e"}
at com.rabbitmq.client.QueueingConsumer.handle(QueueingConsumer.java:198)
at com.rabbitmq.client.QueueingConsumer.nextDelivery(QueueingConsumer.java:214)

How do I gracefully handle the shutdown so that I can get back to consuming? I attempted shutting the channel / connection and creating a new connection from the factory but the shutdown signal exception is still thrown after 'reconnect'. What's going on under the hood that keeps me from successfully consuming again? Is there some sort of message I need to tell RabbitMQ "yes I know I shut you down, but its you're back up so stop crying..."

Thanks,

Matt

[Matthew Ward]

To Give  a few code examples, (I have omitted the non-relevant parts):

try {

this.consumer.nextDelivery();

} catch (ShutdownSignalException e) {

  log.error("ShutdownSignalException", e);

  if (this.channel.isOpen()) {
    try {
      this.channel.close();
    } catch (AlreadyClosedException e1) {}
  }

  if (this.connection.isOpen()) {

    try {
      this.connection.close();
    } catch (AlreadyClosedException e1) {}
  }

  this.connection = factory.newConnection();

  this.channel.basicRecover();

I've got some logic in there to back off during connection refused, but its when the server comes back online that I can't get rid of the ShutDownSignalExceptions.

[Matthew Ward]

Sorry mistake on copy/paste. Replace basicRecover() with 

this.channel = this.connection.createChannel();

[Simon MacMullen]

On 15/03/12 02:46, Matthew Ward wrote:
> As expected, Connection Refused exceptions are thrown (and caught) while
> rabbitmq is off. As soon as I bring the server back online, it throws
> ShutDownSignalExceptions:
>
> com.rabbitmq.client.ShutdownSignalException: connection error; reason:
> {#method<connection.close>(reply-code=320, reply-text=CONNECTION_FORCED
> - broker forced connection closure with reason 'shutdown', class-id=0,
> method-id=0), null, "[B@275e538e"}
> at com.rabbitmq.client.QueueingConsumer.handle(QueueingConsumer.java:198)
> at
> com.rabbitmq.client.QueueingConsumer.nextDelivery(QueueingConsumer.java:214)

Unfortunately there's a bug in the 2.7.1 Java client that obscures the
real error message here. The server logs should tell you more about what
the server is objecting to.

Cheers, Simon

--
Simon MacMullen
RabbitMQ, VMware
_______________________________________________
rabbitmq-discuss mailing list
rabbitmq...@lists.rabbitmq.com
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss

[Matthew Ward]

Is there a work-around / when will the bug fix be available? This is a
pretty crippling bug when trying to handle fault tolerance.

[Simon MacMullen]

On 19/03/12 00:12, Matthew Ward wrote:
> Is there a work-around / when will the bug fix be available? This is a
> pretty crippling bug when trying to handle fault tolerance.

The bug I was referring to was just hiding the error message in the Java
client (and will be fixed in 2.8.0). You need to look in the server logs
to see what the server is objecting to.

Cheers, Simon

--
Simon MacMullen
RabbitMQ, VMware