RabbitMQ channel suddenly closing

[Carleen]

I'm having a problem with my channels suddenly shutting down. I have a shutdown listener, and the error message only says (forgive any errors - I have to retype this):

connection error; reasons {#method<connection.close>(reply-code=320, reply-text=CONNECTION_FORCED - broker connection closure with reason 'shutdown', class-id=0, method-id=0), null, ""}

In my code I have a catch for the AlreadyClosedException, and try to create a new channel like so:

} catch (AlreadyClosedException e) {
  Connection connection = channel.getConnection();
  channel = connection.createChannel();
}

This seems to have no effect. I still continuously get the AlreadyClosedException until I restart the broker. The server log (which I can't post because it's on another network) indicates that rabbitmq is stopping listening, and then shows the same error message that the shutdown listener is giving.

Thanks for any help.

[Michael Klishin]

On 14 November 2014 at 18:29:21, Carleen (dicke...@gmail.com) wrote:
> connection error; reasons {#method(reply-code=320,

> reply-text=CONNECTION_FORCED - broker connection closure
> with reason 'shutdown', class-id=0, method-id=0), null, ""}

This means RabbitMQ was told to shut down (by a human or another program). 

> In my code I have a catch for the AlreadyClosedException, and
> try to create a new channel like so:
>
> } catch (AlreadyClosedException e) {
> Connection connection = channel.getConnection();
> channel = connection.createChannel();
> }
>
> This seems to have no effect

You need to register a shutdown hook. connection.close is dispatched
in a different thread.
--
MK

Staff Software Engineer, Pivotal/RabbitMQ

[Carleen]

Do you mean the ShutdownListener? I have registered that. Is there something I should be doing inside the listener, other than printing the error message?

I can't see how rabbit was told to shutdown though. We have a cluster of 30 boxes, and this was the only one to have a problem. And it happened in the middle of the night.

[Simon MacMullen]

On 14/11/14 15:34, Carleen wrote:
> I can't see how rabbit was told to shutdown though. We have a cluster of
> 30 boxes, and this was the only one to have a problem. And it happened
> in the middle of the night.

The server logs will have more information on this. The client will only
know that the server is shutting down, it doesn't get told why.

Cheers, Simon

[Michael Klishin]

On 14 November 2014 at 18:34:02, Carleen (dicke...@gmail.com) wrote:
> Do you mean the ShutdownListener? I have registered that. Is
> there something I should be doing inside the listener, other
> than printing the error message?

Your entire connection is closed. I am guessing you'd want to do something other
than logging, or use automatic recovery (which recovers from server-sent connection.close
by default):

https://www.rabbitmq.com/api-guide.html

[Carleen]

Unfortunately the server log only gives me the same message that the client did.

--

Carleen

[Carleen]

I'm currently on an older version of Rabbit that doesn't have the auto-recover in the connection factory. Can you provide me some sample code you'd recommend for this shutdown hook and how to restart the connection?

--

Carleen

[Michael Klishin]

On 14 November 2014 at 18:36:31, Carleen (dicke...@gmail.com) wrote:
> Unfortunately the server log only gives me the same message
> that the client did.

Try correlating that message with syslog. Perhaps the entire OS was shut down/rebooted?

[Michael Klishin]

On 14 November 2014 at 18:37:43, Carleen (dicke...@gmail.com) wrote:
> I'm currently on an older version of Rabbit that doesn't have
> the auto-recover in the connection factory. Can you provide
> me some sample code you'd recommend for this shutdown hook and
> how to restart the connection?

You can use the most recent client with older servers (even 2.x). This is certainly
recommended. 

[Carleen]

I know the OS was up because we have other services on that box that didn't have a problem.

--

Carleen

[Michael Klishin]

On 14 November 2014 at 18:40:15, Carleen (dicke...@gmail.com) wrote:
> Yes, and in our next deployment we will upgrade, but for what
> I have fielded now, I can't use a newer client.

The docs explain what automatic recovery does. It's not rocket science. You can
recover your connection and channel(s) with fairly low amount of work, so I'd try with
that and see if you need to recover topology (also explained in the Java client docs).

[Carleen]

I'm sorry I'm still not understanding something. The version of the api I'm using doesn't have

factory.setAutomaticRecoveryEnabled(true);

So what do you recommend I do instead?

--

Carleen

[Michael Klishin]

On 14 November 2014 at 18:46:09, Carleen (dicke...@gmail.com) wrote:
> So what do you recommend I do instead?

Automatic recovery in more recent clients uses a shutdown listener to do basically
the following:

 * Recover connection and its state (shutdown listeners)
 * Recover all channels and their state (shutdown listeners, basic.qos prefetch, publisher confirms, etc)
 * Recover topology (queues, exchanges, bindings, consumers) if enabled

I recommending doing the first two items, and, if needed, the last one. You can use
Java client code as an example, except for topology recovery it's fairly  straightforward:

http://hg.rabbitmq.com/rabbitmq-java-client/file/ab27b7201868/src/com/rabbitmq/client/impl/recovery/AutorecoveringConnection.java#l345

[Carleen]

I really appreciate your patience (*smile*), but it's still not clear to me where to implement this. My version of the API (version 3.2.1) doesn't have AutorecoveringConnection. Are you telling me to look at its code to implement somewhere myself? If so, which part of the code and where do I put it? In the ShutdownListener?

--

Carleen

[Michael Klishin]

On 14 November 2014 at 19:01:58, Carleen (dicke...@gmail.com) wrote:
> My version of the API (version 3.2.1) doesn't have AutorecoveringConnection.
> Are you telling me to look at its code to implement somewhere myself? 

 Yes.

> If so, which part of the code and where do I put it? In the ShutdownListener?

Yes, this is what automatic recovery does, as can be seen from

[Carleen]

In the ShutdownListener, how do I get access to the ConnectionFactory to restart the connection? I only have the Channel.

--

Carleen

[Michael Klishin]

On 14 November 2014 at 19:05:06, Carleen (dicke...@gmail.com) wrote:
> In the ShutdownListener, how do I get access to the ConnectionFactory
> to restart the connection? I only have the Channel.

final ConnectionFactory cf = …;
Connection conn = …;

conn.addShutdownListener(new ShutdownListener() {
  public void shutdownCompleted(ShutdownSignalException cause) {
    // access cf here
  }
})

[Carleen]

Ok, I'm onboard now! I just had to figure out how to get the ConnectionFactory to my class. THANK YOU SO MUCH for all your help!

Carleen

--

Carleen