Rabbit stopped working after the maintenance window. AlreadyClosedException ... reply-code=504, reply-text=CHANNEL_ERROR

[Harrison Avella]

Hello

I am currently using version 3.8.27 of RabbitMQ.  AWS has a maintenance window and after that the application stopped working with the following exception.

 Caused by: com.rabbitmq.client.AlreadyClosedException: connection is already closed due to connection error; protocol method: #method<connection.close>(reply-code=504, reply-text=CHANNEL_ERROR - expected 'channel.open', class-id=40, method-id=10)

Does anyone know what could have happened and how can I prevent this problem from recurring?

[Arnaud Cogoluègnes]

It's likely to be a client-side connection recovery problem, but it's hard to tell with this amount of information.

You could share more information (logs, Java client version, code) and also try to reproduce the issue on a development environment, this would help narrow down the issue.

You should also upgrade, 3.8.x is out of support [1].

 [1] https://rabbitmq.com/versions.html

[Harrison Avella]

Do you know how I can reproduce this error locally?

Java version 11
Code:

@Bean
public ConnectionFactoryProvider connection(RabbitProperties rabbitProperties) {
ConnectionFactory connectionFactory = connectionGlobal(rabbitProperties);
configureSsl(connectionFactory);
return () -> connectionFactory;
}

public ConnectionFactory connectionGlobal(RabbitProperties rabbitProperties){
ConnectionFactory connectionFactory = new ConnectionFactory();
connectionFactory.setHost(rabbitProperties.getHost());
connectionFactory.setPort(rabbitProperties.getPort());
connectionFactory.setUsername(rabbitProperties.getUsername());
connectionFactory.setPassword(rabbitProperties.getPassword());
return connectionFactory;
}

I have another java code but is to register queues.

Library java client 'com.rabbitmq:amqp-client:5.14.0'

Is it valid to send you only the warning logs?

We have three nodes RabbitMQ
Cloudwacth 

Node 1

2023-02-19 09:02:29.232 [warning] <0.17401.2117> This node is being put into maintenance (drain) mode
2023-02-19 09:02:29.233 [warning] <0.17401.2117> Suspended all listeners and will no longer accept client connections
2023-02-19 09:02:29.313 [warning] <0.17401.2117> Closed 6 local client connections
2023-02-19 09:02:33.371 [warning] <0.924.0> HTTP listener registry could not find context rabbitmq_management_tcp
2023-02-19 09:02:34.399 [warning] <0.17516.2117> Asked to close connection <0.22939.2040> (reason: "vhost 'name_host' is down") but no running cluster node reported it as an active connection. Was it already closed?

Node 2

2023-02-19 09:06:24.188 [warning] <0.11105.2123> This node is being put into maintenance (drain) mode
2023-02-19 09:06:24.190 [warning] <0.11105.2123> Suspended all listeners and will no longer accept client connections
2023-02-19 09:06:24.270 [warning] <0.11105.2123> Closed 7 local client connections
2023-02-19 09:06:27.188 [warning] <0.930.0> HTTP listener registry could not find context rabbitmq_management_tcp

Node 3

2023-02-19 09:10:33.179 [warning] <0.11327.2109> This node is being put into maintenance (drain) mode
2023-02-19 09:10:33.181 [warning] <0.11327.2109> Suspended all listeners and will no longer accept client connections
(This one was repeated 10 times because of the queues.) 2023-02-19 09:10:44.285 [warning] <0.11430.2109> Could not transfer leadership of queue queue 'name_queue' in vhost 'name_host' to node 'hostname_node': {not_migrated,undefined}
(This one was repeated 3 times)  [warning] <0.12285.2109> Asked to close connection <0.6240.1922> (reason: "vhost 'name_host' is down") but no running cluster node reported it as an active connection. Was it already closed?


I am attentive if more information is required

[Arnaud Cogoluègnes]

Please be more specific about the environment. Are you using Spring AMQP and Amazon MQ? Could you share the full Java stack trace?

If you're using Spring AMQP, make sure to disable automatic recovery on the custom ConnectionFactory you're creating. See [1]. You could try to run the application locally and just shut down the broker to see if that could cause any problems.

If you're using Amazon MQ, you should check with them the migration steps they're using (apparently there's been a migration) and share the client & server logs with them.

As mentioned above you should upgrade from 3.8.27 (which is no longer supported [2]).

You should also consider moving away from mirrored queues [3] in favor of quorum queues [4].

[1] https://docs.spring.io/spring-amqp/reference/html/#connection-factory

[2] https://rabbitmq.com/versions.html

[3] https://blog.rabbitmq.com/posts/2021/08/4.0-deprecation-announcements/#removal-of-classic-queue-mirroring

[4] https://blog.rabbitmq.com/posts/2023/02/quorum-queues-migration/