Duplicate connections

[Morten Ottestad]

Hi,

I have a problem where RabbitMQ is listing the connections multiple times in the management interface. If I try to click on a connection that is listed multiple times I get the message "Not found". Clicking on a connection that are not listed multiple times works, and I then get the info about the connection. The connections is also listed multiple times when using rabbitmqadmin, and the open channels is also listed multiple times.

I have multiple RabbitMQ servers running as a cluster that stand behind a load balancer. The load balancer sends the traffic to the RabbitMQ server that have the least number of connections.

The traffic from the load balancer is source NATed (every connection will come into RabbitMQ with the same source-IP). The clients use AMQP 1.0.

Version used: RabbitMQ 3.10.5 and Erlang 24.2.2.

Observations:

• The connections listed in the management interface stays there for some time (2-3 minutes), and then dissapear. New connections appear at the same time and it changes between 2- 6 duplicates.
• Running netstat on the server shows that the connection is open and there is one stable connection from the source ip on port 56933.
• I have used tcpdump to see if the tcp connection is closing and then is created again, but the tcp connection is stable and has been open for 24 hours (even if the column "Connected at" in the connections pane is saying that a new connection is connected at some later time)

Does anyone have any suggestions on how to troubleshoot further?

Any help is appreciated :-)

[Luke Bakken]

Hello,

Could you let us know what load balancer you're using? If it's something like haproxy we could try to duplicate your setup.

Do you only use AMQP 1.0 clients?

Do you also see these duplicate connections in the output of rabbitmqctl list_connections?

If it's possible to NOT use your load balancer to see if that resolves the issue, that would be great information to know.

Thanks

Luke

[Morten Ottestad]

Hi Luke,

Thank you for the quick reply, I really appreciate it.

Allmost all the clients use AMQP 1.0, and rabbitmqctl list_connections doesn't seem to work on AMQP 1.0 connections as it is listing only AMQP 0-9-1 connections.

rabbitmqctl list_amqp10_connections does not seem to work either as mentioned in this issue: https://github.com/rabbitmq/rabbitmq-server/issues/4238

After some investigation it seems like the load balancer has nothing to do with the behaviour I'm experiencing.

The behaviour seems to come from that the AMQP 1.0 clients are having multiple sessions open and each session is listed in the management interface as a seperate (duplicate) connection.

If the client opens only one session and multiple links inside that session, everything seems to work as expected.

Does the AMQP 1.0 plugin translate an AMQP 1.0 session to a connection, and an AMQP 1.0 link to a channel?

[Luke Bakken]

Hi Morten,

Thanks for pointing out those limitations in listing connections. I'll make a note to look at fixing rabbitmq/rabbitmq-server#4238

On Sunday, June 12, 2022 at 1:00:04 PM UTC-7 Morten Ottestad wrote:

Allmost all the clients use AMQP 1.0, and rabbitmqctl list_connections doesn't seem to work on AMQP 1.0 connections as it is listing only AMQP 0-9-1 connections.

rabbitmqctl list_amqp10_connections does not seem to work either as mentioned in this issue: https://github.com/rabbitmq/rabbitmq-server/issues/4238

The behaviour seems to come from that the AMQP 1.0 clients are having multiple sessions open and each session is listed in the management interface as a seperate (duplicate) connection.

If the client opens only one session and multiple links inside that session, everything seems to work as expected.

Does the AMQP 1.0 plugin translate an AMQP 1.0 session to a connection, and an AMQP 1.0 link to a channe

Yes, if my reading of this code is correct:

https://github.com/rabbitmq/rabbitmq-server/blob/master/deps/rabbitmq_amqp1_0/src/rabbit_amqp1_0_session_process.erl#L36-L44

Thanks,

Luke 

[Morten Ottestad]

Hi,

It seems like multiple open AMQP 1.0 sessions also affect the possibility to close a connection with both rabbitmqadmin and rabbitmqctl.

If I try to close a connection when multiple sessions is open, I get either an error or the connection is not closed.

Observations:

When using rabbitmqadmin close connection name="the name of the connection" I get a not found message when multiple sessions is open. It works when only one AMQP 1.0 session is open.

When using rabbitmqctl close_connection with the pid listed from rabibtmqctl list_consumers, the command runs without any errors, but the connection is not closed. In the log I see "2022-09-07 07:21:16.368364+00:00 [warning] <0.24195.439> Asked to close connection <0.24074.439> (reason: <<"closing">>) but no running cluster node reported it as an active connection. Was it already closed?"

If I start 3 AMQP 1.0 sessions and then run rabbitmqctl close_all_connections "closing", one of the sessions is closed, but the other 2 is still up.

Does anyone know if there exist a workaround to close an AMQP 1.0 connection that has multiple open sessions?

[Morten Ottestad]

Hi,

Here is a small C# example that can be used to create an AMQP 1.0 connection with 3 sessions and demonstrate the issues i mention over. 

The example uses the AMQPNetLite library. https://github.com/Mott88/Amqp10SessionExample

Does anyone know if there is an erlang expression (rabbitmqctl eval) I can run to close a connection with multiple sessions open?

Any help is appreciated. :-)

[Luke Bakken]

Thank you, I will give this a try.

[Luke Bakken]

Hi Morten,

Thank you for the code. I can see the same behavior you report.

I did a bit of investigation and this issue will take some work to resolve. If you'd like to keep track, please subscribe to this issue - https://github.com/rabbitmq/rabbitmq-server/issues/4238

There may be other issues opened and they will be linked to that one.

Thanks,

Luke

[Morten Ottestad]

Thank you. I will subscribe to the issue.

As a workaround I have tried to use the command below to close an AMQP 1.0 connection that have multiple sessions open.

I use the PID from the AMQP 1.0 connection line in the log: [info] <0.18724.1952> AMQP 1.0 connection <0.32675.1>: user 'myuser' authenticated and granted access to vhost '/'

sudo rabbitmqctl eval 'erlang:exit(c:pid(0,32675,1),kill).'

This seems to work,  but I see some error messages in the log for the subprocesses when they close.

Do you know if closing the connection this way has any negative consequences for the system?

[Luke Bakken]

Hi Morten,

Closing a connection that way shouldn't affect the rest of the system.

Thanks,

Luke