Bug: Multiple MQTT connections using same client_id

[alain....@gmail.com]

We use RabbitMQ primarily as an MQTT broker.

In our production environment, we observe the unexpected behavior that multiple MQTT clients may connect to the cluster using the same MQTT client id. This behavior was not always the case, however we not sure when this started.

In our non-production environment, we observe the expected behavior that multiple MQTT clients may not use the same client id.

Forensics
In non-prod, where the behavior is correct we see log messages like the following when we try to open a second connection using the same client id:

2021-10-21 19:14:15.453 [warning] <0.22343.715> MQTT disconnecting client "144.212.3.4:46336 -> 10.98.8.54:1883" with duplicate id 'Dh4lOywmAxYNIQUBLAQJAhI'

In production, where the behavior is incorrect we DO NOT see log messages like this.

Basic details

RabbitMQ Version: 3.8.5

Erlang Version: 23.3.1

Platform: Ubuntu 16.04.7 LTS on AWS EC2
Cluster of 3 nodes

Thank you,

Alain Kuchta

[alain....@gmail.com]

• Log-level is the same in both environments

[alain....@gmail.com]

I've discovered crash reports in the logs from several weeks ago that probably explain the issue. The raft state machine crashed due to an unhandled case clause. After being restarted three times, it just died for good.

See the attached log for details

[Ryan Pomario]

Hi -

I am experiencing the very same issue.

RabbitMQ 3.8.23 Erlang 24.1.2

[alain....@gmail.com]

Are you seeing similar errors / crash reports?

Based on my reading of the code, I thought this issue might have been resolved by this PR: https://github.com/rabbitmq/rabbitmq-server/pull/2693, which would be part of 3.8.23.

[Ryan Pomario]

Hi,

Yes - exact same error/crash message.  Sometimes it gets so bad that the entire broker crashes.  It is a real concern for me since the number of MQTT clients

 connecting to our infrastructure is growing steadily!

I did read somewhere that it is caused by clients that churn connections frequently and sometimes the connection drops as soon as it is just about to connect to the broker MQTT end-point.  This somehow then causes the erlang engine to crash as it can't handle the closure of this connection cleanly.  Clearly a bug somewhere that needs fixing urgently!

I hope a fix is found soon.

Virus-free. www.avast.com

--
You received this message because you are subscribed to a topic in the Google Groups "rabbitmq-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/rabbitmq-users/hoF5NFAwyv4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/4ad8ba7d-a903-4ede-98e3-8a11dced7551n%40googlegroups.com.

[Alexey Lebedeff]

Hi Alain,

Yes, your analysis for crash log is spot on, and these messages should go away with the given patch. It's also highly probable that the problem of the duplicated clients will also go away. As usual, the recommendation would be to upgrade. This patch is included in 3.8.11.

[Alexey Lebedeff]

Hi Ryan,

Could you please provide your crash logs for that problem? I believe they should be a bit different from what is provided by Alain - the exact "case_clause" error should be impossible in 3.8.23

[Alain Kuchta]

Thanks for confirming Alexey!

--
You received this message because you are subscribed to a topic in the Google Groups "rabbitmq-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/rabbitmq-users/hoF5NFAwyv4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to rabbitmq-user...@googlegroups.com.

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/f8a08253-49d9-40bf-bd9e-a13959e46c84n%40googlegroups.com.