rabbit_mqtt_qos0_queue cannot be declared if the queue leader is down

[Woon Yung Liu]

Hi all,

I have a 3-node cluster, which runs RabbitMQ 3.12.10 and Erlang 26.1.2. The project requires a rather high level of availability, so we use a ha-all policy in an attempt to allow the MQTT client to connect and work with any node, regardless of the QoS settings used. Since the MQTT clients are expected to always work, regardless of whichever node is down.

If the client connects to a node with QoS0 and with the clean session bit set,  the client fails to reconnect and function from another node upon node failure. An error like this appears in the RabbitMQ logs, of the new node:

2023-12-20 22:49:43.372788+08:00 [info] <0.43155.0> Accepted MQTT connection 10.105.0.18:60508 -> 10.105.0.10:1883 for client ID nodered_24e214feb018a232
2023-12-20 22:49:43.375120+08:00 [debug] <0.43155.0> Received a SUBSCRIBE for topic(s) [{mqtt_topic,
2023-12-20 22:49:43.375120+08:00 [debug] <0.43155.0>                                        <<"as923/gateway/+/command/#">>,0}]
2023-12-20 22:49:43.376225+08:00 [error] <0.43155.0> Failed to declare queue 'mqtt-subscription-nodered_24e214feb018a232qos0' in vhost '/': {absent,
2023-12-20 22:49:43.376225+08:00 [error] <0.43155.0>                                                                                         {amqqueue,
2023-12-20 22:49:43.376225+08:00 [error] <0.43155.0>                                                                                          {resource,
2023-12-20 22:49:43.376225+08:00 [error] <0.43155.0>                                                                                           <<"/">>,
2023-12-20 22:49:43.376225+08:00 [error] <0.43155.0>                                                                                           queue,
2023-12-20 22:49:43.376225+08:00 [error] <0.43155.0>                                                                                           <<"mqtt-subscription-nodered_24e214feb018a232qos0">>},
2023-12-20 22:49:43.376225+08:00 [error] <0.43155.0>                                                                                          true,
2023-12-20 22:49:43.376225+08:00 [error] <0.43155.0>                                                                                          false,
2023-12-20 22:49:43.376225+08:00 [error] <0.43155.0>                                                                                          <15486.32690.0>,
2023-12-20 22:49:43.376225+08:00 [error] <0.43155.0>                                                                                          [],
2023-12-20 22:49:43.376225+08:00 [error] <0.43155.0>                                                                                          <15486.32690.0>,
2023-12-20 22:49:43.376225+08:00 [error] <0.43155.0>                                                                                          [],
2023-12-20 22:49:43.376225+08:00 [error] <0.43155.0>                                                                                          [],
2023-12-20 22:49:43.376225+08:00 [error] <0.43155.0>                                                                                          [],
2023-12-20 22:49:43.376225+08:00 [error] <0.43155.0>                                                                                          [{vhost,
2023-12-20 22:49:43.376225+08:00 [error] <0.43155.0>                                                                                            <<"/">>},
2023-12-20 22:49:43.376225+08:00 [error] <0.43155.0>                                                                                           {name,
2023-12-20 22:49:43.376225+08:00 [error] <0.43155.0>                                                                                            <<"ha-all-mqtt">>},
2023-12-20 22:49:43.376225+08:00 [error] <0.43155.0>                                                                                           {pattern,
2023-12-20 22:49:43.376225+08:00 [error] <0.43155.0>                                                                                            <<"^mqtt-">>},
2023-12-20 22:49:43.376225+08:00 [error] <0.43155.0>                                                                                           {'apply-to',
2023-12-20 22:49:43.376225+08:00 [error] <0.43155.0>                                                                                            <<"all">>},
2023-12-20 22:49:43.376225+08:00 [error] <0.43155.0>                                                                                           {definition,
2023-12-20 22:49:43.376225+08:00 [error] <0.43155.0>                                                                                            [{<<"ha-mode">>,
2023-12-20 22:49:43.376225+08:00 [error] <0.43155.0>                                                                                              <<"all">>}]},
2023-12-20 22:49:43.376225+08:00 [error] <0.43155.0>                                                                                           {priority,
2023-12-20 22:49:43.376225+08:00 [error] <0.43155.0>                                                                                            0}],
2023-12-20 22:49:43.376225+08:00 [error] <0.43155.0>                                                                                          undefined,
2023-12-20 22:49:43.376225+08:00 [error] <0.43155.0>                                                                                          [],
2023-12-20 22:49:43.376225+08:00 [error] <0.43155.0>                                                                                          undefined,
2023-12-20 22:49:43.376225+08:00 [error] <0.43155.0>                                                                                          live,
2023-12-20 22:49:43.376225+08:00 [error] <0.43155.0>                                                                                          0,
2023-12-20 22:49:43.376225+08:00 [error] <0.43155.0>                                                                                          [],
2023-12-20 22:49:43.376225+08:00 [error] <0.43155.0>                                                                                          <<"/">>,
2023-12-20 22:49:43.376225+08:00 [error] <0.43155.0>                                                                                          #{user =>
2023-12-20 22:49:43.376225+08:00 [error] <0.43155.0>                                                                                             <<"iottester">>},
2023-12-20 22:49:43.376225+08:00 [error] <0.43155.0>                                                                                          rabbit_mqtt_qos0_queue,
2023-12-20 22:49:43.376225+08:00 [error] <0.43155.0>                                                                                          #{}},
2023-12-20 22:49:43.376225+08:00 [error] <0.43155.0>                                                                                         nodedown}
2023-12-20 22:49:43.376754+08:00 [error] <0.43155.0> MQTT protocol error on connection 10.105.0.18:60508 -> 10.105.0.10:1883: subscribe_error

How do I achieve this level of HA, with the new type of queue?

Thank you for your time.

[David Ansari]

Fixed by https://github.com/rabbitmq/rabbitmq-server/pull/10203

Thanks for reporting this issue.

[David Ansari]

Hey Woon,

actually try out RabbitMQ 3.12.12 (containing https://github.com/rabbitmq/rabbitmq-server/pull/10244).

Let us know whether it solves the issue you reported.

Cheers,

David

[Woon Yung Liu]

Hi, hello.

Thank you and whoever else who was involved in this fix. I have been updating whenever a new release comes out with MQTT fixes mentioned.
I have been noting no/less problems since 3.12.11 and I have not heard of any complaints regarding this matter at 3.12.12. So I think the problem was fixed.

Thank you for helping me solve this huge headache. It was a pleasant Christmas and New Year present.

Yours Sincerely,

Woon Yung

[Michael Klishin]

Thank you for reporting back!

[Michael Klishin]

Note that an "ha-all" policy likely doesn't have the effect you expect it to have, and classic queue mirroring is going away completely later this year.

This is very explicitly mentioned in the docs [1].

Replicated queue types generally do not make sense for QoS 0 clients (since QoS 0 is meant to be used for ephemeral data) but JFYI, you can

set the default queue type on a virtual host to make all queues quorum ones, unless they were explicitly declared as classic queues.

See 'rabbitmqctl help add_vhost' for details.

HTH.

https://rabbitmq.com/ha.html

[Woon Yung Liu]

Hi Michael,

Thank you for bringing this up too. Coincidentally, I've been wondering about that statement about classic queue mirroring going away.

I found myself having to set up classic queue mirroring to allow for High Availability to work for (at least) QoS1 with clean sessions, since the present-day support for quorum queues under the MQTT plugin only supports QoS1 without clean sessions. As described in the original PR: https://github.com/rabbitmq/rabbitmq-server/pull/4254

> Enable the quorum queue for MQTT only if CleanSession is False.
> QQs don't support auto-delete flag so in case Clean session is True
> the queue will be a classic queue.

So, does this mean that the RabbitMQ team intends to change this behaviour, to allow quorum to serve all the QoS1 usecases? If classic queue mirroring will be dropped in its present state, I am concerned that HA will not work for QoS1 with clean sessions.

Thank you for your time!