Unable to add/remove members from quorum queue with no active members

[Pär Dahlman]

We have a cluster with three nodes running RabbitMQ 3.10.19 on ERL 25.3. There are a number of quorum queues where all members are offline (they have been decommissioned). We want these queues to only have members from the currently active nodes, and we are OK with recreating the queues. So far, we've tried the following:

Remove the queues using the management UI. This operation fails and the UI becomes unresponsive

Remove member from the queue using the rabbitmq-queues CLI. This operation fails with the following error message "no_more_servers_to_try" and 3x timeouts

Adding a new (existing) member to the queue also fails, with "no_more_servers_to_try" and 3x nodedown

Attached are screenshots from the management UI.

Any thought on how we can interact with these queues?

Best regards,

Pär

[Luke Bakken]

Hello,

You didn't say which rabbitmq-queues command you specifically tried, so please give this command a try: https://www.rabbitmq.com/rabbitmq-queues.8.html#delete_member

Please attach a transcript showing the command(s) you run and their full output.

Thanks,

Luke

[Luke Bakken]

Hi Pär,

More than likely that is the command you have already tried. I chatted with Karl (who implemented QQs) and we think that you probably have to remove the queue record from Mnesia:

rabbitmqctl eval 'Q = {resource, <<"virtual host name">>, queue, <<"queue name">>}, rabbit_amqqueue:internal_delete(Q, <<"CLI">>).'

Let me know how that goes.

Thanks,

Luke

[Luke Bakken]

PS to prevent this issue from happening in the future, please see this documentation: https://www.rabbitmq.com/quorum-queues.html#replica-management

On Thursday, March 16, 2023 at 8:07:40 AM UTC-7 Luke Bakken wrote:

[Pär Dahlman]

Luke (and Karl, too!). Thanks for taking the time to answer. We've executed the suggested command and it worked as expected. The queues are now deleted and we can continue with our migration! Thanks again!