quorum queue with undefined Snapshot Index

[julien cosmao]

Hello,

I have dozen of queues with a weird status, ended in such state after a network partition (i don't have exact chronology of what happen in details)

Is there a way i can detect and recover queues from this state ?

rabbit-neutron1 $ rabbitmq-queues quorum_status neutron-vo-Trunk-1.1.host1234
Status of quorum queue neutron-vo-Trunk-1.1.host1234 on node rabbit@rabbit-neutron1 ...
┌────────────────────────┬────────────┬───────────┬──────────────┬────────────────┬──────┬─────────────────┐
│ Node Name              │ Raft State │ Log Index │ Commit Index │ Snapshot Index │ Term │ Machine Version │
├────────────────────────┼────────────┼───────────┼──────────────┼────────────────┼──────┼─────────────────┤
│ rabbit@rabbit-neutron3 │ follower   │ 60        │ 60           │ 56             │ 2    │ 3               │
├────────────────────────┼────────────┼───────────┼──────────────┼────────────────┼──────┼─────────────────┤
│ rabbit@rabbit-neutron1 │ leader     │ 60        │ 60           │ 56             │ 2    │ 3               │
├────────────────────────┼────────────┼───────────┼──────────────┼────────────────┼──────┼─────────────────┤
│ rabbit@rabbit-neutron2 │ follower   │ 0         │ 0            │ undefined      │ 0    │ 3               │
└────────────────────────┴────────────┴───────────┴──────────────┴────────────────┴──────┴─────────────────┘

I'm not able to remove a member and add it back.

rabbit-neutron1 $ rabbitmq-queues delete_member neutron-vo-Trunk-1.1.host1234 rabbit@rabbit-neutron2
Removing a replica of queue neutron-vo-Trunk-1.1.host1234.uk1.cloud.ovh.net on node rabbit@rabbit-neutron2...
Error:
{:no_more_servers_to_try, [timeout: {:"%2F_neutron-vo-Trunk-1.1.host1234", :"rabbit@rabbit-neutron2"}, error: :not_member, error: :not_member]}


version: RabbitMQ 3.12.0 on Erlang 25.2.2


Cheers

[Luke Bakken]

Hello,

Without knowing the series of events that lead to this state, we can't explain why.

A colleague suggests that you forget node 2, reset it, re-add it to the cluster, then grow the quorum queues to include it.

Thanks,

Luke