Classic mirror queue rebalance can lose messages if leader downed before sync complete

[Rich Bramante]

We have been testing this scenario on RabbitMQ 3.9.5 and it is reproducible. The issue is almost exactly the same is is described here https://github.com/rabbitmq/rabbitmq-server/issues/2749 which the exception that the rebalance request initiates the processing, otherwise the issue with classic queues and an ha-exactly policy of 2 (less than 3 node cluster) behave similarly.

What has been observed:

1. 3 node cluster. ha-mode exactly 2.

2. Classic mirror Q on A (leader) and B (replica)

3. Create a backlog in the queue (1M 1K events in our tests)

4. Initiate a rebalance via the Mgmt API

5. The rebalance drops A, then B promotes and then adds C as the new replica and starts syncing

6. If before the sync completes, B goes down, C promotes to the leader and adds A as the replica. The queue is now empty.

I realize that this is all legal per our ha-promote-on-failure being the default of always, but should the rebalance be more cautious here and ensure the new replica is in-sync before dropping an existing? Although I suppose that means ha-exactly 2 policy is in violation during the sync?

I just want to make sure our understanding of what is happening here is correct and if there are other steps we could take to avoid this type of issue short of transitioning to quorum queues are forcing replication to all nodes (which won't scale for us on larger deployments).

Would this behavior be considered a bug candidate or is it defined system behavior based on configuration selection of ha-policy with classic queues and leader promote choices?

[Michal Kuratczyk]

Hi,

It certainly sounds like a bug but remember that any solution has trade-offs.

1. If we wait for sync to be complete, we may never complete the rebalance process (it's an edge case but we need to consider that)

2. If you say you cannot have 3 mirrors, then you probably can't have 3 mirrors temporarily either (so adding a new one before dropping the old one is only an option if we rebalance queues one by one or in small batches).

As a workaround, you could probably change the mirroring policy to 3 nodes and then back down to 2 for subsets of the queues to basically achieve rebalancing in small batches.

Please note that classic queue mirroring will be removed in the future. Sticking to classic queue mirroring just because quorum queues require 3 replicas is a dead-end - if you use RabbitMQ at scale then you should be able to provision enough resources to support that scale. If there are issues with quorum queues - please report them.

Best,

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/16c434da-56a5-417b-bc5b-9cde18672448n%40googlegroups.com.

--

Michał

RabbitMQ team

[Michal Kuratczyk]

Just one more thought - not sure if you created a backlog of 1 million messages just as an example or is this a common backlog size in your use case but perhaps Streams is what you are looking for, rather than quorum queues?

--

Michał

RabbitMQ team