Strange behavior with queue expiration

[Arnaud Morin]

Hey team,

We have a cluster of 3 rabbit nodes (r1, r2, r3).
We set a policy on our queues to be deleted after a while (1H) if there
is no consumer anymore.
This prevent us to have too much queues in the system.

We only use quorum and stream queues.

Our cluster is set in pause_minority mode, and our clients are able to
connect to any of the 3 nodes.

So when a cluster node is having issues (e.g. r3), our client reconnects
to another node.
So far, everything is fine.

Now, while r3 is isolated, it is paused: we can see that in logs, it
works.

Other nodes (r1 and r2) are still working as expected, and they apply
expire policy correctly (so some queues are deleted).

Now imagine a new queue is created while r3 is down.
This queue is using the same name as a previous queue that expired.
The cluster accepts the creation, but with only 2 replicas.
It works.

Now r3 comes back, we expect the old queue to be purged from r3 and the
replication of the new queue to be created.
What we observe instead is, in best scenario, the deletion of the queue,
and in worst scenario, the leader election fail and the queue endup in
noproc/broken state and becomes unusable.

What we observe is that {index,term} on the old queue is much higher
that on the new one.
r3 think it has the more recent data, and does not know that the queue
was deleted / recreated in the mean time.
r3 tries to become the leader, but that fail.

I tried to reproduce such behavior here:
https://github.com/arnaudmorin/rabbit-4-quorum-issue

Do you know if there is a easy way to avoid such behavior?
Some workaround?

Regards,
Arnaud.

[Michal Kuratczyk]

There's an ongoing PR to resolve similar situations and this one as well, hopefully:

https://github.com/rabbitmq/rabbitmq-server/pull/14241

Would be great if you could contribute to this PR (not necessarily with code, but

testing and making sure your situations is resolved by it).

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/rabbitmq-users/20251121090023.apne3j6fqmrb2pmo%40sync2.

--

Michal

RabbitMQ Team

[Arnaud Morin]

Nice, thank you, will test ASAP!

> To view this discussion visit https://groups.google.com/d/msgid/rabbitmq-users/CALWErWJk%2BBEXhHpJFqTSkumHEhcL08SmWu__wxLDN8xRJ3HiXw%40mail.gmail.com.