RabbitMQ queue consumer suspected down

[Cristian Rosu]

Hello, 

I have a cluster with 3 nodes on 3 different VMs (Windows server 2022), running RabbitMQ 4.1.0 on Erlang 27.3.3.

I have enabled Khepri on all of them.

I see in the log a lot of messages like below:

2025-08-10 18:48:49.937999+00:00 [info] <0.759267.0> queue 'order_status_q' in vhost '/': detected a new leader {'%2F_order_status_q',rabbit@RMQS02} in term 63

The problem is that when the leader for a queue is changed, the consumer for that queue will change the "activity status" from up to  suspected down and the messages in the queue are not processed anymore.

Has anyone had seen this issue before?

Thank you.

Cristian

[Michal Kuratczyk]

Quorum queues don't perform election without a good reason (loss of the leader, an explicit rebalance, node failure, etc).

If you see "detected a new leader" logs that you don't expect (you didn't attempt any restarts nor rebalances), this suggests

there are intermittent connectivity issues between RabbitMQ nodes. That also explains the "suspected_down" status for consumers.

So the leader change doesn't cause suspected_down status nor the stop of message processing but rather, there's a common

cause for the leader change and the consumer issue.

What happens later? Does message processing resume after some time? If not, then perhaps there's something to be improved

in how the cluster partition is handled but if you see a lot of unexpected leader changes, that's the main issue to be addressed - for some

reason communication between the nodes is not stable.

Best,

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/rabbitmq-users/83c8fcee-0d6a-4daa-8572-321e963eab95n%40googlegroups.com.

--

Michal

RabbitMQ Team

[Cristian Rosu]

Hi Michal,

Thank you for your quick reply.

To answer your question, the  message processing resume only after restarting the consumer service.

As you pointed out, the messages in the log suggests there are intermittent connectivity issues between RabbitMQ nodes.

2025-08-10 18:39:49.739999+00:00 [debug] <0.758825.0> RabbitMQ metadata store: Leader node rabbit@RMQS02 may be down, setting pre-vote timeout

2025-08-10 18:39:54.753999+00:00 [debug] <0.759267.0> queue 'order_status_q' in vhost '/': Leader node rabbit@RMQS02 is back up, cancelling pre-vote timeout

Though I monitor the network connection between the nodes on port 25672 and did not spot any gap.

Where the connectivity issues might come from, besides the networking?

Thank you.

Cristian

[Michal Kuratczyk]

Anything that affects communication between the nodes, so

apart from the network itself, it can be an overloaded distribution link

(https://www.rabbitmq.com/docs/runtime#distribution-buffer)

and/or overloaded nodes (one or both).

If it is the distribution link, you should find "busy_dist_port"

in the logs (you can also see more details with Prometheus/Grafana set up).

If your hypervisor performs live migrations of the VMs, that could also be the cause

(the VM is paused at least for a brief moment during a migration,

but at least on recent vSphere versions, that moment is so short that we haven't

observed any issues while testing this).

Potentially other issues on the hypervisor/OS level.

Best,

To view this discussion visit https://groups.google.com/d/msgid/rabbitmq-users/b36b1681-ee30-4719-abf4-42124150d53fn%40googlegroups.com.

--

Michal

RabbitMQ Team

[Cristian Rosu]

Thank you for the information, Michal. 

I don't find "busy_dist_port" in the logs, I will check with the VMWare team to see if there are any live migration for these VMs.

Cristian