RabbitMQ Consumers Missing Issue on Version 3.9.7

31 views
Skip to first unread message

x yang

unread,
Jun 3, 2024, 9:30:56 PMJun 3
to rabbitmq-users
I am experiencing an issue with my RabbitMQ cluster, which consists of two nodes running version 3.9.7 in autoheal mode.

When a network partition occurs and the network is restored around the nettick_timeout , messages in certain queues start to backlog and are no longer delivered to consumers. Packet capture analysis shows that the corresponding connections are only sending heartbeats to each other.

More information:

1. On the web management page, the queues show non-empty consumers. However, using the rabbitmqctl list_queues consumers command reveals that the queues actually have no consumers.
2. The affected queues share a common characteristic: they use both mirroring and durability attributes.
3. RabbitMQ logs indicate that autoheal triggered a node restart, and there were mnesia database inconsistency warnings before the restart.
This issue is quite critical because there are no clear symptoms indicating cluster or queue anomalies. Has anyone else encountered a similar issue, or is there a known solution without upgrading the RabbitMQ version?

Any insights or solutions are greatly appreciated.

Thank you!

x yang

unread,
Jun 4, 2024, 7:42:18 AMJun 4
to rabbitmq-users

Here are some additional points:

  1. After reading the source code, I found that the consumer information used by message delivery and rabbitmqctl is stored in an internal process record named consumers, while the web management tool reads from an ETS table. This discrepancy might be causing inconsistencies.
  2. From the logs, it appears that after an autoheal restart, the primary replica of the faulty queue changed. I would like to ask if the consumers are only stored on the primary replica of the mirrored queue, and whether they become inaccessible when the primary node changes.
Reply all
Reply to author
Forward
0 new messages