Issue with Queue Mirroring Policy in RabbitMQ Cluster

[sohe...@gmail.com]

Environment Details:

• Cluster Size: 3 (Docker-based)
• RabbitMQ Version: 3.13.7
• Queue Name Pattern: ^routing.delayed..*
• Policy Configuration:
  • dead-letter-exchange: routing.delay
  • ha-mode: exactly
  • ha-sync-mode: automatic
  • ha-params: 2
  • priority: 9

Issue Description:

According to the ha-params in the policy, the queues matching the pattern should have 1 leader and 1 mirror node, and this configuration is usually applied correctly. However, at random times (day or night), the queue suddenly ends up with 1 leader and 2 mirror nodes, despite the policy still being attached (verified via the management console).

To fix this, I delete and reapply the policy, which temporarily restores the expected behavior. However, the issue reappears after some time (usually within a day).

Steps Already Taken:

• Verified that the policy is correctly attached to the queue.
• Manually reapplied the policy to fix the configuration temporarily.

I’m unsure why the queue sometimes doesn’t respect the defined policy. Any insights or suggestions would be greatly appreciated.

[jo...@cloudamqp.com]

Hi,

Do you happen to have another policy with the same priority (in this vhost)? From the docs [0]: "When multiple policies match an entity and they all have equal priorities, the effective one will be chosen undeterministically. Such cases should be avoided by paying attention to what priorities various policies use."

Note that classic mirrored queues (CMQ) are deprecated and removed in version 4 of RabbitMQ.

/Johan

[0] https://www.rabbitmq.com/docs/parameters#operator-policy-definition

[sohe...@gmail.com]

There are no same priority policies in that vhost.  Not even in the cluster.

[jo...@cloudamqp.com]

OK, can you check that the queues actually have that behavior? Run this:

rabbitmqctl list_queues -p YOURVHOST name mirror_pids effective_policy_definition

Once when you have 1 mirror, and once when you have 2. I also suggest you set queue-version:2 in your policy.

/Johan

[sohe...@gmail.com]

Hey, Here is the output 

When 1 (expected):

routing.delayed.600 [<rab...@ip-10-100-33-22.1731572647.119937955.1>] [{<<"dead-letter-exchange">>, ex.routing.post-delay}, {<<"ha-mode">>, exactly}, {<<"ha-params">>, 2}, {<<"ha-sync-mode">>, automatic}]

When 2 (unexpected):

routing.delayed.600 [<rab...@ip-10-100-38-50.1731572650.262150484.0>, <rab...@ip-10-100-33-22.1731572647.189573674.1>] [{<<"dead-letter-exchange">>, ex.routing.post-delay}, {<<"ha-mode">>, exactly}, {<<"ha-params">>, 2}, {<<"ha-sync-mode">>, automatic}]

we have the same queue that has same pattern (e.g: routing.delayed.200) with the same vhost and using the same policy but that never got affected.

[jo...@cloudamqp.com]

Hi,

one more thing you can try is to run

rabbitmqctl eval 'Vhost = <<"VHOSTNAME">>, QName = <<"QUEUENAME">>, QPid = rabbit_amqqueue:pid_of(Vhost, QName), QInfo = recon:info(QPid), QState = recon:get_state(QPid), io:format("QUEUE INFO:~n~p~n~nQUEUE STATE:~n~p~n", [QInfo, QState]).'

when it is the correct policy and when it is the incorrect policy. That might give some hint.

Ideally you would trace this with recon [0], to see what is happening when the policy changes.

/Johan

[0] http://ferd.github.io/recon/recon_trace.html