Problem to connect to cluster. [error] emulator Discarding message

[Сергей Беспалов]

I install cluster in kubernetes with bitnami helm chart 

rabbitmq:3.8.14

I have three node

rab...@rabbitmq-0.rabbitmq-headless.infra-services.svc.cluster.local

rab...@rabbitmq-1.rabbitmq-headless.infra-services.svc.cluster.local

rab...@rabbitmq-2.rabbitmq-headless.infra-services.svc.cluster.local

Cluster have no real load.

rabbitmq-0 logs show infinite loop of 

2021-09-06 17:24:11.953 [error] <0.695.0> Discarding message {'$gen_call',{<0.695.0>,#Ref<0.2044302747.1235746817.160248>},stat} from <0.695.0> to <0.27015.38> in an old incarnation (1630216132) of this node (1630938826)

2021-09-06 17:24:11.954 [error] emulator Discarding message {'$gen_call',{<0.695.0>,#Ref<0.2044302747.1235746817.160248>},stat} from <0.695.0> to <0.27015.38> in an old incarnation (1630216132) of this node (1630938826)

My queue have: 

"policies": [
{
"vhost": "box",
"name": "ha",
"pattern": ".",
"apply-to": "queues",
"definition": {
"ha-mode": "all",
"ha-sync-mode": "automatic"
},
"priority": 0
}
]

In managment console I see : All queues on node rabbitmq-0 have "Unsyncronized mirror: rabbitmq-1, rabbitmq-2"

I try run on rabbitmq-1 

 rabbitmqctl sync_queue celery@disposal_print.celery.pidbox -p box

Synchronising queue 'celery@disposal_print.celery.pidbox' in vhost 'box' ...

Error:

{:absent, {:amqqueue, {:resource, "box", :queue, "celery@disposal_print.celery.pidbox"}, false, true, :none, [{"x-expires", :signedint, 10000}, {"x-message-ttl", :signedint, 300000}], #PID<11901.26443.38>, [#PID<11902.6023.0>, #PID<11613.6021.0>], [], [], [vhost: "box", name: "ha", pattern: ".", "apply-to": "queues", definition: [{"ha-mode", "all"}, {"ha-sync-mode", "automatic"}], priority: 0], :undefined, [{#PID<11613.6022.0>, #PID<11613.6021.0>}, {#PID<11902.6024.0>, #PID<11902.6023.0>}], [], :live, 0, [], "box", %{user: "box"}, :rabbit_classic_queue, %{}}, :timeout}

Most consumer could not connect to their queues. 

Looks like first node became unstable and prevent normal work of all cluster. 

Thanks, Sergey