Mirrored queues cause a node to run out of memory
Bilal Pierre ABDELKADER
Hello
We are running a cluster of three RabbitMQ nodes and noticed one node is always running out of memory after load test.
we have RabbitMQ version 3.8.9, Erlang 23.1, with active federation and mirrored queues.
The problem seems to be related to mirrored queues and is only happening on the mirrored node. The master node [master queue], is maintaing a normal memory profile.
When all messages are done processed by the Master, the mirrored node still running out of memory for an hour later.
The vm highwater mark is set to 0.4 of 16GB of total memory, RabbitMQ is running out to 6.5 GB.
Top plugin on the mirrored node shows a process mirror_queue_slave using normal amount of RAM and CPU Reduction/sec.
Running rabbitmq-diagnostics observer --interval 10 during message processing, shows less than 1GB of Used Memory and 4,7GB of Unused Memory, while Binary still occupy 4,2GB !!!
This problem is reproducible, with 600 000 messages injected during 10 min. The message mean size is 10 KB. After rebooting the cluster, one mirror is elected master, the master changes to mirror, and the same phenomenon occurs.
I've seen version RabbitMQ 3.1.0 fixes a similar behaviour: Bug Fix 25524 fix memory leak in mirror queue mirror with many short-lived publishing channels.
Any help is appreciated.
Thanks in advance