Mirror queue 2/3 nodes out of memory suddently

[王超]

Hello,
I have a cluster of 3 RabbitMQ nodes. 2 of the nodes running out out memory once a week.It was very strange that It happend suddently: the memory of the process which named beam.smp was 200M, but it increase suddently to 16G suddently in several seconds. My prometheus get the metrics every 30s, it didn't even catch the increase.I see the oom in the system log. It increase suddently to 16G and killed by the system. What can I do for this oom issue?

My environment:
    System: CentOS Linux release 7.9.2009 (Core), 8C32G
    Kernal: 3.10.0-1160.6.1.el7.x86_64
    RabbitMQ: 3.8.18
    Erlang: 24.0
    node: all the 3 nodes are disk node
    queue: mirror queue and "ha-mode":"all"

MA config
    vm_memory_high_watermark.relative = 0.2 # there are some java process also in those nodes, so I set 0.2
    vm_memory_high_watermark_paging_ratio = 0.5
    disk_free_limit.absolute = 500MB
    management.tcp.port = 15672
    management.tcp.ip = 0.0.0.0
    log.file.level = info
    log.file.rotation.size = 10485760
    log.file.rotation.count = 7
    consumer_timeout = 86400000

System oom kill log
    Killed process 30128 (beam.smp), UID 1001, total-vm:20165616kB, anon-rss:16865564kB, file-rss:1156kB, shmem-rss:0kB

The metrics of the oom time

[Luke Bakken]

Hello,

RabbitMQ 3.8.X is completely out of support. Please upgrade.

Your issue sounds similar to that reported here - https://groups.google.com/g/rabbitmq-users/c/Kr2QZx1TREA

Thanks -

Luke

[chao wang]

I saw CentOS7 can not use erlang 24 in this page https://github.com/rabbitmq/rabbitmq-server/discussions/4637. Is it possible be the reason of this issue？

[chao wang]

Hello,

I have another question：In my opinion, I set vm_memory_high_watermark.relative=0.2,it means the max memory which rabbitmq use is 32*0.2*2=12.8G. But sometime,the rabbitmq cased system oom, I saw the system log,rabbitmq's memory reached 16GB(Killed process 30128 (beam.smp), UID 1001, total-vm:20165616kB, anon-rss:16865564kB, file-rss:1156kB, shmem-rss:0kB) when the time of oom happened. Is it possible that the maximum memory is more than twice the size of vm_memory_high_watermark?

在2022年9月8日星期四 UTC+8 01:39:24<luker...@gmail.com> 写道：

[4118...@qq.com]

hi 我和你遇到了一模一样的问题 oom，且内存激增，监控抓不到，超水位，这个问题后来，你们有排查出来什么吗？ 

[Michal Kuratczyk]

Newer versions of RabbitMQ handle memory much better. There are multiple reasons for this:

1. Quorum queues don't keep messages in memory anymore

2. Classic queues only keep a very small number of messages in memory (since 3.12)

3. Queue mirroring (which should no longer be used) has some improvements (https://github.com/rabbitmq/rabbitmq-server/issues/5312)

4. Mirrored queues can have a max synchronization throughput set to prevent spikes (mirroring_sync_max_throughput)

Before you look for other solutions, please apply the most obvious one - use a supported version that has multiple improvements.

And if you see this problem in 3.12 - please provide the details to reproduce.

Best,

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/43aad73b-d835-4bd7-97a4-c4147da1b286n%40googlegroups.com.

--

Michał

RabbitMQ team