RabbitMQ running out of memory

[Michael]

Hello,

I have a cluster of three RabbitMQ nodes and I would like to understand why my cluster is running out of memory after few days. When I start the cluster, the used memory is about 400MB but every day the memory is increasing even if the traffic and the number of queues are pretty much the same.

It seems that quorum queue tables are constantly increasing their size, can you explain me what are the quorum queue tables and why they are growing its size ??

After several days the rabbitMQ hit the high watermark so it stops to accept connections. I tried already to increase the watermark to 1.3GB but the only difference was that the problem occurred after more days.

Thank you for help

Please find attached screenshots after running 2 days 

[jo...@cloudamqp.com]

Hi,

Please help us help you: what is the RabbitMQ and Erlang versions? What is the number of QQs (and classic queues)? Message sizes?

What is your "raft.wal_max_size_bytes" ?  See https://www.rabbitmq.com/quorum-queues.html#resource-use for an explanation of why this matters.

If your not on 3.10 set x-max-in-memory-length=0 https://www.rabbitmq.com/quorum-queues.html#memory-limit. And upgrade :)

/Johan

[Michael]

Hello,

The versions are RabbitMQ 3.9.14, Erlang 24.3.3

We have about 240 Quorum Queues and few classic queues. The messages are in general quite small 4 - 50KB in general.

We did not set raft.wal_max_size_bytes so it will be default value.

Thanks for help, I will try to setup the raft.wal_max_size_bytes and the x-max-in-memory-length=0

Dne pátek 5. srpna 2022 v 17:50:49 UTC+2 uživatel jo...@cloudamqp.com napsal:

[Michael]

Hello,

just one more update, when the node is hitting the watermark, the memory is mainly taken by preallocated by runtime, unused memory

Please have a look in the attachment 

Dne pondělí 8. srpna 2022 v 15:55:28 UTC+2 uživatel Michael napsal:

[jo...@cloudamqp.com]

Upgrade to 3.10.7 to get Quorum queues version 2 and see if the issue still remains.

/Johan

[Michael]

Hello,

the issue seems to be still here. This is a screenshot from this morning, the node 0 is running out of memory again (with the 3.10.7 it just took a longer time)

Here is a screenshot of memory details:

It seems that this time the queues are not taking a lot of space but it is the preallocated memory by runtime which takes the majority.

Why the node preallocate so much memory at some point ? 

Dne středa 10. srpna 2022 v 17:29:59 UTC+2 uživatel jo...@cloudamqp.com napsal:

[Luke Bakken]

Hello -

At this point to help us diagnose this issue, please provide...

• Your full, un-edited RabbitMQ configuration files. Attach them to your response.
• An export of your definitions.
• A more detailed description of your workload. Ideally, you would be able to reproduce it (or some of it) using PerfTest - https://rabbitmq.github.io/rabbitmq-perf-test/stable/htmlsingle/
• Are you still using Erlang 24? Can you reproduce this issue using 25.0.3?
• What operating system and version are you using? In what environment?

Thanks -

Luke

[Michal Kuratczyk]

Hi,

Regarding preallocated memory, perhaps you are hitting this issue: https://blog.rabbitmq.com/posts/2022/08/high-initial-memory-consumption-of-rabbitmq-nodes-on-centos-stream-9/

Best,

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/50d26296-8ecd-4f84-bc21-853cf1aa411fn%40googlegroups.com.

--

Michał

RabbitMQ team

[kjnilsson]

Was the raft.wal_max_size_bytes configuration lowered as advised on Aug 5? I suggest setting it to ~128MiB and see how that goes.

Quorum queues will use memory in a "saw tooth" like pattern and the vm is likely to pre-allocate to accommodate this.

The amount of memory on these node is insufficient to support the current memory peaks so you can either lower raft.wal_max_size_bytes as this will also lower the memory use peaks _or_ assign more memory to your nodes.

https://www.rabbitmq.com/quorum-queues.html#resource-use

Cheers

Karl

[Michael]

Hello,

the raft.wal_max_size_bytes value was lowered and we also increased the high watermark value to 1.4 GB and the cluster is running already 30 days without any problem.

Thank you for help !

Dne čtvrtek 1. září 2022 v 11:43:20 UTC+2 uživatel kjnilsson napsal: