RabbitMQ memory growth

71 views
Skip to first unread message

Narendra Sharma

unread,
Jul 6, 2019, 7:05:32 PM7/6/19
to rabbitmq-users
RabbitMQ: 3.7.15
3 node cluster, 16core and 128GB each
100K+ MQTT publishers publishing while AMQP consumers consuming.

The RabbitMQ memory is growing indefinitely in all the tests untill it reaches high watermark (Good that the process is not crashing as it blocks publishers).

I have not set any TTL for the messages. I want AMQP consumers to receive all messages.

If my publish rate is constant (3000 message per second) throughout the test (4hr) and AMQP consumers are reporting latency under 500ms, why would the memory continue to grow? When does RabbitMQ release the memory occupied by message metadata and index? Should I set message TTL for RabbitMQ to release the memory?

I plan to run the test to validate but I can save some time if I someone can throw some light on the implementation.

Karl Nilsson

unread,
Jul 8, 2019, 7:17:47 AM7/8/19
to rabbitm...@googlegroups.com
Hi,

Is your problem at all similar to this recent thread?


Cheers
Karl

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/fca49d2e-32ae-425f-af48-7615d0cd1714%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


--
Karl Nilsson

Pivotal/RabbitMQ

Narendra Sharma

unread,
Jul 15, 2019, 9:40:59 PM7/15/19
to rabbitmq-users
No.

In my test, all 500K connections and subscriptions are setup in first one hour. I am running test in a controlled environment so I am not seeing any connection churn issues (verified ss on clietnt and server machines).

Once all the connections are established, the next 4 hours I am running only Publish (QoS 0) test at 3K messages per second. I would expect the memory usage to reach a peak and stay flat or have a saw pattern. I have also enabled background_gc every 5 minutes.

For 4 hour 3K messages per second, I would expect the memory usage to peak at 32GB (3000 * 4 * 3600 * 780 bytes) + memory used by connection. However, the memory usage hits the high water mark (100GB) and stops all publishers. 
BTW, as per my test memory usage per connection (including connection, channel, queue, consumer) is 385KB. So for 500K connections the usage is approx 185GB across 3 node cluster.

I plan to run some test with message TTL this week and also capture memory metrics. 

If there are other theories for high memory usage, please share. 


 
On Monday, July 8, 2019 at 4:47:47 PM UTC+5:30, Karl Nilsson wrote:
Hi,

Is your problem at all similar to this recent thread?


Cheers
Karl

On Sun, 7 Jul 2019 at 00:05, Narendra Sharma <narendr...@gmail.com> wrote:
RabbitMQ: 3.7.15
3 node cluster, 16core and 128GB each
100K+ MQTT publishers publishing while AMQP consumers consuming.

The RabbitMQ memory is growing indefinitely in all the tests untill it reaches high watermark (Good that the process is not crashing as it blocks publishers).

I have not set any TTL for the messages. I want AMQP consumers to receive all messages.

If my publish rate is constant (3000 message per second) throughout the test (4hr) and AMQP consumers are reporting latency under 500ms, why would the memory continue to grow? When does RabbitMQ release the memory occupied by message metadata and index? Should I set message TTL for RabbitMQ to release the memory?

I plan to run the test to validate but I can save some time if I someone can throw some light on the implementation.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitm...@googlegroups.com.

To post to this group, send email to rabbitm...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/fca49d2e-32ae-425f-af48-7615d0cd1714%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


--
Karl Nilsson

Pivotal/RabbitMQ

Michael Klishin

unread,
Jul 31, 2019, 6:08:50 PM7/31/19
to rabbitmq-users
There are tools that will provide relevant data, please use them [1][2].


To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.

To post to this group, send email to rabbitm...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/ccc62008-fb8e-4e60-90ce-0c1f52f5d202%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.


--
MK

Staff Software Engineer, Pivotal/RabbitMQ
Reply all
Reply to author
Forward
0 new messages