RabbitMQ - MQTT - Memory usage peaks

[Martin Jurák]

Hello, I'm using rabbitmq about 1 year in production. During this time I had issues with memory usage peaks which hit high memory watermark for a few seconds (but not a big problem). However yesterday, rabbitmq memory usage was completely crazy, few seconds over the limit, few seconds under the limit. I tried to restart rabbitmq service, reboot server and chang memory allocators, but without any success.

RabbitMQ instance is:

• Ubuntu 18.04.1, 4 CPU, 16GB RAM.
• Continuously upgraded to RabbitMQ 3.8.2 (started at 3.7.x), erlang 22.2.4 (started at 20.x).
• Instance is used for realtime communication between two clients, so random clients connections/reconnections.
• Over time it had max 200 messages/s and 1000 connections in same time, max about 40 queued messages and message size ~20KB.

When issues appeared there was no high load or something different (50 messages/s and 200 connected clients). There are two issues:

1. It seems that OS keeps a huge amount of memory for the rabbitmq process and rabbitmq repeatedly moves memory between "unallocated, reserved by the OS" and "binaries".
2. When rabbitmq starts it has immediately 2GB binaries without any reason.

Here are screenshots of memory usage when peak appeared and "correct" state:

• peak
  
  

• "normal"
  
  

For more information, I attach stats from recon_alloc (fragmentation, allocators, bin_leak, memory stats).

I read a lot about rabbitmq/erlang memory, allocators and read mailing lists about rabbitmq memory issues, but without any luck with my issues.

I will appreciate if somebody can look at it and check what could be wrong or show me direction where find the issue.

Thank you.

[Johan Rhodin]

Interesting case. I don't have a solution, but am interested in what your flags are? (Verify with "ps aux | grep beam")
How big is the data folder ("du -sh /var/lib/rabbitmq")? How much messages does the node have in message bytes? (i.e. is it possible to load 2GB at startup into binaries)
What are the biggest memory users according to rabbitmq-top? Also (rabbitmqctl eval 'rabbit_diagnostics:top_memory_use().'
and rabbitmqctl eval 'rabbit_diagnostics:top_binary_refs().')
Does manual gc help? (rabbitmqctl eval '[garbage_collect(P) || P <- processes()].')
Anything interesting in the report (rabbitmqctl report)? feel free to attach it to this thread.

/Johan

> On Feb 2, 2020, at 7:17 AM, Martin Jurák <marti...@gmail.com> wrote:
>
> Hello, I'm using rabbitmq about 1 year in production. During this time I had issues with memory usage peaks which hit high memory watermark for a few seconds (but not a big problem). However yesterday, rabbitmq memory usage was completely crazy, few seconds over the limit, few seconds under the limit. I tried to restart rabbitmq service, reboot server and chang memory allocators, but without any success.
>
> RabbitMQ instance is:
> • Ubuntu 18.04.1, 4 CPU, 16GB RAM.
> • Continuously upgraded to RabbitMQ 3.8.2 (started at 3.7.x), erlang 22.2.4 (started at 20.x).
> • Instance is used for realtime communication between two clients, so random clients connections/reconnections.
> • Over time it had max 200 messages/s and 1000 connections in same time, max about 40 queued messages and message size ~20KB.
> When issues appeared there was no high load or something different (50 messages/s and 200 connected clients). There are two issues:
> • It seems that OS keeps a huge amount of memory for the rabbitmq process and rabbitmq repeatedly moves memory between "unallocated, reserved by the OS" and "binaries".
> • When rabbitmq starts it has immediately 2GB binaries without any reason.
>
> Here are screenshots of memory usage when peak appeared and "correct" state:
> • peak

> <memory.png>
> • "normal"

> <memory-ok.png>
>
> For more information, I attach stats from recon_alloc (fragmentation, allocators, bin_leak, memory stats).
>
> I read a lot about rabbitmq/erlang memory, allocators and read mailing lists about rabbitmq memory issues, but without any luck with my issues.
> I will appreciate if somebody can look at it and check what could be wrong or show me direction where find the issue.
>
> Thank you.
>

> --
> You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
> To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/e3d5382c-754f-4de7-aefc-9e08e5c9a43e%40googlegroups.com.
> <allocators><bin_leak><fragmentation-current><fragmentation-max><memory-allocated_types><memory.png><memory-ok.png>