Memory constantly increasing - possibly fragmentation?

[Basti]

Hey,

we are running a RabbitMQ cluster with version 3.6.11 . We are experiencing memory issues: while our queues are empty all the time, memory usage gradually increases a lot. Within a day, one node goes from 500 MB to 8GB. Another issue is that when eventually a memory limit alarm triggers on one node, memory usage will not decrease again. 

We see that almost all of our memory is used in the System/Other category, which is - as I understand from posts in this forum - memory allocated by the runtime.  

So I figure this is either a GC or a runtime (Erlang VM) problem.

In terms of GC, could this be related to some kind of bug and that upgrading to the latest 3.6.X version might fix this behaviour?

In terms of Erlang runtime, how exactly is it possible to set VM flags for RabbitMQ in order to tune the memory allocator? 

I also read here that this kind of behaviour might be a possible memory fragmentation. At the end of this (https://groups.google.com/forum/#!msg/rabbitmq-users/msps9BYUnm0/htedJ1jPAQAJ) thread, one user suggested a code snippet where you can check if you are affected by that. How is it possible to run this inside the RabbitMQ VM ?  

Thanks for all answers.

[Luke Bakken]

Hello,

What version of Erlang are you using? That is an important data point that you did not provide.

If you would like to run the Erlang snippet from that other rabbitmq-users message, run this command:

rabbitmqctl eval '[{{A, N}, Data} || A <- [temp_alloc, eheap_alloc, binary_alloc, ets_alloc, driver_alloc, sl_alloc, ll_alloc, fix_alloc, std_alloc], {instance, N, Data} <- erlang:system_info({allocator,A})].'

If you could redirect the output of that command to a file and provide it, it may show something interesting.

You can also enable background garbage collection in the rabbitmq.config file -

[

    {rabbit, [

        {background_gc_enabled, true}

    ]}

].

Thanks,

Luke

[Basti]

Hey Luke,

thanks for your help!

We are running on Erlang version 17.3, erts 6.2 .

I attached the output of the command in a file.

Thanks,

Basti

[zhenbo zeng]

I met the same problem in some use case, and use recon_alloc:fragmentation/1 and other functions from recon  to show the detail

I read some docs, and still in processing.

http://www.rabbitmq.com/memory-use.html

http://erlang.org/doc/man/erts_alloc.html

if you solved the problem, please note me here

在 2018年2月26日星期一 UTC+8下午5:52:48，Basti写道：

[Basti]

Hey,

thanks a lot for your reply. The command you provided helped us to be sure that it was a memory fragmentation issue.  mbcs_usage was <5% on every node for "binary_alloc" position, which did allocate large amounts of memory. We then upgraded to RabbitMQ 3.6.14 and Erlang 19.3.6 . However that did not improve things, memory would still rise quickly by gigabytes over a few days. We then started RabbitMQ with "ERL_FLAGS=+MBas aobf +MBlmbcs 512" and now we don't experience that at all anymore, it's like night and day. 

There is a very good blog post about why memory fragmentation happens and how these flags help to prevent it.

https://blog.heroku.com/logplex-down-the-rabbit-hole#how-erlang-39-s-memory-stuff-works

[Michael Klishin]

Thank you, Basti.

We will introduce a doc guide around memory allocators at some point. Those values

are useful to have as a specific example that helped.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

[zhenbo zeng]

I try to enable background gc in one RabbitMQ Server, three days later, the memory usage doesn't increase anymore, then enable background gc in others, now everything goes normal. Version: RabbitMQ 3.6.10 and Erlang/OTP 19.3.6

It seems that enable background gc can reduce(or prevent) memory fragmentation.

I have tuned the VM arguments `+MBas aobf +MBlmbcs 512` on test environment before, but still have memory fragmentation. Maybe I should try to tune the arguments on production environment.

{background_gc_enabled, true},
{background_gc_target_interval, 120000},

[Michael Klishin]

I'd not expect background GC to prevent memory fragmentation. It can potentially reduce it

but not a lot of research was done on that. Anyhow, if it helps your case, use it (on all nodes, not selectively).

The downside of having background GC enabled are pretty well studied on this list (higher context switching rate, CPU usage and latency).

--

You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.