RabbitMQ 3.7.0 Memory Usage

[Ilya Smirnov]

Hello, 

during the RabbitMQ 3.7.0 tests with our load(6k connections, 12k channels, 3k exch, 12r queues, 70k erlang proc.), we are faced with the problem of memory allocation. After 1-2 hours, the memory consumption increases several times and exceeds the available size, after which the node falls.

Memory configuration:

{ vm_memory_high_watermark, 0.48 },

{ vm_memory_high_watermark_paging_ratio, 1.1 },

{ vm_memory_calculation_strategy, rss },

[info] <0.332.0> Memory high watermark set to 5686 MiB (5962906337 bytes) of 11847 MiB (12422721536 bytes) total

(We use lazy queues, so we decided to turn off paging, is it correct?)

I found in manual - "Erlang's garbage collector can, in the worst case, cause double the amount of memory to be used"

State from crashdump:

On these screens, you can see that the node has fallen during an attempt to allocate memory in excess of what is available and > then 0.48*2.

Is this behavior erroneous? Or is it a configuration error? 

Thank you in advance.

[Michael Klishin]

Setting `vm_memory_high_watermark_paging_ratio` to values > 1 is not something we ever recommended.

RabbitMQ does not allocate memory directly, the runtime does. There are multiple allocator strategies and paramters

that can be tried by adjusting Erlang VM stats.

rabbitmqctl and management UI provide a way to retrieve a memory breakdown. rabbitmq-top can be enabled

to display top processes by CPU and memory usage on top of that.

All of them have been discussed many times on this list in the past.

TCP connections consume RAM and by default, the OS autotunes TCP buffers to be over 100K in size:

http://www.rabbitmq.com/networking.html

Without that information being monitored over time, there is nothing our team can suggest.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

[Michael Klishin]

Also, the `rabbit.vm_memory_high_watermark` does not limit how much memory the runtime can possibly allocate.

It's a value that, when exceeded, will trigger a resource alarm [1].

Erlang VM allocators can allocate larger or smaller chunks at once. There is absolutely no relation to

the high VM watermark RabbitMQ uses.

1. http://www.rabbitmq.com/alarms.html

On Mon, Oct 23, 2017 at 6:03 PM, Ilya Smirnov <ilya...@gmail.com> wrote:

--

You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Ilya Smirnov]

Thanks for help!

We have changed some parameters according to the following topics, and now rabbit doesn't fall, when memory consumption on the boundary values.

Also, the `rabbit.vm_memory_high_watermark` does not limit how much memory the runtime can possibly allocate.

It's a value that, when exceeded, will trigger a resource alarm [1].

Erlang VM allocators can allocate larger or smaller chunks at once. There is absolutely no relation to

the high VM watermark RabbitMQ uses.

1. http://www.rabbitmq.com/alarms.html

On Mon, Oct 23, 2017 at 6:03 PM, Ilya Smirnov <ilya...@gmail.com> wrote:

Hello, 

during the RabbitMQ 3.7.0 tests with our load(6k connections, 12k channels, 3k exch, 12r queues, 70k erlang proc.), we are faced with the problem of memory allocation. After 1-2 hours, the memory consumption increases several times and exceeds the available size, after which the node falls.

Memory configuration:

{ vm_memory_high_watermark, 0.48 },

{ vm_memory_high_watermark_paging_ratio, 1.1 },

{ vm_memory_calculation_strategy, rss },

[info] <0.332.0> Memory high watermark set to 5686 MiB (5962906337 bytes) of 11847 MiB (12422721536 bytes) total

(We use lazy queues, so we decided to turn off paging, is it correct?)

I found in manual - "Erlang's garbage collector can, in the worst case, cause double the amount of memory to be used"

State from crashdump:

On these screens, you can see that the node has fallen during an attempt to allocate memory in excess of what is available and > then 0.48*2.

Is this behavior erroneous? Or is it a configuration error? 

Thank you in advance.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[V Z]

Ilya, which parameters food you change, if you don't mind sharing, please?

[Sebastian Röbke]

I am also interested in the changes you applied, since we are facing the same issue.

Thanks,
Sebastian

[Ilya Smirnov]

Hi,

In our case, we have many connections and low size of msgs (6000-10000 and 1-3kb), we reduce tcp buffers ->

{ sndbuf,        4096 },

{ recbuf,        4096 }

 then turn off writing msg to queue index ->

{ queue_index_embed_msgs_below, 0 },

most stable watermark options for us ->

{ vm_memory_high_watermark, 0.5 },

{ vm_memory_high_watermark_paging_ratio, 0.99 },

{ vm_memory_calculation_strategy, rss },

and we use lazy queues. 

Then we change additional_erl_args ->

'+A 96(core * 12) +MBas aobf +MBlmbcs 512(reduce bit difference between usage and allocate binary memory) +S 8:8 +sct L0-7c0-7 +stbt ts(good if you have 1 NUMA node) +swt low +sbwt long -smp enable'

all about this args - http://erlang.org/doc/man/erl.html

we try to apply this setting - +MBas aobf +MBlmbcs 512 to all allocators(+Muas aobf +Mulmbcs 512), but get enlargement latency with high rate.

And if you use cluster mode with ha polices, it is better to have a memory reserve for queues - in our case, when one of members down, alloc for queues raise to 2,5 - 3,5 gb from 180mb.

With this configuration, our test stand works ~10hrs with boundary values of memory usage, and doesn't fall. I hope this helps.

[Sebastian Röbke]

Thanks for sharing!