RabbitMQ binaries memory

[Priit Karu]

Hello, fellow RabbitMQ users!

I am very new RabbitMQ user (using version 3.5.1) and struggeling to understand the memory usage with my simple application. I am using Stomp plugin and have 100 clients connected (creating around 500 queues). There is not much activity there from the clients, the system is mainly idle. What i do not understand is the high memory usage of my Rabbit instance. Namely why so much memory is gone to binaries section and why the "Binary references" section does not show the use of that memory. Thanks for any clues that would help me to minimize Rabbit memory use in this case.

Picture from Rabbit Web Console:

Output of "rabbitmqctrl status":

Status of node 'rabbit@pk-pc' ...

[{pid,2194},

 {running_applications,

     [{rabbitmq_management,"RabbitMQ Management Console","3.5.1"},

      {rabbitmq_web_dispatch,"RabbitMQ Web Dispatcher","3.5.1"},

      {webmachine,"webmachine","1.10.3-rmq3.5.1-gite9359c7"},

      {mochiweb,"MochiMedia Web Server","2.7.0-rmq3.5.1-git680dba8"},

      {rabbitmq_stomp,"Embedded Rabbit Stomp Adapter","3.5.1"},

      {rabbitmq_management_agent,"RabbitMQ Management Agent","3.5.1"},

      {rabbit,"RabbitMQ","3.5.1"},

      {os_mon,"CPO  CXC 138 46","2.2.14"},

      {mnesia,"MNESIA  CXC 138 12","4.11"},

      {amqp_client,"RabbitMQ AMQP Client","3.5.1"},

      {sockjs,"SockJS","0.3.4-rmq3.5.1-git3132eb9"},

      {inets,"INETS  CXC 138 49","5.9.7"},

      {xmerl,"XML parser","1.3.5"},

      {sasl,"SASL  CXC 138 11","2.3.4"},

      {stdlib,"ERTS  CXC 138 10","1.19.4"},

      {kernel,"ERTS  CXC 138 10","2.16.4"}]},

 {os,{unix,linux}},

 {erlang_version,

     "Erlang R16B03 (erts-5.10.4) [source] [64-bit] [smp:4:4] [async-threads:30] [kernel-poll:true]\n"},

 {memory,

     [{total,372755696},

      {connection_readers,0},

      {connection_writers,0},

      {connection_channels,0},

      {connection_other,22698048},

      {queue_procs,23044248},

      {queue_slave_procs,0},

      {plugins,11556504},

      {other_proc,11947144},

      {mnesia,1911816},

      {mgmt_db,6136048},

      {msg_index,2087400},

      {other_ets,1717856},

      {binary,265533264},

      {code,20184143},

      {atom,703377},

      {other_system,5235848}]},

 {alarms,[]},

 {listeners,[{clustering,25672,"::"},{amqp,5672,"::"},{stomp,61613,"::"}]},

 {vm_memory_high_watermark,0.4},

 {vm_memory_limit,6681539379},

 {disk_free_limit,50000000},

 {disk_free,97045086208},

 {file_descriptors,

     [{total_limit,924},

      {total_used,105},

      {sockets_limit,829},

      {sockets_used,103}]},

 {processes,[{limit,131072},{used,2840}]},

 {run_queue,0},

 {uptime,4122}]

Thanks for any clues.

BR,

Priit.

[Michael Klishin]

Most of that memory is TCP buffers. You can configure their size to 8-16K to get significant decrease in RAM use for low activity connections.

MK

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Priit Karu]

Thanks, Michael for quick response.

I have already changed the tcp_listen_options for my rabbit instance but have not got any change to the binary size. My ubuntu had default buffer for both tcp_rmem and tcp_wmem set to 87380:

cat /proc/sys/net/ipv4/tcp_rmem
 4096 87380 6291456

Here is my rabbitmq.conf.

[{rabbit,

 [

        {tcp_listen_options, [binary,

                                  {sndbuf,        4096},

                                  {recbuf,        4096},

                                  {buffer,        4096},

                                  {packet,        raw},

                                  {reuseaddr,     true},

                                  {backlog,       128},

                                  {nodelay,       true},

                                  {linger,        {true, 0}},

                                  {exit_on_close, false}]}]}

].

Am i doing something still wrong with this configuration as it does not seem to change the "binary" memory size at all.

[Michael Klishin]

Do you observe a growing number of connections? There was a bug fix in STOMP after 3.5.1 which makes sure we don't leave connections with terminated due to socket exceptions behind.

Your understanding of TCP options seems to be correct and indeed autotuning defaults to about 87K on Linux.

MK

[Priit Karu]

There are no growing connections, both RabbitMQ admin and netstat show steady number of connections.

Is there any way for me to check if the tcp_listen_options are actually applied? I have already checked that the configuration file is correct and other properties do get applied, i have also tried to change the underlying OS defaults of rmem to "4096     4096 6291456" (wmem accordingly) but the binaries memory usage stays the same:-(  The application is almost totally at idle after some initial data exchange between the queues, maybe 1 message get posted to topic in one minute, all the queues are empty.

Would appreciate any other ideas what to test or check...

Thanks,

Priit.

[Michael Klishin]

On 7 May 2015 at 09:29:15, Priit Karu (pka...@gmail.com) wrote:
> There are no growing connections, both RabbitMQ admin and netstat
> show steady number of connections.

rabbitmqctl eval 'application:get_env(rabbitmq_stomp, tcp_listen_options).'

You've configured TCP listen options for RabbitMQ "core", AMQP 0-9-1 and 1.0 connections.

STOMP has its own option (with the same name, just app name differs), as the docs
demonstrate ;)
http://www.rabbitmq.com/stomp.html 
--
MK

Staff Software Engineer, Pivotal/RabbitMQ

[Priit Karu]

:-) 

Thank you! I knew it had to be something simple! Now the binaries part is gone and the memory diagram looks more understandable to me.

But i am still struggling a bit with my real application and RabbitMQ memory usage. The real application creates a lot of queues with different names per connection which are used quite rarely. I wonder if there is anything i can do to on the Rabbit side to reduce the queue processes memory footprint for those queues that do not have any heavy traffic (possible swapping them even away from RAM when idl) or is the only solution to change application code and reduce the number of queues used? 

Currently i see one queue takes more than 20kb even when total empty and without any activity:

BR,

Priit.

[Michael Klishin]

On 7 May 2015 at 14:42:25, Priit Karu (pka...@gmail.com) wrote:
> I wonder if there is anything i can do to on the Rabbit side to reduce
> the queue processes memory footprint for those queues that do
> not have any heavy traffic (possible swapping them even away
> from RAM when idl) or is the only solution to change application
> code and reduce the number of queues used?

You can increase statistics collection interval, that's about it. 

[饶林]

I have same issue with you,  You said it's fine now, just add

 [{rabbit,

 [

        {tcp_listen_options, [binary,

                                  {sndbuf,        4096},

                                  {recbuf,        4096},

                                  {buffer,        4096},

                                  {packet,        raw},

                                  {reuseaddr,     true},

                                  {backlog,       128},

                                  {nodelay,       true},

                                  {linger,        {true, 0}},

                                  {exit_on_close, false}]}]}

].

into rabbitmq.config? Can you explain that specifically?

[Michael Klishin]

Please use new threads for new questions.

You can reduce TCP connection buffer sizes if you have evidence of a lot of

connections in your environment, see http://www.rabbitmq.com/networking.html.

If there is no such evidence, instead of guessing and applying suggestions from threads

that seem "similar", collect memory usage breakdown with `rabbitmqctl status` and rabbitmq-top.

Management UI has a button that provides the same breakdown on the node page.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[饶林]

Thanks, Michael  for quick response. My connection is  5 -10, and consumers is about 100, so I confuse with the 
Binaries is more higher than Binaries references(not any messages in rabbitMQ). Only I kill the process ,the memory is normal now. And last night ,I increase the buffer  size,It's normal now in morning. So if there are any questions I will create a new threads for the question. I will share the details in question. Thank you so much.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[饶林]

This is new question -> (https://groups.google.com/forum/#!topic/rabbitmq-users/Wy0xQtkDY_4). Thank u.

On Wednesday, June 21, 2017 at 9:56:45 PM UTC+8, Michael Klishin wrote:

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.