high memory usage for bean.smp

[Ben P]

I am seeing high memory consumption (6GB+) by the bean.smp process when running rabbitmq server (3.6.10) on ubuntu 16.04.

This is the case even right after I do a fresh start of the the rabbitmq-server service with no clients or messages being communicated through the bus.

I followed some suggestions online to address this issue  (eg: clean up rogue clients sending messages) but to no avail.

Any suggestions?

Thanks,

Ben

[Michael Klishin]

Please see list archives, there is no shortage of threads that discuss memory consumption analysis.

`rabbitmqctl status`, the breakdown button on node page in management UI and rabbitmq-top

are the tools you should consider first.

See http://www.rabbitmq.com/memory-use.html and https://groups.google.com/forum/#!topic/rabbitmq-users/TVZt45O3WzU

as well.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

[Ben P]

Michael,

Thanks for the reply.

I did go through the archives and my situation seems to differ from what I found in that the memory consumption for the beam.smp process hits 6.5 GB out of the box, as soon as I start the rabbitmq daemon with no messages flowing through the bus. The issues I found in the archives mainly relate to cases where the memory usage grows over time as traffic flows the system.

Also, the memory consumption as indicated by rabbitmqctl status indicates total memory consumption of about 65MB, which is way off from what top is showing:

top output:

 9154 rabbitmq  20   0  6.216g 0.096g 0.005g S   1.3  0.3   0:46.27 beam.smp                               

rabbitmqctl status output:

{memory,

     [{total,66185720},

      {connection_readers,29752},

      {connection_writers,1512},

      {connection_channels,5504},

      {connection_other,63160},

      {queue_procs,22200},

      {queue_slave_procs,0},

      {plugins,3191280},

      {other_proc,22666432},

      {mnesia,62448},

      {metrics,149152},

      {mgmt_db,394832},

      {msg_index,41080},

      {other_ets,2466712},

      {binary,2804560},

      {code,26995972},

      {atom,992409},

      {other_system,6445163}]},

On Thursday, August 24, 2017 at 12:48:11 AM UTC-7, Michael Klishin wrote:

Please see list archives, there is no shortage of threads that discuss memory consumption analysis.

`rabbitmqctl status`, the breakdown button on node page in management UI and rabbitmq-top

are the tools you should consider first.

See http://www.rabbitmq.com/memory-use.html and https://groups.google.com/forum/#!topic/rabbitmq-users/TVZt45O3WzU

as well.

On Wed, Aug 23, 2017 at 8:15 PM, Ben P <bnp...@gmail.com> wrote:

I am seeing high memory consumption (6GB+) by the bean.smp process when running rabbitmq server (3.6.10) on ubuntu 16.04.

This is the case even right after I do a fresh start of the the rabbitmq-server service with no clients or messages being communicated through the bus.

I followed some suggestions online to address this issue  (eg: clean up rogue clients sending messages) but to no avail.

Any suggestions?

Thanks,

Ben

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Michael Klishin]

Please post full output of rabbitmqctl status and rabbitmqctl environment

then enable rabbitmq-top (a plugin) and see what processes take most RAM.

[Michael Klishin]

Also, what architecture is used here? What Erlang/OTP version? How was it installed?

Have you tried 19.3.6 or 19.3.6.2 (see http://www.rabbitmq.com/install-debian.html for

the list of repositories that provide recent versions)

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Luke Bakken]

Hi Ben,

That column in the top output is for virtual memory size, which is not the same as what's actually being used physically. On my workstation, a fresh start of RabbitMQ (in development mode) shows the following in top:

12263 lbakken   20   0 5755.4m  72.1m   1.3  0.2   0:05.32 S    beam.smp

There are many articles on the internet that explain the difference in these values. I found this one to be well-written:

https://blogs.kde.org/2005/09/15/measuring-memory-usage

Thanks,

Luke

[Bhanu N Pisupati]

(combining last two replies from Michael and Luke):

> Please post full output of rabbitmqctl status and rabbitmqctl environment (I have attached rabbitmq environment output as a separate file due to its size)

[{pid,27985},

 {running_applications,

     [{rabbit,"RabbitMQ","3.6.11"},

      {mnesia,"MNESIA  CXC 138 12","4.15"},

      {ranch,"Socket acceptor pool for TCP protocols.","1.3.0"},

      {ssl,"Erlang/OTP SSL application","8.2"},

      {public_key,"Public key infrastructure","1.4.1"},

      {asn1,"The Erlang ASN1 compiler version 5.0","5.0"},

      {rabbit_common,

          "Modules shared by rabbitmq-server and rabbitmq-erlang-client",

          "3.6.11"},

      {xmerl,"XML parser","1.3.15"},

      {compiler,"ERTS  CXC 138 10","7.1"},

      {crypto,"CRYPTO","4.0"},

      {syntax_tools,"Syntax tools","2.1.2"},

      {os_mon,"CPO  CXC 138 46","2.4.2"},

      {sasl,"SASL  CXC 138 11","3.0.4"},

      {stdlib,"ERTS  CXC 138 10","3.4"},

      {kernel,"ERTS  CXC 138 10","5.3"}]},

 {os,{unix,linux}},

 {erlang_version,

     "Erlang/OTP 20 [erts-9.0] [source] [64-bit] [smp:12:12] [ds:12:12:10] [async-threads:192] [kernel-poll:true]\n"},

 {memory,

     [{connection_readers,0},

      {connection_writers,0},

      {connection_channels,0},

      {connection_other,0},

      {queue_procs,2744},

      {queue_slave_procs,0},

      {plugins,0},

      {other_proc,22858432},

      {metrics,184888},

      {mgmt_db,0},

      {mnesia,61776},

      {other_ets,1586952},

      {binary,72808},

      {msg_index,42504},

      {code,21355780},

      {atom,891849},

      {other_system,24683707},

      {total,71741440}]},

 {alarms,[]},

 {listeners,[{clustering,25672,"::"},{amqp,5672,"::"}]},

 {vm_memory_calculation_strategy,rss},

 {vm_memory_high_watermark,0.4},

 {vm_memory_limit,13454517862},

 {disk_free_limit,50000000},

 {disk_free,167341670400},

 {file_descriptors,

     [{total_limit,924},{total_used,2},{sockets_limit,829},{sockets_used,0}]},

 {processes,[{limit,1048576},{used,164}]},

 {run_queue,0},

 {uptime,109},

 {kernel,{net_ticktime,60}}]

> Also, what architecture is used here?

x86

> What Erlang/OTP version? 

 1:20.0-1

> How was it installed?

On my ubuntu 16.04 machine, followed instructions from here: https://www.erlang-solutions.com/resources/download.html

Added new repo souces 

deb https://packages.erlang-solutions.com/ubuntu trusty contrib

sudo apt-get update
sudo apt-get install erlang

> Have you tried 19.3.6 or 19.3.6.2

As I wrote above, I upgraded from ver 16 (which had the 6GB memory consumption I mentioned above) to 20:0-1.The virtual memory then went up to 7.1 GB upon starting rabbitmq-server service.
Is there a reason to try the older 19.3.6 instead?

Luke: Thanks for the reply.

As for your comment:

> That column in the top output is for virtual memory size, which is not the same as what's actually being used physically.

Sure, I understand that not all of the memory is going to be in physical memory, but part of it swapped out to disk.

Reading through the link you sent about output from top, specifically:

"VIRT is virtual memory usage, it can probably be best described as the app's used address space - every library the app uses, every data it creates, everything is included here. If the app requests 100M memory from the kernel but actually uses only 1M, VIRT will still increase by 100M."

This suggests that the fact remains that all 7GB will have to be instantiated in virtual memory by erlang and it's collateral (required libraries). Sure all of the 7GB does not have to be in RAM. But this is a larger footprint that what I expected.
Specifically, we have plans to shortly run rabbitMQ on a platform such as Nvidia Jetson TX2 (running ubuntu 16.04), which has 8GB RAM and no swap space support. In such a scenario, a process having virtual memory footprint of 7GB would be a no go.

On a related note, what is the status of RabbitMQ on ARM (Cortex) architectures?

Thanks
Ben

You received this message because you are subscribed to a topic in the Google Groups "rabbitmq-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/rabbitmq-users/KsRH00p-m64/unsubscribe.
To unsubscribe from this group and all its topics, send an email to rabbitmq-users+unsubscribe@googlegroups.com.

[gl...@pivotal.io]

I consider VIRT column in ps/top to be irrelevant for measuring memory usage since it's not real - it's just address space. A processor with 64-bit memory addresses can directly access 264 bytes (=16 exabytes) of byte-addressable memory. Try this experiment: https://www.darkcoding.net/software/resident-and-virtual-memory-on-linux-a-short-example/

"The virtual memory size isn't as important as your resident set. Virtual memory is going to include things that are mapped in such as shared libraries and whatnot (which will only be resident in physical RAM once). RSS tells you what's actually in memory, where as Virt. tells you how much of the available virtual address space has been swallowed." https://serverfault.com/questions/122810/apache-heavy-load-virt-vs-res-memory

VIRT used to be important on 32-bit operating systems, but it's irrelevant on 64-bit machiens. A JVM perspective: https://stackoverflow.com/a/561450

If you want to learn more about Erlang VM memory allocation, take 35 mins to watch this: https://www.youtube.com/watch?v=YuPaX11vZyI

The best thing to do would be to run RabbitMQ on Nvidia Jetson TX2, put the desired load through the system and benchmark. I would love to hear how it goes, never ran RabbitMQ on Nvidia Jetson TX2.

You received this message because you are subscribed to a topic in the Google Groups "rabbitmq-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/rabbitmq-users/KsRH00p-m64/unsubscribe.

To unsubscribe from this group and all its topics, send an email to rabbitmq-user...@googlegroups.com.