Rabbitmq binaries memory usage too high

[po...@platform9.com]

Hi,

I am new to rabbitmq and was troubleshooting an issue with very high rabbitmq memory usage on a server. Attached is the output of rabbitmqctl status command.

I fail to understand why the memory usage of binaries section is about 25G with very few messages in the queue. (see below for current state)

Can someone provide some pointers on how to debug this?

Thanks,

Pooja

# rabbitmqctl list_queues | grep -vw 0

Listing queues ...

notifications.sample 146

reply_5c1b83f5468144b1a8b8e130e4c928f3 1

reply_7911e286809f4d26b530b95e28240cc3 1

...done.

[Michael Klishin]

http://www.rabbitmq.com/memory-use.html contains a number of pointers.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

[Michael Klishin]

Yeah, I hate to say it but 3.3.5 is long out of support, does not provide nearly the number of metrics of 3.6.15 or later,

and on top of that you are running on an Erlang version that has a pretty different GC implementation

from 19.3 (technically 19.x) or later.

There could be something obvious such as but we highly recommend upgrading to at least 3.6.15 and Erlang 19.3.6.5. Then the contents

of the guide above will be that much more helpful.

The only thing I can think of is: you have about 10K connections, are you aware of the fact that connections aren't "free" in terms of memory

and you can drastically decrease the amount of RAM each connection uses with some TCP buffer configuration (which has its downsides)?

Take a look at the Tuning for a Large Number of Connections section in https://www.rabbitmq.com/networking.html.

On Fri, Mar 2, 2018 at 6:43 AM, <po...@platform9.com> wrote:

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[po...@platform9.com]

Thanks for looking into it, Michael!

We have been using latest rabbitmq version from the Centos 7.4 repo so far, will try out latest version available from github.

I had previously looked into the memory usage and networking config guides for rabbitmq performance that you linked here. 

In the current state, my rabbitmq config has no tcp_listen_options set for sndbuf/recbuf/keepalive and backlog defaulted to 128.

Will increasing the backlog to 4k, sndbuf/recbuf to 16k and enabling keepalive on existing setup help? Also would that need rabbitmq-server to be restarted?

My understanding of the no. of connections was that the memory usage would not belong to 'binaries' section, hence did not change those defaults.

How can I verify what is the value of tcp sndbuf/recbuf that rabbitmq is currently using?

JFYI, I also tried manually running garbage collection using below command, but didn't see any change in memory usage reported by top or 'erlang:memory()':

# rabbitmqctl eval '[garbage_collect(P) || P <- processes()].'

#  rabbitmqctl eval 'application:get_env(rabbit, tcp_listen_options).'

{ok,[binary,

     {packet,raw},

     {reuseaddr,true},

     {backlog,128},

     {nodelay,true},

     {linger,{true,0}},

     {exit_on_close,false}]}

...done.

Thanks,

Pooja

On Friday, March 2, 2018 at 9:03:39 AM UTC-8, Michael Klishin wrote:

Yeah, I hate to say it but 3.3.5 is long out of support, does not provide nearly the number of metrics of 3.6.15 or later,

and on top of that you are running on an Erlang version that has a pretty different GC implementation

from 19.3 (technically 19.x) or later.

There could be something obvious such as but we highly recommend upgrading to at least 3.6.15 and Erlang 19.3.6.5. Then the contents

of the guide above will be that much more helpful.

The only thing I can think of is: you have about 10K connections, are you aware of the fact that connections aren't "free" in terms of memory

and you can drastically decrease the amount of RAM each connection uses with some TCP buffer configuration (which has its downsides)?

Take a look at the Tuning for a Large Number of Connections section in https://www.rabbitmq.com/networking.html.

On Fri, Mar 2, 2018 at 6:43 AM, <po...@platform9.com> wrote:

Hi,

I am new to rabbitmq and was troubleshooting an issue with very high rabbitmq memory usage on a server. Attached is the output of rabbitmqctl status command.

I fail to understand why the memory usage of binaries section is about 25G with very few messages in the queue. (see below for current state)

Can someone provide some pointers on how to debug this?

Thanks,

Pooja

# rabbitmqctl list_queues | grep -vw 0

Listing queues ...

notifications.sample 146

reply_5c1b83f5468144b1a8b8e130e4c928f3 1

reply_7911e286809f4d26b530b95e28240cc3 1

...done.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Michael Klishin]

TCP buffers are preallocated and while I’d expect those to count in the “connection” (readers) section, it was something I pointed out nonetheless. So yes, you are likely right in your thinking.

Unfortunately without rabbitmq-top anything I have in mind is a guess that’s not easy to verify.

MK

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

<rabbitmq-status.rtf>