Possible memory leak when using rabbitmq_auth_backend_http

[Alessio Dal Bianco]

Hi all,

I noticed that the memory grows a little, but steady, when i use the rabbitmq_auth_backend_http plugin. 

What could be the cause? I can reduce the memory consumption only when I execute this command:

rabbitmqctl eval 'supervisor2:terminate_child(rabbit_mgmt_sup_sup, rabbit_mgmt_sup),rabbit_mgmt_sup_sup:start_child().'

I have RabbitMQ 3.6.6, Erlang 19.2.

Cheers

[Karl Nilsson]

Hi,

There is a well discussed memory issue to do with the management stats database that the recently released version 3.6.8 addresses. It doesn't have anything to do with the rabbitmq_auth_backend_http plugin however. Does the memory leak still occur when you disable the rabbitmq_auth_backend_http plugin?

Cheers

Karl

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Karl Nilsson

Staff Software Engineer, Pivotal/RabbitMQ

[Alessio Dal Bianco]

At least with RabbitMQ 3.6.3, when I tried to connect thousand of clients using always the internal auth, the memory consumption was normal. 

Vice versa, using the rabbitmq_auth_backend_http plugin, RabbitMQ crashed due the fact that it hadn't memory left to consume.

I updated RabbitMQ to 3.6.6 because I was aware of the problem you described.

For keeping the memory steady, for now, i use rabbit_auth_backend_cache plugin.

Let me know if you need other infos.

Alessio

Il giorno venerdì 17 marzo 2017 16:22:30 UTC+1, Karl Nilsson ha scritto:

Hi,

There is a well discussed memory issue to do with the management stats database that the recently released version 3.6.8 addresses. It doesn't have anything to do with the rabbitmq_auth_backend_http plugin however. Does the memory leak still occur when you disable the rabbitmq_auth_backend_http plugin?

Cheers

Karl

On 17 March 2017 at 15:17, Alessio Dal Bianco <in...@alessiodalbianco.me> wrote:

Hi all,

I noticed that the memory grows a little, but steady, when i use the rabbitmq_auth_backend_http plugin. 

What could be the cause? I can reduce the memory consumption only when I execute this command:

rabbitmqctl eval 'supervisor2:terminate_child(rabbit_mgmt_sup_sup, rabbit_mgmt_sup),rabbit_mgmt_sup_sup:start_child().'

I have RabbitMQ 3.6.6, Erlang 19.2.

Cheers

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Michael Klishin]

With a large number of connections or high connection churn it is a very good idea to use rabbitmq_auth_backend_cache with authn/authz backends that use external services, such as the HTTP one or LDAP.

The memory growth isn't necessarily a leak: more likely it is a backlog of client operations that their downstream service wasn't able to keep up with. In other words the classic consumer/producer problem where the producer outpaces the consumer.

[Alessio Dal Bianco]

Hi Micheal,

you are right, but right now i do not have a lot of concurrent request, and for minutes sometimes there is no request at all (thanks to rabbitmq_auth_backend_cache), and despite this the memory keep growing up (althoug more slowly than without cache plugin).

[Michael Klishin]

Alessio,

I'm not sure how members of this list can help you. You haven't provided any information to work with:

not even what RabbitMQ version is used. Guessing is an incredibly inefficient approach to debugging.

Please collect some data that narrows the problem first:

 * Post full logs

 * Capture memory usage breakdown (via rabbitmqctl status or HTTP API)

 * Use rabbitmq-top to see what processes consume most resources

Without it it is absolutely impossible to suggest anything. Please help us help you.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

[Alessio Dal Bianco]

Michael, 

sorry for bother you and thanks for the tip (i wasn't aware of rabbit_top). I'm going to investigate more deeply.

Alessio

PS: RabbitMQ and Erlang version are in my 1st message, also in my 2nd message i asked if further informations needed. 

I know is frustrating for both of us try to replicate/resolve bugs, especially of this kind. But, at least for me, is also difficult understand even what kind of informations is useful for you guys.

[Alessio Dal Bianco]

Hi,

here are the logs, i think the table aggr_connection_stats_coarse_conn_stats is the one it seems to grow constantly.

- Inside of rabbitmq_status_log.txt you find 3 snapshot of rabbitmqctl status.

Please let me know if need further informations.

Thank you for your time.

Alessio

[Michael Klishin]

OK, that's something a lot more specific, thanks.

3.6.7 and 3.6.8 have a completely different implementation of stats storage. Investigating issues with

the previous iteration is not a priority for our team. Please upgrade and if the issue persists, send us full node

logs and your configuration file off-list.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Alessio Dal Bianco]

Ok I understand.

Meanwhile can I, in some ways, temporary disable the collecting of those stats?

[Karl Nilsson]

The most effective way to deal with this is to set up a scheduled job to periodically restart the mgmt db. See: https://www.rabbitmq.com/management.html#stats-db

Cheers

Karl

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.