Multiple MQTT consumers result in memory leaks

[poagaf...@gmail.com]

Hello everyone,

We are using RabbitMQ  to gather telemetry information from a number of devices.
Devices use MQTT protocol to publish their state (we use rabbitmq_mqtt plugin), on the other side our back-end system reads this information via AMQP (and possibly publishes some information back to devices via MQTT).

We also need to log what is going on on the broker (ideally, every package sent from devices to the server and vice versa should be logged).
We wrote a script in Python which subscribes to # topic via MQTT, receives every message that appears on the broker, extracts its content and writes it to the log file).

We understand that an additional subscription keeps all the messages alive until they are received by all subscribers (both server and diagnostic script).
In the administration web app we also can see that deliver rate is twice higher than publish rate:

The problem is that this situation causes a constant growth of the memory used by the server and results in refused connections.
We have the following idea:
 - The number of our devices constantly grows, this means, the volume of the data transferred through the broker grows too, but our script is always as fast as it has been before
 - Rabbit does not purge messages until they are received by all subscribers - so, new messages could appear faster than they are received by both back-end and diagnostic script.

We also read the output of rabbitmqctl status: binary gets the most memory.
We tried to put the virtual machine that hosts our script in the same virtual network with Rabbit to reduce network latency. It helped a little, but the problem is still here.

What could be a better implementation of such a logging?
We suppose now we should go another way and not increase the number of subscribed clients (resulting in increased messages' lifetime), but log messages just in place (for example, write a Rabbit plugin?)

Any advice?

Thank you!

[Michael Klishin]

Start by collecting data about what actually consumes memory: `rabbitmqctl status` and the rabbitmq_top plugin

are good options.

Binaries is most often message payloads: are your consumers keeping up with the producers or do messages keep piling up?

What MQTT QoS is used?

Delivery rates is writes as higher because a single message is/can be delivered to multiple consumers.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

[Michael Klishin]

Also relevant:

 * lazy queues in 3.6.x can be applied to MQTT subscribers: their queues use a standard naming pattern that makes it easy to apply a policy http://www.rabbitmq.com/lazy-queues.html

 * client connections aren't free and supporting a lot of them might take some tweaking even if they don't publish anything at all http://rabbitmq.com/networking.html

[poagaf...@gmail.com]

Hi Michael,

We are using MQTT QoS 0 (At least once)

We used rabbitmqctl status and it reported binary was using the most memory.
Our back-end servers keep up with producers, but after our diagnostic script comes into play messages start to pile up (we tried to disable the script and memory usage got lower).

That's why we are trying to figure out whether we should not use another client subscription at all for diagnostic purposes, but choose another way (e.g., a plugin)

[Michael Klishin]

with QoS RabbitMQ will try to move messages to disk only when it's approaching VM memory watermark settings.

Lazy queues changes that, of course, and it's much easier to use and perform upgrades compared to a plugin.

What kind of diagnostics is performed? Most kinds do not need a plugin.

--

You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[poagaf...@gmail.com]

We just extract and log the messages' payload (Protobuf-serialized) to the filesystem.

On Monday, June 27, 2016 at 4:00:23 PM UTC+3, poagaf...@gmail.com wrote:

[Michael Klishin]

There is absolutely no need to develop a plugin for that.

Forcing all MQTT subscription backing queues to be lazy is likely what you want.

--

You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[V Z]

I don't understand how lazy queues would help. It seems that the issue is that when there are multiple MQTT consumers on the same topic, they can't keep up, so messages are piling up on the queues. If not in memory, they will be piling up on disk; eventually we will run out of that just like we've run out of RAM.

I've done two tests with qos=0:
1) one MQTT producer / one MQTT consumer -- consumer is keeping up.n
2) one MQTT producer / two MQTT consumers (messages are fanned out) -- consumers can't keep up; messages are piling up on the queues.

I used Paho Java client against Rabbit 3.6.2; prefetch of 1,000 in both cases, and messages were dropped upon receipt by the consumers, so no overhead.

Same arrangement using AMQP works just fine -- consumers are keeping up. Same arrangement with few other MQTT brokers works fine too.

I wonder if something's up with the MQTT plug-in, or how it interacts with the Paho client ...

[V Z]

I gathered a few charts from my testing for your review. There are 2 sets -- one for 1 producer/1 consumer, and one for 1 producer/2 consumers. Each set contains (a) queue to exchange bindings, (b) message rates while the test is running, (c) connections, (d), channels, (e) top processes.

Perhaps, you could spot something worthy of attention? I wonder if 'top' charts explain why consumers can't keep up. Could it be that the messages are not dispatched to them fast enough?

1 producer/1 consumer:

1 producers/2 consumers:

Thanks

[V Z]

To compare, here is the same set of charts for the AMQP use-case (1 producer and 2 consumers bound via topic exchange with messages being fanned out to both consumers) on exactly the same Rabbit node:

We don't see any queuing, connection is not blocked (although it is in flow control as expected), everything is very clean and well-behaved.