RabbitMQ MQTT - Kafka bridge

[Antoine Galataud]

Hi,

I'm working on a project for which potentially millions of devices will stay connected 24/7 to our RabbitMQ clusters.

We haven't benchmarked performances and stability under such load yet, but we need to anticipate and we're thinking of several options to make our architecture scale without troubles.

Among others, one option would be to dedicate RabbitMQ clusters to MQTT clients only and hand over millions of messages generated to an internal Kafka cluster.

That way of doing it would have several benefits:

- other RabbitMQ clusters (internal, mainly AMQP) could have a different scaling approach, and wouldn't suffer from spikes of load on MQTT cluster.

- we could absorb MQTT messages in streaming and/or batching, as long as they are handled by Kafka after going through RabbitMQ

- we'd still have RabbitMQ in front and its great support of MQTT, security, ...

First on the principle: does it make sense or is it over-engineered compared to another solution I'm not aware of?

Secondly, I'm thinking about writing something like a shovel-like plugin for rabbitmq-mqtt that would take messages and publish them on Kafka with the same topic. The important thing here would be to deliver messages as fast as possible to Kafka. Does it sound a reasonable way of handling the problem?

Any suggestions would be welcome.

Thanks in advance

Antoine

[Alvaro Videla]

What volume of messages per second are you thinking about?

For RabbitMQ 3.6.0 (which _should_ come out next month) we are trying to implement a feature called "lazy queues" ie: queues that only load message contents on memory when needed by consumers. These queues have the particularity that for 1MM messages they require around 80mb of RAM instead of 800+ MB required by "normal" queues. Here I'm talking about the broker's total RAM cost, not individual queues. From our tests, the performance of these queues is about the same as for normal queues when publishing persistent messages. Another advantage of these queues is that via policies it will be possible to switch them back and fort from lazy to normal when needed.

Also for 3.6.0 RabbitMQ Queues will support batching of messages  (at least internally), feature that could be used from an exchange plugin or something similar in order to process messages in batches.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Antoine Galataud]

~100k msg/s as a normal traffic expected, with possible spikes between 300k and 500k.

I think the main issues we'll have to deal with:

- unavailability of consumers or offline consumers (batches)

- consumers slowing down for a period of time.

Don't know if lazy queues can help for the second.

Consumers are mainly streaming applications: Spark Streaming, but also "regular" consumers (Java & Scala apps using spring-rabbit).

There could also be some batches involved for some specific topics. We don't have any yet, but we'd like to keep this possibility.

[Alvaro Videla]

On 19 October 2015 at 13:22:26, Antoine Galataud (antoine....@gmail.com) wrote:

- consumers slowing down for a period of time.

Don't know if lazy queues can help for the second.

One of the goals of lazy queues is to have a predictable performance when this happens, since they require almost no paging to disk (since messages are sent to disk right as they enter the queue).

[Antoine Galataud]

That sounds very interesting so I include this in the list of our options.

I'll give it a try when 3.6.0 is out.

[Michael Klishin]

On 19 Oct 2015 at 14:47:39, Antoine Galataud (antoine....@gmail.com) wrote:
> That sounds very interesting so I include this in the list of
> our options.
> I'll give it a try when 3.6.0 is out.

There will be several preview releases of 3.6.0 (the first one coming this week, although without lazy queues),
so you wouldn’t have to wait till late November. We’ll keep you posted.
--
MK

Staff Software Engineer, Pivotal/RabbitMQ

[Pau Freixes]

Sorry for putting in the middle of this conversation, about lazy
queues.Can you explain it a bit more? goal, strategy, etc

Antonie, about your thoughts IMHO is better put in front of your
devices a MQTT layer and apply there the AuthN, AuthZ using this
protocol as a Connection and Transport Protocol. In my experience
implements the AuthZ mechanism over the AMQP topology to be used by
your devices is almost a hard job : exchanges, topics, queues and
dynamic changes.

AMQP infrastructure can be build behind this first ring getting
messages and processing them from the MQTT.

> --
> You received this message because you are subscribed to the Google Groups
> "rabbitmq-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to rabbitmq-user...@googlegroups.com.
> To post to this group, send email to rabbitm...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

--
--pau

[Michael Klishin]

On 19 Oct 2015 at 14:53:07, Pau Freixes (pfre...@gmail.com) wrote:
> IMHO is better put in front of your
> devices a MQTT layer and apply there the AuthN, AuthZ using this
> protocol as a Connection and Transport Protocol.

Keep in mind that MQTT lacks error notifications to clients, making it a usability
disaster when it comes to clients disconnecting for non-obvious reasons (such as duplicate client ids).
In other words, the only way to indicate an error for a server is to close TCP connection, to which clients
react by immediately reconnecting. Good luck debugging a large scale  system like that.

So while using MQTT as “edge protocol” is possible, I’d personally recommend STOMP or AMQP 0-9-1.

[Pau Freixes]

> Keep in mind that MQTT lacks error notifications to clients, making it a usability
> disaster when it comes to clients disconnecting for non-obvious reasons (such as duplicate client ids).
> In other words, the only way to indicate an error for a server is to close TCP connection, to which clients
> react by immediately reconnecting. Good luck debugging a large scale system like that.

To get a more granularity action over each MQTT connection, such as
close TCP connections because of duplicate ids, you
can use something like mosca [1] easy to extend and with a good performance.

Obviously use a two-steps scenario to feed your system would have its
drawbacks, but it can help you to scale each part of your
architecture using different characteristics.

[1] https://github.com/mcollina/mosca

> So while using MQTT as “edge protocol” is possible, I’d personally recommend STOMP or AMQP 0-9-1.

Yes this is why I'm talking about, use it as "edge protocol" and bring
the data to each exchange and topic without take care about the AuthZ
layer scaling up your
AMQP topology thinking only in messages - store, throughput, etc .

Cheers

> --
> MK
>
> Staff Software Engineer, Pivotal/RabbitMQ
>
>

> --
> You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.

> To post to this group, send an email to rabbitm...@googlegroups.com.

[Antoine Galataud]

Hi,

Thank you Michael; I've started to test 3.6.0 when I wrote the topic authorization plugin last week. 

I'd definitely like to test a release candidate when lazy queues are implemented.

@Pau: our intention is to have a RabbitMQ cluster dedicated to external MQTT communications only, with security (including topic auth ;)) applied there. 

Internally we would keep an AMQP cluster for inter-services communication, with very different auth mechanisms.

You received this message because you are subscribed to a topic in the Google Groups "rabbitmq-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/rabbitmq-users/-cvFUCuQeys/unsubscribe.
To unsubscribe from this group and all its topics, send an email to rabbitmq-user...@googlegroups.com.

[Pau Freixes]

On Mon, Oct 19, 2015 at 2:47 PM, Antoine Galataud
<antoine....@gmail.com> wrote:
> Hi,
>
> Thank you Michael; I've started to test 3.6.0 when I wrote the topic
> authorization plugin last week.
> I'd definitely like to test a release candidate when lazy queues are
> implemented.
>
> @Pau: our intention is to have a RabbitMQ cluster dedicated to external MQTT
> communications only, with security (including topic auth ;)) applied there.
> Internally we would keep an AMQP cluster for inter-services communication,
> with very different auth mechanisms.

Which are the points to decide use a RabbitMQ cluster dedicated as
external MQTT communications broker ?

Thanks in advance :)



--
--pau

[Antoine Galataud]

Well, in disorder:
- proven robustness
- lightweight
- eco system
- easy to manage/monitor
- simplicity to find people who knows the product
- simplicity to customize for your needs (plugins)

Antoine

[Alvaro Videla]

Here's a description of Lazy Queues: https://github.com/rabbitmq/rabbitmq-server/issues/351

You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.

[Antoine Galataud]

That are promising results.

I have 2 questions:

- what happens when mode is switched on a highly loaded queue? Does it load everything back in memory or flush everything to disk when executed?

- could it be used in conjunction with sharding plugin? The benefit of having sharded queues with non content kept in memory could be important for performances.

Antoine

To post to this group, send email to rabbitm...@googlegroups.com.

[Alvaro Videla]

If you set mode to "lazy" then the queue will page _almost_ everything to disk, except for the oldest 16k messages. For these messages, it will keep in RAM only their position in the queue, but contents will be paged to disk as well (unless the message is stored in the queue index, see: queue_index_embed_msgs_below).

So making a queue lazy would have similar effects as when the broker decides it needs to page most messages to disk.

When you set the mode to "default" the queue will load messages into RAM as they required by consumers (this is how it works right now so no changes here). It basically works like a queue that had messages paged to disk, the broker was shut down, and then the broker came back. When this happens, a queue will load a batch of 16k msgs in memory.

You can mix lazy queues with sharding by mixing up policies.

In the future we might even further reduce memory usage by keeping the message store index in a different database like elelveldb.

To post to this group, send email to rabbitm...@googlegroups.com.