RabbitMQ on k8s vs bare VMs

[Marek Hübsch]

Hello,

I am working on a plan to replace our Azure IoT Hub with RabbitMQ cluster, which we would host ourselves in Azure.

I have done some usability proof of concept, and also a benchmark with anticipated traffic. 

First some description of what we have:

For evaluation purposes, I set up the cluster myself as 3 nodes on VMs with 4 CPUs, 16GB RAM each, and premium SSDs for mnesia DB. I considered all points in the production checklist like socket limits, tcp buffer sizes, ...

It can handle the traffic I expect with some margin. 

For the reference, our expected setup:

- 50k IoT devices, each having own queue and user credentials

- each device sends 4 msgs per minute, 2-8kB payloads.

- a couple of queues for services handling these messages (under 100), using mostly topic exchanges for message routing.

- occasional message from service to device - for this I configured a direct exchange with each IoT device queue bound to it (= this exchange has 50k bindings)

- if we add more devices, we would add more nodes, so that queues and TCP connections are distributed (each client prefers connection to the node with its queue)

Finally, the question

disclaimer: I don't know that much about kubernetes

My architect does not want us to run RabbitMQ directly on VMs, and I don't blame him as somebody would have to maintain this (probably me anyway). What he wants is to run this on kubernetes cluster, so that we can migrate elsewhere by a click of a button. I am not against this idea, but I am worried about scalability of this approach.

When going through the documentation, I stumbled upon a sentence that the CPU should not be shared with other services for heavy load scenarios, and I brought this up with the architect, so he replied that we can create a separate k8s cluster just for RabbitMQ.

At this point I started to think - does that make even sense? Are there some performance considerations when RabbitMQ runs in k8s? What I am worried about is that with such amount of clients (50k now, 100k in couple of years) we would have connectivity problems, when each connection has to go through ingress. With my VM setup that is not the case.

So I am worried that when running it in kubernetes, we lose too much control, or at least for me, I stop understanding what exactly is happening, and that there might be some overhead, causing this to not scale well.

TL;DR

We want to migrate our IoT Hub of 50-100k devices (ca. 600 million messages per day) to RabbitMQ and tests show we could. Architect is strongly against installing RabbitMQ on VMs but wants it to run on Kubernetes. Does that make sense, considering the traffic and number of concurrent connections (100k) we need?

With regards,

Marek

[Michal Kuratczyk]

Hi,

I don't think you give up much. I think it's more about what's available in your Kubernetes environment, then Kubernetes as such limiting you.

There are Kubernetes APIs to manage pretty much anything these days, including external network devices/layers should you need that.

Of course if you already have a way of simulating your expected workload, repeating these tests on Kubernetes is the ultimate way to answer your question.

As for the CPUs, indeed due to the nature of the Erlang runtime, pinning Erlang schedulers (threads) to CPUs can have a huge impact in some use cases

(otherwise you lose CPU locality). You don't necessarily need a separate cluster though - you can have nodes dedicated to RabbitMQ and

use (anti-)affinity rules to make sure nothing else works on these nodes (there are also ways to express which CPUs should be used by the Erlang VM).

You didn't mention the protocol you use. If that's MQTT, you should keep an eye on https://github.com/rabbitmq/rabbitmq-server/pull/5895.

There are major improvements coming to our MQTT support, especially when it comes to handling many connections.

Best,

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/ba1c2c23-4320-4a6b-bfb9-82a17c385c27n%40googlegroups.com.

--

Michał

RabbitMQ team