The Size of an MQTT Broker

[Tuğrul Kök]

Hello everyone,

I am working on MQTT Broker migration. I am trying to estimate the size of MQTT Broker. Especially when the broker is a docker container. I want to use the most popular brokers, eclipse-mosquitto, HiveMQ...

It is obvious that the size is related to such parameters like message size,  QoS, number of topics, number of publishers... However, I couldn't find a relation like a formula, or an estimation method for given parameters (number of publisher, message size etc.) and the size of MQTT Broker. The best thing for me is finding a relation something like

Size of the broker = base container size +  (number of clients)*(message size) + ...

Is there any way to do it without simulating and observing results? I couldn't find such information on the websites/documentations of the brokers. 

Thanks in advance,

Best regards,

[Bruno]

Hi,

I'm sorry but the size of what? 

Size of RAM / memory needed to run? 

Size of the container snapshot? 

What are your trying to do that needs to evaluate this "size"? 

--
To learn more about MQTT see https://mqtt.org
---
You received this message because you are subscribed to the Google Groups "MQTT" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mqtt+uns...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/mqtt/539b3b9e-b167-45ff-bd72-74d3e0c22ce2n%40googlegroups.com.

[Greg Troxel]

Tuğrul Kök <tugr...@gmail.com> writes:

> I am working on MQTT Broker migration. I am trying to estimate the size of
> MQTT Broker. Especially when the broker is a docker container. I want to
> use the most popular brokers, eclipse-mosquitto, HiveMQ...

I'd ask anyway, but especially becaues you are writing from gmail:
Please describe your situation/purpose-for-asking in terms of:
- homework
- part of a company, job, business plan (more or less, anything for
which you are being paid or stand to gain financially)
- part of a public open-source project
- hobby project
- something else

Then, it would be good to describe your actual problem, not something
that is a detail about how you think you are solving your actual
problem.

I am not the least bit sure I will have any wisdom for you, but I think
people asking for free consulting should be clear about the situation,
and your request comes across as unusual.

(As alawys, follow up on list, not to me directly. It would be rude of
me to quote private mail on the list, and I don't want to provide
offlist help as that doesn't help the next person.)

[Tuğrul Kök]

Thank you for your responses,

I am a master of science student and doing a thesis related to
migration of an MQTT Broker. I am actually interested in all of them.
The size of the database is the most important for me. I want to know
how much data I need to migrate. Size of RAM / Memory is also
important for me, because I want to make sure that the new device,
which MQTT Broker will be migrated to, has enough resources. So I am
also interested in container size. But as far as I understand, Docker
allocates a fixed size of container, the memory changes occur inside
this allocated memory.

It is my first time to use mail groups, apologies for if I used it
inappropriately. I couldn't find any documentation / paper related to
my case. I chose to simulate some cases with different parameters and
try to find a pattern for estimating. I wonder if there is a better
way to do it.

Thank you,
Best regards,
Tuğrul KÖK


Greg Troxel <g...@lexort.com>, 17 Ağu 2024 Cmt, 02:04 tarihinde şunu yazdı:

[Ranjith Kumar Diraviyam]

Hello, 

There are a lot of parameters involved and I believe there is a possibility to build some formula for this. But some parameters will happen at runtime and cannot be predicted. 

1. No of clients with subscription that sets clean session false and no of publishing happening to these topics and their message sizes 
2. No of subscription clients that connects intermittently to get the data ( hourly once or so ) 
3. Retained message size * no of topics
4.  Base Broker size 
5. Whether the broker stores data in memory or on a backend db and how much time the data is kept in memory before it is stored to db. 

Even though 3 & 4 can be predefined, 1,2 & 5 cannot be predicted.  If you could provide more info on your use case, there is some possibility of assessing the foot print needed.  

Ranjith

Bevywise 

www.bevywise.com

--
To learn more about MQTT see https://mqtt.org
---
You received this message because you are subscribed to the Google Groups "MQTT" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mqtt+uns...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/mqtt/539b3b9e-b167-45ff-bd72-74d3e0c22ce2n%40googlegroups.com.

--

Best  Regards,
Ranjith
Mob:- +91 98407 87678
_____________
Whatever the mind of the man conceives and believes, it will achieve.....

[Greg Troxel]

Tuğrul Kök <tugr...@gmail.com> writes:

Thanks for explaining. In general, I think this sort of explanation
should come with requests for help, at least those that aren't
indirectly bug reports about documentation.

> I am a master of science student and doing a thesis related to
> migration of an MQTT Broker. I am actually interested in all of them.
> The size of the database is the most important for me. I want to know
> how much data I need to migrate. Size of RAM / Memory is also
> important for me, because I want to make sure that the new device,
> which MQTT Broker will be migrated to, has enough resources. So I am
> also interested in container size. But as far as I understand, Docker
> allocates a fixed size of container, the memory changes occur inside
> this allocated memory.

You'll have to first define what migrate means. It sounds like
transferring the state of a running broker to another computer, and that
raises issues about keeping TCP connections open or not.

Sometimes, brokers are configured to store topic/payload tuples on disk,
so that they can be reloaded if the machine reboots. This is likely
related.

You may need to implement a save/restore mechanism, and it may or may
not be the same as the persistent storage mechanism. If you want to
migrate across implementations that will have to be an
implementation-neutral format.

For sizing, I think you'll need to experiment with and perhaps also
analyze resource usage vs number of clients, number of topics, payload
size etc. You probably want a test client with controllable/repeatable
synthetic load.

As for docker, it's not clear to me why you are using containers, for
what is just a process, and whether you intend to have storage in some
other container. My not-really-thinking advice would be to simplify
everything you can as much as possible so you can concentrate on your
real problem.

I also have a reaction that for my use of mqtt, the broker process size
(mosquitto) is 31 MB, and the RSS is 5 MB, for a few dozen retained
topics with short payloads. You'll need to pump serious amounts of data
to determine the relationships.

> It is my first time to use mail groups, apologies for if I used it
> inappropriately. I couldn't find any documentation / paper related to
> my case. I chose to simulate some cases with different parameters and
> try to find a pattern for estimating. I wonder if there is a better
> way to do it.

It's not really so much about mail - that part was fine. It's more
that the people who get requests for help get them from all sorts of
places and contexts.

[Andy Stanford-Clark]

This is an interesting question - probably worth an MSc dissertation on it own!

As people have been saying, “it depends”, but the thing I worry most about with a Production broker, is the number of unclaimed messages:
messages at QoS 1 or QoS 2 which are queued in the broker waiting for the subscriber to reconnect to collect them.
Some brokers have a message expiry feature to protect the system from this kind of build-up, and it was one of the reasons we added message time-out in MQTT v5.

The other big area is retained messages: people have a tendency to use (abuse?!) brokers as databases (myself included, so I’m not saying you shouldn’t, before anyone argues!) by storing data on retained topics, and depending on the broker, that will (as Greg mentioned) be stored in some combination of RAM and on disk - a very large retained topic tree can take up a lot of RAM space, which then isn’t available for storing in-flight messages.

That said, I generally work with “constrained memory” devices, so I probably worry more about memory filling up than people who use big cloud-based servers with oodles of memory!

I’m sure for any given broker implementation, though, there could be constructed a formula for how many of what types of messages and subscribers and in-use topics, which pretty accurately predicts the RAM usage for that broker.

As I said at the start - an interesting topic for research!

Regards
Andy S-C

> --
> To learn more about MQTT see https://mqtt.org
> ---
> You received this message because you are subscribed to the Google Groups "MQTT" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to mqtt+uns...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/mqtt/rmiikvxqudx.fsf%40s1.lexort.com.
>