Getting size in bytes of incoming message

[Marshall Royce]

Hello all, I'm currently working on adding a feature to Telegraf's mqtt_consumer plugin, I'm wanting to be able to calculate the message size received by the network interface in order to calculate the throughput (bytes/minute) being sent from the mqtt server. This is all written in Golang, is this possible? Right now I essentially just sum together each part of the incoming message (Payload, topic, QoS, message ID, duplicate, and retained) by using len() on the strings and byte arrays, and Unsafe.sizeof() on the rest. However, I believe this is getting the size that the message takes up in memory rather than the actual size of the message itself. Any suggestions or pointers? thanks in advance!

[Andy Stanford-Clark]

MQTT has a “remaining length” field after the fixed header - if you decode that (it’s variable length encoding, so requires a loop) you get the number of bytes in the variable header and payload that come after it. 

Add that to the length of the fixed header (1 byte) and the number of bytes encoding the remaining length (count them as you decode them), then you have the exact message length.

The MQTT protocol spec explains the message structure and how to decode the remaining length field.

Andy

On 11 Jul 2022, at 13:44, Marshall Royce <marshal...@gmail.com> wrote:

Hello all, I'm currently working on adding a feature to Telegraf's mqtt_consumer plugin, I'm wanting to be able to calculate the message size received by the network interface in order to calculate the throughput (bytes/minute) being sent from the mqtt server. This is all written in Golang, is this possible? Right now I essentially just sum together each part of the incoming message (Payload, topic, QoS, message ID, duplicate, and retained) by using len() on the strings and byte arrays, and Unsafe.sizeof() on the rest. However, I believe this is getting the size that the message takes up in memory rather than the actual size of the message itself. Any suggestions or pointers? thanks in advance!

--
To learn more about MQTT see https://mqtt.org
---
You received this message because you are subscribed to the Google Groups "MQTT" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mqtt+uns...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/mqtt/a0996619-4fc3-486c-a580-8c3d41bf1975n%40googlegroups.com.

[Marshall Royce]

Apologies it seems I replied privately on accident instead of here... Anyways, I understand what you mean and I agree it would work well for what I'm trying to do, but I'm really struggling to decode the incoming packet. I'm working with the mqtt.Message interface which already has all the fields filled in, and I can't really access the actual incoming packet itself, it seems out of scope. I've been digging around all the documentation and can't really find much on how to get the packet from this level. Do you have any recommendations on how to reach it to decode? Or any possible alternate solutions? Thank you in advance!

Marshall 

[Andy Stanford-Clark]

OK - that was an interesting question :) 

How do you reverse engineer the MQTT message size from a received topic, payload and flags set?

And here’s the answer: 

Fixed header: 1 byte

The remaining content comprises: 

Topic length + 2 (for the topic length bytes)

For a qos 1 or 2 message: 2 bytes (for msg ID)

Payload length

And then you work out the length of the “remaining length” field from the size of that remaining content above…

1 byte for remaining length 0  to 127 

2 bytes for rem len 128 to 16,383 

3 ... 16,384 to 2,097,151 

4 … 2,097,152 to 268,435,455 

Add those 3 elements together and that’s the length of your publish message.

Now, if it’s QOS 1

Add 4 bytes for the PUBACK the client sends back to the broker

If It’s QOS 2

Add 12 bytes for the PUBREC, PUBREL, PUBCOMP (@ 4 each) for the handshake which completes the QOS2 message delivery.

That was fun :)

Andy

To view this discussion on the web visit https://groups.google.com/d/msgid/mqtt/e43ee83f-be88-4d5b-9646-2813e2e6b757n%40googlegroups.com.

[Marshall Royce]

Wow... I would have never figured that out on my own, thank you so much! Glad I could give you a little bit of a challenge for your day haha. Just to make sure what I'm doing is right before I commit it, does this look to be about what you explained?

byteCount := 0

qosFlagsSize := 0

topicSize := len(msg.Topic()) + 2

qOsSize := 0

if msg.Qos() == 1 || msg.Qos() == 2 {

     qOsSize = 2

}

payloadSize := len(msg.Payload())

remainingContent := topicSize + qOsSize + payloadSize

remainingLength := 0

if remainingContent < 127 {

     remainingLength += 1

} else if remainingContent < 16383 {

     remainingLength += 2

} else if remainingContent < 2097151 {

     remainingLength += 3

} else {

     remainingLength += 4

}

publishMessageSize := remainingLength + remainingContent + 1

if msg.Qos() == 1 {

     qosFlagsSize += 4

} else if msg.Qos() == 2 {

     qosFlagsSize += 12

}

byteCount += publishMessageSize + qosFlagsSize

[Andy Stanford-Clark]

You are most welcome :)

Very nearly right. 

You want <= on those comparisons for the remaining length boundaries, i.e.

If remainingContent <= 127 

(Or < 128 ;))

etc

Also just as a defensive programming thing, I would *assign* those remainingLength values rather than incrementing them.

Same for the QosFlagsSize values - you’re not going to do it more than once, thanks to the structure of your if/then/elses, so just assign it.

Regards

Andy

To view this discussion on the web visit https://groups.google.com/d/msgid/mqtt/8a0d4567-e47f-4b2b-83f9-c1608117c4b4n%40googlegroups.com.

[Marshall Royce]

I see I see, I think I had just done that because I was confused about if I'd be able to access them after the if statements but I went ahead and got it implemented the way you recommend and it's working well. Thank you for your help again!

Best,

Marshall

[Roger Light]

... and if it's MQTT v5 then you need to add some more on to account for properties, but you won't absolutely get all the information that you need to correctly calculate the size in a user library.

To view this discussion on the web visit https://groups.google.com/d/msgid/mqtt/25432C09-C743-4A78-8A5D-B9CDB64CCDFC%40stanford-Clark.com.

[Andy Stanford-Clark]

Left as an exercise for the reader ;)

To view this discussion on the web visit https://groups.google.com/d/msgid/mqtt/CAH7zdydpiFNWL6bkg9tSSA7KV_vGzTE5qp3OqMx-pudQqBgeRw%40mail.gmail.com.