How many retained messages per topic are stored and delivered

33 views
Skip to first unread message

Cassie C

unread,
May 7, 2020, 12:32:40 PM5/7/20
to MQTT
Hi forum,
Does MQTT only allow a single retained message to be stored per topic? Or can it store more than one retained message, as long as the latest retained message is what is delivered last when a client subscribes?

This link explicitly says that only one retained message is stored per topic: https://www.hivemq.com/blog/mqtt-essentials-part-8-retained-messages/

However, in reading the 3.1.1 spec (http://docs.oasis-open.org/mqtt/mqtt/v3.1.1/os/mqtt-v3.1.1-os.html), I do not see this same limitation. I see this note under section 3.3.1.3: "Retained messages are useful where publishers send state messages on an irregular basis. A new subscriber will receive the most recent state."

Furthermore, I have a unit test that attempts to publish three retained messages to a topic, then subscribes with a client, and expects to receive a single retained message. However, the client in the unit test frequently can receive 2 or 3 messages, but seems to always receive the latest message last.

Cassie C

unread,
May 7, 2020, 12:36:03 PM5/7/20
to MQTT
I see that the latest spec is here, though I do see the same non-normative comment about "a new subscriber will receive the most recent state": https://docs.oasis-open.org/mqtt/mqtt/v5.0/os/mqtt-v5.0-os.html#_Toc3901104

Simon Walters

unread,
May 7, 2020, 12:42:41 PM5/7/20
to mq...@googlegroups.com
" Does MQTT only allow a single retained message to be stored per topic? "

Yes.

The retained message mechanism is there so new (or reconnecting) clients can get the latest message the broker received on that topic
Simon



--
To learn more about MQTT please visit http://mqtt.org
---
You received this message because you are subscribed to the Google Groups "MQTT" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mqtt+uns...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/mqtt/900f3eb1-9718-4956-80b3-d1c4d16bce44%40googlegroups.com.

Cassie Clark

unread,
May 7, 2020, 12:47:32 PM5/7/20
to mq...@googlegroups.com
Should I expect to possibly receive more than one retained message when a client connects to a topic that has been sent three retained messages?

Andy Stanford-Clark

unread,
May 7, 2020, 12:48:31 PM5/7/20
to mq...@googlegroups.com
you can think of the retain mechanism as a one-message-deep buffer on each topic in the broker.

If you publish a retained message, first it goes to all the subscribers that match the published topic, second the message gets stored in the retain buffer.

When a new client connects and subscribes to a topic with a retained message in the buffer, the broker first sends the retained message to the client, with the retain flag set, so you know it’s not a live message, but is the most recently published (and retained) on that topic. After that, subsequent messages on that topic will be delivered to all matching subscribers. 
Note that if messages are published retained, they do NOT have the retained flag set when they are delivered to already-connected clients - they just look like normal messages. This is so clients don’t incorrectly interpret them as “from the past” retained messages. 

The original design point for retained messages was to populate a display with all the current values from a set of topics when the display client first connects to the broker. After getting the initial update of most recently published values (with the retain flag on), one per topic, it then goes on to update with live readings as they are published.

There’s a good explanation of retained messages and some of the things you can do with them here:

Andy

Andy Stanford-Clark

unread,
May 7, 2020, 12:50:08 PM5/7/20
to 'Simon Walters' via MQTT
you should only receive one message with the retain flag set, per topic, after you subscribe.

You might receive other messages that were published with retain set, but they won’t have retain set on them when they arrive at the subscriber.

Andy

Dustin Sallings

unread,
May 7, 2020, 12:54:53 PM5/7/20
to mq...@googlegroups.com
On Thu, May 7, 2020 at 9:50 AM Andy Stanford-Clark <an...@stanford-clark.com> wrote:
you should only receive one message with the retain flag set, per topic, after you subscribe.

You might receive other messages that were published with retain set, but they won’t have retain set on them when they arrive at the subscriber.

Bit of an asterisk here:  * Unless you specifically ask for the "retain as published" flag.

(I just had a small panic attack wondering how my bridge was working and went back and looked at the spec) 

Andy Stanford-Clark

unread,
May 7, 2020, 1:12:17 PM5/7/20
to 'Simon Walters' via MQTT
oops, sorry - I was still thinking about v3 ;)
Thanks for the clarification!

Andy

--
To learn more about MQTT please visit http://mqtt.org
---
You received this message because you are subscribed to the Google Groups "MQTT" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mqtt+uns...@googlegroups.com.

Florian Raschbichler

unread,
May 7, 2020, 1:23:36 PM5/7/20
to mq...@googlegroups.com
Just be clear. This is not actually part of the MQTT spec, right?
Rather sounds like a proprietary bridge configuration provided by a specific broker.

Cheers,
Florian

Dustin Sallings

unread,
May 7, 2020, 1:29:08 PM5/7/20
to mq...@googlegroups.com
On Thu, May 7, 2020 at 10:23 AM Florian Raschbichler <florian.ra...@hivemq.com> wrote:
Just be clear. This is not actually part of the MQTT spec, right?
Rather sounds like a proprietary bridge configuration provided by a specific broker.


"Bit 3 of the Subscription Options represents the Retain As Published option. If 1, Application Messages forwarded using this subscription keep the RETAIN flag they were published with. If 0, Application Messages forwarded using this subscription have the RETAIN flag set to 0. Retained messages sent when the subscription is established have the RETAIN flag set to 1."

This (plus "No Local") makes it possible to write a bridge across arbitrary MQTT brokers easily.

Florian Raschbichler

unread,
May 7, 2020, 1:32:38 PM5/7/20
to mq...@googlegroups.com
Got’ya! 
Seems i missed the second mail of this thread, switching to the MQTT 5 spec.

--
To learn more about MQTT please visit http://mqtt.org
---
You received this message because you are subscribed to the Google Groups "MQTT" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mqtt+uns...@googlegroups.com.

Dustin Sallings

unread,
May 7, 2020, 1:32:52 PM5/7/20
to mq...@googlegroups.com
On Thu, May 7, 2020 at 9:32 AM Cassie C <scurlo...@gmail.com> wrote:
Furthermore, I have a unit test that attempts to publish three retained messages to a topic, then subscribes with a client, and expects to receive a single retained message. However, the client in the unit test frequently can receive 2 or 3 messages, but seems to always receive the latest message last.

Does this happen even if the message is not retained?  I would expect that behavior if you're reusing sessions across invocations.

Subscriptions belong to sessions, not to clients.  Messages should be sent to clients even if they aren't connected. 

Dustin Sallings

unread,
May 7, 2020, 1:34:40 PM5/7/20
to mq...@googlegroups.com
On Thu, May 7, 2020 at 10:32 AM Florian Raschbichler <florian.ra...@hivemq.com> wrote:
Got’ya! 
Seems i missed the second mail of this thread, switching to the MQTT 5 spec.

Yes, sorry, the original specifically was asking about v3.1.1.  MQTT became considerably more interesting to me with v5.  :)

The basic behavior is the same, but you get a bit more granularity in v5 (and can expire retained messages, which is hugely helpful most places I use them).

Cassie C

unread,
May 7, 2020, 2:17:16 PM5/7/20
to MQTT
For some clarity, the implementation of mqtt that we are using, MQTTnet (https://github.com/chkr1011/MQTTnet) uses protocol version 3.1.1 by default. There is a way to specify version 5.0.0 but we are not using it since it does not seem to work for all server behaviors.
So in my case, I am using 3.1.1.

I assumed that the retained message behavior would be similar for 3.1.1 and 5.0.0, but for my case, I will focus on 3.1.1.

Thanks.


On Thursday, May 7, 2020 at 10:32:40 AM UTC-6, Cassie C wrote:

Dustin Sallings

unread,
May 7, 2020, 2:30:35 PM5/7/20
to mq...@googlegroups.com
On Thu, May 7, 2020 at 11:17 AM Cassie C <scurlo...@gmail.com> wrote:
For some clarity, the implementation of mqtt that we are using, MQTTnet (https://github.com/chkr1011/MQTTnet) uses protocol version 3.1.1 by default. There is a way to specify version 5.0.0 but we are not using it since it does not seem to work for all server behaviors.
So in my case, I am using 3.1.1.

I assumed that the retained message behavior would be similar for 3.1.1 and 5.0.0, but for my case, I will focus on 3.1.1.

The differences shouldn't affect you much.  Basically, v5 gives you some more features.  I implemented v5 in a couple of clients such that they degrade by just ignoring v5 specific bits.  If you rely on these features, it might be slightly confusing, but the overall behavior won't matter.

If your test is seeing duplicate messages on connect while messages are not actually being sent, they're probably just sitting in the session queue.  You should set the clean session bit on each request as well as perhaps just use a distinct client name.  Depending on the broker, you can let the broker assign your client ID which makes that a lot easier.

i.e., "retain" may be unrelated to your problem. 
Reply all
Reply to author
Forward
0 new messages