MQTT 5.0 Can [MQTT-3.8.3-3] conflict with [MQTT-3.3.1-9] and what to do about it?

[Philip Couling]

I've tripped up on what looks like conflicting mandatory requirements in MQTT5.0

The standard does not appear to define what to do when "no local" bit 2 and "Retain Handling" bits 4 and 5 conflict.

• [MQTT-3.8.3-3] Bit 2 of the Subscription Options represents the No Local option. If the value is 1, Application Messages MUST NOT be forwarded to a connection with a ClientID equal to the ClientID of the publishing connection.
  
• [MQTT-3.3.1-9]
  If Retain Handling is set to 0 the Server MUST send the retained messages matching the Topic Filter of the subscription to the Client.
  
• [MQTT-3.8.3-1] The Topic Filters MUST be a UTF-8 Encoded String.
  "Each Topic Filter is followed by a Subscription Options byte."
  

It seems clear from the standard that a "topic filter" is only the string that's used to pattern match the topic and does not include the subscription options byte.  So [MQTT-3.3.1-9] does not account for the subscription options. 

So what happens when a client's own message is the retained message and it then subscribes with no-local 1 (bit 2) and retain handling 0 (bits 4,5):

• If the retained message is sent to the client and [MQTT-3.8.3-3] is violated
• If the retained message is NOT sent to the client and [MQTT-3.3.1-9] is violated

It seems to me that either course of action could break things.  Sending a client it's own message when it explicitly asked not to receive them could break something.  Failing to send a topic's retained message could just leave the client hanging.

Is there any existing consistency in implementation for this scenario?

Kind regards

[Roger Light]

Hi Philip,

In the section 3.3.1.3 where the [MQTT-3.3.1-9] requirement is
described, this paragraph precedes the list of requirements:

> When a new Non‑shared Subscription is made, the last retained message, if any, on each matching topic name is sent
> to the Client as directed by the Retain Handling Subscription Option. These messages are sent with the RETAIN flag set
> to 1. Which retained messages are sent is controlled by the Retain Handling Subscription Option. At the time of the Subscription:

This makes it clear to me that [MQTT-3.3.1-9] only applies when a
subscription is being made, not in direct response to a PUBLISH.

I believe the intent of the spec is that once a message is accepted as
a retained message, the original client is no longer considered to be
the publishing client, hence if it made a later subscription that
matched that retain message it would still expect to receive it. In
other words, the [MQTT-3.8.3-3] only applies to live connections. I
don't believe that is written down, but it goes hand in hand with the
decoupled nature.

Having said that, some people contend that there is a security issue
here. For example, if a client publishes a retained message and at a
later point its authorisation to publish to that topic is removed, the
contention is that this retained message should no longer be published
to new subscribers. This means that keeping the association between a
retained message and the originating client id is required, which
confuses the point above.

Mosquitto does have the capability to record the originating client in
retained messages, but does not consider that when sending retained
messages on a subscribe. In other words it obeys [MQTT-3.3.1-9].

Regards,

Roger

> --
> To learn more about MQTT see https://mqtt.org
> ---
> You received this message because you are subscribed to the Google Groups "MQTT" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to mqtt+uns...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/mqtt/CANWftzK6hwsaTNV8z4EZLSvW5j%2BWuUP3V87RacznxXEOGxjM0w%40mail.gmail.com.

[Andy Stanford-Clark]

Well explained, Roger :)
I agree with that… based on the point that a retained message becomes a property of the *topic*, not of the originating client, and therefore should be sent back to the originating client as a retained topic if it subscribes to it later.

The original intention of “no local” was that there’s no point in sending a message back to the client that just published it, as it would already know the thing that it just sent.
But later on, it would most likely benefit from the last-known publication on topic (no matter who published it), so its world-view is up to date.

Andy

> To view this discussion on the web visit https://groups.google.com/d/msgid/mqtt/CAH7zdyempLigeEdiZ6MyQP6fdohdA367y-vokeEWbhirj1d7PQ%40mail.gmail.com.
>

[Philip Couling]

Thank you both.

Okay so you are saying the spirit of the standard (if perhaps not the letter) is for retained messages to be "re-published" by the server as if the server itself is a connection with a unique client id.

I especially appreciate the insight that "no local" is intended as a performance booster and perhaps clients should be written to not break if they do receive their own messages back.

Thanks

To view this discussion on the web visit https://groups.google.com/d/msgid/mqtt/6CE38EF4-5CCA-45A6-AE58-4A155F61EDAC%40stanford-Clark.com.

[Andy Stanford-Clark]

Yes, that’s a good way to think about it.

Bear in mind there’s also the cost of sending the message back over a slow expensive network (I know there aren’t so many of those these days ;))… 

Andy

To view this discussion on the web visit https://groups.google.com/d/msgid/mqtt/CANWftzJg-y9R_Honhj%2Bk6g5%2BAo4YS_9mnbgDchLesWDwbJBhoQ%40mail.gmail.com.

[Dustin Sallings]

The local thing was super helpful when building a bridge to avoid making loops.  In the most simple case, one can subscribe to the same topics on two sides and then convey anything that comes across.  For this usage, picking up retained messages is actually critical.

To view this discussion on the web visit https://groups.google.com/d/msgid/mqtt/26511D6E-AF4E-4039-9B47-CE646AC29F52%40stanford-Clark.com.

[Andy Stanford-Clark]

Yes... “avoiding looping on a bridge” was actually the original use case - I’d forgotten that! 

I think it was @icraggs’s idea :)

Andy

To view this discussion on the web visit https://groups.google.com/d/msgid/mqtt/CAFnwT5_OdCPj--a68kXZLwJx1jwu2GP3owvSd5GLwmnDaNocQA%40mail.gmail.com.