MQTT Message Ordering - MQTT 3.1.1 vs 3.1

[Dominik Obermaier]

Hello dear MQTT community,

I recently had an interesting discussion with a colleague about MQTT message ordering while implementing a low-level MQTT 3.1.1 client. The MQTT 3.1.1 specification does not say anything about an in-flight windows, in contrast to MQTT 3.1 when it comes to message ordering (see [1] and [2]). My understanding of the passages is, that MQTT 3.1.1 has a default in-flight window of 1 *for every single topic*, while MQTT 3.1 has a *global* in-flight window (with no size recommendation). 

This makes it certainly very easy for 3.1.1 clients to handle async publishes with QoS 1 and 2 but I find this major change rather confusing, especially if it comes to MQTT 3.1.1 compliance, since a good MQTT API would need to have the ability to define “unordered” topics somehow.

After checking a few implementations, I noticed, that MQTT 3.1.1 clients like Eclipse Paho (at least the Java lib) define a global in-flight window and not a per-topic window. Actually, every implementation I checked does something similar and does ignore the fact that there are ordered topics. I’m aware of the fact that the spec explicitly states that the server must treat each topic as ordered topic but in my understanding this should also be applied to clients (when it comes to resends; see the client resend rules in the spec), because the resend rules for clients are de-factor topic in-flight windows of 1.

So my question is: How did other people solve the ordered vs unordered topic challenge, especially when it comes to client API design. Is there even any client library out there which implements the MQTT 3.1.1 spec in regards to the message ordering correctly? Or am I just misunderstanding a key aspect of the 3.1.1 spec?

All the best,

Dominik

[1] http://docs.oasis-open.org/mqtt/mqtt/v3.1.1/os/mqtt-v3.1.1-os.html#_Toc398718105

[2] http://public.dhe.ibm.com/software/dw/webservices/ws-mqtt/mqtt-v3r1.html#ordering

[Michael Combs]

I'm not a fan of the wording of that section (4.6), especially considering that 'ease of implementation' is supposed to be one of the benefits of MQTT. I think it implies a more complex requirement than is needed to implement. I have implemented both client and broker and my reasoning, however bad, for the single/global inflight window for clients (and per client for brokers) is as follows....

It's much simpler to implement and meets the requirements of ordered topics.

This simply requires one queue/backlog and the state of the last message per client (i.e. send a message from the queue, if QoS > 0, then queue messages until an ACK is received for the previous message). The result is that each message, per client, regardless of topic has to wait on all previous messages Ordering messages per topic per client would require maintaining a queue/backlog and state per client per subscribed topic (i.e. send a message from each subscribed topics queue, if QoS > 0, then queue messages for that topic until an ACK is received for the previous message). This would allow, at the cost of complexity and memory, each topics messages to be sent and received without waiting on a different topics messages. 

The single inflight queue per client doesn't generally impact broker or client performance though. 

At the client consider receiving 2 messages, topic A and B resp., QoS > 0, from a broker. If the client is using asynchronous IO it's likely queuing and processing (or sending) messages on a dedicated thread in order anyway (i.e. one message at a time). Even if it is running multiple processing queues/threads, the most likely reason for a delayed/unsent ack to the broker (i.e. causing all messages for this client to wait at broker ) is probably client failure or connection loss. So it doesn't matter if the broker can continue to send topic A while waiting on an ACK for topic B because the client can't receive messages anyway. This wait at the broker is only per client, so no other clients are affected since they each have their own queue.

At the broker, consider a receiving a publish message,  QoS > 0. The broker looks up the subscriptions and finds 2 clients. If there is no pending QoS for a client then send the message, otherwise, place the message on the clients queue. Process the next message. The broker is free to continue processing request for other clients.

I have implemented this both ways though. I maintain a tree of topic nodes, each of which contain a list subscriber nodes (i.e. client, date, etc...). For the simple implementation I keep a queue and QoS state on the client. When a publish arrives, I lookup the topic node, then for each subscriber node just check the subscriber.client QoS state and either send or place in queue. The more complicated version maintains a queue and state on the subscriber node rather than the client. The method is the same, except check the state and queue on the subscriber node (not the subscriber.client)  before sending. This works at the client and the broker. A topic/subscription tree on the client is also handy for storing callbacks/delegates per subscription as well.

As far as unordered topics go, you just ignore any pending QoS publish messages and always send. Just let the messages fly from either client or broker.

Hope this helps,

-Michael

[Ian Craggs]

Dominik,

my reading of the MQTT 3.1.1 standard is:

1) it does not specify any inflight window size, or whether it should be per client or per topic.   I think that is up to the implementer.
2) it allows a topic to be unordered (but not by default), which means that the ordering rules do not have to be abided by.
3) [MQTT-4.6.0-6] says that publish packets should be sent in the same order that they were received on ordered topic.  This can't be strictly met on session reestablishment for duplicated packets, but holds otherwise.

This is the view that I have taken with the Paho clients - and that consquently no changes were necessary from their 3.1 support.

Ian