MQTT-ultralight? or: does MQTT-SN go far enough?

[Thorsten von Eicken]

I'm looking into implementing MQTT-SN on tiny embedded nodes in a home automation setting. Mostly JeeNodes (arduino + rfm12b) or wimpy ARM uC with 16KB flash and 4KB RAM. In these systems every byte counts. It seems to me that MQTT-SN is still too complex. I'm new to MQTT, so I expect there are some things I mis-understand or simply don't understand.

What I mean by "too complex" is the following. MQTT provides solutions for a number of issues: the core pub/sub routing, dynamic topic management, state retention (the Will notion), QoS/reliability, gateway/router detection, node/client presence/liveness, and perhaps more. MQTT-SN simplifies the verbosity of the message format and of the topics, but it doesn't really simplify the rest much, or perhaps not enough for me. And it doesn't offer a way to say "I want state retention and QoS but I don't want gateway detection nor node liveness". (The QoS -1 mode is one step in this direction, but it's a fixed package deal.) As a result it doesn't simplify the protocol state machine and that just ends up in a ton of code on a tiny uC. For example, as far as I can tell the arduino implementations out there use up the majority of program memory for MQTT. That's great for a demo, for real life the comm protocol has to use a minority of program space. At least in my book...

I believe the worst part of MQTT-SN is the QoS. The first problem is that it's solved anew for each message type: PUB has a PUBACK, SUB has a SUBACK, CONNECT has a CONNACK, etc. Trivial on a real machine, but on a uC just the dispatch of all the message types and the code fragment for each one becomes significant. But the bigger problem is that MQTT-SN doesn't allow QoS to be solved at the link layer (yes, I know that QoS=2 can't really be solved there). This is subtle until you try to implement retransmission. The key question is: if I just transmitted a packet can I sit there waiting for an ACK or rexmit timeout (whichever comes first) and drop everything else on the floor or can't I?

If the answer is "no, you have to accept other incoming messages" then you have to either buffer them (yet you have almost no memory) or deliver them to the app (which ends up requiring dynamic dispatch). If you deliver to the app chances are that the app will want to send something which will need to be queued (here goes the memory again). So you inherently buy into dynamic dispatch and buffering of a somewhat unbounded number of messages (the peer could easily feed you a dozen data packets before sending you the ACK you're waiting for).

In order for the answer to be "yes, you can just sit there" both ends must prioritize ACKs, which means that if they receive a message that requires an ACK they must send that ACK before sending anything else they may want to send. Unless I'm missing something this is the only way to ensure you don't have deadlock (or is that livelock?) where two nodes don't make progress because they're waiting on each other's ACK and neither can send it because they're repeatedly retransmitting another data packet.

Coming back to MQTT-SN, the ACKs are not at the transport level, they're higher up, which means that you actually can't generate immediate ACKs. In fact a forwarder or a GW can't generate most of the ACKs, only the broker can (right?). Thus I would claim that you cannot write a proper MQTT-SN implementation without dynamic scheduling and/or packet buffering. And the result of that is that by the time you've implemented MQTT-SN you have very little room left.

The simplification of the topics in MQTT-SN is nice, but I feel like I'm falling through the cracks. With "pre-defined" topics something still needs to pre-define the topics with the GW, and with "short topics" there's no wildcarding. Why is there no wildcarding? I was thinking about mapping the physical topology into the topic space: for each node there's a network id and a node id, and each message could have a 1-byte sub-topic and all this could be mapped into something like "/rfm12/<network_id>/<node_id>/<sub-topic>", where <xxx> refers to the decimal representation of the value. In addition each node could be auto-subscribed to it's part of the sub-tree (assuming this wouldn't cause it to receive its own messages). In the end this is like short topics but using fewer bytes and providing more functionality...

In the end I feel like I want a MQTT-UL (ultralight) or more flexibility in MQTT-SN that would have the following features:

 - make gateway location/detection optional

 - make node liveness optional

 - allow retransmissions to be implemented at the link layer

 - give me "handshake-less" topics that support wildcarding plus auto-subscription

 - eliminate the flags byte in the publish message (after moving retransmission into the link layer the only flag that's left is the Will flag and that could be associated with the topic)

Whoa, this is a long post already. Apologies for that, this stuff has been swirling in my head for the past 2 days. Am I missing the point of MQTT and MQTT-SN completely perhaps? Do others feel similarly? Am I trying to shove a square peg into a round hole?

NB: I still see not MQTT-SN tag, I do see an mqtt and a MQTT tag now, so something probably went awry...

[Tomoaki Yamaguchi]

My implementation of  MQTT-SN client over XBee S2 for Arduino is only 20Kbytes.

I think MQTT-SN is not so complicated.  

https://github.com/TomoakiYAMAGUCHI/MQTT-S

2014/1/3 Thorsten von Eicken <tvone...@gmail.com>

--
To learn more about MQTT please visit http://mqtt.org
---
You received this message because you are subscribed to the Google Groups "MQ Telemetry Transport" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mqtt+uns...@googlegroups.com.
To post to this group, send email to mq...@googlegroups.com.
Visit this group at http://groups.google.com/group/mqtt.
For more options, visit https://groups.google.com/groups/opt_out.

--

         

               Tomoaki  YAMAGUCHI

[Thorsten von Eicken]

Tomoaki, that's great! But 20KB out of 32KB is 62%! That means that before someone starts writing any application, loading any other library, doing anything else 62% are already consumed. That doesn't make sense to me.

[Ian Craggs]

Thorsten,

for both MQTT and MQTT-SN, you don't need to implement the features of the whole protocol.  A particular application need implement only the features it needs.  A client library could be structured so that only the needed features are included in the application.  As the specification for both is public, you don't have to use a library, if you don't want to.  

A few years ago we had a little competition within IBM to write the smallest program to publish an MQTT QoS 0 message, in either a complied or interpreted language.  The approach in both cases was to open a socket, then send a sequence of bytes down it which corresponded to a connect packet followed immediately by a publish packet.  We didn't have to bother with the disconnect packet even, just close the socket.  The compiled competition was won with a Linux binary of less than 400 bytes for the whole program (as far as I remember), while I won the interpreted competition with about 90 bytes of Ruby (I know there is the interpreter - this is just for illustrative purposes).

For MQTT-SN, it would be shorter, as you don't have to bother with the connect if you use QoS -1.  We have thought about enumerating reduced function MQTT "profiles" from time to time, but we weren't sure that was necessary.  

We are open to suggestions for improving MQTT-SN, so anything that does improve its ability to be implemented in small environments would be considered.  In many circumstances however, I would say don't implement what you don't need.

Ian 

[Thorsten von Eicken]

Ian, thanks for taking the time to respond. I would contend that while your challenge to send a QoS 0 packet is cool, I don't think it's all that useful. If I'm using QoS 0 and a "blind connect" then I end up not really benefiting from MQTT. Something interesting would be the size of a small controller that sends out a temperature value every few seconds and subscribes to two topics to receive QoS 1 commands with a binary flag (e.g. to turn on/off two relays). Now you start dealing with all the node liveness, subscription, ACKs, etc.

I understand your point about MQTT (and -SN) being open and that I can implement a subset, but to me a big plus of using a std protocol is that I can plug into existing software. In particular, I find UDP very useful in gatewaying between various RF environments and servers. Nicholas (njh) wrote a UDP-mqtt-sn to mqtt gateway, but if my RF nodes don't implement the full MQTT-SN protocol I can't really interface directly with his gateway. The usefulness of reduced function profiles would be to make such interfacing more standard.

[Karl P]

On 01/05/2014 02:30 AM, Thorsten von Eicken wrote:
> Ian, thanks for taking the time to respond. I would contend that while your
> challenge to send a QoS 0 packet is cool, I don't think it's all that useful. If
> I'm using QoS 0 and a "blind connect" then I end up not really benefiting from
> MQTT.

Says you :)

Not every node needs to subscribe to anything, not everything needs high qos.
Some things would make more sense if they sent out state at qos0, rather than
state changes at qos>0. All of a sudden you don't care about acks, you don't
care about subscribing, you don't care about maintaining state. And you still
have all the _rest_ of the mqtt goodness for the _rest_ of your application
environment. Other applications/nodes can subscribe, repub, resub, whatever to
the topic you are "blindly publishing" too, without your "blind publisher"
having to know _anything_ beyond the network method to deliver that publish.

Something interesting would be the size of a small controller that sends
> out a temperature value every few seconds and subscribes to two topics to
> receive QoS 1 commands with a binary flag (e.g. to turn on/off two relays). Now
> you start dealing with all the node liveness, subscription, ACKs, etc.
>

Again, it's not appropriate for every node, but you don't always need to make
this high qos. You can do things in the smart side of the app by listening for
periodically published state, and republishing a command if the state isn't what
you expect. This is the repeatable/idempotent style used on some webservices,
ie, "make the bank balance $10", which can be sent over and over again, vs
"reduce bank balance by $1", which clearly you only want to send once.

Twisting the thinking a little bit and you might find that even regular mqtt
isn't nearly as difficult as you think.

Cheers,
Karl P

[Thorsten von Eicken]

Karl, what you say is totally correct, and I'm aware of that. However it doesn't contradict what I stated. I'm looking for a pub/sub protocol that makes it easy for me to deploy a few dozen arduino-level nodes. Your answer basically boils down to: they will either have QoS=0 and a library that does not properly implement the protocol or they will have >60% of memory devoted to the protocol (and even that version has some serious protocol issues if my reading is correct). That's not interesting to me. In order to feed a few values to other apps that may use MQTT I'm better off building my own protocol and gatewaying to MQTT than to either hack partial MQTT into the nodes or use MQTT-SN. The responses I got here indicate that the MQTT group isn't interested in solving the problem domain I'm in, which is fine.

BTW, is there any reference full implementation of MQTT-SN for tiny devices? Preferably one that doesn't require dynamic message dispatch and doesn't assume a deep buffer queue.

[Ian Craggs]

Thorsten,

On Sunday, 5 January 2014 17:37:47 UTC, Thorsten von Eicken wrote:

Karl, what you say is totally correct, and I'm aware of that. However it doesn't contradict what I stated. I'm looking for a pub/sub protocol that makes it easy for me to deploy a few dozen arduino-level nodes. Your answer basically boils down to: they will either have QoS=0 and a library that does not properly implement the protocol or they will have >60% of memory devoted to the protocol (and even that version has some serious protocol issues if my reading is correct). That's not interesting to me. In order to feed a few values to other apps that may use MQTT I'm better off building my own protocol and gatewaying to MQTT than to either hack partial MQTT into the nodes or use MQTT-SN. The responses I got here indicate that the MQTT group isn't interested in solving the problem domain I'm in, which is fine.

I don't agree with you on this.  There is no need for a "client" to support the whole of MQTT or MQTT-SN to "properly" implement the protocol.  Even a server need not implement everything.  An MQTT server might only implement QoS 0, and this would conform to the protocol perfectly well, as any subscription request would receive a suback saying that that the granted QoS is 0.  The test is whether that client or server can communicate with a fully featured remote end.

Each QoS is a balance of message delivery reliability against the cost of implementation and execution.  Otherwise all messages may as well be QoS 2.

 

BTW, is there any reference full implementation of MQTT-SN for tiny devices? Preferably one that doesn't require dynamic message dispatch and doesn't assume a deep buffer queue.

The IBM Zurich lab has a reference implementation of MQTT-SN for ZigBee.  I don't know whether it is full, but we intend to contribute it to the Eclipse Paho or Mosquitto projects soon.

Ian 

[Ian Craggs]

Thorsten,

On Sunday, 5 January 2014 02:30:24 UTC, Thorsten von Eicken wrote:

Ian, thanks for taking the time to respond. I would contend that while your challenge to send a QoS 0 packet is cool, I don't think it's all that useful. If I'm using QoS 0 and a "blind connect" then I end up not really benefiting from MQTT.

I contend that it is useful.  It's then down to the design of your application.  If you are expecting messages from a sensor on a regular basis, but the sensor can only support QoS 0, then another application could monitor those messages to detect missing ones and take action - which could be notifying an administrator for example.  If the underlying transport is TCP/IP, then even QoS 0 messages will reach their destination, except in the case of loss of connection.

 

Something interesting would be the size of a small controller that sends out a temperature value every few seconds and subscribes to two topics to receive QoS 1 commands with a binary flag (e.g. to turn on/off two relays). Now you start dealing with all the node liveness, subscription, ACKs, etc.

That's an interesting challenge!  I think I'll see if anyone has an example already.

 

I understand your point about MQTT (and -SN) being open and that I can implement a subset, but to me a big plus of using a std protocol is that I can plug into existing software. In particular, I find UDP very useful in gatewaying between various RF environments and servers. Nicholas (njh) wrote a UDP-mqtt-sn to mqtt gateway, but if my RF nodes don't implement the full MQTT-SN protocol I can't really interface directly with his gateway. The usefulness of reduced function profiles would be to make such interfacing more standard.

Your RF nodes can interface with Nichlolas' gateway without implementing MQTT-SN completely.   Or they can if the gateway does what I think it does.  I have also implemented a UDP MQTT-SN to MQTT gateway (in the Eclipse Mosquitto project http://git.eclipse.org/c/mosquitto/org.eclipse.mosquitto.rsmb.git/)/.   An MQTT-SN client could use QoS -1 publications only - in which case they don't need to even connect.   A client which implements only QoS -1 or 0 publications, is perfectly well conforming - it is not going to receive any MQTT or MQTT-SN messages it can't handle

Ian

[x13dev]

Look at our implementation, supported JeeNode(ATMega328+RFM12) & PanStamp(ATMega328 +CC1101).

https://github.com/X13home/X13.devices

пятница, 3 января 2014 г., 7:07:58 UTC+1 пользователь Thorsten von Eicken написал:

I'm looking into implementing MQTT-SN on tiny embedded nodes in a home automation setting. Mostly JeeNodes (arduino + rfm12b) or wimpy ARM uC with 16KB flash and 4KB RAM. In these systems every byte counts.

regards, Wassili Hense.

http://x13home.github.com/  Open Source Home Automation system. 

MQTT, MQTT-SN over UDP & RF 

[Ian Craggs]

I've started implementing an MQTT C client structured as a set of helper functions, so that it can be small.  Hopefully as small as most people will need. http://git.eclipse.org/c/paho/org.eclipse.paho.mqtt.embedded-c.git/  Some description is here: http://modelbasedtesting.co.uk/?p=69.

I've created an example of a simple application that subscribes to one topic to receive QoS 1 command messages and publishes QoS 0 readings, at http://git.eclipse.org/c/paho/org.eclipse.paho.mqtt.embedded-c.git/tree/samples/pub0sub1.c.  It doesn't cover all the error situations (like losing a connection), but it shows how you might start.

Ian