MQTT with dynamic subscriptions

[Ian M]

Hi,

I'd like to setup an MQTT input node with the Node-Red flow determining which topic to subscribe to.  Can, and how do you use a variable in the MQTT subscription node?

Thanks,
Ian

[Dave C-J]

Hi Ian,

currently no. But you can set up a wildcard subscription that will cover all your needs and then use a switch node to filter/fork...

[Dave C-J]

... or more likely a function... 

[Ian M]

What about using a var within context.global?

[Dean Sellers]

From my understanding of MQTT the wildcard system is what you want.
The node makes a connection to the broker and subscribes to '/atopic/#'. Any messages published to topics under This will be delivered to the downstream node with the topic in the message object.
eg:
a remote device publishes 'data' to /atopic/device1. You get a msg with payload 'data' and topic /atopic/device1.You can then use a switch or function node to route this data as needed based on the topic.

Make sense? Google MQTT wildcard for examples of this. I'd post links but i am typing on my phone.

This is much more efficient than the node opening up multiple connections, and one of the beauties of the pub/sub model.

If you would like more specific help maybe post more details of your use case.

[Ian M]

Hi Dean, thanks for your reply.

I've been using wildcards in this project and others and they work very well, but in this particular instance I'd like to avoid them.

I'm working on a kiosk system for a local museum where each kiosk is used as a display system, similar to a digital signage system.  Each kiosk exists in a stateless condition between reboots and they are all booted from a single read only NFS mount.  The catch is that each kiosk needs to display different information.  The idea is that the Node-Red instance on each kiosk would monitor a unique MQTT topic that would deliver it information to control the display of content.  The Node-Red instance discovers which topic to follow by searching a central configuration file for its MAC address.  Because of the potential number of kiosks on the network I would prefer to avoid having all kiosks subscribed to all of the topics so I'm trying to find a solution.

As I see it there are four potential courses of action:
a) Adapt the normal MQTT node to facilitate this use case,
b) Write a custom node for it,
c) Execute code to run a seperate MQTT client and feed the results back into Node-Red,
d) Subscribe to everything and use switches.

I think COA a) makes the most sense.  The MQTT output node can use different topics, so it would make sense for the input nodes to be able to change topics as well.  Since I posted this question I found one other use-case where this would be useful, in that case a need for stateless clients to subscribe to topics based on geographic considerations, and I imagine there are others.  I've experimented with adapting the MQTT node to facilitate this, and while I now have a node with an input and an output, I haven't been able to make it re-connect when it receives a new topic, probably due mostly to my ineptitude with JavaScript.  I've been trying to set it up so that when it receives a msg.topic it would close any existing connection and establish a new connection with the new topic.  Because I imagine this only happening once shortly after the client boots, there wouldn't be a concern about missing traffic during the process.

I'll probably give it another try tonight before moving onto COA c)...

Ian

[Hans Jespersen]

This exact use case came up in another thread.  Another option is to use an MQTT broker than allows each individual client's topic subscriptions to be dynamically modified on the MQTT broker. This would allow all the kiosks to connect on the same topic and the broker would use some unique identifier from the client connection (MAC address, client id, whatever) to add new topic subscriptions to the list it maintains for that clients session. Then each node-red client would start recieving messages that are unique to that kiosk. This pattern is similar to the use case of how foreign exchange rates get distributed by big banks to their customers. Everyone subscribes to the US Dollar/Euro exchange topic but the good customers get better rates than the bad customers. Sending all the exchange rates to everyone and then having them see that someone else is getting a better rate would be counter productive to their business ;-)

-hans

[Dave C-J]

If the topic happened to contain the mac address somewhere then at boot time you could run a simple script that could search and replace a dummy entry in the flow file with the local mac address, before starting Node-RED.

[Dean Sellers]

Hi Ian,

Sounds interesting. Slightly off-topic, what hardware are you (or planning to) using?

[Ian M]

Hans,
Your suggestion is interesting but I don't think that I want to complicate things too far beyond my understanding...  I didn't realize that the MQTT protocol was malleable enough to to facilitate such a solution.

Dave,
I'm using a three-letter hash of the last four digits of the MAC address as a client ID.  I think I may try to implement this at-boot swaparoo as a solution.  I'm already running some scripts at boot to generate the hash and confirm that the client is registered and not a new addition, so this might be a simple although not the most elegant solution.

Dean,
We were intending to use the Raspberry Pi B+, but we just began development on the Raspberry Pi 2 which is a crazy improvement.  Our goal is to keep the cost low enough that is as affordable to use this solution as it is to have traditional museum print displays produced.  We'll present a fully functional demo of an exhibit to the Board this summer and we are hoping to roll out the system to a half dozen exhibits this coming winter and work from there.  We're mostly volunteer based so keeping things as simple, inexpensive and resilient as possible is important.  The local Node-Red instance will monitor the GPIOs which will act as a tactile UI with physical objects to interact with and send those events via MQTT to a central instance of Node-Red running on a server.  The server, again via MQTT sends back 'instructions' to the client instance of Node-Red to control GPIOs and through a websocket connection on the local client to interact with the browser to control content on the screen as well as audio.  Lots of pieces still in the air but it is coming together nicely.


Does anyone know why it isn't possible to change the subscription of an MQTT input node?
 
Thanks,
Ian

[Nicholas O'Leary]

HI Ian,

> Does anyone know why it isn't possible to change the subscription of an MQTT input node?

It isn't possible because it isn't a use case that we've chosen to address so far. There are a number of technical ways it could be done, the question is what is the right way for node-red to do it.

It also depends on what the actual use case is:

 1. an mqtt node that can receive its configuration from some external source at start-up, but not change whilst running.

 2. an mqtt node that can, at runtime, change what topic it is subscribed to.

As Dave says, we solve #1 today in real uses by templating the flow file and updating it at boot time before running node-red. An alternative solution would be to modify the node implementation to pick up its topic from the settings file. However, that is a slippery slope - why just do that with the MQTT node... why not other properties of other nodes? And you still need to update the settings file before running node-red, so on balance, is a worse option than just updating the flow file.

#2 is a bigger change. The only way to modify a node's state at runtime is to send it a message as part of a flow. That means we would need an MQTT node that sits in the middle of a flow; messages it receives set the topic (triggering an unsubscribe/subscribe sequence) and then terminate. The node then emits the MQTT messages it receives. 

 Hans - what broker's support that sort of topic manipulation? Not any that I know of... but the list is growing faster than I can keep up with.

Nick

--
http://nodered.org
---
You received this message because you are subscribed to the Google Groups "Node-RED" group.
To unsubscribe from this group and stop receiving emails from it, send an email to node-red+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Dave C-J]

The other other way is to do your option b) - a custom node... and it's wouldn't have to be "that" custom... if your scenario is Nick's use case 1)...  ie configure once at start...    Basically this would do the same as the external customise the flow step but as part of the node startup step... - ie - when the node is initialised - go look at some property of the system and prepend/append/modify the actual specified topic. 

However I think an extra line in your bash script may be easier :-)

    perl -pe "s/MY_ID/$HASH/" -i /home/pi/.node-red/myflow.json

sort of thing....

[Mark Setrem]

+1 for an inflow MQTT node.

As per twitter conv recently, a MQTT node that gets triggered in flow subscribes to a topic with the Retain-Flag set would simplify my setup.

Currently I have 1 flow that subscribes to MQTT topic and saves it to global context, for the second flow to pick up when its run.

[Nicholas O'Leary]

Mark, in your use case, would you expect it disconnect after receiving that retained message, or stay connected for any subsequent message?

[Hans Jespersen]

I know that the Solace Message Broker supports this feature (called "on behalf of" subscriptions because it allows a client or admin app to subscribe to topics on behalf of another client). I have seen this pattern supported in other enterprise message brokers like TIBCO Rendezvous and EMS but they do not support MQTT. Really this is an application of the general pattern called "Control Bus" in the EIP book ( see http://www.eaipatterns.com/ControlBus.html). I haven't yet used a lot of MQTT brokers other than Solace (my employer for full disclosure), Mosquitto, and Mosca but this kind of dynamic, per connection, topic manipulation is absolutely supported by the MQTT spec. The whole concept of the MQTT "clean session" vs a durable subscription is meant to control the retention or purging of each clients subscription state. I think most MQTT brokers have special $SYS topics that allow monitoring and control of the broker but these administrative topics have not yet been standardized so this kind of control is unfortunately a product specific capability at the moment. 

Sorry to take a deep dive into messaging internals but I think this is a general use case that should be supported easily on either the client or the MQTT broker. I can see a lot of use cases where thousands of identically configured "things" all need to connect back to a central point and recieve unique streams of data. 

-hans

On Friday, March 13, 2015 at 3:24:48 AM UTC-7, Nicholas O'Leary wrote:

[...]

[Mark Setrem]

Yes, disconnect.

[Julian Knight]

And +2

I think that a 3rd mqtt node might be best? And in keeping with the node-red way, surely the topic should be taken as the mqtt topic to subscribe to?


With NR on a Pi, it should be trivial to use the file read node to read the mac address from /sys

    /sys/class/net/eth0/address

contains the mac address of the eth0 link for example, wlan0 for the first wi-fi connection.

You should be able to use that as a lookup key to your database, retrieve the topic and set it. Job done.

[skavan]

Did anything come of this?

There are two mqtt related additions that I would love to see:

1. dynamic subscription - just like the current mqtt input node, but with an input node. when a message with a specified topic arrives on the input, the mqtt unsubscribes from its current subscription and re-subscribes to the specified topic.
2. this one may be already implicit in the implementation, but rather than have mqtt nodes in every flow, it would seem to be more efficient in many use cases, to have a global subscription -- and then child nodes that subscribe to the global subscription within a flow (kinda like a split function for mqtt messages). maybe this is redundant -- but I subscribe to the same mqtt topic in multiple flows - and I wonder if its creating redundant traffic.

the first is way more useful than the second...

s.