Get data from MQTT broker usining inject node

[Greg EVA]

Hello,

I am working on getting to know Node-RED and trying to build a little energy management application and am having trouble wrapping my head around how this whole event driven flow approach works.

What I want to do is set some periodic cron type triggers (hourly, daily, monthly) which will start a flow, allow me to grab some data from an MQTT broker, and then store that somewhere.

Currently I am using the MQTT input node to subscribe to topic updates, but I can't figure out how to do a point in time read.

Help please!

Cheers,

Greg

[ryan...@gmail.com]

Hi Greg,

MQTT uses a publish-subscribe messaging pattern, so your subscribing MQTT input node will fire and start the connected flow every time it receives a topic update, immediately. If this is too frequent you can use the delay nodes to throw some samples away and only progress the flow after a suitable period.

Ryan

[Greg EVA]

Hi Ryan,

Thanks for your advice. The rate limiting delay is not something I was aware of.

The frequency is however not the issue.

I am trying to have the flows input trigger be periodic using the inject node with a from scheduled expression, say every hour. Then I want to be able to grab the data from an MQTT topic at that moment.... Not when it eventually updates.

The use case would be the same for an HTTP get.... However I gather the implementations are different due to the pub/sub approach of MQTT.

Cheers,

Greg

--
http://nodered.org
---
You received this message because you are subscribed to a topic in the Google Groups "Node-RED" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/node-red/SZd1M3nU3gM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to node-red+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Mark Setrem]

One way of approaching this would be to use a function node to write the value from MQTT to a variable named context.global.whatever.
This would then be updated everytime a new value was broadcast over MQTT.

Then in a separate flow you can use a timed trigger linked to a function node to pick up the value from context.global.whatever

Take a look at http://nodered.org/docs/writing-functions.html to find out details of the context.global

[Julian Knight]

I'd agree with that.

This is an area I'd like to see some improvements on since this would seem to be a common requirement - setting/getting a global var. Seems a shame we have to resort to a function node just to do that.

Developing with Node-Red (and Node.JS in general) often requires some lateral thinking! One of the reasons it is fun.

[Nicholas O'Leary]

Julian, you know we have the item to make context more widely used by nodes. It has come up a number of times, you've commented on the issue itself. We will get there.

Greg, the only way a 'point in time' read would work with MQTT is if you are publishing every message with the retained flag set. The broker stores the most recent message with the retained flag set for each topic. Our MQTT nodes do not enable this sort of dynamic connect & subscribe pattern. Mark's suggested approach is the way to do it with the nodes we have.

Nick

--
http://nodered.org
---
You received this message because you are subscribed to the Google Groups "Node-RED" group.
To unsubscribe from this group and stop receiving emails from it, send an email to node-red+u...@googlegroups.com.

[Greg EVA]

@Mark and @Julian: thanks for this clarification.  I had resorted to this global context approach for the Meter ID which never changes.... and then I started adding a couple for the meter indexes to be able to do some basic math on them when various topics triggered events.  I just didn't want to run off on a global variable tangent if there was a better approach to this sort of thing.  (I am reassured that I am at least picking things up reasonably normally).

@Nick: I am using the retained flag with persistent storage at the broker.  My thinking is that upon Node-RED startup, it would fire a flow which could say load the electric meter ID into a global context variable; which I could then use everywhere else, knowing that the flow would be retriggered and the topic updated should someone say, replace my electric meter.  OK... perhaps not the best example... ;-P Energy consumption tarification period indexes are a better example as they may or may not change, but I would like to know what they are at particular times.

Thanks for the help!

Cheers,

Greg

[Julian Knight]

I'm actually having a go at creating a node specifically to easily set/get global variables. Not quite there as I'm struggling to work out exactly how they work at a node level. I think they have a getter/setter in

RED.settings.functionGlobalContext

but I didn't quite get the time to work it out last night.

The node UI looks like this (dratted Google Groups web sticks it in the wrong place - might appear at the top of this post!):

[Greg EVA]

Nice!!!  I think this will be a good addition to the node library as this seems to help cross-flow data sharing.

[Nicholas O'Leary]

Julian, by all means play, but please consider listening to what we already have planned and collaborate with us rather than go your own way.

This node will be completely redundant very soon and in fact will be unhelpful as it will not factor in the changes to context we have planned.

Nick

On Mon, 25 May 2015 12:24 Greg EVA <ge...@ge-volution.eu> wrote:

Nice!!!  I think this will be a good addition to the node library as this seems to help cross-flow data sharing.

--

[Julian Knight]

Not a problem Nick. Learning as I go. I now know a lot more about how global and context work so it is a useful exercise and hasn't taken long. I can see that the set/get context part isn't worth me trying to deal with anyway & knowing how to set/get global vars from a node is useful.

Happy to contribute this or other nodes or just some code to core if anyone thinks any of it worth while. Corporate IT is my thing so programming is "play" for me :) I'm just keen for NR to get some simpler ways of working for people without JavaScript skills.

[Dave C-J]

Going back to the original question... part of the point of being event driven (and indeed part of philosophy behind why MQTT was invented) is that polling for things is inefficient if you have lots of endpoints that only need to update occasionally. IE there is no point asking 10,000 sensors for data every minute if 9999 of them haven't changed. Hence in MQTT you subscribe (in theory once) to the broker - and then when a client publishes something that matches your subscription you get it straight away. And if you don't get anything then you can assume it hasn't changed. Of course this means that if you come late to the party (subscribe after last value changed) then you may not see anything for while... Clients can send a value with the Retain flag set - which then means hold onto this for any latecomers... but it is an old value.

This suits the node.js event model also in that it can happily sit and wait for things to happen - it is this asynchronism that takes a bit of getting used to. So asking for something every minute (also of course by an event every minute..) then has to mesh with the "I'll deal with it when it arrives" world.... hence the need for a buffer or something to hold state. This could be the broker (as previously mentioned using retained messages) - or a lcoal database (if you need more than one value) - or some form of local "in-memory" store that we call context. Node-RED has local context within a single function or global context across the whole runtime. You could use either depending how you design your flows (obviously to use local within a single function both "paths" of your flow would need to go through the same function - using something like the topic property to tell apart actual data arriving (to be stored) and injects (to read out the stored data). This would then not "pollute" or leak to the global space. Using global context allows the two paths to be separate but be able to share the data - but any other function could also read it... which may or may not be desirable (typically for small flows it won't be an issue).

As Nick has said we are looking at how best to expose some of this context more widely - but we also want to be a bit careful... the whole point of a wiring type tool is to show what is connected to what - and what can affect what - and just allowing things to talk globally (ie not through wires drawn on the canvas) may make things more confusing rather than less. Hence why currently only functions support it - so the user is explicitly acting to use it.

[Greg EVA]

As Nick has said we are looking at how best to expose some of this context more widely - but we also want to be a bit careful... the whole point of a wiring type tool is to show what is connected to what - and what can affect what - and just allowing things to talk globally (ie not through wires drawn on the canvas) may make things more confusing rather than less. Hence why currently only functions support it - so the user is explicitly acting to use it.

This is just an idea for you guys to ponder... but what if there was a node.... in my brain I called it a micro-node as it may appear smaller in the workspace.... which would get an input and set a global context type value.  Then you'd take the output; once or many times, and wire it to a connection on the top (left) of another node (like function node), and then you'd use a node local context to reference that value.... and the flow execution would "pull" the current value from the "global micro-node".  You could then wire certain values up to many nodes, see their connections, wire/unwire, etc.  It would essentially be like passing variables to a function.

[Nicholas O'Leary]

Hi,

If I understand your suggestion correctly, you would have some nodes pulling values across wires rather than the normal push? We don't want to confuse the model by doing that.

The use of a shared context solves this problem in a very straight forward way. Once the Change and Switch nodes are able to access Context, which is the plan, it all becomes much easier to achieve without writing code in function blocks.

Nick

On Mon, 25 May 2015 15:24 Greg EVA <ge...@ge-volution.eu> wrote:

As Nick has said we are looking at how best to expose some of this context more widely - but we also want to be a bit careful... the whole point of a wiring type tool is to show what is connected to what - and what can affect what - and just allowing things to talk globally (ie not through wires drawn on the canvas) may make things more confusing rather than less. Hence why currently only functions support it - so the user is explicitly acting to use it.

This is just an idea for you guys to ponder... but what if there was a node.... in my brain I called it a micro-node as it may appear smaller in the workspace.... which would get an input and set a global context type value.  Then you'd take the output; once or many times, and wire it to a connection on the top (left) of another node (like function node), and then you'd use a node local context to reference that value.... and the flow execution would "pull" the current value from the "global micro-node".  You could then wire certain values up to many nodes, see their connections, wire/unwire, etc.  It would essentially be like passing variables to a function.

--

[Greg EVA]

What I was thinking of would be comparable to water pipes connected together in a line, with one at the end going down a bit.  The water would normally flow though the pipes and out the end.

If you put a short pipe stub (open at the top) vertically, the flow of water through the pipes would cause suction in this little chimney, and if there was water sitting in a reservoir there, it would be pulled into the flow.  This was what I was thinking to visually provide wiring and context for global values into the flow.  The event flow would in effect pull the global context value as it passed through the node.

... not sure if this is what you understood... but that is in essence what I meant.

                                                                           

Greg Eva

ge...@ge-volution.eu

www.gE-volution.eu

France : +33 4 81 68 09 64

Canada : +1 226 476 1192

iNum : +833 51 00 09 90 28 81

--
http://nodered.org
---
You received this message because you are subscribed to a topic in the Google Groups "Node-RED" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/node-red/SZd1M3nU3gM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to node-red+u...@googlegroups.com.

[Greg EVA]

And we're back... after having just read this other thread about Julian's red wine, and the upcoming plans for the Change node... here is an adapted idea to bridge the visual idea about the plumbing and the Change node.

What if the change node had "flying leads" that came up from the top and down from the bottom of the node.  They could be short, and say end in an arrow... indicating that the wire goes somewhere.   Hovering over it could show the value it is refering to.  You would then quickly be able to look at a change node and see that it grabs a global context variable, and sets three of them.  The leads would obviously be obviously be drawn based on how the change node was configured inside.