General question about Node-RED capabilities

[Bryan Creel]

I've been a DIY home automation hobbyist for a while now. I wrote a C# rules engine some years ago that does a ton of stuff for me. Basically it takes an XML config file, creates stateful objects based on the nodes and their attributes (representing physical or virtual devices) and exposes the objects to a rules system in hosted JavaScript. Events generated by the objects can trigger JS functions, which can interrogate any other object before (maybe) making changes. These objects include things like Insteon devices, Ubiquiti mPower plugs, weather, users (occupancy, etc), IP devices, and so on. Since then I've really fallen in love with Node.js and JavaScript so when I saw Node-RED it got me thinking whether the time had come to migrate.

My question is, would this be a reasonable use-case for Node-RED? I know I would need to write a number of nodes for it. Something I wasn't sure about from playing around is how this type of example would be handled:

When my garage motion sensor is triggered, a JavaScript handler is called. Before that handler can decide what action to take, it needs to check the state of several other objects, such as whether my wife and I are home, the current state of the garage door, the time, and the sunrise/sunset for the current day.

What is the proper way to have stateful objects in Node-RED that can be accessed from rules? I could imagine putting them in global state but that feels wrong to me. Would they need to be managed in a node.js module placed on the global object using the config file? Is this type of thing beyond the goals of Node-RED?

I'm still in the consideration stage about this migration, so any input would be really helpful.

[Colin Law]

Hi Bryan
Absolutely this is the sort of thing node-red excels at. Function
nodes can be stateful by saving stuff in the node context, so your
garage motion sensor code can remember the state of the garage door
and so on. I recommend using MQTT in conjunction with node red for
managing states.

Colin

> --
> http://nodered.org
>
> Join us on Slack to continue the conversation: http://nodered.org/slack
> ---
> You received this message because you are subscribed to the Google Groups
> "Node-RED" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to node-red+u...@googlegroups.com.
> To post to this group, send email to node...@googlegroups.com.
> Visit this group at https://groups.google.com/group/node-red.
> To view this discussion on the web, visit
> https://groups.google.com/d/msgid/node-red/10816d8d-423d-4865-98b4-2e27122010fc%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Julian Knight]

You beat me to it Colin almost word for word!

I'd also say that the global vars problem isn't so excessive in NR anyway as, generally, only your flows should be setting global vars.

[Bryan Creel]

Thanks for the quick replies!

However, it's not so much about one rule remembering its own context - in my current system it's about having the garage door object itself manage its own state and then that state being available to any rules that need it. Or maybe I'm thinking about this the wrong way. Let's take an example of how I think it might work in an MQTT context:

- Update comes in through MQTT saying that garage door is now in the "open" state.

- This triggers some rule in Node-RED which may or may not do something, but it needs to make the current state of the garage door available to other rules so they aren't continually having to query it.

Another example would be mPower devices which transmit their state through websocket messages. How would one share that state so that another rule could access it? What is a good way to do this type of thing for a large number of devices?

I hope it makes sense what I'm trying to understand; I'm just trying to figure out how to see it from a Node-RED context.

[Julian Knight]

Hi. MQTT is certainly a good route for this. MQTT can also retain the last msg so it survives restarts and you can subscribe to a topic as many times as you like so multiple flows are not a problem.

The change of MQTT topic will trigger a new flow from an MQTT in node.

With the websocket example. You would use a websocket listener node to get the update and then attach that to an MQTT output so everything goes into a standard structure.

[Julian Knight]

This is a bit dated but may give you some ideas on how it could all work:

https://github.com/TotallyInformation/Home-Automation-Hub-Design

[Colin Law]

On 2 May 2017 at 16:49, Bryan Creel <scyta...@gmail.com> wrote:
> Thanks for the quick replies!
>
> However, it's not so much about one rule remembering its own context - in my
> current system it's about having the garage door object itself manage its
> own state and then that state being available to any rules that need it. Or
> maybe I'm thinking about this the wrong way. Let's take an example of how I
> think it might work in an MQTT context:
>
> - Update comes in through MQTT saying that garage door is now in the "open"
> state.
> - This triggers some rule in Node-RED which may or may not do something, but
> it needs to make the current state of the garage door available to other
> rules so they aren't continually having to query it.

Feed the current door state from MQTT into any nodes that need to know
it and in those nodes save it in the node context. Then in those nodes
the current state will always be available in the context. No need to
query it.

Colin

>
> Another example would be mPower devices which transmit their state through
> websocket messages. How would one share that state so that another rule
> could access it? What is a good way to do this type of thing for a large
> number of devices?
>
> I hope it makes sense what I'm trying to understand; I'm just trying to
> figure out how to see it from a Node-RED context.
>

> --
> http://nodered.org
>
> Join us on Slack to continue the conversation: http://nodered.org/slack
> ---
> You received this message because you are subscribed to the Google Groups
> "Node-RED" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to node-red+u...@googlegroups.com.
> To post to this group, send email to node...@googlegroups.com.
> Visit this group at https://groups.google.com/group/node-red.
> To view this discussion on the web, visit

> https://groups.google.com/d/msgid/node-red/d49a4057-a25b-4581-ab24-8d1828374fd5%40googlegroups.com.

[Bryan Creel]

Okay, this has been really instructive! I did some more reading on MQTT and went through Julian's flows (very nice, thanks for publishing them!). Let me try and lay out a possible approach for my example in the methodology you guys are using, where I think you are essentially using JSON in MQTT in a similar way to how I'm currently using stateful objects:

- Add an MQTT broker to my setup that retains messages.

- Refactor my C# Insteon objects (since they work pretty well) to publish and subscribe to MQTT topics. Their events become pubs and they listen to subs for commands that result in sending the Insteon commands to the actual devices. Do something similar for other device types, either in Node-RED or out of it if I keep something in C#.

- In my example, the flow would subscribe to several MQTT topics: the garage door state, the motion sensor, user occupancy. The motion sensor is the only one that would actually initiate a change - the others would simply store those values in context. 

- When the motion sensor sub is triggered, it can examine context for the other values. Because of the pub/sub model, the MQTT broker will handle getting the subs out to all the flows that need it without anything needing to directly query the hardware.

Does that sound workable?

Finally, as an alternative: I really like the idea of MQTT but I'm just wondering if it would be workable to do something like this for a family of devices like Insteon:

- Create my own node. It would have a number of config nodes, one for each Insteon object.

- All of the config nodes would share a single Insteon driver that actually manages the serial communications and messaging. 

- Then I could have input nodes that would deliver events from an individual device (config node) and output nodes that would actuate a device.

That approach would still need context to store state, so I think the MQTT way is actually better. I'm just wondering if it would also work.

Thanks again for taking the time to respond; it has been really helpful.

[Julian Knight]

Always happy to share ;-)

Couple of things. 

Your C# stuff is similar to the way a lot of us use ESP8266 devices (wi-fi enabled microprocessors) which are usually programmed using C++ in a similar (if not identical) way to an Arduino. I know there is a .net library for MQTT as I tried to create an MQTT client as a universal windows app last year (and failed dismally, I am certainly not a C-anything programmer!), so that may work for you.

In regard to the variables, you will probably need flow vars rather than context. Flow vars are global to a specific tab in your admin UI. Context vars only exist for the node that creates them.

Trying to do this in your own node is going to be painful, a lot of work and will get you little reward I suspect. As you begin to use MQTT, you will likely find many other benefits just as lots of us have gone through that learning experience. If you are familiar with enterprise service buses (ESB) and similar message queue services, you may "get" the benefits rather more quickly. MQTT enables you to disaggregate your devices and processing, enabling future changes a lot easier and more quickly. So certainly you could get it to work but I doubt it would be worth it.

[Colin Law]

On 2 May 2017 at 20:18, Bryan Creel <scyta...@gmail.com> wrote:
> Okay, this has been really instructive! I did some more reading on MQTT and
> went through Julian's flows (very nice, thanks for publishing them!). Let me
> try and lay out a possible approach for my example in the methodology you
> guys are using, where I think you are essentially using JSON in MQTT in a
> similar way to how I'm currently using stateful objects:

Not sure exactly what you mean by using JSON, usually I publish simple
values to MQTT and use the addressing scheme to structure the data
where necessary. So you might have home/garage/door_position "open" or
"closed" for example.

Colin

[Colin Law]

On 2 May 2017 at 21:06, Julian Knight <j.kni...@gmail.com> wrote:
> ...

> In regard to the variables, you will probably need flow vars rather than
> context. Flow vars are global to a specific tab in your admin UI. Context
> vars only exist for the node that creates them.

I have a slightly different philosophical approach to Julian on this
one. I virtually always feed the required values into any node that
needs them and use node context, rather than using flow context. I
like it to be obvious what is used where. It does add extra wires into
the flow though.

Colin

[Bryan Creel]

I think I have a decent understanding of the approach you guys are using and what my options are. Thanks again for patiently explaining things to me; it has been really helpful for comparing the Node-RED and MQTT ideas against my current setup.

[Julian Knight]

I have a mix. I use JSON with some standardised content across my various sensors but then republish with simple values. That's to allow multi-stage flows with minimal effort and to provide for a number of different uses. Starts with fairly complex (for MQTT) JSON and ends with single value topics.

[Colin Law]

Yes, I can see that could make sense. Probably my requirements have
not been complex enough to need more complex data structures in MQTT.
Thanks Julian.

Colin

> --
> http://nodered.org
>
> Join us on Slack to continue the conversation: http://nodered.org/slack
> ---
> You received this message because you are subscribed to the Google Groups
> "Node-RED" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to node-red+u...@googlegroups.com.
> To post to this group, send email to node...@googlegroups.com.
> Visit this group at https://groups.google.com/group/node-red.
> To view this discussion on the web, visit

> https://groups.google.com/d/msgid/node-red/91cb94a0-34df-43b5-b107-c120661bf29c%40googlegroups.com.

[Jeff Tapia]

Brian,

I would be very intersted in your progress.

I tried using node-red to directly query my Hub2014 (similar to the way you would the PLM) but I got bogged down with deciphering hex Standard Messages (0x50), Extended Messages (0x51), etc. and was spinning my wheels.

I am currently using the Insteon web api http://docs.insteon.apiary.io/#

which in turn uses the Hub 2014 and I can query all my devices, get individual status, device details (like schedule, LED backlight brightness, etc). and I'm getting close but I can't connect to what they call the Event Stream (or Servicer-Side Events) which I believe should allow me to keep up to date with device changes rather than constantly polling every single device - but I can't get that to work. I don't know how to handle the associated HTTP Redirect nor the Event Streaming.

I am also currently successfully using NR moving data between an online service (online.wonderware.com) and MQTT and Rainforest Power Meter (Smartmeter) and Ecobee Thermostat with occupancy, furnace status, and all kinds of other data.

The insteon has remained a challenge for me.

I'd welcome any help or insight you may have on Insteon. Thx.

[Bryan Creel]

Jeff, I'd be happy to help. Do you need documentation on the format of the messages? I have some PDFs with the protocol spec. Or I could send you my data structure declarations - if you are familiar with C# then they might make even more sense than the PDFs. Or if you have specific questions just shoot me those and I'll do my best to answer.

[Jeff Tapia]

Thanks Bryan - I have all the PDF / documentation.

I'm not a C# guy but, hey I'm not a javascript or node-red guy either, but I'm learning.

Could you send the data structure you mention?

I'm wondering if I can convert the concepts to node-red/javascript.

[Bryan Creel]

I think this will make sense to you - I like to write comments.

[Jeff Tapia]

Thank you Bryan - I think I'm going to learn a lot.!

Again, Thx.

[Jeff Tapia]

Bryan, I appreciate you sharing your code a few months back but I crashed and burned trying to understand how to translate it to node-red.

My brain was unable to rise to the occasion.

I was wondering if you have done anything since to more directly integrate insteon to node-red through new insteon nodes (I don't see anything on flows.nodered.org but was hoping you did something that you haven't yet shared).

[Bryan Creel]

Jeff, I've pretty much converted my existing C# rules engine into an MQTT client. It handles all of my Insteon devices and and exposes status, commands, and logs through MQTT and I've moved all of the rules into node-red. That's working well for me so I don't currently have any plans to rewrite the Insteon logic in NR.

--

http://nodered.org
 
Join us on Slack to continue the conversation: http://nodered.org/slack
---

You received this message because you are subscribed to a topic in the Google Groups "Node-RED" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/node-red/UlPQ_2UmdIk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to node-red+u...@googlegroups.com.

To post to this group, send email to node...@googlegroups.com.
Visit this group at https://groups.google.com/group/node-red.

To view this discussion on the web, visit https://groups.google.com/d/msgid/node-red/88b61b91-2f12-4379-a972-703a35ac4a97%40googlegroups.com.

[Jeff Tapia]

@Tarun Sharma,

Regarding your PM about sending data to Wonderware online...

I am looking for a SIMPLE version and having trouble finding it.

I currently pump a bit of data (perhaps something every 10 seconds or so).

Everything was working find but at some point I thought "this is not very efficient" since some things don't change much, I don't care about 0.01 degree F changes, I don't need to send data if it doesn't change, I should probably queue data and send all of it a maximum of once a minute (timestamped appropriately).

So I currently create a queue in node red that consists of a timestamp and anything received at that timestamp, along with it's tagname (in MQTT-like format for no reason other than I ALSO send to an MQTT broker) and it's value.

Then once every minute a different "thread" reads the queue, sends the data and clears the queue.

This works great too and makes me "feel better" because it is efficient - but it is quite a bit more complex.

The reason I did that was that:

I started adding around 15 environment settings (outdoor temp, 4 different rooms of indoor temp, humidity, wind speed and dir, precip, pressure)

I also added a power meter that reads every 5 seconds instantaneous and totalized kwh

Then added a couple other temperatures, battery levels, light levels, and attempted the insteon stuff (had some things working but the inefficiency of polling devices caused me to give up.

Anyway, I implemented the queue because, although I had no problems when I was sending data pretty much every 5 seconds, it seemed inefficient.

Finally, I will try to find my original, very simple code to send things to online.wonderware.com and post but I'm not having any luck.

If I post the queue thing I think it will take more explaining than I can afford (or remember, it is not well documented and I would have to re-learn).

[Jeff Tapia]

Here is the node red function that formats my queue to json:

/*20171208 

input message should contain:

msg.timestamp ie 1514764800000 is an epoch milliseconds timestamp representing 2018.01.01. at 00:00:00.000

 (if msg.timestamp does not exist then "now" will be used)

either single value-based payload:

    msg.payload ie 72.1 or any string or number representing the value for the topic

    msg.topic ie diningroom/temperature

or multi-variable object-based payload containing property : value pairs (use . as delimiter)

    https://stackoverflow.com/questions/1661197/what-characters-are-valid-for-javascript-variable-names

    msg.payload ie {basement$temperature : 72.1, diningroom$temperature: 73.2}

    msg.topic ie wisconsin/summerhome becomes the prefix for each item, creating final items such as wisconsin/summerhome/basement/temperature

*/

var wwQueue = flow.get('wwQueue') ||{"data":[]};

//var dt = new Date(); // defaults to now //var dtstring = dt.toISOString();

//use timestamp if it exists otherwise default to now

if ((typeof msg.timestamp !== 'undefined')){dtstring = new Date(msg.timestamp).toISOString();} 

else {dtstring = new Date().toISOString();}

i = 0;

while (

    (typeof wwQueue.data !== 'undefined') &&

    (typeof wwQueue.data[i] !== 'undefined') &&

    (typeof wwQueue.data[i].dateTime !== 'undefined') &&

    (new Date(wwQueue.data[i].dateTime).getTime() > new Date(dtstring).getTime()) //while we find newer dates keep looking 

) {i++;}

var obj = {};

if (

    (typeof wwQueue.data !== 'undefined')&&

    (typeof wwQueue.data[i] !== 'undefined')&&

    (typeof wwQueue.data[i].dateTime !== 'undefined')&&

    (new Date(wwQueue.data[i].dateTime).getTime() == new Date(dtstring).getTime()) //if exact date found then we need to handle existing

) {

    obj = wwQueue.data.splice(i,1)[0];

} else {

    obj.dateTime = dtstring;

}

if (typeof msg.payload !== "object"){

    var tag = msg.topic.replace(/\//g,"."); //globally replace / with .

    obj[tag]=msg.payload;//add new tag and value to this dateTime record

} else {

    var prefix = msg.topic.replace(/\//g,".") + ".";

    var tag = "";

    for(var propertyName in msg.payload) {

        tag = prefix  + propertyName;//propertyName.replace(/\$/g,".") was going to use $ delimiter but dot "." works 

        obj[tag] = msg.payload[propertyName];

//        node.error(JSON.stringify(obj,null, 4));

    }

}

wwQueue.data.splice(i,0,obj); //add new, full dateTime record back to all record data in the right spot

flow.set('wwQueue',wwQueue);

msg.topic = "tapia/home";

msg.payload = wwQueue;

return msg;

/*

example from online.wonderware.com

{

  "data": [

    {

      "dateTime": "2017-03-20T12:21:12.199Z",

      "Reactor3.Level": 1.51860072870498,

      "Reactor3.Temp": 27.1360543141452,

      "Reactor3.InletValve": 0,

      "Reactor3.Step": 7,

      "Line3.Units": 1518,

      "Line3.Product": "BRN"

    }

  ]

}

*/

THen the code to turn it to json:

var wwQueue = flow.get('wwQueue') || null;

flow.set('wwQueue',null); //delete processed json from global context

msg.payload = wwQueue;

if (!msg.payload) {

    msg = null;

}

//msg.payload = JSON.stringify(wwQueue,null,"\t");

return msg;

*** AND ***

msg.payload = JSON.stringify(msg.payload,null,"\t");

return msg;

** AND ** then the message is passed through a secure Bearer token node where it stores my private authorization info

*** AND *** then I form the http request:

// configure http (headers,url,method)

msg.headers = {

//    "Authorization" : msg.authorization,

    "Content-type" : "application/json"

};

msg.url = "https://online.wonderware.com/apis/upload/datasource";

msg.method = "POST";

msg.headers.Authorization = msg.Authorization;  //copy Authorization to be under headers

delete msg.Authorization;                       // and remove from old location

return msg;

/*

The following works (replace token with real one):

msg.headers = {

    "Authorization" : "Bearer aThisIsMyFakeTokenYourWillBeLonger-AjsTIEvRKKQ",

    "Content-type" : "application/json"

    };

msg.url = "https://online.wonderware.com/apis/upload/datasource";

msg.method = "POST";

*/

*** AND *** Send to https request node

[Jeff Tapia]

Bryan,

It sounds like you have a reliable C# app "converting" Insteon to MQTT.

Do you see any way that I could compile/run that app on my raspberry PI (ARM processor) that is currently my Node-RED node and MQTT Broker?

I found a Python library that delivers the full, unparsed insteon payload to MQTT - but that still takes a bit of work to unparse, requires a config file with link definitions, doesn't support any configuration or linking, etc.

It seems like you have figured this out and my best bet would be to somehow run your code on RPI (or perhaps a dedicated PC if x86/x64/Windows is required).

Today, everything works in Node RED in my house (a bunch of MQTT, Ecobee, Power meters, weather, online.wonderware.com, etc.) except Insteon.

I am revisiting trying to breath life back into my $5k insteon "investment" - which has languished.

It sounds like you have evolved the code that you shared with me last year to become an Insteon<>MQTT bridge with better parsing and features than anything I can find.

Might I be able to reuse your work?

Raspberry? or should I just move to Windows PC?

[Bryan Creel]

Jeff, I don't think my project could be compiled for PI. 

[Jeff Tapia]

I didn't think so either - looks like Dot Net dependancies (perhaps C# is microsoft only?), anyway...

I found this Python project that seems to work:

https://github.com/TD22057/insteon-mqtt

I need to figure out how to add features like battery and light level from motion sensors, FastOn/FastOff, etc. But it is a good start.