Simplifying a Function (jsonata?)

skavan

unread,

Feb 14, 2018, 3:41:22 PM2/14/18

to Node-RED

Hi,

I have successfully written a tcp flow creator that takes a "device definition file" as input and then builds a flow to handle sending and receiving messages to and from that device.

My code is messy and large (I'm not a native js developer). I want to simplify it, before releasing it to the wild.

I was wondering whether:

a) jsonata can be used inside a function and

b) whether jsonata is even the way to go.

Consider this use case:

A message arrives.

message.payload ="IIS:HD1"

I have a flow object, deviceMap that hold the following json (simplified):

{
    "projector": {
        "panasonic_ae":{
            "commands": {
                "power": {"deviceCommand": "P$","deviceStatus": "(?:^P)(.{2}$)","deviceRequestStatus": "QPW", "category":"main", "sph":"end"},
                "power status": {"deviceCommand": "QPW","deviceStatus": "(^000$|^001$)","deviceRequestStatus": "QPW", "category":"main", "topic": "power", "requestReply": "power", "sph":"end"},
                "input": {"deviceCommand": "IIS:$","deviceStatus": "(?:^IIS:)(.{3})","deviceRequestStatus": "QIN", "category":"main", "sph":"end"},
                "input status": {"deviceCommand": "QIN","deviceStatus": "(^.{3}$)","deviceRequestStatus": "QIN", "category":"main", "topic": "input", "requestReply": ["input","input status"], "sph":"end", "link": "input"},
                "picture mode": {"deviceCommand": "VPM:$","deviceStatus": "(?:^VPM:)(.{3})","deviceRequestStatus": "QPM", "category":"main", "sph":"end"},
                "dpad": {"deviceCommand": "$","deviceStatus": "","deviceRequestStatus": "", "category":"navigation", "link":"navigation"},
                "menu": {"deviceCommand": "$","deviceStatus": "","deviceRequestStatus": "", "category":"navigation"}
            }
		}
	}
}

I wish to iterate through the "commands" objects and find which one or more commands.deviceStatus regex matches the message payload.

For a match, we create a new message, or append an array of messages to this message, that holds the matched Command.

In this case "command":"input", "value":"HD1".

For bonus points, if the matching object has a "sph":"end", then we stop processing the array of commands.

My clunky code (which works) is as follows:

/**
 * given a message object, look up all the potential commands and iterate through the deviceStatus regex
 * so that we can figure out what sort of message we have received (message identification)
 * we may have more than one match, so we create an array of returned message(s) or a zero length array
 * we also check for a mqtt clear topic command and pass it through as the clear flag. the matching command is placed in the node property
 * @param {obj} thisMessage 
 * @returns {array} outputMsgs 
 */
function indentifyStatusMessage(thisMessage, deviceMap){
    let commands = getChildObjectByPath("commands", deviceMap);     //get all the commands
    let outputMsgs = [];
    for (let item in commands){                                     // iterate through them looking for a regex match
        let command = commands[item];                               

        let match = getRegexMatch(command.deviceStatus, thisMessage.payload) || [];
        if (match.length > 1){                                      // a legit match returns two or three elements
            let outItem;
            let category = makeSubTopic(command.category, null, command.subTopic);      // build the topic/subtopic string
            if (command.requestReply || false){                     // If this is a special message that is a reply to a request/command then 
                let found = false;                                  // check the last request/command (saved in the LAST_COMMAND flow var)
                if (Array.isArray(command.requestReply)){
                    for (let rr of command.requestReply){
                        if (rr === lastCommand){found = true; break;}
                    }
                } else {
                    if (command.requestReply === lastCommand){found = true;}
                }
                if (found === false){continue;}
            }
            // handle unknown status messages -- or not!
            if (category === MQTT_UNKNOWN_ROOT){
                if (!flowSettings.showUnknowns){
                    continue;
                }
            }
            let clear = command.clear || '';                    // pass through the clear flag into the returned outItem
            if (match.length ===2){
                // if there is a topic attribute on the command/status, then use that instead of the "item" name.
                outItem = {"topic":command.topic || item, "value":match[1], "category": category, "clear": clear, "node": commands[item], "valuemap": item};       
            } else {
                // this is for when the regex returns two matches. the first is presumed to be status/topic, the second is value
                outItem = {"topic":match[1], "value":match[2], "category": category, "clear": clear, "node": commands[item], "valuemap": item};       
            }
            if (!(outputMsgs.length>0 && item.toUpperCase().startsWith('UNKNOWN'))){
                // process all messages except when outputmsg.length>0 and its unknown
                outputMsgs.push(outItem);
                if ((command.sph || false) === "end"){
                    // it the command metadata (sph) instructs us to terminate, do so.
                    return outputMsgs;
                }
            }
        }
    }
    if (outputMsgs.length === 0){
        //node.warn("parseStatusMessage processing failure:" + thisMessage);
    }
    return outputMsgs;
}

All thoughts on how to make this cleaner, more efficient and more node-red-ish, much appreciated.

There is a second part to this (which is handling and translating the value half of the message---but let's save that for another day/post....

s.

AIOT MAKER

unread,

Feb 15, 2018, 12:15:28 AM2/15/18

to Node-RED

Hello,

In regards to these points:

a) jsonata can be used inside a function and

b) whether jsonata is even the way to go.

My thoughts:

(a) I don´t think it is possible to use Jsonata inside a function node

(b) It is likely that Jsonata can help you to simplify a little bit your flow

Your code seems to be well written, robust (error manipulation included) and more importantly you state it works. However as your post is a request for comments I provide below some ideas (draft) that could (eventually) be explored.

The core of the problem seems to be expressed in your statement "I wish to iterate through the "commands" objects and find which one or more commands.deviceStatus regex matches the message payload."

What always comes to my mind is how to manipulate the data set to make the problem resolution easier.

I imagined creating a new JSON data structure (without changing the original message) that holds the relevant data : (a) the regular expression (value from key deviceStatus) (b) the keys (seven) from the command object and finally a new boolean element that will say if msg.payload is matched by the regular expression.

This can be done with a single change node but with many rules(most of them relies on Jsonata expressions). I did a testing by creating a new array, named "transformed", like shown below. Then I used a function node to iterate over the array elements and apply the test() method from Javascript RegExp object (https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/RegExp/exec) to test the regex match (perhaps it is what you do in your function getRegexMatch ?).

This is how the code looks like inside the function node. It takes the regex from the JSON and test against the string in msg.payload.

msg.transformed.forEach(function(item){

var re = new RegExp (item[1]);

var result = re.test(msg.payload);

item[2] = result;

});

return msg;

Of course this is half baked idea but it is straightforward and easy to read (few lines of code and extensive use of the change node).

Flow:

[{"id":"1339ae17.104482","type":"inject","z":"e25c4f6.dca52b","name":"Start","topic":"","payload":"","payloadType":"date","repeat":"","crontab":"","once":false,"x":110,"y":140,"wires":[["e1e8b2af.6ae55"]]},{"id":"1e4017c0.4e9f58","type":"debug","z":"e25c4f6.dca52b","name":"","active":true,"console":"false","complete":"true","x":890,"y":140,"wires":[]},{"id":"30b32f31.3ee72","type":"function","z":"e25c4f6.dca52b","name":"Test Match","func":"\nmsg.transformed.forEach(function(item){\nvar re = new RegExp (item[1]);\nvar result = re.test(msg.payload);\nitem[2] = result;\n\n});\nreturn msg;","outputs":1,"noerr":0,"x":750,"y":140,"wires":[["1e4017c0.4e9f58"]]},{"id":"e1e8b2af.6ae55","type":"function","z":"e25c4f6.dca52b","name":"Data Set hard Coded for testing","func":"msg.payload =\"IIS:HD1\";\nmsg.device = {\n \"projector\": {\n \"panasonic_ae\":{\n \"commands\": {\n \"power\": {\"deviceCommand\": \"P$\",\"deviceStatus\": \"(?:^P)(.{2}$)\",\"deviceRequestStatus\": \"QPW\", \"category\":\"main\", \"sph\":\"end\"},\n \"power status\": {\"deviceCommand\": \"QPW\",\"deviceStatus\": \"(^000$|^001$)\",\"deviceRequestStatus\": \"QPW\", \"category\":\"main\", \"topic\": \"power\", \"requestReply\": \"power\", \"sph\":\"end\"},\n \"input\": {\"deviceCommand\": \"IIS:$\",\"deviceStatus\": \"(?:^IIS:)(.{3})\",\"deviceRequestStatus\": \"QIN\", \"category\":\"main\", \"sph\":\"end\"},\n \"input status\": {\"deviceCommand\": \"QIN\",\"deviceStatus\": \"(^.{3}$)\",\"deviceRequestStatus\": \"QIN\", \"category\":\"main\", \"topic\": \"input\", \"requestReply\": [\"input\",\"input status\"], \"sph\":\"end\", \"link\": \"input\"},\n \"picture mode\": {\"deviceCommand\": \"VPM:$\",\"deviceStatus\": \"(?:^VPM:)(.{3})\",\"deviceRequestStatus\": \"QPM\", \"category\":\"main\", \"sph\":\"end\"},\n \"dpad\": {\"deviceCommand\": \"$\",\"deviceStatus\": \"\",\"deviceRequestStatus\": \"\", \"category\":\"navigation\", \"link\":\"navigation\"},\n \"menu\": {\"deviceCommand\": \"$\",\"deviceStatus\": \"\",\"deviceRequestStatus\": \"\", \"category\":\"navigation\"}\n }\n\t\t}\n\t}\n}\nreturn msg;","outputs":1,"noerr":0,"x":310,"y":140,"wires":[["a3499630.db4398"]]},{"id":"a3499630.db4398","type":"change","z":"e25c4f6.dca52b","name":"","rules":[{"t":"set","p":"commands","pt":"msg","to":"**.commands.$keys()","tot":"jsonata"},{"t":"set","p":"status","pt":"msg","to":"**.deviceStatus","tot":"jsonata"},{"t":"set","p":"match","pt":"msg","to":"$map(status, function($v, $i, $a) {\t false\t})","tot":"jsonata"},{"t":"set","p":"transformed","pt":"msg","to":"$zip(commands,status,match)","tot":"jsonata"},{"t":"delete","p":"commands","pt":"msg"},{"t":"delete","p":"status","pt":"msg"},{"t":"delete","p":"match","pt":"msg"}],"action":"","property":"","from":"","to":"","reg":false,"x":560,"y":140,"wires":[["30b32f31.3ee72"]]}]

skavan

unread,

Feb 15, 2018, 2:18:37 AM2/15/18

to Node-RED

Thank you for taking all this time and trouble. Your change node is terrific.

A few questions ---

What does this function mean/do? what is the $vm $i, $a? :

$map(status, function($v, $i, $a) {

false

})

2. Can jsonata apply the regex to the payload (IIS:HD1)?

This has certainly got me thinking! Thanks again.

Rewe Node

unread,

Feb 15, 2018, 5:06:01 AM2/15/18

to Node-RED

Hi,

I have not dealt with your specific problem now. Nor can I say whether jsonata is a suitable way to solve their problem.

I just want to go into a detail.

(a) I don´t think it is possible to use Jsonata inside a function node

That is quite possible. Provided that jsonate is registered in the. node-red/settings. js:

    functionGlobalContext: {
        jsonata:require('jsonata'), 
        // os:require('os'),
        // octalbonescript:require('octalbonescript'),
        // jfive:require("johnny-five"),
        // j5board:require("johnny-five").Board({repl:false})
    },

Here is a simple example (see https://github.com/jsonata-js/jsonata )

Flow:

[{"id":"b1386288.52e03","type":"inject","z":"fc8d3b46.2198","name":"","topic":"","payload":"","payloadType":"date","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":120,"y":820,"wires":[["77ce92.edd1017"]]},{"id":"77ce92.edd1017","type":"function","z":"fc8d3b46.2198","name":"test JSONata","func":"var jsonata = global.get('jsonata'); \n\nvar data = {\n example: [\n {value: 4},\n {value: 7},\n {value: 13}\n ]\n};\n\nvar expression = jsonata(\"$sum(example.value)\");\nvar result = expression.evaluate(data); // returns 24\n\nmsg.payload = result ; // returns 24\n\nreturn msg;","outputs":1,"noerr":0,"x":320,"y":820,"wires":[["e8d41f8f.e9a7e"]]},{"id":"e8d41f8f.e9a7e","type":"debug","z":"fc8d3b46.2198","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","x":510,"y":820,"wires":[]}]

AIOT MAKER

unread,

Feb 15, 2018, 9:16:37 AM2/15/18

to Node-RED

Hello Skavan,

In general what I proposed is to use Jasonata inside the change node and the methods from Javascript RegExp object to accomplish one of the core functionalities. Having a second look today (after a good sleep) it seems that (apparently) the rules in the change node could be simplified. For sure the Regex method has to be replaced.

The map function, from Jasonata or from Javascript, is probably one of the most powerful tools to manipulate arrays. It allows one to iterate of the array elements applying a function to each of the elements. The variable $v will hold the value of each element and the variable $i will hold the index of this element. This can be handy in many different ways. My initial idea was to use this map to apply the regex match inside the change node (not needing a downstream function node). However I failed in my attempt to use Regex with Jasonata (but it should work). At the end of the day I left this map only to set the boolean value "false" to each element of the array (initialization).

In regards to the second point. It will work by replacing the method test(0 with the method exec() from Regex Object (https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/RegExp/exec). I tested it and show below the results. You can recover the value from transformed[2][2][1]

msg.transformed.forEach(function(item){

var re = new RegExp (item[1]);

var result = re.exec(msg.payload);

item[2] = result;

});

return msg;

Checkout out below the results. All in all this is a very interesting use case that is helping me to learn more (by the way thanks to Rewe Node for showing how to use Jsonata inside function nodes. I will try it later on).

Modified flow using excec():

[{"id":"1339ae17.104482","type":"inject","z":"e25c4f6.dca52b","name":"Start","topic":"","payload":"","payloadType":"date","repeat":"","crontab":"","once":false,"x":110,"y":140,"wires":[["e1e8b2af.6ae55"]]},{"id":"1e4017c0.4e9f58","type":"debug","z":"e25c4f6.dca52b","name":"","active":true,"console":"false","complete":"true","x":890,"y":140,"wires":[]},{"id":"30b32f31.3ee72","type":"function","z":"e25c4f6.dca52b","name":"Test Match","func":"\nmsg.transformed.forEach(function(item){\nvar re = new RegExp (item[1]);\nvar result = re.exec(msg.payload);\nitem[2] = result;\n\n});\nreturn msg;","outputs":1,"noerr":0,"x":750,"y":140,"wires":[["1e4017c0.4e9f58"]]},{"id":"e1e8b2af.6ae55","type":"function","z":"e25c4f6.dca52b","name":"Data Set hard Coded for testing","func":"msg.payload =\"IIS:HD1\";\nmsg.device = {\n \"projector\": {\n \"panasonic_ae\":{\n \"commands\": {\n \"power\": {\"deviceCommand\": \"P$\",\"deviceStatus\": \"(?:^P)(.{2}$)\",\"deviceRequestStatus\": \"QPW\", \"category\":\"main\", \"sph\":\"end\"},\n \"power status\": {\"deviceCommand\": \"QPW\",\"deviceStatus\": \"(^000$|^001$)\",\"deviceRequestStatus\": \"QPW\", \"category\":\"main\", \"topic\": \"power\", \"requestReply\": \"power\", \"sph\":\"end\"},\n \"input\": {\"deviceCommand\": \"IIS:$\",\"deviceStatus\": \"(?:^IIS:)(.{3})\",\"deviceRequestStatus\": \"QIN\", \"category\":\"main\", \"sph\":\"end\"},\n \"input status\": {\"deviceCommand\": \"QIN\",\"deviceStatus\": \"(^.{3}$)\",\"deviceRequestStatus\": \"QIN\", \"category\":\"main\", \"topic\": \"input\", \"requestReply\": [\"input\",\"input status\"], \"sph\":\"end\", \"link\": \"input\"},\n \"picture mode\": {\"deviceCommand\": \"VPM:$\",\"deviceStatus\": \"(?:^VPM:)(.{3})\",\"deviceRequestStatus\": \"QPM\", \"category\":\"main\", \"sph\":\"end\"},\n \"dpad\": {\"deviceCommand\": \"$\",\"deviceStatus\": \"\",\"deviceRequestStatus\": \"\", \"category\":\"navigation\", \"link\":\"navigation\"},\n \"menu\": {\"deviceCommand\": \"$\",\"deviceStatus\": \"\",\"deviceRequestStatus\": \"\", \"category\":\"navigation\"}\n }\n\t\t}\n\t}\n}\nreturn msg;","outputs":1,"noerr":0,"x":310,"y":140,"wires":[["a3499630.db4398"]]},{"id":"a3499630.db4398","type":"change","z":"e25c4f6.dca52b","name":"","rules":[{"t":"set","p":"commands","pt":"msg","to":"**.commands.$keys()","tot":"jsonata"},{"t":"set","p":"status","pt":"msg","to":"**.deviceStatus","tot":"jsonata"},{"t":"set","p":"match","pt":"msg","to":"$map(status, function($v, $i, $a) {\t false\t})","tot":"jsonata"},{"t":"set","p":"transformed","pt":"msg","to":"$zip(commands,status,match)","tot":"jsonata"},{"t":"delete","p":"commands","pt":"msg"},{"t":"delete","p":"status","pt":"msg"},{"t":"delete","p":"match","pt":"msg"}],"action":"","property":"","from":"","to":"","reg":false,"x":560,"y":140,"wires":[["30b32f31.3ee72"]]}]

QUOTE

steve rickus

unread,

Feb 15, 2018, 10:20:46 AM2/15/18

to Node-RED

I'm wondering if the "device definition file" input is a requirement that cannot be changed and must be input into the flow -- or is it just what you have and you are trying to use it in a function (or maybe change) node?

The more "node-red-ish" way I can suggest is to simply build a switch node with all of the regex tests built in, and an output port for each test. The ones that match will send the msg (unchanged) to its matching output port, and then you can enrich the initial msg object using a specific change node for that type of command. As for the extra credit, just test for the "sph":"end" case first, and don't connect any wires to it's output port -- that will effectively stop that msg.

Or am I missing some other requirements here?

As much as I love (and use!) jsonata for most data manipulations, the complexity of that expression will probably be unintelligible to the next person looking at it (or even yourself next month). Your javascript solution is probably more maintainable, and tolerant to unexpected inputs. But I think if you can break the problem up into pieces (switch on regex, then modify msg), it will be more readable, even if it takes 10 times as many nodes to accomplish. Function nodes are great for special tasks, but are opaque as to their logic from within the flow editor, so I like to trade off the size of the flow for more readability/maintainability.

--

Steve

On Wednesday, February 14, 2018 at 3:41:22 PM UTC-5, skavan wrote:

Hi,

I have successfully written a tcp flow creator that takes a "device definition file" as input and then builds a flow to handle sending and receiving messages to and from that device.

My code is messy and large (I'm not a native js developer). I want to simplify it, before releasing it to the wild.
I was wondering whether:
a) jsonata can be used inside a function and
b) whether jsonata is even the way to go.
Consider this use case:

A message arrives.
message.payload ="IIS:HD1"

I have a flow object, deviceMap that hold the following json (simplified):
{ "projector": { "panasonic_ae":{ "commands": { "power": {"deviceCommand": "P$","deviceStatus": "(?:^P)(.{2}$)","deviceRequestStatus": "QPW", "category":"main", "sph":"end"}, "power status": {"deviceCommand": "QPW","deviceStatus": "(^000$|^001$)","deviceRequestStatus": "QPW", "category":"main", "topic": "power", "requestReply": "power", "sph":"end"}, "input": {"deviceCommand": "IIS:$","deviceStatus": "(?:^IIS:)(.{3})","deviceRequestStatus": "QIN", "category":"main", "sph":"end"}, "input status": {"deviceCommand": "QIN","deviceStatus": "(^.{3}$)","deviceRequestStatus": "QIN", "category":"main", "topic": "input", "requestReply": ["input","input status"], "sph":"end", "link": "input"}, "picture mode": {"deviceCommand": "VPM:$","deviceStatus": "(?:^VPM:)(.{3})","deviceRequestStatus": "QPM", "category":"main", "sph":"end"}, "dpad": {"deviceCommand": "$","deviceStatus": "","deviceRequestStatus": "", "category":"navigation", "link":"navigation"}, "menu": {"deviceCommand": "$","deviceStatus": "","deviceRequestStatus": "", "category":"navigation"} } } } }

I wish to iterate through the "commands" objects and find which one or more commands.deviceStatus regex matches the message payload.
For a match, we create a new message, or append an array of messages to this message, that holds the matched Command.
In this case "command":"input", "value":"HD1".

All thoughts on how to make this cleaner, more efficient and more node-red-ish, much appreciated.

Walter

unread,

Feb 15, 2018, 10:24:11 AM2/15/18

to Node-RED

Here is a way to check the commands - it does not implement the whole function.

[{"id":"609fec03.c698b4","type":"debug","z":"96082429.7476f8","name":"","active":true,"console":"false","complete":"true","x":1026,"y":4843,"wires":[]},{"id":"69c403a.15302fc","type":"inject","z":"96082429.7476f8","name":"Start","topic":"IIS:HD1","payload":" {     \"projector\": {         \"panasonic_ae\":{             \"commands\": {                 \"power\": {\"deviceCommand\": \"P$\",\"deviceStatus\": \"(?:^P)(.{2}$)\",\"deviceRequestStatus\": \"QPW\", \"category\":\"main\", \"sph\":\"end\"},                 \"power status\": {\"deviceCommand\": \"QPW\",\"deviceStatus\": \"(^000$|^001$)\",\"deviceRequestStatus\": \"QPW\", \"category\":\"main\", \"topic\": \"power\", \"requestReply\": \"power\", \"sph\":\"end\"},                 \"input\": {\"deviceCommand\": \"IIS:$\",\"deviceStatus\": \"(?:^IIS:)(.{3})\",\"deviceRequestStatus\": \"QIN\", \"category\":\"main\", \"sph\":\"end\"},                 \"input status\": {\"deviceCommand\": \"QIN\",\"deviceStatus\": \"(^.{3}$)\",\"deviceRequestStatus\": \"QIN\", \"category\":\"main\", \"topic\": \"input\", \"requestReply\": [\"input\",\"input status\"], \"sph\":\"end\", \"link\": \"input\"},                 \"picture mode\": {\"deviceCommand\": \"VPM:$\",\"deviceStatus\": \"(?:^VPM:)(.{3})\",\"deviceRequestStatus\": \"QPM\", \"category\":\"main\", \"sph\":\"end\"},                 \"dpad\": {\"deviceCommand\": \"$\",\"deviceStatus\": \"\",\"deviceRequestStatus\": \"\", \"category\":\"navigation\", \"link\":\"navigation\"},                 \"menu\": {\"deviceCommand\": \"$\",\"deviceStatus\": \"\",\"deviceRequestStatus\": \"\", \"category\":\"navigation\"}             } \t\t} \t} }","payloadType":"json","repeat":"","crontab":"","once":false,"x":214,"y":4766,"wires":[["f5ebb2c3.a8b82"]]},{"id":"f5ebb2c3.a8b82","type":"change","z":"96082429.7476f8","name":"","rules":[{"t":"set","p":"payload","pt":"msg","to":"payload.projector.panasonic_ae.commands","tot":"msg"}],"action":"","property":"","from":"","to":"","reg":false,"x":376,"y":4782,"wires":[["36fe26a.406d7da"]]},{"id":"36fe26a.406d7da","type":"split","z":"96082429.7476f8","name":"","splt":"\\n","spltType":"str","arraySplt":1,"arraySpltType":"len","stream":false,"addname":"","x":554,"y":4796,"wires":[["9321ce10.df877"]]},{"id":"9321ce10.df877","type":"switch","z":"96082429.7476f8","name":"","property":"topic","propertyType":"msg","rules":[{"t":"regex","v":"payload.deviceStatus","vt":"msg","case":false}],"checkall":"true","outputs":1,"x":702,"y":4810,"wires":[["aa4c4b22.13d148"]]},{"id":"aa4c4b22.13d148","type":"join","z":"96082429.7476f8","name":"","mode":"custom","build":"array","property":"payload","propertyType":"msg","key":"topic","joiner":"\\n","joinerType":"str","accumulate":false,"timeout":"0.5","count":"","x":857,"y":4823,"wires":[["609fec03.c698b4"]]}]

skavan

unread,

Feb 15, 2018, 11:32:59 AM2/15/18

to Node-RED

Steve, thanks for your comments. This challenge really breaks into two parts;

1. The architectural approach and

2. The nits and grits of particular approaches - like jsonata etc..

On the first, the "device definition json" is the core of the system. The idea is that by changing the device definition file and ONLY the device definition file you can teach the flow how to support a new device.

I have it happily working with a denon receiver, a pioneer receiver, a harmony hub controlling a firetv, a panasonic projector and a matrix switcher. The flows are identical (except for the harmony hub, where a customized harmony node is swapped in for a tcp node). The only difference between them is the definition file. In my example, I just cut out a small excerpt...related to the command identification.

The architectural question is whether to build a function based solution (which I am now porting to a custom node), or whether to break the problem into chunks into a more node-redy kind of way. Right now, I have everything built into a 485 line function. As I move to putting that function into a custom node, I am faced with the question of "port it as is" or refactor it - and if the latter - deciding on the refactor approach.

Good point about the illegibility of jsonata! very powerful but quite a syntax shift.

s.

skavan

unread,

Feb 15, 2018, 11:53:32 AM2/15/18

to Node-RED

Very clever approach...thank you.

In your set-message-payload function, you hardcode the extraction of the commands objects like this:

payload.projector.panasonic_ae.commands

I don't actually know "projector" or "panasonic_ae" (depends on the definition file). But I do know it will always be the third level of the payload. Is there a way of doing the same thing?

I stole a line from AIOT Maker and changed that node to a Jsonata expression: **.commands

But I wonder if there is a different way of writing your line like: payload[0][0].commands (which doesn't work)!

Then the final trick, is to only take the first match if sph === "end".

Wow - the power of community. I am learning a lot.

Thanks.

Walter

unread,

Feb 15, 2018, 1:12:15 PM2/15/18

to Node-RED

You are asking to address a level in the object tree, because the name is variable - maybe you could give it a constant name and put the info (device type) in the value of another attribute?

The jsonata expression is: payload[0][0] (so you were pretty close :)

About the sph = "end": I'm not sure, but i think that the order of attributes in json is not guaranteed - so you would need a separate attribute to ensure the order - or change the approach.

To filter the msgs you could use a switch node based on a flow variable, indicating if a "end" msg is already in the array. Or a simple function node...

I'm also a node-red newbie, but intuitively i think that the current approach is too complicated and does not use the advantages of node-red and msg based programming.

But, it's easier to discuss it, if the whole problem is known...

AIOT MAKER

unread,

Feb 15, 2018, 1:49:54 PM2/15/18

to Node-RED

I never used the matches regex capability inside a switch node. It is awesome. Simply brilliant.

In regards to the point :

In your set-message-payload function, you hardcode the extraction of the commands objects like this:

payload.projector.panasonic_ae.commands

I don't actually know "projector" or "panasonic_ae" (depends on the definition file). But I do know it will always be the third level of the payload. Is there a way of doing the same thing?

Probably there are a lot of different ways to implement (as always in Node-RED) but I can think of two ..

(a) Adding three of split nodes.

Testing flow:

[{"id":"cecfdfa1.ab29f","type":"inject","z":"258218b5.64e358","name":"Start","topic":"IIS:HD1","payload":" { \"projector\": { \"panasonic_ae\":{ \"commands\": { \"power\": {\"deviceCommand\": \"P$\",\"deviceStatus\": \"(?:^P)(.{2}$)\",\"deviceRequestStatus\": \"QPW\", \"category\":\"main\", \"sph\":\"end\"}, \"power status\": {\"deviceCommand\": \"QPW\",\"deviceStatus\": \"(^000$|^001$)\",\"deviceRequestStatus\": \"QPW\", \"category\":\"main\", \"topic\": \"power\", \"requestReply\": \"power\", \"sph\":\"end\"}, \"input\": {\"deviceCommand\": \"IIS:$\",\"deviceStatus\": \"(?:^IIS:)(.{3})\",\"deviceRequestStatus\": \"QIN\", \"category\":\"main\", \"sph\":\"end\"}, \"input status\": {\"deviceCommand\": \"QIN\",\"deviceStatus\": \"(^.{3}$)\",\"deviceRequestStatus\": \"QIN\", \"category\":\"main\", \"topic\": \"input\", \"requestReply\": [\"input\",\"input status\"], \"sph\":\"end\", \"link\": \"input\"}, \"picture mode\": {\"deviceCommand\": \"VPM:$\",\"deviceStatus\": \"(?:^VPM:)(.{3})\",\"deviceRequestStatus\": \"QPM\", \"category\":\"main\", \"sph\":\"end\"}, \"dpad\": {\"deviceCommand\": \"$\",\"deviceStatus\": \"\",\"deviceRequestStatus\": \"\", \"category\":\"navigation\", \"link\":\"navigation\"}, \"menu\": {\"deviceCommand\": \"$\",\"deviceStatus\": \"\",\"deviceRequestStatus\": \"\", \"category\":\"navigation\"} } \t\t} \t} }","payloadType":"json","repeat":"","crontab":"","once":false,"x":90,"y":100,"wires":[["6f9dd943.850818","b30bf1d6.fb6e2"]]},{"id":"6f9dd943.850818","type":"split","z":"258218b5.64e358","name":"","splt":"\\n","spltType":"str","arraySplt":1,"arraySpltType":"len","stream":false,"addname":"","x":210,"y":100,"wires":[["2b24912a.33d3fe","79465b43.b87154"]]},{"id":"2b24912a.33d3fe","type":"split","z":"258218b5.64e358","name":"","splt":"\\n","spltType":"str","arraySplt":1,"arraySpltType":"len","stream":false,"addname":"","x":350,"y":100,"wires":[["1c27b656.67416a","358ccfce.95e4d"]]},{"id":"d29db7b3.6a3608","type":"debug","z":"258218b5.64e358","name":"","active":true,"console":"false","complete":"true","x":610,"y":100,"wires":[]},{"id":"1c27b656.67416a","type":"split","z":"258218b5.64e358","name":"","splt":"\\n","spltType":"str","arraySplt":1,"arraySpltType":"len","stream":false,"addname":"","x":470,"y":100,"wires":[["d29db7b3.6a3608"]]},{"id":"79465b43.b87154","type":"debug","z":"258218b5.64e358","name":"","active":true,"console":"false","complete":"true","x":310,"y":180,"wires":[]},{"id":"b30bf1d6.fb6e2","type":"debug","z":"258218b5.64e358","name":"","active":true,"console":"false","complete":"true","x":130,"y":200,"wires":[]},{"id":"358ccfce.95e4d","type":"debug","z":"258218b5.64e358","name":"","active":true,"console":"false","complete":"true","x":490,"y":180,"wires":[]}]

(b) using a function node with the following code to skip levels. It is a kind of ugly but it will provide the desired outcome. I provide a testing flow comparing the outcomes.

var level1 = Object.entries(msg.payload);

var level2 = Object.entries(level1[0][1]);

var level3 = Object.entries(level2[0][1]);

msg= {"topic": msg.topic, "payload": level3[0][1]};

return msg;

Testing flow :

[{"id":"bc27ce7d.a0496","type":"inject","z":"31171de6.05bfc2","name":"Start","topic":"IIS:HD1","payload":" { \"projector\": { \"panasonic_ae\":{ \"commands\": { \"power\": {\"deviceCommand\": \"P$\",\"deviceStatus\": \"(?:^P)(.{2}$)\",\"deviceRequestStatus\": \"QPW\", \"category\":\"main\", \"sph\":\"end\"}, \"power status\": {\"deviceCommand\": \"QPW\",\"deviceStatus\": \"(^000$|^001$)\",\"deviceRequestStatus\": \"QPW\", \"category\":\"main\", \"topic\": \"power\", \"requestReply\": \"power\", \"sph\":\"end\"}, \"input\": {\"deviceCommand\": \"IIS:$\",\"deviceStatus\": \"(?:^IIS:)(.{3})\",\"deviceRequestStatus\": \"QIN\", \"category\":\"main\", \"sph\":\"end\"}, \"input status\": {\"deviceCommand\": \"QIN\",\"deviceStatus\": \"(^.{3}$)\",\"deviceRequestStatus\": \"QIN\", \"category\":\"main\", \"topic\": \"input\", \"requestReply\": [\"input\",\"input status\"], \"sph\":\"end\", \"link\": \"input\"}, \"picture mode\": {\"deviceCommand\": \"VPM:$\",\"deviceStatus\": \"(?:^VPM:)(.{3})\",\"deviceRequestStatus\": \"QPM\", \"category\":\"main\", \"sph\":\"end\"}, \"dpad\": {\"deviceCommand\": \"$\",\"deviceStatus\": \"\",\"deviceRequestStatus\": \"\", \"category\":\"navigation\", \"link\":\"navigation\"}, \"menu\": {\"deviceCommand\": \"$\",\"deviceStatus\": \"\",\"deviceRequestStatus\": \"\", \"category\":\"navigation\"} } \t\t} \t} }","payloadType":"json","repeat":"","crontab":"","once":false,"x":130,"y":200,"wires":[["224b05b5.46e6fa","38f5c641.f0901a"]]},{"id":"38f5c641.f0901a","type":"change","z":"31171de6.05bfc2","name":"","rules":[{"t":"set","p":"payload","pt":"msg","to":"payload.projector.panasonic_ae.commands","tot":"msg"}],"action":"","property":"","from":"","to":"","reg":false,"x":300,"y":100,"wires":[["b685869e.298cd8"]]},{"id":"464101ef.057fc","type":"debug","z":"31171de6.05bfc2","name":"","active":true,"console":"false","complete":"true","x":490,"y":200,"wires":[]},{"id":"b685869e.298cd8","type":"debug","z":"31171de6.05bfc2","name":"","active":true,"console":"false","complete":"true","x":490,"y":100,"wires":[]},{"id":"224b05b5.46e6fa","type":"function","z":"31171de6.05bfc2","name":"Skip Levels","func":"var level1 = Object.entries(msg.payload);\n\nvar level2 = Object.entries(level1[0][1]);\n\nvar level3 = Object.entries(level2[0][1]);\n\n\nmsg= {\"topic\": msg.topic, \"payload\": level3[0][1]};\n\nreturn msg;","outputs":1,"noerr":0,"x":300,"y":200,"wires":[["464101ef.057fc"]]}]

steve rickus

unread,

Feb 15, 2018, 3:40:18 PM2/15/18

to Node-RED

On Thursday, February 15, 2018 at 11:32:59 AM UTC-5, skavan wrote:

On the first, the "device definition json" is the core of the system. The idea is that by changing the device definition file and ONLY the device definition file you can teach the flow how to support a new device.
I have it happily working with a denon receiver, a pioneer receiver, a harmony hub controlling a firetv, a panasonic projector and a matrix switcher. The flows are identical (except for the harmony hub, where a customized harmony node is swapped in for a tcp node). The only difference between them is the definition file. In my example, I just cut out a small excerpt...related to the command identification.

Ah, ok, that's what I was getting at... I do love the approach of having a "definition" of all the mappings, and letting some fixed logic handle all the routing and reformatting. In fact, one of my experiments last year was taking the "definition" of a flow from rows of database query results -- conceptually similar to your definition file, but all i had to do to add more "routes" and "handlers" was add more rows to the database.

The idea was borrowed (and expanded) from this article detailing how to use node-red flows to dynamically build and deploy node-red flows. Sounds like it would be a topic that you would find interesting. It worked well, as long as you don't deploy the newly generated flows to your own node-red server (trust me)! I may be able to dig some of it up, but suspect it's mostly obsolete now. In fact, I bet a lot of the javascript I wrote could now be replaced with some clever jsonata -- hmmm... that's an idea.