Mustache expressions in Change-Node

[Bart Butenaers]

Dear,

I would like to request a new feature in the Change-Node, to allow Mustache expressions to be used.

It can be implemented using a single line of code (see below), similar to the HttpRequest node.

Problem description

Suppose there is a global variable 'topics' containing following data:

A message arrives with following two fields:

• msg.payload: containing 'new value'
  
• msg.topic: containing 'TOPIC2'
  

I want to use a single Change-node to apply the new value to topic2 in the global json array. 

If I'm not mistaken this is currently not possible, but correct me if I'm wrong !!!!!!!!

Possible solution

I have added following code to the 15-change.js file:

The result

When I use now following Mustache formatted expression:

In following flow:

[{"id":"cce10850.07aec8","type":"inject","z":"d0d4f76e.209b68","name":"Inject new value","topic":"TOPIC2","payload":"new value","payloadType":"str","repeat":"","crontab":"","once":false,"x":803.6668033599854,"y":805.6666679382324,"wires":[["5a39220.3d2f7e"]]},{"id":"5a39220.3d2f7e","type":"change","z":"d0d4f76e.209b68","name":"Set last value","rules":[{"t":"set","p":"topics['{{topic}}'].last_value","pt":"global","to":"payload","tot":"msg"}],"action":"","property":"","from":"","to":"","reg":false,"x":997.6668643951416,"y":805.6666679382324,"wires":[[]]},{"id":"f78c767c.de6eb8","type":"inject","z":"d0d4f76e.209b68","name":"Initialize global context","topic":"MAGN_KEUK","payload":"{\"TOPIC1\":{\"last_value\":\"old value\"},\"TOPIC2\":{\"last_value\":\"old value\"},\"TOPIC3\":{\"last_value\":\"old value\"}}","payloadType":"json","repeat":"","crontab":"","once":false,"x":784.6666259765625,"y":760.6666259765625,"wires":[["9ce689a6.09fe38"]]},{"id":"9ce689a6.09fe38","type":"change","z":"d0d4f76e.209b68","name":"Set global variable","rules":[{"t":"set","p":"topics","pt":"global","to":"payload","tot":"msg"}],"action":"","property":"","from":"","to":"","reg":false,"x":1009,"y":760.6666259765625,"wires":[[]]}]

The result seems to be correct:

Open issues

I opened an issue for this feature, but Nick already found a major drawback of this solution:

Introducing mustache support is something we've chosen not to do in general - but I can see the use case you're raising isn't easily handled otherwise (short of using a Function node). A significant issue will be doing this in a way that doesn't break existing flows - however unlikely, "{{payload}}" is a valid identifier that an existing flow may be using. If we start parsing it as mustache it will break.

If somebody is aware of other drawbacks, alternatives, solutions, ... I would be glad to hear your opinions about this!!

Perhaps an extra checkbox (to enable Mustache content), but that might be too complex for non-technical users? 

Thanks in advance !!!

Bart Butenaers

[Bart Butenaers]

Hi folks,

I thought that mustache expressions would be a nice (and powerful) addition to the Change node.

However the number of replies on this thread was not overwhelming ...

So I tried to find a more "Node-Red standard" way to implement this functionality.

Instead of Mustache expressions, it would be even more powerful if a Jsonata expression could be used.

So something like this (add Jsonata type in the first dropdown, where it is now only available in the second 'to' dropdown):

This would even be more powerful compared to Mustache: the payload could e.g. be set to multiple Json fields using a single Jsonata expression.

Moreover it would be more visible to the user (instead of Mustache),  since the user explictly has to choose the 'Jsonata expression' type.

However the Jsonata team confirmed me in this issue that currently Jsonata doesn't allow to update JSON data ;-(

So this is currently not possible...  

As a result, I assume that Mustache expressions are currently the only option.

Could following checkbox be a good addition, or a bad or useless idea??

I would appreciate to get more insights on this issue!!

Kind regards,

Bart

[Julian Knight]

Personally, I think that the idea of JSONata in the what to update field seems a bit heavy but I do like the idea of being able to use data from the msg and data from an environment value as standard in such cases, the use case you give is one that I hit fairly regularly and have to use a function node to deliver.

In terms of JSONata, the lack of the ability to update a specific object property is really frustrating as it blocks the use of JSONata in a number of otherwise useful cases.

[Nick O'Leary]

I think you have a good use case here for us to look at; determining the property you want to set dynamically is only possible with the Function node today.

Within the context of a Set property action, a JSONata expression is not a good fit - it gets evaluated and returns an value. That value could be any type... an object, an array, a string, a number... how that would then map to either a msg, flow or global property isn't obvious.

I think the mustache route is probably the right way to go - the only question is how it is exposed. I had my original reservation that introducing it could break a flow without an option to enable it. I wonder what the true risk of that is; adding a checkbox to enable it is ugly. Or maybe there's another way to tackle - open to suggestions.

Nick

--
http://nodered.org
 
Join us on Slack to continue the conversation: http://nodered.org/slack
---
You received this message because you are subscribed to the Google Groups "Node-RED" group.
To unsubscribe from this group and stop receiving emails from it, send an email to node-red+unsubscribe@googlegroups.com.
To post to this group, send email to node...@googlegroups.com.
Visit this group at https://groups.google.com/group/node-red.
To view this discussion on the web, visit https://groups.google.com/d/msgid/node-red/d7455243-81fa-47bd-a453-4374ce7d8735%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Julian Knight]

Struggling to think how it might break a flow though clearly you could get unintended consequences - getting a string that you didn't expect or plan for. But as long as the input is constrained to return a string that is valid for the selected set then I don;t think it would seriously "break" anything?

On the other hand, it would probably be quite tricky to debug an unexpected string in some cases. You would have to attach a forked flow to the upstream node with a change node that returned a payload with the same expression & a debug. So not insurmountable perhaps.

You would probably need to check the validation and error checking of the change node too. For example, if I did the following:

myGlobalVar["{{payload.property1}}"]["{{payload.property2}}"]

And myGlobalVar.<property1> didn't exist, what would happen?

Which, thinking about it, is probably what you were thinking about anyway :-/

On Wednesday, 9 August 2017 13:18:52 UTC+1, Nick O'Leary wrote:

I think you have a good use case here for us to look at; determining the property you want to set dynamically is only possible with the Function node today.

Within the context of a Set property action, a JSONata expression is not a good fit - it gets evaluated and returns an value. That value could be any type... an object, an array, a string, a number... how that would then map to either a msg, flow or global property isn't obvious.

I think the mustache route is probably the right way to go - the only question is how it is exposed. I had my original reservation that introducing it could break a flow without an option to enable it. I wonder what the true risk of that is; adding a checkbox to enable it is ugly. Or maybe there's another way to tackle - open to suggestions.

Nick

On 9 August 2017 at 12:58, Julian Knight <j.kni...@gmail.com> wrote:

Personally, I think that the idea of JSONata in the what to update field seems a bit heavy but I do like the idea of being able to use data from the msg and data from an environment value as standard in such cases, the use case you give is one that I hit fairly regularly and have to use a function node to deliver.

In terms of JSONata, the lack of the ability to update a specific object property is really frustrating as it blocks the use of JSONata in a number of otherwise useful cases.

On Tuesday, 8 August 2017 22:51:15 UTC+1, Bart Butenaers wrote:

Hi folks,

I thought that mustache expressions would be a nice (and powerful) addition to the Change node.

However the number of replies on this thread was not overwhelming ...

--
http://nodered.org
 
Join us on Slack to continue the conversation: http://nodered.org/slack
---
You received this message because you are subscribed to the Google Groups "Node-RED" group.

To unsubscribe from this group and stop receiving emails from it, send an email to node-red+u...@googlegroups.com.

To post to this group, send email to node...@googlegroups.com.
Visit this group at https://groups.google.com/group/node-red.

[Keith Kelly]

For what it's worth, I also ran into a case where mustache expressions would have been handy in a Change node, and ended up using a function instead.  

[Dave C-J]

for clarity - the edge case Nick is thinking about is if an existing flow happens to have the target property like

msg["{{my_silly_variable}}"]   ie a variable name that includes {{ }} for some reason... so in reality pretty unlikely... but...

[Julian Knight]

Urgh! Haha, that would be problematic indeed. Might also be hard to debug then since you would probably misinterpret what was happening (basic psychology). I would think some of these edge cases could be dealt with through documentation though? e.g. "Don't!"

[Dave C-J]

Well going forward docs would help make it Ok - it's just if someone has already done it... they would broken by this change.

personally I think it's a very very slight concern and better to break it now rather than later - (ie before 1.0)

[Bart Butenaers]

Hi Julian, Dave, Keith, Nick,

thank you all for joining this discussion.  

It seems to me there are only a couple of options available:

• Don't enhance the Change node.  But then we all keep on writing Function nodes for stuff that could be fairly easily done by the Change node ...
• Let the user explicitly select that he wants to enable Mustache expressions.  But then we end up with something like my 'ugly' checkbox ...
• Automatically detect Mustache placeholders and let all the magic happen in the background.  But then it could be theoretically that someone his flow is broken ...

I personally would prefer the last option.  

P.S. To avoid breaking existing flows, perhaps a setting could be added to the settings file.  

Like 'enableMustacheInSwitch' which is disabled by default.  Or is this kind of stuff not allowed in the settings file ?

A disadvantage would be that almost nobody (including myself) will ever figure out that there is a setting available ;-)

Kind regards,

Bart

[Dave C-J]

I would also vote for 3

[Keith Kelly]

Is there ever a "breaking changes" section of the changelog or update warning popup/message?  With AutoHotkey, I check that section each time I upgrade, and that helps me know what I need to change if I rely on those components that are changing.

What about #3, with a setting behind the scenes (enableMustacheInSwitch) defaulted to enabled.  Then, a note that this legacy mode would be removed in 1.0?

[Julian Knight]

I'd happily vote for #3 as well.

Personally, I'd think the chance of anyone using {{aaaaa}} type property names is vanishingly small - not worth the effort of putting in switches - but if one were to be put in, it could go in the settings.js file.

[Fabien Marchewka]

Any news about this improvement ? It will be a fine feature.

[Bart Butenaers]

Hi Fabien,

I tried to get this feature implemented in Node-Red by discussing it here, since that is a mandatory part of the contribution process.  If you have more persuasiveness as me, please be my guest to get enough votes to get it implemented...  Don't really know how much votes are minimal required :-)

Good luck with it!

Bart

[Julian Knight]

I think that Nick and Dave are snowed under with work right now. I know that Nick in particular is working to try and take Node-RED to v1.

Bottom line is that the best chance of this being implemented is probably for someone to have a go at doing it then submitting an PR.

I certainly didn't hear any dissenting voices and several of us indicated it would be worthwhile.

[Nick O'Leary]

> Bottom line is that the best chance of this being implemented is probably for someone to have a go at doing it then submitting an PR.

If anyone is interested, then please let us know first. This discussion was a almost year ago and should be reviewed/revised in light of where things are at now.

--
http://nodered.org
 
** We're moving this mailing list over to the new Node-RED Forum: https://discourse.nodered.org **

 
Join us on Slack to continue the conversation: http://nodered.org/slack
---
You received this message because you are subscribed to the Google Groups "Node-RED" group.
To unsubscribe from this group and stop receiving emails from it, send an email to node-red+u...@googlegroups.com.
To post to this group, send email to node...@googlegroups.com.
Visit this group at https://groups.google.com/group/node-red.

To view this discussion on the web, visit https://groups.google.com/d/msgid/node-red/b6af8439-6704-4c1e-938e-e588a6b1e81c%40googlegroups.com.

[Bart Butenaers]

Hi Julian,

I'm VERY aware of Nick and Dave having an overflow of work.  Don't even understand how on earth they can manage it.

But back to this issue.  The 'discussion' part of the contribution process is not clear to me: you start a discussion, get some usefull feedback and then ... ?  When is the discussion complete (i.e. GO or NO-GO)?  It is very obvious that option 3 gets most votes.  But should I 'interpret' that as an implicit approval for a pull request?   

Don't know ...  Would prefer some kind of explicit approval from the Node-Red boss, or the Node-Red-approval-committee ;-)

Kind regards,

Bart