[ANNOUNCE] A Roadmap to 1.0

[Nick O'Leary]

Hi,

we've been thinking about what's next for the project.  We thought it was time to have a bit more of a roadmap that sets out where we think we're headed.

It's on the blog and we very much value your feedback.

https://nodered.org/blog/2017/07/17/roadmap-to-1-dot-0

Nick

[Joseph Martine]

This was a great write up. I was wondering about the big pic and this gives me a warm fuzzy about the direction. I am pushing this technology to everyone I know and am betting part of my career on its success. I too will be pushing a couple quasi-production apps out in the sept/oct timeframe so this couldn't come at a better time!!! I am pushing my aquarium automation out sometime in the next coming weeks. Things are feeling solid enough to place other "lives" in its care lol.

[Colin Law]

The road map looks great. A couple of questions and points for consideration:

Will the flow diff tool show line by line changes of the js in function nodes?

Will the proposed structure allow different projects to use different
versions of the same node in different projects?

A minor point on the welcome screen. it may not be obvious whether
cloning an existing project should be under Create or Open Existing.
One could produce arguments for either. An alternative might be to
have clone as a third option on the welcome screen.

Possibly the Version Control screen could do with a Discard
Uncommitted Changes option.

I am not sure about the choice of words "Commit unstaged changes" as
unstaged has a specific meaning in git. Perhaps it should just be
"Commit changes".

Will I be allowed to put a nodes folder inside the project to put
home-brewed nodes or development versions of contrib nodes into?

The .node-red folder. The choice of .node-red as the default top
level folder is something that causes me some grief. It is by default
a hidden folder which can be a pain. Standard system file search
utilities may not index it. Has the idea of changing this (to
node-red for example) been considered? Possibly with the release of
1.0.0.

Cheers

Colin

> --
> http://nodered.org
>
> Join us on Slack to continue the conversation: http://nodered.org/slack
> ---
> You received this message because you are subscribed to the Google Groups
> "Node-RED" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to node-red+u...@googlegroups.com.
> To post to this group, send email to node...@googlegroups.com.
> Visit this group at https://groups.google.com/group/node-red.
> To view this discussion on the web, visit
> https://groups.google.com/d/msgid/node-red/CAF%3Dvhqco7WvkyOoncBVMfnvVbmoa_n8-RpvYf%2BHx-ZDve3HQLg%40mail.gmail.com.
> For more options, visit https://groups.google.com/d/optout.

[Nick O'Leary]

Hi Colin, thanks for the comments.

The wireframes are not intended to be absolute final designs, so any specific wording or arrangement of buttons is subject to change as it is developed.

The question of whether cloning an existing project comes under opening an existing project or creating a new one is something I bounced between a number of times. Again, the exact detail will come as we create the UI and test it.

We don't have any specific plans for improvements to the diff tool, but clearly there are more things it could do. The challenge is how the tool knows when to provide a more detailed diff of a node property - we wouldn't want to hardcode knowledge of the Function, Template, Comment, UI-Template nodes, etc etc.

What platform do you run on? Using a dot-directory in the user home is a widely used standard approach on Linux/OSX systems. We have no intention of changing that.

Projects will *not* be able to have different versions of nodes installed. The nodes are installed at the top-level as we cannot switch between versions of nodes at runtime. This does mean edge cases exist where projects specify different versions and some thought will go into how that will be managed.

As for home brew nodes... undecided where they sit in the projects.

Nick

> email to node-red+unsubscribe@googlegroups.com.

> To post to this group, send email to node...@googlegroups.com.
> Visit this group at https://groups.google.com/group/node-red.
> To view this discussion on the web, visit
> https://groups.google.com/d/msgid/node-red/CAF%3Dvhqco7WvkyOoncBVMfnvVbmoa_n8-RpvYf%2BHx-ZDve3HQLg%40mail.gmail.com.
> For more options, visit https://groups.google.com/d/optout.

--
http://nodered.org

Join us on Slack to continue the conversation: http://nodered.org/slack
---
You received this message because you are subscribed to the Google Groups "Node-RED" group.

To unsubscribe from this group and stop receiving emails from it, send an email to node-red+unsubscribe@googlegroups.com.
To post to this group, send an email to node...@googlegroups.com.

Visit this group at https://groups.google.com/group/node-red.

To view this discussion on the web, visit https://groups.google.com/d/msgid/node-red/CAL%3D0gLui3YTT9B1tFoVOknTkpRfoS_rH7koSbMUcFuXbLGCM0A%40mail.gmail.com.

[Garry Hayne]

Looks good Nick, I would like to see the possibility to update node-red itself from within the editor.

Garry

[Colin Law]

Hi Nick

Thanks for the clarifications.

I use various flavours of Linux. As I understand it dot folders are
intended for saving config and settings data that the user would not
normally have to access directly. They are hidden, so the ls command
does not, by default, show them, nor do GUI file managers I believe.
To me this does not seem appropriate for the node-red folder. I
realise that as node-red develops the intention is that the user will
use the command line less and less as the UI becomes more like an IDE.
There will still be times when the GUI will not do everything though.
For example if I have additional javascript files I wish to be
included I currently put those in a folder named static. I imagine I
will always have to do that manually. Similarly I guess that the
node-red UI is unlikely to handle git branches nor provide the
capabilities of utilities like gitk so to perform such tasks will
require the user to run commands from within the .node-red folder.

Cheers

Colin

>> > email to node-red+u...@googlegroups.com.

>> > To post to this group, send email to node...@googlegroups.com.
>> > Visit this group at https://groups.google.com/group/node-red.
>> > To view this discussion on the web, visit
>> >
>> > https://groups.google.com/d/msgid/node-red/CAF%3Dvhqco7WvkyOoncBVMfnvVbmoa_n8-RpvYf%2BHx-ZDve3HQLg%40mail.gmail.com.
>> > For more options, visit https://groups.google.com/d/optout.
>>
>> --
>> http://nodered.org
>>
>> Join us on Slack to continue the conversation: http://nodered.org/slack
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "Node-RED" group.
>> To unsubscribe from this group and stop receiving emails from it, send an

>> email to node-red+u...@googlegroups.com.

>> To post to this group, send an email to node...@googlegroups.com.
>> Visit this group at https://groups.google.com/group/node-red.
>> To view this discussion on the web, visit
>> https://groups.google.com/d/msgid/node-red/CAL%3D0gLui3YTT9B1tFoVOknTkpRfoS_rH7koSbMUcFuXbLGCM0A%40mail.gmail.com.
>> For more options, visit https://groups.google.com/d/optout.
>
>

> --
> http://nodered.org
>
> Join us on Slack to continue the conversation: http://nodered.org/slack
> ---
> You received this message because you are subscribed to the Google Groups
> "Node-RED" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to node-red+u...@googlegroups.com.

> To post to this group, send email to node...@googlegroups.com.
> Visit this group at https://groups.google.com/group/node-red.
> To view this discussion on the web, visit

> https://groups.google.com/d/msgid/node-red/CAF%3DvhqfAzjn4tgsqZifEEMUGc8GZUFxSuFu%3DLTh8t2s2%3DHxHNg%40mail.gmail.com.

[steve rickus]

Great write-up, Nick -- thanks for putting that together...

I do agree that the diff tool should have no hard-coded behavior based on the containing node type. I'm wondering if it would be sufficient to simply check each string value for any "\n" newline characters?

If found, specific multi-line logic (like splitting into separate lines and displaying line numbers) could then be employed. I think this feature is already built-in to the debug sidebar display, correct? Would the newline check carry too much overhead, or cast too wide a net and pick up more than just those 4 nodes you listed?

--

Steve

On Wednesday, July 19, 2017 at 7:28:26 AM UTC-4, Nick O'Leary wrote:

We don't have any specific plans for improvements to the diff tool, but clearly there are more things it could do. The challenge is how the tool knows when to provide a more detailed diff of a node property - we wouldn't want to hardcode knowledge of the Function, Template, Comment, UI-Template nodes, etc etc.

Nick

[Nick O'Leary]

Steve, not looking to solve it right now. There are various approaches we could take. The net would be cast as wide as it makes sense - arguably any field that includes newlines would benefit from a more detailed diff view.

Nick

--

http://nodered.org
 
Join us on Slack to continue the conversation: http://nodered.org/slack
---
You received this message because you are subscribed to the Google Groups "Node-RED" group.
To unsubscribe from this group and stop receiving emails from it, send an email to node-red+unsubscribe@googlegroups.com.
To post to this group, send email to node...@googlegroups.com.
Visit this group at https://groups.google.com/group/node-red.

To view this discussion on the web, visit https://groups.google.com/d/msgid/node-red/b8b9415f-3232-4785-bfad-c3cf3cf22446%40googlegroups.com.

[Julian Knight]

All looks pretty good there Nick and I love that NR will get to v1 as I think you are right about it impacting production deployment. Many organisations would not generally wish to deploy something with a version <1. Indeed, many that I've worked with over the years wouldn't deploy a v1.0 either. Management rarely has the luxury of spending time working out whether <v1 products are worth the effort so assumptions are made.

The features mentioned in your slide deck seem to reflect the kinds of things people have been looking for in this group and on Slack so that's also great.

My only comment at this stage would be not to loose sight (not that I think you would) of the ability to install NR locally rather than globally. In my view, that is a more likely production deployment option since it allows multiple instances of NR at different version numbers to be deployed which gives the maximum operational flexibility.

[Bart Butenaers]

Hi Nick,

Amazing what your team wants to achieve in the next releases.  Great stuff !!!  Looking forward to it.

Extensibility - providing the right extension points and APIs to adapt Node-RED

I really like Node-Red because it is extremely easy to develop a custom node (compared to other systems like e.g. OpenHab), and publish it to the user. However lot's of other stuff in Node-Red is very fixed:

• Dashboard: limited number of widgets (button, chart, gauge, slider ...), with fixed events in main.js file.
• Node property types: types are fixed (json, number, date, string ...) in util.js file.
• Red.validators: limited validator choice (number, regex, typedinput) in validators.js file.
• ...

To extend such functionality as a community member, we need to create a pull request (which is less accessible for hobbyists !!).  Is this a conscious choice for stability (e.g. avoid that my custom node X crashes, because it depends on a custom validator Y from another developer, which is not installed on somebody's Node-Red instance ...), or is this something that might become more extensible (via extra API's) in a future version?  Remark: the stability issue could be solved by some kind of Node-Red dependency mechanism (Node X has a dependency Validator Y) ...

Most popular missing functionality

It would be great to know which extra functionality users want to have in Node-Red.  Are there any plans to add some kind of voting system, to vote for the most popular missing functionality?  Or perhaps the query results (statistics) from flows.nodered.org could be made public, so we could e.g. analyze which search criteria gave no results: the community could develop extra nodes based on this information, or extra keywords could be added to the package.json file of existing nodes (to route users correctly to the nodes they need).

Kind regards,

Bart

[Mike Blackstock]

Hi Nick,

Thanks for keeping us up to date on your thinking.  We really appreciate having the feature flags and hooks needed to disable features, or do things differently in FRED.  That said our goal is to support as many of the new features as possible :-)

Some questions and comments for you roughly categorized.

> Editor/Runtime separation

Curious to learn more about the editor/runtime separation and how it will work when you are ready.  Will we be able to deploy just the editor, without a runtime so I can edit/validate flow files and deploy later, even on a non-Node-RED runtime?  I assume so, just want to confirm.

> Version Control 'Commit' Deployment Option and Runtime

One issue i have had since the beginning is that there is no way to save/edit my work without deploying/running the flow using the editor, understanding the potential UI issues around this.  So somewhat related to this and the runtime separation, I notice that you mention a new 'Commit changes' deployment option.  How will this interact with the runtime?  Will committing also restart (parts of) the flow if it has changed since last deployed?   Will it be possible to 'deploy' or save/commit the flow file e.g. using the editor component without deploying it to a local runtime?

> Project/Version Control and Keys

Currently FRED manages and hides the encryption key for credentials. Will we need to make that visible/changeable to the user for the new 'project' features?

> Editing non-flow files

Just a comment.  You noted that we don't want NR to be a generic IDE, but you have some README, settings and package.json editing.  I understand that it's hard to find the right balance there.  I'm torn because I want to let users do everything they need or want to do with Node-RED on our system, but don't want them to edit a file that somehow causes Node-RED to fail on startup.  Not sure how much file access and text editing capability to give them...perhaps the hook up to repos is all they need to be able to view/edit their files offline in a text editor.  Not sure.

> Access to logs/stdout/err

There is a lot of useful information in stdout still.  In FRED we provide an easy way to access this.  Any thoughts about somehow displaying this console information in the UI somewhere?

> Node install hooks

We handle node installation ourselves currently.  Will there be a way for us to hook into the node install features in the system so that we can take advantage of the project node install feature without using NPM to do it?

Of course there will be cases where we cannot support certain nodes in an imported project repo.  Makes me wonder if we can we substitute nodes for different implementations somehow? E.g. to simulate a Pi's GPIO pins or an external service for testing.

If I think of anything else I will get it to you as soon as I can.  Best of luck in SF with your presentations this week.

Mike

[Dean Cording]

Hi Nick

I have a bit of experience here I'd like to share.  About 10 years ago I wrote a very similar system to Node-Red but entirely in Java.  This system was used to control a fully automated astronomical observatory - everything from moving the telescope and dome to controlling the lights and building security.  Probably a couple of hundred flows with several hundred nodes.

Here are some lessons learned:

* Things can get complex very quickly. More complex than any one person can keep in their head, and much quicker than any one person can develop.  Being able to have several developers working simultaneously on different parts of the system is essential for any serious production use.

* Distributed code is essential.  When you have sensors, actuators, switches, and displays scattered around your environment, having to bring everything back to a central point is a major handicap.  Being able to distribute your code over a collection of devices, large and small, and being able to easily move a flow from one device to another or split it up over two or more devices is an essential capability.

Related to this is being able to easily move flows between development, test, and production environments.  It also needs to be able to be done without disruption - restarting your physical access controls every time you do an lighting control update isn't a good option.

Dynamic scalability and failover fit in here as well.

The roadmap covers these points to a large extent but I would make one comment - you talk about Node-Red applications; think more along the lines of Node-Red systems.  Node-Red's strength is being able to tie together disparate inputs and outputs to implement complex intelligent systems.

In the short term, I'd like to make two suggested improvements:

1. Improved error reporting.  Debugging new nodes is a real pain in the arse.  Just trying to find a missing bracket or semicolon can take hours.  I've been forced to use debugging techniques I haven't had to use since my assembler days.  Just telling me which line number a syntax or runtime error occurs on would be a major step forward.

2. Have a good think about the structure of messages.  The msg.topic/msg.payload clearly originates from the constraints of MQTT.  Some nodes only expect a single value to be in the payload, others handle embedded objects, whilst others a completely flexible about what message/flow/global properties they work with.  This results in a lot of property wrangling to get nodes to fit along a flow - work that is an artifact of the implementation and does not contribute to the solution. A good example of this is the dashboard - gauges can only display msg.payload values. If I have a complex data object then I have to break it up to display values.  And too bad if I want to display a global or flow property.

At some point there was the idea that msg.payload should contain the data and other msg properties contain metadata - topic, timestamp, request id, etc.  However from experience, one node's metadata is the next node's data at some point.  

Outside of MQTT, msg.topic serves little purpose. With any sort of compound data, the concept of a topic covering the entire data object becomes somewhat meaningless.

Personally, I believe the current quasi-standard is a hinderance and Node-Red would benefit from dropping the msg.topic and msg.payload concepts completely and just have messages as arbitrary objects.  It isn't that hard to make nodes completely configurable as to which properties they act upon. Node Red even makes functionality available to make it quite straight forward.


Cheers,

Dean

[Michael Hogan]

I would disagree a little with your views on the utility of the ".topic" field.  We are starting to use it even in a non MQTT context (or, seamlessly, in an MQTT mediated context) for just the sort of hierarchical organization and routing that you describe at the outset of your post.  We even use the field in web clients, since the topic fragment be easily re-written into legitimate javascript to dig into an object hierarchy, or to generate an ID for an element in a web page (e.g. web page receives message over WS or WS/MQTT, unpacks and re-writes it as a javascript fragment, and then evals the script to update the right field on the page, all without any need to understand the content of the message).

As an example in the client-side, we use this to animate displays of automated processes on an event driven basis.  If multiple clients are looking at the process, and something changes, the displays all update dynamically.  Surprisingly simple to write if the design is well organized.  A few lines of code can manage this sort of thing for a nearly unlimited number of points under observation.

Within node-red itself, we use this for properly dispatching messages.  E.g. the MQTT block can be followed by a function that strips off the head of the topic and dynamically dispatches the message based on the tail of the topic.  The receiving blocks can determine if the message was intended for them, strip off their part of the topic and dispatch to subordinate blocks and so forth ..so this dispatch mechanism can be neatly layered.  Wild-cards can be used for broadcasts that have carefully controlled scope etc.  

I'd go as far as suggesting that the topic idea is probably worth formalizing in node-red, so that there are some relatively standard ways of doing these hierarchical organization schemes.

One thing I'd love is if a function within a sub-flow could figure out the name of its parent.  Then you could easily make hierarchies using nested sub-flows.  Incoming messages could be easily filtered by comparing the first field in the topic to the parent's name, and outgoing messages could easily be "topified" by pre-pending the parent sub-flow's name to the topic field, and forwarding the message out an output port.

[Nick O'Leary]

Hi Dean, on your two specific suggestions:

1. Improved error reporting - I agree there's more to do here, but we are somewhat limited by the nature of certain errors and what information we have access to. This is a particular problem within the editor.

2. msg.payload is the main focal point of data passing between nodes. That is not going to change; it is that convention that allows things to Just Work far more often than not. If there was no convention and every node picked an arbitrary property, a user would have to configure every single node to ensure the data was accessed properly at each stage. The Change node is there to help restructure messages as needed - including putting information into or pulling out of flow/global context. Making every node completely configurable will, in my experience and opinion, make it more complicated.

Nick

--
http://nodered.org
 
Join us on Slack to continue the conversation: http://nodered.org/slack
---
You received this message because you are subscribed to the Google Groups "Node-RED" group.
To unsubscribe from this group and stop receiving emails from it, send an email to node-red+unsubscribe@googlegroups.com.
To post to this group, send email to node...@googlegroups.com.
Visit this group at https://groups.google.com/group/node-red.

To view this discussion on the web, visit https://groups.google.com/d/msgid/node-red/d8c2e75f-bec6-4afe-825f-da216669200b%40googlegroups.com.

[Julian Knight]

See my comments below...

On Saturday, 29 July 2017 10:42:22 UTC+1, Dean Cording wrote:

...

* Things can get complex very quickly. More complex than any one person can keep in their head, and much quicker than any one person can develop.  Being able to have several developers working simultaneously on different parts of the system is essential for any serious production use.

JK> I fully agree. One way to help would be to include some additional self-documentation features built in to every node (e.g. not needing node developers). We now have the ability to add labels to ports. Adding the ability to document each node instance would help further. A description tab with the ability to use Markdown would work I think.

* Distributed code is essential.  When you have sensors, actuators, switches, and displays scattered around your environment, having to bring everything back to a central point is a major handicap.  Being able to distribute your code over a collection of devices, large and small, and being able to easily move a flow from one device to another or split it up over two or more devices is an essential capability.

JK> Again I agree. This would benefit from some additional development. 

Related to this is being able to easily move flows between development, test, and production environments.  It also needs to be able to be done without disruption - restarting your physical access controls every time you do an lighting control update isn't a good option.

Dynamic scalability and failover fit in here as well.

The roadmap covers these points to a large extent but I would make one comment - you talk about Node-Red applications; think more along the lines of Node-Red systems.  Node-Red's strength is being able to tie together disparate inputs and outputs to implement complex intelligent systems.

In the short term, I'd like to make two suggested improvements:

1. Improved error reporting.  Debugging new nodes is a real pain in the arse.  Just trying to find a missing bracket or semicolon can take hours.  I've been forced to use debugging techniques I haven't had to use since my assembler days.  Just telling me which line number a syntax or runtime error occurs on would be a major step forward.

JK> I think this has already been agreed. Node.JS isn't always the easiest to debug at the best of times. 

2. Have a good think about the structure of messages.  The msg.topic/msg.payload clearly originates from the constraints of MQTT.  Some nodes only expect a single value to be in the payload, others handle embedded objects, whilst others a completely flexible about what message/flow/global properties they work with.  This results in a lot of property wrangling to get nodes to fit along a flow - work that is an artifact of the implementation and does not contribute to the solution. A good example of this is the dashboard - gauges can only display msg.payload values. If I have a complex data object then I have to break it up to display values.  And too bad if I want to display a global or flow property.

At some point there was the idea that msg.payload should contain the data and other msg properties contain metadata - topic, timestamp, request id, etc.  However from experience, one node's metadata is the next node's data at some point.  

Outside of MQTT, msg.topic serves little purpose. With any sort of compound data, the concept of a topic covering the entire data object becomes somewhat meaningless.

Personally, I believe the current quasi-standard is a hinderance and Node-Red would benefit from dropping the msg.topic and msg.payload concepts completely and just have messages as arbitrary objects.  It isn't that hard to make nodes completely configurable as to which properties they act upon. Node Red even makes functionality available to make it quite straight forward.

JK> This is where I can't fully agree with you I'm afraid. I can see where you are coming from but actually the standard of using msg.topic and msg.payload is really helpful in a lot of situations and particularly useful for beginners.

JK> In particular, msg.topic is not only helpful for MQTT, it is a key piece of metadata that helps identify and separate out data as it flows through. This is totally independent of the actual payload and it is common to keep metadata separate to data in many systems.

JK> While it is indeed possible to make nodes that use any data, this actually goes against your earlier argument in my view. Purely arbitrary data structures are MUCH harder to understand and Node-RED would need massive restructuring to apply schema controls throughout in order to bring sense to fully arbitrary structures. That would also add significant overheads in processing that are unnecessary for the majority of use cases.

JK> The use of .topic and .payload are useful standards that need no special controls since most people agree to use them. But they don't restrict the use of Node-RED since they can be ignored or augmented where it is appropriate. They are easy for beginners to understand and useful guides.

[Julian Knight]

One thing I'd love is if a function within a sub-flow could figure out the name of its parent.  Then you could easily make hierarchies using nested sub-flows.  Incoming messages could be easily filtered by comparing the first field in the topic to the parent's name, and outgoing messages could easily be "topified" by pre-pending the parent sub-flow's name to the topic field, and forwarding the message out an output port.

Michael, though I agree with the rest of your comments, I think that last piece goes against the idea of flow processing which is that any individual node does not need to handle any "knowledge" about its parent. I don't believe we need to add that kind of complexity to Node-RED since anyone can already do that for themselves if it makes sense but to make all nodes add that information would result in many vary large messages for no purpose since it is rare indeed that you would want to programmaticaly calculate the path a msg has taken, using the topic is usually enough.

[Michael Hogan]

Mmmm, hierarchical tagging is incredibly handy for certain kinds of large configurations, and it helps with code reuse.  A generically useful sub-flow could be instantiated multiple times, and it's position in a fully qualified path tells you which particular instance you are dealing with (e.g. "TheGraduate\DustinHoffman" vs "Tootsie\DustinHoffman").

Just to clarify the suggestion, the contained is not handling any knowledge about its parent -- it is presuming that the parent has created a context value that conveys its name, and that value is (in this particular example) used for filtering incoming messages and contextualizing outgoing messages.

At any rate, it's not too hard to implement something akin to this using the current capabilities of the system, so it's not a super-important feature.

[Dean Cording]

Thanks Nick and Julian,

I know I have argued the case against topic and payload a couple of time now, but I would just like to counter a couple of your reasonings.  I should also clarify that my main gripe is with nodes that work only if msg.payload contains a single value.

NO'L> If there was no convention and every node picked an arbitrary property, a user would have to configure every single node to ensure the data was accessed properly at each stage. The Change node is there to help restructure messages as needed

But isn't having to use a Change node in front of every functional node just as complicated.  There is nothing stopping every node defaulting to using msg.payload, just don't restrict it to only using msg.payload for advanced users who are working with structured data (trust us, we can do complicated). 

JK> using msg.topic and msg.payload is really helpful in a lot of situations and particularly useful for beginners.

As an advanced user, it's a hindrance.  As I suggested above, make msg.payload the default, but don't restrict it to just that.

JK> Node-RED would need massive restructuring to apply schema controls throughout in order to bring sense to fully arbitrary structures.

Why would you need schema controls?  Node Red already provides support for fully configurable properties, both in the editor and the framework.   There are just a handful of the standard nodes that need modification to have configurable properties. 

JK>That would also add significant overheads in processing that are unnecessary for the majority of use cases.

Having to now use multiple Change nodes and having to unmarshal and  remarshal data in messages is a significant overhead that is unnecessary.  The only reason they are needed is because of an arbitrary design limitation.

The really silly thing is that I spent some time trying to come up with a real life example where this was a problem but couldn't.  The reality is that only a very small number of nodes actually adhere to the msg.property standard (RBE and Range are two examples).  The notable exceptions are the dashboard nodes Gauge and Chart which only work with msg.payload containing a single numeric value.

So I suppose my strongest argument against the msg.payload standard is that in reality not many nodes actually use it.

Dean

[Dave C-J]

Hi Dean

what do you mean by "not many nodes use" the msg.payload "standard" - all nodes (afaik) expect the "interesting data" to be in msg.payload. For example all the io type nodes (mqtt, http, serial, tcp, udp, etc) deliver and expect the data as payload ...

Also not sure what you mean by chart and gauge being exceptions - certainly chart uses the topic to identify a different data series and other ui nodes just expect values in payload (but can use angular filters to modify them)

[Julian Knight]

Hi Dean. Thanks for the response, I've added some more info below.

On Monday, 31 July 2017 15:06:10 UTC+1, Dean Cording wrote:

Thanks Nick and Julian,

I know I have argued the case against topic and payload a couple of time now, but I would just like to counter a couple of your reasonings.  I should also clarify that my main gripe is with nodes that work only if msg.payload contains a single value.

NO'L> If there was no convention and every node picked an arbitrary property, a user would have to configure every single node to ensure the data was accessed properly at each stage. The Change node is there to help restructure messages as needed

But isn't having to use a Change node in front of every functional node just as complicated.  There is nothing stopping every node defaulting to using msg.payload, just don't restrict it to only using msg.payload for advanced users who are working with structured data (trust us, we can do complicated). 

JK> Interesting. Just checking my own live code and I node that the majority of change nodes are to handle things that DONT work off the payload (e.g. the filename in the file node) or are changing some specific value either on the payload or restructuring the topic (pretty standard extract/transformation use cases) or setting a global/flow variable. Not using topic/payload wouldn't change any of those.

 

JK> using msg.topic and msg.payload is really helpful in a lot of situations and particularly useful for beginners.

As an advanced user, it's a hindrance.  As I suggested above, make msg.payload the default, but don't restrict it to just that.

JK> I don't think you are restricted? You CAN use anything you want can't you? As you point out, some nodes do use something different already. A good example is Max's rfxcom nodes. They do have the main data on the payload as expected but may also set a msg.status as additional data. I don't really have a problem with that - other than the fact that I generally end up feeding that data back onto the payload! ;-) 

 

JK> Node-RED would need massive restructuring to apply schema controls throughout in order to bring sense to fully arbitrary structures.

Why would you need schema controls?  Node Red already provides support for fully configurable properties, both in the editor and the framework.   There are just a handful of the standard nodes that need modification to have configurable properties. 

JK> I was thinking there that, without standards, far more nodes would not be predictable in nature and so the need for schema checks would likely increase. I might have been wrong about that. 

 

JK>That would also add significant overheads in processing that are unnecessary for the majority of use cases.

Having to now use multiple Change nodes and having to unmarshal and  remarshal data in messages is a significant overhead that is unnecessary.  The only reason they are needed is because of an arbitrary design limitation.

The really silly thing is that I spent some time trying to come up with a real life example where this was a problem but couldn't.  The reality is that only a very small number of nodes actually adhere to the msg.property standard (RBE and Range are two examples).  The notable exceptions are the dashboard nodes Gauge and Chart which only work with msg.payload containing a single numeric value.

JK> Hmm, well I wont make the obvious comment for fear of offending. But again, I've just checked my "live" code and note that by far the majority of everything I've used has the main data on the payload. And the majority also use the topic as metadata or at least allow it to flow through. 

So I suppose my strongest argument against the msg.payload standard is that in reality not many nodes actually use it.

JK> I have 24 node modules installed, 127 nodes on the pallet in total. Of which I am using 17 modules (must tidy up!) and somewhere around 50 node types in use. Actually, I cannot spot a single node in use that doesn't at least default to using msg.payload for main data though of course, many of them can in fact be changed. 

[Julian Knight]

By the way, I really appreciate this discussion as it is making me think carefully about how I use Node-RED. As I mainly use it for a hobby, it is often hard to go back and think rationally about how it should be used, its strengths and weaknesses.

[Dave C-J]

Dean, You also mention "a handful of nodes would need modification". Which would they be ? Maybe this is only a small problem, that only affects a certain category of nodes. 

(As per JK, good discussion)

--

Sent from phone.