File nodes suggestion

123 views
Skip to first unread message

Mike Bell

unread,
Oct 20, 2016, 11:43:16 AM10/20/16
to Node-RED
All,

I would like to be able to move flows between users and machines with a minimum of editing. Since the file in/out nodes require complete path names, I always find myself editing them. Would it be possible to have those nodes recognize the $HOME (or ~) environmental variable? I realize that I could use exec nodes (with cat or echo), but that seems to defeat the purpose of having file i/o nodes.

Mike

Sebastian Barwe

unread,
Oct 20, 2016, 1:34:08 PM10/20/16
to Node-RED

Julian Knight

unread,
Oct 20, 2016, 1:59:41 PM10/20/16
to Node-RED
Yes, the file nodes are a little simplistic. I started to create a new set of nodes to address these issues but I'm afraid the work stalled after the file lister. This could be added to the wish list, it wouldn't be hard to do.

Mike Bell

unread,
Oct 20, 2016, 5:03:44 PM10/20/16
to Node-RED
Sebastian,

Thanks for the suggestion. I generally use ftp to move files between hosts, but this could be useful. Unfortunately, it doesn't seem to automate the host-specific or user-specific changes that need to be made in the file in/out nodes.

Mike

Mike Bell

unread,
Oct 20, 2016, 5:20:49 PM10/20/16
to Node-RED
Julian,

I noticed your work on the file lister (and good intentions on the copier), but I haven't had a chance to use it yet. I have posted before on suggestions for enhancements of the file nodes, so if you think it would be helpful, I'll raise an issue (or two) on your repository on GitHub.

Thanks,
Mike

Nicholas O'Leary

unread,
Oct 20, 2016, 5:27:05 PM10/20/16
to Node-RED Mailing List
I think we'd much rather ensure the default file nodes are as useful as they can be.

If you have specific feature requests, lets look at them in the context of the core nodes in the first instance.

We have a not-well-documented feature of allowing you to set any node property to an environment variable using the following syntax:

 $(MY_ENV_VAR)

The runtime automatically substitutes such a value with the corresponding environment variable if its set.

This technique does support partial substitutions such as $(HOME)/fred ... it has to be the entire property value.

So with the current node-red, you could use that and set environment variables on each machine as needed. For example, in your settings.js file you can do:

    process.env.MY_ENV_VAR = "foo!";

or set them outside of the node-red process using whatever method is appropriate for the platform you're running on.





--
http://nodered.org
 
Join us on Slack to continue the conversation: http://nodered.org/slack
---
You received this message because you are subscribed to the Google Groups "Node-RED" group.
To unsubscribe from this group and stop receiving emails from it, send an email to node-red+unsubscribe@googlegroups.com.
To post to this group, send email to node...@googlegroups.com.
Visit this group at https://groups.google.com/group/node-red.
To view this discussion on the web, visit https://groups.google.com/d/msgid/node-red/2666bc84-0240-4bc5-8f9d-5ec3e72f2f4a%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Dave C-J

unread,
Oct 20, 2016, 5:54:36 PM10/20/16
to node...@googlegroups.com
that should be "... This technique does NOT support partial substitutions..."

also the file nodes do support the filename being set using msg.filename as a property of the msg - (as long as you leave the field in the config blank) - so you can compose it in a previous node dynamically if you wish.

Julian Knight

unread,
Oct 20, 2016, 6:03:06 PM10/20/16
to Node-RED
Feel free Mike though I'm not sure when I'll get to do anything with it ;-)

Mike Bell

unread,
Oct 20, 2016, 6:09:36 PM10/20/16
to Node-RED
Nick and Dave,

Thanks very much for the help. I figured out the "NOT" by trial and error. I have been using the msg.filename property. Now all I need to do is find a not-too-devious way of building the filenames I need.

I had hoped you would be willing to consider enhancements of the default file nodes. I have started a wishlist and will post it here as soon as it stops growing.

Regards,
Mike

Dave C-J

unread,
Oct 20, 2016, 6:25:10 PM10/20/16
to node...@googlegroups.com
Mike, we're always willing to consider enhancements, as long as they don't add too much complexity, or diverge from the original purpose.

Julian Knight

unread,
Oct 20, 2016, 6:27:52 PM10/20/16
to Node-RED
I'd be happy to help work on feeding back into the core file nodes. I'm certainly not precious about things - its just that I can see some gaps that really do need to be filled especially in relation to file nodes.

It would be good to have a place to put some thoughts on file nodes - is there somewhere you prefer?

Nicholas O'Leary

unread,
Oct 20, 2016, 6:30:16 PM10/20/16
to Node-RED Mailing List
It would be good to have a place to put some thoughts on file nodes - is there somewhere you prefer?

Please share them here.

--
http://nodered.org
 
Join us on Slack to continue the conversation: http://nodered.org/slack
---
You received this message because you are subscribed to the Google Groups "Node-RED" group.
To unsubscribe from this group and stop receiving emails from it, send an email to node-red+unsubscribe@googlegroups.com.
To post to this group, send email to node...@googlegroups.com.
Visit this group at https://groups.google.com/group/node-red.

Dave C-J

unread,
Oct 20, 2016, 6:47:08 PM10/20/16
to node...@googlegroups.com
Mike

a "quick" way to get ENV variables in would be to us an inject node and set the topic field to $(YOUR_ENV_VARIABLE)  then in a following function node extend / manipulate / store it as necessary before combining with the real flow as msg.filename 

Mike Bell

unread,
Oct 20, 2016, 7:05:09 PM10/20/16
to Node-RED
Dave,

Perfect -- or at least good enough. I had that working just as you posted. I plan to use the function node to construct several filenames and send them out as flow variables. At startup, I want to have an exec node touch those files so that subsequent reads before writing will not generate errors. (Perhaps not the best logic, but something that can happen in one of my flows.) Thanks again for your help.

Mike

Julian Knight

unread,
Oct 21, 2016, 4:42:37 AM10/21/16
to Node-RED
Here is my starter list then:

  • We need a node to do file/folder list/search operations - with multi-msg and array options for output. Supporting wildcards/globbing.
  • A node to do file copy operations
  • A node to do file move operations
  • A node to do file delete operations
  • A node to create folders
  • A node to create links
  • A node to return file/folder information including size, type, ACL
  • An updated version of Tail that supports Windows
  • All file nodes should be able to take inputs such as the file name to operate on & should recognise the current OS users home folder where appropriate (probably only on Linux, maybe Mac? since on Windows environment vars can be used) with a note in the docs to make it clear that the "user" concerned is the one running Node-RED.
  • All the nodes should support globbing where appropriate

On Thursday, 20 October 2016 23:30:16 UTC+1, Nick O'Leary wrote:
It would be good to have a place to put some thoughts on file nodes - is there somewhere you prefer?

Please share them here.
On 20 October 2016 at 23:27, Julian Knight <j.kni...@gmail.com> wrote:
I'd be happy to help work on feeding back into the core file nodes. I'm certainly not precious about things - its just that I can see some gaps that really do need to be filled especially in relation to file nodes.

It would be good to have a place to put some thoughts on file nodes - is there somewhere you prefer?

On Thursday, 20 October 2016 22:27:05 UTC+1, Nick O'Leary wrote:
I think we'd much rather ensure the default file nodes are as useful as they can be.

If you have specific feature requests, lets look at them in the context of the core nodes in the first instance.

--
http://nodered.org
 
Join us on Slack to continue the conversation: http://nodered.org/slack
---
You received this message because you are subscribed to the Google Groups "Node-RED" group.
To unsubscribe from this group and stop receiving emails from it, send an email to node-red+u...@googlegroups.com.

To post to this group, send email to node...@googlegroups.com.
Visit this group at https://groups.google.com/group/node-red.

Dave C-J

unread,
Oct 21, 2016, 4:51:26 AM10/21/16
to node...@googlegroups.com
Good list - though they are all file manipulation things (apart from tail) - rather than file IO which is what the existing nodes are. So feel different to me. Let's see what Mike has also.

Julian Knight

unread,
Oct 21, 2016, 5:07:26 AM10/21/16
to Node-RED
Well that was my original thought which is why I started work on them - but I do agree that whether contributed or core, it is best to have nodes have similar approaches.

Sadly, my workloads have gone crazy since I started work on node-red-contrib-fs which is typical I guess. Always happy to take contributions though or to contribute to other work.

steve rickus

unread,
Oct 21, 2016, 5:42:37 PM10/21/16
to Node-RED
+1 for tail -follow to receive the content of (e.g. log) files as they grow...

I agree Dave, these feel different to me, too -- perhaps they are not so much "file IO" operations as "directory" operations?

Oh, and I would add the ability to create a tmp file (in a given folder or the system tmp) to Julian's list, to avoid having to generate unique names in an upstream function node. But perhaps that would be the default behavior if the filename was left empty.
--
Steve

Dave C-J

unread,
Oct 21, 2016, 5:53:11 PM10/21/16
to node...@googlegroups.com
Yes - that sort of classification. Which then naturally fits more like enhancements to the node(s) Julian started - so we should look to help enhance those.

(plus fixing tail - though I have yet to find a decent one that worked cross platform rather than have to install two dependancies- one specially for Windows which is/was the problem - will have another look though as it has been a while)

Julian Knight

unread,
Oct 24, 2016, 3:41:06 AM10/24/16
to Node-RED
So it looks like the outcome of this conversation is - yes, please add any suggestions to that repo. And if anyone has any code to contribute, I'm happy to add people to the repo as needed or to do pull requests, any additions will be suitably attributed of course.

I will also add an "issue" with the things I listed.


On Thursday, 20 October 2016 22:20:49 UTC+1, Mike Bell wrote:
Reply all
Reply to author
Forward
0 new messages