storage of flows & nodes

[Julian Lyndon-Smith]

I'm looking into allowing node-red to use a database such as mongo to store the flow and node information. Been having a look at storage/localfilesystem.js and have a few of questions

#1) Am I right in assuming that a "deploy" does both a save of the flow & deploy the nodes ?

#2) All flows are reads from and written to a single file (specified in flowFile)

#3) the "saveFlows" function seems to get called even if I deploy just flows with changed nodes , rather than the expected "saveFlow" function. Is this by design or a bug ?

#4) the "getAllFlows" function seems to be called for all reads, I haven't seen a call to getFlow() in use yet (same as #3)

#5) are sheets just a ux to allow flows to be designed in comfort ? :) ie they don't serve any purpose but to seperate flows from a ui point of view

#6) looks like there is no seperate storage for nodes - from the contents of the .json file it looks like all the nodes are stored in the flow itself, correct ?

#7) what are the getLibraryEntry and saveLibraryEntry functions for ? I can only find a reference to them in api/library.js

thanks - sorry for so many questions but I want to make sure that I have the architecture mapped out in my mind correctly ;)

Julian

[Nicholas O'Leary]

Hi,

Quick response as I'm out and about.

This topic has come up a couple times in the last week.

I'm currently documenting this api - will be publishing it in the next day so.

We already have a mongo storage plugin in our node-red-bluemix repository - although it's slightly out of date.

Will reply more fully when I'm near the laptop this afternoon.

Nick

--
http://nodered.org
---
You received this message because you are subscribed to the Google Groups "Node-RED" group.
To unsubscribe from this group and stop receiving emails from it, send an email to node-red+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Julian Knight]

I'm really interested in this too. I'd very much like to use MongoDB to store flows.

Even better if the pluggable storage module created a persistent connection that was available to function nodes and reused by the appropriate db nodes ;)

[Dave C-J]

The storage of flows is an admin function... accessing data is a runtime function... they may well need to be secured differently and held in different databases so access would be via a different connection.

[Julian Knight]

Spoil-sport! OK, I get that.

[Nicholas O'Leary]

Some specific responses to your questions...

>

#1) Am I right in assuming that a "deploy" does both a save of the flow & deploy the nodes ?

Yes

>

#2) All flows are reads from and written to a single file (specified in flowFile)

Yes

>

#3) the "saveFlows" function seems to get called even if I deploy just flows with changed nodes , rather than the expected "saveFlow" function. Is this by design or a bug ?

>

#4) the "getAllFlows" function seems to be called for all reads, I haven't seen a call to getFlow() in use yet (same as #3)

saveFlows is used to save your flow configuration, regardless of the type of deploy that is performed. The "deploy only changes" only affects how NR decides what nodes need to be stopped/started.

saveFlow/getFlow/getAllFlows are functions of the flow library - the import/export to library option in the menu. These functions are about to be deprecated as they are overly specific versions of get/Save/LibraryEntry (more on those later).

#5) are sheets just a ux to allow flows to be designed in comfort ? :) ie they don't serve any purpose but to seperate flows from a ui point of view

Currently yes.

#6) looks like there is no seperate storage for nodes - from the contents of the .json file it looks like all the nodes are stored in the flow itself, correct ?

The node configuration is stored in the flow - that is what the flow is.

#7) what are the getLibraryEntry and saveLibraryEntry functions for ? I can only find a reference to them in api/library.js

The function/template nodes allow you load/save content - make it easier to reuse content locally. There is an api for nodes to register a type that should make use of the library - if you look at the "how to create a node" documentation overview, you'll see one of the todo items is "library documentation". Anyway, this pair of functions is how the library content is saved/loaded on a per-type basis.

As I mentioned, Im in the process of updating and documenting this API - which is what drove the deprecation of bits of it as I realised how unnecessary they were. We will also be moving the couchdb and mongo storage plugins we have hidden in the bluemix repository into their own repo to make them easier for people to reuse.

Nick

[Julian Lyndon-Smith]

Thanks for this Nick.

I've added a comment to your latest commit (hope you don't mind) , but I was also wondering if there is any documentation on what the expected return value is on each of the functions - or is this part of the "updating and documenting this API" ?

For example, I see that listFlows returns an object containing a set of "d" objects and an "f" array . Are these directories and files ? If so, what is the significance of the Directories ? If the storage layer is a database, are directories relevant , or are they used as some sort of category ?

thanks

julian

[Julian Lyndon-Smith]

now that I've put some diagnostics in, looks to me as if listFlows is actually "list flows in library" and the "f" stands for flows, not files

onto the "d" ;)

julian

[Nicholas O'Leary]

Hi Julian,

the first pass of the storage API docs is now available here: http://nodered.org/docs/api/index.html

The library content is represented as a virtual file system that can be browsed, hence the representation of 'directories' and 'files' (or indeed 'flows')

The (now deprecated) getAllFlows() function of the storage api would return a complete listing of all flows in the library. This used the format of an object with 'f' properties and 'd' properties to separate files (flows) and directories at each level of the listing. I was never a fan of that format, it was a quick hack early in the days of node-red that I never got around to tidying up. It was also inconsistent with the getLibraryEntry format.

As part of documenting the API, we have deprecated getAllFlows, so that new implementations should not implement it. Instead, they only need to implement getLIbraryEntry. However, we have to maintain backwards compatibility with existing implementations, so the storage layer checks for the existence of getAllFlows and uses it if it is there. If getAllFlows isn't there, it uses the internal function 'listFlows'  to build up an equivalent response that getAllFlows would have returned, but using calls to getLibraryEntry. That is a temporary measure - I will be changing the format the internal code uses shortly, at which point there will be code in the storage layer to reformat what getAllFlows returns. But that is all internal detail that isn't relevant to storage implementations.

I'm also in the process of publishing our mongodb and couchdb storage plugins to their own repositories and npm modules.

Nick

[Julian Lyndon-Smith]

cool. thanks for this.

julian

[Rahul Kumar Prajapati]

Hi Julian 

 did you success to store to flows in mongodb. i am facing same issue please help me how to do it?

Thanks 

Rahul

[|\|370]

Here is someone actually interested in storing Nodes inside a DB instead of inside a FS. I'd like to have all the node files stored as objects inside, for instance, MongoDB. Does that make sense to any of you guys?

[Julian Knight]

Hi, yes it makes sense certainly!

MongoDB seems like a fairly sensible choice apart from one really annoying issue. There still doesn't seem to be an ARM version for Raspberry Pi's. Currently, I have to run my MongoDB instance on my Synology NAS (using Docker now, a neat addition by Synology). I would much prefer to have everything on my Pi and use the NAS as a backup.

[Timur Fatykhov]

I have mongodb on raspberry pi - http://c-mobberley.com/wordpress/2013/10/14/raspberry-pi-mongodb-installation-the-working-guide/
it takes time to compile but it is working.

[Julian Knight]

Thanks Timur, I think I tried that previously and it didn't work for me, perhaps I'll try again. However, you should see some of the issue comments from the developers, they don't make for happy reading re ARM code. The Pi2 uses ARMv7 I believe? They specifically call that out as being never fully supported due to the way MongoDB is coded.

[Julian Knight]

I've just noticed that the version in that article is now 4 years old. So it equates to around v2, maybe 1.8

The current version is v3.2

[Julian Knight]

Of course, I should have checked apt-get for Rasbian. In fact, mongodb server v2.4 appears to be included. Installing now.

[Shyam Nagarajan]

Hi Nick,

I have just started to play around with Node-RED. I would like to use the storage API to store the flow and node information in a database as against the local filesystem. I have deployed Node-RED on Google Cloud. Can you kindly point me to the approach that I should follow to use MongoDB to store the Node-RED information?

Regards,

Shyam