Node for storing/caching reference-able JSON Objects

[Scott C. Lemon]

As we are creating more and more flows for a diverse set of sensors, a common pattern is that we see data from different sensors at very different rates.  Because of this, we end up having to "cache" the values of the incoming JSON sensor data into a context (flow or global) and then create other flows, triggered by inject nodes, that gather the "latest" data values from the various sensors to then perform their computation or other processing of the array of sensor data.

I've been wondering if there might be a good way to create a node, or have some other Node-RED feature, that would allow me to flow the entire JSON object, along with an identifier, into some new "context node" that would store the entire JSON object in either a flow or global context that would allow it's properties to be referenced from anywhere else in my functions and flows.  The reference would use some context reference, the identifier, and then the path to the property we want to reference.

This would allow us to automatically "cache" the entire JSON object from the sensors at whatever rate they arrive, and then have our "processing" flow have access to them when needed.

Before we started to develop something like this ourselves, I wanted to start a discussion, and see if anyone else has a different pattern that might be better, or if this is something that might be on the roadmap for Node-RED itself?

Thoughts?

[Neil Kolban]

Howdy Scott,

A while back I had a similar puzzle and solved it using CouchDB.  CouchDB is a NoSQL database.  We can create a record in the database that is a JavaScript object and later retrieve it or delete it.  It strikes me that this may be exactly the cache you are looking for.  When you insert a record into the CouchDB database, you can either specify a key or have one generated for you.  Since the database is running in its own process, it is accessible by all the different flows.

What is your insert/retrieval/deletion rate of records?

[Dave C-J]

Scott

the delay node does have a "topic based fair queue" mode which will store messages into "queues" by msg.topic - and then release the next topic on an interval tick. If a new message arrives on that topic it replaces the value in that slot - but doesn't lose it's position in the release rota.

So each topic gets it's turn - and only the most recent value is sent. This may be a good place to start. However - it has been pointed out in a different thread on this mailing list that an extra option could be to release "all" the topics at the same time - so that it wouldn't matter if you had two topics or twenty - you would get whatever was there. Currently this isn't there... it's one topic per tick.

Obviously this is an in line thing so doesn't allow general access to properties from elsewhere - but maybe that isn't so necessary, if the release rate is now part of the event flow.

However what you describe is also almost exactly what the global or flow contexts can be used for today - you can store anything you like in there - remembering of course that at present they are volatile.

[Nicholas O'Leary]

Scott,

the Change node can be used to store values in context for this sort of scenario. If you have a predefined set of possible keys, you could use a Switch node to route messages to appropriate preconfigured Change nodes to achieve what you want.

What it really needs is the ability to dynamically determine the property name - so you can do it all with one Change node that varies what flow/global property it sets based on a property of the message. We already support {{mustache}} syntax in some nodes, so it wouldn't be a great leap for us to add that sort of thing to the Change node.

With that you wouldn't need to create a new node.

Nick

--
http://nodered.org
 
Join us on Slack to continue the conversation: http://nodered.org/slack
---
You received this message because you are subscribed to the Google Groups "Node-RED" group.
To unsubscribe from this group and stop receiving emails from it, send an email to node-red+u...@googlegroups.com.
To post to this group, send email to node...@googlegroups.com.
Visit this group at https://groups.google.com/group/node-red.
For more options, visit https://groups.google.com/d/optout.

[Scott C. Lemon]

Neil,

We've looked at using an external database, but love the convenience of the flow and global contexts and not having to add code to make the external calls.  Our data rates are not significant at this point ... human speeds of minutes.  :-)

The issue that we have with the current flow and global context model is that there is so much overhead in setting up the storage on a property.  That being said I just realized that we have never tried to simply store the entire payload object into a context!

We're going to do a little more experimenting before we jump outside to a full blown DB ... but I do agree we see the potential benefits.

[Scott C. Lemon]

Dave,

It's funny but your post triggered the realization of what we've been doing - potentially - wrong!

Per the documentation examples, we've been thinking only of storing simple variables in the flow and global contexts ... not whole objects!  Duh!

We're now going to go back and refactor some flows for testing, and simply store the entire msg.payload into the context so that we can retrieve and access it later.  Now, this still doesn't allow us to simply reference a property of that stored object as we still have to do the get() operation ... but it's closer and better than what we've been doing!

Thanks!

[Scott C. Lemon]

Ok ... and now I'm reading up on the Change node ...  :-)