Proprietary Nodes?

[Aron Semle]

Node RED is a really cool tool with a ton of potential. One thing that could accelerate it's growth are proprietary nodes for protocols, databases, etc. The catch is these are not easy to write, and someone probably wouldn't do it for free. Is there a market for Node RED nodes? If no should there be?

Interested in your thoughts. Nick, I think I asked you this over twitter awhile back.

- Aron

[Shem Jamieson]

Interesting concept. I guess if there were nodes that solved big problems for people that they would definitely consider paying.

[Dave C-J]

Certainly our intention all along has been if people see value in adding extra to it then they may do so. I'm not sure it would accelerate the growth that much unless it was in a particular area that is very popular but currently unserved - though I'm happy to be proved wrong. 

Of course currently the nodes are just written in node.js (javascript) and html so "protecting" them - apart from general honesty and corporate audits could prove tricky, unless it was down in the 99c not worth ripping off range.

One of our "models" has always been that other products may like to provide a Node-RED node as a way to access their main product - whether they charge for the node and/or the product is down to them. IE give away the node as it's only of use if you buy the product.... Indeed some existing nodes only make sense alongside (or running inside) the product they were written for.

Are there any particular nodes you had in mind ?

[Aron Semle]

Great point about securing the source, and vendors providing nodes to their own APIs.

I was looking for nodes to push data into Hadoop, Cassandra, InfluxDB, etc. Some of these support HTTP interfaces, so with a little work you could do it, but companies looking to get data into these would pay for something 'easy'. Other nodes that would be interesting are those that pull data from IoT and other intelligent devices. A lot of times these devices are proprietary and 'opening' up access to the data is not the companies business model, but still very valuable for end users.

[Josh Conway]

Aron,

I've found there is software that already handles this, along with much more. There is something called Apache Nifi, which appears to be a Node-red-like system for running big data jobs. One of the neatest areas it handles is data provenance. So, you have data coming in, being transformed, modified, copied, what have you. Data provenance shows you the path your data took from intake to output.

So now, why not just use Apache Nifi? At least in my case, I'm using Node-red for building sensor acquisition. MySensors combined with serial input, along with simple javascript programming really is just that powerful, and easy for front-end web programmers to get. I am looking on using Apache Nifi for log ingestion and processing, as Node-red doesn't have constructs like "load to HDFS" and "process dataset on Hadoop".

Sincerely,
Josh Conway

[Josh Conway]

> I'm using Node-red for building sensor acquisition.

Ok, that wasn't very clear how I said it. I'll explain in some detail.

I'm using an Arduino nano, hooked up to a nRF24L01+ with the MySensors library (npm:node-red-contrib-mysensors). I have about 200 arduinos outputting data to a gateway per floor (3 floors).

Each gateway is running node-red, converts data using the contrib-mysensors, and then sends the data via MQTT to a main node-red flow.

The main node-red flow is the "building node-red controller", where the data is processed and transformed to a simple website for a "view from above" at all the sensors.

Where Apache NiFi can come in to play is all of those logs that are generated: I ingest them into HDFS and process on Hadoop, and I can start looking at historical trends and other "interesting" data. Node-red doesn't really do this, but that's OK. It does everything before this like a champ.

On Tuesday, September 29, 2015 at 9:06:49 PM UTC-4, Aron Semle wrote:

[Robert Weiman]

I was kind of surprised that there wasn't already an InfluxDB node.  I started working on one that wrapped one of the InfluxDB node.js libraries but got diverted into doing a bunch of python stuff instead of node-red for the PoC.  I'll get it sorted out, or just build one from scratch talking directly to the HTTP layer eventually.

If there really is proprietary code that you'd want to protect, I'd expect a valid approach would be to put it in a compiled library (C/C++) with a node.js wrapper.  I'm considering that approach for some of the sensors we're playing around with.  Have low level libraries and/or daemons as necessary, and then have a node.js/node-red control layer.  That way even "non-devs" can play with configuration and data flows, while the lower level code handles the tighter timing constraints.

On Wed, Sep 30, 2015 at 10:40 AM Aron Semle <seml...@hotmail.com> wrote:

Great point about securing the source, and vendors providing nodes to their own APIs.

I was looking for nodes to push data into Hadoop, Cassandra, InfluxDB, etc. Some of these support HTTP interfaces, so with a little work you could do it, but companies looking to get data into these would pay for something 'easy'. Other nodes that would be interesting are those that pull data from IoT and other intelligent devices. A lot of times these devices are proprietary and 'opening' up access to the data is not the companies business model, but still very valuable for end users.

--
http://nodered.org
---
You received this message because you are subscribed to the Google Groups "Node-RED" group.
To unsubscribe from this group and stop receiving emails from it, send an email to node-red+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Julian Knight]

Ack! I really wish Apache would give up their insistence on using Java for everything. I've seen so many issues, version lockin and excessive costs associated with Java based products that I go out of my way to avoid it (Sorry IBM'ers I know that IBM is heavily invested into the Sun/Oracle maze, indeed that is where some of my aversion comes from - dealing with bespoke IBM developments).

It is a particular killer for embedded platforms where resources are at a premium.

Of course, NR stems from a desire to encourage experimentation with IoT and MQTT. Being Node.js and JavaScript based enables it to be pretty agile and usable on minimal platforms like the Pi and inevitably makes it a great platform for hobby use.

It would be great to see more integration for NR though personally, I'd like a focus on open protocols and databases not proprietary. 

There is already pretty good support for databases in NR: MariaDB/MySQL, SQLITE, REDIS, MongoDB, LevelDB, etc.

Protocols? MQTT, Websockets, HTTP(S), TCP - what else is needed? :)

Personally again, I'd like to see more consolidated processing nodes - ways to do things in a single step that currently take many steps & so make creating things in NR more complex. I'd certainly like to see more transform type nodes that I think would take NR from just an IoT tool to being a more rounded and generalised information transformation tool and give it some ETL credence (Extract, Transformation & Load). An area that really lends itself to flow-based programming as the Apache tool indicates.

Makes me wish I was retired so I could spend more time creating stuff!

[David Robinson]

Hi Josh,

I also have been looking and playing with Apache Nifi, and was hoping you could further clarify why you use both Node Red and Apache Nifi, as Nifi appears it can do (or integrate with things that can do) anything that Node Red can do.   There must be some 'dimension' I have not thought through yet on why Red is needed in addition to Nifi.

What is your take on maturity or breadth of support of connectors of one vs. the other ?

Thanks !

[Josh Conway]

Gladly.

All of this research is based upon my work and experimenting in this area, so YMMV. I dabble in a few different areas. Mainly IoT, machine learning, and computer vision.

So, the first thing is I look at the current state of the art in IoT. And a few things came to mind. There are a ton of online based flow controllers. IFTTT and friends. Then there were these node.js controllers. My first experience with them was with Hark.jp sound audition stack. That was my first foray into node.js graphical programming. So far, so good.

I came across Node-Red after (enter work stuff with Citrix/Octoblu). Long story short, I understand the basics rapidly, and start making things in NR rather than Octoblu. Because this ecosystem is more mature than theirs... and node-red-contrib-npm (yeowsa).

I also came across Apache NiFi. I ran across it when it was still in incubator, perhaps 3 months after being decommissioned from the NSA. I also play around with this kit as well, however documentation is a lot more scarce.. No, Apache has better 'Apache Documentation', but less on what you can do with it rapidly. And it also offers less local services, as you need "Big Data/Processor/API" sources set up to really use this kit. The gist is that ApacheNiFi feels like a Strategic Data Controller whereas Node-Red feels like a Tactical Data Controller.

As an example. Say I run all the data for all corporately owned McDonalds. Each store may have dozens or hundreds of sensors and datapoints. These sensors connect via wifi, USB-serial, Bluetooth. You have camera systems available. Audio recording is done. (We're assuming the worst here...) Now, With these local datapoints, it is easy to imagine what nodes and how your flows would be constructed within this store Node-Red. And most stores within the region would be roughly the same.

So, At a store level, we run NR. And at the district level (group of stores), is where you would first see NiFi running. We now acquire data from, say, 10 stores. Now, we can connect the store data in to our analytics. And if we have alerts, we can fire them off as we need within NiFi. Above this, would be a regional level. So a group of districts' NiFi flows would then fed into a regional NiFi flow. And on up to international.

NiFi just feels heavy-handed, but well placed with large data jobs. I've fed it 20MB/s for hours on a Intel NUC with no computer slowdown. I've done similar tests to Node-Red, when I piped webcam data through it from multiple devices... Lets just say Node.js didn't fare so well. But when it comes to connecting web APIs, local hardware stuffs, and glue logic, Node-Red really can't be beat. Going up to higher abstractions is difficult though. Fortunately, you pass that heavy workload from NR to NiFi and problem solved.

Sincerely,
Josh Conway

[Julian Knight]

Great summary and overview Josh, thanks for laying that out for us.

On Thursday, 7 January 2016 21:00:25 UTC, Josh Conway wrote:

Gladly.

...