Node-Red usage for home automation

[Oliv']

Hi,

 

This email is not directly related to NR but on one of its application that people seems to discuss on the group : home automation, and how people are interfacing it with NR

 

 

Home automation is composed by

1. Sensor & actuators (Huge choice from stock, home made, etc...)
2. Database
3. Man user interface
4. Data processing
5. Server/computer/cloud

1. Sensor & actuators

Huge choice from stock, home made, etc...

I am currently waiting for some PCB  designed

2. Database

I tried SQLite for some embedded projects, and it is good for some data, but not for storing too much value from sensors

I don't know how MySQL and equivalent performs on small embedded devices

I just discovered timeseries database like influxDB which are great for storing sensors values, but I don't know how it performs too

Some people are using MongoDB , I never used it

3. Man user interface

I am still seeking for a great tool, Freeboard might be one but never tested.

Also I might do something from scratch

4. Data processing

That is the difficult part, how to analyse all the data to have something useful and not just some temperature graphs on one page, power on another etc....

Is seems that Open Energy Monitor and Open Sens.se are exploring that way but the later is only on the cloud, closed source.

5. Server/computer/cloud

Will have an impact on the database and data processing choice regarding the computing power available

Cloud might not be good as you will depend from other people an your internet connection

So if some people already developed such system around NR  or are planning to do it, which technologies/software do you use ?

Thanks

Oliv'

[ryan...@gmail.com]

Hi Oliv'

My project is very much work in progress and is mostly 'home control' rather than 'home automation' at this stage, but I have played with various different technologies and interfaces whilst exploring possibilities, which may be interesting to some.

I had aspirations of designing the system to be modular, scalable, decoupled etc. It is some of the way there but I have taken a few short-cuts because it was becoming overkill for home use and I wanted to make use of some existing tools. To give you an overview of my design:

I have a distributed system, all talking a JSON data model over MQTT. There is a NR status server, which subscribes to MQTT messages published from other devices (Android apps, Arduino sensors/actuators, RPis etc.). These messages are JSON objects that encapsulate the necessary command information to do something: request a status, update a status, trigger an actuation etc. This command information is functionally abstracted, no interface specific content is present. The NR server has a simple flow to act on these messages, such as retrieving a state and sending a reply, or sending a command on to a specific interface for output, whilst updating its own internal state as it goes. The interface devices interpret these messages and translate into device-specific commands. To give an example:

- Android app publishes a message to dim the lights: {"command":"dim","level":{value}, "device":"lwrf/mainroom/central/spots"}.

- NR status server is subscribed to all messages, stores the new state and then constructs a status update to return to the Android app.

- Arduino LWRF controller is subscribed to all lwrf/# messages and translates the command to an interface specific command, which is transmitted to the LWRF dimmer over 433mhz.

The abstraction becomes useful when adding multiple interfaces to the system that have similar functionality. If for example I want to dim some LED strip that is controlled by a different interface (not LWRF), I can publish the same {"command":"dim","level":{value}... command from the Android app but with a different target device string (e.g. led/bedroom/ambient). An appropriate controller, in this case a different Arduino running an LED strip driver sketch, is subscribed to led/bedroom/# and will translate the command message into an output specific for driving the LED strip.

Unfortunately a big compromise I have made is the use of TCP sockets rather than MQTT on the Android remote control app. I have opted to use a web-based app design tool rather than making my own and this is limited to TCP/UDP/HTTP communication, no MQTT. This complicates things a bit with a TCP->MQTT bridge on the NR server. In time I might 'do it properly', but this works well for now.

To follow your categories and elaborate on hardware/software:

1. Sensors & actuators

1-wire (RPi host, over CAT5): temperature sensors, TTL input, output relays

LightwaveRF: PIR sensors, proximity sensors, socket relays, in-line relays, lighting dimmers, radiator TRVs

Z-Wave: lighting dimmers

Custom (Arduino with ethernet & MQTT): Contact sensor, humidity sensor, LightWaveRF 433mhz controller, curtain open/close motor, RGB LED strip driver

Android: Owntracks location reporting, NetIO control GUI

2. Database
Currently I use a hashmap data structure in global.context to store states against topics. Will probably move to Redis DB for persistence when I have time. I also use maps on the Arduino's to match devices against specific commands.

3. Man user interface

I use NetIO (Android & iOS app) as the primary control interface. This app lets you build simple control GUIs using a drag & drop web interface, you can then create connections in TCP/UDP sockets or HTTP/S requests (I have requested MQTT support but no luck!). It's really quite good and flexible enough to create a basic interface with  no knowledge of Android development.

4. Data processing & 5. Server/computer/cloud

I'm not quite at this stage yet. Freeboard is on my list of things to play with. 

I'm personally not so interested in having metrics on a page, I'm more interested in giving the actual automation processing some structure & formality and this is an area I'm most interested in hearing other's views/experience on. I don't want a messy system with lots of chained conditional logic to decide actions (e.g. If mode is 1 location is x and date/time is y and light level is z etc...). I have been playing with a rules engine in node, which on the surface seems like it could help. Can't help but feel there is a better option for dealing with this sort of logic and I've just not found it though...

Cheers,

Ryan.

[Bluefox Fox]

Hi Oliv,

we are working too on the Home Automation solution - ioBroker, that includes node-red, but not as a main server, only for logic.

Project has predecessor, known as ccu.io (The documentation unfortunately only in German/Russian) and has some community in german.

The new ioBroker-Project is in progress and eliminates all disadvantages of ccu.io. It is modular, can use DB, file or something else as storage (now used text storage, but redis+CouchDB will work too). It can run on different HW and can be distributed over different hosts. E.g. main server runs on cubietruck and 1-wire temperature sensors can run on rPi and rPi sends data to cubietruck.

Some basic description can be found here http://www.iobroker.org

We are working now on the "Man user interface" - ioBroker.vis. It will look like this one http://dashui.ccu.io/ , but much better and comfortable to use.

1. Sensor & actuators

Just now supported Homematic, Homegear, KNX, z-wave and everything that can be connected via node-red.

2. Database

As Database is used now just the text file(JSON). But we had the running system with Redis and CounchDB. It was disabled, because users have a problems with installing these DBs. Will be enabled later again.

3. Man user interface

Just now GUI is ioBroker.vis (the next version of DashUI) and it has main priority, to get it work. We want to develop some GUI for mobile devices too. The predecessor for it exists too. (http://homematic-forum.de/forum/viewtopic.php?f=48&t=13965&p=108417&hilit=yahui#p108417)

4. Data processing

What exactly is desired here? We plan to connect Grafana+InfluxDB for this purpose. Or you can use now "Graphite"  http://graphite.wikidot.com/screen-shots. ioBroker supports it too.

5. Server/computer/cloud

ioBroker can run on any linux or windows system, where node.js is installed. Even on rPI (but installation is very long)

Cloud... Does not planned, but why knows.

You can install ioBroker like this on linux or windows:

mkdir /somewhere/iobroker

cd /somewhere/iobroker

npm install iobroker

Of course node.js and npm must be installed before.

You can ask about ioBroker here http://forum.iobroker.org/. 

And please don't forget it is still very much work in progress.

Greetings,

Bluefox

 

---------------------------------------

[Scott Penrose]

On Monday, January 26, 2015 at 9:03:59 PM UTC+11, Oliv' wrote:

1. Sensor & actuators

Huge choice from stock, home made, etc...

I am currently waiting for some PCB  designed

My attraction to Node-Red is the ease of which I can add sensors. Many other projects allow you to add modules, but Node-Red I just created my own flow - no waiting.

 

2. Database

I tried SQLite for some embedded projects, and it is good for some data, but not for storing too much value from sensors

Very surprised you are having trouble with large data with SQLite. SQLite limitation is more about multiple concurrent users. Not really an issue with NodeJS solutions, but could be. What it should not be is a problem with size or performance, not on a Raspberry Pi.

 

I don't know how MySQL and equivalent performs on small embedded devices

I have successfully run MySQL on a Pi but with the extra memory and separate process I don't get as good a performance as SQlite. But I would try and be flexible and grow as necessary. eg.. you could run a second Raspberry Pi and MySQL, or use a Beagle Bone (faster, more memory) and run MySQL and Node-Red.

 

I just discovered timeseries database like influxDB which are great for storing sensors values, but I don't know how it performs too

Some people are using MongoDB , I never used it

Once you start using aggregate queries to do your reporting you will find it easier and faster to use SQL. But it does take a little more effort at the start to add the objects. Adding just JSON to MongoDB is obviously easier than creating tables etc. All of them have their place and you should probably think about end to end and where your existing knowledge is. Or what is supported by a project. 

 

3. Man user interface

I am still seeking for a great tool, Freeboard might be one but never tested.

Also I might do something from scratch

Today I built a flow that allows me to read/write Freeboards to files managed by Node-Red - it has tripled my productivity with Freeboard. I am finding Freeboard frustrating in some ways - e.g. wanted to just count objects coming in? or show a graph with some old data (from dB) then update from WS. Or splitting data so one WS has all the sensor updates but I show RSSI in one graph and Temperature in another - I am sure it is all possible, I am just finding it difficult and hard to get support.

But then there is everything else. A dashboard of switches for example - to allow manual turning on and off, or seeing the state of things. Freeboard doesn't seem the best for that (would require a lot of plugins).

I would like a graph processor that would allow me to filter on sensor type, to filter by node id etc - this is something I would attach to the database.

 

4. Data processing

That is the difficult part, how to analyse all the data to have something useful and not just some temperature graphs on one page, power on another etc....

Is seems that Open Energy Monitor and Open Sens.se are exploring that way but the later is only on the cloud, closed source.

All the processing I have required has been possible with a combination of either an SQL query or using a flow. Of course if I wanted to do OpenCV processing of an image or something, neither of those would help.

 

5. Server/computer/cloud

Will have an impact on the database and data processing choice regarding the computing power available

Cloud might not be good as you will depend from other people an your internet connection

So if some people already developed such system around NR  or are planning to do it, which technologies/software do you use ?

Writing a monolithic application is always easier (to write). And with that ease comes more power, at least more power more quickly. But I am a unix type of guy, do one thing well. As such I am looking for the right sensor dashboard, and the right graphing / charting application for analysis of my database data.

Good luck.

[Julian Knight]

1. Sensor & actuators
Totally agree that one of the real draws of NR is the flexibility it brings - I initially tried out Domoticz - Liked it for the first few days then got totally fed up. It is slow, sometimes hangs totally & is hard to do even simple things like getting combined charts. Great if you don't want to get your hands dirty though.

2. Database
While SQL is easier to do "straight" aggregations, that assumes that: a) you know SQL, b) you have data that fits comfortably into a relational table format. Also, performance is highly dependent on having enough memory for indexes once you reach into the many thousands of entries. I've never used SQLlite for really large numbers or records, certainly not on an embedded platform so I'm not sure how it handles. Certainly something like LevelDB (which I now know that InfluxDB uses) is likely to be more performant for time series.

In terms of aggregation queries in MongoDB, this is actually pretty straight forwards as well. There is a section in the manual that covers it well. Map-Reduce can be a bit hard to get your head around initially but no worse than learning SQL. I think we probably need some additional nodes in NR to handle this well though.

3. Man user interface
Always going to be hard. I've tried Freeboard but again, it is very limiting, better off just writing something yourself.

In the end, I always find myself starting something with plain HTML/CSS/JS. That's what I'm doing now. With a bit of help from JQuery, Twitter Bootstrap and a library for doing nice, compact, bullet graphs. I'll add some sparkline charts for time series data in a bit.

I've used JQuery to dynamically create a list of floors & rooms with all the display hidden until I receive some sensor data for a specific room. Then it magically appears just by taking the "hidden" class of the appropriate DIV's.

4. Data processing
Yes, this is a pain. Cube looks like it might have been helpful if you go down the MongoDB route however, it is no longer being developed. It is probably that aggregation queries in MongoDB may be sufficient. There is an interesting example based on US road camera data. It also took me a while to find a useful Google search :)

However, I feel that the main thing to do from the outset is to think about and plan out how to normalise all of your data. Since NR relies heavily on MQTT and WS & Topics, it is sensible to ensure that you have a topic/path that makes sense to the data and to store that with the data. Consistent timestamps are also helpful. Consistent naming conventions will also help you a lot.

5. Server/computer/cloud
There is a lot of stuff I don't want on the cloud when we are talking about home automation. Also, I don't want to accidentally leak information about whether I'm home or not and I certainly don't want anyone to be able to control things in the house!!

So much better I think to do home automation & monitoring in-house - literally! I use my Synology 412+ NAS box with Node-Red and MongoDB installed to do most of the higher level work and produce the final user interface. I use a Raspberry Pi with Node-Red & Mosquitto (for MQTT, on the Pi because it isn't so easy to install on the Synology) to monitor actual sensors and control switches, etc. I have a Ciesco radio module for low cost radio comms to custom Arduino sensors and an RFXTRX433 for HomeEasy/LightwaveRF/Oregon Scientific devices.

I totally disagree I'm afraid that monolithic apps are easier to write. That's only true while the code is no more than a couple of screens long. In any case, the whole reason for NR is to make this a non-issue. The whole ethos of Node.JS is to use lots of small modules, NR follows this pattern using flow programming instead of traditional code.

[Scott Penrose]

On Wednesday, January 28, 2015 at 12:27:03 AM UTC+11, Julian Knight wrote:

All good points Julian. I also agree that as an off the shelf solution, SQL is not the right answer for people. Just like you can inject any JSON into MQTT, a DB the same I think is the right fit for standard NR.

I totally disagree I'm afraid that monolithic apps are easier to write. That's only true while the code is no more than a couple of screens long. In any case, the whole reason for NR is to make this a non-issue. The whole ethos of Node.JS is to use lots of small modules, NR follows this pattern using flow programming instead of traditional code.

Didn't explain myself very well sorry. I certainly agree with you. I think writing smaller special purpose things and joining them is far better, faster and easier. BUT what I mean is from a large company perspective. A large company can fix on their own protocol, and their own database, and their own queues etc. Imagine that Node-Red fixed exactly what a sensor looked like. This would then allow easy additions of a dashboard. While Node-Red is more flexible you can't therefore have an automatic designed dashboard. So I didn't really mean monolithic applications  - I really mean closed wall garden, where you have control over everything. You can build graphs, buttons everything - and then everyone must work your way.

Node-Red is more like Bash. You don't draw graphs with Bash, you use PGPlot - you can generate SVG in Bash, but is it the right tool, not really. Long story short, I like NR as it is - open to any type of integration and one small part of a big picture. Sorry I wasn't clear before.

Scott

[Julian Knight]

No problem Scott, there is always room for robust discussion! Glad we've clarified things.

As I generally work for large organisations, I've come more and more to appreciate the way of open source, open standards, open data and also the "Node way" of creating lots of small, dedicated tools linked together and the "web way" of using standards-based UI's (HTML/CSS/JS) with wel defined REST API's. An awful lot of big corporate projects fail due to monolithic design & build using proprietory tools and data schema's that vendors protect aggressively. As I now work for a public organisation, part of my role is to ensure good value for money for the public purse and I've already begun to change our IT standards to reflect the need for "open" where possible and sensible and to require a business case to justify why open hasn't been used where the option exists.

In fact, one of my concerns with Node-Red is the liberal use of IBM copyright. Thankfully IBM have a pretty good track record of supporting open source but it is still disconcerting to see.

Sorry, I've totally gone off at a tangent now :} I'll get off my hobby horse, normal service will now be resumed!

[Dave C-J]

Julian,

it's IBM copyright as Nick and myself (the co-creators) both work at IBM. However the license is Apache2 which is open and allows anyone to re-use it, including taking it and building into their own products. So basically "it's out there" .

[Olivier Gimenez]

Thank you all for your information, it is now clear that there is no universal dashboard :)

2015-01-26 15:03 GMT+01:00 Bluefox Fox <dog...@googlemail.com>:

[...]

Thanks, I will watch your progress

2015-01-27 11:24 GMT+01:00 Scott Penrose <scoot...@gmail.com>:

My attraction to Node-Red is the ease of which I can add sensors. Many other projects allow you to add modules, but Node-Red I just created my own flow - no waiting.

 

Yes, I totally agree 

2. Database

I tried SQLite for some embedded projects, and it is good for some data, but not for storing too much value from sensors

Very surprised you are having trouble with large data with SQLite. SQLite limitation is more about multiple concurrent users. Not really an issue with NodeJS solutions, but could be. What it should not be is a problem with size or performance, not on a Raspberry Pi.

It was on a Gumstix, an ARMv7 with eMMC. I don't remember exactly but we fed  

 

you should probably think about end to end and where your existing knowledge is.

 for myself, it is definitely SQL, but I like to learn new things :)

 

3. Man user interface

Freeboard doesn't seem the best for that (would require a lot of plugins).

Yes, in fact that is what in think missing in Freeboard, but maybe I could write something :) 

But if it must write too much code, maybe a design from scratch can be better...

I would like a graph processor that would allow me to filter on sensor type, to filter by node id etc - this is something I would attach to the database.

Yes it would be great ! Maybe have a look on some things like Grafana ? 

2015-01-27 14:27 GMT+01:00 Julian Knight <j.kni...@gmail.com>:

I've used JQuery to dynamically create a list of floors & rooms with all the display hidden until I receive some sensor data for a specific room. Then it magically appears just by taking the "hidden" class of the appropriate DIV's.

Really good idea ! 

2015-01-28 10:26 GMT+01:00 Dave C-J <dce...@gmail.com>:

it's IBM copyright as Nick and myself (the co-creators) both work at IBM. However the license is Apache2 which is open and allows anyone to re-use it, including taking it and building into their own products. So basically "it's out there" .

I am not very close to all those license/copyright and intellectual property things. But is there a possibility than IBM changes its mind on such subjects ? What can be the consequences ? 

 

[Julian Knight]

I got that Dave, don't mind me I'm too prone to preach about copyright issues (urr, and security issues, and ....) :}

I'm not that concerned here though I would note something of a chilling effect of having to sign and send in an "official" IBM form to contribute to an open source project.

[Nicholas O'Leary]

Julian,

Contributor License Agreements are by no means unique to IBM. You have to sign CLA's to contribute to projects maintained by both the Eclipse and Apache Foundations and many other open source projects out there.

This is a useful overview of why they are a good thing: http://oss-watch.ac.uk/resources/cla

Nick

--
http://nodered.org
---
You received this message because you are subscribed to the Google Groups "Node-RED" group.
To unsubscribe from this group and stop receiving emails from it, send an email to node-red+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Artyom Syomushkin]

Hi there, I'm happy user of OpenHab, but looking for some other software, because if my concept, shown below - OH is not very suitable for this. I would like to know of Node-Red is ther right thing for me.

1. Sensor & actuators

I'm currently using Z-Wave. In plans I want to use MQTT for ESP8266 based sensors and EnOcean for powerless sensors and switches.

2. Database

I think the main thing here is not the database itself, but logs with timestamp - e.g. it should be easy to see what happened with house and when, to diagnose any mistakes and errors. The logs can be produces from database, but also can be created much easier, for example using MQTT client, subscribed to all messages from Broker. In this case if all traffic goes through Broker, the logs will be created automatically.

Would be nice to know, how Node-Red supports this.

3. Man user interface

I have quite easy solution - WiFi tablet or Smartphone on the wall with GUI app. I stopped on Commandfusion right now, because I have made MQTT client for it, which works quite well - https://groups.google.com/forum/#!topic/commandfusion/vfPByGiBd2g

So all you need is to create your own Dashboard graphically and put a couple of MQTT calls there. Works quite fast and stable. 
As future solution I was thinking about Google Voice recognition using Tasker - my tests shown, that it works quite well. To extend the Mic Range I want to use Conference Bluetooth Speakers - then it should hear in whole room.

4. Data processing

If we talk about scenarios or rules - e.g. "real" automation, like opening and closing of Shutters on schedule, or intelligent thermostat control, alarm management, I have two solutions - local and cloud.
Local is done using Codesys runtime, running on Raspberry or any other PC. With this you can create logic and rules using one of the industry standard IEC 61131-3 languages and integrated IDE. With this IDE you can also easily debug your code in real-time on your controller. As interface with sensors/actuators I was thinking of using MQTT or Modbus TCP. The last one is supported from the box. Here I see Node-Red as important thing to convert all those different interfaces(Z-wave, Enocean, REST API for Web) to single one to be interfaces to automation controller.
For cloud control I was planning to use Thingspeak. It would be nice to use MQTT for that as well.

5. Server/computer/cloud

Here I have significant doubts on reliability in case of local server. Currently I run everything on Raspberry PI, but maybe will move something to my Synology NAS. Anyway redundancy is a must.

So my question to you guys - I see Node-Red as a tool, which will manage connections between hardware and software and bring them all to some standard single interface. It should help me to manage all those signals easily. Basically what I want is that internally I should not care to which sensor or which actuator belongs particular signal in my automation software. Only ant the Node-Red level I should connect them to real software. Am I right in my choice?

[Nicholas O'Leary]

> Am I right in my choice?

Yes. Sounds like just what Node-RED was created for.

Nick

--
http://nodered.org
 
Join us on Slack to continue the conversation: http://nodered.org/slack

---
You received this message because you are subscribed to the Google Groups "Node-RED" group.
To unsubscribe from this group and stop receiving emails from it, send an email to node-red+u...@googlegroups.com.

To post to this group, send email to node...@googlegroups.com.
Visit this group at https://groups.google.com/group/node-red.

[Tri Sooma]

Hi,

I am currently working on a home automation tool, it's in the pre-beta state - so no code has been published yet. Let me tell you a little bit about what I have envisioned for this project.

The concept for this system is that it consists of a server that consists of an API and a web interface, then there are modules - these can be physical nodes (RPi's, arduino's etc.) that perform specific functions like measuring or controlling things orsoftware modules that implements specific API's (Philips Hue, Vitadock, etc.). Communication between different parts is (preferably) done through MQTT and time series data is stored in RRD files. Basic configuration is done using text (.ini) files and other stuff ends up in a database.

The backend is written in Python and uses SqlAlchemy to interface with a database, so users have choice to use any database supported by SqlAlchemy.

For making this system move out of the realm of home control into home automation I want to use NR as the engine that processes messages and acts upon them by calling specific endpoints in my applications API. For this i want to create some NR-modules that will allow users to use my applications API without the need to write any javascript code. More tech-savvy users can roll their own nodes.

Then using you categories, let me tell you what I have so far:

1. Sensor & actuators

Stock: Philips Hue

Stock: Vitadock integration (let's users automatically track measurements taken by a body composition scale)

Custom: trying to figure out how to get some arduino and RPi nodes up (hardware is not my strong suit) - think simple temperature/humidity monitoring; PIR sensors maybe

2. Database

I have tested storing data in SQLite and MySQL so these work (I use MySQL for all my RDBMS storage)

3. Main user interface

custom built web interface that allows users to configure/manager the system and contains a simple dashboard with even simpler widgets (CommonJs / Marionette)

4. Data processing

Nothing here yet, users will set thresholds so the system can act upon them.

5. Server/computer/cloud

The server might run on a RPi (should test this really), but I am assuming users will have access to a server.

I really don't want to store anything in the cloud or do any processing there, but it's up to the user.

My 2c,

Trisooma

[Julian Knight]

See my comments below:

On Monday, 14 March 2016 08:38:02 UTC, Artyom Syomushkin wrote:

Hi there, I'm happy user of OpenHab, but looking for some other software, because if my concept, shown below - OH is not very suitable for this. I would like to know of Node-Red is ther right thing for me.

1. Sensor & actuators

I'm currently using Z-Wave. In plans I want to use MQTT for ESP8266 based sensors and EnOcean for powerless sensors and switches.

Obviously, the more network types you have, the harder it is to maintain them all. However, NR excels here since it becomes fairly easy to take each input type and normalise the data. That's what I do. See https://github.com/TotallyInformation/Home-Automation-Hub-Design for some info.

MQTT is the glue that enables you to de-couple inputs from processes from outputs. It also shows you a snapshot of the current status of everything.

2. Database

I think the main thing here is not the database itself, but logs with timestamp - e.g. it should be easy to see what happened with house and when, to diagnose any mistakes and errors. The logs can be produces from database, but also can be created much easier, for example using MQTT client, subscribed to all messages from Broker. In this case if all traffic goes through Broker, the logs will be created automatically.

Would be nice to know, how Node-Red supports this.

This always seems tricky. I've gone through several databases so far in my explorations. I want everything to run on a Raspberry Pi so things are a little limited. I was using MongoDB but it really doesn't work that well on 32bit platforms and I found out the hard way that it just crashes when it reaches some fairly arbitrary size. 

I'm now using InfluxDB which is designed specifically for timeseries data and currently (touch wood) seems to have no artificial limits on the data size. Though you do need to simplify the data somewhat - a good thing really. I'm using InfluxDB with Grafana which directly interfaces and it is easy to output tables and charts showing exactly what has been happening.

My hardware input goes to MQTT after being cleansed and normalised. A listener flow picks up new hardware inputs and republishes a few different ways in parallel to outputting to Influx.

3. Man user interface

I have quite easy solution - WiFi tablet or Smartphone on the wall with GUI app. I stopped on Commandfusion right now, because I have made MQTT client for it, which works quite well - https://groups.google.com/forum/#!topic/commandfusion/vfPByGiBd2g

So all you need is to create your own Dashboard graphically and put a couple of MQTT calls there. Works quite fast and stable. 
As future solution I was thinking about Google Voice recognition using Tasker - my tests shown, that it works quite well. To extend the Mic Range I want to use Conference Bluetooth Speakers - then it should hear in whole room.

This is rather less polished in NR. Many of us started with custom web pages which are very easy to build in NR. Some of us went further and started to build more dynamic pages with websockets based comms. However, in recent months an excellent set of nodes from Andrei Tătar came along and which is now due to be adopted into the core. UI provides a set of simple but effective elements for building UI's that are fairly mobile friendly and very easy to use.

There are a couple of people in this group who have experimented with voice inputs and outputs. Sounds like you have some experience of whole-room recording. My own professional experience indicates that it is a very hard thing to do well without specialist equipment. Echo's are the thing that kills conference room recording. The ear/brain filters them out totally but microphones, even good ones, don't. You can get specialist microphones of course and there are some great conference phone/video units that work really well, many of them now compatible with things like Microsoft Lync/Skype for Business. But they do cost a lot of money. Well spent money though if you need low-friction, high fidelity conference room recordings. Speaking as someone who spends inordinate amounts of time in remote conferences, I can say that a good setup means good communications, better understanding and a lot less tiredness at the end of the day.

4. Data processing

If we talk about scenarios or rules - e.g. "real" automation, like opening and closing of Shutters on schedule, or intelligent thermostat control, alarm management, I have two solutions - local and cloud.
Local is done using Codesys runtime, running on Raspberry or any other PC. With this you can create logic and rules using one of the industry standard IEC 61131-3 languages and integrated IDE. With this IDE you can also easily debug your code in real-time on your controller. As interface with sensors/actuators I was thinking of using MQTT or Modbus TCP. The last one is supported from the box. Here I see Node-Red as important thing to convert all those different interfaces(Z-wave, Enocean, REST API for Web) to single one to be interfaces to automation controller.
For cloud control I was planning to use Thingspeak. It would be nice to use MQTT for that as well.

No matter what you use to coordinate the comms, you will need to ensure that you have redundency built in. Thankfully, MQTT fully allows for this in the standards. 

Node-Red is, as you've seen, ideal for acting as a glue system to pull together the inputs from many different sources, normalise them and output them as required. This is bread-and-butter for NR. However, what isn't so good is that you seem to be centring much of this project around professional-level kit and services. If this were my project, I'd be slightly concerned about how I could make NR resilient. I expect it is doable but I've not looked into it as I'm only running some lighting and sensors at home. But this is particularly important if you are going to run all this on a Raspberry Pi as the weakest part of the Pi is the reliance on a MicroSD card. Those do fail with fairly monotonous regularity. So look to move all the writes off to an SSD or HDD and work out how to use a mesh of Pi's so that others can take up the load when one fails.

5. Server/computer/cloud

Here I have significant doubts on reliability in case of local server. Currently I run everything on Raspberry PI, but maybe will move something to my Synology NAS. Anyway redundancy is a must.

Ah, I should have read this part first :-)

The Synology NAS's are excellent pieces of kit for this kind of thing and they seem very reliable. Again though, make sure you are using high quality HDD's or SDD's, use dual network cards (which many of them have) and use dual NAS's for redundency. They don't have a lot of memory though typically so make sure you turn off and strip out everything you don't need & don't try to do too many things with them - I've fallen foul of that myself a couple of times! 

So my question to you guys - I see Node-Red as a tool, which will manage connections between hardware and software and bring them all to some standard single interface. It should help me to manage all those signals easily. Basically what I want is that internally I should not care to which sensor or which actuator belongs particular signal in my automation software. Only ant the Node-Red level I should connect them to real software. Am I right in my choice?

With the caveat about making NR itself resilient, yes I would say so. That is what I do though on a smaller scale of course. NR along with MQTT for transport and immediate status info, a suitable timeseries database such as InfluxDB for historical recording, node-red-contrib-UI for a simple but effective control interface and a charting/dashboard tool such as Grafana for pretty front-end output.

And I know that there will be several of us who would be very interested in the outcome in general but specifically in how you make NR resilient.

Oh, and don;t let me forget to say: DON'T FORGET SECURITY! This is too easy to leave to the end and get wrong.

[Artyom Syomushkin]

Thank you for comments. Just one about HMI
 

3. Man user interface

I have quite easy solution - WiFi tablet or Smartphone on the wall with GUI app. I stopped on Commandfusion right now, because I have made MQTT client for it, which works quite well - https://groups.google.com/forum/#!topic/commandfusion/vfPByGiBd2g

So all you need is to create your own Dashboard graphically and put a couple of MQTT calls there. Works quite fast and stable. 
As future solution I was thinking about Google Voice recognition using Tasker - my tests shown, that it works quite well. To extend the Mic Range I want to use Conference Bluetooth Speakers - then it should hear in whole room.

This is rather less polished in NR. Many of us started with custom web pages which are very easy to build in NR. Some of us went further and started to build more dynamic pages with websockets based comms. However, in recent months an excellent set of nodes from Andrei Tătar came along and which is now due to be adopted into the core. UI provides a set of simple but effective elements for building UI's that are fairly mobile friendly and very easy to use.

I tried Web-based UI, but found that it has significant problems with keeping connection alive. Also using GUI apps with REST API requires continuous pooling, slowing reaction and loading the network.. That's why I ended up with MQTT, as it is message-based and client doesn't need to poll all the time. 

[Artyom Syomushkin]

Anyway UI IMHO should be decoupled from everything else as much as possible to be portable and independent from connection stuff.

[Tony Neal]

I am using Imperihome as a nice dashboard on a mobile device. The api allows you to create virtual devices and scenes populated from Node Red and is tablet friendly. Pete Scargill did a nice intro here: http://tech.scargill.net/putting-it-all-together-in-imperihome/

[Greg EVA]

Some thoughts below...

On Monday, 14 March 2016 09:38:02 UTC+1, Artyom Syomushkin wrote:

1. Sensor & actuators

 

I'm currently using Z-Wave. In plans I want to use MQTT for ESP8266 based sensors and EnOcean for powerless sensors and switches.

See redundancy comments.

2. Database

I think the main thing here is not the database itself, but logs with timestamp - e.g. it should be easy to see what happened with house and when, to diagnose any mistakes and errors. The logs can be produces from database, but also can be created much easier, for example using MQTT client, subscribed to all messages from Broker. In this case if all traffic goes through Broker, the logs will be created automatically.

Would be nice to know, how Node-Red supports this.

It depends on what you are trying to accomplish.   Looking at raw MQTT publishes would certainly not be ideal for certain circumstances.  But you are talking about decoupling from Node-RED... you still need a client to write the data somewhere, and that somewhere could well be a database even if it passes through an MQTT broker.  The desired use of the data will depend on how you structure your data model and files.  Log messages and time-series data are quite different.  You might have a look at HiveMQ as it has tons of modules/options/capabilities and can write data to a database for you.

4. Data processing

If we talk about scenarios or rules - e.g. "real" automation, like opening and closing of Shutters on schedule, or intelligent thermostat control, alarm management, I have two solutions - local and cloud.
Local is done using Codesys runtime, running on Raspberry or any other PC. With this you can create logic and rules using one of the industry standard IEC 61131-3 languages and integrated IDE. With this IDE you can also easily debug your code in real-time on your controller. As interface with sensors/actuators I was thinking of using MQTT or Modbus TCP. The last one is supported from the box. Here I see Node-Red as important thing to convert all those different interfaces(Z-wave, Enocean, REST API for Web) to single one to be interfaces to automation controller.
For cloud control I was planning to use Thingspeak. It would be nice to use MQTT for that as well.

Node-RED certainly wins my vote here as this is really the sort of thing it is designed for.  You can do loads with OpenHABs DSL scripting language, however it is much harder to use and is poorly documented.
 

5. Server/computer/cloud

Here I have significant doubts on reliability in case of local server. Currently I run everything on Raspberry PI, but maybe will move something to my Synology NAS. Anyway redundancy is a must.

Regardless of what platform you will run this on, there are a significant number of possible points of failure in such a system.  You obviously need to ensure that all network components (devices, switches, etc.) have backup power in case of a power outage or dead batteries.  One reason I like TCP/IP and Ethernet is that you can build a pretty resilient network and be able to access its devices from all over the network.  This is not the case for Z-Wave (not sure about EnOcean) as you need to have a unique Z-Wave controller (as I understand things): if your Z-Wave interface node  dies, you cannot easily have another take over for it.

There are some who have setup very high-availability OpenHAB systems using multiple virtualization servers, coupled with network attached home automation network interfaces (which can be connected to over TCP/IP).  You can then have the VMs of your server components fail over from one to the other server.

Full redundancy will likely not be possible, so it is important to design your system with that in mind.  For example providing a physical means of turning lights on and off without requiring the network or wireless or the automation controller.  Distributing the control and giving nodes more responsibility over there proper operation will mean that the overall system will work better if any parts fail.

[Julian Knight]

Indeed, decoupling is virtually always a good thing. But you can have both. I've had significant success by using NR to deliver a single-page app and then using either websockets or MQTT websockets (or both). Of course, if I were doing an industrial scale or pro delivery, I would almost certainly create a separate microservice to deliver the user interface and have that behind a proper web server with Phusion Passenger tying the web server to the Node.js microservice. You can still get all that squeezed into a low-cost platform such as a NAS.

[Julian Knight]

Indeed. You can either give a lot more smarts to each end-unit - but then they are more prone to failure probably as they are more complex. Alternatively, you can give them a little more smarts by making sure you send periodic keepalives and have the end-units self-check to see if they've received anything recently. If not, they need to look for an alternative control source. Just watch out for MitM attacks that knock your legitimate controller offline and try to substitute their own. There are toolkits for doing that kind of attack with WiFi networks for example. 

[acoliveira Oliveira]

sorry for my poor english. this flow can send & receive msg relay board(sensor/actuator) -->http://flows.nodered.org/flow/071ec99f89a19b63ac07

best regards,

ac

[Olivier Gimenez]

Thank you all for your answers, since my first post I still not have find a decent UI. For now I still use Grafana but I am looking to node-red-contrib-ui since few weeks :)

[Timur Fatykhov]

I ended up creating my own bootstrap responsive ui. Works on ipad, android, regular desktop/laptop. (https://github.com/tfatykhov/WinkRedNode)

[Timur Fatykhov]

Slowly moving to material design - app config page - 

[Artyom Syomushkin]

Hi there,

I'm thinking if Node Red is useful for home automation as rule engine.
E.g I would like to create some simple rules, like movement-light, or roller/shutter alarm control and asking if Node Red can be used for this.
Why Node-Red? Because it is quite independent of all home automation servers - like OpenHAB, FHEM etc. So the flows, developed here, are quire universal and portable and don't need to be changed when migrating to other HAsoftware. All communication is done via MQTT.  Also debugging with NodeRED seems to be quite fun, without a risk to crash whole system. This is not the case with many of open-source HA software.

I found almost no flows dedicated to HA, so does it mean, that noone uses Noder Red for such purpose?

There is nice library, posted in this thread https://groups.google.com/forum/#!searchin/node-red/home$20automation%7Csort:relevance/node-red/czMlUYvDsmg/6r1PWluLBQAJ
So it looks at least somebody was thinking about it.

[Colin Law]

On 15 August 2016 at 09:05, Artyom Syomushkin <syom...@gmail.com> wrote:
> Hi there,
>
> I'm thinking if Node Red is useful for home automation as rule engine.
> E.g I would like to create some simple rules, like movement-light, or
> roller/shutter alarm control and asking if Node Red can be used for this.
> Why Node-Red? Because it is quite independent of all home automation servers
> - like OpenHAB, FHEM etc. So the flows, developed here, are quire universal
> and portable and don't need to be changed when migrating to other
> HAsoftware. All communication is done via MQTT. Also debugging with NodeRED
> seems to be quite fun, without a risk to crash whole system. This is not the
> case with many of open-source HA software.
>
> I found almost no flows dedicated to HA, so does it mean, that noone uses
> Noder Red for such purpose?

Many use node red for home automation. What sort of flows are you
looking for that you have not found?

Colin

[lingvo]

Many use node red for home automation. What sort of flows are you
looking for that you have not found?

Well, first of all I was trying to search for some typical HA rules. For example:
- Room Thermostat/climate control
- Light control on movement/darkness and scenarios
- Roller/shutter control on sun, jalousie angle
- Water Irrigation control
- Alarm system

I found something, but not like a library, ready to use.

Then I tried to dig into lower level of automation - for example nodes for creation of some if/then/else logic, conditions, comparisons, states, DQ Triggers. The simplest way off course is to use Javascript nodes, but I wanted to see a kind of toolbox, where I could connect such functions  graphically.
I also found not so much, so therefore I raised a question - maybe Node Red is not suitable for such thing at all?
 

[Julian Knight]

Hi there, NR isn't actually a home automation "system" like the others you have mentioned which is why you won't find something completely pre-canned. One of its great benefits is that it allows you to build just what you want without the weight and restrictions of a full system. You can also use it (at the same time if you like) for many other purposes.

If you have a look on flows.nodered.org, you will find a number of examples including my own (look for TotallyInformation as the author). I've published a set of information including screenshots but because my system, like most here, evolved over time it is hard to publish the code since it may contain information I don't want shared but also it is specific to my own collection of sensors and controls.

So you have some work to do I'm afraid to connect up your own systems and to create your own logic. But there are some examples on the flows site, some blog posts and in this group.