Espruino-red: running node-red nodes on an esp8266

[Thorsten von Eicken]

I'm been working for a while with embedded stuff and lately a lot with the Espruino javascript interpreter running on an esp8266 (it runs well on little ARM boards also). The espruino-red project is my attempt to use node-red to program/configure and operate the embedded boards.

At a high level espruino-red runs real node-red nodes on the embedded system. However, it does not run arbitrary node-red nodes on Espruino, rather a special set of nodes is used. The standard nodes (i.e. the ones you're familiar with) can be freely interconnected with the espruino-red nodes, so they really all function the same way from a user's perspective. It's just that the implementation of the embedded nodes is slightly different so they can actually run on espruino. Moreover, if you put a wire between a standard node and an espruino node the messages carried on that wire will be sent over the network, which is MQTT in the current implementation. This connectivity works between node-red and espruino-red as well as between espruino-red nodes running on different boards.

To be clear about the implementation up-front: this is a proof of concept! I built it for two primary reasons: the first is that I have been dreaming about this type of visual program composition applied to embedded systems for a long time and I want to try it out to see whether I actually like to use it in reality or whether it's just a misguided dream. The second is that doing this type of stuff in node-red without modifying node-red itself requires some ugly hacks, plus I'm not a real node.js programmer so I'm looking for feedback. The espruino-red code reaches into some internals, in particular the flows module, to grab some information it needs to function. I expect to hear that I did it all wrong and it should instead be done so and so :-). Finally, I have not paid attention to performance or efficiency: there's a lot of room for improvement!

The node-red part of the espruino-red implementation is neither tied to Espruino nor to MQTT. It just happens that espruino makes things easy because I can just use JSON messages and MQTT over wifi. I personally am looking forward to creating an implementation in Forth (!) but doing a Lua version would be rather easy as well. The MQTT portion is totally pluggable at the flow level (explained further down). For this reason I'm calling the node-red portion of all this "e-red" for "embedded-red". So think of e-red as being the overall framework that allows embedded boards to be connected to node-red and espruino-red as a specific instantiation for espruino.

Enough preliminaries, here's a very simple demo. The following flow uses a DS18B20 temperature sensor to set the blinking rate of an LED. It also provides manual overrides to fake the temperature and it has a debug node to print the blinking rate in the debug console.

All the yellow nodes run on Espruino: I can shut down node-red on my laptop and they continue running just fine. The e-red timer outputs a message at a fixed rate configured into it using the std node-red UI. The ds18b20 node reads the temperature sensor each time it receives a message and outputs the temp in degrees F. The calc rate is a simple function node with a small snippet of javascript code that converts the temperature into a blinking rate in Hz. The blinker node runs an internal timer to make an LED attached to a gpio pin blink at the rate specified by input messages.

The three grey nodes at the top are standard node-red inject nodes that override the temperature. Clicking on them injects a message into the calc rate node on espruino. The blink rate debug node is also a standard node that prints the blink rate into the node-red debug log.

This example should show how the espruino-red nodes fit very naturally into the node-red framework and how they can be interconnected at will with standard nodes. The only thing a user notices is that each e-red node has a "board" input, which is the name of the board it needs to run on. This means that it is possible to run nodes on multiple boards and interconnect them with one-another. In those cases messages are carried over MQTT as well without going through the node-red core. This screen shot shows the edit pane of the ds18b20 node, nothing special: that's the point.

I hope it's clear by now how all this differs from a firmata type of approach: the code runs on the embedded systems and new types of nodes can be added almost in the same way they're added to node red. The main difference is that in addition to the standard html and js parts each espruino-red node type needs a piece of code that runs in espruino and currently that code needs to be pre-loaded onto the board.

The communication via MQTT is exposed to the flows in a somewhat peculiar manner. The following flow (which needs to be running) interconnects node-red with espruino:

The mqtt-in node is configured to subscribe to MQTT messages destined to the core node-red and it forwards them to the espruino-red gateway, which contains the plumbing to deliver the messages to the destination nodes in node-red, i.e., this routes messages from boards to node-red The gateway also produces messages that are destined to the espruino-red boards and they are routed via the mqtt-out node, i.e. this routes messages from node-red to the boards. The queue node is a hack that's there 'cause mqtt-out drops messages when it's disconnected from the broker and that breaks things. Swapping MQTT for a different transport is mostly a matter of changing this flow. (The e-red gateway node is confusing in that messages coming in have nothing to do with messages going out, I should have used two separate nodes for the two parts.)

You can read more about the internals of espruino-red at https://github.com/jeelabs/espruino-red which has a detailed readme. There is also a demo.md that explains how to run a demo of the above flow on linux (no esp8266 needed). I have not done a full walk-through of the demo from scratch myself and I haven't looked up what the easiest way is to get espruino for linux, so if you want to actually play with this hit me up on gitter https://gitter.im/jeelabs/espruino-red

Overall I'd most appreciate feedback on how this meshes with the node-red vision, if at all, and how to make it fit more naturally into node-red. The current implementation is just a few hundred lines of code, which is a testament to the flexibility of the node-red implementation!

[Librae]

Hi, Thorsten,

It's a dramatic long story, and I opportunely have played with espruino on esp8266 before.

Not sure if I catch your point, just share a bit of my thoughts live in mind for several years.

In my understanding e-red would be some kind of tool for firmware developing, by abstracting hardware logic modules into node-red nodes, is it true?

That is something I've ever imagine, drag and drop hardware logic modules on the paper and deploy with one click then it works as it is :D

According to my previous experience, we can achieve such object nicely on platforms like Raspberry Pi, with application processors and enough memory, by running a complete Node-RED instance on it.

Then we can use hardware peripheral nodes for Pi, to develop more complicated application, together with a good interaction with more nodes for other purpose.

For espruino you mentioned, as it's a simplified Javascript port on microcontrollers(constrained hardware resource), it's easy to build small networking applications with lines of code.

So, what about the memory status in your experiment? Is it enough for deploying a bit more nodes?

I guess if there is no critical memory issue(it's a headache on NodeMCU with LUA), we may go further, and consider more about realtime performance.

Nevertheless, I still guess in the future, hardware is going to be more powerful and smaller in size, Node-RED could still be a great idea for developing hardware because of its ability for logical nodes connectivity.

Thanks,

Librae

--
http://nodered.org
 
Join us on Slack to continue the conversation: http://nodered.org/slack
---
You received this message because you are subscribed to the Google Groups "Node-RED" group.
To unsubscribe from this group and stop receiving emails from it, send an email to node-red+unsubscribe@googlegroups.com.
To post to this group, send email to node...@googlegroups.com.
Visit this group at https://groups.google.com/group/node-red.
For more options, visit https://groups.google.com/d/optout.

[Thorsten von Eicken]

Librae, thanks for your thoughts! For stuff that hangs on A/C power I'm increasingly using a odroid/raspi, or CHIP and they can run full-blown node-red. But for stuff that does not hang on A/C power or where the complexity of linux device drivers and interrupts to get realtime behavior are too much of a hassle it's different. There I still use smaller uC's that just don't have enough memory or horsepower to run node.js or linux.

There's no question that espruino is primarily memory limited on the esp8266 and most STM32 processors as well. But there's enough there that one can implement interesting things.

Overall, what I'm looking for is a way to define small functional modules that run on embedded system and be able to compose them in node-red like fashion and also to be able to build flows that span multiple embedded systems and "server-side" logic.

[Bruce Richardson]

Hi,

I really like this idea of doing it all from node-red, it appeals to me as a hobbyist with only a surface knowledge of such systems. 

With the limited resources of microcontrollers in mind, would it be an idea to have a more minimal firmware and compile for that firmware during the deploy stage in node-red. 

You could then avoid having the overhead of javascript interpreters and serial shells on such tiny devices. An 'e-node' firmware would only need a way of talking to its node-red control system and the ability to run something closer to its native machine code. I'm just thinking out loud here, I don't have any experience of programming at that level so I might be missing the point. 

Bruce

[Thorsten von Eicken]

The node-red side of the espruino-red implementation is not tied to espruino in any way whatsoever. I personally want to play with forth, which is sort-of compiled, very small, and very fast. But other implementations are also possible. If you use a compiled language you have to face the issue of the function node: how do you implement that? Some interpreted language, compile on demand and dynamically link, ...?

Just to be clear, the microcontroller firmware for espruino-red is under 200 lines of javascript, I'm not running any node-red code there.

[Librae]

Yes, I think so, for sure if we can put the heaviest application logic on server side (node-red), it can significantly save a lot of hardware resource for both code & memory.

Maybe we can imagine in the near future, with even better networking facilities, the realtime performance would not be one of our major worry, for this idea.

So as an objective,  in uC's firmware(espruino), do we only need to handle critical hardware timing stuffs, together with the javascript node interface to Node-RED server, including MQTT connection?

Anyway I must say it worth a try, keep us posted, Thorsten!

--

[Julian Knight]

Funny that FORTH should come back out. That reminds me of when I started with personal computers learning all manner of languages on a BBC Micro - FORTH was the one I settled on most because you can do so much with so little.

[Marko S]

So Thorsten, what kind of stuff do you support on the esp8266, can you post a screenshot of which nodes are available. For example can you read an "analog" pin,  can you control neopixels, what about max7219 matrix displays? what about 16x2 or 20x4 lcd's?

I'm hoping the answer to those questions is yes, or planned in the future :D. Nice work btw!

Dne sobota, 13. avgust 2016 22.22.51 UTC+2 je oseba Thorsten von Eicken napisala:

[Thorsten von Eicken]

Ha, I just managed to get the core working and a demo! :-) The esp8266 doesn't have any analog pin worth reading, unless you're in for torture :-). I tend to use i2c or spi chips to do analog, but mostly I try to use sensors with an i2c or spi interface. Espruino has quite a lot of JS modules, so for many external peripherals there's already something to start with.

[ghe...@me.com]

This is just a fantastic idea! It really fits exactly what I have been doing ie node-red to do all kinds of home control things and espurino to setup esp8266 (wemos D1 mini) boards to control actual things. However being newbyish would it be possible to 

run through a procedure to actually setup this. How to install the node? Can I just use npm install? How to setup the espurino bit; which files go where? Does it use the web IDE in some way? Arre you saying that if node-red runs on say rasp pi

it is not working yet? What about OSX?

[Thorsten von Eicken]

To be honest, I'm not sure I have the time this week-end to do a complete how-to. The short version is to install node-red in a completely normal way. Should work fine on a raspi (there are notes in the docs about that, it's now a package in raspbian, I think). Then clone my github repo into your home dir or elsewhere and edit the node-red settings file (typically ~/.node-red/settings.js) and set nodesDir to point to the espruino-red directory (something like nodesDir: '/home/src/espruino-red',). If you now start node-red you should see an e_red section of nodes in the left side-bar.

You will also need an mqtt broker, such as mosquitto. I use pretty much a plain set-up (assuming your network is reasonably secure).

For espruino use 1v86 or the latest build I advertised http://forum.espruino.com/comments/13093133/ and use whatever method you're comfortable with to load e-red-esp8266.js (mhh, search for 'voneicken' in the source first and change to point to your mqtt broker).

In node-red hook-up 4 nodes: mqtt-in -> e-red gw -> mqtt queue -> mqtt out and configure them to your mqtt broker. After that you can add a trigger node -> e-red function node -> debug node, add some simple code to the function (like msg.payload = "hello world"; return msg) and when you hit the trigger it should show in the debug.

Sorry, even this "short" version is already long. Lots of moving parts...

[ghe...@me.com]

Thanks for your reply

So.. I put the files in home/pi/.node-red/ as

home/

-pi/

--.node-red/

---espurino-red/ (directory name changed from espruino-red-master)

----e-red/

----Espruino/

----others etc..

and I changed  ~/.node-red/settings.js

to set nodesDir to point to the espruino-red directory

/home/pi/.node-red/e-red/

as

    // Node-RED scans the `nodes` directory in the install directory to find nodes.

    // The following property can be used to specify an additional directory to scan.

    nodesDir: '/home/pi/.node-red/espruino-red/e-red',

but the nodes didnt show up on restart

you say set nodesDir to espruino-red so I assumed e-red

but this didnt work so I tried

    // Node-RED scans the `nodes` directory in the install directory to find nodes.

    // The following property can be used to specify an additional directory to scan.

    nodesDir: '/home/pi/.node-red/espruino-red/',

but this didnt work either

[Thorsten von Eicken]

I have it pointing to the espruino-red dir, but when I argue with npm it wins every time... :-(

[ghe...@me.com]

so what would I do now to work out where I went wrong?

[Marko S]

So should all of these work - http://www.espruino.com/Modules -> and are they all somehow exposed in a node?

Dne ponedeljek, 15. avgust 2016 07.08.30 UTC+2 je oseba ghe...@me.com napisala:

[Thorsten von Eicken]

On Monday, August 15, 2016 at 12:23:22 AM UTC-7, Marko S wrote:

So should all of these work - http://www.espruino.com/Modules -> and are they all somehow exposed in a node?

They should all work if you wrap them into a node ;-)