Node-Red and ESP8266

[Toshi Bass]

Hi 

I am new to Node-Red only 2nd day, but I am getting along fine with it, however I have one problem.

I have several ESP8266 which currently I retrieve the data from them by making a http call from a python script with python requests module like :

r = requests.get('http://192.168.0.179:8080', timeout=8)  

then I parse the text response and it all works fine.

However I would like to do the same with Node-Red, so I used a http request node with method GET and URL 192.168.0.179:8080 but when I fire it

I get :

 HPE_INVALID_HEADER_TOKEN 

and in the debug screen :

 [msg.payload] : (string)

Error: Parse Error : http://192.168.0.179:8080
Tried googling this problem but I don't seam to be able find a solution, any help / solution / explanation would be appreciated.
Thanks
Toshi 

[Nicholas O'Leary]

Hi,

a quick google of the error finds this: https://github.com/joyent/node/issues/4394 - which suggests the error is because there is an invalid header field in the http response that the http library in node.js is "unforgiving of such spec violations".

If you run:

 curl -v http://192.168.0.179:8080

you can see the complete request/response - have a look for anything invalid (or share here and we can take a look).

Nick

--
http://nodered.org
---
You received this message because you are subscribed to the Google Groups "Node-RED" group.
To unsubscribe from this group and stop receiving emails from it, send an email to node-red+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Toshi Bass]

Hi Nick 

Thanks for the pointer, originally I got the output below from    curl -v http://192.168.0.199:8080
that made it obvious that the headers were causing the problem, so with a bit of trial and error I found that removing the red ones but leaving the blue one, resulted in Node-Red getting the values but I am still unsure if its correct because I still get 1  * additional stuff not fine transfer.c:1037: 0 0 see bottom result

Toshi

root@client38 ~ # curl -v http://192.168.0.179:8080
* About to connect() to 192.168.0.179 port 8080 (#0)
* Trying 192.168.0.179...
* connected
* Connected to 192.168.0.179 (192.168.0.179) port 8080 (#0)
> GET / HTTP/1.1
> User-Agent: curl/7.26.0
> Host: 192.168.0.179:8080
> Accept: */*
>
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* HTTP 1.1 or later with persistent connection, pipelining supported
< HTTP/1.1 200 OK
< Access-Control-Allow-Origin:http://192.168.0.34:8000
< Access-Control-Allow- Methods", "POST, GET
< Access-Control-Allow-Headers *AUTHORISED*
< Content-type: text/html
< Server: ESP8266-0
* no chunk, no close, no size. Assume close to signal end
<
* nread <= 0, server closed connection, bailing
* Closing connection #0
28345d2c060000e5,22.8125,28a0f52b06000003,23.8750,root@client38 ~ #

Now I get :

root@client38 ~ # curl -v http://192.168.0.199:8080

* About to connect() to 192.168.0.199 port 8080 (#0)

*   Trying 192.168.0.199...

* connected

* Connected to 192.168.0.199 (192.168.0.199) port 8080 (#0)

> GET / HTTP/1.1

> User-Agent: curl/7.26.0

> Host: 192.168.0.199:8080

> Accept: */*

>

* additional stuff not fine transfer.c:1037: 0 0

* HTTP 1.1 or later with persistent connection, pipelining supported

< HTTP/1.1 200 OK

* no chunk, no close, no size. Assume close to signal end

<

* nread <= 0, server closed connection, bailing

* Closing connection #0

1,1,1,25.75

[Julian Knight]

But those headers look valid to me? The access control ones are for CORS. Content-Type looks right. Maybe the last one, not sure what that is?

[Toshi Bass]

Hi Julian

In light of what you said I went through the trial and error process again just to be sure, result the only combination of headers specified in the lua program on the ESP that allows node-red to receive a reply is

conn:send('HTTP/1.1 200 OK\r\n\n')

If I include none or or any combination of the others on there own or more than one I get:  HPE_INVALID_HEADER_CONSTANT 

if I include the last one I get:

 HPE_INVALID_HEADER_TOKEN

So whilst it is clear I have no idea what I am doing, it does work now !

Thanks

Toshi

 

[Julian Knight]

Odd, I wonder if that is a problem in NR's http processing?

[Nicholas O'Leary]

In your original response, these headers look malformed to me:

Access-Control-Allow- Methods", "POST, GET

Access-Control-Allow-Headers *AUTHORISED*

They should be:

Access-Control-Allow-Methods: POST, GET   << this has a space before 'Methods' , was missing the colon and had quotes that shouldn't be there
Access-Control-Allow-Headers: *AUTHORISED* << this was missing the colon

[Julian Knight]

Ah, good spot Nick. Don't know why I didn't see that. Too many late nights. That might then have thrown out any following headers?

[Toshi Bass]

Agreed well spotted, Python must be more forgiving than node-red because these headers have been working fine for months, anyway I modified the mistakes and also added Content-type:"text/html" and it works in both python and node-red,  still get 1 instance of  * additional stuff not fine transfer.c:1037: 0 0 but it doesn't mater what I do I cannot get rid of it, like I said without at least HTTP/1.1 200 OK node-red will not get the values which incidentally on this ESP are temperature, humidity, light level and control of 2 relays.... I will now convert and test rest of the ESP's, wanted to program them with mqtt instead of http calls but mqtt just uses up to much of the available memory.

curl -v http://192.168.0.56:8080

* About to connect() to 192.168.0.56 port 8080 (#0)

*   Trying 192.168.0.56...

* connected

* Connected to 192.168.0.56 (192.168.0.56) port 8080 (#0)

> GET / HTTP/1.1

> User-Agent: curl/7.26.0

> Host: 192.168.0.56:8080

> Accept: */*

>

* additional stuff not fine transfer.c:1037: 0 0

* HTTP 1.1 or later with persistent connection, pipelining supported

< HTTP/1.1 200 OK

< Access-Control-Allow-Origin:http://192.168.0.34:8000

< Access-Control-Allow- Methods:"POST, GET"

< Access-Control-Allow-Headers:*AUTHORISED*

< Content-type:"text/html"

* no chunk, no close, no size. Assume close to signal end

<

* nread <= 0, server closed connection, bailing

* Closing connection #0

25.7,40.9,165,0,0

Thanks again for your Help

Toshi

[Julian Knight]

Glad that's sorted. I'd love to hear more about how you set up the ESP's, hardware, software and the like. How difficult it was, how you get on with power and so on. Obviously off-topic here though, will you be writing up a blog post?

[Dave C-J]

See this stackoverflow question also... may be relevant.

http://stackoverflow.com/questions/13943181/why-is-curl-returning-additional-stuff-not-fine-transfer

[Toshi Bass]

Not planning to write a blog just yet, everything is a little fluid at the moment, the biggest problem with the ESP modules is the availability of usable memory (or perhaps I am over ambitious and trying to build in to much in each module) every time a new firmware upgrade is released it eats some more of the usable space for example, I wanted to use MQTT but the size of the code needed on the ESP for that takes up more ram than using a tcp connection so its limited to simple things like returning data from a couple of DS18B20 or a button press my other frustration is programming in lua.

On the control side of things I want to use Python / Webiopi but the requests module that's needed to communicate is slow and not that reliable, although its great when its working, that's why I am here, trying Node-Red with the Paho-MQTT Python client, first impression now I sorted out the Invalid header issue is its great, much faster and up its not missed a beat, couple that with node red's flexibility and I am feeling quite happy today, just need to spend some time to figure stuff out, as I said its only like my 3rd day with this, but I am already getting data and switching relays on the esp modules so ....

Finally from a power point of view, currently the rule is more rather than less, I think its going to be quite a while before anyone's seriously powering the ESP from a battery for a reasonable amount of time, I don't think anyone has worked out how to sleep and wake these thing properly yet and the wifi on these things are quite greedy, but these modules are great for the right application, for example I have a freezer and a beer-fridge and I want to monitor the internal temperatures (not so worried about the freezer temp getting warm but I hate warm beer) getting wires to them would have been difficult, but I have been happily monitoring the temps with a ESP module for several months now.

Hope that's not to off-topic I did mention Node-Red a few of times.

Toshi  

[Toshi Bass]

Hi Dave

Thanks for the reply, not sure what to make of stackoverflow thing, it opens more questions than it closes, so I am going to monitor my setup and see if I can spot any problems, but if not then fine its working.

Toshi

[Luis Montes]

I looked a bit at using nodemcu/lua and share some of your frustrations.  

Starting to play with this now https://github.com/5shekel/ESP8266-transparent-bridge It makes the ESP just a super cheap generic serial relay.  In theory node-red-contrib-gpio should be able to communicate through the relay to any board with RX/TX pins (arduino UNO or otherwise) that can run a standardfirmata sketch.  Similar to how the the regular node-red arduino node uses a serial cable but over TCP instead.

Ajfisher and I managed to get it working with node directly and running johnny-five code:  https://gist.github.com/ajfisher/5fe60fe7d8c49b3223f0 

I'd love to see this new firmware updated for mqtt support and better remote configuration, but this will suffice for now :)

--

[Julian Knight]

Really helpful stuff, thanks Toshi.

From my reading, the LUA firmware is just too greedy of memory to be really useful. People seem to have lots of issues with stability too. Have you tried the Arduino compiler for the ESP that Dave suggested? It claims to be compatible with the MQTT library.

For my use, perhaps I was being too ambitious with battery only modules. In fact, having thought it through, nearly all the places I want to put sensors will be not that far from power. In any case, it is probably a lot simpler to grab a bunch of cheap low voltage power supplies.

Anyway, thanks again for the info on power use, very helpful.

[Toshi Bass]

Not really arduino fan and it kinda defeats the object in my view, these ESP modules have everything on-board already they just need to have a reasonable amount of usable memory instead of the pitiful amount currently available, they cost < £3 so why don't they double or more the memory and charge £6, the sdk already allows for that so I think sooner rather than later we should start to see new higher capacity modules available, it takes me back to the zx80 days and 1k of ram it always needed that 16k ram expansion pack !

Toshi

[Peter Scargill]

Hope I'm not off topic here.. The Lua software through clever eats too much memory.

I took TUAN's MQTT code and used that as the base for my own programming.. I've since put all sorts of stuff in - it just keeps growing but I've yet to hit any memory barrier programming in C (Eclipse environment on the PC).  

Pete

[Luis Montes]

I'd love to have something like like firmata running on the chip itself, but using them as a cheap wifi relay is fine if you want to attach another cheap device(arduino or otherwise) so that you can have more than 3 GPIOs.  Would definitely be a waste for the more expensive esp boards that have several GPIOs.
There seems to be plenty of memory on the ESP if you want to write in C. 

On Mon, Apr 27, 2015 at 3:44 AM, Toshi Bass <toshib...@gmail.com> wrote:

Not really arduino fan and it kinda defeats the object in my view, these ESP modules have everything on-board already they just need to have a reasonable amount of usable memory instead of the pitiful amount currently available, they cost < £3 so why don't they double or more the memory and charge £6, the sdk already allows for that so I think sooner rather than later we should start to see new higher capacity modules available, it takes me back to the zx80 days and 1k of ram it always needed that 16k ram expansion pack !

Toshi

[Gaz]

Hi LuisI

 have a fully working Arduino UNO with Ethernet shield running MQTT and various other libraries.
This has been running perfectly for over a year, I know want to replace the Ethernet with an ESP8266.

You mention you have this working, I want to keep all my code I have on my UNO and then somehow send the data out via  the ESP8266.

You say "It makes the ESP just a super cheap generic serial relay" does this mean 2 way communication. Can I send out and in, MQTT messages ?



Basically all I want to do is remove the Ethernet shield and have the ESp8266 do the same job but Wireless !

Have you any tips or help or code you can give me ?


Thanks for your time

Regards


Gary