Question re: handling a chunked https response that never ends

[William Lindsay]

Hello Node-RED group

My name is Bill Lindsay and I'm in California.

I am a new Node-RED user and I'm doing various IoT application demos interfacing to a Wireless Sensor Network Gateway.  The particular gateway I'm connecting to requires https with basic authentication.  No problem there, I installed the contrib-https node and that's working perfectly.  I can do a simple call-response GET and see the response in the debug pane so I know that's all working.  

The place where I'm stuck is getting sensor data notifications.  This particular gateway is intended for reasonably high traffic, so the GET/notifications opens up a single, never ending connection.  As I kind of expected, I do the get, and a little text message under my https node says 'sending request' because the response is never 'complete'.  What I need is a way to handle each chunk.  An individual wireless sensor data notification is its own JSON object with new line characters at the end.  I imagine it like a worker at the noodle factory.  Miles of noodle come slithering out the machine, and he chops them off every once in a while.  I want a little noodle worker who will chop off this data every time it sees 0x0D0A and send that along for me to parse.  

Is there a standard or built-in way to take in a never ending chunked https response?

thanks

Bill Lindsay

El Cerrito, CA 

[Nick O'Leary]

Hi Bill,

I'm not aware of an HTTP Request node that supports that sort of chunked response. It would be something we could consider adding to the core HTTP Request node (which supports https and basic auth, so no real need to install the contrib-https node, unless I'm missing something about it).

If you wanted to raise an issue on github, it would get onto the backlog so we don't forget about it. Can't guarantee when it would get done - but at least it would be in the mix. Of course if anyone wanted to pick it up and contribute the change, that would be welcome (just discuss the design/approach first...)

Nick

--
http://nodered.org
 
Join us on Slack to continue the conversation: http://nodered.org/slack
---
You received this message because you are subscribed to the Google Groups "Node-RED" group.
To unsubscribe from this group and stop receiving emails from it, send an email to node-red+unsubscribe@googlegroups.com.
To post to this group, send email to node...@googlegroups.com.
Visit this group at https://groups.google.com/group/node-red.
To view this discussion on the web, visit https://groups.google.com/d/msgid/node-red/bacdd0da-3b89-4eb2-948d-e1d29af9c881%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Bart Butenaers]

Hi Bill,

I'm not familiar with the contrib-https node, but I'm using the core http request node (which also allows to create https connections).

And that node also doesn't support endless streaming of data ;-(

It can retrieve chunks of data, append them together, and return the complete response at the end.  But no streaming ...

When I saw your question this weekend, it reminded me that I have a similar problem on my todo list: implementing MJPEG streaming in Node-red.

In short, I want to get an infinite stream of camera images over HTTP, and inject those images as messages into my Node-Red flow (for image processing).

I added some code to the original http request node to support Multipart Http streaming: you can find my test version here.

Hopefully Dave and/or Nick can find some time to review my solution.  Otherwise I need to create a custom node, but it would be nice if it could become part of the core ...

Below my analysis of the issue.  Hopefully it can be of any help to your problem also !

Kind regards,

Bart

Context

My http request node is used to get every second an image from my IP camera, which is then (base-64 encoded) displayed in my dashboard UI

Works fine.  However, the image is updated at a low data rate because we have to send a request for every image, and wait for the answer:

1. Http request to camera (for image snapshot)
   
2. Wait for http response (containing image)
   
3. Http request to camera  (for image snapshot)
   
4. Wait for http response (containing image)
   
5. .......... 
   

My camera supports MJPEG streaming, which sends an endless stream of images after a single http request:

1. Http request to camera (for mjpeg stream)
2. Http response (containing image)
3. Http response (containing image)
4. ........

This way the number of images we can receive is much larger!

Multipart streaming

 A lot of implementations available on the web, are designed specifically for MJPEG (by detecting MJPEG specific bytes in the stream).

However, since MJPEG is only one of the many use cases of Multipart http streaming, I decided to implement that as a general solution.

Summarized: the stream can be automatically be detected, since the header 'content-type' contains the text 'multipart'.  Moreover, it contains a boundary value.

That boundary value is repeated afterwards as a separator between the data parts.  For example:

* HTTP/1.0 200 OK

* Server: en.code-bude.net example server

* Cache-Control: no-cache

* Cache-Control: private

* Content-Type: multipart/x-mixed-replace;boundary=--boundary

* 

* --boundary

* Content-Type: image/jpeg

* Content-Length: [length of the content data bytes]

* 

* [content data bytes e.g. camera image]

*

* --boundary

* Content-Type: image/jpeg

* Content-Length: [length of the content data bytes]

* 

* [content data bytes e.g. camera image]

*

* [...]

So, if the boundary value is read (at the start of the stream) from the header, it can be used to detect when a new data part is received (e.g. camera image).

Result on github

I have tested my patch on my ip camera's, and it works great.  My camera images are now arriving at very high speed, resulting in a smooth video.

Open issues

1. When multipart is detected and no boundary value is available (on the header): the error in the code should get another description.
2. Some kind of throttling should be available: currently the images are arriving as fast as my camera can deliver.  It should be possible to let the user (in the property view of the http request node) enter some delay (in microseconds) that puts the node in sleep, to limit the number of data parts arriving.  However, since I'm a Javascript beginner I don't have a clue how to accomplish such a wait mechanism without affecting performance ...
3. I have tested my code on a raspberry pi, running on linux raspbian.  However, it should also be tested on Windows and Mac.  Reason is that I need to detect the end-of-line characters (to separate the headers from the data bodies).  However, the end-of-line characters differ between operating systems: I have added this in the code, but it should be tested ...

[Nick O'Leary]

Best way to get us to review proposed changes is to open a pull request against the code is question. We can then use GitHub's built in tools to provide feedback in context.

Nick

--

http://nodered.org
 
Join us on Slack to continue the conversation: http://nodered.org/slack
---
You received this message because you are subscribed to the Google Groups "Node-RED" group.

To unsubscribe from this group and stop receiving emails from it, send an email to node-red+u...@googlegroups.com.

To post to this group, send email to node...@googlegroups.com.
Visit this group at https://groups.google.com/group/node-red.

To view this discussion on the web, visit https://groups.google.com/d/msgid/node-red/37f8c3e4-8195-4e92-a63c-dbc610e6c847%40googlegroups.com.

[William Lindsay]

Thanks for the reply, Nick.  I will attempt to add a feature request through github.  I think I can work around it with a python tool that can grab these chunks one at a time and throw them at a NodeRED url.  That should be good enough for the demo-level tasks I'm doing now.

I will also revisit whether the standard HTTP request node works or not with this particular gateway.  I had tried to get it working and was receiving errors.  I will give it another look.

Bill

On Sunday, February 5, 2017 at 3:54:12 PM UTC-8, Nick O'Leary wrote:

Hi Bill,

I'm not aware of an HTTP Request node that supports that sort of chunked response. It would be something we could consider adding to the core HTTP Request node (which supports https and basic auth, so no real need to install the contrib-https node, unless I'm missing something about it).

If you wanted to raise an issue on github, it would get onto the backlog so we don't forget about it. Can't guarantee when it would get done - but at least it would be in the mix. Of course if anyone wanted to pick it up and contribute the change, that would be welcome (just discuss the design/approach first...)

Nick

On 3 February 2017 at 01:45, William Lindsay <tape...@gmail.com> wrote:

Hello Node-RED group

My name is Bill Lindsay and I'm in California.

I am a new Node-RED user and I'm doing various IoT application demos interfacing to a Wireless Sensor Network Gateway.  The particular gateway I'm connecting to requires https with basic authentication.  No problem there, I installed the contrib-https node and that's working perfectly.  I can do a simple call-response GET and see the response in the debug pane so I know that's all working.  

The place where I'm stuck is getting sensor data notifications.  This particular gateway is intended for reasonably high traffic, so the GET/notifications opens up a single, never ending connection.  As I kind of expected, I do the get, and a little text message under my https node says 'sending request' because the response is never 'complete'.  What I need is a way to handle each chunk.  An individual wireless sensor data notification is its own JSON object with new line characters at the end.  I imagine it like a worker at the noodle factory.  Miles of noodle come slithering out the machine, and he chops them off every once in a while.  I want a little noodle worker who will chop off this data every time it sees 0x0D0A and send that along for me to parse.  

Is there a standard or built-in way to take in a never ending chunked https response?

thanks

Bill Lindsay

El Cerrito, CA 

--
http://nodered.org
 
Join us on Slack to continue the conversation: http://nodered.org/slack
---
You received this message because you are subscribed to the Google Groups "Node-RED" group.

To unsubscribe from this group and stop receiving emails from it, send an email to node-red+u...@googlegroups.com.