Binary response from POST appears corrupt

[Kevin Jones]

I need some smart minds to help me out.

I am calling a web service that returns an audio stream as a response to a POST request.

First, I retrieve an access token via a POST call to the authentication service.  My second POST call supplies the token and the wav audio I want to retrieve.

I have this working fine with PHP and Curl.  I am using the exact same headers for Content-Type and Accept as I do with my NR flow.

Here is the prep (in a function node) to the POST call in NR. access_token is set in the aforementioned first POST call.

msg.headers = {'Authorization':'Bearer ' + access_token,'Accept':'audio/x-wav','Content-Type':'text/plain'};

msg.payload = "This is a test message from St. Louis Park, Minnesota";

return msg;

The PHP version returns a correctly formatted wav audio response which I save to a file (and it plays fine).  By the way, in this case the audio file is 100,524 bytes

The NR version returns an incorrectly formatted wav audio response, though the status is valid (200).  I save the payload to a file (using file node), but the wav header looks close, but not quite.  The file size if 225,548 bytes (over double the size returned by the PHP version).

I am not modifying the response, and the header calling the web service is request the same "Accept:plain/text" header.

Any ideas of why the PHP curl call returns the correct wav audio stream size and format while the NR HTTP "POST" node does not?

Thanks in advance!  Very active and helpful forum here.

msg.headers = {'Authorization':'Bearer ' + access_token,'Accept':'audio/x-wav','Content-Type':'text/plain'};
msg.payload = "This is a test message from St. Louis Park, Minnesota";

return msg;

msg.headers = {'Authorization':'Bearer ' + access_token,'Accept':'audio/x-wav','Content-Type':'text/plain'};
msg.payload = "This is a test message from St. Louis Park, Minnesota";

return msg;

msg.headers = {'Authorization':'Bearer ' + access_token,'Accept':'audio/x-wav','Content-Type':'text/plain'};
msg.payload = "This is a test message from St. Louis Park, Minnesota";

return msg;

[Nicholas O'Leary]

The http request node currently assumes the response will be a UTF8 encodable string - which is going to do odd things when the response is a binary payload.

I'll have a look to see what we can do about it.

Nick

--
http://nodered.org
---
You received this message because you are subscribed to the Google Groups "Node-RED" group.
To unsubscribe from this group and stop receiving emails from it, send an email to node-red+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Kevin Jones]

Thanks Nick.

I made additional progress.  I now receive the proper binary (audio payload in this case) byte count, having replicated in PHP to compare.  So I have a node.js buffer now.  However, the file output node appears to be doubling the byte count on me.  So, I sense the file write is the culprit here.  Since I cannot set any properties to the current file output node, I'll dig into it and see what I find.

With regard to the file output node, I have UNchecked the newline property, and I have checked the overwrite property.  Per the current documentation, that should result in a binary write.  It looks like the buffer (payload) from the POST is being doubled in size upon file creation.  Comparing a properly formatted audio (wav) file header to that created here indicates a similar but different encoding.  I don't see newlines being added (od -h), so it appears the property to NOT add newline is being honored.

I will continue to investigate.  Now seeking counsel from anyone who has taken a response buffer and written verbatim to disk.  I'd rather not dive into the node, but certainly will if necessary.

Kevin

[László Jakab]

Hi!

Same problem there. I've downloaded binary data with http request node and saved with file node, and the file content was corrupted. If I use browser to download the file I get a file with ASCII encoding, but when I download it with http request and file node I get a file with UTF8 encoding.

Were there any progress in this issue? 

Thanks!

[Dave C-J]

Do you have an example flow? Or an example URL we can point at to recreate the problem.

[Dave C-J]

Foe example this works fine for me...

[{"id":"c0b5b52c.22d938","type":"http request","z":"5abb0ba2.b32384","name":"","method":"GET","ret":"bin","url":"https://shipitsquirrel.github.io/images/ship%20it%20squirrel.png","tls":"","x":310,"y":660,"wires":[["562f5152.1f106"]]},{"id":"8b3abcea.a7de3","type":"inject","z":"5abb0ba2.b32384","name":"","topic":"","payload":"","payloadType":"str","repeat":"","crontab":"","once":false,"x":150,"y":660,"wires":[["c0b5b52c.22d938"]]},{"id":"562f5152.1f106","type":"file","z":"5abb0ba2.b32384","name":"","filename":"/tmp/image.png","appendNewline":false,"createDir":false,"overwriteFile":"true","x":500,"y":660,"wires":[]}]

[Bart Butenaers]

Hi Dave,

this might be related to a similar problem I ran into.  For another discussion I have created my own local version of the HttpRequest node, to implement HTTP multipart streaming.  Currently it seems to work fine: I can do MJPEG streaming from all kind of ip camera's.  The only problem is performance on my Raspberry when Node-Red needs to process the high amount of images that keeps arriving ...

Current situation

After some investigation, I found that the HttpRequest node was consuming lots of memory and CPU.

What happens: If you specify to return a binary buffer, it will set encoding to 'binary'.  However 'binary' is obsolete and should be replaced by encoding 'null'.  However, due to an issue in NodeJs the 'null' will be ignored and it will keep its default stringencoder.  As a result: a HTTP response arrives in NodeJs as a binary buffer, which is converted by NodeJs to an UTF8 string, which is again afterwards converted to a binary buffer by Node-Red.  This will consume a lot of CPU and memory...  

I will show it here in detail for your test flow.  

1) Node-Red sets the encoding to 'binary':

2) When the request has been send, the response data will arrive as string chunks (which are appended to one large string):

3) At the end, the msg.payload contains a large string (with all the data chunks).  Then Node-Red will convert it back to a Buffer, since we asked it:

So in your example this will work fine: the output of the HttpRequest node is a Buffer, and the image is stored on disc.  

However, a lot of useless conversions took place (which moreover might corrupt our bytes like newline characters).

My solution

Your example flow is only about a single image.  However, in my endless HTTP streaming this becomes a real performance issue.  Here in detail how I worked around the issue:

1) I implemented the workaround (from the NodeJs issue link) by removing the string encoder in NodeJs:

2) Next step is to initialize the msg.payload as an array (of Buffers).  

3) When the request has been send, the response data will arrive as buffer chunks (which are appended to the Buffer array) since NodeJs doesn't have a stringencoder anymore:

     Remark: I used Buffer.concat  first, but then the entire buffer will need to be copied underneath every time a new chunk arrives.  This would become again a performance killer. 

 

 4) At the end, the msg.payload contains an array (with all the data chunk buffers).  We will concatenate all the buffers into one large buffer:

The end result will be the same: the output of the HttpRequest node is a Buffer, and the image is stored on disc. 

But it will run much smoother.

Conclusion

Hopefully this can solve the problem of Kevin, since I also had corrupted images in my multipart stream (due to characters being removed in the string conversion).   But I'm not sure ...

However, it would be great if this could be fixed anyway in Node-Red.

I have run out of spare time to create a pull-request for this, but this is the part of 21-httprequest.js that I have fixed:

            var req = ((/^https/.test(urltotest))?https:http).request(opts,function(res) {

                //(node.ret === "bin") ? res.setEncoding('binary') : res.setEncoding('utf8');

                if (node.ret === "bin") {

                    res.setEncoding(null);

                    delete res._readableState.decoder;               

                }

                msg.statusCode = res.statusCode;

                msg.headers = res.headers;

                msg.responseUrl = res.responseUrl;

                if (node.ret === "bin") {

                    msg.payload = [];

                }

                else {                

                    msg.payload = "";

                }

                // msg.url = url;   // revert when warning above finally removed

                res.on('data',function(chunk) {

                    if (node.ret === "bin") {  

                        msg.payload.push(chunk);

                    }

                    else {  

                        msg.payload += chunk;

                    }

                });

                res.on('end',function() {

                    if (node.metric()) {

                        // Calculate request time

                        var diff = process.hrtime(preRequestTimestamp);

                        var ms = diff[0] * 1e3 + diff[1] * 1e-6;

                        var metricRequestDurationMillis = ms.toFixed(3);

                        node.metric("duration.millis", msg, metricRequestDurationMillis);

                        if (res.client && res.client.bytesRead) {

                            node.metric("size.bytes", msg, res.client.bytesRead);

                        }

                    }

                    if (node.ret === "bin") {

                        //msg.payload = new Buffer(msg.payload,"binary");

                        msg.payload = Buffer.concat(msg.payload);

                    }

                    else if (node.ret === "obj") {

                        try { msg.payload = JSON.parse(msg.payload); }

                        catch(e) { node.warn(RED._("httpin.errors.json-error")); }

                    }

                    node.send(msg);

                    node.status({});

                });

            });

Kind regards,

Bart Butenaers

[László Jakab]

The file is on a password protected site, I'll try to reproduce it on a public if I have same time.

[László Jakab]

Forget my issue. I haven't set my http request node return type:( After setting it to binary buffer it works fine.

Thanks!

[Bart Butenaers]

Hey Dave,

I'm not a Git master yet, but I tried to create a pull request to contribute my fix.  Hopefully you find it useful.

Perhaps a rule of thumb for Node-Red users/developers having this kind of issues: always return a Buffer (instead of a text) in case you want to get media (audio, video, images ...).  This way you can avoid corrupt data, and performance will be a lot better.

Bart

[Dave C-J]

As an old hardware engineer I see nothing wrong with binary... but for some reason most people can't just read it. so prefer text by default...  ;-)

But yes - thanks for the Pull Request - will have a look later. Sounds sensible.

[Dave C-J]

Bart

I see as part of that PR some multipart code ;-) 

At the risk of opening u a can of worms... What is/was the behaviour of that ? 

[Bart Butenaers]

Dave, as I said before I'm not a git master yet :-(  I will have a look at it tonight and remove the multipart from my pull request.  Thanks for letting me know

Op 4 mrt. 2017 12:23 schreef "Dave C-J" <dce...@gmail.com>:

Bart

I see as part of that PR some multipart code ;-) 

At the risk of opening u a can of worms... What is/was the behaviour of that ? 

--
http://nodered.org
 
Join us on Slack to continue the conversation: http://nodered.org/slack
---
You received this message because you are subscribed to a topic in the Google Groups "Node-RED" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/node-red/enoYrQMm5B8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to node-red+unsubscribe@googlegroups.com.
To post to this group, send email to node...@googlegroups.com.
Visit this group at https://groups.google.com/group/node-red.
To view this discussion on the web, visit https://groups.google.com/d/msgid/node-red/CACXWFwJ5z9cKPR2%3DWcFNkOUZFmc4%3DvXXsFS%2BdHj8VxDvugD3BQ%40mail.gmail.com.

[Dave C-J]

No no. You did it just fine. PR is good as is. (Just it shows history ... So can see you deleted it... Was just wondering about it, how it was exposed to user, what were possible consequences, etc.)

May will also be useful... But one thing at a time.

sent from phone

You received this message because you are subscribed to the Google Groups "Node-RED" group.

To unsubscribe from this group and stop receiving emails from it, send an email to node-red+unsubscribe@googlegroups.com.

To post to this group, send email to node...@googlegroups.com.
Visit this group at https://groups.google.com/group/node-red.

To view this discussion on the web, visit https://groups.google.com/d/msgid/node-red/CANE1cgv5RRrbX66kEqw89rkRfTvFeCO0TeBkrrmA%3DLsBFsCUHA%40mail.gmail.com.