[Help] Serving binary data from redis

[NinjaTux]

Hi all,

i'm doing some experiment with nodejs these days and i'm trying to
build a service for image sharing.

I'm using node_redis to get images from Redis and serve them to
clients but i'm having some issues to view the images in the browser.
I think i'm missing something when i build the response using the
binary data returned from the cache.

When storing image data in cache i'm doing:

client.hset(key, "data", data, redis.print);
client.hset(key, "type", mimetype, redis.print);

and everything (i think) work without problems. To retrieve them i do:

client.hget(req.params.key, "data", function (err, data) {
if(err) {
console.log("ERROR");
} else {
res.writeHead(200, {
'Content-Type': type
});
res.end(data, 'binary');
}
}

but i see this

<img src="http://127.0.0.1:3000/get/241550564245760" alt="The image
“http://127.0.0.1:3000/get/241550564245760” cannot be displayed
because it contains errors.">

instead of the original image.
If i use WGET to access the URL the image is still corrupted and i
can't open it using "image viewer" in Ubuntu.

The headers of the response received when requesting the image are:

X-Powered-By Express
Content-Type image/jpeg
Connection keep-alive
Transfer-Encoding chunked

What i'm missing?I'm not sending the full binary data as response??

[Liam]

Have you tried a Content-Length header?

[NinjaTux]

Yes,

and i found out that file size differs when i retrieve the binary data
from the cache.

I'm using "connect-form" to parse the file upload using readable
stream:

app.post('/upload', function (req, res, next) {
if (req.form) {
req.form.complete(function (err, fields, files){
if(err) console.log(err);
var key = generateId();
res.writeHead(200, {
"X-Key-Received": key
});
console.log(files.file.size);
var mimetype = mime.lookup(files.file.path);
// from http://blog.andrewvc.com/node-js-and-binary-data
var chunkSize = 64 * 1024;
var bufSize = files.file.size;
var bufPos = 0;
var buf = new Buffer(bufSize);
fs.createReadStream(files.file.path,{
'flags': 'r',
'encoding':'binary',
'mode': 0666,
'bufferSize': chunkSize})
.addListener("data", function(chunk){
var bufNextPos = bufPos + chunk.length;
if (bufNextPos == bufSize) {
buf.write(chunk,'binary',bufPos);
res.write(buf);
bufPos = 0;
} else {
buf.write(chunk,'binary',bufPos);
bufPos = bufNextPos;
}
})
.addListener("close",function() {
if (bufPos != 0) {
console.log("bufPos!=0");
} else {

client.hset(key, "type", mimetype, redis.print);

client.hset(key, "data", buf, redis.print);
console.log(buf.length);
res.end();
}
});
});
}
});



and to retrieve images:



app.get('/get/:key', function (req, res) {
client.hget(req.params.key, "type", function (err, type) {
if(type != null){

client.hget(req.params.key, "data", function (err, data) {
if(err) {
console.log("ERROR");
} else {
res.writeHead(200, {

'Content-Type': type,
'Content-Length': data.length
});
res.end(data, 'binary');
console.log(data.length);
}
});
} else {
res.writeHead(404, {});
res.end("File Not Found!");
}
});
});


but data.length differs between uploading and serving.
What's happening when i retrieve data?i build the response too soon?

[Liam]

Can you come up with a simpler code segment which reproduces the
error? That would help you isolate it...

On Nov 14, 1:16 pm, NinjaTux <valerio.barr...@gmail.com> wrote:
> Yes,
>
> and i found out that file size differs when i retrieve the binary data
> from the cache.
>
> I'm using "connect-form" to parse the file upload using readable
> stream:
>
> app.post('/upload', function (req, res, next) {
>   if (req.form) {
>     req.form.complete(function (err, fields, files){
>       if(err) console.log(err);
>       var key = generateId();
>       res.writeHead(200, {
>         "X-Key-Received": key
>       });
>       console.log(files.file.size);
>       var mimetype = mime.lookup(files.file.path);

>       // fromhttp://blog.andrewvc.com/node-js-and-binary-data

[Jann Horn]

Is 'data' a string or a buffer?

--
Job Board: http://jobs.nodejs.org/
Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nod...@googlegroups.com
To unsubscribe from this group, send email to
nodejs+un...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en

[Matt Ranney]

This is probably because node_redis defaults to sending back JavaScript Strings for all commands.  You can change this behavior at client create time like this:

var client = require("redis").createClient(null, null, { return_buffers: true });

All Redis "bulk" responses on this client will be sent back as Buffers.

The next release of node_redis will also provide a way to select Strings or Buffers on a per-command basis.