Streaming POST data over HTTP

[Jesper Nøhr]

Hi list,

I'm experimenting with a somewhat simple HTTP "load balancer". It
takes an incoming request, picks a random server, opens a connection
to the server, reads the response, and delivers it back to the client.

In its simplest form (w/ comments):

var server = http.createServer(function (req, res) {
// I can read the data here, but then I need to buffer it.

pick_server(req, function(host, port) {
var conn = http.createClient(port, host);
var client = conn.request(req.method, req.url, req.headers);

// I'd like to listen for +data on 'req' here.

client.addListener("response", function (resp) {
res.writeHeader(resp.statusCode, resp.headers);

resp.addListener("data", function (chunk) {
res.write(chunk, "binary");
});
});

conn.addListener("end", function() {
res.close();
});

client.close();
});
});

Now the problem is POST data. It's possible to read the data by
listening on +data for 'req', but to read it all, it seems I need to
buffer it. That becomes a problem when the request is very large, and
it would be much better if I could somehow stream it straight to the
client. Can I do that somehow?

I'm quite new to Node, and I'm probably missing something obvious. Any
help is greatly appreciated.

Jesper

[Lincoln Stoll]

Where you said you'd like to listen to it worked - after adding a few
end listeners to clean up and move it on to the next phase. So close!

This is something which I got working:

var sys = require('sys')
var http = require('http');

var server = http.createServer(function (req, res) {
// I can read the data here, but then I need to buffer it.

//

pick_server(req, function(host, port) {
var conn = http.createClient(port, host);

var headers = req.headers;
//headers["Transfer-Encoding"] = "chunked"; //docs say we should.
But not here

var client = conn.request(req.method, req.url, req.headers);

// I'd like to listen for +data on 'req' here.

// You can!
req.addListener("data", function(chunk) {
client.write(chunk, "binary");
});

// however, listening for data didn't work until this was added.
req.addListener("end", function() {
// we are done sending - close and flush client
client.close();
});

// this is being invoked multiple times with couch. It shouldn't
be, according
// to the docs, however the HTTP responses coming back imply it should

client.addListener("response", function (resp) {

if (resp.statusCode == 100) {
// this is a continue, we will anyway thankyou - so don't send to client
// Seriously though, look in to this more - I'm not sure
ignoring is OK, but
// it seems right
return;
}
res.writeHeader(resp.statusCode, resp.headers);

resp.addListener("data", function (chunk) {
res.write(chunk, "binary");
});

// to wrap it up
resp.addListener('end', function () {
// close the response
res.close();
});
});

// This end goes on the response
//conn.addListener("end", function() {
// res.close();
//});

// I think this close off the client request before we could send data
//client.close();
});
});

// run
server.listen(4040);

// Mock version, talking to couch.
function pick_server(req, callback) {
callback('127.0.0.1', '5984');
}

(http://gist.github.com/311813)

It may not be the best way, I'm no expert myself - but it's working,
and it probably a good starting point. And seems to be chunking
through nicely.

Linc

> --
> You received this message because you are subscribed to the Google Groups "nodejs" group.
> To post to this group, send email to nod...@googlegroups.com.
> To unsubscribe from this group, send email to nodejs+un...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/nodejs?hl=en.
>
>

[jespern]

On Feb 23, 5:42 am, Lincoln Stoll <lst...@lstoll.net> wrote:
> Where you said you'd like to listen to it worked - after adding a few
> end listeners to clean up and move it on to the next phase. So close!
>
> This is something which I got working:

[...]

> function pick_server(req, callback) {
>   callback('127.0.0.1', '5984');
>
> }
>
> (http://gist.github.com/311813)
>
> It may not be the best way, I'm no expert myself - but it's working,
> and it probably a good starting point. And seems to be chunking
> through nicely.

Indeed, that does work. Now here's the twist, and the little detail I
left out (didn't think it mattered, but apparently it does):

pick_server is talking to memcached, to figure out where to go. For
some reason, this breaks the entire logic and I'm back to square 1.

E.g.:

function pick_server(req, callback) {
mc.get(<some key we figured out from req>, function (result) {
callback(result);
}
}

That hangs the server. I've debugged it as far as I can, and memcached
returns its response right away, and pick_server receives it just
fine. I'm guessing since we're calling out for a moment, 'req' gets
upset, and doesn't know what to do with its payload? It breaks both
GET and POST, fwiw.

I'm using a memcache library that didn't work with 1.3.0 (used
promises), so I rewrote it, if anyone wants to experiment:
http://gist.github.com/311911

It's of course entirely possible I screwed up the rewrite with my
feeble understanding of this (here's the original:
http://github.com/elbart/node-memcache/blob/master/memcache.js).

Jesper