Most efficient way of piping HTTP body through a tunneling proxy

[axs]

I'm making an http proxy that tunnels data at the transport layer. Here is the complete code:

var http = require('http');

var net = require('net');

var url = require('url');

var proxy = http.createServer();

// proxy an HTTP request

proxy.on('request', function(req, res){

var uri = url.parse(req.url);

var httpMessage = req.method + ' ' + uri.path + ' HTTP/'+ req.httpVersion + '\r\n';

for(var header in req.headers){

httpMessage += header + ': ' + req.headers[header] + '\r\n';

}

httpMessage += '\r\n';

req.on('data', function(data){

httpMessage += data;

});

req.on('end', function(data){

httpMessage += data;

var client = net.connect(uri.port || 80, uri.hostname, function(){

client.write(httpMessage);

client.pipe(req.connection);

req.connection.pipe(client);

});

});

});

// proxy an HTTPS request

proxy.on('connect', function(req, socket, head) {

  var uri = req.url.split(':')

var tunnel = net.connect({

port: uri[1],

host: uri[0]

}, function() {

socket.write('HTTP/1.1 200 Connection Established\r\n\r\n');

tunnel.write(head);

tunnel.pipe(socket);

socket.pipe(tunnel);

});

});

proxy.listen(8080);

My question has to do with http proxy section: I haven't run into this problem yet, but I can see that for potentially large HTTP bodies (such as file transfers), httpMessage may dramatically increase in size. What is the best way to pipe this data over the client tcp socket as it comes in, instead of caching the whole thing and sending all at once? Would be even better if I could just pipe all the parts of the HTTP message (request line, headers, body, ...) as they come into the server.

Thank for any help in advance.

[Mikeal Rogers]

request does all of this, including SSL tunneling.

req.pipe(request(req.url, {proxy:'https://site.com'})).pipe(resp)

-Mikeal

--
Job Board: http://jobs.nodejs.org/
Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nod...@googlegroups.com
To unsubscribe from this group, send email to
nodejs+un...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en

[axs]

Thank you, Mikeal. I use request quite a bit in my projects, and it's great. However, I won't be using request for this project, because I need to write this one at the transport layer. Eventually I will be redirecting various requests to other servers and examining data at the transport layer. I just need to make this barebones version function and then branch from it. Any ideas on how to pipe the request into the socket? 

I thought of listening to the 'connection' event on the proxy server, and piping the socket into the tunnel. But this won't differentiate between http and https requests without parsing the request header, which I'm not sure how to do.

Regards,

Alex

[Diogo Resende]

I'm not sure why you don't use .pipe().. you don't need to buffer all the request before sending it. The headers should have a content-length so your endpoint will know about it. Just send the data directly to the other end instead of buffering.

-- 

Diogo Resende

[Dominic Tarr]

@john I applaud your efforts for FREE WIFI!

does the firewall permit http pipelining?

also, have you heard of dnstunneling? http://dnstunnel.de/

with a little bit more work you could make your http-tunnel into a
reliable stream
that maintained stream semantics over disconnects.

[ajc]

Looks great! I'll need to play around with that. I can't be bothered to make my http tunnel disconnect proof though, because I've noticed SSH connections stay open when my wifi has dropped out.

John

> nodejs+unsubscribe@googlegroups.com

> For more options, visit this group at
> http://groups.google.com/group/nodejs?hl=en?hl=en
>
>
> --
> Job Board: http://jobs.nodejs.org/
> Posting guidelines:
> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
> You received this message because you are subscribed to the Google
> Groups "nodejs" group.
> To post to this group, send email to nod...@googlegroups.com
> To unsubscribe from this group, send email to

> nodejs+unsubscribe@googlegroups.com

[axs]

 I tried piping but was having issues: The following code throws an "Cannot call method 'on' of undefined" error at the last line.

proxy.on('request', function(req, res){

var uri = url.parse(req.url);

var httpMessage = req.method + ' ' + uri.path + ' HTTP/'+ req.httpVersion + '\r\n';

for(var header in req.headers){

httpMessage += header + ': ' + req.headers[header] + '\r\n';

}

httpMessage += '\r\n';

var client = net.connect(uri.port || 80, uri.hostname);

client.write(httpMessage);

req.connection.pipe(client);

client.pipe(res.connection);

});

When I change that last line to client.pipe(req.connection); , it works, but for some requests I get multiple warnings:

 "(node) warning: possible EventEmitter memory leak detected. 11 listeners added. Use emitter.setMaxListeners() to increase limit." 

at the last two lines. I think I'm piping correctly in the first scenario, but I'm not sure what the source of errors/warnings is. 

Regards,

Alex