My question has to do with http proxy section: I haven't run into this problem yet, but I can see that for potentially large HTTP bodies (such as file transfers), httpMessage may dramatically increase in size. What is the best way to pipe this data over the client tcp socket as it comes in, instead of caching the whole thing and sending all at once? Would be even better if I could just pipe all the parts of the HTTP message (request line, headers, body, ...) as they come into the server.
> My question has to do with http proxy section: I haven't run into this problem yet, but I can see that for potentially large HTTP bodies (such as file transfers), httpMessage may dramatically increase in size. What is the best way to pipe this data over the client tcp socket as it comes in, instead of caching the whole thing and sending all at once? Would be even better if I could just pipe all the parts of the HTTP message (request line, headers, body, ...) as they come into the server.
Thank you, Mikeal. I use request quite a bit in my projects, and it's great. However, I won't be using request for this project, because I need to write this one at the transport layer. Eventually I will be redirecting various requests to other servers and examining data at the transport layer. I just need to make this barebones version function and then branch from it. Any ideas on how to pipe the request into the socket?
I thought of listening to the 'connection' event on the proxy server, and piping the socket into the tunnel. But this won't differentiate between http and https requests without parsing the request header, which I'm not sure how to do.
+1 for request. It will help you massively on the client side. I tried to do what you're doing a few weeks ago so I could use SSH in cafes. I found that I had to do write a blank string into the http stream on the client side to force request to send the request headers though. However, I think request's 'data' event might give you a String, not a Buffer. This means that any binary data might get corrupted. You can get round that by making a base64 stream encoder and decoder, but they have to pass binary data around as Buffer objects and not strings - stringstream didn't work for me.
Another problem you might get is that some firewalls might make sure that your full request is sent to the server before any response is delivered, which would prevent streaming. For instance, vodafone's 3g content filter will block that implementation. Here is my solution:
The client polls the server 5 times a second, and they relay all the Stream events to eachother, keeping track of the streams. Gets through most unhardcore firewalls. Unfortunately it still gets blocked by McAfee's firewall, so I can't use it in my local library :(
In the end I just gave up and worked from somewhere devoid of technophobic luddites :)
> My question has to do with http proxy section: I haven't run into this > problem yet, but I can see that for potentially large HTTP bodies (such as > file transfers), httpMessage may dramatically increase in size. What is the > best way to pipe this data over the client tcp socket as it comes in, > instead of caching the whole thing and sending all at once? Would be even > better if I could just pipe all the parts of the HTTP message (request > line, headers, body, ...) as they come into the server.
I'm not sure why you don't use .pipe().. you don't need to buffer all the request before sending it. The headers should have a content-length so your endpoint will know about it. Just send the data directly to the other end instead of buffering.
On Monday, July 30, 2012 at 22:40 , axs wrote:
> Thank you, Mikeal. I use request quite a bit in my projects, and it's great. However, I won't be using request for this project, because I need to write this one at the transport layer. Eventually I will be redirecting various requests to other servers and examining data at the transport layer. I just need to make this barebones version function and then branch from it. Any ideas on how to pipe the request into the socket?
> I thought of listening to the 'connection' event on the proxy server, and piping the socket into the tunnel. But this won't differentiate between http and https requests without parsing the request header, which I'm not sure how to do.
> Regards,
> Alex
> On Monday, July 30, 2012 5:21:04 PM UTC-4, Mikeal Rogers wrote:
> > request does all of this, including SSL tunneling.
> > -Mikeal
> -- > Job Board: http://jobs.nodejs.org/ > Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines > You received this message because you are subscribed to the Google
> Groups "nodejs" group.
> To post to this group, send email to nodejs@googlegroups.com (mailto:nodejs@googlegroups.com)
> To unsubscribe from this group, send email to
> nodejs+unsubscribe@googlegroups.com (mailto:nodejs+unsubscribe@googlegroups.com)
> For more options, visit this group at
> http://groups.google.com/group/nodejs?hl=en?hl=en
On Fri, Aug 3, 2012 at 10:37 AM, Diogo Resende <drese...@thinkdigital.pt> wrote:
> I'm not sure why you don't use .pipe().. you don't need to buffer all the
> request before sending it. The headers should have a content-length so your
> endpoint will know about it. Just send the data directly to the other end
> instead of buffering.
> --
> Diogo Resende
> On Monday, July 30, 2012 at 22:40 , axs wrote:
> Thank you, Mikeal. I use request quite a bit in my projects, and it's great.
> However, I won't be using request for this project, because I need to write
> this one at the transport layer. Eventually I will be redirecting various
> requests to other servers and examining data at the transport layer. I just
> need to make this barebones version function and then branch from it. Any
> ideas on how to pipe the request into the socket?
> I thought of listening to the 'connection' event on the proxy server, and
> piping the socket into the tunnel. But this won't differentiate between http
> and https requests without parsing the request header, which I'm not sure
> how to do.
> Regards,
> Alex
> On Monday, July 30, 2012 5:21:04 PM UTC-4, Mikeal Rogers wrote:
> request does all of this, including SSL tunneling.
Looks great! I'll need to play around with that. I can't be bothered to make my http tunnel disconnect proof though, because I've noticed SSH connections stay open when my wifi has dropped out.
> with a little bit more work you could make your http-tunnel into a > reliable stream > that maintained stream semantics over disconnects.
> On Fri, Aug 3, 2012 at 10:37 AM, Diogo Resende <drese...@thinkdigital.pt> > wrote: > > I'm not sure why you don't use .pipe().. you don't need to buffer all > the > > request before sending it. The headers should have a content-length so > your > > endpoint will know about it. Just send the data directly to the other > end > > instead of buffering.
> > -- > > Diogo Resende
> > On Monday, July 30, 2012 at 22:40 , axs wrote:
> > Thank you, Mikeal. I use request quite a bit in my projects, and it's > great. > > However, I won't be using request for this project, because I need to > write > > this one at the transport layer. Eventually I will be redirecting > various > > requests to other servers and examining data at the transport layer. I > just > > need to make this barebones version function and then branch from it. > Any > > ideas on how to pipe the request into the socket?
> > I thought of listening to the 'connection' event on the proxy server, > and > > piping the socket into the tunnel. But this won't differentiate between > http > > and https requests without parsing the request header, which I'm not > sure > > how to do.
> > Regards, > > Alex
> > On Monday, July 30, 2012 5:21:04 PM UTC-4, Mikeal Rogers wrote:
> > request does all of this, including SSL tunneling.
On Friday, August 3, 2012 4:37:48 AM UTC-4, Diogo Resende wrote:
> I'm not sure why you don't use .pipe().. you don't need to buffer all the > request before sending it. The headers should have a content-length so your > endpoint will know about it. Just send the data directly to the other end > instead of buffering.
