Possible to aggregate chunks as a full request/response before sending them to their destination?

[Zackary Whipkey]

Hi,

Is it possible to aggregate chunks as a full request/response before sending them to their destination, without having to declare a max size of request/response buffers in the Http filters? 

Currently, my proxy works like this:

received clientToProxy chunk!
GET
http://www.google.com/
send proxyToServer chunk!
received clientToProxy chunk!
send proxyToServer chunk!
received serverToProxy chunk!
send proxyToClient chunk!
received serverToProxy chunk!

I'd like to get it to work like this! 

received clientToProxy request!
GET
http://www.google.com/
send proxyToServer request!
received clientToProxy response!
send proxyToClient response!

However, LittleProxy's implementation of HttpObjectAggregator seems to require a static number in regards to max request/response size.

Id like all HTML traffic to flow through this buffer/aggregator, but all requests with a content type of a file or other large formats to follow through as normal chunks/etc.

Is this possible? I realize that even inspecting the content type would require inspection of the first chunk/packet. 

[Frank Ganske]

Hi,

some quick ideas here:

Respect: it's not assembling chunks only, but inflating/deflating too
what declare a max size handles for you and it's not Gzip only.

I think, you could answer every text/html (but there are more content
types) chunk with a no content response and spool your data somewhere.
On last chunk, you can answer a new FullResponse. I've never tried it.

You are free to set a max size or not depending the URL on creating
FiltersSource. This means, you have to try to detect the content by URL.


Regards Frank

> --
> You received this message because you are subscribed to the Google
> Groups "LittleProxy" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to littleproxy...@googlegroups.com
> <mailto:littleproxy...@googlegroups.com>.
> For more options, visit https://groups.google.com/d/optout.

[Mark Feeney]

Hi Zackary.

I don't think there's an built-in way to do conditional response buffering with the current API.  You can still do it -- you'll just have to do the buffering yourself in your HttpFilters implementation.  If you search this group for "stateful" you'll find some additional discussion about how to do this.

I have the same issue in my app but so far have taken the lazy route and just buffer everything. It's a non-solution but for my use case it happens to work (for now).

Mark

[Frank Ganske]

Nonsense:

> You are free to set a max size or not depending the URL on creating
> FiltersSource. This means, you have to try to detect the content by URL.

Sorry, this answer was to quick. I had a closer look. You are free to
return different Filter implementations depending the URL.

What I've done: no max size, chunks spooled for caching, but forwarded
with no modification, later answered in offline mode with a new
FullResponse. This means, the first request is answered unmodified.

I prefer to set a max size depending the system now, for Java 10MB, for
Android 2 MB. This allows me to answer modified to the first request.


But I think, answering some chunks with state CONTINUE_100 should be
possible. You have to detect the chunk before last, to replace with your
modified content. The last chunk is necessary and has always no data.
Sounds like a hack. :-(


In the FiltersSource is a fairly new method filterRequest(HttpRequest,
ChannelHandlerContext) with the context added. This makes it possible,
to modify the pipeline and removing the aggregation handler before
answering. And this means, you have to try to detect the content by URL
again. Nice idea. I will give it a try. :-)

[Zackary Whipkey]

I was thinking about playing with the context object. However, I was a little fearful of playing with the pipeline. Is the pipeline meant to be dynamic? Can we add and remove handlers/adapters based upon a request? 

I will investigate this too.

[Frank Ganske]

Am 13.07.2015 um 21:01 schrieb Zackary Whipkey:
> I was thinking about playing with the context object. However, I was a
> little fearful of playing with the pipeline. Is the pipeline meant to be
> dynamic? Can we add and remove handlers/adapters based upon a request?
>
> I will investigate this too.

No way, since this is the context of the client to proxy, but we need
the context of the proxy to server connection.


This was introduced with:

commit 5e04c7315eae741abae9cf548fc3ca666af9b143
Author: Ox To A Cart <ox.to....@gmail.com> 2014-01-13 14:32:23

Added ability to access ChannelHandlerContext from filters


Yes, a pipeline is dynamic, but I'm not very familiar with this Netty
stuff, for now...


The serverConnection.ctx could be a parameter at
org.littleshoot.proxy.HttpFilters.proxyToServerConnectionSucceeded()

Yes, this works. It's worth a pull request, but I'm on a railway with a
very intermittent internet connection.

[Frank Ganske]

This issue is handled with
https://github.com/adamfisk/LittleProxy/pull/185 now.