How to intercept raw bytes while proxying

[Andy Jackson]

I'm attempting to intercept the raw bytes of a proxied connection. I'm currently able to add inbound and outbound channel handlers as shown here:

https://github.com/anjackson/warc-proxy/blob/master/src/main/java/net/anjackson/warc/proxy/WarcProxyFiltersSourceAdapter.java#L60-L77

However, this seems to miss the initial HTTP request from the client, as this seems to be consumed elsewhere in the code and all my code can see is the HttpObject rather than the raw ByteBuf that preceded it. If, on the other hand, I make a HTTPS connection, then I can record the client-to-proxy inbound request just fine (both the CONNECT and the GET, I think).

Is this a bug? Or is my approach mistaken?

Also, ideally, I'd like to be able to record the proxy-to-server request and response in the same way. Is it possible to modify the proxy-to-server pipeline somehow?

(b.t.w. I'm using https://github.com/ganskef/LittleProxy-parent for this).

Thanks.

Andy Jackson

[Frank Ganske]

Hello Andy,

second question first - the context of the server connection is given to
you here:

HttpFilters.proxyToServerConnectionSucceeded(ChannelHandlerContext)

There is it possible to modify the proxy-to-server pipeline.
https://github.com/adamfisk/LittleProxy/blob/master/src/main/java/org/littleshoot/proxy/HttpFilters.java#L210

I'm not shure with the first question, but I think it's the
HttpRequestDecoder added here:

https://github.com/adamfisk/LittleProxy/blob/master/src/main/java/org/littleshoot/proxy/impl/ClientToProxyConnection.java#L696

This is consuming the bytes you are searching for.


Regards Frank

> --
> You received this message because you are subscribed to the Google
> Groups "LittleProxy" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to littleproxy...@googlegroups.com
> <mailto:littleproxy...@googlegroups.com>.
> For more options, visit https://groups.google.com/d/optout.

[Andy Jackson]

Hi Frank,

Thanks for your help. I spotted (https://github.com/adamfisk/LittleProxy/pull/185) just after I posted and gave it a whirl, but I seemed to hit the same issue the other way around, as in I seems to get access to the channel too late to intercept the bytes of the initial call out to the remote server. I'm guessing that by the time my proxyToServerConnectionSucceed implementation was called, the ByteBuf had already been and gone.

Thanks for the pointer to the decoder. I'll try sub-classing the two decoders and replacing the 'vanilla' ones with ones that copy off the bytes before the decoding happens. I guess that will help test if I'm right about the HttpFilters not being applied early early enough in the process to catch the initial requests. Or, of course, I may have misunderstood how Netty's pipelines and channels work, which is entirely plausible!

Thanks again.

Andy

> <mailto:littleproxy+unsub...@googlegroups.com>.

[Frank Ganske]

Hello Andy,

you've seen the example how to modify the pipeline at the README.md?

ChannelPipeline pipeline = serverCtx.pipeline();
if (pipeline.get("inflater") != null) {
pipeline.remove("inflater");
}

This removes the inflater before the first chunk is handled by Netty. It
should be possible this way to replace the HTTP handlers with handlers
for your protocol. I suppose to dig into the Netty examples and try a
little one direction client handler first.

Absolutely I know the feeling, I may have to misunderstood...


Regards Frank

> > an email to littleproxy...@googlegroups.com <javascript:>
> > <mailto:littleproxy...@googlegroups.com <javascript:>>.

> > For more options, visit https://groups.google.com/d/optout

> <https://groups.google.com/d/optout>.

>
> --
> You received this message because you are subscribed to the Google
> Groups "LittleProxy" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to littleproxy...@googlegroups.com

> <mailto:littleproxy...@googlegroups.com>.

[Andy Jackson]

Hi Frank,

Thanks for that. I'm not sure what I was doing wrong before, but that method works perfectly for intercepting both the in-bound and out-bound proxy-to-server traffic. I still can't seem to intercept the initial user-to-proxy request, but I don't think I actually need that now that I have full access to the proxy-to-server communications.

Thanks again for your help - much appreciated.

Best wishes,
Andy

>      > <mailto:littleproxy+unsub...@googlegroups.com <javascript:>>.

>      > For more options, visit https://groups.google.com/d/optout
>     <https://groups.google.com/d/optout>.
>
> --
> You received this message because you are subscribed to the Google
> Groups "LittleProxy" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to littleproxy...@googlegroups.com

> <mailto:littleproxy+unsub...@googlegroups.com>.