Filter HTTPS/SSL requests

[Vitaly Polonetsky]

Hi,

First of all thanks for a wonderful proxy. After some research I ended up using LittleProxy because of the easy embedding and stable implementation.

I need to use LittleProxy to intercept HTTP and HTTPS requests. HTTP request intercepting worked out of the box. But HTTPS was only opening a TCP connection and the client/browser could use the connection to proxy any TCP connection. LittleProxy wasn't aware of the data that was transmitted there.

So I forked the proxy and tried to implement some kind of SSL MITM proxy filtering. So far I got to the situation where SelfSignedKeyStoreManager is used to generate a certificate and using SSLHandler I'm getting the plain HTTP request in HttpConnectRelayingHandler, instead of an encrypted traffic.

I had a few problems with my changes and I though you could help me. The first problem is that in some situations I'm getting:
javax.net.ssl.SSLException: Received close_notify during handshake
I believe it's some kind of a timing issue.

And the second problem is that I need to add filtering to or in addition to HttpConnectRelayingHandler. Can somebody help me with it.

My fork is here, intercept-ssl branch:
https://github.com/mvitaly/LittleProxy.git

Thanks.

[Vitaly Polonetsky]

No, I didn't solve that problem yet.
I believe the problem is some race condition between server and client channels, like client is done with the handshake and starts receiving HTTP request while the server is still in the middle of the handshake. But that's only my guess and I didn't have time to investigate it further.

On Nov 1, 2012 1:25 AM, "Jon Southall" <jondso...@gmail.com> wrote:

Hi Vitaly,

Did you resolve this problem? I am looking to do something similar and would be interested to know the challenges you faced

Thanks

Jon 

[Nir Asis]

Hi Vitaly,

any new here? Did you move to a different solution?

As I wrote in the past I need the exact same thing and I'm going to work on it this week. I have started looking into your code and the test you have created. I'll let you know if I'll be able to fix it. Please let me know if you were able to solve this issue differently or using other tools.

Nir

[Nir Asis]

So, I was able to implement an initial solution for SSL interception. It is initial since it doesn't generate certificates yet. I currently trust any server certificate and every client connected to the proxy should do that to. Here is my fork of the proxy:

https://github.com/nasis/LittleProxy/commits/master

I will really appreciate some review of the code especially from you Adam and you Vitaly.

Please notice that there are two commits where the second one is simply a different way for keeping track on open connections. Adam, I'll appreciate your opinion here on the better method of the two.

I also have two other open issues:

- will this configuration still work with proxy chaining? (I'm not so sure)

- is the current flow of connecting and handshaking the best one or should it be managed differently?

Please send any questions you have and please lets have a discussion on what's the best way to keep track on open connections and which handlers should to that.

Nir

[Adam Fisk]

This is really great Nir! Will review as soon as I have a second. Will be a great addition. Also you may have noticed I just submitted this bug, which is a pretty big one:

https://github.com/adamfisk/LittleProxy/issues/53

and may impact how channels are tracked in SSL intercept as well, depending on the cause of the issue (not clear yet).

Thanks -- this is an exciting change.

-Adam

[Nir Asis]

Thanks for letting me know on this bug. We actually run some tests using Apache Bench back when we have evaluated LittleProxy and didn't find any such issue even with large files but it seems that you came across some very specific scenario.

I'm going to keep on working on the full SSL implementation and hopefully you'll be able to fix this bug as well.

Notice that I have moved Netty to version 3.6.1 mainly since many changes to SSLHanlder were made.

Nir

[Adam Fisk]

This is fixed for the most important case -- basically for downloads but not uploads.

I was hoping to get to your SSL patch over the weekend but didn't make it. Unfortunately I'm also really tied up this week meeting with my team, so I won't be able to look at it until a week from now at the earliest.

-Adam

[Adam Fisk]

I think this generally looks cool in terms of adding in the SSL handshake with the client and the separate SSL handshake with the target server. It looks like it just does the handshake now, is that correct? It doesn't actually relay any data once the handshake is complete? I just want to make sure I'm not missing something.

I think that actually relates to the connection handling question. It should not ever be the case that an incoming request does not have a host/port pair. Of course the middle of a request will not (uploading a large file, for example), but any new request should have a host and a port in either the request URI itself or in the headers.

So if the host and port are blank on line 561 of HttpRequestHandler, there's just something wrong. We definitely don't want to just blindly return the old connection, as presumably a new incoming request indicates either we should have a new connection or a request to the same existing connection (so would be right in some cases).

Can you describe your thinking behind this a little bit? It seems like having a better test that actually relays data back and forth along the MITMed connection might make some of that interplay clearer.

Overall, though, great addition and good stuff! Would love to ultimately integrate it with some sort of config flag.

-Adam

[Nir Asis]

So, first, the implementation does relay data over the connections, it
is not just a handshake.
Take a look at HttpRequestHandler#writeConnectResponse I am not
changing the chain anymore.
There is also a test to demonstrate that called HttpsFilterTest.

The connection issue we see here is very similar to what you describe
as uploading a large file.
The CONNECT and GET requests are coming over the same TCP connection.
There is no case
where a request arrives without the hostport in the URL. I don't
really understand why the browser (or maybe it's LittleProxy??)
bothers to remove the hostport from the URL and move them to a HOST header.

More generally I didn't understand why
you chose to implement the map<hostport, connection>. To my
understanding every HTTP request will arrive on a different TCP socket
and therefore will have it's own pipeline. Therefore all we need to
keep track of is the connection/channel to the web server. If it
exists
we simply use it if not we create it. Please correct me if I'm wrong
here since HTTPRequestHandler could be simplified greatly if it is the
case.

If you'll integrate the current code to LittleProxy, browsers
connecting to it will reject the certificate the proxy provides.
I'm currently working on solving that but in any case any user using
this proxy will have to plant a new CA Root certificate its browser.
Therefore having this functionality controlled by a flag makes lots of
sense but will require more development.
Anyhow, this is a discussion by itself. Lets work on the first issues
first and we'll close this one on a different thread.

[Joe Bowbeer]

What is the current state of support for transparent proxying of HTTPS connections?

Has that discussion been continued on another thread?

The approach taken by Burp Suite and TcpCatcher is to add a trusted CA at the originating client.  Is this the approach envisioned for LittleProxy?