Problem with using a proxy for http client requests - handshakefailed for https

[Dominic Rübelzahn]

Hello,

I don't get a proxy for http client requests working when using https. I'm using a proxy which directly passes all data, it does not break SSL or switch certificates or anything else. When doing GETs with http everything works fine but when using https I get an 500 and "handshakefailed" error. I also downloaded the certificate and put it into a keystore but the problem remains.

Here is what I am doing:

// setup options
HttpClientOptions httpClientOptions = new HttpClientOptions();
httpClientOptions.setVerifyHost(false);
httpClientOptions.setTrustAll(true); // should normally solve our problem...
httpClientOptions.setSsl(false); // proxy is available via http

JksOptions jksOptions = new JksOptions();
jksOptions.setPassword("password");
jksOptions.setPath("keystore.jks");
httpClientOptions.setTrustStoreOptions(jksOptions);

jksOptions = new JksOptions();
jksOptions.setPassword("password");
jksOptions.setPath("keystore.jks");
httpClientOptions.setKeyStoreOptions(jksOptions);

// create request
HttpClient httpClient = vertx.createHttpClient(httpClientOptions);
HttpClientRequest request = httpClient.get(proxyPort, proxyHost, absoluteURL); // absolute url starts with https://...
request.putHeader("host", host); // host is extracted from the absolute url

request.handler(responseHandler -> {
    responseHandler.endHandler(v -> {
        System.out.println(responseHandler.statusCode() +" : " +responseHandler.statusMessage());
    });
});

// execute
request.end();

What am I doing wrong? What do I have to do to get it running?

Greetings,
Dominic

[Daniel Lindberg]

What happens if you change to setSsl(true);?

[Dominic Rübelzahn]

Hello,

an exception is thrown because the proxy is accessable via http

Exception:
io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record

[Daniel Lindberg]

I would try running your first example with -Djavax.net.debug=all

to see exactly why the handshake is failing. 

[Dominic Rübelzahn]

That doesn't print anything. I think it is because the proxy is accessable via http, only our target url is using https. I also configured logging to print as much as possible but no information about my problem is contained.
To test the proxy I configured it in my browser and called our url, everything works fine.

[Tim Fox]

This means you are trying to connect using SSL/TLS to a server that is not expecting SSL/TLS connections.

--
You received this message because you are subscribed to the Google Groups "vert.x" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.
Visit this group at https://groups.google.com/group/vertx.
To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/be0315c6-e24a-4c3c-8f53-41faefafb039%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Tim Fox]

If you want to tunnel HTTPS over a proxy I think you want to use HTTP CONNECT tunnelling:

https://en.wikipedia.org/wiki/HTTP_tunnel#HTTP_CONNECT_tunneling

You don't want to connect to the proxy via https otherwise the https would have to be terminated there.

--
You received this message because you are subscribed to the Google Groups "vert.x" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.
Visit this group at https://groups.google.com/group/vertx.

To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/81ac6d15-6054-4def-ba84-1d89e628e384%40googlegroups.com.

[Tim Fox]

See https://groups.google.com/forum/#!searchin/vertx/http$20connect$20proxy/vertx/rmFHRA9MHeQ/7SI-44pN07IJ

[Dominic Rübelzahn]

That looks interessting, I'll have a look and will let you know then. Many thanks so far!

[Dominic Rübelzahn]

Sadly I don't get it running... I tried the tunneling, created a connection to the proxy and then upgraded to ssl. Then i created a new GET request but when I call end() and error is thrown saying that the connection was closed. I have no glue so far what I am doing wrong and how to get it running.

Does anyone have a working tunneling example for ssl in vertx 3?

[Alexander Lehmann]

Unfortunately there is no code in vert.x currently to do that and you cannot use the vert.x classes for https when you implement the proxy CONNECT method since its not possible to tack a http request onto an already opened connection (I think at least its not possible).

Doing a proxy request involves the following steps:

Socket connect to proxy ip/port, write "CONNECT example.com:443 HTTP/1.1\r\n\r\n" to the socket, read the reply as http headers, it will be something like
"HTTP/1.0 200 Connection established\r\n\r\n"
(it might be possible to do that request as CONNECT request with the http client)

Now, you can do upgradeToSsl, do some validation of the certificate and after that you should be able to write the complete http request into the NetSocket and read the response headers and the response body from the socket.

[Julien Viet]

in the http2 branch, work has been done to have either http/1.x or http/2 protocol, both at the protocol level and at the connection pool because http/1.x uses a pool of several connections and http/2 uses a single connection with multiplexing.

The ConnectionManager.Pool has been introduced so the ConnectionManager (used directly by the HttpClient) can either uses http/1.x or http/2.

It seems that this proxy case could be implemented as a particular ConnectionManager.Http1xPool with an existing connection.

The api itself could provide an HttpClient after a connect request has been made.

The current pseudo code for using CONNECT is:

client.connect(…, resp -> {

   // Check response status then

   NetSocket socket = resp.netSocket();

});

In this case we would have something that would transform the NetSocket into an HttpClient that proxies that NetSocket:

client.connect(…, resp -> {

   // Check response status then

   NetSocket socket = resp.netSocket();

   HttpClient proxyClient = HttpClient.create(socket);

});

--
You received this message because you are subscribed to the Google Groups "vert.x" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.
Visit this group at https://groups.google.com/group/vertx.

To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/89091c88-0799-41fe-84cc-e8fcb9ed37b3%40googlegroups.com.

[Alexander Lehmann]

I have written an unit test for a https request to google.com using NetClient, the code cuts a few corners (e.g. it assumes that the complete response is in one Buffer), but it should be an example for the necessary protocol steps at least

https://gist.github.com/alexlehm/d0f401792f66b6aaa9aa

bye, Alexander

On Monday, March 21, 2016 at 4:10:47 PM UTC+1, Dominic Rübelzahn wrote:

[Tim Fox]

On 22/03/16 22:55, Alexander Lehmann wrote:

Unfortunately there is no code in vert.x currently to do that and you cannot use the vert.x classes for https when you implement the proxy CONNECT method since its not possible to tack a http request onto an already opened connection (I think at least its not possible).

+1. It would be a nice feature to support HTTP CONNECT tunnelling directly in the Http client so the client could be continued to be used in the normal way after doing the CONNECT.
PR anyone? ;)

Doing a proxy request involves the following steps:

Socket connect to proxy ip/port, write "CONNECT example.com:443 HTTP/1.1\r\n\r\n" to the socket, read the reply as http headers, it will be something like
"HTTP/1.0 200 Connection established\r\n\r\n"
(it might be possible to do that request as CONNECT request with the http client)

Now, you can do upgradeToSsl, do some validation of the certificate and after that you should be able to write the complete http request into the NetSocket and read the response headers and the response body from the socket.



On Monday, March 21, 2016 at 4:10:47 PM UTC+1, Dominic Rübelzahn wrote:

Sadly I don't get it running... I tried the tunneling, created a connection to the proxy and then upgraded to ssl. Then i created a new GET request but when I call end() and error is thrown saying that the connection was closed. I have no glue so far what I am doing wrong and how to get it running.

Does anyone have a working tunneling example for ssl in vertx 3?

--

You received this message because you are subscribed to the Google Groups "vert.x" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.
Visit this group at https://groups.google.com/group/vertx.

To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/89091c88-0799-41fe-84cc-e8fcb9ed37b3%40googlegroups.com.

[Alexander Lehmann]

I sortof have this on my "things I should do when I have time" list for vertx, but didn't get around to anything except complaining that it is not implemented.

The ssl code should be possible in netty, not sure if they have support for a https proxy

[Manish Kumar]

Alexander,

We also have exactly same use case where we need to tunnel to HTTPS resource via HTTP proxy.

I don't have good low level knowledge on what changes should be done in Vertx toolkit/API as suggested by Tim, but I am happy to help if any way possible.

[Fuzz]

+1 

In v3.3.3 we're seeing the same issue, attempting to use the corporate proxy to reach a public host. 

Getting rather close to go-live and I'm wondering how best to achieve this.

Julien, did you say that this may be in the http2 branch?

thanks and kind regards

Fuzz

[Julien Viet]

Hi,

this thread is quite old the http2 branch does not exist anymore

contribs and fixes have been done for client proxy support since then.

could you explain exactly what is the problem you are seing and maybe provide a reproducer ?

thanks

--
You received this message because you are subscribed to the Google Groups "vert.x" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.
Visit this group at https://groups.google.com/group/vertx.

To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/ac090015-c542-4611-8273-87f4d942eede%40googlegroups.com.

[Alexander Lehmann]

In 3.3.3 the proxy request should work for https with http 1.1 correctly, there was an issue with cert checking before.

If you have a reproducer, that would be great.