Failed to create SSL Connection with authenticating HTTP CONNECT proxy calling a HTTPS service

[Fuzz]

Hello,

For vert.x 3.3.3, do we expect the following configuration to work?

* A http client 

* Calling a HTTPS service

* Via an authenticating HTTP/CONNECT capable proxy

* Accessing a HTTPS service

I find that with our corporate proxy, the call fails with:

Caused by: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 485454502f312e31203430302042616420526571756573740d0a43616368652d436f6e74726f6c3a206e6f2d63616368650d0a507261676d613a206e6f2d63616368650d0a436f6e74656e742d547970653a20746578742f68746d6c3b20636861727365743d7574662d380d0a50726f78792d436f6e6e656374696f6e3a20636c6f73650d0a436f6e6e656374696f6e3a20636c6f73650d0a436f6e74656e742d4c656e6774683a20333134360d0a0d0a3c48544d4c3e3c484541443e0d0a20203c5449544c453e52657175657374204572726f723c2f5449544c453e0d0a20203c2f484541443e0d0a20203c424f44593e0d0a20203c7461626c65206267636f6c6f723d2257484954452220626f726465723d2230222063656c6c70616464696e673d2230222063656c6c73706163696e673d2230222077696474683d2231303025223e0d0a20203c747220616c69676e3d2263656e746572223e0d0a20203c74643e0d0a20203c7461626c65206267636f6c6f723d2257484954452220626f726465723d2230222063656c6c70616464696e673d2230222063656c6c73706163696e673d2230222077696474683d22353230223e0d0a202020202020202020203c74626f64793e0d0a2020202020202020202020203c747220616c69676e3d227269676874223e0d0a202020202020202020202020202020202020203c74643e0d0a202020202020202020202020202020202020203c494d47205352433d22687474703a2f2f666d2d65752d6c6f6e2d7061632e666d2e7262736772702e6e65743a383038302f696d616765732f5242536c6f676f312e4749462220424f524445523d2230223e0d0a2020202020202020202020202020202020203c2f74643e0d0a2020202020202020202020203c2f74723e0d0a2020202020202020202020203c74723e0d0a2020202020202020202020202020202020203c74643e0d0a2020202020202020202020202020202020203c464f4e5420666163653d22415249414c2c48454c56223e3c6269673e3c7374726f6e673e3c2f7374726f6e673e3c2f6269673e3c42523e3c2f464f4e543e3c62723e0d0a2020202020202020202020202020202020203c2f74643e0d0a2020202020202020202020203c2f74723e0d0a202020202020202020203c626c6f636b71756f74653e0d0a202020202020202020203c54523e202020203c54443e0d0a2020202020202020202020202020202020203c464f4e5420666163653d22415249414c2c48454c56223e203c6269673e52657175657374204572726f722028696e76616c69645f72657175657374293c2f6269673e3c42523e3c42523e3c2f464f4e543e0d0a2020202020202020202020202020202020203c2f54443e0d0a202020202020202020203c2f54

...

Sep 20, 2016 4:28:04 PM io.vertx.core.http.impl.HttpClientRequestImpl

SEVERE: javax.net.ssl.SSLHandshakeException: Failed to create SSL connection

The client code is this:

import io.vertx.core.AbstractVerticle;
import io.vertx.core.Vertx;
import io.vertx.core.http.HttpClientOptions;
import io.vertx.core.http.HttpClientRequest;
import io.vertx.core.net.ProxyOptions;
import io.vertx.core.net.ProxyType;

public class Client extends AbstractVerticle {

  // Convenience method so you can run it in your IDE
  public static void main(String[] args) {
    Vertx.vertx().deployVerticle(new Client());
  }

  @Override
  public void start() throws Exception {
    HttpClientRequest request = vertx.createHttpClient(new HttpClientOptions().
      setProxyOptions(new ProxyOptions()
        .setType(ProxyType.HTTP)
        .setHost("localhost")
        .setPort(8080)
        .setUsername("me")
        .setPassword("pwd"))
      .setSsl(true)
      .setTrustAll(true)
      .setVerifyHost(false)
    )
      .put(4443, "localhost", "/", resp -> {
        System.out.println("Got response " + resp.statusCode());
        resp.bodyHandler(body -> System.out.println("Got data " + body.toString("ISO-8859-1")));
      });
    request.end();
  }
}

I've tried to create a standalone reproducer using an off-the-shelf proxy (littleproxy) but I haven't as yet been able to setup the certificates to behave correctly (which may be a clue!).

Any pointers much appreciated. 

Thanks

Fuzz.

[Julien Viet]

it should work:

http://vertx.io/docs/vertx-core/java/#_using_a_proxy_for_client_connections

--
You received this message because you are subscribed to the Google Groups "vert.x" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.
Visit this group at https://groups.google.com/group/vertx.
To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/a35f3f74-60d9-446e-9226-10a6e9e846a3%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Fuzz]

I think what's going on is that vert.x is attempting to create a TLS connection to the proxy, when in fact the proxy doesn't support client TLS connections. 

The hex string returned decodes to:

HTTP/1.1 400 Bad Request

Cache-Control: no-cache

Pragma: no-cache

Content-Type: text/html; charset=utf-8

Proxy-Connection: close

Connection: close

Content-Length: 3146

<HTML><HEAD>

  <TITLE>Request Error</TITLE>

  </HEAD>

  <BODY>

  <table bgcolor="WHITE" border="0" cellpadding="0" cellspacing="0" width="100%">

  <tr align="center">

  <td>

  <table bgcolor="WHITE" border="0" cellpadding="0" cellspacing="0" width="520">

          <tbody>

            <tr align="right">

                   <td>

                   <IMG SRC="logo1.GIF" BORDER="0">

                  </td>

            </tr>

            <tr>

                  <td>

                  <FONT face="ARIAL,HELV"><big><strong></strong></big><BR></FONT><br>

                  </td>

            </tr>

          <blockquote>

          <TR>    <TD>

                  <FONT face="ARIAL,HELV"> <big>Request Error (invalid_request)</big><BR><BR></FONT>

                  </TD>

...

So, what's the right configuration of the client to request a HTTPS request to a HTTP-only proxy?

If I configure with .setSsl(false), then the connection to the HTTP-only proxy is HTTP, but the service URL protocol is HTTP also.

[Julien Viet]

I remember I had a discussion with Alex during this summer and we agreed that CONNECT for clear text is not supported.

I tried to find the conversation on echelog IRC but I cannot find it.

Alex would you mind to elaborate ?

-- 
You received this message because you are subscribed to the Google Groups "vert.x" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.
Visit this group at https://groups.google.com/group/vertx.

To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/a5db0d00-b930-41b8-9c39-f4b8f274e1bf%40googlegroups.com.

[Fuzz]

Thanks Julien.

It would be certainly useful to understand the rational. Persuading our corporation to change the behaviour of the proxy will be interesting!

In the meantime, perhaps I can work around this by having another implementation of HttpClient ... hmm ..

[Julien Viet]

On Sep 20, 2016, at 7:58 PM, Fuzz <farzad.pe...@gmail.com> wrote:

Thanks Julien.

It would be certainly useful to understand the rational. Persuading our corporation to change the behaviour of the proxy will be interesting!

you can search on echelog the discussion but the search engine does not work very well

In the meantime, perhaps I can work around this by having another implementation of HttpClient ... hmm ..

or contribute to Vert.x Core :-)

To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/8f150d4b-c660-4c15-a9ed-093f70c08343%40googlegroups.com.

[Alexander Lehmann]

I would think it should work with vert.x 3.3.3, the implementation for proxy supports ssl/tls connections, which isn't handled by the proxy directly, rather it forwards the connection to the origin server and that server will handle the tls connection.

It would also support non-ssl connections, but that is handled differently by the proxy, it doesn't use the CONNECT method.

Two things I think are not quite right, first of all, the Bad Request response should be handled by netty and returned as an proxy connect exception instead of NotSslRecordException, this probably means that it tried to start the ssl protocol without checking the reply from the proxy, second I would expect a Forbidden or Unauthorized reponse if the proxy rejects the connection, not Bad Request.

I will do some tests and post my findings, I usually use a Squid proxy to check the correct operation of the client.

[Alexander Lehmann]

Ok, the tests I have implemented work correctly, both with a Squid running with user auth and with Littleproxy without authentication.

Maybe the issue is with the proxy, but I would guess that if the proxy can be accessed with e.g. curl, it should work with the vertx/netty implementation as well.

When running Squid on port 3128, I can access https sites with the follow code:

import io.vertx.core.AbstractVerticle;
import io.vertx.core.Vertx;
import io.vertx.core.http.HttpClientOptions;
import io.vertx.core.http.HttpClientRequest;
import io.vertx.core.net.ProxyOptions;
import io.vertx.core.net.ProxyType;

public class Client extends AbstractVerticle {

  // Convenience method so you can run it in your IDE
  public static void main(String[] args) {
    Vertx.vertx().deployVerticle(new Client());
  }

  @Override
  public void start() throws Exception {
    HttpClientRequest request = vertx.createHttpClient(new HttpClientOptions().
      setProxyOptions(new ProxyOptions()
        .setType(ProxyType.HTTP)
        .setHost("localhost")

        .setPort(3128)
        .setUsername("user")
        .setPassword("user"))
      .setSsl(true)
      .setTrustAll(true)
      .setVerifyHost(false)
    )
      .get(443, "www.lehmann.cx", "/", resp -> {

        System.out.println("Got response " + resp.statusCode());
        resp.bodyHandler(body -> System.out.println("Got data " + body.toString("ISO-8859-1")));
      });
    request.end();
  }
}

(the same works with curl -s -x localhost:3128 -U user:user --head https://www.lehmann.cx)

maybe it would be possible to the check your proxy config with curl as well, that should give a log of the protocol lines.

If that works, the next step would be to run the vertx proxy test with logging enabled on netty and on vertx

[Alexander Lehmann]

If you can do tcpdump or wireshark on your computer, it would be probably helpful if you could sniff the connection to the proxy and take a look at the tcp stream for that.

Maybe there is an issue in vertx 3.3.3 with proxy handling when some conditions are different then what I am testing.

[Alexander Lehmann]

I thought about the different possible errors that may cause this error message and it is possible that the proxy is working correctly, but you can connecting to a http server instead of a https server by accident (though port 4443 would suggest a https server).

Maybe it would be worthwhile if you could check if the same connect works with curl with the proxy

[Fuzz]

Thanks Alex.

Definitely, it's a https service (we wrote it with vert.x).

 

I couldn't get the wireshark anonymiser tools on my work desktop. 

Instead have provided redacted curl verbose output (GET to https://www.redhat.com/en)

Is this useful?

Can you share your LittleProxy configuration please?

thanks

Fuzz

[Alexander Lehmann]

The curl dump looks quite normal, I would expect that this should work with vertx/netty as.

My littleproxy project is this (really just a main to configure and start the server, I had to put that into a separate project since it uses netty 4.0 I think that will not work with vertx since that uses netty 4.1 now)

https://github.com/alexlehm/littleproxy-example

My vert.x project to access the proxy and the server that is accessed as origin server is this.

https://github.com/alexlehm/vertx-proxy-example

[Mumuney Abdlquadri]

Hi Fuzz,

I am not sure this might help but once I had issue connecting to an HTTPS service within a VPN setup.

I had to set -Djsse.enableSNIExtension=false in the command line of java -jar...

If it fixes the issue kindly let us know.

--

You received this message because you are subscribed to the Google Groups "vert.x" group.

To unsubscribe from this group and stop receiving emails from it, send an email to vertx+unsubscribe@googlegroups.com.

Visit this group at https://groups.google.com/group/vertx.

To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/5e4b820d-cb8e-470b-9d60-f02f93e0efc3%40googlegroups.com.