HTTPS proxy issue using CONNECT method

[Mauro Monteiro]

Hello all,

I have been facing an issue when I try to create a HTTP client which needs to connect through a HTTPS proxy using the HTTP CONNEC method. I know that it can be achieved setting my own http.Transport object. However the issue seems to be in the current implementation of /net/http/transport.go code.

In my environment, I am developing a HTTP client which ALWAYS use a HTTPS proxy using HTTP CONNECT method. This client is allowed to reach HTTP or HTTPS targets. Therefore, I noticed that when I try to reach a HTTPS target, the the transport layer works as expected and it uses the HTTP CONNECT method. However, when I try to reach a HTTP target, the transport does not use the CONNECT  method.

Looking at the transport.go code, I realized that the check to use the CONNECT method is based on the protocol of the target instead of being on the protocol of the proxy URL. Below is a link showing that:

1. HTTP check

https://cs.opensource.google/go/go/+/refs/tags/go1.9.5:src/net/http/transport.go;l=1092

2. HTTPS check

https://cs.opensource.google/go/go/+/refs/tags/go1.9.5:src/net/http/transport.go;l=1099

As can be seen on the links above, the condition is based on cm.targetScheme instead of cm.proxyURL.Scheme. Is it a bug?

Go version: go version go1.19.3 linux/amd64

Mauro

[Sean Liao]

How are you setting the proxy?

- sean

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/d06cd78b-c012-425f-8b5e-52d4bd7a3cbcn%40googlegroups.com.

[Mauro Monteiro]

Hello Sean

The proxy is a HTTPS proxy which exposes an endpoint that ONLY accepts HTTP CONNECT method. The code checks with the HTTP method informed is CONNECT and then it tries to establish the TCP connection with the target informed. Otherwise the request is rejected. Therefore, the proxy is ALWAYS HTTPS however it should be able to relay request to HTTP or HTTPS targets.

Mauro

[gbarr]

On Saturday, December 3, 2022 at 4:46:54 PM UTC maumon...@gmail.com wrote:

Hello all,

I have been facing an issue when I try to create a HTTP client which needs to connect through a HTTPS proxy using the HTTP CONNEC method. I know that it can be achieved setting my own http.Transport object. However the issue seems to be in the current implementation of /net/http/transport.go code.

In my environment, I am developing a HTTP client which ALWAYS use a HTTPS proxy using HTTP CONNECT method. This client is allowed to reach HTTP or HTTPS targets. Therefore, I noticed that when I try to reach a HTTPS target, the the transport layer works as expected and it uses the HTTP CONNECT method. However, when I try to reach a HTTP target, the transport does not use the CONNECT  method.

This is normal. The CONNECT method allows a client to create a TCP tunnel through your gateway. This allows your client to perform all TLS negotiation.

However for HTTP requests this extra layering is not required. In the standard library you can see the setting of pconn.isProxy=true at https://cs.opensource.google/go/go/+/refs/tags/go1.9.5:src/net/http/transport.go;l=1093  this is later used when writing the request at https://cs.opensource.google/go/go/+/refs/tags/go1.9.5:src/net/http/request.go;l=521

Essentially it changes the form of the http method from GET /path/to/resource to GET http://hostname/path/to/resource so your gateway would then know that this is a proxy request and perform the external request

Graham.

[Mauro Monteiro]

Hello Graham,

Would not make more sense to deal with both cases HTTPS and HTTP in the same way? The transport could send the CONNECT method to establish the TCP tunnel and then send the request after that. Of course, if the target request is based on HTTPS, the TLS handshake would start otherwise the request would be sent in plain text. For instance, I tested both scenarios using curl command and they  behave exactly the same way, sending a CONNECT request.

curl -v --proxytunnel --proxy-insecure -x https://localhost:7443 http://localhost:8001 -> The proxy is HTTPS and the target is HTTP, curl sends a CONNECT and then the GET method

curl -v --proxytunnel --proxy-insecure -x https://localhost:7443 https://www.google.com -> The proxy is HTTPS and the target is HTTPS, curl sends a CONNECT and then the GET method

Based on what i understood from you comment, please sorry whether I misunderstood, the proxy would need to have a logic to check whether the target is HTTPS or HTTP and the act differently depending on the case. For HTTPS it would create the TCP tunnel while for HTTP it would forward the request received. Is it true?

Mauro

[gbarr]

Hi Mauro,

In your curl examples the CONNECT method was used in both cases because you forced it to with the --proxytunnel option. If you run those same commands without that option then you will find that curl will operate the same as Go and will only use the CONNECT method when the target is HTTPS

So yes the transport could operate in a way that you describe, as demonstrated by your curl commands. But doing so would add overhead that is unnecessary for a plain HTTP request. By using CONNECT you impose an extra round trip between the client and the proxy because first it has to establish the connection to the target site with the CONNECT then it sends the request.

Graham.

[Mauro Monteiro]

Hello Graham

Thanks a lot for your explanation ! 

I really appreciate.

Mauro