HTTPS connection through proxy (CONNECT HTTP Method)

[Sean Kemplay]

Hi All,

Sending an http request through our corporate proxy works as follows for http requests ->

(define-values (x y z)
(http-sendrecv "10.0.0.200" "http://www.example.com"
#:port 8080
#:headers '(
"Proxy-Authorization: Basic base64encodedcredentials"
"Proxy-Connections: keep-alive"
)
#:ssl? #f
#:method "GET"))

However fails for HTTPS requests (as expected).

What I need to do is make a request like the above using the #:method "CONNECT" and then make a secondary request through a returned connection.

Does anyone know how I would go about doing that in Racket?

Kind regards,
Sean

[Jay McCarthy]

http-sendrecv combines calls to http-conn-open, http-conn-send!,
http-conn-recv!, then http-conn-close!. I suspect that you just need
to break up that one big call into a few calls like open, send, recv,
send, recv, close. I'd be happy to work on it with you, but I don't
have such a proxy on hand, so I'll need helping testing it.

Jay

>
> Kind regards,
> Sean
>
> --
> You received this message because you are subscribed to the Google Groups "Racket Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to racket-users...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.



--
Jay McCarthy
http://jeapostrophe.github.io

"Wherefore, be not weary in well-doing,
for ye are laying the foundation of a great work.
And out of small things proceedeth that which is great."
- D&C 64:33

[Sean Kemplay]

Hi Jay,

Thanks for that, yes I think you are right. I have just installed squid at home which also supports http tunnelling. I'll see how I get on and post my results - whether they be good or bad!

It would be really good to at least get an example on the wiki for others to build from, as I suspect a lot of corporate networks are behind proxies and this would be problematic in using Racket to make calls to REST APIs etc which my job at least requires a lot of.

Kind regards,
Sean

[Sean Kemplay]

Hi Jay,

Unfortunately I am not getting very far with this.

Our app servers where our production code sits are not behind a proxy, so at the end of the day it doesn't rule out using Racket for some of the tasks I have in mind. It would be nice to be able to get through the proxy from my desktop to test code though.

I tried the following but the connection is ending early -

#lang racket
(require net/http-client)

(define conn (http-conn-open "10.0.0.200" #:port 8080))

(http-conn-send! conn "https://news.ycombinator.com/" #:method "CONNECT" #:headers '("Proxy-Authorization: Basic base64encodedcredentials" "Connection: Keep-Alive") #:version #"1.1")
(define-values (a b c)(http-conn-recv! conn #:close? #f))
(http-conn-send! conn "/" #:method "GET"); #:headers '("Proxy-Authorization: Basic Y3hnXHNlYW4ua2VtcGxheTpBdWd1c3QyMDE0" "Connection: Keep-Alive") #:version #"1.1")
(define-values (e f g) (http-conn-recv! conn))
(http-conn-close! conn)

I am basing the above on this S/O post but am not entirely sure if this is even the path I should be pursuing!

http://stackoverflow.com/questions/11697943/when-should-one-use-connect-and-get-http-methods-at-http-proxy-server

Here is an example using Node, it opens the connection to the uri through the proxy, then writes to that connection through an SSL connection which is something I can't see how to do in Racket.

http://blog.vanamco.com/proxy-requests-in-node-js/

I will keep digging, if you have any Ideas or anything I could test against our network it would be much appreciated.

Kind regards,
Sean

[Sean Kemplay]

Just an update on this, looking at the code for http-client I now understand that http-conn is a struct with an input and output port from a tcp connection.

Based on the node.js example I am thinking of instead of calling http-conn-send! a second time with a different method I need to write a function along the lines of http-conn-tunnel! which somehow pipes ssl input and output ports from the tcp input and output ports from http-conn's input and output pipes.

[Sean Kemplay]

I haven't given up... yet!

I exported http-conn-from and http-conn-to in a new racket instance so I could grab the ports and have tried the following -

#lang racket
(require net/http-client)
(require openssl)
;;(require openssl)
(define conn (http-conn-open "127.0.0.1" #:port 3128))
(http-conn-send! conn "news.ycombinator.com:443" #:method "CONNECT" #:headers '("Connection: Keep-Alive"))
(define-values (a b c) (http-conn-recv! conn))
;(tunnel conn)
(http-conn-live? conn)
(let-values ([(i o) (ports->ssl-ports
(http-conn-from conn)
(http-conn-to conn)
#:context
(ssl-make-client-context 'auto)
#:mode 'connect
#:hostname "news.ycombinator.com")])
(fprintf o "GET / HTTP/1.1\r\nHost: news.ycombinator.com:443\r\n\r\n")

If I check the value of a it is #"HTTP/1.1 200 Connection established" - which means is step one working correctly.

Step 2 is to pipe ssl traffic over the existing connection which is failing with -

port->ssl-ports: connect failed (input terminated prematurely)

Has anyone used port->ssl-ports before?

I have made http-conn transparent and there is definitely an input-port and output-port available as per the contract of port->ssl-ports so I must be providing the correct types of arguments.

It's doing my head in a bit but the functionality is important to me and I am learning new things so I will push on :-) If anyone can spot any fundamental flaws in my assumptions on how an ssl tunnel should work or in what I am doing in racket it would be much appreciated.

On a final note, if this could be made to work and
1) a tunnel function was exported from http-client for explicit control
2) the http_proxy environment variable was picked up if present and http-conn-send! and friends tunnelled ssl trafic transparently

The following benefits would be realised -
- higher level libraries would benefit by default (url and anything that builds on that out in userland)
-raco pkg might work behind corporate proxies (that is a serious pain)
- it would break down one of the barriers I come up against in using racket in enterprise (we do a ton of integration with Salesforce, finance systems, AWS, Jira, our client's systems which require external connections).

Cheers,
Sean

[Jay McCarthy]

It sounds like you are on the right track. If you need help and can
provide a test case, then I'd be able to be more useful. I think the
function you're going to want to get out is something like:

http-conn-tunnel/ssl : http-conn [arguments to ports->ssl-ports] -> http-conn

where the input port is a plain HTTP port and the one that comes out
is the SSL version. Then you'd use the second one to actually make the
requests.

From your email, I can't tell if you are using the results from
ports->ssl-ports back in a new http-conn object.

Another design decision would be whether the tunnel/ssl function makes
the CONNECT call too. But for now, you should just assume not and just
return a new http-conn object assuming the connect has been done.

Jay

[P. Baillet]

For the newcomers like me, a small update here to say that proxy support has almost landed in racket. https://github.com/racket/racket/pull/1411

Being behind a corporate firewall, I'm eager to test that!

Cheers,
Pierre.

[ZY ZV]

For someone reach here from search engine like me, below is a example.

https://gist.github.com/vy0b0x/4b6a175019954e370bba9594a47830e6

在 2016年8月31日星期三 UTC+8下午7:19:11，P. Baillet写道：

[Neil Van Dyke]

BTW, if anyone wants to implement more proxy support, in addition to
HTTP `CONNECT`, SOCKS 5 would also be useful (for SSH tunneling, Tor,
and possibly other things).