HTTPS problem with net/url

[David Storrs]

Hi folks,

tl;dr:  How do I make HTTPS calls from within Racket?

Background:

I co-write a play-by-post RPG (https://forums.sufficientvelocity.com/threads/marked-for-death-a-rational-naruto-quest.24481/ -- stop by if you're curious; the barrier to entry is low).   The players all vote to control a single character, so being able to easily tally the votes is a big thing.  As part of my "learning Racket" efforts, I'm writing a web spider that will crawl the forum starting from a given location and tally up votes.

In this I have the following method:

(define (web/call url-string #:method [:method get-pure-port] )
  (string->xexp
   (call/input-url (string->url url-string)
                   (curry :method #:redirections 5)
                   port->string)))

(NB:  That originally hardcoded get-pure-port; I put the keyword in just as an exercise, but it wouldn't actually work if you gave it an impure port.  Will fix when tuits are available.)

When I do this:

(define u "https://forums.sufficientvelocity.com/threads/marked-for-death-a-rational-naruto-qu\
est.24481/page-6")

(web/call u)

I get this:

[dstorrs@MacBook-Pro:~/personal/study/scheme/sv_vote_tally:<master>]$ racket tallyho.rkt
racket tallyho.rkt
ssl-connect: connect failed (error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure)
  context...:
   /Applications/Racket_v6.3/collects/openssl/mzssl.rkt:1401:8: loop
   /Applications/Racket_v6.3/collects/openssl/..:261:28
   /Applications/Racket_v6.3/collects/openssl/..:259:25
   /Applications/Racket_v6.3/collects/net/http-client.rkt:224:0
   /Applications/Racket_v6.3/collects/racket/contract/private/arrow-val-first.rkt:324:3
   /Applications/Racket_v6.3/collects/net/url.rkt:77:0: http://getpost-impure-port
   /Applications/Racket_v6.3/collects/net/url.rkt:179:2: redirection-loop
   /Applications/Racket_v6.3/collects/net/url.rkt:143:0: getpost-pure-port
   /Applications/Racket_v6.3/collects/net/url.rkt:245:4: call/input-url
   /Users/dstorrs/personal/study/scheme/spider/spider.rkt:204:0: web/call19
   /Users/dstorrs/personal/study/scheme/sv_vote_tally/tallyho.rkt: [running body]

I've just spent a whole lot of time Googling around.  There are a lot of tutorials about how to write a web *server* in Racket, and some of those touch on SSL and/or HTTPS.  There's not so much for web *clients* though, and the actual web-client module doesn't seem to handle HTTPS.

When I read the docs for net/url I saw this bit:

Beware: By default, "https" scheme handling does not verify a server’s certificate (i.e., it’s equivalent of clicking through a browser’s warnings), so communication is safe, but the identity of the server is not verified. To validate the server’s certificate, set current-https-protocol to a context created with ssl-make-client-context, and enable certificate validation in the context with ssl-set-verify!.

When I look at 'current-https-protocol' I see this:

Changed in version 6.1 of package base: Added 'tls11 and 'tls12. Changed in version 6.1.1.3: Default to new 'auto and disabled SSL 2.0 and 3.0 by default.

So it should be attempting to negotiate the protocol on its own.

Help me, wisdom of crowds.  What is it that I don't know?

Dave

[Matthew Flatt]

Racket is using the too-old version of "libssl.dylib" that is provided
by the OS. The too-old version doesn't work with some servers.

For that server, I get the same error in v6.3. It works for me with the
development version of Racket --- but only because I've been working on
this problem (and related issues) for the past day. The next Racket
snapshot will include its own copy of "libssl.1.0.0dylib" to solve the
problem.

To fix a v6.3 installation, you can download

https://racket-packages.s3-us-west-2.amazonaws.com/pkgs/cfaf0f27a375dbdac2e6f68d3863328b64b84eb2/racket-x86_64-macosx-2.zip

and copy the two ".dylib" files from the "racket" folder into

/Applications/Racket_v6.3/lib/

Thanks for delaying this question until the first day that I know the
answer!

> <http://docs.racket-lang.org/net/url.html#%28def._%28%28lib._net%2Furl-connect.
> .rkt%29._current-https-protocol%29%29>

> to a context created with ssl-make-client-context

> <http://docs.racket-lang.org/openssl/index.html#%28def._%28%28lib._openssl%2Fma
> in..rkt%29._ssl-make-client-context%29%29>,

> and enable certificate validation in the context with ssl-set-verify!

> <http://docs.racket-lang.org/openssl/index.html#%28def._%28%28lib._openssl%2Fma
> in..rkt%29._ssl-set-verify%21%29%29>
> .

>
> When I look at 'current-https-protocol' I see this:
>
> Changed in version 6.1 of package base: Added 'tls11 and 'tls12. Changed in
> version 6.1.1.3: Default to new 'auto and disabled SSL 2.0 and 3.0 by
> default.
>
> So it should be attempting to negotiate the protocol on its own.
>
> Help me, wisdom of crowds. What is it that I don't know?
>
> Dave
>

> --
> You received this message because you are subscribed to the Google Groups
> "Racket Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to racket-users...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[David Storrs]

Wow.  You're fantastic Matthew, thank you.

> Thanks for delaying this question until the first day that I know the answer!

No problem.  I'm nice like that. ;>

On Wed, Jan 6, 2016 at 2:15 PM, Matthew Flatt <mfl...@cs.utah.edu> wrote:

Racket is using the too-old version of "libssl.dylib" that is provided
by the OS. The too-old version doesn't work with some servers.

For that server, I get the same error in v6.3. It works for me with the
development version of Racket --- but only because I've been working on
this problem (and related issues) for the past day. The next Racket
snapshot will include its own copy of "libssl.1.0.0dylib" to solve the
problem.

To fix a v6.3 installation, you can download

https://racket-packages.s3-us-west-2.amazonaws.com/pkgs/cfaf0f27a375dbdac2e6f68d3863328b64b84eb2/racket-x86_64-macosx-2.zip

and copy the two ".dylib" files from the "racket" folder into

 /Applications/Racket_v6.3/lib/

Thanks for delaying this question until the first day that I know the
answer!

At Wed, 6 Jan 2016 13:50:51 -0800, David Storrs wrote:

> <http://docs.racket-lang.org/net/url.html#%28def._%28%28lib._net%2Furl-connect.

> .rkt%29._current-https-protocol%29%29>
> to a context created with ssl-make-client-context
> <http://docs.racket-lang.org/openssl/index.html#%28def._%28%28lib._openssl%2Fma
> in..rkt%29._ssl-make-client-context%29%29>,
> and enable certificate validation in the context with ssl-set-verify!
> <http://docs.racket-lang.org/openssl/index.html#%28def._%28%28lib._openssl%2Fma
> in..rkt%29._ssl-set-verify%21%29%29>

> .
>

> When I look at 'current-https-protocol' I see this:
>
> Changed in version 6.1 of package base: Added 'tls11 and 'tls12. Changed in
> version 6.1.1.3: Default to new 'auto and disabled SSL 2.0 and 3.0 by
> default.
>
> So it should be attempting to negotiate the protocol on its own.
>
> Help me, wisdom of crowds.  What is it that I don't know?
>
> Dave
>