Unknown CA when performing HTTP POST from within controller code

[Brad O'Hearne]

I am attempting to perform an HTTP POST to a remote web service from within Phoenix controller code. To try to sum up the nasties of the problem: 

• Using httpoison 0.8.1 and underneath, hackney 1.4.9.
• Using HTTPS.
• Getting "unknown CA" error when attempting a POST. 
• I have the required root CA installed, but note, I'm running on OS X, so this is installed into my Keychain.
• Hitting the URL over HTTPS from Safari or curl works just fine, no errors. 
• Attempting to show certs using OpenSSL fails on my OS X machine. 
• Attempting to show certs using OpenSSL on a different Windows machine with our root CA works just fine, no errors. 
• Sending the :insecure to hackney on the httpoison POST request (which turns off cert validation) causes the request to succeed just fine. 

From an Elixir API standpoint, I have traced this from Phoenix through httpoison down to hackney, but I believe the problem may be even below that, and this is where I get a little foggy as to where the proper intervention point is. Apps on OS X which do leverage the Keychain seem to work just fine, and things seem to work fine using OpenSSL on Windows. What I think, given the info above, is that OpenSSL and hackney (or perhaps Elixir, or Erlang) is not seeing our root CA which is installed in the OS X Keychain.  

I'm not sure if this is ultimately an issue of Phoenix, or httpoison, or OpenSSL, or Erlang, or Elixir, or hackney. But I think it clearly is an issue of visibility of our installed CA cert on OS X. Does anyone know whether OpenSSL uses the Keychain on OS X or not? Some documentation I've found suggests that it does, but something is clearly not right. 

I suppose I could have asked the question another way -- what keystore does Phoenix use on OS X, but that question would seem to drill way down under the hood, hence all of the above detail. Any help is greatly appreciated. 

Thanks, 

Brad

[Alex Shneyderman]

I doubt keychain stored certs will work. Hackney uses erlang ssl application

https://github.com/benoitc/hackney/blob/master/doc/hackney.md#request-1

you need to provide your certificate with the ssl_options, so that hackney (via ssl lib of erlang) can present this cert to the server when the server tries to verify your client's identity.Those are the same ssl options that erlang's ssl lib's connect method uses:

http://erldocs.com/current/ssl/ssl.html?i=0&search=ssl:conn#connect/2

httpoison also has the options on request.

I do not have a sample ready but I guess something like this should do the trick:

HTTPoison.get "http://url", [],  cacertfile: '/path/to/certs'

you will need to export your CA certificates to that file at the specified path

Cheers,

Alex.

[Alex Shneyderman]

On Thursday, February 11, 2016 at 3:52:27 PM UTC-5, Alex Shneyderman wrote:

I doubt keychain stored certs will work. Hackney uses erlang ssl application

https://github.com/benoitc/hackney/blob/master/doc/hackney.md#request-1

=== start crap ===

 

you need to provide your certificate with the ssl_options, so that hackney (via ssl lib of erlang) can present this cert to the server when the server tries to verify your client's identity.Those are the same ssl options that erlang's ssl lib's connect method uses:

=== end crap  ===

forget this explanation, my wires got crossed. You need CA roots. You can grab a file for example from here:

https://github.com/hexpm/hex/blob/master/lib/hex/api/ca-bundle.crt

it seems like a sizeable list.

 

http://erldocs.com/current/ssl/ssl.html?i=0&search=ssl:conn#connect/2

httpoison also has the options on request.

I do not have a sample ready but I guess something like this should do the trick:

HTTPoison.get "http://url", [],  cacertfile: '/path/to/certs'

this part is correct.

[Brad O'Hearne]

Alex, 

I really appreciate the reply. Hopefully I've just got something minimal out of skew and we can get it worked out quickly. 

I tried what you said...unfortunately, I had no luck -- same exact error. First, I tried indicating the path to the cacert file in an httpoison call: 

HTTPoison.post(@body_login, @http_headers, hackney: [cacertfile: "/Users/brad/Desktop/MyCertificates.pem"])

Then I tried going one level deeper into hackney and trying it directly: 

iex(1)> :hackney.get("https://<host>:<port>", cacertfile: "/Users/brad/Desktop/MyCertificates.pem")

The result was exactly the same in both cases: 

15:28:26.171 [error] SSL: :certify: ssl_handshake.erl:1490:Fatal error: unknown ca

I'm a bit befuddled. The cert file has both the intermediate certificate and and root CA contained within. Any ideas? 

Thanks, 

Brad

[Brad O'Hearne]

Alex, 

One additional question -- do you know if hackney / Erlang is using OpenSSL underneath the hood, or is it entirely separate? It isn't that this should affect troubleshooting per se, but right now I basically have apps on OSX which work against this site (ones that use the OS X keychain, like Safari, etc.) and other apps / libraries (like FireFox and OpenSSL) which do not (presumably those that do not use the OS X keychain). 

Since I don't have the first clue about how to debug an Erlang SSL problem, I'm trying to determine whether getting OpenSSL working will solve the Erlang problem too. Just trying to figure out all resources and avenues to troubleshoot this. 

Thanks, 

Brad

[Alex Shneyderman]

Erlang ssl uses open ssl.

is that ssl site public? Can you provide its exact address? If not can you fetch server's cert and paste it here?

Cheers,

Alex.

[Alex Shneyderman]

Also what erlang version are you using? 

[Brad O'Hearne]

Hey Alex, 

Thanks so much for the reply.

> Also what erlang version are you using? 

Erlang/OTP 18

is that ssl site public? Can you provide its exact address? If not can you fetch server's cert and paste it here?

Unfortunately, no, the site isn't public and I cannot post the cert for reasons beyond my control. However, I've made some progress with respect to OpenSSL, which may provide a clue. I was finally able to get this command to work: 

I found two possible explanations for the problem I was having with OpenSSL. A resource that discusses this as a possible OpenSSL bug: 

https://bugs.launchpad.net/ubuntu/+source/openssl/+bug/965371

….or a buggy server…

https://www.openssl.org/docs/manmaster/apps/s_client.html

The key part of the s_client OpenSSL doc: 

If the handshake fails then there are several possible causes, if it is nothing obvious like no client certificate then the -bugs, -ssl3, -tls1, -no_ssl3, -no_tls1 options can be tried in case it is a buggy server. In particular you should play with these options before submitting a bug report to an OpenSSL mailing list.

Regardless of whether the bug is with OpenSSL, or the server — both resources above agreed with the solution, which is forcing TLS with the –tls1 option. That did it. This command allowed the OpenSSL CLI to succeed in validating the cert chain: 

openssl s_client -CAfile MyCertificates.pem -tls1 -connect <host>:<port>

…where “MyCertificates.pem” is a bundle of exported certificates from the OS X Keychain app. Note that specifying the –CApath to the OpenSSL certs folder with this file, nor letting it default to the certs folder with this file in it (or individual root CA file) worked. The above was the only combination which validated. 

So that appears to have given some explanation for OpenSSL's woes. Now the question becomes what's the analog for httpoison and hackney, or deeper Erlang. I'm not sure if there's a similar means of forcing TLS or not, but I've tried using the certificate file above in an httpoison and hackney call, but both fail with the same "unknown ca" error. 

First with httpoison:

HTTPoison.post(@body_login, @http_headers, hackney: [cacertfile: "/Users/brad/Desktop/MyCertificates.pem"])

then with hackney: 

:hackney.get("https://<host>:<port>", cacertfile: "/Users/brad/Desktop/MyCertificates.pem").

Both fail identically: 

17:41:04.117 [error] SSL: :certify: ssl_handshake.erl:1490:Fatal error: unknown ca

{:error, {:tls_alert, 'unknown ca'}}

I'm not sure if you can see anything wrong with my code....any ideas would help. 

Thanks, 

Brad

[Alex Shneyderman]

:hackney.get("https://<host>:<port>", cacertfile: "/Users/brad/Desktop/MyCertificates.pem").

you are calling from elixir to erlang. I believe cacertfile has to be erlang string so it probably needs to be '/Users/brad/Desktop/MyCertificates.pem' ... although I would expect ssl app to raise an exception or something. Still worth a try, I think.

[Alex Shneyderman]

also if tls1 is what you'd like to force there is an ssloption for it {versions, [protocol()]} where protocol is one of sslv3 | tlsv1 | 'tlsv1.1' | 'tlsv1.2' 

so adding additional 

versions: [:tls1]

might help in your circumstance.

As far as how much of openssl erlang ssl uses - I believe this is only limited to crypto functionality from openssl. I am not certain what other functionality is inherited from openssl but handshaking is all erlang.

[Brad O'Hearne]

Alex, 

Awesome advice....thanks! I think I may have this whittled down to a final problem of a newbie (me) needing syntax help. I am able to get the httpoison call to work now, using this code: 

opts = [{:ssl_options, [{:cacertfile, "/Users/bohearne/Desktop/MyCertificates.pem"}]}]

HTTPoison.post(@body_login, @http_headers, hackney: opts)

That now succeeds. However, I tried to pass the same options to hackney, and it didn't work: 

opts = [{:ssl_options, [{:cacertfile, "/Users/bohearne/Desktop/MyCertificates.pem"}]}]

:hackney.get("https://<host>:<port>", opts)

That does not work. I'm guessing this may have something to do with the format of opts getting changed due to dealing with Erlang vs. Elixir. I tried the above with single quotes, but no luck. Do you happen to know how to massage the opts to fix this? I feel like we are on the precipice of victory here....

Brad

[Alex Shneyderman]

Hi, Brad!

erlang's [{key1, "Test"},{key2, "Test2"}] is equivalent to elixir's [key1: 'Test', key2: 'Test1'] this only applies if keys are atoms. Erlang's proplists are a lot more relaxed in terms of types for the keys.

if options is the last parameter of the call you are making like :hackney.request then they look like named parameters:

method = :get

headers = []

body = ""

url = "http://url"

:hackney.request method, url, headers, body, 

                            option1: "Value for option1", 

                            option2: "Value for option2"

erlang's string - i.e. Path = "/Users/path" - is expressed as path = '/Users/path' in elixir. Elixir's strings "string" are equivalent to erlang's <<"string">>. Sometimes erlang's lib will take string or binary in that case it does not matter what you pass from elixir code. Other times it does. Sometimes Elixir wrapper (HTTPoison for example) will do the conversion for you so you can be elixir happy camper. Other times it will not - you gotta check the docs. If you use erlang's lib from elixir you will need to follow erlang's types.

:hackney.request method, url, headers, body, ssl_options: [cacertfile: '/Users/bohearne/Desktop/MyCertificates.pem']

which is the same as 

:hackney.get url, ssl_options: [cacertfile: '/Users/bohearne/Desktop/MyCertificates.pem']

and should be the same as 

opts = [{:ssl_options, [{:cacertfile, '/Users/bohearne/Desktop/MyCertificates.pem'}]}]

:hackney.get("https://<host>:<port>", opts)

So, I am not sure why the last one does not work for you. Except for single quotes I do not see much of a difference.

Cheers,

Alex.

[Brad O'Hearne]

Alex....thanks for the reply. I really do appreciate it, you've been a great help!

As for the issue, I'm not seeing the difference either. I've spent several hours testing different variations, with no luck. I'm starting to suspect it may have something to do with trying to test hackney from iex, the interactive elixir shell. Things are working from httpoison, so there's something taking place in the moving through that layer that must be invoking hackney differently than I am attempting to. I've had issues posted on both httpoison and hackney GitHub projects for the past day with no response there, perhaps I'll shake that tree again to hopefully get the authors of those layers to weigh in. I'm sure the answer is probably trivial, though non-obvious. 

Again Alex, I really appreciate your help through what was a hair-pulling ordeal on my end, not helped by some unexpected rocky terrain with OpenSSL and OS X (what a kluge that is...if you ever go off into those weeds, if you hear the spirits of the fallen whispering "force TLS", obey, lest you be pulled into a time-sink of root CA validation netherworld).

If anyone else has any Erlang / Elixir / httpoison / hackney wisdom to lend to the problem, it would be extremely welcome. 

[Alex Shneyderman]

you can see what exact call httpoison is making with dbg module. Then compare it to your direct calls with hackney:

iex(1)> :hackney.start

:ok

iex(2)> HTTPoison.get!("http://cnn.com")

%HTTPoison.Response{body: "<html>\r\n<head><title>301 Moved Permanently</title></head>\r\n<body bgcolor=\"white\">\r\n<center><h1>301 Moved Permanently</h1></center>\r\n<hr><center>nginx</center>\r\n</body>\r\n</html>\r\n",

 headers: [{"Server", "nginx"}, {"Date", "Fri, 12 Feb 2016 18:01:57 GMT"},

  {"Content-Type", "text/html"}, {"Transfer-Encoding", "chunked"},

  {"Connection", "keep-alive"}, {"Set-Cookie", "CG=US:FL:Sarasota; path=/"},

  {"Location", "http://www.cnn.com/"}, {"X-UA-Profile", "desktop"}],

 status_code: 301}

iex(3)> :dbg.tracer

{:ok, #PID<0.252.0>}

iex(4)> :dbg.p :all, :c

{:ok, [{:matched, :nonode@nohost, 82}]}

iex(5)> :dbg.tpl :hackney, :x

{:ok, [{:matched, :nonode@nohost, 169}, {:saved, :x}]}

iex(6)> HTTPoison.get!("http://cnn.com")

(<0.233.0>) call hackney:request(get,<<"http://cnn.com">>,[],<<>>,[])

(<0.233.0>) call hackney:request(get,{hackney_url,hackney_tcp_transport,http,<<"cnn.com">>,<<"/">>,<<"/">>,<<>>,

             <<>>,"cnn.com",80,<<>>,<<>>},[],<<>>,[])

(<0.233.0>) call hackney:maybe_proxy(hackney_tcp_transport,"cnn.com",80,[])

(<0.233.0>) returned from hackney:maybe_proxy/4 -> {ok,#Ref<0.0.8.261>}

(<0.233.0>) call hackney:make_request(get,{hackney_url,hackney_tcp_transport,http,<<"cnn.com">>,<<"/">>,<<"/">>,<<>>,

             <<>>,"cnn.com",80,<<>>,<<>>},[],<<>>,[],false)

(<0.233.0>) call hackney:host_header({hackney_url,hackney_tcp_transport,http,<<"cnn.com">>,<<"/">>,<<"/">>,<<>>,

             <<>>,"cnn.com",80,<<>>,<<>>},[])

(<0.233.0>) returned from hackney:host_header/2 -> [{<<"Host">>,

                                                     <<"cnn.com">>}]

(<0.233.0>) returned from hackney:make_request/6 -> {get,<<"/">>,

                                                     [{<<"Host">>,

                                                       <<"cnn.com">>}],

                                                     <<>>}

(<0.233.0>) call hackney:send_request(#Ref<0.0.8.261>,{get,<<"/">>,[{<<"Host">>,<<"cnn.com">>}],<<>>})

(<0.233.0>) call hackney:send_request({client,{1455,300157,25381},

        hackney_dummy_metrics,hackney_tcp_transport,"cnn.com",80,

        <<"cnn.com">>,[],#Port<0.9310>,

        {default,#Ref<0.0.8.261>,

                 {"cnn.com",80,hackney_tcp_transport},

                 <0.248.0>,hackney_tcp_transport},

        #Ref<0.0.8.261>,true,hackney_pool,5000,false,5,false,5,nil,nil,nil,

        connected,start,nil,normal,false,false,false,undefined,false,nil,

        waiting,nil,4096,<<>>,[],undefined,nil,nil,nil,nil,undefined,nil},{get,<<"/">>,[{<<"Host">>,<<"cnn.com">>}],<<>>})

(<0.233.0>) call hackney:maybe_redirect({ok,301,

    [{<<"Server">>,<<"nginx">>},

     {<<"Date">>,<<"Fri, 12 Feb 2016 18:02:37 GMT">>},

     {<<"Content-Type">>,<<"text/html">>},

     {<<"Transfer-Encoding">>,<<"chunked">>},

     {<<"Connection">>,<<"keep-alive">>},

     {<<"Set-Cookie">>,<<"CG=US:FL:Sarasota; path=/">>},

     {<<"Location">>,<<"http://www.cnn.com/">>},

     {<<"X-UA-Profile">>,<<"desktop">>}],

    {client,{1455,300157,25381},

            hackney_dummy_metrics,hackney_tcp_transport,"cnn.com",80,

            <<"cnn.com">>,[],#Port<0.9310>,

            {default,#Ref<0.0.8.261>,

                     {"cnn.com",80,hackney_tcp_transport},

                     <0.248.0>,hackney_tcp_transport},

            #Ref<0.0.8.261>,true,hackney_pool,5000,false,5,false,5,nil,

            <<"http://www.cnn.com/">>,

            {hparser,response,4096,10,0,on_body,

                     <<"b2\r\n<html>\r\n<head><title>301 Moved Permanently</title></head>\r\n<body bgcolor=\"white\">\r\n<center><h1>301 Moved Permanently</h1></center>\r\n<hr><center>nginx</center>\r\n</body>\r\n</html>\r\n\r\n0\r\n\r\n">>,

                     {1,1},

                     undefined,[],undefined,<<"chunked">>,<<"keep-alive">>,

                     <<"text/html">>,<<"http://www.cnn.com/">>,waiting},

            connected,waiting,nil,normal,false,false,false,undefined,false,

            #Fun<hackney_request.send.2>,waiting,nil,4096,<<>>,[],

            {1,1},

            nil,<<"chunked">>,<<"keep-alive">>,<<"GET">>,<<"/">>,

            <<"text/html">>}},{get,<<"/">>,[{<<"Host">>,<<"cnn.com">>}],<<>>})

(<0.233.0>) returned from hackney:maybe_redirect/2 -> {ok,301,

                                                       [{<<"Server">>,

                                                         <<"nginx">>},

                                                        {<<"Date">>,

                                                         <<"Fri, 12 Feb 2016 18:02:37 GMT">>},

                                                        {<<"Content-Type">>,

                                                         <<"text/html">>},

                                                        {<<"Transfer-Encoding">>,

                                                         <<"chunked">>},

                                                        {<<"Connection">>,

                                                         <<"keep-alive">>},

                                                        {<<"Set-Cookie">>,

                                                         <<"CG=US:FL:Sarasota; path=/">>},

                                                        {<<"Location">>,

                                                         <<"http://www.cnn.com/">>},

                                                        {<<"X-UA-Profile">>,

                                                         <<"desktop">>}],

                                                       {client,

                                                        {1455,300157,25381},

                                                        hackney_dummy_metrics,

                                                        hackney_tcp_transport,

                                                        "cnn.com",80,

                                                        <<"cnn.com">>,[],

                                                        #Port<0.9310>,

                                                        {default,

                                                         #Ref<0.0.8.261>,

                                                         {"cnn.com",80,

                                                          hackney_tcp_transport},

                                                         <0.248.0>,

                                                         hackney_tcp_transport},

                                                        #Ref<0.0.8.261>,true,

                                                        hackney_pool,5000,

                                                        false,5,false,5,nil,

                                                        <<"http://www.cnn.com/">>,

                                                        {hparser,response,

                                                         4096,10,0,on_body,

                                                         <<"b2\r\n<html>\r\n<head><title>301 Moved Permanently</title></head>\r\n<body bgcolor=\"white\">\r\n<center><h1>301 Moved Permanently</h1></center>\r\n<hr><center>nginx</center>\r\n</body>\r\n</html>\r\n\r\n0\r\n\r\n">>,

                                                         {1,1},

                                                         undefined,[],

                                                         undefined,

                                                         <<"chunked">>,

                                                         <<"keep-alive">>,

                                                         <<"text/html">>,

                                                         <<"http://www.cnn.com/">>,

                                                         waiting},

                                                        connected,waiting,nil,

                                                        normal,false,false,

                                                        false,undefined,false,

                                                        #Fun<hackney_request.send.2>,

                                                        waiting,nil,4096,<<>>,

                                                        [],

                                                        {1,1},

                                                        nil,<<"chunked">>,

                                                        <<"keep-alive">>,

                                                        <<"GET">>,<<"/">>,

                                                        <<"text/html">>}}

(<0.233.0>) call hackney:reply_response({ok,301,

    [{<<"Server">>,<<"nginx">>},

     {<<"Date">>,<<"Fri, 12 Feb 2016 18:02:37 GMT">>},

     {<<"Content-Type">>,<<"text/html">>},

     {<<"Transfer-Encoding">>,<<"chunked">>},

     {<<"Connection">>,<<"keep-alive">>},

     {<<"Set-Cookie">>,<<"CG=US:FL:Sarasota; path=/">>},

     {<<"Location">>,<<"http://www.cnn.com/">>},

     {<<"X-UA-Profile">>,<<"desktop">>}],

    {client,{1455,300157,25381},

            hackney_dummy_metrics,hackney_tcp_transport,"cnn.com",80,

            <<"cnn.com">>,[],#Port<0.9310>,

            {default,#Ref<0.0.8.261>,

                     {"cnn.com",80,hackney_tcp_transport},

                     <0.248.0>,hackney_tcp_transport},

            #Ref<0.0.8.261>,true,hackney_pool,5000,false,5,false,5,nil,

            <<"http://www.cnn.com/">>,

            {hparser,response,4096,10,0,on_body,

                     <<"b2\r\n<html>\r\n<head><title>301 Moved Permanently</title></head>\r\n<body bgcolor=\"white\">\r\n<center><h1>301 Moved Permanently</h1></center>\r\n<hr><center>nginx</center>\r\n</body>\r\n</html>\r\n\r\n0\r\n\r\n">>,

                     {1,1},

                     undefined,[],undefined,<<"chunked">>,<<"keep-alive">>,

                     <<"text/html">>,<<"http://www.cnn.com/">>,waiting},

            connected,waiting,nil,normal,false,false,false,undefined,false,

            #Fun<hackney_request.send.2>,waiting,nil,4096,<<>>,[],

            {1,1},

            nil,<<"chunked">>,<<"keep-alive">>,<<"GET">>,<<"/">>,

            <<"text/html">>}},{client,{1455,300157,25381},

        hackney_dummy_metrics,hackney_tcp_transport,"cnn.com",80,

        <<"cnn.com">>,[],#Port<0.9310>,

        {default,#Ref<0.0.8.261>,

                 {"cnn.com",80,hackney_tcp_transport},

                 <0.248.0>,hackney_tcp_transport},

        #Ref<0.0.8.261>,true,hackney_pool,5000,false,5,false,5,nil,nil,nil,

        connected,start,nil,normal,false,false,false,undefined,false,nil,

        waiting,nil,4096,<<>>,[],undefined,nil,nil,nil,nil,undefined,nil})

(<0.233.0>) returned from hackney:reply_response/2 -> {ok,301,

                                                       [{<<"Server">>,

                                                         <<"nginx">>},

                                                        {<<"Date">>,

                                                         <<"Fri, 12 Feb 2016 18:02:37 GMT">>},

                                                        {<<"Content-Type">>,

                                                         <<"text/html">>},

                                                        {<<"Transfer-Encoding">>,

                                                         <<"chunked">>},

                                                        {<<"Connection">>,

                                                         <<"keep-alive">>},

                                                        {<<"Set-Cookie">>,

                                                         <<"CG=US:FL:Sarasota; path=/">>},

                                                        {<<"Location">>,

                                                         <<"http://www.cnn.com/">>},

                                                        {<<"X-UA-Profile">>,

                                                         <<"desktop">>}],

                                                       #Ref<0.0.8.261>}

(<0.233.0>) returned from hackney:send_request/2 -> {ok,301,

                                                     [{<<"Server">>,

                                                       <<"nginx">>},

                                                      {<<"Date">>,

                                                       <<"Fri, 12 Feb 2016 18:02:37 GMT">>},

                                                      {<<"Content-Type">>,

                                                       <<"text/html">>},

                                                      {<<"Transfer-Encoding">>,

                                                       <<"chunked">>},

                                                      {<<"Connection">>,

                                                       <<"keep-alive">>},

                                                      {<<"Set-Cookie">>,

                                                       <<"CG=US:FL:Sarasota; path=/">>},

                                                      {<<"Location">>,

                                                       <<"http://www.cnn.com/">>},

                                                      {<<"X-UA-Profile">>,

                                                       <<"desktop">>}],

                                                     #Ref<0.0.8.261>}

(<0.233.0>) returned from hackney:send_request/2 -> {ok,301,

                                                     [{<<"Server">>,

                                                       <<"nginx">>},

                                                      {<<"Date">>,

                                                       <<"Fri, 12 Feb 2016 18:02:37 GMT">>},

                                                      {<<"Content-Type">>,

                                                       <<"text/html">>},

                                                      {<<"Transfer-Encoding">>,

                                                       <<"chunked">>},

                                                      {<<"Connection">>,

                                                       <<"keep-alive">>},

                                                      {<<"Set-Cookie">>,

                                                       <<"CG=US:FL:Sarasota; path=/">>},

                                                      {<<"Location">>,

                                                       <<"http://www.cnn.com/">>},

                                                      {<<"X-UA-Profile">>,

                                                       <<"desktop">>}],

                                                     #Ref<0.0.8.261>}

(<0.233.0>) returned from hackney:request/5 -> {ok,301,

                                                [{<<"Server">>,<<"nginx">>},

                                                 {<<"Date">>,

                                                  <<"Fri, 12 Feb 2016 18:02:37 GMT">>},

                                                 {<<"Content-Type">>,

                                                  <<"text/html">>},

                                                 {<<"Transfer-Encoding">>,

                                                  <<"chunked">>},

                                                 {<<"Connection">>,

                                                  <<"keep-alive">>},

                                                 {<<"Set-Cookie">>,

                                                  <<"CG=US:FL:Sarasota; path=/">>},

                                                 {<<"Location">>,

                                                  <<"http://www.cnn.com/">>},

                                                 {<<"X-UA-Profile">>,

                                                  <<"desktop">>}],

                                                #Ref<0.0.8.261>}

(<0.233.0>) returned from hackney:request/5 -> {ok,301,

                                                [{<<"Server">>,<<"nginx">>},

                                                 {<<"Date">>,

                                                  <<"Fri, 12 Feb 2016 18:02:37 GMT">>},

                                                 {<<"Content-Type">>,

                                                  <<"text/html">>},

                                                 {<<"Transfer-Encoding">>,

                                                  <<"chunked">>},

                                                 {<<"Connection">>,

                                                  <<"keep-alive">>},

                                                 {<<"Set-Cookie">>,

                                                  <<"CG=US:FL:Sarasota; path=/">>},

                                                 {<<"Location">>,

                                                  <<"http://www.cnn.com/">>},

                                                 {<<"X-UA-Profile">>,

                                                  <<"desktop">>}],

                                                #Ref<0.0.8.261>}

(<0.233.0>) call hackney:body(#Ref<0.0.8.261>)

(<0.233.0>) call hackney:'-body/1-fun-0-'({client,{1455,300157,25381},

        hackney_dummy_metrics,hackney_tcp_transport,"cnn.com",80,

        <<"cnn.com">>,[],#Port<0.9310>,

        {default,#Ref<0.0.8.261>,

                 {"cnn.com",80,hackney_tcp_transport},

                 <0.248.0>,hackney_tcp_transport},

        #Ref<0.0.8.261>,true,hackney_pool,5000,false,5,false,5,nil,

        <<"http://www.cnn.com/">>,

        {hparser,response,4096,10,0,on_body,

                 <<"b2\r\n<html>\r\n<head><title>301 Moved Permanently</title></head>\r\n<body bgcolor=\"white\">\r\n<center><h1>301 Moved Permanently</h1></center>\r\n<hr><center>nginx</center>\r\n</body>\r\n</html>\r\n\r\n0\r\n\r\n">>,

                 {1,1},

                 undefined,[],undefined,<<"chunked">>,<<"keep-alive">>,

                 <<"text/html">>,<<"http://www.cnn.com/">>,waiting},

        connected,waiting,nil,normal,false,false,false,undefined,false,

        #Fun<hackney_request.send.2>,waiting,nil,4096,<<>>,[],

        {1,1},

        nil,<<"chunked">>,<<"keep-alive">>,<<"GET">>,<<"/">>,<<"text/html">>})

(<0.233.0>) call hackney:reply({ok,<<"<html>\r\n<head><title>301 Moved Permanently</title></head>\r\n<body bgcolor=\"white\">\r\n<center><h1>301 Moved Permanently</h1></center>\r\n<hr><center>nginx</center>\r\n</body>\r\n</html>\r\n">>,

    {client,{1455,300157,25381},

            hackney_dummy_metrics,hackney_tcp_transport,"cnn.com",80,

            <<"cnn.com">>,[],nil,nil,#Ref<0.0.8.261>,true,hackney_pool,5000,

            false,5,false,5,nil,<<"http://www.cnn.com/">>,nil,closed,done,nil,

            normal,false,false,false,undefined,false,

            #Fun<hackney_request.send.2>,done,nil,4096,<<>>,[],

            {1,1},

            nil,<<"chunked">>,<<"keep-alive">>,<<"GET">>,<<"/">>,

            <<"text/html">>}},{client,{1455,300157,25381},

        hackney_dummy_metrics,hackney_tcp_transport,"cnn.com",80,

        <<"cnn.com">>,[],#Port<0.9310>,

        {default,#Ref<0.0.8.261>,

                 {"cnn.com",80,hackney_tcp_transport},

                 <0.248.0>,hackney_tcp_transport},

        #Ref<0.0.8.261>,true,hackney_pool,5000,false,5,false,5,nil,

        <<"http://www.cnn.com/">>,

        {hparser,response,4096,10,0,on_body,

                 <<"b2\r\n<html>\r\n<head><title>301 Moved Permanently</title></head>\r\n<body bgcolor=\"white\">\r\n<center><h1>301 Moved Permanently</h1></center>\r\n<hr><center>nginx</center>\r\n</body>\r\n</html>\r\n\r\n0\r\n\r\n">>,

                 {1,1},

                 undefined,[],undefined,<<"chunked">>,<<"keep-alive">>,

                 <<"text/html">>,<<"http://www.cnn.com/">>,waiting},

        connected,waiting,nil,normal,false,false,false,undefined,false,

        #Fun<hackney_request.send.2>,waiting,nil,4096,<<>>,[],

        {1,1},

        nil,<<"chunked">>,<<"keep-alive">>,<<"GET">>,<<"/">>,<<"text/html">>})

(<0.233.0>) call hackney:maybe_update_req({client,{1455,300157,25381},

        hackney_dummy_metrics,hackney_tcp_transport,"cnn.com",80,

        <<"cnn.com">>,[],nil,nil,#Ref<0.0.8.261>,true,hackney_pool,5000,false,

        5,false,5,nil,<<"http://www.cnn.com/">>,nil,closed,done,nil,normal,

        false,false,false,undefined,false,#Fun<hackney_request.send.2>,done,

        nil,4096,<<>>,[],

        {1,1},

        nil,<<"chunked">>,<<"keep-alive">>,<<"GET">>,<<"/">>,<<"text/html">>})

(<0.233.0>) returned from hackney:maybe_update_req/1 -> ok

(<0.233.0>) returned from hackney:reply/2 -> {ok,

                                              <<"<html>\r\n<head><title>301 Moved Permanently</title></head>\r\n<body bgcolor=\"white\">\r\n<center><h1>301 Moved Permanently</h1></center>\r\n<hr><center>nginx</center>\r\n</body>\r\n</html>\r\n">>}

(<0.233.0>) returned from hackney:'-body/1-fun-0-'/1 -> {ok,

                                                         <<"<html>\r\n<head><title>301 Moved Permanently</title></head>\r\n<body bgcolor=\"white\">\r\n<center><h1>301 Moved Permanently</h1></center>\r\n<hr><center>nginx</center>\r\n</body>\r\n</html>\r\n">>}

(<0.233.0>) returned from hackney:body/1 -> {ok,

                                             <<"<html>\r\n<head><title>301 Moved Permanently</title></head>\r\n<body bgcolor=\"white\">\r\n<center><h1>301 Moved Permanently</h1></center>\r\n<hr><center>nginx</center>\r\n</body>\r\n</html>\r\n">>}

%HTTPoison.Response{body: "<html>\r\n<head><title>301 Moved Permanently</title></head>\r\n<body bgcolor=\"white\">\r\n<center><h1>301 Moved Permanently</h1></center>\r\n<hr><center>nginx</center>\r\n</body>\r\n</html>\r\n",

 headers: [{"Server", "nginx"}, {"Date", "Fri, 12 Feb 2016 18:02:37 GMT"},

  {"Content-Type", "text/html"}, {"Transfer-Encoding", "chunked"},

  {"Connection", "keep-alive"}, {"Set-Cookie", "CG=US:FL:Sarasota; path=/"},

  {"Location", "http://www.cnn.com/"}, {"X-UA-Profile", "desktop"}],

 status_code: 301}

[Brad O'Hearne]

Alex,

WOW. First off, I didn't know how to use the dbg module, so that was informative in and of itself. Second, I made what in hindsight was a very stupid assumption that options on GET requests were being handled the same as on POST requests, which they clearly aren't, now that I can see the debug output. This is something that no one could have caught in my above posts, because I've had to do some scrubbing of the code posted for various reasons (again, outside my control). But in the context I'm actually testing, the httpoison call was a POST, while the call to hackney was a GET. Here's the rub: hackney just completely discards GET options, but it passes on the POST options. As soon as I saw that, I backed up and issued a GET request from httpoison with the same options, and it fails too. Wild. 

Unless there is something I have missed in the API usage of either httpoison or hackney, there is a problem here. I've reported it as an issue under both httpoison and hackney GitHub projects. Hopefully it is just my usage issue, but it looks like something may actually be amiss here. 

Thanks for your help, Alex. I don't think I would have come this far without your help. 

Brad

[Alex Shneyderman]

this is what hackney has:

-define(METHOD_TPL(Method),

        Method(URL) ->

            hackney:request(Method, URL)).

-include("hackney_methods.hrl").

-define(METHOD_TPL(Method),

        Method(URL, Headers) ->

            hackney:request(Method, URL, Headers)).

-include("hackney_methods.hrl").

-define(METHOD_TPL(Method),

        Method(URL, Headers, Body) ->

            hackney:request(Method, URL, Headers, Body)).

-include("hackney_methods.hrl").

-define(METHOD_TPL(Method),

        Method(URL, Headers, Body, Options) ->

            hackney:request(Method, URL, Headers, Body, Options)).

-include("hackney_methods.hrl").

so for each method (get, post, ... etc) you can have 1..4 arguments. Options is always #4 ... so you gotta do this

method = :get

headers = []

body = ""

url = "http://url"

opts = [{:ssl_options, [{:cacertfile, "/Users/bohearne/Desktop/MyCertificates.pem"}]}]

:hackney.get url, headers, body, opts

or just use request/5 

[Brad O'Hearne]

Yep....at the end of the day, all of this ended up being a parameter issue. Strange that it was able to pass through two layers of API without manifesting itself as a problem and even with debug output it wasn't immediately clear what the problem was. Interesting....lesson learned -- and  bunch of other stuff too, thanks to your help, Alex. I really appreciate your help. Time well spent.

[Alex Shneyderman]

At the end of the day it is the lack of documentation on hackney side ... the get/post/... variety is not documented at all. So, if anything you should raise an issue with hackney and probably submit a patch with documentation.

[Bradley O'Hearne]

> At the end of the day it is the lack of documentation on hackney side ... the get/post/... variety is not documented at all. So, if anything you should raise an issue with hackney and probably submit a patch with documentation.

That was basically the conclusion in the responses on the hackney Github project too — I think it is going to lead to some improved logging / error reporting, so its been worth the effort.

Take care,

Brad