RabbitMQ always fails to verify client certificate when fail_if_no_peer_cert is set to true

[Renat Nurgaliyev]

Hi everyone!

I'm trying to establish TLS enabled federation link using self-signed CA using RabbitMQ 3.6.14 and Erlang 19.3. TLS stuff is configured by following manual (https://www.rabbitmq.com/ssl.html). Everything works if client certificate verification is disabled by setting fail_if_no_peer_cert to false. But if I set it to true, rabbit will always fail to verify client certificate, throwing following error messages:

upstream_1    | =INFO REPORT==== 9-Nov-2017::07:58:58 ===
upstream_1    | SSL WARNING: Ignoring a CA cert as it could not be correctly decoded.


upstream_1    | =ERROR REPORT==== 9-Nov-2017::07:59:14 ===
upstream_1    | SSL: certify: ssl_connection.erl:484:Fatal error: handshake failure


downstream_1  | =ERROR REPORT==== 9-Nov-2017::07:59:14 ===
downstream_1  | SSL: cipher: ssl_alert.erl:88:Fatal error: handshake failure


downstream_1  | =WARNING REPORT==== 9-Nov-2017::07:59:14 ===
downstream_1  | Federation exchange 'federation.exchange' in vhost '/' did not connect to exchange 'federation.exchange' in vhost '/' on amqps://upstream
downstream_1  | {error,{tls_alert,"handshake failure"}}

Since TLS is handled only by Erlang, I investigated OTP code and I have the feeling that OTP code fails to parse ASN.1 data. I've tried different formats of certificate file, different public key lengths and certificate options, but no luck.

To easily reproduce this issue, I've prepared test code around official rabbitmq:3.6-management-alpine container.

Steps to reproduce:

1. Clone https://github.com/rnurgaliyev/rabbitmq-fed-tls

2. Generate CA by running `create-ca.sh`

3. Generate client keys and certificates by running `create-keys-upstream.sh` and `create-keys-downstream.sh`

4. Bring up test environment by running `docker-compose up`

5. Configure federation link by running `setup-federation.sh`

6. Subscribe to a message queue by running `go run receiver.go`

7. Confirm bug observing messages in container log output.

I've also tried to reproduce problem with Erlang 18.3 and RabbitMQ 3.5.7 from latest ubuntu xenial repo. I get exactly same behavior, but log is a bit more verbose:

=INFO REPORT==== 8-Nov-2017::19:33:17 ===
accepting AMQP connection <0.346.0> (192.168.33.10:37528 -> 192.168.33.20:5671)
                           
=ERROR REPORT==== 8-Nov-2017::19:33:17 ===
** State machine <0.347.0> terminating
** Last message in was {tcp,#Port<0.9579>,
                            <<21,3,3,0,26,10,210,174,236,125,223,121,161,133,
                              1,181,98,137,179,81,246,136,40,27,182,45,40,101,
                              202,218,188>>}
** When State == connection
**      Data  == [{data,
                      [{"StateData",
                        {state,server,
                            {#Ref<0.0.1.1389>,<0.346.0>},
                            gen_tcp,tls_connection,tcp,tcp_closed,tcp_error,
                            "localhost",5671,#Port<0.9579>,
                            {ssl_options,tls,
                                [{3,3},{3,2},{3,1}],
                                verify_peer,undefined,#Fun<ssl.8.92387908>,
                                false,false,undefined,1,<<"/tls/cert.pem">>,
                                "***",<<"/tls/key.pem">>,"***","***","***",
                                <<"/tls/cacert.pem">>,"***",undefined,
                                undefined,"***","***",
                                [<<"�,">>,<<"�0">>,<<"�$">>,<<"�(">>,<<"�.">>,
                                 <<"�2">>,<<"�&">>,<<"�*">>,
                                 <<0,159>>,
                                 <<0,163>>,
                                 <<0,107>>,
                                 <<0,106>>,
                                 <<0,157>>,
                                 <<0,61>>,
                                 <<"�+">>,<<"�/">>,<<"�#">>,<<"�'">>,<<"�-">>,
                                 <<"�1">>,<<"�%">>,<<"�)">>,
                                 <<0,158>>,
                                 <<0,162>>,
                                 <<0,103>>,
                                 <<0,64>>,
                                 <<0,156>>,
                                 <<0,60>>,
                                 <<"�\n">>,
                                 <<192,20>>,
                                 <<0,57>>,
                                 <<0,56>>,
                                 <<192,5>>,
                                 <<192,15>>,
                                 <<0,53>>,
                                 <<"�\b">>,
                                 <<192,18>>,
                                 <<0,22>>,
                                 <<0,19>>,
                                 <<192,3>>,
                                 <<"�\r">>,
                                 <<0,10>>,
                                 <<"�\t">>,
                                 <<192,19>>,
                                 <<0,51>>,
                                 <<0,50>>,
                                 <<192,4>>,
                                 <<192,14>>,
                                 <<0,47>>,
                                 <<0,21>>,
                                 <<0,9>>],
                                #Fun<ssl.1.92387908>,true,268435456,false,
                                true,undefined,false,undefined,undefined,
                                undefined,undefined,true,undefined,[],
                                undefined,false,true,undefined,false,
                                {ssl_crl_cache,{internal,[]}}},
                            {socket_options,binary,0,0,0,once},
                            "***","***","***",282706,"***",307288,
                            ssl_session_cache,
                            {ssl_crl_cache,{{294997,299094},[]}},
                            {3,3},
                            false,ecdhe_rsa,
                            {sha512,rsa},
                            undefined,undefined,"***","***","***",undefined,
                            "***","***","***",286803,#Ref<0.0.2.323>,
                            undefined,"***",undefined,undefined,undefined,
                            {[],[]},
                            false,true,false,false,undefined,
                            {{elliptic_curves,
                                 [{1,3,132,0,39},
                                  {1,3,132,0,38},
                                  {1,3,132,0,35},
                                  {1,3,36,3,3,2,8,1,1,13},
                                  {1,3,132,0,36},
                                  {1,3,132,0,37},
                                  {1,3,36,3,3,2,8,1,1,11},
                                  {1,3,132,0,34},
                                  {1,3,132,0,16},
                                  {1,3,132,0,17},
                                  {1,3,36,3,3,2,8,1,1,7},
                                  {1,3,132,0,10},
                                  {1,2,840,10045,3,1,7},
                                  {1,3,132,0,3},
                                  {1,3,132,0,26},
                                  {1,3,132,0,27},
                                  {1,3,132,0,32},
                                  {1,3,132,0,33},
                                  {1,3,132,0,24},
                                  {1,3,132,0,25},
                                  {1,3,132,0,31},
                                  {1,2,840,10045,3,1,1},
                                  {1,3,132,0,1},
                                  {1,3,132,0,2},
                                  {1,3,132,0,15},
                                  {1,3,132,0,9},
                                  {1,3,132,0,8},
                                  {1,3,132,0,30}]},
                             {ec_point_formats,[0]}},
                            undefined,undefined}}]}]
** Reason for termination =
** {badarg,[{ets,update_counter,[286803,#Ref<0.0.2.323>,-1],[]},
            {ssl_pkix_db,ref_count,3,[{file,"ssl_pkix_db.erl"},{line,207}]},
            {ssl_connection,handle_trusted_certs_db,1,
                            [{file,"ssl_connection.erl"},{line,1801}]},
            {ssl_connection,terminate,3,
                            [{file,"ssl_connection.erl"},{line,938}]},
            {tls_connection,terminate,3,
                            [{file,"tls_connection.erl"},{line,336}]},
            {gen_fsm,terminate,7,[{file,"gen_fsm.erl"},{line,610}]},
            {proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]}


=INFO REPORT==== 8-Nov-2017::19:33:17 ===
closing AMQP connection <0.346.0> (192.168.33.10:37528 -> 192.168.33.20:5671)

Any sort of help or directions where to dig further will be really appreciated!

[Michael Klishin]

> upstream_1    | SSL WARNING: Ignoring a CA cert as it could not be correctly decoded

is definitely something that I'd investigate.

The entire trace is from OTP's ssl/public_key applications. Curiously it's an ETS operation that fails.

A `badarg` from ETS means one of two things:

 * A table does not exist

 * The node is out of file descriptors

Feel free to try OTP 19.3.6.3 or 20.x. RabbitMQ does not implement TLS, Erlang/OTP does.

There is a reasonably extensive TLS troubleshooting guide that documents a process that

quickly helps narrow the issue down:

http://www.rabbitmq.com/troubleshooting-ssl.html

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

[Michael Klishin]

One possible sequence of events that I can think of is:

 * The client (a federation downstream?) fails to decode its certificate

 * It then does not send it (I'm not sure if this is correct per TLS RFCs)

 * Since the server is configured to require client certificate, it fails TLS upgrade

however, the ETS except happens in the server log, so it's definitely not a clean TLS connection

shutdown using a TLS alert.

To post to this group, send email to rabbitm...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

[Renat Nurgaliyev]

According to log file and source here https://github.com/erlang/otp/blob/f6e336f7818a04e411a5e20e7d48df89f35e679c/lib/ssl/src/ssl_pkix_db.erl#L312, it tries to load CA certificates from file specified in ssl_opts, option "cacerts".

When I remove this configuration line, it stop complaining that it cannot decode CA certificate, but it still cannot verify client certificate, because as far as I understand, this option sets valid CAs for client verification.

I will try different OTP versions, and I'll also try to catch what kind of error is rasied in this try/catch block.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.

To post to this group, send email to rabbitm...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

[Michael Klishin]

It's a lot more important to understand why your certificate cannot be read (for example, is it in the PEM format?)

first, then try a different OTP version.

http://www.rabbitmq.com/troubleshooting-ssl.html explains how server and client certificates can be tested in relative isolation.

For example, do your certificate/key pairs work with openssl s_client against openssl s_server? If not then surely you have

to take care of that first.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Renat Nurgaliyev]

Cert is in PEM format, I've also tried with DER, troubleshooting procedure described in this link is working perfectly, all steps, the stunnel also works perfectly with that certificate. Erlang can read this certificate successfully when it is defined in { cacertfile, "/tls/cacert.pem" } but not when in { cacerts, "/tls/cacert.pem" }

[Michael Klishin]

the cacerts option requires a list of paths (otherwise you can simply concatenate all certificates

into one file and use `cacertfile`).

Therefore { cacerts, "/tls/cacert.pem" } is not a valid value.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Michael Klishin]

Actually, according to http://erlang.org/doc/man/ssl.html `cacerts` expects a list of decoded certificates,

not file paths.

Therefore it's a much better idea to concatenate all certificates into a single file and use the `cacertfile` option,

like our docs and 99% of Erlang TLS examples on the Web do.

[Renat Nurgaliyev]

Yep, according to that documentation one should not use this option when 'cacertfile' is defined, because it will override it. I've removed it, but now i silently get this:

=INFO REPORT==== 9-Nov-2017::15:03:04 ===
TLS server: In state certify at ssl_connection.erl:491 generated SERVER ALERT: Fatal - Handshake Failure

It is now latest RabbitMQ and OTP: RabbitMQ 3.6.14 on Erlang 21.0-rc0

And just as before, if I set fail_if_no_peer_cert to false, everything works.

[Michael Klishin]

"Handshake Failure" is the most generic TLS alert out there. Without having logs from both sides

and full TLS configuration it's anyone's guess what may be going on.

While I doubt it is significant but we do not support preview releases such as 21.0-rc0.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Renat Nurgaliyev]

I agree, but how can I make erlang more verbose? Because rabbit isn't giving me more details. I've checked https://github.com/erlang/otp/blob/30c3930fb296f46bac334cb6a5856a747984f69d/lib/ssl/src/ssl_connection.erl#L491 , doesn't look like I could improve verbosity anyhow.

Configuration of both rabbit servers is very straightforward:

[
        { rabbit, [
                { loopback_users, [ ] },
                { tcp_listeners, [ 5672 ] },
                { ssl_listeners, [ 5671 ] },
                { ssl_options, [
                        { cacertfile, "/tls/cacert.pem" },
                        { certfile, "/tls/cert.pem" },
                        { fail_if_no_peer_cert, true },
                        { keyfile, "/tls/key.pem" },
                        { verify, verify_peer },
                ] },

                { default_pass, <<"mypass">> },
                { default_user, <<"myuser">> },
                { hipe_compile, false }
        ] },
        { rabbitmq_management, [ { listener, [
                { port, 15671 },
                { ssl, true },
                { ssl_opts, [
                        { cacertfile, "/tls/cacert.pem" },
                        { certfile, "/tls/cert.pem" },
                        { fail_if_no_peer_cert, true },
                        { keyfile, "/tls/key.pem" },
                        { verify, verify_none }
                ] }
        ] } ] }
].

TLS stuff is pretty default self signed CA and it is generated using very basic openssl commands.

[Michael Klishin]

I'm asking someone who has experience with debugging Erlang's TLS implementation to chime in.

On an unrelated note, are your HTTP API clients guaranteed to provide a certificate? It's a common

requirement for client connections but not e.g. Web browsers.

[Renat Nurgaliyev]

Yep, you are definitely correct. I just copied tls configuration there and decided not to worry about it a lot before federation is working. fail_if_no_peer_cert needs to be changed to false there. 

You received this message because you are subscribed to a topic in the Google Groups "rabbitmq-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/rabbitmq-users/Xdgv9Y4pl6Q/unsubscribe.
To unsubscribe from this group and all its topics, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.

[Luke Bakken]

Hi Renat,

I'll take some time today to generate certs using the same project you did to try and reproduce this issue.

In your environment, have you tried connecting an AMQP client library using a client certificate, rather than only testing using federation?

I like Michael's suggestion to connect to the Management UI using a client certificate. Have you tried that? curl supports sending a client certificate (https://curl.haxx.se/docs/manpage.html). Combined with -vvv for verbose output, you may get more information that way.

Even though you're not using intermediate CAs, could you give the {depth, 1} and {depth, 2} settings a try to see if they resolve your issue:

https://www.rabbitmq.com/ssl.html#certificate-chain

Thanks -

Luke

[Luke Bakken]

Hi Renat,

I reproduced the handshake error you are seeing using Cowboy's ssl_hello_world example. I then updated the example to use a couple additional settings, and can connect successfully using the openssl s_client command.

Here is the example code with my changes:

https://github.com/lukebakken/cowboy/blob/lrb-test/examples/ssl_hello_world/src/ssl_hello_world_app.erl

Basically, this is the important line:

https://github.com/lukebakken/cowboy/blob/lrb-test/examples/ssl_hello_world/src/ssl_hello_world_app.erl#L27

The corresponding openssl command that successfully connects is this (I added upstream and downstream to my hosts file):

openssl s_client -tls1 -connect upstream:8443 -CAfile ./tls-downstream/cacert.pem -cert ./tls-downstream/cert.pem -key ./tls-downstream/key.pem

Without the -tls1 argument, you will see the handshake error. I am assuming this is due to issues surrounding version negotiation. For some reason, SSLv3 is attempted which causes the error.

I think if you add the versions setting in your environment (both upstream and downstream) you will solve this issue. Let me know how it goes.

Thanks,

Luke

[Renat Nurgaliyev]

Hi Luke,

I've already tried setting versions manually, with no luck, but previously I gave only one version as a value, thought it could be the issue. So I set versions with exactly same line from your example, on both ends of the federation link, but still getting same error. But your example code makes sense, I'll play with it and try to figure out what is happening. Will let you know as soon as I have any results.

Thanks!

[Renat Nurgaliyev]

Hi Guys,

So I managed to sniff TLS between hosts, and actually problem is client not sending it's certificate at all. Hosts use TLS1.2 as expected, client sends hello, server replies with it's certificates and requests client's certificate. Client replies with 0 certificates, and off course server replies with handshake failure. I'm attaching pcap file for reference. 

[Michael Klishin]

Is RabbitMQ Erlang client configured to use TLS per http://rabbitmq.com/ssl.html

and http://www.rabbitmq.com/uri-query-parameters.html?

That’s what Federation and Shovel plugins use.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

<p.pcap>

[Renat Nurgaliyev]

Hi guys,

Sorry for being quiet. So I realized that TLS configuration in configuration file is used only when RabbitMQ acts a server, not when as a client. For federation link, path to certificate and key files should be included in URI like this, as Michael mentioned:

amqps://rabbit-upstream-host:5671?cacertfile=/etc/ssl/ca.crt&certfile=/etc/ssl/rabbit.crt&keyfile=/etc/ssl/rabbit.key&verify=verify_peer&fail_if_no_peer_cert=true

I was still struggling with 'Unknown CA' problem on upstream for a while though. It turned out, that in file specified in 'cacertfile' option may be not only the CA certificate itself, but all certificates from chain.

Thanks everyone for help!:)

[Michael Klishin]

> It turned out, that in file specified in 'cacertfile' option may be not only the CA certificate itself, but all certificates from chain

because it's the chain that's validated, not a single certificate. I'll try to clarify that in the docs.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.