Has anyone configured RabbitMQ cluster with TLSed etcd?

[Liviu Ungureanu]

I am trying to setup a RabbitMQ cluster using etcd and TLS secured.

Since RabbitMQ nodes need to connect to etcd in order to adhere to the cluster so far I have:

cluster_partition_handling = pause_minority

cluster_formation.peer_discovery_backend = rabbit_peer_discovery_etcd

cluster_formation.etcd.host = 10.232.122.160

cluster_formation.etcd.port = 4011

cluster_formation.etcd.scheme = https

ssl_options.certfile = /etc/ssl/cert.pem

ssl_options.keyfile = /etc/ssl/key.pem

ssl_options.cacertfile = /etc/ssl/ca-certs.pem

With the config above,  etcd's TLS  fails the handshake:

2020-03-26 22:13:55.510 [debug] <0.12225.0> Response: [{error,{failed_connect,[{to_address,{"10.232.122.160",4011}},{inet,[inet],{tls_alert,{bad_certificate,"TLS client: In state cipher received SERVER ALERT: Fatal - Bad Certificate\n "}}}]}}]

Note: I trust the certs above in ssl_options are correct since I have successfully validated them against my etcd:

openssl s_client  -connect  10.232.122.160:4011   -CAfile  /etc/ssl/ca-certs.pem -cert /etc/ssl/cert.pem  -key  /etc/ssl/key.pem

Does this mean that there are different clustering parameters for TLS (i.e. separated from ssl_options.xxx tls params above? )

The RabbitMQ documentation looks like it does not detail etcd clustering with TLS.

[Michael Klishin]

In the current plugin it's a matter of configuring the standard library HTTP client, we will try to come up with an example soon.

However, for 3.8.4 etcd peer discovery will be completely reworked to be based on the v3 API and client. It does not use

HTTP the same way, and neither uses the same client, so there will be a separate set of options similar to those you use in other TLS

contexts with RabbitMQ. This is just a heads-up that things will change.

Also note that there's hardly any sensitive data that is being read from and written to etcd, only locks and node names.

You likely run etcd with TLS connections from clients across the board but for RabbitMQ peer discovery, it makes little difference

besides an ability to authenticate (which a v3 API client will also provide independently).

[Michael Klishin]

The error simply says that the client did not provide a certificate the server trusts [1].

Which is true because it hasn't provided any. If you don't use TLS for authentication,

a short term solution can be disable peer verification on the etcd side (which the Internet will scorn on and tell you that it is "really unsafe" but

in reality it disables only one main feature of TLS, and does not affect traffic encryption, which can be good enough).

1. https://www.rabbitmq.com/ssl.html#peer-verification

On Friday, March 27, 2020 at 1:26:43 AM UTC+3, Liviu Ungureanu wrote:

[Luke Bakken]

Hello Liviu,

Please be patient when requesting assistance from the RabbitMQ community. There is no need to open a GitHub issue that duplicates a message to this list (https://github.com/rabbitmq/rabbitmq-peer-discovery-etcd/issues/23). The people who assist do so on their free time and generally try to answer questions within a week.

You will have to create the file /etc/rabbitmq/advanced.config with this content:

[

    {ssl, [

        {verify, verify_peer},

        {cacertfile, "/path/to/CA/certificate"},

        {depth, 99}

    ]}

].

The file pointed to by cacertfile must contain one CA certificate that was used to sign the certificate used by your etcd server.

Note that depth is only required when you are using intermediate certificates.

The above settings will affect all https requests that are made by RabbitMQ. More than likely in your environment this will just be the etcd peer discovery backend.

You will have to restart RabbitMQ after creating this file. Check the RabbitMQ log to ensure this file is being used in conjunction with rabbitmq.conf. It will be logged just after the banner text.

Thanks,

Luke

[Liviu Ungureanu]

Thank you Luke for your response.


I have applied your config, but  the `etcd` I am using has mutual tls enabled which means  RabbitMQ is also requested its identity cert.

I have further added `certfile` and `keyfile` in `advanced.config` :

 {ssl, [

        {verify, verify_peer},

        {cacertfile, "/etc/ssl/ca-certs.pem"},

        {certfile, "/etc/ssl/cert.pem"},

        {keyfile, "/etc/ssl/key.pem"},

        {depth, 99}

 ]},


As whireshark show

Again I've made sure the certs are correct since the following mutual tls invokation works:

openssl s_client    -connect 10.231.18.59:4011 -showcerts  -CAfile /etc/ssl/ca-certs.pem  -cert /etc/ssl/cert.pem  -key /etc/ssl/key.pem

Have I specified    {certfile, "/etc/ssl/cert.pem"} and   {keyfile, "/etc/ssl/key.pem"} correctly? Or what is the correct way to specify them?

Regards,

  Liviu

[Luke Bakken]

Hello,

Please attach your complete RabbitMQ log file, and the complete contents of all of your RabbitMQ configuration files.

RabbitMQ logs the effective configuration when it starts up.

Thanks,

Luke

[Liviu Ungureanu]

Hello Luke,

I have attached logs and all the config files used by RabbitMQ.

In the start up logs one can see:

{"date":"2020-04-28","file":"Undefined","function":"register","id":"Undefined","line":"93","message":"Failed to register node with etcd: {failed_connect,\n    [{to_address,{\"10.231.18.59\",4011}},\n     {inet,\n         [inet],\n         {tls_alert,\n             {bad_certificate,\n                 \"TLS client: In state cipher received SERVER ALERT: Fatal - Bad Certificate\\n \"}}}]}","module":"rabbit_peer_discovery_etcd","pid":"<0.345.0>","severity":"error","time":"15:37:17.321"}
{"date":"2020-04-28","file":"Undefined","function":"register","id":"Undefined","line":"212","message":"Failed to register with peer discovery backend rabbit_peer_discovery_etcd: \"{failed_connect,\\n    [{to_address,{\\\"10.231.18.59\\\",4011}},\\n     {inet,\\n         [inet],\\n         {tls_alert,\\n             {bad_certificate,\\n                 \\\"TLS client: In state cipher received SERVER ALERT: Fatal - Bad Certificate\\\\n \\\"}}}]}\"","module":"rabbit_peer_discovery","pid":"<0.345.0>","severity":"error","time":"15:37:17.322"}

Regards,

  Liviu

[Luke Bakken]

Hello,

That is interesting information. What is the full output of the openssl command you ran?

openssl s_client -connect 10.231.18.59:4011 -showcerts  -CAfile /etc/ssl/ca-certs.pem  -cert /etc/ssl/cert.pem  -key /etc/ssl/key.pem

You may have to enable more TLS ciphers - https://www.rabbitmq.com/ssl.html#cipher-suites

Thanks -

Luke

[Luke Bakken]

In addition, your rabbitmq.conf file does not have a ssl_options.cacertfile setting. That is almost certainly an error.

[Liviu Ungureanu]

Hi Luke,

On "your rabbitmq.conf file does not have a ssl_options.cacertfile setting".

RabbitMQ connections(i.e. AMQP protocol) is simple TLS secured so that's why I haven't put a ssl_options.cacertfile setting. Plus since this addresses AMQP it should not interfere with etcd which uses https

Please also find the output from openssl you have requested in  openssl_output.txt

Regards,

   Liviu

[Luke Bakken]

Hi Liviu,

If leaving that setting out doesn't affect your AMQPS connections keep it that way. You're the first person I've seen who has left it out :-)

My next suggestion is to enable all ciphers in the ssl section of advanced.config

Run this command:

rabbitmq-diagnostics cipher_suites --openssl-format=false --formatter=erlang -q

Then, take the output, and add it to advanced.config, like this:

 {ssl, [

        {verify, verify_peer},

        {cacertfile, "/etc/ssl/ca-certs.pem"},

        {certfile, "/etc/ssl/cert.pem"},

        {keyfile, "/etc/ssl/key.pem"},

        {depth, 99},

        {ciphers, [

            % LIST OF CIPHERS GOES HERE

        ]}

 ]},

Restart RabbitMQ, and see if the following error is still logged:

TLS client: In state cipher received SERVER ALERT: Fatal - Bad Certificate

[Liviu Ungureanu]

Not clear why RabbitMQ does not send its identity cert (i.e. certfile) to etcd.

 {ssl, [

        {verify, verify_peer},

        {cacertfile, "/etc/ssl/ca-certs.pem"},

        {certfile, "/etc/ssl/cert.pem"},

        {keyfile, "/etc/ssl/key.pem"},

        {depth, 99}

 ]},

[Liviu Ungureanu]

HI Luke,

The cypher log still appears.

Please find attached the following contents:

advanced.config_with_cyphers.txt

rabbitmq_start_up_with_cyphers.txt

Regards,

  Liviu

[Luke Bakken]

If a cipher suite can't be agreed upon, TLS handshake stops well before certs are exchanged.

[Luke Bakken]

Hi Liviu,

That's probably not the complete list of cipher suites you can enable. I may have given you an outdated command. Here is what I ran in my RabbitMQ 3.8.3 environment:

rabbitmq-diagnostics cipher_suites --format erlang --all > ~/cipher_suites.txt

I have attached the generated file. I would expect the list to be at least that long in your environment.

Sorry this is a hassle. All evidence points to etcd wanting to use a cipher suite that isn't currently enabled.

Thanks -

Luke

[Liviu Ungureanu]

Hi Luke,

Executing 

rabbitmq-diagnostics cipher_suites --format erlang --all > ~/cipher_suites.txt

on my RabbitMQ produces an identical list of ciphers.

Unfortunately the cipher complain still appears in the logs.

Attached:

advanced.config_with_2nd_set_of_cyphers.txt

rabbitmq_start_up_with_2nd_set_of_cyphers.txt

So openssl has the cypher requested by etcd whereas RabbitMQ does not?

Regards,

  Liviu

[Liviu Ungureanu]

Also,

Here is the handshake failure in wireshark:

I've also attached the wireshark capture:

Filtering by  tcp.port == 4011 && ssl gets you to the package in the image above.

[Luke Bakken]

Hello,

OK here's what I think is going on after doing some searching. I found this issue which got me thinking - https://github.com/esl/MongooseIM/issues/2291

You have configured the etcd plugin to connect via an IP address and have told the Erlang TLS implementation to verify the etcd server's identity.

From rabbitmq.conf, the IP is specified:

cluster_formation.etcd.host = 10.231.18.59

cluster_formation.etcd.port = 4011

cluster_formation.etcd.scheme = https

From advanced.config, peer verification is enabled:

{ssl, [

    {verify, verify_peer},

    ...

]},

But, the server certificate used by etcd has no CN or SAN value that corresponds to a host name or the IP address being used:

Server certificate

subject=C = UK, ST = England, L = Enfield, O = Metaswitch Networks, CN = CNMS CI OAM Test 1

In short, there is no way that the Erlang TLS implementation can do peer validation. The openssl s_client command "succeeds" because you're not telling it to do peer verification, either. Here's how you would do that (https://www.rabbitmq.com/troubleshooting-ssl.html):

openssl s_client -connect HOST:PORT -cert client_certificate.pem -key client_key.pem -CAfile ca_certificate.pem -verify 8 -verify_hostname ETCD_HOSTNAME

You need to ensure your etcd certificate's CN value corresponds to the host name, and then use that host name in rabbitmq.conf. Or, I believe you can use the IP address in the CN field. Or, disable peer verification.

Final question - how have you configured etcd? The openssl output suggests that it has been configured for client certificate authentication. Can you share your complete etcd config?

Thanks -

Luke

[Liviu Ungureanu]

Hi Luke,

If I remove the verify from advanced.config:

 {ssl, [

        {cacertfile, "/etc/ssl/ca-certs.pem"},

        {certfile, "/etc/ssl/cert.pem"},

        {keyfile, "/etc/ssl/key.pem"},

        {depth, 99},

        {ciphers, [

           {ecdhe_ecdsa,aes_256_gcm,aead,sha384}

          ,{ecdhe_rsa,aes_256_gcm,aead,sha384}

          ,{ecdhe_ecdsa,aes_256_cbc,sha384,sha384}

          ,{ecdhe_rsa,aes_256_cbc,sha384,sha384}

          ,{ecdh_ecdsa,aes_256_gcm,aead,sha384}

          ,{ecdh_rsa,aes_256_gcm,aead,sha384}

          ,{ecdh_ecdsa,aes_256_cbc,sha384,sha384}

          ,{ecdh_rsa,aes_256_cbc,sha384,sha384}

          ,{dhe_rsa,aes_256_gcm,aead,sha384}

          ,{dhe_dss,aes_256_gcm,aead,sha384}

          ,{dhe_rsa,aes_256_cbc,sha256}

          ,{dhe_dss,aes_256_cbc,sha256}

          ,{ecdhe_ecdsa,aes_128_gcm,aead,sha256}

          ,{ecdhe_rsa,aes_128_gcm,aead,sha256}

          ,{ecdhe_ecdsa,aes_128_cbc,sha256,sha256}

          ,{ecdhe_rsa,aes_128_cbc,sha256,sha256}

          ,{ecdh_ecdsa,aes_128_gcm,aead,sha256}

          ,{ecdh_rsa,aes_128_gcm,aead,sha256}

          ,{ecdh_ecdsa,aes_128_cbc,sha256,sha256}

          ,{ecdh_rsa,aes_128_cbc,sha256,sha256}

          ,{dhe_rsa,aes_128_gcm,aead,sha256}

          ,{dhe_dss,aes_128_gcm,aead,sha256}

          ,{dhe_rsa,aes_128_cbc,sha256}

          ,{dhe_dss,aes_128_cbc,sha256}

          ,{ecdhe_ecdsa,aes_256_cbc,sha}

          ,{ecdhe_rsa,aes_256_cbc,sha}

          ,{dhe_rsa,aes_256_cbc,sha}

          ,{dhe_dss,aes_256_cbc,sha}

          ,{ecdh_ecdsa,aes_256_cbc,sha}

          ,{ecdh_rsa,aes_256_cbc,sha}

          ,{ecdhe_ecdsa,aes_128_cbc,sha}

          ,{ecdhe_rsa,aes_128_cbc,sha}

          ,{dhe_rsa,aes_128_cbc,sha}

          ,{dhe_dss,aes_128_cbc,sha}

          ,{ecdh_ecdsa,aes_128_cbc,sha}

          ,{ecdh_rsa,aes_128_cbc,sha}

          ,{ecdhe_ecdsa,chacha20_poly1305,aead,sha256}

          ,{ecdhe_rsa,chacha20_poly1305,aead,sha256}

          ,{dhe_rsa,chacha20_poly1305,aead,sha256}

          ,{rsa_psk,aes_256_gcm,aead,sha384}

          ,{rsa_psk,aes_256_cbc,sha384,sha384}

          ,{rsa_psk,aes_128_gcm,aead,sha256}

          ,{rsa_psk,aes_128_cbc,sha256}

          ,{rsa_psk,aes_256_cbc,sha}

          ,{rsa_psk,aes_128_cbc,sha}

          ,{rsa_psk,'3des_ede_cbc',sha}

          ,{rsa_psk,rc4_128,sha}

          ,{srp_rsa,'3des_ede_cbc',sha}

          ,{srp_dss,'3des_ede_cbc',sha}

          ,{srp_rsa,aes_128_cbc,sha}

          ,{srp_dss,aes_128_cbc,sha}

          ,{srp_rsa,aes_256_cbc,sha}

          ,{srp_dss,aes_256_cbc,sha}

          ,{ecdhe_ecdsa,rc4_128,sha}

          ,{ecdhe_rsa,rc4_128,sha}

          ,{ecdh_ecdsa,rc4_128,sha}

          ,{ecdh_rsa,rc4_128,sha}

          ,{rsa,rc4_128,sha}

          ,{rsa,rc4_128,md5}

          ,{dhe_rsa,des_cbc,sha}

          ,{rsa,des_cbc,sha}

          ,{ecdhe_ecdsa,'3des_ede_cbc',sha}

          ,{ecdhe_rsa,'3des_ede_cbc',sha}

          ,{dhe_rsa,'3des_ede_cbc',sha}

          ,{dhe_dss,'3des_ede_cbc',sha}

          ,{ecdh_ecdsa,'3des_ede_cbc',sha}

          ,{ecdh_rsa,'3des_ede_cbc',sha}

          ,{rsa,aes_256_gcm,aead,sha384}

          ,{rsa,aes_256_cbc,sha256}

          ,{rsa,aes_128_gcm,aead,sha256}

          ,{rsa,aes_128_cbc,sha256}

          ,{rsa,aes_256_cbc,sha}

          ,{rsa,aes_128_cbc,sha}

          ,{rsa,'3des_ede_cbc',sha}     

        ]}  

 ]},

I still get:

{"date":"2020-04-28","file":"Undefined","function":"register","id":"Undefined","line":"93","message":"Failed to register node with etcd: {failed_connect,\n    [{to_address,{\"10.231.18.59\",4011}},\n     {inet,\n         [inet],\n         {tls_alert,\n             {bad_certificate,\n                 \"TLS client: In state cipher received SERVER ALERT: Fatal - Bad Certificate\\n \"}}}]}","module":"rabbit_peer_discovery_etcd","pid":"<0.347.0>","severity":"error","time":"22:21:07.316"}

{"date":"2020-04-28","file":"Undefined","function":"register","id":"Undefined","line":"212","message":"Failed to register with peer discovery backend rabbit_peer_discovery_etcd: \"{failed_connect,\\n    [{to_address,{\\\"10.231.18.59\\\",4011}},\\n     {inet,\\n         [inet],\\n         {tls_alert,\\n             {bad_certificate,\\n                 \\\"TLS client: In state cipher received SERVER ALERT: Fatal - Bad Certificate\\\\n \\\"}}}]}\"","module":"rabbit_peer_discovery","pid":"<0.347.0>","severity":"error","time":"22:21:07.316"}

It looks like issue remains around the cipher.

Thanks,

   Liviu

[Luke Bakken]

Hi Liviu,

Please attach your complete etcd configuration file. I will try to find time to reproduce this using certificates generated by this project - https://github.com/michaelklishin/tls-gen

For the time being you will have to use http:// communication because I can't figure out why you're having this issue.

Thanks -

Luke

[Liviu Ungureanu]

Hi Luke,

etcd is started in docker container with the following parameters:

   "Entrypoint": [

                "etcd",

                "--advertise-client-urls=https://127.0.0.1:2379",

                "--cert-file=/etc/kubernetes/pki/etcd/server.crt",

                "--client-cert-auth=true",

                "--data-dir=/var/lib/etcd",

                "--initial-advertise-peer-urls=https://127.0.0.1:2380",

                "--initial-cluster=localhost=https://127.0.0.1:2380",

                "--key-file=/etc/kubernetes/pki/etcd/server.key",

                "--listen-client-urls=https://127.0.0.1:2379",

                "--listen-peer-urls=https://127.0.0.1:2380",

                "--name=localhost",

                "--peer-cert-file=/etc/kubernetes/pki/etcd/peer.crt",

                "--peer-client-cert-auth=true",

                "--peer-key-file=/etc/kubernetes/pki/etcd/peer.key",

                "--peer-trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt",

                "--snapshot-count=10000",

                "--trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt"

            ],

Of particular interest is --client-cert-auth=true

which I guess enables mutual TLS by requesting the clients (i.e. RabbitMQ) to send their certs.

I have also attached my etcd test certs in order to be closer to the problem.

As RabbitMQ  is co-hosted with etcd in my setup,  it "mounts" the same etcd certs in  the docker running RabbitMQ.

Regards,

 Liviu

On Friday, March 27, 2020 at 12:26:43 AM UTC+2, Liviu Ungureanu wrote:

[Luke Bakken]

Thanks a lot! I have enough information to test in my environment.

[Luke Bakken]

Hi Liviu,

Unfortunately I have not yet had time to investigate. Please note that the next release of the etcd plugin will be using the v3 API.

Thanks -

Luke