Unable to get TLS Auth to work

[Byte Flinger]

Hi guys

I am attempting to get rabbitmq working with TLS auth but struggling a bit.

The setup is simple. I have generated a Root CA and from that Root CA I generated a Rabbit CA.

I then generate 2 key/certificate pairs, one is for the server and one for the client signed by the Rabbit CA.

I configure rabbit's advanced configuration

{rabbit, [

               {auth_mechanisms, ['EXTERNAL']},

               {ssl_options, [{cacertfile,"/etc/tls/ca.crt"},

                               {certfile,  "/etc/tls/tls.crt"},

                               {keyfile,   "/etc/tls/tls.key"},

                                %% We only trust TLS 1.2

                               {versions, ['tlsv1.2']},

                               

                               {verify,verify_peer},

                               

                               {fail_if_no_peer_cert,true},

                               

                               %% The server should dictate its preference of cipher suites on TLS negotiation

                               {honor_cipher_order, true},

                               {honor_ecc_order, true},

                               

                               %% We only accept these cipher suites

                               {ciphers,  [

                                           {ecdhe_ecdsa,aes_256_gcm,null,sha384},

                                           {ecdhe_rsa,aes_256_gcm,null,sha384},

                                           {ecdhe_ecdsa,aes_256_cbc,sha384,sha384},

                                           {ecdhe_rsa,aes_256_cbc,sha384,sha384},

                                           {ecdh_ecdsa,aes_256_gcm,null,sha384},

                                           {ecdh_rsa,aes_256_gcm,null,sha384},

                                           {ecdh_ecdsa,aes_256_cbc,sha384,sha384},

                                           {ecdh_rsa,aes_256_cbc,sha384,sha384},

                                           {dhe_rsa,aes_256_gcm,null,sha384},

                                           {dhe_dss,aes_256_gcm,null,sha384},

                                           {dhe_rsa,aes_256_cbc,sha256},

                                           {dhe_dss,aes_256_cbc,sha256},

                                           {rsa,aes_256_gcm,null,sha384},

                                           {rsa,aes_256_cbc,sha256},

                                           {ecdhe_ecdsa,aes_128_gcm,null,sha256},

                                           {ecdhe_rsa,aes_128_gcm,null,sha256},

                                           {ecdhe_ecdsa,aes_128_cbc,sha256,sha256},

                                           {ecdhe_rsa,aes_128_cbc,sha256,sha256},

                                           {ecdh_ecdsa,aes_128_gcm,null,sha256},

                                           {ecdh_rsa,aes_128_gcm,null,sha256},

                                           {ecdh_ecdsa,aes_128_cbc,sha256,sha256},

                                           {ecdh_rsa,aes_128_cbc,sha256,sha256},

                                           {dhe_rsa,aes_128_gcm,null,sha256},

                                           {dhe_dss,aes_128_gcm,null,sha256},

                                           {dhe_rsa,aes_128_cbc,sha256},

                                           {ecdh_rsa,aes_128_gcm,null,sha256}

                                         ]}

                             ]}

               ]},

Where ca.crt  is the Rabbit CA certificate (I also attempted with having both the Rabbit Ca and the Root CA in the same file but no difference). Then I start rabbitmq (Dockerhub 3.7-management-alpine) and attempt to connect using a client with the other certificate but I get "remote error: tls: unknown certificate authority" in the client side and in the rabibt server logs I see the error

2018-08-02 14:03:57.507 [info] <0.844.0> TLS server: In state certify at ssl_handshake.erl:1626 generated SERVER ALERT: Fatal - Unknown CA

I am also seeing this message often in the server

2018-08-02 14:05:08.710 [info] <0.905.0> TLS server: In state certify received CLIENT ALERT: Fatal - Bad Certificate

though I suspect this might be a clent attempting to connect without a certificate

I am don't quite understand why it does not work. I have double checked the rabbit support EC certificates (Which is my case), I have double checked the client certificate is signed by the same CA loaded in rabbit. Am I missing something?

Any help would be appreciated.

[Luke Bakken]

Hello,

I suggest starting simple and removing all of the "restrictions" in your configuration. Remove the ciphers list, remove versions, set fail_if_no_peer_cert to false.

Then see what happens. We also have a comprehensive troubleshooting guide here: https://www.rabbitmq.com/troubleshooting-ssl.html

Thanks,

Luke

[Byte Flinger]

Thank you Luke.

I went through the guide and can confirm that openssl seems to be able to connect to rabbit (And client/server test with openssl only works fine). I also disabled the 3 parameters you mentioned but the issue is still there.

I get a feeling that something might be wrong on the client side but I cannot find any issues with my as it is quite simple so I am a bit at a loss on how to proceed right now

[Luke Bakken]

Hello,

openssl seems to be able to connect to rabbit

Could you please attach the output of the openssl s_client command you're running, as well as the command itself? The reason I ask is the last time someone thought openssl connected successfully it actually didn't do the handshake.

I re-read your initial post and notice that you have an intermediate CA in place. You should use the depth TLS/SSL option and set it to at least 2 - http://erlang.org/doc/man/ssl.html

Thanks,

Luke

[Luke Bakken]

For what it's worth, intermediate CAs are mentioned here: https://www.rabbitmq.com/troubleshooting-ssl.html#certificate-chains

[Byte Flinger]

Hi

Here's what I get on the client side (Data below is masked)

$ openssl s_client -connect myserver:5671 -cert client.pem -key client.key   -CAfile ca.crt
CONNECTED(00000003)
depth=2 C = US, L = Michigan, O = MyCompany, CN = MyCompany Root
verify return:1
depth=1 C = US, L = Michigan, O = MyCompany, CN = MyCompany Rabbit
verify return:1
depth=0 C = US, L = Michigan, O = MyCompany, CN = rabbitserver
verify return:1
---
Certificate chain
 0 s:/C=US/L=Michigan/O=MyCompany/CN=rabbitserver
   i:/C=US/L=Michigan/O=MyCompany/CN=MyCompany Rabbit
 1 s:/C=US/L=Michigan/O=MyCompany/CN=MyCompany Rabbit
   i:/C=US/L=Michigan/O=MyCompany/CN=MyCompany Root
 2 s:/C=US/L=Michigan/O=MyCompany/CN=MyCompany Root
   i:/C=US/L=Michigan/O=MyCompany/CN=MyCompany Root
---
Server certificate
-----BEGIN CERTIFICATE-----
<--BASE64 DATA-->
-----END CERTIFICATE-----
subject=/C=US/L=Michigan/O=MyCompany/CN=rabbitserver
issuer=/C=US/L=Michigan/O=MyCompany/CN=MyCompany Rabbit
---
Acceptable client certificate CA names
/C=US/L=Michigan/O=MyCompany/CN=MyCompany Root
/C=US/L=Michigan/O=MyCompany/CN=MyCompany Rabbit
Client Certificate Types: ECDSA sign, RSA sign, DSA sign
Requested Signature Algorithms: ECDSA+SHA512:RSA+SHA512:ECDSA+SHA384:RSA+SHA384:ECDSA+SHA256:RSA+SHA256:ECDSA+SHA224:RSA+SHA224:ECDSA+SHA1:RSA+SHA1:DSA+SHA1
Shared Requested Signature Algorithms: ECDSA+SHA512:RSA+SHA512:ECDSA+SHA384:RSA+SHA384:ECDSA+SHA256:RSA+SHA256:ECDSA+SHA224:RSA+SHA224:ECDSA+SHA1:RSA+SHA1:DSA+SHA1
Peer signing digest: SHA512
Server Temp Key: ECDH, P-521, 521 bits
---
SSL handshake has read 2280 bytes and written 2115 bytes
Verification: OK
---
New, TLSv1.2, Cipher is ECDHE-ECDSA-AES256-GCM-SHA384
Server public key is 256 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-ECDSA-AES256-GCM-SHA384
    Session-ID: 7DA984EFC27DCF3646A2851C169E17AB36CC0E56B86299528339656EBFD4EA80
    Session-ID-ctx:
    Master-Key: ...
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1533223225
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
    Extended master secret: no
---
closed

And this is what I see in the rabbit logs

2018-08-02 15:20:35.420 [info] <0.2215.0> accepting AMQP connection <0.2215.0> (10.56.1.195:46486 -> 10.56.1.256:5671)
2018-08-02 15:20:35.421 [warning] <0.2215.0> closing AMQP connection <0.2215.0> (10.56.1.195:46486 -> 10.56.1.256:5671):
{handshake_timeout,handshake}

There is a pause after running openssl and after 10 seconds it exits with the "closed" message at which point rabbit prints the data above in the logs.

I did find the depth section and attempted setting it to 3 however that did not seem to make a difference

[Luke Bakken]

Hello,

Thanks for the output. Everything appears correct, so at this time I suggest testing a scenario that does not involve an intermediate certificate. For what it's worth, the RabbitMQ team often uses this project to generate test certs, including certs for scenarios involving intermediates:

https://github.com/michaelklishin/tls-gen/

And just as I was typing this, I remembered my response here:

https://groups.google.com/d/msg/rabbitmq-users/1Pyowi3exro/s1nu9MgAAgAJ

Basically, if you are requiring client certificates (which {fail_if_no_peer_cert, true} does) the client must send all intermediate CA certs. I have my notes here:

https://gist.github.com/lukebakken/582806c8844d3fc08c22e898b272325e

I think you're running into the same issue. Trying a scenario without intermediate certs will confirm it.

Thanks,

Luke

On Thursday, August 2, 2018 at 8:31:46 AM UTC-7, Byte Flinger wrote:

Hi

Here's what I get on the client side (Data below is masked)

[Byte Flinger]

Thank you Luke.

I did a lot of testing on my side and arrived at no better situation. Reading your notes and comments I am uncertain on how to go about this if one wants to keep the intermediate CA. I attempted sending both the client certificate and the intermediate CA from the client as you suggested but got the same error.

I also loaded rabbit with both CA certificates using the cacertifle and then attempted to connect sending the client certificate and intermediate and got another error

"Exception (403) Reason: "SASL could not negotiate a shared mechanism"

I guess not using an intermediate CA would be the way to go but is there any way whatsoever to use an intermediate CA?

[Luke Bakken]

Hello again -

Yes, using intermediate CAs is supported if the entire cert chain is in the file used as cacertfile by RabbitMQ, and sent by the client when connecting.

"SASL could not negotiate a shared mechanism" is a different issue. It means that RabbitMQ expected to authenticate your client one way, but your client said it was going to authenticate a different way. Since I don't know anything about your client code, I can't say what's wrong. What I would try is to configure your client to use PLAIN authentication (which is usually the default) and then provide a username / password. I suspect you might be trying to authenticate using a client cert at this point?

Luke

[Byte Flinger]

The idea was to basically do some sort of client certificate auth.

At the moment what I am trying to achieve is to allow the client to connect and read/write any topic as long as it connects with a certificate belonging to the intermediate chain (without having to create the user in rabbit's own internal database) which is what I expected this configuration would do.

How would I go about that?

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Luke Bakken]

Hello -

Before going on to that, you should get client certificate auth working. Note that a user will still have to be created in RabbitMQ with a username that matches either the full DN or CN value of the certificate (see the plugin config).

What client library are you using? You will have to configure TLS/SSL to pass the client cert and the full cert chain, and also to specify EXTERNAL auth.

Finally, this plugin must be enabled - there is good info there about configuration:

https://github.com/rabbitmq/rabbitmq-auth-mechanism-ssl

Let me know how it goes. I have tested this scenario so I know it works, it's just involved in getting it working.

Luke

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.

[Byte Flinger]

The plugin itself is already enabled. This is the one I have been trying to use.

I had hope to avoid creating the user as it is not very dynamic and requires running a cli and writes to an internal database but I'll give it a try first

Also I have configured EXTERNAL auth only in the rabbit config, are you saying the client also needs to specify that somehow?

I am using the Go client library found at https://github.com/streadway/amqp

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.

[Luke Bakken]

Hello,

Yes the client needs to support EXTERNAL auth as well. I did a quick glance through the streadway/amqp code, and believe you need to do this:

* Create a type that implements the Authentication interface: https://github.com/streadway/amqp/blob/e5adc2ada8b8efff032bf61173a233d143e9318e/auth.go#L12-L17

* That type should return the string EXTERNAL for Mechanism(), and I believe it can return the empty string for Response()

* When you instantiate the Config struct, set the Authentication array to include the type you just created, in the same way as how the default is used here: https://github.com/streadway/amqp/blob/e5adc2ada8b8efff032bf61173a233d143e9318e/connection.go#L171-L173

* Then, ensure that your client TLS config is set correctly: https://github.com/streadway/amqp/blob/e5adc2ada8b8efff032bf61173a233d143e9318e/connection.go#L51-L55

You will have to create at least one user, however each of your client certificates can use the same user in their DN / CN value.

Let me know how it works and I'll create a task to add support for EXTERNAL natively within streadway/amqp

Thanks -

Luke

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.

[Byte Flinger]

Hi Luke

Mostly appreciated.

So I have now gotten pretty far with this. I setup the rabbitmq with the full CA chain (root ca and intermediate ca), added configuration to make use of CN parameter from the plugin and kept EXTERNAL as the only authentication method in the rabbit config. Then I created a user in the database with name "myuser" and on the client side I configured as you specified above and present both the client certificate (Containing CN=myuser) and the intermediate ca certificate.

The issue now is that the server seems to refuse the user throwing "invalid credentials" back. I see the following message on the server

2018-08-03 07:45:58.754 [info] <0.1012.0> accepting AMQP connection <0.1012.0> (10.46.6.132:51384 -> 10.46.6.213:5671)
2018-08-03 07:45:58.795 [error] <0.1012.0> Error on AMQP connection <0.1012.0> (10.46.6.132:51384 -> 10.46.6.213:5671, state: starting):
EXTERNAL login refused: user 'myuser' - invalid credentials
2018-08-03 07:46:01.796 [info] <0.1012.0> closing AMQP connection <0.1012.0> (10.46.6.132:51384 -> 10.46.6.213:5671)

The user "myuser" has a password set on him however as per documentation on the plugin github page that should not matter as it is not checked. It seems the plugin is trying to somehow check the certificate towards the user however I see nowhere on how one can tie a certificate to a user. Did I miss something?

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.

[Byte Flinger]

Actually disregard the above post. I had mistakenly commented away the user creation part in the rabbit config.

It is working now.

The next step is figure out how to handle the internal database since the plugin requires a user in it. Do you know where the user/password database is stored? Is there documentation on this?

[Luke Bakken]

Hello,

You should use rabbitmqctl add_user or the HTTP API to add a user to RabbitMQ.

https://www.rabbitmq.com/passwords.html

Don't forget to set the user's permissions as well.

If you add a user with a username that doesn't correctly correspond to what is in your certificate, you'll get a log message showing what actual username was tried, and you can adjust.

Thanks,

Luke