Federation: Insufficient Security - no_suitable_ciphers

[Wolfgang Wiessler]

Hello,

I am trying to set up federation between my main RabbitMQ server (which has 11 clients connected to it successfully) and a new RabbitMQ server that I want to use as a message relay to another client. I want the new server to pick up messages from a certain queue on the main server.

So far I enabled the federation plugins and managed to configure the policies and upstreams in a way that the "Federation status" shows that it attempts to connect to the main server. I do however get an error "closed".

On the main server, in the log files I see these errors:

TLS server: In state hello at tls_handshake.erl:197 generated SERVER ALERT: Fatal - Insufficient Security - no_suitable_ciphers

Both nodes are running the most recent RabbitMQ and Erlang versions (on Windows). And both use identical config files (except of course where local paths differ). I am currently specifying the ciphers like this:

                    {ciphers, ["ECDHE-ECDSA-AES256-GCM-SHA384","ECDHE-RSA-AES256-GCM-SHA384",
                        "ECDHE-ECDSA-AES256-SHA384","ECDHE-RSA-AES256-SHA384", "ECDHE-ECDSA-DES-CBC3-SHA",
                        "ECDH-ECDSA-AES256-GCM-SHA384","ECDH-RSA-AES256-GCM-SHA384","ECDH-ECDSA-AES256-SHA384",
                        "ECDH-RSA-AES256-SHA384","DHE-DSS-AES256-GCM-SHA384","DHE-DSS-AES256-SHA256",
                        "AES256-GCM-SHA384","AES256-SHA256","ECDHE-ECDSA-AES128-GCM-SHA256",
                        "ECDHE-RSA-AES128-GCM-SHA256","ECDHE-ECDSA-AES128-SHA256","ECDHE-RSA-AES128-SHA256",
                            "ECDH-ECDSA-AES128-GCM-SHA256","ECDH-RSA-AES128-GCM-SHA256","ECDH-ECDSA-AES128-SHA256",
                        "ECDH-RSA-AES128-SHA256","DHE-DSS-AES128-GCM-SHA256","DHE-DSS-AES128-SHA256",
                        "AES128-GCM-SHA256","AES128-SHA256","ECDHE-ECDSA-AES256-SHA",
                        "ECDHE-RSA-AES256-SHA","DHE-DSS-AES256-SHA","ECDH-ECDSA-AES256-SHA",
                        "ECDH-RSA-AES256-SHA","AES256-SHA","ECDHE-ECDSA-AES128-SHA",
                        "ECDHE-RSA-AES128-SHA","DHE-DSS-AES128-SHA","ECDH-ECDSA-AES128-SHA",
                        "ECDH-RSA-AES128-SHA","AES128-SHA"]},
                  {honor_cipher_order, true}

I had to do that in an older version due to some Erlang issue. Not sure if that is still needed. I tried it with that list and without it and neither case worked.

I also already ran werl.exe and the ssl:versions(). and ssl:cipher_suites(openssl). and ssl:cipher_suites(erlang) commands and the output is the same on both servers.

Would anybody be able to point me into the right direction for debugging this?

Best regards,

Wolfgang

[Luke Bakken]

Hi Wolfgang,

What I would do at this point is the following:

* Remove the ciphers and honor_cipher_order settings from your main RabbitMQ server and restart it.

* Use this troubleshooting guide and run openssl s_client to verify that a connection can be made and handshake completed - https://www.rabbitmq.com/troubleshooting-ssl.html

* (optional) Use the following Python program to test publishing a message to your main RabbitMQ server - note that paths to cert files will have to be modified: https://gist.github.com/lukebakken/56c115c5fe5d329763ca367e2ce16d01

If all the above steps work, please provide your federation policy definitions.

Thanks,

Luke

[Wolfgang Wiessler]

Hi Luke,

I followed all the steps. Removed the ciphers list, did the SSL troubleshooting (noticed that I used the .cer version instead of the .pem version of the CA in my new broker. Fixed that but that did not help). Except for stunnel since I do not have an non-TLS port open and other clients can connect to the broker without issues. I get Verification: OK and I can see in the remote broker log that connections are accepted. When I try sending the random bytes, I can see in the broker log the bytes that I sent. One time I got an "AMQP    read:errno=0" back, the other times just "read:errno=0". Not sure if I am sending the wrong byes (e.g. "12345678") or whether that indicates a problem.

Tried your python script and it runs without errors (no console output). In the broker log, I can see:

2018-08-02 14:44:21.044 [info] <0.1643.0> accepting AMQP connection <0.1643.0> (XXX:52494 -> YYY:5672)
2018-08-02 14:44:21.559 [info] <0.1643.0> connection <0.1643.0> (XXX:52494 -> YYY:5672): user 'ZZZ' authenticated and granted access to vhost '/'
2018-08-02 14:44:22.575 [info] <0.1643.0> closing AMQP connection <0.1643.0> (XXX:52494 -> YYY:5672, vhost: '/', user: 'ZZZ')

Does that look OK?

Here is my federation URI on the broker that is supposed to pick up messages from the remote broker:

amqps://MyUser:MyPwd@RemoteIP:5672?cacertfile=/rabbitmqcerts/ca.pem&certfile=/rabbitmqcerts/client/cert.pem&keyfile=/rabbitmqcerts/client/key.pem

I am running TLS on port 5672.

[Luke Bakken]

Hi Wolfgang,

I forgot to ask some basic questions at the outset - what version of RabbitMQ and Erlang are you using, and on what version of Windows? Could you provide your complete RabbitMQ configuration file?

I'll follow up on this issue as time allows.

Thanks,

Luke

[Wolfgang Wiessler]

I am using RabbitMQ 3.7.7 and Erlang 21.0.1 on both brokers. This is the configuration file (the new broker that wants to connect via federation is not using the ip range plugin yet):

[
 {ssl, [{versions, ['tlsv1.2', 'tlsv1.1', tlsv1]}]},
 {rabbit, [
    {tcp_listeners, []},
    {ssl_listeners, [5672]},
      
    {ssl_options, [{cacertfile,"C:/ca/ca.pem"},
                    {certfile, "C:/rabbitmqserver/cert.pem"},
                    {keyfile,"C:/rabbitmqserver/key.pem"},
                    {verify,verify_peer},
                    {fail_if_no_peer_cert,true}
                  ]},   
    {ssl_handshake_timeout, 60000},
    {handshake_timeout, 60000},
    {disk_free_limit, 1000000000},
    {auth_backends, [{rabbit_auth_backend_internal, [rabbit_auth_backend_internal, rabbit_auth_backend_ip_range]}]}
 ]},
 {rabbitmq_auth_backend_ip_range, [
        {default_masks, [a list of IPs, inlcuding the one of the other broker]}
 ]}
].

Thanks for looking into this! I am sure I am missing just something trivial but I cannot seem to find it. These SSL and network related problems are tough to debug.

-Wolfgang

[Wolfgang Wiessler]

Forgot to mention that I am running Windows Server 2012 R2 DataCenter on the main broker and Windows Server 2016 on the other one.

[Luke Bakken]

Hi Wolfgang -

Could you double-check all of your certificates to ensure that the keyUsage attribute includes both digitalSignature and keyEncipherment? Without both, you will run into the issue I describe here:

https://github.com/michaelklishin/tls-gen/pull/8

I bet this is what is going on.

Thanks,

Luke

[Wolfgang Wiessler]

Hi Luke,

assuming I understood your suggestions correctly (not an SSL expert), I checked the openssl config file that I am using for my CA and certificates and found these sections:

[ root_ca_extensions ]                                    
basicConstraints = CA:true                                
keyUsage = keyCertSign, cRLSign                           

[ client_ca_extensions ]                                  
basicConstraints = CA:false                               
keyUsage = digitalSignature                               
extendedKeyUsage = 1.3.6.1.5.5.7.3.2                      

[ server_ca_extensions ]                                  
basicConstraints = CA:false                               
keyUsage = keyEncipherment                                
extendedKeyUsage = 1.3.6.1.5.5.7.3.1

Are you saying that "client_ca_extensions" and "server_ca_extensions" should use "keyUsage = digitalSignature,keyEncipherment" ?

If so, does that mean that I need to regenerate all my certificates? Is there a way I can configure two server/client pairs in RabbitMQ? So I can maintain the existing ones with my customer base and try the new ones for the federation?

[Luke Bakken]

Hi Wolfgang,

Are you saying that "client_ca_extensions" and "server_ca_extensions" should use "keyUsage = digitalSignature,keyEncipherment" ?

Yep, that's exactly it. If keyUsage is present, it must have both of those values set (client and server certs) or else you will see the cipher issue you report.

I suspect you can use a separate set of server / client certs as long as the CA certs are concatenated into a single file that cacertfile points to.

Thanks,

Luke

On Friday, August 3, 2018 at 7:17:40 AM UTC-7, Wolfgang Wiessler wrote:

Hi Luke,

[Michael Klishin]

We should really document this keyUsage pitfall :)

--

You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

[Wolfgang Wiessler]

The good news is that your documentation is up to date about the correct OpenSSL configuration: https://www.rabbitmq.com/ssl.html  Several years ago, I copied the config file from your documentation and it did not have "keyUsage = digitalSignature,keyEncipherment" specified. Since everything was working fine outside of federation, I never bothered to double-check if anything changed.

As for the resolution of this for my case: I currently cannot just replace all certificates. I might be able to with the next release, but that will take a while. For now, could you tell me if one of those approaches could work:

- It is possible to configure multiple SSL listener ports. Is it also possible to specify different SSL Options per listener? Based on the structure of the configuration file, it does not look like it

- Can I specify two separate server cert/key combinations in the configuration file?

- You wrote earlier: "I suspect you can use a separate set of server / client certs as long as the CA certs are concatenated into a single file that cacertfile points to". I am not 100% clear on the approach. Are you saying that I can put two separate server certificates into one .pem file and point RabbitMQ to that combined .pem file? Assuming that both certs are from the same CA?

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Luke Bakken]

Hi Wolfgang,

The issue with keyUsage is recent and our docs were updated after we discovered it.

On Monday, August 6, 2018 at 4:36:03 AM UTC-7, Wolfgang Wiessler wrote:

- It is possible to configure multiple SSL listener ports. Is it also possible to specify different SSL Options per listener? Based on the structure of the configuration file, it does not look like it

No, it isn't possible to do this.

 

- Can I specify two separate server cert/key combinations in the configuration file?

No.

  

- You wrote earlier: "I suspect you can use a separate set of server / client certs as long as the CA certs are concatenated into a single file that cacertfile points to". I am not 100% clear on the approach. Are you saying that I can put two separate server certificates into one .pem file and point RabbitMQ to that combined .pem file? Assuming that both certs are from the same CA?

Sorry, I think I misunderstood your environment and thought that there were multiple CA certificates involved. You use only one server certificate, the one with the correct values in keyUsage. For Federation clients, they also have to use the correct values for keyUsage.

Thanks,

Luke

[Wolfgang Wiessler]

I regenerated my certificates with the new openssl.cnf and now I get a Federation upstream in status "running". So that seems to do the trick - thanks!

Here is what I want to achieve:

- Broker A and Broker B both have a queue named "Wolfgang_Test"

- Broker A has Broker B's queue configured as upstream (including a policy that matches the queue "Wolfgang_Test" to the upstream and the upstream points to Broker B's "Wolfgang_Test")

- I published 2 messages on Broker B's "Wolfgang_Test" queue

My expectation is that those two messages would automatically appear in Broker A's "Wolfgang_Test" queue. At the moment, they are not and are staying in Broker B's queue.

Is my expectation about federation correct?

Regards,

Wolfgang

[Wolfgang Wiessler]

I can answer my own question in this case. Those two articles were useful for understanding what queue federation does:

https://www.cloudamqp.com/blog/2015-03-24-rabbitmq-federation.html

https://stackoverflow.com/questions/27220766/rabbitmq-how-to-federate-mirror-messages

When federating queues, the messages are only being pushed to the downstream broker (in this case Broker A), when there is an active consumer on Broker A's federated queue. If not, then the messages just stay in the original queue on Broker B.

So after having a consumer listen on "Wolfgang_Test" on Broker A, the messages started flowing.

Thanks for all your help!

[Michael Klishin]

Thank you for reporting back to the list, Wolfgang!

--

You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.