Rabbitmq server fails to change the cipher spec

[Turbocoder]

I also have posted this issue on stack-overflow  https://stackoverflow.com/questions/49927848/rabbitmq-server-fails-to-change-the-cipher-spec

I am running rabbitmq:3-management with ssl enabled. Without SSL things work fine.

Version

Java 1.8

spring-cloud-starter-bus-amqp => 1.3.1.RELEASE

Spring boot parent => 1.5.9.RELEASE

My client is not able to finish ssl handshake. Looks like server is failing to change the cipher spec. Below are java logs.

update handshake state: change_cipher_spec
upcoming handshake states: client finished[20]
upcoming handshake states: server change_cipher_spec[-1]
upcoming handshake states: server finished[20]
springCloudBus.anonymous.aZM2olJcQ2GjJmGdOiE1mA-2, WRITE: TLSv1.1 Change Cipher Spec, length = 1
[Raw write]: length = 6
0000: 14 03 02 00 01 01                                  ......
*** Finished
verify_data:  { 190, 239, 197, 246, 182, 0, 21, 204, 12, 16, 169, 239 }
***
update handshake state: finished[20]
upcoming handshake states: server change_cipher_spec[-1]
upcoming handshake states: server finished[20]
springCloudBus.anonymous.aZM2olJcQ2GjJmGdOiE1mA-2, WRITE: TLSv1.1 Handshake, length = 32
Padded plaintext before ENCRYPTION:  len = 64
0000: 3D 2C 53 62 10 27 46 71   C1 A7 A1 5A D3 08 DC EC  =,Sb.'Fq...Z....
0010: 14 00 00 0C BE EF C5 F6   B6 00 15 CC 0C 10 A9 EF  ................
0020: 30 4C 37 F8 98 A6 17 0E   A9 D3 45 2F C8 F6 66 3E  0L7.......E/..f>
0030: A3 9D 5C 25 0B 0B 0B 0B   0B 0B 0B 0B 0B 0B 0B 0B  ..\%............
[Raw write]: length = 69
0000: 16 03 02 00 40 42 30 D1   B4 DF 8E 22 CE 8D 72 E4  ....@B0...."..r.
0010: 24 9B 31 22 3E 6C 43 FA   63 9A 19 7E D4 7C 13 BE  $.1">lC.c.......
0020: 11 12 43 B2 20 ED EC 71   D9 BA 11 79 34 C3 A9 C9  ..C. ..q...y4...
0030: 0A AE 45 26 1D 00 2B 94   04 33 95 3B CD EB 11 FD  ..E&..+..3.;....
0040: EC C1 57 84 D0                                     ..W..
[Raw read]: length = 5
0000: 15 03 02 00 02                                     .....
[Raw read]: length = 2
0000: 02 28                                              .(
springCloudBus.anonymous.aZM2olJcQ2GjJmGdOiE1mA-2, READ: TLSv1.1 Alert, length = 2
springCloudBus.anonymous.aZM2olJcQ2GjJmGdOiE1mA-2, RECV TLSv1.1 ALERT:  fatal, handshake_failure
%% Invalidated:  [Session-7, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA]

I visited the server which is running in docker container. Used command: docker logs -f <containerID>. Enabled debug logs through dockerbuild.

2018-04-19 18:47:37.176 [debug] <0.685.0> Supervisor {<0.685.0>,rabbit_connection_sup} started rabbit_connection_helper_sup:start_link() at pid <0.686.0>
2018-04-19 18:47:37.176 [debug] <0.685.0> Supervisor {<0.685.0>,rabbit_connection_sup} started rabbit_reader:start_link(<0.686.0>, {acceptor,{0,0,0,0,0,0,0,0},5671}, {sslsocket,{gen_tcp,#Port<0.28884>,tls_connection,<0.465.0>},<0.684.0>}) at pid <0.687.0>
2018-04-19 18:47:37.530 [info] <0.684.0> TLS server: In state certify at ssl_connection.erl:647 generated SERVER ALERT: Fatal - Handshake Failure


2018-04-19 18:47:42.630 [debug] <0.689.0> Supervisor {<0.689.0>,rabbit_connection_sup} started rabbit_connection_helper_sup:start_link() at pid <0.690.0>
2018-04-19 18:47:42.631 [debug] <0.689.0> Supervisor {<0.689.0>,rabbit_connection_sup} started rabbit_reader:start_link(<0.690.0>, {acceptor,{0,0,0,0,0,0,0,0},5671}, {sslsocket,{gen_tcp,#Port<0.28893>,tls_connection,<0.465.0>},<0.688.0>}) at pid <0.691.0>
2018-04-19 18:47:42.989 [info] <0.688.0> TLS server: In state certify at ssl_connection.erl:647 generated SERVER ALERT: Fatal - Handshake Failure


2018-04-19 18:47:48.195 [debug] <0.693.0> Supervisor {<0.693.0>,rabbit_connection_sup} started rabbit_connection_helper_sup:start_link() at pid <0.694.0>
2018-04-19 18:47:48.196 [debug] <0.693.0> Supervisor {<0.693.0>,rabbit_connection_sup} started rabbit_reader:start_link(<0.694.0>, {acceptor,{0,0,0,0,0,0,0,0},5671}, {sslsocket,{gen_tcp,#Port<0.28902>,tls_connection,<0.465.0>},<0.692.0>}) at pid <0.695.0>
2018-04-19 18:47:48.546 [info] <0.692.0> TLS server: In state certify at ssl_connection.erl:647 generated SERVER ALERT: Fatal - Handshake Failure


2018-04-19 18:47:53.659 [debug] <0.697.0> Supervisor {<0.697.0>,rabbit_connection_sup} started rabbit_connection_helper_sup:start_link() at pid <0.698.0>
2018-04-19 18:47:53.659 [debug] <0.697.0> Supervisor {<0.697.0>,rabbit_connection_sup} started rabbit_reader:start_link(<0.698.0>, {acceptor,{0,0,0,0,0,0,0,0},5671}, {sslsocket,{gen_tcp,#Port<0.28912>,tls_connection,<0.465.0>},<0.696.0>}) at pid <0.699.0>
2018-04-19 18:47:54.015 [info] <0.696.0> TLS server: In state certify at ssl_connection.erl:647 generated SERVER ALERT: Fatal - Handshake Failure

[Luke Bakken]

Hello,

What version of RabbitMQ and Erlang are you using? It's not obvious from the name of the docker container.

Is your client code trying to use a specific cipher? Are you using any custom RabbitMQ configuration? Can you provide a set of certificates and Java code that reproduces this issue every time?

Thanks,

Luke

[Turbocoder]

How do i find the version of erlang and rabbitmq from docker image or container ?

I have attached docker and rabbitmq.conf files.

Below is the java application properties. I have not added a single line of code since I am using the spring cloud bus and amqp framework.

spring:

  rabbitmq:

    host: my.rabbitmq.com

    port: 5671

    username: guest

    password: guest

    ssl:

      enabled: true

      key-store: keystore.pfx

      key-store-password: keystorepassword

      trust-store: trust.jks

      trust-store-password: truststorepassword

  cloud:

    bus:

      enabled: true

Thank you,

Parth Patel.

[Turbocoder]

I changed docker image to rabbitmq:3.6.15-management and now logs show me the version.

 

=INFO REPORT==== 19-Apr-2018::20:53:23 ===

Starting RabbitMQ 3.6.15 on Erlang 19.2.1

This version give me more detailed failure.

=ERROR REPORT==== 19-Apr-2018::20:53:39 ===

SSL: certify: ssl_handshake.erl:1623:Fatal error: handshake failure - {bad_cert,invalid_ext_key_usage}

Anyone know what this means ?

On Thursday, April 19, 2018 at 12:07:00 PM UTC-7, Turbocoder wrote:

[Luke Bakken]

Hello -

It means that the keyUsage field in your certificates isn't valid.

Here are the values that we use in a project that will generate certificates for use in testing:

https://github.com/michaelklishin/tls-gen/blob/master/basic/openssl.cnf#L46-L54

Thanks,

Luke

[Turbocoder]

Thanks Luke. 

So i poked around the KeyUsage fields in my certificates. I see these in java logs.

[7]: ObjectId: 2.5.29.15 Criticality=false

KeyUsage [

  DigitalSignature

  Key_Encipherment

]

[8]: ObjectId: 2.5.29.15 Criticality=true

KeyUsage [

  Key_CertSign

  Crl_Sign

]

Are these wrong ?

On Thursday, April 19, 2018 at 12:07:00 PM UTC-7, Turbocoder wrote:

[Turbocoder]

Made some progress but conclusion is possible bug in erlang ssl handshake implementation for validating the extended key usage field. The rabbitmq server using erlang ssl handshake fails for the certificates I used complaining that the extended Key Usage is not valid marking it as bad certificates. But I created openssl server with same certificates as I used for rabbitmq server and had java code set up ssl connection with openssl server and it worked. It failed due to amqp time out which is expected.

openssl s_server -accept 5671 -cert crt.pem -key key.pem -CAfile ca.pem

Logs indicating the ssl handshake completion.

springCloudBus.anonymous.SXulBI9JQfKH4oTgrdpgqg-1, READ: TLSv1 Handshake, length = 48

Padded plaintext after DECRYPTION:  len = 48

0000: 14 00 00 0C 43 B9 F9 34   93 AF 6E 55 C1 BE 33 E0  ....C..4..nU..3.

0010: CD CF F7 31 D0 3A FF AE   70 2E 39 2B BB E5 B5 30  ...1.:..p.9+...0

0020: 2B 68 96 2B 0B 0B 0B 0B   0B 0B 0B 0B 0B 0B 0B 0B  +h.+............

check handshake state: finished[20]

update handshake state: finished[20]

*** Finished

verify_data:  { 67, 185, 249, 52, 147, 175, 110, 85, 193, 190, 51, 224 }

***

Can anyone suggest what can be other alternate ? Can we use less strict ssl handshake that does not perform validation on extensions of certificates ? How can I get an image of rabbitmq server with a fix or custom config solving this issue ?

On Thursday, April 19, 2018 at 12:07:00 PM UTC-7, Turbocoder wrote:

On Thursday, April 19, 2018 at 12:07:00 PM UTC-7, Turbocoder wrote:

[Michael Klishin]

RabbitMQ does not implement TLS, Erlang/OTP does. Please post this to erlang-questions

first and if there's consensus that this is not a standard/expected behavior, file a JIRA ticket.

I'm not a TLS expert but according to RFC 5280 [1], extensions designated to be "critical" must be verified

by a conforming implementation. I don't think changing the implementation is going to be an easy sell.

Changing your certificates is what I'd do since it is supposedly under your control and Erlang TLS implementation

change would be a long shot.

1. https://tools.ietf.org/html/rfc5280#section-4.2

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

[Luke Bakken]

Turbocoder -

The openssl command does only minimal validation of the keyUsage field, and can't be considered a definitive check when issues like this are seen. I know this because I spent quite a bit of time trying to figure out an issue where using a small set of configured ciphers in RabbitMQ was rejecting certificates that openssl did not. It turned out that the value in the keyUsage field is used to limit available cipher suites in the Erlang TLS implementation, but no such filtering is done in openssl.

As Michael points out, TLS is implemented in the Erlang VM. In fact, every vendor's implementation has subtle differences, and none can be considered the "standard", except maybe libnss

Your easiest solution is to generate certificates that have digitalSignature,keyEncipherment in the keyUsage field. We recommend this in our documentation now as well.

If you decide to file a bug report, the site is bugs.erlang.org. Provide a complete set of certificates and the code you're running to test them out, or a procedure that can reproduce this every time.

Thanks,

Luke

[Turbocoder]

This is the image obtained from google chrome>developer console->security->view certificate. I visited the rabbitmq default port 15671 of same container with same certs I am testing. Are the Key Usage and Ext Key Usage values not correct ? Port 15671 I know is only one way but still it has to do the work of validating the cert. doesn't it ? I am facing {bad_cert,invalid_ext_key_usage}. Does it mean invalid Key Usage or invalid Ext Key Usage or they are used based on each other so we consider Key Usage invalid ?

                                       

On Thursday, April 19, 2018 at 12:07:00 PM UTC-7, Turbocoder wrote:

[Luke Bakken]

Hello,

You also have RabbitMQ configured to require a client certificate, so the error {bad_cert,invalid_ext_key_usage} may be due to the Erlang VM validating the client certificate -

ssl_options.fail_if_no_peer_cert = true

I would be interested in the output of this command:

openssl x509 -noout -text -in client_certificate.pem

The client certificate will be available in the Java trust store you are using. In my test environment, the above command outputs these lines as part of the total output:

        X509v3 extensions:
            X509v3 Basic Constraints: 
                CA:FALSE
            X509v3 Key Usage: 
                Digital Signature, Key Encipherment
            X509v3 Extended Key Usage: 
                TLS Web Client Authentication

Thanks,
Luke

[Turbocoder]

Thank Luke and Micheal. You have been very helpful. I think I got the issue. The client cert that I send has 3 chain. Please see attached file. I have removed other fields that I am not authorized to share. But the focus is that the chain have different key usages value compared to each other. Chain 0 has valid Key Usage value and No Basic Contraints. Chain 1 and 2 does not have valid Key Usage value and Basic Contraints. Can this be issue ?

On Thursday, April 19, 2018 at 12:07:00 PM UTC-7, Turbocoder wrote:

[Michael Klishin]

There is only one chain with 3 certificates.

Certificates 1 and 2 are CA certificates. They do have key usage and basic constraints according to the attached file.

Certificate 0 is a "leaf" one, that one should have a key usage extension that allows it to authenticate either a server (if this is a server certificate chain)

or a client (if this is a client certificate chain).

E.g. use tls-gen [1]'s basic profile to generate a chain, then run `make info` and you will see that the client certificate has

            X509v3 Extended Key Usage:
                TLS Web Client Authentication

and the server one has

        X509v3 extensions:
            X509v3 Basic Constraints:
                CA:FALSE
            X509v3 Key Usage:
                Digital Signature, Key Encipherment
            X509v3 Extended Key Usage:

                TLS Web Server Authentication

1. https://github.com/michaelklishin/tls-gen

...

[Message clipped]  

[Michael Klishin]

Sent an incomplete draft after a network hiccup.

Basically it looks like you are trying to use a certificate with extended key usage = TLS Web Server Authentication

on the client end, which is not supposed to be valid.

On Sat, Apr 21, 2018 at 3:00 AM, Turbocoder <parth....@gmail.com> wrote:

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.