RabbitMQ SSL - server not responding to ClientHello

[Alexander B]

Hi,

I'm trying to set up RabbitMQ to work with SSL/TLS. I generated certificates using tls-gen as described in the RabbitMQ docs.

Environment: Centos7, RabbitMQ 3.6.6, rabbitmqctl status output:

[{pid,32233},
 {running_applications,
     [{rabbitmq_management,"RabbitMQ Management Console","3.6.6"},
      {rabbitmq_web_dispatch,"RabbitMQ Web Dispatcher","3.6.6"},
      {rabbitmq_management_agent,"RabbitMQ Management Agent","3.6.6"},
      {rabbit,"RabbitMQ","3.6.6"},
      {mnesia,"MNESIA  CXC 138 12","4.14.2"},
      {os_mon,"CPO  CXC 138 46","2.4.1"},
      {webmachine,"webmachine","1.10.3"},
      {mochiweb,"MochiMedia Web Server","2.13.1"},
      {compiler,"ERTS  CXC 138 10","7.0.3"},
      {amqp_client,"RabbitMQ AMQP Client","3.6.6"},
      {rabbit_common,[],"3.6.6"},
      {xmerl,"XML parser","1.3.12"},
      {ssl,"Erlang/OTP SSL application","8.1"},
      {public_key,"Public key infrastructure","1.3"},
      {asn1,"The Erlang ASN1 compiler version 4.0.4","4.0.4"},
      {syntax_tools,"Syntax tools","2.1.1"},
      {inets,"INETS  CXC 138 49","6.3.4"},
      {crypto,"CRYPTO","3.7.2"},
      {ranch,"Socket acceptor pool for TCP protocols.","1.2.1"},
      {sasl,"SASL  CXC 138 11","3.0.2"},
      {stdlib,"ERTS  CXC 138 10","3.2"},
      {kernel,"ERTS  CXC 138 10","5.1.1"}]},
 {os,{unix,linux}},
 {erlang_version,
     "Erlang/OTP 19 [erts-8.2] [source] [64-bit] [async-threads:64] [hipe] [kernel-poll:true]\n"},
 {memory,
     [{total,50205176},
      {connection_readers,0},
      {connection_writers,0},
      {connection_channels,0},
      {connection_other,2688},
      {queue_procs,19352},
      {queue_slave_procs,0},
      {plugins,1205752},
      {other_proc,12841184},
      {mnesia,67272},
      {mgmt_db,1062336},
      {msg_index,49920},
      {other_ets,1429160},
      {binary,76712},
      {code,24821242},
      {atom,1033401},
      {other_system,7596157}]},
 {alarms,[]},
 {listeners,[{clustering,25672,"::"},{amqp,5672,"::"},{'amqp/ssl',5671,"::"}]},
 {vm_memory_high_watermark,0.4},
 {vm_memory_limit,205093273},
 {disk_free_limit,50000000},
 {disk_free,9437691904},
 {file_descriptors,
     [{total_limit,924},{total_used,2},{sockets_limit,829},{sockets_used,0}]},
 {processes,[{limit,1048576},{used,240}]},
 {run_queue,0},
 {uptime,2574},
 {kernel,{net_ticktime,60}}]

rabbitmq.config:

[
  {rabbit, [
     {ssl_listeners, [5671]},
     {ssl_options, [{cacertfile,"/etc/rabbitmq/certs/ca_certificate.pem"},
                    {certfile,"/etc/rabbitmq/certs/server_certificate.pem"},
                    {keyfile,"/etc/rabbitmq/certs/server_key.pem"},
                    {password,  "bunnies"},
                    {verify,verify_none},
                    {fail_if_no_peer_cert,false}]}
   ]}
].

I tested certificates using openssl s_server + s_client and verified that client can connect to the server.

when I try 

 openssl s_client -connect rabbit-tls:5671 -cert client_certificate.pem -key client_key.pem -CAfile ca_certificate.pem -debug -msg

I get:

CONNECTED(00000003)

write to 0x239bd70 [0x239bdf0] (247 bytes => 247 (0xF7))

0000 - 16 03 01 00 f2 01 00 00-ee 03 03 58 7d 31 c2 b2   ...........X}1..

0010 - a2 df 1b ab 1c f5 35 59-8d 45 18 e4 ef f9 5a 80   ......5Y.E....Z.

0020 - c6 cc 54 8a 19 7d 48 7f-d7 0e 5f 00 00 84 c0 30   ..T..}H..._....0

0030 - c0 2c c0 28 c0 24 c0 14-c0 0a 00 a3 00 9f 00 6b   .,.(.$.........k

0040 - 00 6a 00 39 00 38 00 88-00 87 c0 32 c0 2e c0 2a   .j.9.8.....2...*

0050 - c0 26 c0 0f c0 05 00 9d-00 3d 00 35 00 84 c0 2f   .&.......=.5.../

0060 - c0 2b c0 27 c0 23 c0 13-c0 09 00 a2 00 9e 00 67   .+.'.#.........g

0070 - 00 40 00 33 00 32 c0 12-c0 08 00 9a 00 99 00 45   .@.3.2.........E

0080 - 00 44 00 16 00 13 c0 31-c0 2d c0 29 c0 25 c0 0e   .D.....1.-.).%..

0090 - c0 04 c0 0d c0 03 00 9c-00 3c 00 2f 00 96 00 41   .........<./...A

00a0 - 00 0a 00 07 c0 11 c0 07-c0 0c c0 02 00 05 00 04   ................

00b0 - 00 ff 01 00 00 41 00 0b-00 04 03 00 01 02 00 0a   .....A..........

00c0 - 00 08 00 06 00 19 00 18-00 17 00 23 00 00 00 0d   ...........#....

00d0 - 00 20 00 1e 06 01 06 02-06 03 05 01 05 02 05 03   . ..............

00e0 - 04 01 04 02 04 03 03 01-03 02 03 03 02 01 02 02   ................

00f0 - 02 03 00 0f 00 01 01                              .......

>>> TLS 1.2 Handshake [length 00f2], ClientHello

    01 00 00 ee 03 03 58 7d 31 c2 b2 a2 df 1b ab 1c

    f5 35 59 8d 45 18 e4 ef f9 5a 80 c6 cc 54 8a 19

    7d 48 7f d7 0e 5f 00 00 84 c0 30 c0 2c c0 28 c0

    24 c0 14 c0 0a 00 a3 00 9f 00 6b 00 6a 00 39 00

    38 00 88 00 87 c0 32 c0 2e c0 2a c0 26 c0 0f c0

    05 00 9d 00 3d 00 35 00 84 c0 2f c0 2b c0 27 c0

    23 c0 13 c0 09 00 a2 00 9e 00 67 00 40 00 33 00

    32 c0 12 c0 08 00 9a 00 99 00 45 00 44 00 16 00

    13 c0 31 c0 2d c0 29 c0 25 c0 0e c0 04 c0 0d c0

    03 00 9c 00 3c 00 2f 00 96 00 41 00 0a 00 07 c0

    11 c0 07 c0 0c c0 02 00 05 00 04 00 ff 01 00 00

    41 00 0b 00 04 03 00 01 02 00 0a 00 08 00 06 00

    19 00 18 00 17 00 23 00 00 00 0d 00 20 00 1e 06

    01 06 02 06 03 05 01 05 02 05 03 04 01 04 02 04

    03 03 01 03 02 03 03 02 01 02 02 02 03 00 0f 00

    01 01

And nothing else, until connection times out after some time. 

From Wireshark I can see the TCP handshake and the this ClientHello message sent, and ACK received. And then nothing.

I also checked that certificate path in rabbitmq.config is correct. But not sure how to check if rabbit (Erlang) is actually happy with those certificates or what else to try. 

Also tried different values for ssl_options to verify peer and fail on invalid certificate. Same result. Please help.

[Michael Klishin]

It's not clear whether you tested an `openssl s_client` connection to RabbitMQ or an `openssl s_client` connection

to `openssl s_server`.

How about inspecting server logs and trying to connect to RabbitMQ

with a RabbitMQ client?

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

[Alexander B]

Hi Michael, thank you for the response.

I tried both: openssl s_client connection to RabbitMQ Server, and .NET client connection to RabbitMQ Server. Results are the same, no response to ClientHello.

With default logging configuration, RabbitMQ logs contain nothing about those connection attempts. Should I try and set log level to debug?

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Alexander B]

Set RabbitMQ log_levels:

[

  {rabbit, [

     {log_levels, [{connection, debug},{channel, debug}]},

     {ssl_listeners, [5671]},

     {ssl_options, [{cacertfile,"/etc/rabbitmq/certs/ca_certificate.pem"},

                    {certfile,"/etc/rabbitmq/certs/server_certificate.pem"},

                    {keyfile,"/etc/rabbitmq/certs/server_key.pem"},

                    {password,  "bunnies"},

                    {verify,verify_none},

                    {fail_if_no_peer_cert,false}]}

   ]}

].

Still nothing get logged when trying to connect via openssl or .NET client. Issue remains, no answer to ClientHello.

[Michael Klishin]

If RabbitMQ does not log any connections, it means that either it doesn't even receive one

(from the runtime) or that there is no data sent after it is opened.

Try Erlang 19.0.x or 19.1.x or 18.2.x and a different TLS version for comparison. RabbitMQ does implement TLS

handshake or state machine.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[jan.go...@enervalis.com]

Yes ! It doesn't hang when Using Erlang 18.3. Although I'm not sure whether th

$ openssl s_client -pass "pass:**************" -connect localhost:5671 -cert ../certs/client.cert.pem -key ../private/client.key.pem -CAfile ./ca.cert.pem -state -nbio
CONNECTED(00000003)
Turned on non blocking io
SSL_connect:before SSL initialization
SSL_connect:SSLv3/TLS write client hello
SSL_connect:error in SSLv3/TLS write client hello
write R BLOCK
SSL_connect:error in SSLv3/TLS write client hello
read:errno=104
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 0 bytes and written 176 bytes
Verification: OK
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : 0000
    Session-ID:
    Session-ID-ctx:
    Master-Key:
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1484641717
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
    Extended master secret: no
---

[Michael Klishin]

Please report this to the OTP team: https://bugs.erlang.org/secure/Dashboard.jspa.

Thank you.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[jan.go...@enervalis.com]

However, now I'm getting the old problem again. There is an unspecified error with the key file ?

=CRASH REPORT==== 17-Jan-2017::10:23:30 ===
  crasher:
    initial call: rabbit_reader:init/4
    pid: <0.265.0>
    registered_name: []
    exception error: {keyfile,function_clause}
      in function  ranch_ssl:accept_ack/2 (src/ranch_ssl.erl, line 131)
      in call from rabbit_net:accept_ack/2 (src/rabbit_net.erl, line 261)
      in call from rabbit_reader:init/4 (src/rabbit_reader.erl, line 230)
    ancestors: [<0.263.0>,<0.250.0>,<0.249.0>,<0.248.0>,rabbit_sup,
                  <0.128.0>]
    messages: []
    links: [<0.263.0>]
    dictionary: []
    trap_exit: false
    status: running
    heap_size: 610
    stack_size: 27
    reductions: 142
  neighbours:


=SUPERVISOR REPORT==== 17-Jan-2017::10:23:30 ===
     Supervisor: {<0.263.0>,rabbit_connection_sup}
     Context:    child_terminated
     Reason:     {keyfile,function_clause}
     Offender:   [{pid,<0.265.0>},
                  {name,reader},
                  {mfargs,
                      {rabbit_reader,start_link,
                          [<0.264.0>,
                           {acceptor,{0,0,0,0},5671},
                           {sslsocket,
                               {gen_tcp,#Port<0.12049>,tls_connection,
                                   <0.252.0>},
                               <0.262.0>}]}},
                  {restart_type,intrinsic},
                  {shutdown,30000},
                  {child_type,worker}]




=SUPERVISOR REPORT==== 17-Jan-2017::10:23:30 ===
     Supervisor: {<0.263.0>,rabbit_connection_sup}
     Context:    shutdown
     Reason:     reached_max_restart_intensity
     Offender:   [{pid,<0.265.0>},
                  {name,reader},
                  {mfargs,
                      {rabbit_reader,start_link,
                          [<0.264.0>,
                           {acceptor,{0,0,0,0},5671},
                           {sslsocket,
                               {gen_tcp,#Port<0.12049>,tls_connection,
                                   <0.252.0>},
                               <0.262.0>}]}},
                  {restart_type,intrinsic},
                  {shutdown,30000},
                  {child_type,worker}]

[Michael Klishin]

It's not "unspecified"m it is "function_clause", which means something in the parser fails to load your key file.

The original issue seems to be addressed in https://github.com/erlang/otp/commit/1364c7308e17d43d1a2244e3f2bf11cfec3789ef,

at least it looks highly relevant.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[jan.go...@enervalis.com]

I could make it work in the end ... with keys without a password. :-(

Even though the key's password is specified in the config file, it won't work. 

Perhaps the password is encoded in some way ? 

[Michael Klishin]

No, it's a clear text value and we know of people who use keys with passwords.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.

[Alexander B]

Ok, I got it sorted. Placing it here so it may be of help to others.

So, my problem was: Centos7 + Erlang 19.0.4 + RabbitMQ 3.6.6 - rabbit is not responding to ClientHello (tried TLS 1.1, TLS 1.2, SSL3). Nothing was logged.

I followed suggestion to try erlang 18. I installed erlang Erlang 18.1 release from Rabbit website and tried coonecting .net client.

I received error: Errno=104 Connection reset by peer. 

Luckily, this time file rab...@rabbit-sasl.log contained an entry for the connection!

Entry looked like this:

=SUPERVISOR REPORT==== 17-Jan-2017::03:50:23 ===
     Supervisor: {<0.468.0>,rabbit_connection_sup}
     Context:    child_terminated
     Reason:     {options,{cacertfile,"/etc/rabbitmq/certs/ca_certificate.pem",
                                      {error,eacces}}}
     Offender:   [{pid,<0.470.0>},
                  {name,reader},
                  {mfargs,
                      {rabbit_reader,start_link,
                          [<0.469.0>,
                           {acceptor,{0,0,0,0,0,0,0,0},5671},
                           {sslsocket,
                               {gen_tcp,#Port<0.20772>,tls_connection,
                                   <0.275.0>},
                               <0.467.0>}]}},

                  {restart_type,intrinsic},
                  {shutdown,30000},
                  {child_type,worker}]

Then the problem was clear - incorrect permissions on CA certificate file.

But I made sure rabbitmq can read those files by setting correct permissions on the file and path. Further investigation showed that the problem was due to SELinux security framework being enabled and blocking Rabbit from reading certificate files.

To disable SELinux I modified /etc/sysconfig/selinux  config file as descirbed here . Then I have to reboot the machine. After that TLS worked. Then I installed erlang 19.0.4 from rabbitmq website, and it continued to work.

[Michael Klishin]

Thank you for reporting back.

SELinux is a common reason for obscure issues, not just with RabbitMQ :)

[Alexander B]

Hi Michael, thank you for help and for tls-gen, its very handy!