RabbitMQ SSL Setup Issues

780 views
Skip to first unread message

Matthias Thubauville

unread,
Aug 14, 2016, 10:08:23 AM8/14/16
to rabbitmq-users
Hi there,

For the past days I've been trying to setup RabbitMQ and to secure traffic with TLS (for now only the traffic: Broker <-> Client is important - sooner or later it would be interesting to also have broker <-> broker communication secured).

What I did so far:

0) I installed RabbitMQ and Erlang (I tried the Erlang which is shipped by default and the latest Erlang) on Ubuntu 14.04.5
# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 14.04.5 LTS
Release:        14.04
Codename:       trusty


# dpkg -l | grep rabbitmq
ii  rabbitmq-server                  3.6.5-1                             all          Multi-protocol messaging broker


# dpkg -l | grep erlang
[...]
ii  erlang-base                      1:19.0-1                            amd64        Erlang/OTP virtual machine and base applications
[...]


1) I followed https://www.rabbitmq.com/ssl.html to generate the the CA and certificates. (I also tried https://github.com/Berico-Technologies/CMF-AMQP-Configuration)

2) I configured the rabbitmq.config file
# cat /etc/rabbitmq/rabbitmq.config
[ { rabbit, [
        {loopback_users, [ ] },
        { tcp_listeners, [ 5672 ] },
        { ssl_listeners, [ 5671 ] },
        { ssl_options, [
                { cacertfile, "/etc/rabbitmq/ssl/ca/cacert.pem" },
                { certfile, "/etc/rabbitmq/ssl/server/rabbit-00.key.pem" },
                { keyfile, "/etc/rabbitmq/ssl/server/rabbit-00.cert.pem" },
                { fail_if_no_peer_cert, false },
                { verify, verify_none }
        ] },
        { default_pass, <<"user">> },
        { default_user, <<"password">> },
        { log_levels, [
                {connection, debug}
        ] }
] } ].


3) I checked https://www.rabbitmq.com/troubleshooting-ssl.html. Here the step Attempt SSL connection to broker fails:
# openssl s_client -msg -ssl3 -state -showcerts -connect localhost:5671 -cert client/rabbit-00.cert.pem -key client/rabbit-00.key.pem   -CAfile ca/cacert.pem
CONNECTED(00000003)
SSL_connect:before/connect initialization
>>> SSL 3.0 Handshake [length 007b], ClientHello
    01 00 00 77 03 00 d7 a3 31 a5 5f c1 1e 8c 2d df
    0b 17 dc 1a b3 83 8a c7 b7 e9 63 7e 3a 74 81 0e
    03 a5 de 13 b5 17 00 00 50 c0 14 c0 0a 00 39 00
    38 00 88 00 87 c0 0f c0 05 00 35 00 84 c0 12 c0
    08 00 16 00 13 c0 0d c0 03 00 0a c0 13 c0 09 00
    33 00 32 00 9a 00 99 00 45 00 44 c0 0e c0 04 00
    2f 00 96 00 41 c0 11 c0 07 c0 0c c0 02 00 05 00
    04 00 15 00 12 00 09 00 ff 01 00
SSL_connect:SSLv3 write client hello A
There is no more output. The previous tests perform OK.

Logs don't show anything. (Literally nothing. The startup section of the log is without warnings or errors).

Here is some more debug information.
1) TCPdump on the port while running the openssl command from above. (and hitting CTRL+C)
# tcpdump -i lo port 5671
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on lo, link-type EN10MB (Ethernet), capture size 65535 bytes
13:52:52.553215 IP rabbit-00.vagrant.47629 > rabbit-00.vagrant.5671: Flags [S], seq 464237873, win 43690, options [mss 65495,sackOK,TS val 1605102 ecr 0,nop,wscale 7], length 0
13:52:52.553224 IP rabbit-00.vagrant.5671 > rabbit-00.vagrant.47629: Flags [S.], seq 1548985028, ack 464237874, win 43690, options [mss 65495,sackOK,TS val 1605102 ecr 1605102,nop,wscale 7], length 0
13:52:52.553232 IP rabbit-00.vagrant.47629 > rabbit-00.vagrant.5671: Flags [.], ack 1, win 342, options [nop,nop,TS val 1605102 ecr 1605102], length 0
13:52:52.554674 IP rabbit-00.vagrant.47629 > rabbit-00.vagrant.5671: Flags [P.], seq 1:129, ack 1, win 342, options [nop,nop,TS val 1605102 ecr 1605102], length 128
13:52:52.554688 IP rabbit-00.vagrant.5671 > rabbit-00.vagrant.47629: Flags [.], ack 129, win 350, options [nop,nop,TS val 1605102 ecr 1605102], length 0




13:52:55.259631 IP rabbit-00.vagrant.47629 > rabbit-00.vagrant.5671: Flags [F.], seq 129, ack 1, win 342, options [nop,nop,TS val 1605779 ecr 1605102], length 0
13:52:55.298958 IP rabbit-00.vagrant.5671 > rabbit-00.vagrant.47629: Flags [.], ack 130, win 350, options [nop,nop,TS val 1605789 ecr 1605779], length 0

2) service rabbitmq-server status (I tried the rabbitmq_auth_mechanism_ssl plugin out of desperation, but it doesn't help.
Status of node 'rabbit@rabbit-00' ...
[{pid,1504},
 {running_applications,
     [{rabbitmq_auth_mechanism_ssl,
          "RabbitMQ SSL authentication (SASL EXTERNAL)","3.6.5"},
      {rabbitmq_management,"RabbitMQ Management Console","3.6.5"},
      {rabbitmq_management_agent,"RabbitMQ Management Agent","3.6.5"},
      {rabbit,"RabbitMQ","3.6.5"},
      {os_mon,"CPO  CXC 138 46","2.4.1"},
      {ranch,"Socket acceptor pool for TCP protocols.","1.2.1"},
      {amqp_client,"RabbitMQ AMQP Client","3.6.5"},
      {rabbit_common,[],"3.6.5"},
      {rabbitmq_web_dispatch,"RabbitMQ Web Dispatcher","3.6.5"},
      {webmachine,"webmachine","1.10.3"},
      {mochiweb,"MochiMedia Web Server","2.13.1"},
      {ssl,"Erlang/OTP SSL application","8.0"},
      {public_key,"Public key infrastructure","1.2"},
      {crypto,"CRYPTO","3.7"},
      {inets,"INETS  CXC 138 49","6.3"},
      {xmerl,"XML parser","1.3.11"},
      {syntax_tools,"Syntax tools","2.0"},
      {compiler,"ERTS  CXC 138 10","7.0"},
      {mnesia,"MNESIA  CXC 138 12","4.14"},
      {asn1,"The Erlang ASN1 compiler version 4.0.3","4.0.3"},
      {sasl,"SASL  CXC 138 11","3.0"},
      {stdlib,"ERTS  CXC 138 10","3.0"},
      {kernel,"ERTS  CXC 138 10","5.0"}]},
 {os,{unix,linux}},
 {erlang_version,
     "Erlang/OTP 19 [erts-8.0] [source] [64-bit] [async-threads:64] [kernel-poll:true]\n"},
[...]
{alarms,[]},
 {listeners,[{clustering,25672,"::"},{amqp,5672,"::"},{'amqp/ssl',5671,"::"}]},
 {vm_memory_high_watermark,0.4},
 {vm_memory_limit,416887603},
 {disk_free_limit,50000000},
 {disk_free,38356254720},
 {file_descriptors,
     [{total_limit,924},{total_used,2},{sockets_limit,829},{sockets_used,0}]},
 {processes,[{limit,1048576},{used,238}]},
 {run_queue,0},
 {uptime,2907},
 {kernel,{net_ticktime,60}}]

3) Some general stuff:
# hostname
rabbit-00

# hostname -f
rabbit-00.vagrant

root@rabbit-00:~# su rabbitmq -s /bin/bash
rabbitmq@rabbit-00:/root$ cat /etc/rabbitmq/ssl/server/*
-----BEGIN CERTIFICATE-----
[...]
-----END CERTIFICATE-----

-----BEGIN RSA PRIVATE KEY-----

[...]
-----END RSA PRIVATE KEY-----
rabbitmq@rabbit-00:/root$ cat /etc/rabbitmq/ssl/ca/*
-----BEGIN CERTIFICATE-----
[...]
-----END CERTIFICATE-----


To sum up:
* I can connect via 5672/TCP and send messages
* I cannot connect via 5671/TCP and don't know why.

Any help is appreciated. Thanks a lot for now.
Matthias

PS. I tried the same setup using the official docker container, however if i use the container, I can't even use rabbitmqctl any more - IDK why, but I guess that's a different issue.

Michael Klishin

unread,
Aug 14, 2016, 11:08:26 AM8/14/16
to rabbitm...@googlegroups.com
RabbitMQ will only log connections that sent data, which may or may not be the case with `openssl s_client`.
This is so that TCP load balancer/proxy health checks do not pollute the logs.

You posted no information of any kind beyond "I cannot connect via 5671", what does your code look like,
what does the error say (if any), what do you observe and expect to happen?

Lastly, I'd like to clarify that there is no official RabbitMQ Docker image: there is one maintained by Docker, Inc but none from
Pivotal.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.



--
MK

Staff Software Engineer, Pivotal/RabbitMQ

Matthias Thubauville

unread,
Aug 14, 2016, 3:35:44 PM8/14/16
to rabbitmq-users
Dear Michael,

thanks a lot for the quick reply. I didn't know that the docker image was not by Pivotal - yet another reason to build our own images sooner or later.

According to https://www.rabbitmq.com/troubleshooting-ssl.html the openssl s_client should give a log message. It would be great to state it on the page if this is not the case. (Note that I also didn't receive an output similar to the case where port 8443 was used).

Never the less: I'm trying to use either of the following implementations. (Please excuse any bad coding style - it's supposed to work as a simple proof of concept and I'm not really experienced in either of the two languages.)

The gist also includes the error message. I've tried to contact the server via localhost, its short and its full name. All tree give the same error message. (I also added the output of what's happening if I restart the server while trying to contact it.)

Again I tried various hostnames.

Both scripts work fine if I use the 5672 port without TLS and neither of the two cause a log entry in /var/log/rabbitmq/* (if used via 5671).

If I had to guess I'd say that either packages are lost (therefore the tcpdump) or the application is not running/not answering.

Please let me know if there is anything more I can provide. I know debugging via mailing lists is hard ;)

Thanks!
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Michael Klishin

unread,
Aug 15, 2016, 10:53:45 AM8/15/16
to rabbitmq-users
I certainly expect `openssl s_client` to leave a trace in the logs, even if it's a handshake timeout (since it just opens a TCP connection
and performs a TLS upgrade, it's not a RabbitMQ client library).

Can you do a Wireshark traffic capture? [1]. ECONNREFUSED is fairly unambiguous and means that TCP connection to port 5671 failed.

I suspect that the fact that you run in a VM and use contains could have an effect here (as in, your connections go to a different host/process
than you expect).

Michael Klishin

unread,
Aug 15, 2016, 10:54:02 AM8/15/16
to rabbitmq-users

      {rabbit,"RabbitMQ","3.6.5"<span s

Matthias Thubauville

unread,
Aug 15, 2016, 2:08:33 PM8/15/16
to rabbitmq-users
Hi Michael,


I'm not sure if the the ssh decryption worked, but it looks pretty much the same as my tcpdump: There is no more communication after the Client Hello. If it helps I can also upload the original dump files somewhere.

Please note: The ECONNREFUSED you quote above only occurs if I restart the service while trying to connect. If do not restart the service I only get a timeout.

All tests I perform give the same results, no matter if I perform them from on the machine running RabbitMQ or from my outside Laptop.

I can connect to the port via telnet (on the host running RabbitMQ), however port 5671 does not terminate the connection, while 5672 does.

# telnet localhost 5672
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.








AMQP    Connection closed by foreign host.

# telnet localhost 5671
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.














^]


telnet> Connection closed.

Maybe something is not started correctly:

# netstat -tulpen | grep 567
tcp        0      0 0.0.0.0:15672           0.0.0.0:*               LISTEN      109        17790       5317/beam
tcp        0      0 0.0.0.0:25672           0.0.0.0:*               LISTEN      109        17695       5317/beam
tcp6       0      0 :::5671                 :::*                    LISTEN      109        17785       5317/beam
tcp6       0      0 :::5672                 :::*                    LISTEN      109        17779       5317/beam

# ps aux | grep beam
rabbitmq  5317  0.1  5.8 1672132 59368 ?       Sl   17:15   0:05 /usr/lib/erlang/erts-8.0/bin/beam -W w -A 64 -P 1048576 -t 5000000 -stbt db -K true -B i -- -root /usr/lib/erlang -progname erl -- -home /var/lib/rabbitmq -- -pa /usr/lib/rabbitmq/lib/rabbitmq_server-3.6.5/ebin -noshell -noinput -s rabbit boot -sname rabbit@rabbit-00 -boot start_sasl -config /etc/rabbitmq/rabbitmq -kernel inet_default_connect_options [{nodelay,true}] -sasl errlog_type error -sasl sasl_error_logger false -rabbit error_logger {file,"/var/log/rabbitmq/rab...@rabbit-00.log"} -rabbit sasl_error_logger {file,"/var/log/rabbitmq/rab...@rabbit-00-sasl.log"} -rabbit enabled_plugins_file "/etc/rabbitmq/enabled_plugins" -rabbit plugins_dir "/usr/lib/rabbitmq/lib/rabbitmq_server-3.6.5/plugins" -rabbit plugins_expand_dir "/var/lib/rabbitmq/mnesia/rabbit@rabbit-00-plugins-expand" -os_mon start_cpu_sup false -os_mon start_disksup false -os_mon start_memsup false -mnesia dir "/var/lib/rabbitmq/mnesia/rabbit@rabbit-00" -kernel inet_dist_listen_min 25672 -kernel inet_dist_listen_max 25672


Matthias Thubauville

unread,
Aug 17, 2016, 8:45:08 AM8/17/16
to rabbitmq-users
Hi Michael,

for me adding the line
domainComponent = optional
in the
[ testca_policy ]
section of the openssl.cnf file fixed the problem. 

It seems I'm not the only one (at least some friends remembered that they had the same problem).

Maybe you can find the time and update the docs. I can also send a pull request if this is easier for you.

Thanks for your help and time

PS. It's quite funny that you refuse to see any misstakes on your side: Your repo (https://github.com/michaelklishin/tls-gen/) pointed me in the right direction, as the openssl.cnf contains the missing line).

Michael Klishin

unread,
Aug 17, 2016, 8:47:14 AM8/17/16
to rabbitm...@googlegroups.com, Matthias Thubauville
Feel free to submit a PR against the `live` branch in https://github.com/rabbitmq/rabbitmq-website.

Thank you very much! 
> --
> You received this message because you are subscribed to the Google Groups "rabbitmq-users"
> group.
> To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
> To post to this group, send an email to rabbitm...@googlegroups.com.
Reply all
Reply to author
Forward
0 new messages