Rabbitmq 3.7 - Issues with Let's Encrypt, Config File for Cluster...

564 views
Skip to first unread message

Casey Havenor

unread,
Jun 17, 2019, 5:15:47 PM6/17/19
to rabbitmq-users
Any help is greatly appreciated.

Setup:
  • Letsencrypt for a subdomain mq1.ob.com and mq2.ob.com. 
  • Rabbitmq 3.7 
  • Erlang 22 
  • All Standard ports are fine.

Goals:
  • Use OLD config file format as it seemed there were fewer issues. 
  • No non-secure ciphers should be used. 
  • All nodes connecting via cookie but secured with TLS 1.2
  • AMPQ data secured with TLS 1.2
  • Serve web manager portal over TLS 1.2
  • I do not want to have to use HAproxy as I want to keep things as simple as possible. 

Config File:
Without the config, I can connect to the mq1 web portal without issues.  With config, I can not connect but on node 2 I can see that node 1 has joined the cluster. 

[
 {ssl,    [
           {versions,           ['tlsv1.2']},
           {secure_renegotiate, true}
          ]},
 {rabbit, [
           {ssl_listeners, [5671]},
           {ssl_options, [{cacertfile, "/etc/letsencrypt/live/mq1.ob.com/fullchain.pem"},
                          {certfile,   "/etc/letsencrypt/live/mq1.ob.com/cert.pem"},
                          {keyfile,    "/etc/letsencrypt/live/mq1.ob.com/privkey.pem"},
                          {versions, ['tlsv1.2']},
                          {ciphers,  [
                                       {ecdhe_ecdsa,aes_256_gcm,aead,sha384},
                                       {ecdhe_rsa,aes_256_gcm,aead,sha384},
                                       {ecdh_ecdsa,aes_256_gcm,aead,sha384},
                                       {ecdh_rsa,aes_256_gcm,aead,sha384},
                                       {dhe_rsa,aes_256_gcm,aead,sha384},
                                       {dhe_dss,aes_256_gcm,aead,sha384},
                                       {ecdhe_ecdsa,aes_128_gcm,aead,sha256},
                                       {ecdhe_rsa,aes_128_gcm,aead,sha256},
                                       {ecdh_ecdsa,aes_128_gcm,aead,sha256},
                                       {ecdh_rsa,aes_128_gcm,aead,sha256},
                                       {dhe_rsa,aes_128_gcm,aead,sha256},
                                       {dhe_dss,aes_128_gcm,aead,sha256}
                                     ]},
                          {honor_cipher_order,   true},
                          {honor_ecc_order,      true},
                          {client_renegotiation, false},
                          {secure_renegotiate,   true},
                          {verify,               verify_peer},
                          {fail_if_no_peer_cert, false}]}
          ]},
{rabbitmq_management,
    [
     {listener, [{port, 15672},
                 {ssl, true},
                 {ssl_opts, [{cacertfile, "/etc/letsencrypt/live/mq1.ob.com/fullchain.pem"},
                                {certfile, "/etc/letsencrypt/live/mq1.ob.com/cert.pem"},
                               {keyfile, "/etc/letsencrypt/live/mq1.ob.com/privkey.pem"},
                             {verify,               verify_none},
                             {fail_if_no_peer_cert, false},

                             {client_renegotiation, false},
                             {secure_renegotiate,   true},
                             {honor_ecc_order,      true},
                             {honor_cipher_order,   true},

                             {versions,[ 'tlsv1.2']},
                             {ciphers, ["ECDHE-ECDSA-AES256-GCM-SHA384",
                                        "ECDHE-RSA-AES256-GCM-SHA384",
                                        "ECDHE-ECDSA-AES256-SHA384",
                                        "ECDHE-RSA-AES256-SHA384",
                                        "ECDH-ECDSA-AES256-GCM-SHA384",
                                        "ECDH-RSA-AES256-GCM-SHA384",
                                        "ECDH-ECDSA-AES256-SHA384",
                                        "ECDH-RSA-AES256-SHA384",
                                        "DHE-RSA-AES256-GCM-SHA384"
                                        ]}
                             ]}
                ]}
    ]}
].

Outcome:

2019-06-17 20:52:00.565 [info] <0.385.0> rabbit on node rabbit@mq2 up
2019-06-17 20:52:00.569 [warning] <0.547.0> Setting Ranch options together with socket options is deprecated. Please use the new map syntax that allows specifying socket options separately from other options.
2019-06-17 20:52:00.570 [info] <0.561.0> started TCP listener on [::]:5672
2019-06-17 20:52:00.572 [warning] <0.562.0> Setting Ranch options together with socket options is deprecated. Please use the new map syntax that allows specifying socket options separately from other options.
2019-06-17 20:52:00.576 [info] <0.577.0> started TLS (SSL) listener on [::]:5671
2019-06-17 20:52:00.579 [info] <0.280.0> Running boot step direct_client defined by app rabbit
2019-06-17 20:52:01.075 [info] <0.628.0> Management plugin: HTTPS listener started on port 15672
2019-06-17 20:52:01.076 [info] <0.735.0> Statistics database started.
2019-06-17 20:52:01.150 [notice] <0.106.0> Changed loghwm of /var/log/rabbitmq/rab...@mq1.log to 50
2019-06-17 20:52:01.559 [info] <0.8.0> Server startup complete; 3 plugins started.
 * rabbitmq_management
 * rabbitmq_management_agent
 * rabbitmq_web_dispatch



2019-06-17 05:36:28.263 [info] <0.307.0> Running boot step notify_cluster defined by app rabbit
2019-06-17 05:36:28.263 [info] <0.307.0> Running boot step networking defined by app rabbit
2019-06-17 05:36:28.334 [error] <0.306.0> CRASH REPORT Process <0.306.0> with 0 neighbours exited with reason: no match of right hand value noport in rabbit_networking:record_distribution_listener/0 line 305 in application_master:init/4 line 138
2019-06-17 05:36:28.335 [info] <0.43.0> Application rabbit exited with reason: no match of right hand value noport in rabbit_networking:record_distribution_listener/0 line 305
2019-06-17 05:36:28.337 [info] <0.365.0> Closing all connections in vhost '/' on node 'rabbit@mq2' because the vhost is stopping
2019-06-17 05:36:28.338 [info] <0.384.0> Stopping message store for directory '/var/lib/rabbitmq/mnesia/rabbit@mq2/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent'
2019-06-17 05:36:28.353 [info] <0.384.0> Message store for directory '/var/lib/rabbitmq/mnesia/rabbit@mq2/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent' is stopped
2019-06-17 05:36:28.353 [info] <0.381.0> Stopping message store for directory '/var/lib/rabbitmq/mnesia/rabbit@mq2/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_transient'
2019-06-17 05:36:28.367 [info] <0.381.0> Message store for directory '/var/lib/rabbitmq/mnesia/rabbit@mq2/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_transient' is stopped

This page isn’t working

mq1.ob.com didn’t send any data.

ERR_EMPTY_RESPONSE


Need anything else?  Let me know.   

Thanks! 

Casey 




Luke Bakken

unread,
Jun 17, 2019, 6:46:46 PM6/17/19
to rabbitmq-users
Hi Casey,

Just to be precise, what's the exact version of RabbitMQ you're using. 3.7.15 is the most recent, for instance.

Did you create an /etc/rabbitmq/rabbitmq-env.conf file or did you set any custom environment variables for RabbitMQ?

You said this in your description:

With config, I can not connect but on node 2 I can see that node 1 has joined the cluster.

You don't specifically say you're running a cluster of RabbitMQ nodes but the above statement suggests you have at least a two node cluster. Is that correct? If you are using a cluster, did you at one point start all cluster nodes without issue without using these TLS settings?

2019-06-17 05:36:28.334 [error] <0.306.0> CRASH REPORT Process <0.306.0> with 0 neighbours exited with reason: no match of right hand value noport in rabbit_networking:record_distribution_listener/0 line 305 in application_master:init/4 line 138

That error message means that one RabbitMQ node attempted to connect to another to get it's node name, but the request failed. Most likely this is due to TCP port 4369 being blocked between nodes in your cluster, or the other node isn't running.

Thanks -
Luke

Casey Havenor

unread,
Jun 18, 2019, 12:36:38 PM6/18/19
to rabbitm...@googlegroups.com
Luke thank you for getting back to me in a timely manner -- greatly appreciated. 

Version is 3.7.15.
No custom rabbitmq-env.conf has been set. 
I have only a 2 node cluster.  I'll likely add new nodes later.
Port 4369 is open on both nodes. 

I did set up the cluster outside of the config file.
Without TLS enabled then nodes connect to each other.   The config file attached doesn't seem to allow a cowboy to start serving the web interface.  But on node 2 (no config) I can see node 1 has joined the cluster.   Does that mean that node 1 connected TLS? 

Thank you,

Casey



--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/d0f3bcac-ffe8-4655-afb8-f9c3785a86bc%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Luke Bakken

unread,
Jun 18, 2019, 1:05:22 PM6/18/19
to rabbitmq-users
Hi Casey,

I don't see how enabling TLS could cause rabbit_networking:record_distribution_listener to return noport because you are not using TLS for distributed Erlang, only for AMQP and HTTPS connections.

If this is a new environment, I suggest doing at this point is stopping both nodes, deleting the data directory (/var/lib/rabbitmq/*) and then re-starting both nodes. All I can think of at this point is that one node started with a different name at some point and this is confusing the clustering.

If that doesn't resolve your issue please make the complete log files available from mq1 and mq2.

Thanks -
Luke


On Tuesday, June 18, 2019 at 9:36:38 AM UTC-7, Casey Havenor wrote:
Luke thank you for getting back to me in a timely manner -- greatly appreciated. 

Version is 3.7.15.
No custom rabbitmq-env.conf has been set. 
I have only a 2 node cluster.  I'll likely add new nodes later.
Port 4369 is open on both nodes. 

I did set up the cluster outside of the config file.
Without TLS enabled then nodes connect to each other.   The config file attached doesn't seem to allow a cowboy to start serving the web interface.  But on node 2 (no config) I can see node 1 has joined the cluster.   Does that mean that node 1 connected TLS? 

Thank you,

Casey



On Mon, Jun 17, 2019 at 5:53 PM Luke Bakken <lba...@pivotal.io> wrote:
Hi Casey,

Just to be precise, what's the exact version of RabbitMQ you're using. 3.7.15 is the most recent, for instance.

Did you create an /etc/rabbitmq/rabbitmq-env.conf file or did you set any custom environment variables for RabbitMQ?

You said this in your description:

With config, I can not connect but on node 2 I can see that node 1 has joined the cluster.

You don't specifically say you're running a cluster of RabbitMQ nodes but the above statement suggests you have at least a two node cluster. Is that correct? If you are using a cluster, did you at one point start all cluster nodes without issue without using these TLS settings?

2019-06-17 05:36:28.334 [error] <0.306.0> CRASH REPORT Process <0.306.0> with 0 neighbours exited with reason: no match of right hand value noport in rabbit_networking:record_distribution_listener/0 line 305 in application_master:init/4 line 138

That error message means that one RabbitMQ node attempted to connect to another to get it's node name, but the request failed. Most likely this is due to TCP port 4369 being blocked between nodes in your cluster, or the other node isn't running.

Thanks -
Luke

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.

Casey Havenor

unread,
Jun 18, 2019, 6:11:29 PM6/18/19
to rabbitm...@googlegroups.com
Yes - there was a node name change in the middle of setting it up -- I made a mistake setting the static hostnames and had to swap them post clustering.  Do you think that is why I can't get into the web interface with TLS enabled?   Does the config look correct?

Do you need to copy the config file to each node that is the same?  Or does having one config enabled on just one node overrides the rest of nodes as they connect? 

Thanks! 

Casey Havenor 


To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/e6f98f55-8286-47dc-94f4-18737439ecbf%40googlegroups.com.

Casey Havenor

unread,
Jun 20, 2019, 5:20:34 PM6/20/19
to rabbitm...@googlegroups.com
wiped (/var/lib/rabbitmq/*)   
I re-setup the cluster. 
The nodes are connecting.   However, when I try to access the management site over Https I'm getting an error.  Empty response.  Or connection reset. 

Logs attached. 

Nothing pops out in the logs? 

Thanks, 

Casey 

rabbit@mq1_upgrade.log
rabbit@mq1.log
rabbit@mq2.log

Casey Havenor

unread,
Jun 24, 2019, 10:52:33 AM6/24/19
to rabbitm...@googlegroups.com
Luke - or anyone any ideas on what could be afoul. 

Thanks! 

Casey Havenor 

Luke Bakken

unread,
Jun 24, 2019, 3:54:20 PM6/24/19
to rabbitmq-users
Hi Casey,

It looks like HTTPS should be up and running on port 15672:

2019-06-20 21:00:12.251 [info] <0.622.0> Management plugin: HTTPS listener started on port 15672

Can you follow the troubleshooting guide here? https://www.rabbitmq.com/troubleshooting-ssl.html

Please use it to test port 15672. Also, opening your browser's developer tools may be helpful.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.

Casey Havenor

unread,
Jun 27, 2019, 1:39:09 PM6/27/19
to rabbitmq-users
So trying to connect with the following command... 

root@mq1:/etc/rabbitmq# openssl s_client -connect localhost:5671
140400506638784:error:02002068:system library:connect:Connection reset by peer:../crypto/bio/b_sock2.c:110:
140400506638784:error:2008A067:BIO routines:BIO_connect:connect error:../crypto/bio/b_sock2.c:111:
connect:errno=104

Is the problem that Letsencrypt only wants to secure web traffic? 
Why do I need to use the key as shown in the rabbitmq docs -- Letsencrypt sites don't require the client to have those public generated keys installed?

Not sure where to go from here.   Does anyone anywhere have this actually working with Let's Encrypt? 
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitm...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitm...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.

Luke Bakken

unread,
Jun 27, 2019, 1:42:26 PM6/27/19
to rabbitmq-users
Hi Casey,

I thought that the intent was to test HTTPS connections to port 15672. To do that, you use this command:

openssl s_client -connect localhost:15672

Casey Havenor

unread,
Jun 27, 2019, 1:48:43 PM6/27/19
to rabbitm...@googlegroups.com
I'm looking to get SSL enabled on the web management control interface for sub.domain.com:15672 -- Should I change that to port 443? 

Result of the below. 

root@mq1:/etc/rabbitmq# openssl s_client -connect localhost:15672
CONNECTED(00000005)
write:errno=0
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 0 bytes and written 311 bytes
Verification: OK
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)


Thanks! 

Casey Havenor 


--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.

To post to this group, send email to rabbitm...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/939f7255-1168-475f-9060-0079f502f545%40googlegroups.com.

Luke Bakken

unread,
Jun 27, 2019, 3:43:53 PM6/27/19
to rabbitmq-users
Hi Casey,

That output means that there is something wrong with your configuration for TLS. Either a cert file isn't readable, is incorrectly created, etc.

The TLS troubleshooting guide has further steps for you to try, including steps to run openssl s_server and openssl s_client to test the certs without using RabbitMQ.

You should also use the simplified configuration that I have attached here.

Thanks,
Luke
rabbitmq-simplified.config

Casey Havenor

unread,
Jun 27, 2019, 4:55:34 PM6/27/19
to rabbitm...@googlegroups.com
Not working.  I think it is something wrong with my certs.   I'm not sure LetsEncrypt makes a CA level cert?   Looking into this.  I'll keep you posted. 

Thanks! 

Casey Havenor 


--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/55962799-6b8b-4af4-a101-1163418b37ad%40googlegroups.com.

Luke Bakken

unread,
Jun 28, 2019, 2:03:23 PM6/28/19
to rabbitmq-users
Hi Casey,

You might be able to find the appropriate Lets Encrypt signing CA here: https://letsencrypt.org/certificates/
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.
Reply all
Reply to author
Forward
0 new messages