Rabbitmctl on Windows with inter-node TLS

[Lexi Elliott]

Hi,

I am trying to set up inter-node TLS on a Windows RabbitMQ cluster (as per https://www.rabbitmq.com/clustering-ssl.html). To configure it, I put the rabbitmq-env-conf.bat on each node, and restart the RabbitMQ windows service on each. After this I can no longer run commands using rabbitmqctl. Before configuring TLS, the commands worked. The erlang cookie is the same on the install user profile and C:\Windows\System32\config\systemprofile on each node. I have previously configured client TLS (as per https://www.rabbitmq.com/ssl.html) and it works fine, and they use the same certificate files.

In the user guide, it states "CLI tools such as rabbitmqctl and rabbitmq-diagnostics also must use TLS to talk to the node". How can I make these tools use TLS? And how can I verify that the system is using TLS once configured?

I have read many pages of RabbitMQ docs and tried various things so apologies in advance if I've missed something obvious!

Thanks in advance for any help.

----------------------

Additional info:

The error when running any rabbitmqctl command is as follows:

Error: unable to perform an operation on node 'rabbit@server'. Please see diagnostics information and suggestions below.

Most common reasons for this are:

 * Target node is unreachable (e.g. due to hostname resolution, TCP connection or firewall issues)

 * CLI tool fails to authenticate with the server (e.g. due to CLI tool's Erlang cookie not matching that of the server)

 * Target node is not running

In addition to the diagnostics info below:

 * See the CLI, clustering and networking guides on https://rabbitmq.com/documentation.html to learn more

 * Consult server logs on node rabbit@server

 * If target node is configured to use long node names, don't forget to use --longnames with CLI tools

DIAGNOSTICS

===========

attempted to contact: ['rabbit@server']

rabbit@server:

  * connected to epmd (port 4369) on server

  * epmd reports node 'rabbit' uses port 25672 for inter-node and CLI tool traffic

  * TCP connection succeeded but Erlang distribution failed

  * suggestion: check if the Erlang cookie identical for all server nodes and CLI tools

  * suggestion: check if all server nodes and CLI tools use consistent hostnames when addressing each other

  * suggestion: check if inter-node connections may be configured to use TLS. If so, all nodes and CLI tools must do that

   * suggestion: see the CLI, clustering and networking guides on https://rabbitmq.com/documentation.html to learn more

Current node details:

 * node name: 'rabbitmqcli-6452-rabbit@server'

 * effective user's home directory: C:\Users\myuser

 * Erlang cookie hash: Tgfip9t+sPqsYPSR6dR6RQ==

The rabbit environment config is as follows:

@echo off

rem NOTE: If spaces are present in any of these paths,

rem double quotes must be used.

rem NOTE: the following path is **system dependent** and will vary between Erlang versions

rem       and installation paths

set SSL_PATH="C:/Program Files/erl10.5/lib/ssl-9.4/ebin"

rem -pa $ERL_SSL_PATH prepends the directory ERL_SSL_PATH points at to the code path

rem -proto_dist inet_tls tells the runtime to encrypt inter-node communication

rem -ssl_dist_optfile tells the runtime where to find its inter-node TLS configuration file

set SERVER_ADDITIONAL_ERL_ARGS=-pa %SSL_PATH% ^

    -proto_dist inet_tls ^

    -ssl_dist_optfile C:/ProgramData/RabbitMQ/inter_node_tls.config

rem Same as above but for CLI tools

set CTL_ERL_ARGS=-pa %SSL_PATH% ^

    -proto_dist inet_tls ^

    -ssl_dist_optfile C:/ProgramData/RabbitMQ/inter_node_tls.config

The inter_node_tls is as follows (tried with both verify_peer and verify_none):

[

  {server, [

        {cacertfile,"C:/ProgramData/RabbitMq/MyCA.pem"},

        {certfile, "C:/ProgramData/RabbitMq/server.pem"},

        {keyfile, "C:/ProgramData/RabbitMq/server.key"},

        {secure_renegotiate, true},

        {verify, verify_none},

        {fail_if_no_peer_cert, true}

    ]},

    {client, [

        {cacertfile,"C:/ProgramData/RabbitMq/MyCA.pem"},

        {certfile, "C:/ProgramData/RabbitMq/server.pem"},

        {keyfile, "C:/ProgramData/RabbitMq/server.key"},

        {secure_renegotiate, true},

        {verify, verify_none},

        {fail_if_no_peer_cert, true}

    ]}

].

[Luke Bakken]

Hello,

This all looks correct. 

Please be certain that C:\Users\myuser\.erlang.cookie is the exact same file as C:\Windows\System32\config\systemprofile\.erlang.cookie

Thanks,

Luke

[Luke Bakken]

One more question - is myuser the administrative account that was used to install RabbitMQ?

[Lexi Elliott]

Hi,

I have confirmed that the same cookie file is in both locations.

myuser is the administrative user that was used to install RabbitMQ.

Thanks,

Lexi

[Luke Bakken]

Hello,

I'm assuming that you saved rabbitmq-env-conf.bat in C:\Users\myuser\AppData\Roaming\RabbitMQ

Did you re-install the Windows service after creating the rabbitmq-env-conf.bat file? It is required for the service to pick up the new arguments in the file. Any time the environment variables change for RabbitMQ the Windows service must be stopped, removed, re-installed and re-started.

Give this a try:

* Open the "RabbitMQ Command Prompt (sbin dir)" start menu item

* Run these commands:

.\rabbitmq-service.bat stop

.\rabbitmq-service.bat remove

.\rabbitmq-service.bat install

.\rabbitmq-service.bat start

You can verify the arguments passed to the erl.exe virtual machine when it is running as a Windows service by opening this registry key:

HKLM\software\ericsson\erlang\erlsrv\1.1\rabbitmq

Look for a sub-key named "Args". You should see the -pa, -proto_dist and other arguments specified in rabbitmq-env-conf.bat.

Then, re-try your rabbitmqctl command.

Thanks,

Luke

[Lexi Elliott]

Hi,

We were previously installing into a different Rabbit MQ base location. I have changed it back to the default location and placed the env conf file in rabbitmq-env-conf.bat and reinstalled the service. Rabbitmqctl still has the same problem (I've started a new cmd session and everything). Reinstalling the service removes the node from the cluster, presumably this mechanism should work with a single node though? I cannot add it back into the cluster without rabbitmqctl.

The Erlang args in the registry are as follows:

  -pa "C:/Program Files/RabbitMQ Server/rabbitmq_server-3.8.2/ebin" -boot start_sasl -s "rabbit" boot -config "C:\\Users\\myuser\\AppData\\Roaming\\RabbitMQ\\rabbitMq"  +W w +A "64" +MBas ageffcbf +MHas ageffcbf +MBlmbcs 512 +MHlmbcs 512 +MMmcs 30 +P 1048576 +t 5000000 +stbt db +zdbbl 128000  -kernel inet_default_connect_options "[{nodelay,true}]" -pa "C:/Program Files/erl10.5/lib/ssl-9.4/ebin"     -proto_dist inet_tls     -ssl_dist_optfile C:/ProgramData/RabbitMQ/inter_node_tls.config -sasl errlog_type error -sasl sasl_error_logger false -rabbit lager_log_root "\"C:/Users/myuser/AppData/Roaming/RabbitMQ/log\"" -rabbit lager_default_file "\"C:/Users/myuser/AppData/Roaming/RabbitMQ/log/rab...@server.log\"" -rabbit lager_upgrade_file "\"C:/Users/myuser/AppData/Roaming/RabbitMQ/log/rabbit@server_upgrade.log\"" -rabbit feature_flags_file "\"C:/Users/myuser/AppData/Roaming/RabbitMQ/db/rabbit@server-feature_flags\"" -rabbit enabled_plugins_file "\"C:/Users/myuser/AppData/Roaming/RabbitMQ/enabled_plugins\"" -rabbit plugins_dir "\"C:/Program Files/RabbitMQ Server/rabbitmq_server-3.8.2/plugins\"" -rabbit plugins_expand_dir "\"C:/Users/myuser/AppData/Roaming/RabbitMQ/db/rabbit@server-plugins-expand\"" -rabbit windows_service_config "\"C:/Users/myuser/AppData/Roaming/RabbitMQ/rabbitmq.config\"" -mnesia dir "\"C:/Users/myuser/AppData/Roaming/RabbitMQ/db/rabbit@server-mnesia\"" -os_mon start_cpu_sup false -os_mon start_disksup false -os_mon start_memsup false -ra data_dir \""C:/Users/myuser/AppData/Roaming/RabbitMQ/db/RABBIT~1/quorum"\"  -kernel inet_dist_listen_min 25672 -kernel inet_dist_listen_max 25672 

I don't know if it helps but I tried connecting to the inter-node port using tls using openssl:

c:\openssl>openssl s_client -connect server.domain.com:25672 -cert c:\certs\server.pem -key c:\certs\server.key -CAfile c:\certs\myca.pem -verify 8 -verify_hostname *.domain.com

verify depth is 8

Enter pass phrase for c:\certs\server.key:

CONNECTED(00000003)

write:errno=104

---

no peer certificate available

---

No client certificate CA names sent

---

SSL handshake has read 0 bytes and written 306 bytes

---

New, (NONE), Cipher is (NONE)

Secure Renegotiation IS NOT supported

Compression: NONE

Expansion: NONE

No ALPN negotiated

SSL-Session:

    Protocol  : TLSv1.2

    Cipher    : 0000

    Session-ID:

    Session-ID-ctx:

    Master-Key:

    Key-Arg   : None

    PSK identity: None

    PSK identity hint: None

    SRP username: None

    Start Time: 1580142523

    Timeout   : 300 (sec)

    Verify return code: 0 (ok)

---

Thanks,

Lexi

[Luke Bakken]

Hello,

Yes, that openssl output is helpful. You should see a successful TLS connection. You should double-check that your cert paths in C:/ProgramData/RabbitMQ/inter_node_tls.config are correct and readable, and that openssl s_client is using the same certificate files

We have instructions for using s_server and s_client to double-check your certs here: https://www.rabbitmq.com/troubleshooting-ssl.html

If your certificates are for wildcard domains they may not work using the default TLS host verification methods provided by Erlang/OTP. If you have the option of generating per-server certificates you should do that. If you'd like, we could use this example as a starting point for verifying whether or not your certs will work with Erlang's TLS implementation: https://erlang.org/doc/apps/ssl/using_ssl.html

If your certificates are for servers using fully-qualified host names (server1.domain.com, server2.domain.com, etc) but your cluster is formed using short host names (server1, server2) that is probably causing issues as well. Please see the documentation with regard to "long names": https://www.google.com/search?q=site%3Arabbitmq.com+longnames

Thanks,

Luke

On Monday, January 27, 2020 at 8:33:44 AM UTC-8, Lexi Elliott wrote:

Hi,

We were previously installing into a different Rabbit MQ base location. I have changed it back to the default location and placed the env conf file in rabbitmq-env-conf.bat and reinstalled the service. Rabbitmqctl still has the same problem (I've started a new cmd session and everything). Reinstalling the service removes the node from the cluster, presumably this mechanism should work with a single node though? I cannot add it back into the cluster without rabbitmqctl.

The Erlang args in the registry are as follows:

  -pa "C:/Program Files/RabbitMQ Server/rabbitmq_server-3.8.2/ebin" -boot start_sasl -s "rabbit" boot -config "C:\\Users\\myuser\\AppData\\Roaming\\RabbitMQ\\rabbitMq"  +W w +A "64" +MBas ageffcbf +MHas ageffcbf +MBlmbcs 512 +MHlmbcs 512 +MMmcs 30 +P 1048576 +t 5000000 +stbt db +zdbbl 128000  -kernel inet_default_connect_options "[{nodelay,true}]" -pa "C:/Program Files/erl10.5/lib/ssl-9.4/ebin"     -proto_dist inet_tls     -ssl_dist_optfile C:/ProgramData/RabbitMQ/inter_node_tls.config -sasl errlog_type error -sasl sasl_error_logger false -rabbit lager_log_root "\"C:/Users/myuser/AppData/Roaming/RabbitMQ/log\"" -rabbit lager_default_file "\"C:/Users/myuser/AppData/Roaming/RabbitMQ/log/rabbit@server.log\"" -rabbit lager_upgrade_file "\"C:/Users/myuser/AppData/Roaming/RabbitMQ/log/rabbit@server_upgrade.log\"" -rabbit feature_flags_file "\"C:/Users/myuser/AppData/Roaming/RabbitMQ/db/rabbit@server-feature_flags\"" -rabbit enabled_plugins_file "\"C:/Users/myuser/AppData/Roaming/RabbitMQ/enabled_plugins\"" -rabbit plugins_dir "\"C:/Program Files/RabbitMQ Server/rabbitmq_server-3.8.2/plugins\"" -rabbit plugins_expand_dir "\"C:/Users/myuser/AppData/Roaming/RabbitMQ/db/rabbit@server-plugins-expand\"" -rabbit windows_service_config "\"C:/Users/myuser/AppData/Roaming/RabbitMQ/rabbitmq.config\"" -mnesia dir "\"C:/Users/myuser/AppData/Roaming/RabbitMQ/db/rabbit@server-mnesia\"" -os_mon start_cpu_sup false -os_mon start_disksup false -os_mon start_memsup false -ra data_dir \""C:/Users/myuser/AppData/Roaming/RabbitMQ/db/RABBIT~1/quorum"\"  -kernel inet_dist_listen_min 25672 -kernel inet_dist_listen_max 25672 

[Lexi Elliott]

Hi,

I have changed it to use long names and with the wildcard certificates, it is working, providing that verify_none is set on the client certificate settings. If I use a per-server certificate, I can use verify_peer for both. So I presume it is how you said and Erlang's TLS verification doesn't work with wildcard certificates. 

Thanks so much for your help,

Lexi

[Luke Bakken]

Thanks for reporting back.

If you're using Erlang version 21 or later and would like to try to get the hostname verification working with wildcards, I think the following will work for you:

http://erlang.2086793.n4.nabble.com/Wildcard-certificate-rejected-with-hostname-check-failed-td4726616.html

Use this as your inter_node_tls.config file:

[

    {server, [

        {cacertfile,"C:/ProgramData/RabbitMq/MyCA.pem"},

        {certfile, "C:/ProgramData/RabbitMq/server.pem"},

        {keyfile, "C:/ProgramData/RabbitMq/server.key"},

        {secure_renegotiate, true},

        {verify, verify_peer},

        {fail_if_no_peer_cert, true},

        {customize_hostname_check, [

            {match_fun, public_key:pkix_verify_hostname_match_fun(https)}

        ]}

    ]},

    {client, [

        {cacertfile,"C:/ProgramData/RabbitMq/MyCA.pem"},

        {certfile, "C:/ProgramData/RabbitMq/server.pem"},

        {keyfile, "C:/ProgramData/RabbitMq/server.key"},

        {secure_renegotiate, true},

        {verify, verify_peer},

        {fail_if_no_peer_cert, true},

        {customize_hostname_check, [

            {match_fun, public_key:pkix_verify_hostname_match_fun(https)}

        ]}

    ]}

].

You will have to restart the RabbitMQ service, but you won't have to re-install it. Let me know if the above works for you.

Thanks,

Luke

On Tuesday, January 28, 2020 at 3:31:45 AM UTC-8, Lexi Elliott wrote:

Hi,

[Lexi Elliott]

Just tried it and it works.

Thanks for that!

Lexi

[Luke Bakken]

Aha! Again, thanks for taking the time to test our suggestions and report back to the list.

I will update our documentation.