rabbitmqctl fails saying "TCP connection succeeded but Erlang distribution failed"

[Narendran G]

Hello Team,

RabbitMQ & Erlang Version:

---

## ## RabbitMQ 3.12.7

## ##

########## Copyright (c) 2007-2023 VMware, Inc. or its affiliates.

###### ##

########## Licensed under the MPL 2.0. Website: https://rabbitmq.com

Erlang: 26.1.2 [emu]

TLS Library: OpenSSL - OpenSSL 3.0.8 7 Feb 2023

Release series support status: supported

---

We're trying to deploy RabbitMQ with fips enabled in a k8s pod. So far, rabbitmq-server is up and running w/o any issues. But when we try to reach the rabbitmq-server using rabbitmqctl, we fail with the below error:-

rabbitmqctl status -n rab...@xxxx-rabbitmq-0.xxxx-rabbitmq.xxxx.svc.cluster.local --longnames

---

Most common reasons for this are:

    * Target node is unreachable (e.g. due to hostname resolution, TCP connection or firewall issues)
    * CLI tool fails to authenticate with the server (e.g. due to CLI tool's Erlang cookie not matching that of the server)
    * Target node is not running

DIAGNOSTICS
===========

    attempted to contact: ['rab...@xxxx-rabbitmq-0.xxxx-rabbitmq.xxxx.svc.cluster.local']

    rab...@xxxx-rabbitmq-0.xxxx-rabbitmq.xxxx.svc.cluster.local:
    * connected to epmd (port 4369) on xxxx-rabbitmq-0.xxxx-rabbitmq.xxxx.svc.cluster.local
    * epmd reports node 'rabbit' uses port 25672 for inter-node and CLI tool traffic
    * TCP connection succeeded but Erlang distribution failed
    * suggestion: check if the Erlang cookie is identical for all server nodes and CLI tools
    * suggestion: check if all server nodes and CLI tools use consistent hostnames when addressing each other
    * suggestion: check if inter-node connections may be configured to use TLS. If so, all nodes and CLI tools must do that
    * suggestion: see the CLI, clustering and networking guides on https://rabbitmq.com/documentation.html to learn more

---

Some of the above suggestions tell us the issue could be due to 

1. erlang cookie mismatch (CLI tool fails to authenticate with the server)
2. hostname resolution, consistent hostname
3. Target node is not running
   
4. Target node is unreachable (e.g. due to hostname resolution, TCP connection or firewall issues)
   
5. check if inter-node connections may be configured to use TLS. If so, all nodes and CLI tools must do that
   

For suggestions 1-4, we can be able to verify and is in the expected state. 

1. We've the same cookie value hash for both server & cli-tool (z0CZ8wlEB4HFdxH01POdWQ==)

2. Hostname is consistent for both server & client and it can be resolved as below:-

xxxx-rabbitmq-0:/$ rabbitmq-diagnostics resolve_hostname $(hostname -f) --address-family IPv4 --offline

Resolving hostname xxxx-rabbitmq-0.xxxx-rabbitmq.xxxx.svc.cluster.local to IPv4 addresses...

10.x.xxx.xx

xxxx-rabbitmq-0:/$ rabbitmq-diagnostics resolve_hostname $(hostname) --address-family IPv4 --offline

Resolving hostname xxxx-rabbitmq-0 to IPv4 addresses...

10.x.xxx.xx

3. Target node is running as we can see it in the server log (log attached)

4. Target node is unreachable (e.g. due to hostname resolution, TCP connection or firewall issues)

- Hostname resolution works good as we see earlier.

- Can't be firewall issue (as server & client running in the same pod)

- TCP connection is successful as we see in the succeeding logs, that says "TCP connection succeeded but Erlang distribution failed"

5. check if inter-node connections may be configured to use TLS. If so, all nodes and CLI tools must do that

This could be an issue. But we didn't see this issue in RabbitMQ w/o fips enabled.

We don't know what the issue is about and how to verify it. Can someone please take a look at this issue and help us debug it further? Let me know if you need any further information.

Thanks,

Naren

[Narendran G]

Can someone give me a heads-up to debug further?

[Luke Bakken]

Hi Naren,

There is no need to bump your message. If someone is willing and able to assist you, they will.

You haven't provided your RabbitMQ or k8s configuration, nor any details about how you enabled FIPS, so there's really very little to work with here.

It sounds like you are able to use inter-node TLS without FIPS enabled, is that true? I am also assuming that if you use non-TLS inter-node communication your environment also works correctly.

Thanks -

Luke

[Narendran G]

Hello Luke,

Thanks for the reply!

It sounds like you are able to use inter-node TLS without FIPS enabled, is that true?

>> RMQ was up and running w/o FIPS enabled. The inter-node TLS was configured as below :-

export ERL_SSL_PATH=/usr/lib/erlang/lib/ssl-8.1.1/ebin
export RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS="-pa ${ERL_SSL_PATH} -proto_dist inet_tls -ssl_dist_opt server_certfile /etc/rabbitmq/ssl/server.pem -ssl_dist_opt server_keyfile /etc/rabbitmq/ssl/server.key -ssl_dist_opt server_secure_renegotiate true client_secure_renegotiate true -ssl_dist_opt server_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256"

I am also assuming that if you use non-TLS inter-node communication your environment also works correctly.

>> Yes, I'd to unset the env variable RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS  to disable TLS for inter-node communication and it works completely fine.

how you enabled FIPS?

>> FYR, I've attached the config file - advanced.config.

we added below configuration 

- crypto.fips_mode as true

- ssl.versions as ['tlsv1.2']

I tried to configure TLS for CLI tools using the below configuration, but it was still failing.

---

export ERL_SSL_PATH=/usr/local/lib/erlang/lib/ssl-11.0.3/ebin

export RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS="-pa ${ERL_SSL_PATH} -proto_dist inet_tls -ssl_dist_optfile ${RABBITMQ_HOME}/etc/rabbitmq/inter_node_tls.config"

export RABBITMQ_CTL_ERL_ARGS="-pa ${ERL_SSL_PATH} -proto_dist inet_tls -ssl_dist_optfile ${RABBITMQ_HOME}/etc/rabbitmq/inter_node_tls.config"

cat ${RABBITMQ_HOME}/etc/rabbitmq/inter_node_tls.config

[
    {server, [
        {cacertfile, "/etc/rabbitmq/ssl/cacert.pem"},
        {certfile,   "/etc/rabbitmq/ssl/server.pem"},
        {keyfile,    "/etc/rabbitmq/ssl/server.key"},
        {secure_renegotiate, true},
        {verify, verify_peer},
        {fail_if_no_peer_cert, true}
    ]},
    {client, [
        {cacertfile, "/etc/rabbitmq/ssl/cacert.pem"},
        {certfile,   "/etc/rabbitmq/ssl/server.pem"},
        {keyfile,    "/etc/rabbitmq/ssl/server.key"},
        {secure_renegotiate, true},
        {verify, verify_peer}
    ]}
].

---

As a workaround, we've disabled the TLS for inter-node communication to have the RMQ working.

Thanks,

Naren

[Luke Bakken]

Hello,

I suggest enabling TLS and not using FIPS if you'd like to secure inter-node communication.

OpenSSL 3 / FIPS support will be part of VMware's commercial RabbitMQ distribution.

Thanks,

Luke

[Narendran G]

OK, thank you!

--
You received this message because you are subscribed to a topic in the Google Groups "rabbitmq-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/rabbitmq-users/ij3eSV8SnVw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/69e58e39-d723-452c-a376-a7cac51d04ddn%40googlegroups.com.