TCP connection succeeded but Erlang distribution failed

[slee7791]

I know there have been a million posts about this error and the likely culprit is mismatched .erlang.cookie between the rabbitmq home dir and the server.

However, I am unable to resolve the issue as I will explain below.

I upgraded rabbitmq from 3.11.28 to 3.12.13 running on erlang 25.2 on RHEL8.  Rabbitmq came up fine and the rabbitmqctl command worked successfully.

I then proceeded to upgrade erlang from 25.2 to 26.1.3 in preparation to upgrade rabbitmq to 3.13.0 as the minimum erlang version for 3.13.0 is erlang 26.0.

The upgrade of erlang to 26.1.3 was successful and rabbitmq came up fine.  I am able to log into the admin console, look at objects such as queues, exchanges, etc and publish messages successfully, etc.

When I run the rabbitmqctl command on the command line however, I continue to get the error in the title.

Ok, let's update the .erlang.cookie in the /home/rabbitmq dir and also the .erlang.cookie in the /var/lib/rabbitmq dir to match what the current node details states is the Erlang cookie hash.

However, every time I run the rabbitmqctl command, it spits out a new Erlang cookie hash, and even though I update the .erlang.cookie's to match this new cookie hash, I continue to get the error with a new cookie hash.

So, why does the rabbitmqctl command continue to spit out new Erlang cookie hashes even though the .erlang.cookie's match between the /home/rabbitmq and /var/lib/rabbitmq dir's?

[Luke Bakken]

Hello, thanks for using RabbitMQ.

Without having a way to reproduce what you're seeing, I can only guess.

What is the value of HOME when you are running the rabbitmqctl command as a "normal" user?

What happens when you run rabbitmqctl as the root user?

[slee7791]

Hello,

Thanks for the reply.  We only allow root and the service id "rabbitmq" permissions to run the rabbitmqctl command.  The value of HOME for the rabbitmq user is /home/rabbitmq.

When running rabbitmqctl as root, that is when the Erlang error gets thrown.

Just as an FYI, running rabbitmqctl as both 'rabbitmq' and 'root' work fine before upgrading Erlang from 25.2 to 26.1.3.

Thanks.

[Julio Polo]

Erlang 26 changes the SSL default to verify_peer and supposedly makes fail_if_no_peer_cert default to true if verify_peer is set on the server.   You probably have a ssl_dist.config file in /etc/rabbitmq (configured in /etc/rabbitmq/rabbitmq-env.conf).  Try changing verify to verify_none and see if that gets rid of the problem.  Of course, this will turn off peer verification between nodes.   I was able to get rid of this error when rabbit rabbitmqctl by setting fail_if_no_peer_cert to false, so maybe that will also work for you.  Doing the latter instead of verify_none I think allows CLI tools to work without doing peer verification while the Erlang nodes are still doing peer verification (because I set verify=verify_peer instead of verify_none)

[Julio Polo]

You also have to update ERL_SSL_PATH in /etc/rabbitmq/rabbitmq-env.conf to the new version for Erlang 26.  I went from Erlang 25.3 ERL_SSL_PATH="/usr/lib64/erlang/lib/ssl-10.8.5/ebin" to Erlang 26.0 ERL_SSL_PATH="/usr/lib64/erlang/lib/ssl-11.0/ebin"