rabbitmqctl status says "TCP connection succeeded but Erlang distribution failed"

[Barry Kaplan]

What does that mean?

I run:

[barry@ops-elk-1]/var/log/rabbitmq❯ sudo rabbitmqctl status
Status of node 'rabbit@ops-elk-1' ...
Error: unable to connect to node 'rabbit@ops-elk-1': nodedown


DIAGNOSTICS
===========


attempted to contact: ['rabbit@ops-elk-1']


rabbit@ops-elk-1:
  * connected to epmd (port 4369) on ops-elk-1
  * epmd reports node 'rabbit' running on port 25672
  * TCP connection succeeded but Erlang distribution failed
  * suggestion: hostname mismatch?
  * suggestion: is the cookie set correctly?
  * suggestion: is the Erlang distribution using TLS?


current node details:
- node name: 'rabbitmq-cli-23825@ops-elk-1'
- home dir: /var/lib/rabbitmq
- cookie hash: wgoXOJ59K9hu4w40T3FMbQ==

But I can connect to the management console and clients are happily sending and receiving messages.

The hostname is not wrong:

[barry@ops-elk-1]/var/log/rabbitmq❯ hostname
ops-elk-1

[Barry Kaplan]

But my bigger problem is that 'service rabbitmq-server status' fails and hence my ansible task tries to start a running a rabbit:

[barry@ops-elk-1]/var/log/rabbitmq❯ sudo service rabbitmq-server status

Status of node 'rabbit@ops-elk-1' ...
Error: unable to connect to node 'rabbit@ops-elk-1': nodedown


DIAGNOSTICS
===========


attempted to contact: ['rabbit@ops-elk-1']


rabbit@ops-elk-1:
  * connected to epmd (port 4369) on ops-elk-1
  * epmd reports node 'rabbit' running on port 25672
  * TCP connection succeeded but Erlang distribution failed
  * suggestion: hostname mismatch?
  * suggestion: is the cookie set correctly?
  * suggestion: is the Erlang distribution using TLS?


current node details:

- node name: 'rabbitmq-cli-24185@ops-elk-1'

- home dir: /var/lib/rabbitmq
- cookie hash: wgoXOJ59K9hu4w40T3FMbQ==

I have the cookie set in /var/lib/rabbitmq/.erlang.cookie. This is working for other nodes that I have clustered (this one is not clustered) 

Does the cookie need to be multiple places?

[Barry Kaplan]

Ok, it was indeed a cookie problem. The server was started before the cookie was changed, so the command line was using a different cookie.

[arghya sadhu]

Hi Barry,

How did you solve this issue?

Thanks,

Arghya

[Michael Klishin]

See "How Nodes (and CLI Tools) Authenticate to Each Other"

http://www.rabbitmq.com/clustering.html.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

[Clark Wilkerson]

The quick answer on windows was to copy c:\windows\.erlang.cookie to %USERPROFILE%\.erlang.cookie

[Nathan Owen]

I ran into this as well on windows server using 2 vms in our data center.

We have windows firewall turned on and I opened up incoming & outgoing ports 4369,5672,5671,25672 on both nodes I was clustering and it then clustered fine.

[Shweta Lokhande]

Hi Barry 
Can you please share how did you resolve this cookie issue?

[Shweta Lokhande]

PS : I am working on RHEL  server . 

[Michael Klishin]

There is a link to the docs in this very thread. Please put in some effort of your own.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Staff Software Engineer, Pivotal/RabbitMQ

[mimy...@gmail.com]

Nice answer my friend Michael Klishin Staff Software Engineer, Pivotal/RabbitMQ

Very helpful.

[Michael Klishin]

This is one of the most commonly asked questions on this list and elsewhere. It is covered decently in the docs,

there are hints right in the error message you get from CLI tools.

There are tens of thousands of emails in the archives and over 100 threads on this list alone

every month. The information is there for those who are interested in understanding the problem

and solving it.

I'm sorry but we don't deserve or need more sarcasm on this list.

If you don't have basic respect towards open source software maintainers,

you don't deserve a minute of their time.

mimyrtek is therefore permanently banned from this list.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

[Michael Klishin]

Apparently this thread gets quite a few hits, presumably from Google.

So I'm going to answer it in more detail even though there's a decent amount of information

available in the docs linked to in this thread (after all, what self respecting developer reads docs, right?)

Here are some lines produced by rabbitmqctl (or rabbitmq-plugins) that you may be staring at right now:

> * epmd reports node 'rabbit' running on port 25672 
> * TCP connection succeeded but Erlang distribution failed 
> * suggestion: hostname mismatch? 
> * suggestion: is the cookie set correctly? 
> * suggestion: is the Erlang distribution using TLS? 

So what the hell do they mean? Let's find out.

> * epmd reports node 'rabbit' running on port 25672 

This line means that rabbitmqctl connected to the epmd process (see [1]) on the target server node

and found out what RabbitMQ node's port is. Now it knows how to (try to) connect to RabbitMQ proper.

> * TCP connection succeeded but Erlang distribution failed 

it then proceeded to successfully open a TCP connection on that port. However, even though TCP connection

succeeded, rabbitmqctl could not authenticate with the target node and therefore cannot proceed

with executing the command.

What can be the reason? rabbitmqctl lists a few very common ones (sadly, in the order that puts

the most common reason at number 2):

> * suggestion: hostname mismatch? 

RabbitMQ node name has a hostname part. if CLI contact the node using a different name, authentication

will fail. This is a relatively rare scenario.

> * suggestion: is the cookie set correctly? 

This is the meat and potatoes part. As [2] explains in a dedicated sections,

nodes and CLI tools authenticate using a shared secret — called the Erlang cookie — to each other.

The cookie value must match exactly or authentication will fail.

Where is the secret value retrieved from?

By default it is read from a local file, $HOME/.erlang.cookie (on Windows it is more involved, see [4]), as a single string

of up to 255 characters in size.

It is therefore important to make sure that all server nodes have that file (provisioning tools such as Chef, Puppet, Ansible or Docker

can help with that) as well as all machines *and all users* that will run CLI tools.

Since this is a shared secret, the file must have the permissions of 0600 (accessible only to the user).

When CLI tools are executed via `sudo`, their effective user changes (typically to `root`), and thus changes the value of the cookie file path, $HOME/.erlang.cookie.

Almost always the failure described above is due to a non-existent or misplaced cookie file, or an incorrect (mismatching) value in it. More often

than not it's on the CLI tool end.

Starting with 3.7.0, rabbitmqctl and other CLI tools support setting the cookie in two more ways:

 * via the --erlang-cookie [value] CLI flag which accepts a string

 * via the `RABBITMQ_ERLANG_COOKIE` environment variable

> * suggestion: is the Erlang distribution using TLS? 

If RabbitMQ is set up to encrypt inter-node connections using TLS [3],

CLI tools also must use TLS and therefore require additional options.

Non-TLS connections from other nodes and CLI tools will fail.

There's a CLI tool that uses RabbitMQ HTTP API [4] and provides a subset of features

(plus a few unique ones) compared to rabbitmqctl. It may be an easier option but it requires

HTTP API port [1] to be open on the target node and management plugin to be enabled.

1. http://www.rabbitmq.com/networking.html

2. http://www.rabbitmq.com/clustering.html

3. http://www.rabbitmq.com/clustering-ssl.html

4. http://www.rabbitmq.com/management-cli.html

[Michael Klishin]

We added a new doc guide that goes into more detail and focuses on CLI-to-node interaction cases:

http://www.rabbitmq.com/cli.html.

Everything above is applicable for node-to-node connections as well (except for the sudo part, perhaps, since none of

RabbitMQ packages run nodes as root and it is something we absolutely do not recommend doing e.g. with the generic binary UNIX build).