Rabbitmq nodes not able to join cluster?

[Nishant Mishra]

[root@DGTLSITTIBCO2 ~]# rabbitmqctl join_cluster rabbit@DGTLSITTIBCO1 Clustering node rabbit@DGTLSITTIBCO2 with rabbit@DGTLSITTIBCO1 Error: unable to perform an operation on node 'rabbit@DGTLSITTIBCO1'. Please see diagnostics information and suggestions below.

Most common reasons for this are:

• Target node is unreachable (e.g. due to hostname resolution, TCP connection or firewall issues)
• CLI tool fails to authenticate with the server (e.g. due to CLI tool's Erlang cookie not matching that of the server)
• Target node is not running

In addition to the diagnostics info below:

• See the CLI, clustering and networking guides on rabbitmq documentation to learn more
• Consult server logs on node rabbit@DGTLSITTIBCO1

DIAGNOSTICS

attempted to contact: [rabbit@DGTLSITTIBCO1]

rabbit@DGTLSITTIBCO1:

• connected to epmd (port 4369) on DGTLSITTIBCO1
• epmd reports node 'rabbit' uses port 25672 for inter-node and CLI tool traffic

• TCP connection succeeded but Erlang distribution failed

• TCP connection to remote host has timed out. Is the Erlang distribution using TLS?

Current node details:

• node name: rabbitmqcli57@DGTLSITTIBCO2
• effective user's home directory: /var/lib/rabbitmq
• Erlang cookie hash: ci1chJoiyrDKCLOj2TDxaA==

[Luke Bakken]

Hi Nishant,

Generally, it's not very helpful to paste an error into your message without any other information. The people helping out on this list like to know what version of RabbitMQ and Erlang you're using and on what operating system, and a description of what you're trying to do.

Anyway, in this case, your issue is one of the most commonly reported on this list. If you search for the string Error: unable to perform an operation on node, you'll see many replies with suggestions on how to fix your issue.

The RabbitMQ team also maintains documentation, including a troubleshooting document: https://www.rabbitmq.com/troubleshooting.html

You should also read the Clustering Guide, specifically this section: https://www.rabbitmq.com/clustering.html#erlang-cookie

Thanks,

Luke

[Michael Klishin]

> TCP connection succeeded but Erlang distribution failed

is the line you are looking for. It has a dedicated documentation section in more than one guide [1].

1. http://www.rabbitmq.com/clustering.html#erlang-cookie

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

[Nishant Mishra]

I have copied the erlang cookie from one server to another in order to have same cookie in both the nodes using below command :

echo -n "KGHHFIOLNZMAWIFNOCKI" > /var/lib/rabbitmq/.erlang.cookie

erlang version : erlang-21.0.2-1.el6.x86_64.

rabbitmw version : rabbitmq-server-3.7.7-1.el6.noarch

socat : socat-1.7.1.3-1.el6.rf.x86_64.rpm

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Nishant Mishra]

Hi Luke,

Please find below the details and issues which i am facing while joining the nodes in a cluster:

erlang version : erlang-21.0.2-1.el6.x86_64.

rabbitmw version : rabbitmq-server-3.7.7-1.el6.noarch

socat : socat-1.7.1.3-1.el6.rf.x86_64.rpm

I copied the erlang cookie from one server to another in order to have same cookie in both the nodes using below command :

echo -n "KGHHFIOLNZMAWIFNOCKI" > /var/lib/rabbitmq/.erlang.cookie

#run the following command 

rabbitmqctl reset

#stopped both the node 

rabbitmqctl stop_app

#now tried joining the nodes :

rabbitmqctl join_cluster rabbit@DGTLSITTIBCO1

Below is error faced :

Clustering node rabbit@DGTLSITTIBCO2 with rabbit@DGTLSITTIBCO1

Error: unable to perform an operation on node 'rabbit@DGTLSITTIBCO2'. Please see diagnostics information and suggestions below.

Most common reasons for this are:

 * Target node is unreachable (e.g. due to hostname resolution, TCP connection or firewall issues)

 * CLI tool fails to authenticate with the server (e.g. due to CLI tool's Erlang cookie not matching that of the server)

 * Target node is not running

In addition to the diagnostics info below:

 * See the CLI, clustering and networking guides on http://rabbitmq.com/documentation.html to learn more

 * Consult server logs on node rabbit@DGTLSITTIBCO2

DIAGNOSTICS

===========

attempted to contact: [rabbit@DGTLSITTIBCO2]

rabbit@DGTLSITTIBCO2:

  * connected to epmd (port 4369) on DGTLSITTIBCO2

  * epmd reports: node 'rabbit' not running at all

                  no other nodes on DGTLSITTIBCO2

  * suggestion: start the node

Current node details:

 * node name: rabbitmqcli34@DGTLSITTIBCO2

 * effective user's home directory: /var/lib/rabbitmq

 * Erlang cookie hash: ci1chJoiyrDKCLOj2TDxaA==

When i start the current node using "rabbitmqctl start_app" and retry joining node using "rabbitmqctl join_cluster rabbit@DGTLSITTIBCO1", getting another error :

[root@DGTLSITTIBCO2 ~]# rabbitmqctl join_cluster rabbit@DGTLSITTIBCO1

Error: this command requires the 'rabbit' app to be stopped on the target node. Stop it with 'rabbitmqctl stop_app'.

Arguments given:

        join_cluster rabbit@DGTLSITTIBCO1

Usage:

rabbitmqctl [-n <node>] [-l] [-q] join_cluster [--disc|--ram] <existing_cluster_member_node>

Please suggest some solution.

[Michael Klishin]

Please read the messages CLI tools produce. Now they report that the node is NOT running at all.

So rabbitmqctl cannot even get to the authentication phase.

--

You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Nishant Mishra]

Yes, but when i start the current node using "rabbitmqctl start_app" and retry joining node using "rabbitmqctl join_cluster rabbit@DGTLSITTIBCO1", getting another error :

[root@DGTLSITTIBCO2 ~]# rabbitmqctl join_cluster rabbit@DGTLSITTIBCO1

Error: this command requires the 'rabbit' app to be stopped on the target node. Stop it with 'rabbitmqctl stop_app'.

Arguments given:

        join_cluster rabbit@DGTLSITTIBCO1

Usage:

rabbitmqctl [-n <node>] [-l] [-q] join_cluster [--disc|--ram] <existing_cluster_member_node>

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Nishant Mishra]

SO i believe the current node should remain stop while the master node should be running.

[Michael Klishin]

`rabbitmqctl start_app` needs a running node (an Erlang VM node) to start the RabbitMQ application on it.

If you shut down RabbitMQ and the runtime with `rabbitmqctl stop` it will terminate the VM (as the last message in the RabbitMQ

log will say).

Use the means of starting services on your OS (`sudo service …` on most recent distributions that are based on systemd) to start the node.

You can even stop the node using the same mechanism (which should do nothing and simply say that the service isn't running to begin with)

and then start it.

Then you would be able to use CLI tools and see if your cookie changes had an effect.

`rabbitmqctl stop_app` and `start_app` stop/start RabbitMQ ("pause" and "unpause' it, if you will) without shutting down the runtime

so that certain CLI commands that do not require a running RabbitMQ node (or even mandate that the local node is not running) can operate.

`rabbitmqctl stop` shuts down the OS process that runs RabbitMQ, so once that happens, you cannot "unpause" and need to start that process again

using the system service management tool or `./sbin/rabbitmq-server` in the binary RabbitMQ distribution.

It's a pretty confusing design decision at first but increasingly developers and operators alike can use peer discovery [1]

100% of the time, so those commands aren't needed.

1. http://www.rabbitmq.com/cluster-formation.html

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.