Cannot create a quorum queue - fails with error cluster_not_formed

[Diego Maravankin]

Hi guys! 

Hope everyone is doing well. I am having some issues creating a quorum queue (I'll use QQ to refer to these) in a newly created 3 node cluster.

The rabbitmqctl cluster_status command shows the same info on each cluster node, with all three nodes online and connected. 

When attempting to create a new QQ, I get the following error:

I put all nodes in debug to get more information, and logs show the following entries:
mb-01 (where I issue the queue creation)

mb-02

db-arb

Config files are the same in the three nodes, and Node names are resovled via hosts file:

rabbitmq.conf

cluster_formation.peer_discovery_backend = classic_config

cluster_formation.classic_config.nodes.1 = rabbit@mb-01

cluster_formation.classic_config.nodes.2 = rabbit@mb-02

cluster_formation.classic_config.nodes.3 = rabbit@db-arb

advanced.config

[

 %% five replicas by default, only makes sense for nine node clusters

 {rabbit, [

            {quorum_cluster_size, 2},

            {quorum_commands_soft_limit, 512}

           ]

        }

].

I understand the issue is that mb-01 is not receiving db-arb's vote, thus timing out and the QQ does not created. I checked and TCP connections are established and traffic is flowing through port 25672.

I would really appreciate any lead on how to troubleshoot further to find the cause of this.

Thanks in advance,

Diego

[Karl Nilsson]

{quorum_cluster_size, 2},

Don't use even cluster sizes, Try 3.

Cheers

Karl

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/d4ddd4ae-51b6-430e-af07-4868b747a55cn%40googlegroups.com.

--

Karl Nilsson

[Diego Maravankin]

Hi Karl,

Thanks for the quick response. I thought that parameter was the minimum amount of nodes. I tried setting quorum_cluster_size to 3 nodes, but still get the same error indicating that one node (it is always the same node) timed out.

Cheers,

Diego

[Diego Maravankin]

I did some more tests. I removed the offending node from the cluster, did a force_reset and added it back. 

After doing so, the node timing out was the one that wasn't removed. I am thinking I might have something wrong with my cluster definition. 

 I am using RabbitMQ 3.8.14 on Erlang 23.3.1, installed in Debian 10 from the bionic repos (buster repos did not work)

I followed the Clustering Guide in the documentation, using classic cluster definition and declaring the nodes in the rabbitmq.conf file. 

Then I bootstraped the cluster using:

rabbitmqctl stop_app

rabbitmqctl join_cluster rabbit@mb-01

rabbitmqctl start_app

I am missing something?