rabbitmq-autocluster - cluster not creating

Mike Brannigan

unread,

Apr 25, 2016, 2:33:38 PM4/25/16

to rabbitmq-users

I have the rabbitmq-autocluster plugin enabled on my rabbitmq nodes. I'm trying to get them to join together in a cluster using consul. The nodes are correctly registering with consul yet they don't seem to be attempting to contact one another. Here's the rabbitmq.config file that I'm working with:

[{autocluster,

[

{consul_host, "localhost"},

{consul_port, 8500},

{consul_service, "rabbitmq"},

{longname, "true"},

{cluster_name, "staging"}

]}

].

Any recommendations on how I can debug this?

Gavin M. Roy

unread,

Apr 25, 2016, 2:38:02 PM4/25/16

to rabbitm...@googlegroups.com

Have you also setup RabbitMQ long name support as well?

See https://www.rabbitmq.com/networking.html#dns

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Michael Klishin

unread,

Apr 25, 2016, 2:42:14 PM4/25/16

to rabbitm...@googlegroups.com

Is your config file loaded? See http://rabbitmq.com/configure.html for details.

Is the plugin enabled?

--

Mike Brannigan

unread,

Apr 25, 2016, 3:09:19 PM4/25/16

to rabbitmq-users

The config file is loaded:

rab...@ip-10-10-2-20.log:config file(s) : /etc/rabbitmq/rabbitmq.config

The plugin is enabled:

# rabbitmq-plugins list

[e] amqp_client 3.2.4

[E] autocluster 0.4.1

[ ] cowboy 0.5.0-rmq3.2.4-git4b93c2d

[ ] eldap 3.2.4-gite309de4

[e] mochiweb 2.7.0-rmq3.2.4-git680dba8

[ ] rabbitmq_amqp1_0 3.2.4

[ ] rabbitmq_auth_backend_ldap 3.2.4

[ ] rabbitmq_auth_mechanism_ssl 3.2.4

[ ] rabbitmq_consistent_hash_exchange 3.2.4

[ ] rabbitmq_federation 3.2.4

[ ] rabbitmq_federation_management 3.2.4

[ ] rabbitmq_jsonrpc 3.2.4

[ ] rabbitmq_jsonrpc_channel 3.2.4

[ ] rabbitmq_jsonrpc_channel_examples 3.2.4

[E] rabbitmq_management 3.2.4

[e] rabbitmq_management_agent 3.2.4

[ ] rabbitmq_management_visualiser 3.2.4

[ ] rabbitmq_mqtt 3.2.4

[ ] rabbitmq_shovel 3.2.4

[ ] rabbitmq_shovel_management 3.2.4

[ ] rabbitmq_stomp 3.2.4

[ ] rabbitmq_tracing 3.2.4

[e] rabbitmq_web_dispatch 3.2.4

[ ] rabbitmq_web_stomp 3.2.4

[ ] rabbitmq_web_stomp_examples 3.2.4

[ ] rfc4627_jsonrpc 3.2.4-git5e67120

[ ] sockjs 0.3.4-rmq3.2.4-git3132eb9

[e] webmachine 1.10.3-rmq3.2.4-gite9359c7

Michael Klishin

unread,

Apr 25, 2016, 3:12:11 PM4/25/16

to rabbitm...@googlegroups.com, Mike Brannigan

Gavin is the source of truth on this but I'm not sure if this plugin supports RabbitMQ 3.2.x.

> To post to this group, send an email to rabbitm...@googlegroups.com.

> For more options, visit https://groups.google.com/d/optout.
>

--
MK

Staff Software Engineer, Pivotal/RabbitMQ

Mike Brannigan

unread,

Apr 25, 2016, 3:44:38 PM4/25/16

to rabbitmq-users

I updated RabbitMQ to 3.6.1. Also, in the rabbitmq-env.config file, I set the following:

RABBITMQ_USE_LONGNAME=true

rabbitmqctl cluster_status still only shows the local node:

root@ip-10-10-2-20:/var/log/rabbitmq# rabbitmqctl cluster_status

Cluster status of node 'rabbit@ip-10-10-2-20' ...

[{nodes,[{disc,['rabbit@ip-10-10-2-20']}]},

{running_nodes,['rabbit@ip-10-10-2-20']},

{cluster_name,<<"rabbit@ip-10-10-2-20">>},

{partitions,[]},

{alarms,[{'rabbit@ip-10-20-12-7',[]}]}]

Mike Brannigan

unread,

Apr 25, 2016, 3:46:56 PM4/25/16

to rabbitmq-users

Here's the startup log on one of the nodes:

=INFO REPORT==== 25-Apr-2016::19:45:02 ===

Starting RabbitMQ 3.6.1 on Erlang R16B03

Licensed under the MPL. See http://www.rabbitmq.com/

=INFO REPORT==== 25-Apr-2016::19:45:02 ===

node : rabbit@ip-10-20-11-33

home dir : /var/lib/rabbitmq

config file(s) : /etc/rabbitmq/rabbitmq.config

cookie hash : HASH

log : /var/log/rabbitmq/rab...@ip-10-20-11-33.log

sasl log : /var/log/rabbitmq/rab...@ip-10-20-11-33-sasl.log

database dir : /var/lib/rabbitmq/mnesia/rabbit@ip-10-20-11-33

=ERROR REPORT==== 25-Apr-2016::19:45:03 ===

autocluster: Can not communicate with cluster nodes: ['rabbit@ip-10-20-13-182',

'rabbit@ip-10-20-12-7']

=INFO REPORT==== 25-Apr-2016::19:45:03 ===

Memory limit set to 1580MB of 3952MB total.

=INFO REPORT==== 25-Apr-2016::19:45:03 ===

Disk free limit set to 50MB

=INFO REPORT==== 25-Apr-2016::19:45:03 ===

Limiting to approx 924 file handles (829 sockets)

=INFO REPORT==== 25-Apr-2016::19:45:03 ===

FHC read buffering: OFF

FHC write buffering: ON

=INFO REPORT==== 25-Apr-2016::19:45:03 ===

Priority queues enabled, real BQ is rabbit_variable_queue

=INFO REPORT==== 25-Apr-2016::19:45:03 ===

Management plugin: using rates mode 'basic'

=INFO REPORT==== 25-Apr-2016::19:45:04 ===

msg_store_transient: using rabbit_msg_store_ets_index to provide index

=INFO REPORT==== 25-Apr-2016::19:45:04 ===

msg_store_persistent: using rabbit_msg_store_ets_index to provide index

=INFO REPORT==== 25-Apr-2016::19:45:04 ===

started TCP Listener on [::]:5672

=INFO REPORT==== 25-Apr-2016::19:45:04 ===

autocluster: Starting Consul Health Check TTL Timer

=INFO REPORT==== 25-Apr-2016::19:45:04 ===

Management plugin started. Port: 15672

=INFO REPORT==== 25-Apr-2016::19:45:04 ===

Statistics database started.

=INFO REPORT==== 25-Apr-2016::19:45:04 ===

Server startup complete; 7 plugins started.

* rabbitmq_management

* rabbitmq_web_dispatch

* webmachine

* mochiweb

* autocluster

* rabbitmq_management_agent

* amqp_client

=INFO REPORT==== 25-Apr-2016::19:45:05 ===

accepting AMQP connection <0.330.0> (127.0.0.1:45902 -> 127.0.0.1:5672)

=INFO REPORT==== 25-Apr-2016::19:45:05 ===

accepting AMQP connection <0.339.0> (127.0.0.1:45903 -> 127.0.0.1:5672)

=INFO REPORT==== 25-Apr-2016::19:45:06 ===

accepting AMQP connection <0.370.0> (127.0.0.1:45904 -> 127.0.0.1:5672)

Gavin M. Roy

unread,

Apr 25, 2016, 4:03:22 PM4/25/16

to rabbitm...@googlegroups.com

3.2.x and 3.6.x are not supported with the version you're running. 3.5.6 is your best bet. Also, you can set the log level in the rabbitmq config to get debug logs from the autocluster plugin.

--

You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.

To post to this group, send email to rabbitm...@googlegroups.com.

Mike Brannigan

unread,

Apr 25, 2016, 5:53:19 PM4/25/16

to rabbitmq-users

Ok. I installed 3.5.6 on the nodes and enabled debug logging:

=INFO REPORT==== 25-Apr-2016::21:51:19 ===

Starting RabbitMQ 3.5.6 on Erlang R16B03

Licensed under the MPL. See http://www.rabbitmq.com/

=INFO REPORT==== 25-Apr-2016::21:51:19 ===

node : rabbit@ip-10-20-12-151

home dir : /var/lib/rabbitmq

config file(s) : /etc/rabbitmq/rabbitmq.config

cookie hash : uTHWz8EK0NSx8oXuvnUpEw==

log : /var/log/rabbitmq/rab...@ip-10-20-12-151.log

sasl log : /var/log/rabbitmq/rab...@ip-10-20-12-151-sasl.log

database dir : /var/lib/rabbitmq/mnesia/rabbit@ip-10-20-12-151

=INFO REPORT==== 25-Apr-2016::21:51:19 ===

autocluster: GET http://localhost:8500/v1/health/service/rabbitmq?passing&tag=staging

=INFO REPORT==== 25-Apr-2016::21:51:19 ===

autocluster: Response: [{ok,{{"HTTP/1.1",200,"OK"},

[{"date","Mon, 25 Apr 2016 21:51:19 GMT"},

{"content-length","2376"},

{"content-type","application/json"},

{"x-consul-index","181142"},

{"x-consul-knownleader","true"},

{"x-consul-lastcontact","0"}],

"[{\"Node\":{\"Node\":\"ip-10-20-11-61\",\"Address\":\"10.20.11.61\",\"TaggedAddresses\":{\"wan\":\"10.20.11.61\"},\"CreateIndex\":181005,\"ModifyIndex\":181140},\"Service\":{\"ID\":\"rabbitmq\",\"Service\":\"rabbitmq\",\"Tags\":[\"staging\"],\"Address\":\"\",\"Port\":5672,\"EnableTagOverride\":false,\"CreateIndex\":181030,\"ModifyIndex\":181140},\"Checks\":[{\"Node\":\"ip-10-20-11-61\",\"CheckID\":\"serfHealth\",\"Name\":\"Serf Health Status\",\"Status\":\"passing\",\"Notes\":\"\",\"Output\":\"Agent alive and reachable\",\"ServiceID\":\"\",\"ServiceName\":\"\",\"CreateIndex\":181005,\"ModifyIndex\":181005},{\"Node\":\"ip-10-20-11-61\",\"CheckID\":\"service:rabbitmq\",\"Name\":\"Service 'rabbitmq' check\",\"Status\":\"passing\",\"Notes\":\"RabbitMQ Auto-Cluster Plugin TTL Check\",\"Output\":\"\",\"ServiceID\":\"rabbitmq\",\"ServiceName\":\"rabbitmq\",\"CreateIndex\":181030,\"ModifyIndex\":181140}]},{\"Node\":{\"Node\":\"ip-10-20-12-151\",\"Address\":\"10.20.12.151\",\"TaggedAddresses\":{\"wan\":\"10.20.12.151\"},\"CreateIndex\":181024,\"ModifyIndex\":181142},\"Service\":{\"ID\":\"rabbitmq\",\"Service\":\"rabbitmq\",\"Tags\":[\"staging\"],\"Address\":\"\",\"Port\":5672,\"EnableTagOverride\":false,\"CreateIndex\":181040,\"ModifyIndex\":181142},\"Checks\":[{\"Node\":\"ip-10-20-12-151\",\"CheckID\":\"serfHealth\",\"Name\":\"Serf Health Status\",\"Status\":\"passing\",\"Notes\":\"\",\"Output\":\"Agent alive and reachable\",\"ServiceID\":\"\",\"ServiceName\":\"\",\"CreateIndex\":181024,\"ModifyIndex\":181024},{\"Node\":\"ip-10-20-12-151\",\"CheckID\":\"service:rabbitmq\",\"Name\":\"Service 'rabbitmq' check\",\"Status\":\"passing\",\"Notes\":\"RabbitMQ Auto-Cluster Plugin TTL Check\",\"Output\":\"\",\"ServiceID\":\"rabbitmq\",\"ServiceName\":\"rabbitmq\",\"CreateIndex\":181040,\"ModifyIndex\":181142}]},{\"Node\":{\"Node\":\"ip-10-20-13-77\",\"Address\":\"10.20.13.77\",\"TaggedAddresses\":{\"wan\":\"10.20.13.77\"},\"CreateIndex\":180926,\"ModifyIndex\":181139},\"Service\":{\"ID\":\"rabbitmq\",\"Service\":\"rabbitmq\",\"Tags\":[\"staging\"],\"Address\":\"\",\"Port\":5672,\"EnableTagOverride\":false,\"CreateIndex\":180943,\"ModifyIndex\":181139},\"Checks\":[{\"Node\":\"ip-10-20-13-77\",\"CheckID\":\"serfHealth\",\"Name\":\"Serf Health Status\",\"Status\":\"passing\",\"Notes\":\"\",\"Output\":\"Agent alive and reachable\",\"ServiceID\":\"\",\"ServiceName\":\"\",\"CreateIndex\":180926,\"ModifyIndex\":180926},{\"Node\":\"ip-10-20-13-77\",\"CheckID\":\"service:rabbitmq\",\"Name\":\"Service 'rabbitmq' check\",\"Status\":\"passing\",\"Notes\":\"RabbitMQ Auto-Cluster Plugin TTL Check\",\"Output\":\"\",\"ServiceID\":\"rabbitmq\",\"ServiceName\":\"rabbitmq\",\"CreateIndex\":180943,\"ModifyIndex\":181139}]}]"}}]

=INFO REPORT==== 25-Apr-2016::21:51:19 ===

autocluster: Node is only node in the cluster

=ERROR REPORT==== 25-Apr-2016::21:51:19 ===

autocluster: Can not communicate with cluster nodes: ['rabbit@ip-10-20-11-61',

'rabbit@ip-10-20-13-77']

=INFO REPORT==== 25-Apr-2016::21:51:19 ===

Memory limit set to 1580MB of 3952MB total.

=INFO REPORT==== 25-Apr-2016::21:51:19 ===

Disk free limit set to 50MB

=INFO REPORT==== 25-Apr-2016::21:51:19 ===

Limiting to approx 924 file handles (829 sockets)

=INFO REPORT==== 25-Apr-2016::21:51:19 ===

FHC read buffering: ON

FHC write buffering: ON

=INFO REPORT==== 25-Apr-2016::21:51:19 ===

Priority queues enabled, real BQ is rabbit_variable_queue

=INFO REPORT==== 25-Apr-2016::21:51:19 ===

Management plugin: using rates mode 'basic'

=INFO REPORT==== 25-Apr-2016::21:51:19 ===

msg_store_transient: using rabbit_msg_store_ets_index to provide index

=INFO REPORT==== 25-Apr-2016::21:51:19 ===

msg_store_persistent: using rabbit_msg_store_ets_index to provide index

=INFO REPORT==== 25-Apr-2016::21:51:19 ===

started TCP Listener on [::]:5672

=INFO REPORT==== 25-Apr-2016::21:51:19 ===

autocluster: Starting Consul Health Check TTL Timer

=INFO REPORT==== 25-Apr-2016::21:51:20 ===

Management plugin started. Port: 15672

=INFO REPORT==== 25-Apr-2016::21:51:20 ===

Statistics database started.

=INFO REPORT==== 25-Apr-2016::21:51:20 ===

Server startup complete; 7 plugins started.

* rabbitmq_management

* rabbitmq_web_dispatch

* webmachine

* mochiweb

* autocluster

* rabbitmq_management_agent

* amqp_client

=INFO REPORT==== 25-Apr-2016::21:51:20 ===

accepting AMQP connection <0.290.0> (127.0.0.1:40425 -> 127.0.0.1:5672)

=INFO REPORT==== 25-Apr-2016::21:51:24 ===

accepting AMQP connection <0.301.0> (127.0.0.1:40426 -> 127.0.0.1:5672)

=INFO REPORT==== 25-Apr-2016::21:51:24 ===

accepting AMQP connection <0.310.0> (127.0.0.1:40427 -> 127.0.0.1:5672)

=INFO REPORT==== 25-Apr-2016::21:51:34 ===

autocluster: GET http://localhost:8500/v1/agent/check/pass/service%3Arabbitmq

=INFO REPORT==== 25-Apr-2016::21:51:34 ===

autocluster: Response: [{ok,{{"HTTP/1.1",200,"OK"},

[{"date","Mon, 25 Apr 2016 21:51:34 GMT"},

{"content-length","0"},

{"content-type","text/plain; charset=utf-8"}],

[]}}]

On Monday, April 25, 2016 at 3:03:22 PM UTC-5, Gavin M. Roy wrote:

3.2.x and 3.6.x are not supported with the version you're running. 3.5.6 is your best bet. Also, you can set the log level in the rabbitmq config to get debug logs from the autocluster plugin.

On Mon, Apr 25, 2016 at 3:46 PM, Mike Brannigan <mikejbr...@gmail.com> wrote:

Here's the startup log on one of the nodes:

=INFO REPORT==== 25-Apr-2016::19:45:02 ===
Starting RabbitMQ 3.6.1 on Erlang R16B03
Copyright (C) 2007-2016 Pivotal Software, Inc.
Licensed under the MPL. See http://www.rabbitmq.com/

=INFO REPORT==== 25-Apr-2016::19:45:02 ===
node : rabbit@ip-10-20-11-33
home dir : /var/lib/rabbitmq
config file(s) : /etc/rabbitmq/rabbitmq.config
cookie hash : HASH

log : /var/log/rabbitmq/rabbit@ip-10-20-11-33.log
sasl log : /var/log/rabbitmq/rabbit@ip-10-20-11-33-sasl.log

Gavin M. Roy

unread,

Apr 25, 2016, 8:15:17 PM4/25/16

to rabbitm...@googlegroups.com

So you have long name support turned on in for the plugin, but not RabbitMQ. Just turn that setting off in the plugin and you should be good to go.

log : /var/log/rabbitmq/rab...@ip-10-20-11-33.log
sasl log : /var/log/rabbitmq/rab...@ip-10-20-11-33-sasl.log

Gavin M. Roy

unread,

Apr 25, 2016, 8:16:57 PM4/25/16

to rabbitm...@googlegroups.com

Alternatively, you could have other issues not limited to:

- Not having the same .erlang.cookie file in /var/lib/rabbitmq

- Security group issues preventing the nodes from talking to each other

Mike Brannigan

unread,

Apr 26, 2016, 3:38:42 PM4/26/16

to rabbitmq-users

I ensured that the contents of /var/lib/rabbitmq/.erlang.cookie matched across the three nodes; apparently it was different. I also disabled the long name support in the rabbitmq.config and the rabbitmq-env.config files. What ports should be permitted between the nodes? I am permitting the following between the RabbitMQ nodes:

TCP/4369, TCP/5555, TCP/5672, TCP/15672, TCP/25672

I also verified connectivity between the nodes on these ports with netcat.

=INFO REPORT==== 26-Apr-2016::19:36:14 ===

Starting RabbitMQ 3.5.6 on Erlang R16B03

Licensed under the MPL. See http://www.rabbitmq.com/

=INFO REPORT==== 26-Apr-2016::19:36:14 ===

node : rabbit@ip-10-20-13-140

home dir : /var/lib/rabbitmq

config file(s) : /etc/rabbitmq/rabbitmq.config

cookie hash : u6aPKvfa57KsVpYKu7cVpA==

log : /var/log/rabbitmq/rab...@ip-10-20-13-140.log

sasl log : /var/log/rabbitmq/rab...@ip-10-20-13-140-sasl.log

database dir : /var/lib/rabbitmq/mnesia/rabbit@ip-10-20-13-140

=INFO REPORT==== 26-Apr-2016::19:36:15 ===

autocluster: GET http://localhost:8500/v1/health/service/rabbitmq?passing&tag=staging

=INFO REPORT==== 26-Apr-2016::19:36:15 ===

autocluster: Response: [{ok,{{"HTTP/1.1",200,"OK"},

[{"date","Tue, 26 Apr 2016 19:36:15 GMT"},

{"content-length","1586"},

{"content-type","application/json"},

{"x-consul-index","194629"},

{"x-consul-knownleader","true"},

{"x-consul-lastcontact","0"}],

"[{\"Node\":{\"Node\":\"ip-10-20-11-93\",\"Address\":\"10.20.11.93\",\"TaggedAddresses\":{\"wan\":\"10.20.11.93\"},\"CreateIndex\":194413,\"ModifyIndex\":194621},\"Service\":{\"ID\":\"rabbitmq\",\"Service\":\"rabbitmq\",\"Tags\":[\"staging\"],\"Address\":\"\",\"Port\":5672,\"EnableTagOverride\":false,\"CreateIndex\":194429,\"ModifyIndex\":194621},\"Checks\":[{\"Node\":\"ip-10-20-11-93\",\"CheckID\":\"serfHealth\",\"Name\":\"Serf Health Status\",\"Status\":\"passing\",\"Notes\":\"\",\"Output\":\"Agent alive and reachable\",\"ServiceID\":\"\",\"ServiceName\":\"\",\"CreateIndex\":194413,\"ModifyIndex\":194413},{\"Node\":\"ip-10-20-11-93\",\"CheckID\":\"service:rabbitmq\",\"Name\":\"Service 'rabbitmq' check\",\"Status\":\"passing\",\"Notes\":\"RabbitMQ Auto-Cluster Plugin TTL Check\",\"Output\":\"\",\"ServiceID\":\"rabbitmq\",\"ServiceName\":\"rabbitmq\",\"CreateIndex\":194429,\"ModifyIndex\":194621}]},{\"Node\":{\"Node\":\"ip-10-20-12-108\",\"Address\":\"10.20.12.108\",\"TaggedAddresses\":{\"wan\":\"10.20.12.108\"},\"CreateIndex\":194428,\"ModifyIndex\":194629},\"Service\":{\"ID\":\"rabbitmq\",\"Service\":\"rabbitmq\",\"Tags\":[\"staging\"],\"Address\":\"\",\"Port\":5672,\"EnableTagOverride\":false,\"CreateIndex\":194447,\"ModifyIndex\":194629},\"Checks\":[{\"Node\":\"ip-10-20-12-108\",\"CheckID\":\"serfHealth\",\"Name\":\"Serf Health Status\",\"Status\":\"passing\",\"Notes\":\"\",\"Output\":\"Agent alive and reachable\",\"ServiceID\":\"\",\"ServiceName\":\"\",\"CreateIndex\":194428,\"ModifyIndex\":194428},{\"Node\":\"ip-10-20-12-108\",\"CheckID\":\"service:rabbitmq\",\"Name\":\"Service 'rabbitmq' check\",\"Status\":\"passing\",\"Notes\":\"RabbitMQ Auto-Cluster Plugin TTL Check\",\"Output\":\"\",\"ServiceID\":\"rabbitmq\",\"ServiceName\":\"rabbitmq\",\"CreateIndex\":194447,\"ModifyIndex\":194629}]}]"}}]

=INFO REPORT==== 26-Apr-2016::19:36:15 ===

autocluster: Registering node with consul

=INFO REPORT==== 26-Apr-2016::19:36:15 ===

autocluster: POST http://localhost:8500/v1/agent/service/register ["{\"ID\":\"rabbitmq\",\"Name\":\"rabbitmq\",\"Port\":5672,\"Tags\":[\"staging\"],\"Check\":{\"Notes\":\"RabbitMQ Auto-Cluster Plugin TTL Check\",\"TTL\":\"30s\"}}"]

=INFO REPORT==== 26-Apr-2016::19:36:15 ===

autocluster: Response: [{ok,{{"HTTP/1.1",200,"OK"},

[{"date","Tue, 26 Apr 2016 19:36:15 GMT"},

{"content-length","0"},

{"content-type","text/plain; charset=utf-8"}],

[]}}]

=INFO REPORT==== 26-Apr-2016::19:36:15 ===

autocluster: Registered node

=INFO REPORT==== 26-Apr-2016::19:36:15 ===

autocluster: Node is only node in the cluster

=ERROR REPORT==== 26-Apr-2016::19:36:15 ===

autocluster: Can not communicate with cluster nodes: ['rabbit@ip-10-20-11-93',

'rabbit@ip-10-20-12-108']

=INFO REPORT==== 26-Apr-2016::19:36:15 ===

Memory limit set to 1580MB of 3952MB total.

=INFO REPORT==== 26-Apr-2016::19:36:15 ===

Disk free limit set to 50MB

=INFO REPORT==== 26-Apr-2016::19:36:15 ===

Limiting to approx 924 file handles (829 sockets)

=INFO REPORT==== 26-Apr-2016::19:36:15 ===

FHC read buffering: ON

FHC write buffering: ON

=INFO REPORT==== 26-Apr-2016::19:36:15 ===

Priority queues enabled, real BQ is rabbit_variable_queue

=INFO REPORT==== 26-Apr-2016::19:36:15 ===

Management plugin: using rates mode 'basic'

=INFO REPORT==== 26-Apr-2016::19:36:15 ===

msg_store_transient: using rabbit_msg_store_ets_index to provide index

=INFO REPORT==== 26-Apr-2016::19:36:15 ===

msg_store_persistent: using rabbit_msg_store_ets_index to provide index

=INFO REPORT==== 26-Apr-2016::19:36:15 ===

started TCP Listener on [::]:5672

=INFO REPORT==== 26-Apr-2016::19:36:15 ===

autocluster: Starting Consul Health Check TTL Timer

=INFO REPORT==== 26-Apr-2016::19:36:15 ===

Management plugin started. Port: 15672

=INFO REPORT==== 26-Apr-2016::19:36:15 ===

Statistics database started.

=INFO REPORT==== 26-Apr-2016::19:36:15 ===

Server startup complete; 7 plugins started.

* rabbitmq_management

* rabbitmq_web_dispatch

* webmachine

* mochiweb

* autocluster

* rabbitmq_management_agent

* amqp_client

=INFO REPORT==== 26-Apr-2016::19:36:19 ===

accepting AMQP connection <0.291.0> (127.0.0.1:56070 -> 127.0.0.1:5672)

=INFO REPORT==== 26-Apr-2016::19:36:30 ===

autocluster: GET http://localhost:8500/v1/agent/check/pass/service%3Arabbitmq

=INFO REPORT==== 26-Apr-2016::19:36:30 ===

autocluster: Response: [{ok,{{"HTTP/1.1",200,"OK"},

[{"date","Tue, 26 Apr 2016 19:36:30 GMT"},

{"content-length","0"},

{"content-type","text/plain; charset=utf-8"}],

[]}}]

=INFO REPORT==== 26-Apr-2016::19:36:45 ===

autocluster: GET http://localhost:8500/v1/agent/check/pass/service%3Arabbitmq

=INFO REPORT==== 26-Apr-2016::19:36:45 ===

autocluster: Response: [{ok,{{"HTTP/1.1",200,"OK"},

[{"date","Tue, 26 Apr 2016 19:36:45 GMT"},

{"content-length","0"},

{"content-type","text/plain; charset=utf-8"}],

[]}}]

=INFO REPORT==== 26-Apr-2016::19:37:00 ===

autocluster: GET http://localhost:8500/v1/agent/check/pass/service%3Arabbitmq

=INFO REPORT==== 26-Apr-2016::19:37:00 ===

autocluster: Response: [{ok,{{"HTTP/1.1",200,"OK"},

[{"date","Tue, 26 Apr 2016 19:37:00 GMT"},

{"content-length","0"},

{"content-type","text/plain; charset=utf-8"}],

[]}}]

=INFO REPORT==== 26-Apr-2016::19:37:15 ===

autocluster: GET http://localhost:8500/v1/agent/check/pass/service%3Arabbitmq

=INFO REPORT==== 26-Apr-2016::19:37:15 ===

autocluster: Response: [{ok,{{"HTTP/1.1",200,"OK"},

[{"date","Tue, 26 Apr 2016 19:37:15 GMT"},

{"content-length","0"},

{"content-type","text/plain; charset=utf-8"}],

[]}}]

=INFO REPORT==== 26-Apr-2016::19:37:30 ===

autocluster: GET http://localhost:8500/v1/agent/check/pass/service%3Arabbitmq

=INFO REPORT==== 26-Apr-2016::19:37:30 ===

autocluster: Response: [{ok,{{"HTTP/1.1",200,"OK"},

[{"date","Tue, 26 Apr 2016 19:37:30 GMT"},

{"content-length","0"},

{"content-type","text/plain; charset=utf-8"}],

[]}}]

Thanks for your assistance!

On Monday, April 25, 2016 at 7:16:57 PM UTC-5, Gavin M. Roy wrote:

Alternatively, you could have other issues not limited to:

- Not having the same .erlang.cookie file in /var/lib/rabbitmq
- Security group issues preventing the nodes from talking to each other

On Mon, Apr 25, 2016 at 8:15 PM, Gavin M. Roy <gavi...@gmail.com> wrote:

So you have long name support turned on in for the plugin, but not RabbitMQ. Just turn that setting off in the plugin and you should be good to go.

On Mon, Apr 25, 2016 at 5:53 PM, Mike Brannigan <mikejbr...@gmail.com> wrote:

Ok. I installed 3.5.6 on the nodes and enabled debug logging:

=INFO REPORT==== 25-Apr-2016::21:51:19 ===
Starting RabbitMQ 3.5.6 on Erlang R16B03
Copyright (C) 2007-2015 Pivotal Software, Inc.
Licensed under the MPL. See http://www.rabbitmq.com/

=INFO REPORT==== 25-Apr-2016::21:51:19 ===
node : rabbit@ip-10-20-12-151
home dir : /var/lib/rabbitmq
config file(s) : /etc/rabbitmq/rabbitmq.config
cookie hash : uTHWz8EK0NSx8oXuvnUpEw==

log : /var/log/rabbitmq/rabbit@ip-10-20-12-151.log
sasl log : /var/log/rabbitmq/rabbit@ip-10-20-12-151-sasl.log

log : /var/log/rabbitmq/rabbit@ip-10-20-11-33.log
sasl log : /var/log/rabbitmq/rabbit@ip-10-20-11-33-sasl.log

Mike Brannigan

unread,

Apr 28, 2016, 9:26:46 AM4/28/16

to rabbitmq-users

Could it be that the nodes cannot resolve names like "ip-10-20-11-93"?

On Tuesday, April 26, 2016 at 2:38:42 PM UTC-5, Mike Brannigan wrote:

I ensured that the contents of /var/lib/rabbitmq/.erlang.cookie matched across the three nodes; apparently it was different. I also disabled the long name support in the rabbitmq.config and the rabbitmq-env.config files. What ports should be permitted between the nodes? I am permitting the following between the RabbitMQ nodes:

TCP/4369, TCP/5555, TCP/5672, TCP/15672, TCP/25672

I also verified connectivity between the nodes on these ports with netcat.

=INFO REPORT==== 26-Apr-2016::19:36:14 ===
Starting RabbitMQ 3.5.6 on Erlang R16B03
Copyright (C) 2007-2015 Pivotal Software, Inc.
Licensed under the MPL. See http://www.rabbitmq.com/

=INFO REPORT==== 26-Apr-2016::19:36:14 ===
node : rabbit@ip-10-20-13-140
home dir : /var/lib/rabbitmq
config file(s) : /etc/rabbitmq/rabbitmq.config
cookie hash : u6aPKvfa57KsVpYKu7cVpA==

log : /var/log/rabbitmq/rabbit@ip-10-20-13-140.log
sasl log : /var/log/rabbitmq/rabbit@ip-10-20-13-140-sasl.log

...

Michael Klishin

unread,

Apr 28, 2016, 10:18:03 AM4/28/16

to rabbitmq-users

Nodes need to be able to resolve each other's domains, yes. This is a runtime requirement.

...

Mike Brannigan

unread,

Apr 28, 2016, 1:21:42 PM4/28/16

to rabbitmq-users

How is this normally done with AWS, as the hostname that is assigned to boxes is in the format "ip-10-20-11-93" and isn't regularly resolvable. Is there a way to force the nodes to assume the format: ip-10-20-11-65.us-west-2.compute.internal? I believe that is resolvable within AWS.

...

Michael Klishin

unread,

Apr 28, 2016, 1:23:37 PM4/28/16

to rabbitm...@googlegroups.com, Mike Brannigan

See Hostname Changes on http://www.rabbitmq.com/ec2.html. Hostnames is something that you can control, despite EC2 defaults.

> > log : /var/log/rabbitmq/rab...@ip-10-20-13-140.log
> > sasl log : /var/log/rabbitmq/rab...@ip-10-20-13-140-sasl.log

> > On Mon, Apr 25, 2016 at 8:15 PM, Gavin M. Roy wrote:
> >
> > So you have long name support turned on in for the plugin, but not
> > RabbitMQ. Just turn that setting off in the plugin and you should be good
> > to go.
> >
> > On Mon, Apr 25, 2016 at 5:53 PM, Mike Brannigan

> > wrote:
> >
> > Ok. I installed 3.5.6 on the nodes and enabled debug logging:
> >
> > =INFO REPORT==== 25-Apr-2016::21:51:19 ===
> > Starting RabbitMQ 3.5.6 on Erlang R16B03
> > Copyright (C) 2007-2015 Pivotal Software, Inc.
> > Licensed under the MPL. See http://www.rabbitmq.com/
> >
> > =INFO REPORT==== 25-Apr-2016::21:51:19 ===
> > node : rabbit@ip-10-20-12-151
> > home dir : /var/lib/rabbitmq
> > config file(s) : /etc/rabbitmq/rabbitmq.config
> > cookie hash : uTHWz8EK0NSx8oXuvnUpEw==

> > log : /var/log/rabbitmq/rab...@ip-10-20-12-151.log
> > sasl log : /var/log/rabbitmq/rab...@ip-10-20-12-151-sasl.log

> --
> You received this message because you are subscribed to the Google Groups "rabbitmq-users"
> group.
> To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.

> To post to this group, send an email to rabbitm...@googlegroups.com.

> For more options, visit https://groups.google.com/d/optout.
>

--

Reply all

Reply to author

Forward