RE: Auto-Clustering Not working

[Skrzypek, Jonathan]

Hi,

Did a couple tests and still can't get the autoclustering working.
Config says :

[
{rabbit, [{tcp_listeners, [29010]},{loopback_users, []},
{cluster_nodes,
{[
'test_...@host1.gs.com',
'test_...@host2.gs.com',
'test_...@host3.gs.com'
], disc}},
{cluster_partition_handling,pause_minority}]},
{rabbitmq_management, [{listener, [{port, 29031}]}]}

].

rabbitmqctl report
Cluster status of node 'test_dev_1@d152549-004' ...
[{nodes,[{disc,['test_dev_1@host1']}]},
{running_nodes,['test_dev_1@host1']},
{cluster_name,<<"test_...@host1.gs.com">>},
{partitions,[]}]

./rabbitmqctl eval 'application:get_env(rabbit, cluster_nodes).'
{ok,{['test_...@host1.gs.com','test_...@host2.gs.com',
'test_...@host3.gs.com'],
disc}}

Tried to reset a few nodes to force the config to be re-read, no chance.
The erlang cookie is the same, file is 400.

On the same cluster, manual clustering works.
From host1 :
./rabbitmqctl join_cluster test_dev_2@host2
Clustering node 'test_dev_1@host1' with 'test_dev_2@host2' ...

./rabbitmqctl cluster_status
Cluster status of node 'test_dev_1@host1' ...
[{nodes,[{disc,['test_dev_1@host1','test_dev_2@host2']}]},
{running_nodes,['test_dev_2@host2','test_dev_1@host1']},
{cluster_name,<<"test_...@host2.gs.com">>},
{partitions,[]}]

Don't know what we are missing.
Any ideas ?



-----Original Message-----
From: Michael Klishin [mailto:mkli...@pivotal.io]
Sent: Friday, March 27, 2015 5:25 PM
To: jean-se...@rabbitmq.com; Dogra, Rajesh [Tech]; si...@rabbitmq.com
Cc: gs-rabbitmq-engineering
Subject: Re: Auto-Clustering Not working

On 27 March 2015 at 12:10:14, Dogra, Rajesh (rajesh...@gs.com) wrote:
> config file(s) : /home/rabbit/rabbit/dev_2/conf/rabbitmq.config

 * Are you sure that the file above is the one you're editing?
 * What does `rabbitmqctl report` output?
 *  What about rabbitmqctl eval 'application:get_env(rabbit, cluster_nodes).' ?
--
MK

Staff Software Engineer, Pivotal/RabbitMQ

[Jean-Sébastien Pédron]

On 31.03.2015 11:53, Skrzypek, Jonathan wrote:
> Hi,

Hi!

> [
> {rabbit, [{tcp_listeners, [29010]},{loopback_users, []},
> {cluster_nodes,
> {[
> 'test_...@host1.gs.com',
> 'test_...@host2.gs.com',
> 'test_...@host3.gs.com'

Your configuration lists nodes with a long name (note the FQDN), but...

> rabbitmqctl report
> Cluster status of node 'test_dev_1@d152549-004' ...
> [{nodes,[{disc,['test_dev_1@host1']}]},
> {running_nodes,['test_dev_1@host1']},

You seem to use short name here (note the lack of ".gs.com").

> On the same cluster, manual clustering works.
> From host1 :
> ./rabbitmqctl join_cluster test_dev_2@host2
> Clustering node 'test_dev_1@host1' with 'test_dev_2@host2' ...

Here, you are clustering using short names.

> Don't know what we are missing.
> Any ideas ?

As all your nodes are on the same domain, you should use short names
(the default). Try to remove ".gs.com" from your configuration file.

--
Jean-Sébastien Pédron
Pivotal / RabbitMQ

[Skrzypek, Jonathan]

Hi,

Sorry those are copy/paste mistake, I suspected the FQDN thingy so I changed from FQDN to shortnames but didn't paste the correct config.
I'm using shortnames everywhere.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send an email to rabbitm...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Jean-Sébastien Pédron]

On 31.03.2015 12:40, Skrzypek, Jonathan wrote:
> Sorry those are copy/paste mistake, I suspected the FQDN thingy so I changed from FQDN to shortnames but didn't paste the correct config.
> I'm using shortnames everywhere.

Automatic clustering only happens when a node is started for the very
first time (or just after a reset). If you started the node, then
stopped it, added the configuration and restarted it, automatic
clustering is not effective.

See the third paragraph in the automatic clustering documentation:
http://www.rabbitmq.com/clustering.html#auto-config

If you are using Debian for instance, where the common choice is to
automatically start a service during package install, then you need to
reset the node.

[Skrzypek, Jonathan]

Yes I know about the first start or reset thing, I call stop_app / reset / start_app when I perform the changes.
Just to make sure I stopped all the nodes, scrapped the mnesia on all of them, and started them again

Config :

[

{rabbit, [{tcp_listeners, [29010]},{loopback_users, []},
{cluster_nodes,
{[

'test_dev_1@host1',
'test_dev_2@host2',
'test_dev_3@host3'

], disc}},
{cluster_partition_handling,pause_minority}]},
{rabbitmq_management, [{listener, [{port, 29031}]}]}

].

Report gives me the following :

{rabbit,
[{auth_backends,[rabbit_auth_backend_internal]},
{auth_mechanisms,['PLAIN','AMQPLAIN']},
{backing_queue_module,rabbit_priority_queue},
{channel_max,0},
{cluster_keepalive_interval,10000},
{cluster_nodes,
{['test_dev_1@host1','test_dev_2@host2',
'test_dev_3@host3'],
disc}},

and further down :

Cluster status of node 'test_dev_2@host2' ...
[{nodes,[{disc,['test_dev_2@ host2']}]},
{running_nodes,['test_dev_2@ host2']},
{cluster_name,<<"test_...@host2.gs.com">>},
{partitions,[]}]

Clustername still has the FQDN, I assume it picks it up from hostname

The config definitely looks like it's being read and picked up, but the clustering doesn't happen.
The EPMD seems alright as well, able to spawn a shell from a node and ping another one ;

(testuser2@host2)1> net_adm:ping('test_dev_1@host1').
pong
(testuser2@ host2)2>

-----Original Message-----
From: rabbitm...@googlegroups.com [mailto:rabbitm...@googlegroups.com] On Behalf Of Jean-Sébastien Pédron
Sent: 31 March 2015 11:50
To: rabbitm...@googlegroups.com
Subject: Re: [rabbitmq-users] RE: Auto-Clustering Not working

[Michael Klishin]

 On 31 March 2015 at 15:21:11, Skrzypek, Jonathan (jonathan...@gs.com) wrote:
> Cluster status of node 'test_dev_2@host2' ...
> [{nodes,[{disc,['test_dev_2@ host2']}]},
> {running_nodes,['test_dev_2@ host2']},
> {cluster_name,<<"test_...@host2.gs.com">>},
> {partitions,[]}]

What does

rabbitmqctl report | grep -B 7 -A 7 cluster_nodes

output?

[Skrzypek, Jonathan]

cluster_nodes
{start_os_sup,false}]},

{rabbit,
[{auth_backends,[rabbit_auth_backend_internal]},
{auth_mechanisms,['PLAIN','AMQPLAIN']},
{backing_queue_module,rabbit_priority_queue},
{channel_max,0},
{cluster_keepalive_interval,10000},
{cluster_nodes,
{['test_dev_1@host1','test_dev_2@host2',
'test_dev_3@host3'],
disc}},

{cluster_partition_handling,pause_minority},
{collect_statistics,fine},
{collect_statistics_interval,5000},
{default_permissions,[<<".*">>,<<".*">>,<<".*">>]},

[Michael Klishin]

On 31 March 2015 at 16:04:59, Skrzypek, Jonathan (jonathan...@gs.com) wrote:
> {cluster_nodes,
> {['test_dev_1@host1','test_dev_2@host2',
> 'test_dev_3@host3'],
> disc}},

Thanks, may I ask you to post the entire `report` output? (feel free to edit out sensitive info such as users, vhosts, queue names)

Seeing what's in SASL log would be helpful, too.

[Skrzypek, Jonathan]

Sure.
The config is pretty vanilla since we're trying to isolate the issue.

Sasl log is empty, will try to enable logging

Report :
...
Status of node 'test_dev_2@host2' ...
[{pid,26939},
{running_applications,
[{rabbitmq_management,"RabbitMQ Management Console","3.5.0"},
{rabbitmq_web_dispatch,"RabbitMQ Web Dispatcher","3.5.0"},
{webmachine,"webmachine","1.10.3-rmq3.5.0-gite9359c7"},
{mochiweb,"MochiMedia Web Server","2.7.0-rmq3.5.0-git680dba8"},
{rabbitmq_management_agent,"RabbitMQ Management Agent","3.5.0"},
{rabbit,"RabbitMQ","3.5.0"},
{mnesia,"MNESIA CXC 138 12","4.9"},
{os_mon,"CPO CXC 138 46","2.2.12"},
{inets,"INETS CXC 138 49","5.9.5"},
{amqp_client,"RabbitMQ AMQP Client","3.5.0"},
{xmerl,"XML parser","1.3.3"},
{sasl,"SASL CXC 138 11","2.3.2"},
{stdlib,"ERTS CXC 138 10","1.19.2"},
{kernel,"ERTS CXC 138 10","2.16.2"}]},
{os,{unix,linux}},
{erlang_version,
"Erlang R16B01 (erts-5.10.2) [source] [64-bit] [smp:4:4] [async-threads:30] [hipe] [kernel-poll:false]\n"},
{memory,
[{total,37261232},
{connection_readers,0},
{connection_writers,0},
{connection_channels,0},
{connection_other,5600},
{queue_procs,2800},
{queue_slave_procs,0},
{plugins,816864},
{other_proc,4177704},
{mnesia,61376},
{mgmt_db,386480},
{msg_index,47168},
{other_ets,1310048},
{binary,182776},
{code,20348676},
{atom,711569},
{other_system,9210171}]},
{alarms,[]},
{listeners,[{clustering,56671,"::"},{amqp,29010,"0.0.0.0"}]},
{vm_memory_high_watermark,0.4},
{vm_memory_limit,13465588531},
{disk_free_limit,50000000},
{disk_free,39788703744},
{file_descriptors,
[{total_limit,65435},
{total_used,3},
{sockets_limit,58889},
{sockets_used,1}]},
{processes,[{limit,262144},{used,180}]},
{run_queue,0},
{uptime,3480}]

Cluster status of node 'test_dev_2@host2' ...
[{nodes,[{disc,['test_dev_2@host2']}]},

{running_nodes,['test_dev_2@host2']},
{cluster_name,<<"test_...@host2.gs.com">>},
{partitions,[]}]

Application environment of node 'test_dev_2@host2' ...
[{amqp_client,[{prefer_ipv6,false},{ssl_options,[]}]},
{inets,[]},
{kernel,[{error_logger,tty},{inet_default_connect_options,[{nodelay,true}]}]},
{mnesia,[{dir,"/local/data/rabbitmq//test_dev_2/mnesia"}]},
{mochiweb,[]},
{os_mon,
[{start_cpu_sup,false},
{start_disksup,false},
{start_memsup,false},

{start_os_sup,false}]},
{rabbit,
[{auth_backends,[rabbit_auth_backend_internal]},
{auth_mechanisms,['PLAIN','AMQPLAIN']},
{backing_queue_module,rabbit_priority_queue},
{channel_max,0},
{cluster_keepalive_interval,10000},

{cluster_nodes,
{['test_dev_1@host1','test_dev_2@host2',
'test_dev_3@host3'],
disc}},

{cluster_partition_handling,pause_minority},
{collect_statistics,fine},
{collect_statistics_interval,5000},
{default_permissions,[<<".*">>,<<".*">>,<<".*">>]},

{default_user,<<"guest">>},
{default_user_tags,[administrator]},
{default_vhost,<<"/">>},
{delegate_count,16},
{disk_free_limit,50000000},
{enabled_plugins_file,
"/local/data/rabbitmq//test_dev_2/conf/enabled_plugins"},
{error_logger,
{file,"/local/data/rabbitmq//test_dev_2/logs/rabbitmq.log"}},
{frame_max,131072},
{halt_on_upgrade_failure,true},
{handshake_timeout,10000},
{heartbeat,580},
{hipe_compile,false},
{hipe_modules,
[rabbit_reader,rabbit_channel,gen_server2,rabbit_exchange,
rabbit_command_assembler,rabbit_framing_amqp_0_9_1,rabbit_basic,
rabbit_event,lists,queue,priority_queue,rabbit_router,rabbit_trace,
rabbit_misc,rabbit_binary_parser,rabbit_exchange_type_direct,
rabbit_guid,rabbit_net,rabbit_amqqueue_process,
rabbit_variable_queue,rabbit_binary_generator,rabbit_writer,
delegate,gb_sets,lqueue,sets,orddict,rabbit_amqqueue,
rabbit_limiter,gb_trees,rabbit_queue_index,
rabbit_exchange_decorator,gen,dict,ordsets,file_handle_cache,
rabbit_msg_store,array,rabbit_msg_store_ets_index,rabbit_msg_file,
rabbit_exchange_type_fanout,rabbit_exchange_type_topic,mnesia,
mnesia_lib,rpc,mnesia_tm,qlc,sofs,proplists,credit_flow,pmon,
ssl_connection,tls_connection,ssl_record,tls_record,gen_fsm,ssl]},
{log_levels,[{connection,info}]},
{loopback_users,[]},
{mnesia_table_loading_timeout,30000},
{msg_store_file_size_limit,16777216},
{msg_store_index_module,rabbit_msg_store_ets_index},
{plugins_dir,
"/gns/software/infra/messaging/rabbitmq/rabbitmq-server-snapshot/plugins"},
{plugins_expand_dir,
"/local/data/rabbitmq//test_dev_2/mnesia/plugin-expand"},
{queue_index_embed_msgs_below,4096},
{queue_index_max_journal_entries,65536},
{reverse_dns_lookups,false},
{sasl_error_logger,
{file,"/local/data/rabbitmq//test_dev_2/logs/rabbitmq-sasl.log"}},
{server_properties,[]},
{ssl_allow_poodle_attack,false},
{ssl_apps,[asn1,crypto,public_key,ssl]},
{ssl_cert_login_from,distinguished_name},
{ssl_handshake_timeout,5000},
{ssl_listeners,[]},
{ssl_options,[]},
{tcp_listen_options,
[binary,
{packet,raw},
{reuseaddr,true},
{backlog,128},
{nodelay,true},
{linger,{true,0}},
{exit_on_close,false}]},
{tcp_listeners,[29010]},
{trace_vhosts,[]},
{vm_memory_high_watermark,0.4},
{vm_memory_high_watermark_paging_ratio,0.5}]},
{rabbitmq_management,
[{http_log_dir,none},
{listener,[{port,29031}]},
{load_definitions,none},
{rates_mode,basic},
{sample_retention_policies,
[{global,[{605,5},{3660,60},{29400,600},{86400,1800}]},
{basic,[{605,5},{3600,60}]},
{detailed,[{10,5}]}]}]},
{rabbitmq_management_agent,[]},
{rabbitmq_web_dispatch,[]},
{sasl,[{errlog_type,error},{sasl_error_logger,false}]},
{stdlib,[]},
{webmachine,[{error_handler,rabbit_webmachine_error_handler}]},
{xmerl,[]}]

Connections:

Channels:

Queues on /:

Exchanges on /:
name type durable auto_delete internal arguments policy
direct true false false []
amq.direct direct true false false []
amq.fanout fanout true false false []
amq.headers headers true false false []
amq.match headers true false false []
amq.rabbitmq.log topic true false true []
amq.rabbitmq.trace topic true false true []
amq.topic topic true false false []

Bindings on /:

Consumers on /:

Permissions on /:
user configure write read
guest .* .* .*

Policies on /:

Parameters on /:

-----Original Message-----
From: Michael Klishin [mailto:mkli...@pivotal.io]

Sent: 31 March 2015 14:10
To: rabbitm...@googlegroups.com; Skrzypek, Jonathan [Tech]
Subject: RE: [rabbitmq-users] RE: Auto-Clustering Not working

[Michael Klishin]

On 31 March 2015 at 16:18:15, Skrzypek, Jonathan (jonathan...@gs.com) wrote:
> Sasl log is empty, will try to enable logging

Autoclustering should leave messages in the regular log. Please post it as well. 

[Skrzypek, Jonathan]

No messages regarding clustering whatsoever in the regular logs, logging params are default, do I need to add a flag ?

-----Original Message-----
From: Michael Klishin [mailto:mkli...@pivotal.io]
Sent: 31 March 2015 14:21
To: rabbitm...@googlegroups.com; Skrzypek, Jonathan [Tech]
Subject: RE: [rabbitmq-users] RE: Auto-Clustering Not working

[Michael Klishin]

On 31 March 2015 at 16:33:42, Skrzypek, Jonathan (jonathan...@gs.com) wrote:
> No messages regarding clustering whatsoever in the regular
> logs, logging params are default, do I need to add a flag ?

No, the logging is done using info and warn levels.

This likely means that your local node database is initialised.

Can you please clear the logs, remove the database on all nodes and start
them one by one, posting all logs (in the order of start)? 

[Skrzypek, Jonathan]

Ok, node3 started first :

=INFO REPORT==== 31-Mar-2015::14:52:37 ===
Starting RabbitMQ 3.5.0 on Erlang R16B01
Copyright (C) 2007-2014 GoPivotal, Inc.
Licensed under the MPL. See http://www.rabbitmq.com/

=INFO REPORT==== 31-Mar-2015::14:52:37 ===
node : test_dev_3@host3
home dir : /home/rabbitadm
config file(s) : /local/data/rabbitmq/test_dev_3/conf/rabbitmq.config
cookie hash : pQd/hWnOwVFO2kgjQVqlgQ==
log : /local/data/rabbitmq//test_dev_3/logs/rabbitmq.log
sasl log : /local/data/rabbitmq//test_dev_3/logs/rabbitmq-sasl.log
database dir : /local/data/rabbitmq/test_dev_3/mnesia

=WARNING REPORT==== 31-Mar-2015::14:52:37 ===
Kernel poll (epoll, kqueue, etc) is disabled. Throughput and CPU utilization may worsen.

=INFO REPORT==== 31-Mar-2015::14:52:38 ===
Limiting to approx 65435 file handles (58889 sockets)

=INFO REPORT==== 31-Mar-2015::14:52:38 ===
Priority queues enabled, real BQ is rabbit_variable_queue

=INFO REPORT==== 31-Mar-2015::14:52:38 ===
Memory limit set to 6377MB of 15944MB total.

=INFO REPORT==== 31-Mar-2015::14:52:38 ===
Disk free limit set to 50MB

=INFO REPORT==== 31-Mar-2015::14:52:38 ===
msg_store_transient: using rabbit_msg_store_ets_index to provide index

=INFO REPORT==== 31-Mar-2015::14:52:38 ===
msg_store_persistent: using rabbit_msg_store_ets_index to provide index

=WARNING REPORT==== 31-Mar-2015::14:52:38 ===
msg_store_persistent: rebuilding indices from scratch

=INFO REPORT==== 31-Mar-2015::14:52:38 ===
Adding vhost '/'

=INFO REPORT==== 31-Mar-2015::14:52:38 ===
Creating user 'guest'

=INFO REPORT==== 31-Mar-2015::14:52:38 ===
Setting user tags for user 'guest' to [administrator]

=INFO REPORT==== 31-Mar-2015::14:52:38 ===
Setting permissions for 'guest' in '/' to '.*', '.*', '.*'

=INFO REPORT==== 31-Mar-2015::14:52:38 ===
started TCP Listener on 0.0.0.0:29010

=INFO REPORT==== 31-Mar-2015::14:52:38 ===
Management plugin started. Port: 29031

=INFO REPORT==== 31-Mar-2015::14:52:38 ===
Statistics database started.

=INFO REPORT==== 31-Mar-2015::14:52:38 ===
Server startup complete; 6 plugins started.
* rabbitmq_management
* rabbitmq_web_dispatch
* webmachine
* mochiweb
* rabbitmq_management_agent
* amqp_client

Then node 2

=INFO REPORT==== 31-Mar-2015::14:52:56 ===
Starting RabbitMQ 3.5.0 on Erlang R16B01
Copyright (C) 2007-2014 GoPivotal, Inc.
Licensed under the MPL. See http://www.rabbitmq.com/

=INFO REPORT==== 31-Mar-2015::14:52:56 ===
node : test_dev_2@host2
home dir : /home/rabbitadm
config file(s) : /local/data/rabbitmq/test_dev_2/conf/rabbitmq.config
cookie hash : pQd/hWnOwVFO2kgjQVqlgQ==
log : /local/data/rabbitmq//test_dev_2/logs/rabbitmq.log
sasl log : /local/data/rabbitmq//test_dev_2/logs/rabbitmq-sasl.log
database dir : /local/data/rabbitmq/test_dev_2/mnesia

=WARNING REPORT==== 31-Mar-2015::14:52:56 ===
Kernel poll (epoll, kqueue, etc) is disabled. Throughput and CPU utilization may worsen.

=INFO REPORT==== 31-Mar-2015::14:52:57 ===
Limiting to approx 65435 file handles (58889 sockets)

=INFO REPORT==== 31-Mar-2015::14:52:57 ===
Priority queues enabled, real BQ is rabbit_variable_queue

=INFO REPORT==== 31-Mar-2015::14:52:57 ===
Memory limit set to 12841MB of 32104MB total.

=INFO REPORT==== 31-Mar-2015::14:52:57 ===
Disk free limit set to 50MB

=INFO REPORT==== 31-Mar-2015::14:52:57 ===
msg_store_transient: using rabbit_msg_store_ets_index to provide index

=INFO REPORT==== 31-Mar-2015::14:52:57 ===
msg_store_persistent: using rabbit_msg_store_ets_index to provide index

=WARNING REPORT==== 31-Mar-2015::14:52:57 ===
msg_store_persistent: rebuilding indices from scratch

=INFO REPORT==== 31-Mar-2015::14:52:57 ===
Adding vhost '/'

=INFO REPORT==== 31-Mar-2015::14:52:57 ===
Creating user 'guest'

=INFO REPORT==== 31-Mar-2015::14:52:57 ===
Setting user tags for user 'guest' to [administrator]

=INFO REPORT==== 31-Mar-2015::14:52:57 ===
Setting permissions for 'guest' in '/' to '.*', '.*', '.*'

=INFO REPORT==== 31-Mar-2015::14:52:57 ===
started TCP Listener on 0.0.0.0:29010

=INFO REPORT==== 31-Mar-2015::14:52:57 ===
Management plugin started. Port: 29031

=INFO REPORT==== 31-Mar-2015::14:52:57 ===
Statistics database started.

=INFO REPORT==== 31-Mar-2015::14:52:57 ===
Server startup complete; 6 plugins started.
* rabbitmq_management
* rabbitmq_web_dispatch
* webmachine
* mochiweb
* rabbitmq_management_agent
* amqp_client

Then node 1

=INFO REPORT==== 31-Mar-2015::14:53:19 ===
Starting RabbitMQ 3.5.0 on Erlang R16B01
Copyright (C) 2007-2014 GoPivotal, Inc.
Licensed under the MPL. See http://www.rabbitmq.com/

=INFO REPORT==== 31-Mar-2015::14:53:19 ===
node : test_dev_1@host1
home dir : /home/rabbitadm
config file(s) : /local/data/rabbitmq/test_dev_1/conf/rabbitmq.config
cookie hash : pQd/hWnOwVFO2kgjQVqlgQ==
log : /local/data/rabbitmq//test_dev_1/logs/rabbitmq.log
sasl log : /local/data/rabbitmq//test_dev_1/logs/rabbitmq-sasl.log
database dir : /local/data/rabbitmq/test_dev_1/mnesia

=WARNING REPORT==== 31-Mar-2015::14:53:19 ===
Kernel poll (epoll, kqueue, etc) is disabled. Throughput and CPU utilization may worsen.

=INFO REPORT==== 31-Mar-2015::14:53:20 ===
Limiting to approx 65435 file handles (58889 sockets)

=INFO REPORT==== 31-Mar-2015::14:53:20 ===
Priority queues enabled, real BQ is rabbit_variable_queue

=INFO REPORT==== 31-Mar-2015::14:53:20 ===
Memory limit set to 12841MB of 32104MB total.

=INFO REPORT==== 31-Mar-2015::14:53:20 ===
Disk free limit set to 50MB

=INFO REPORT==== 31-Mar-2015::14:53:20 ===
msg_store_transient: using rabbit_msg_store_ets_index to provide index

=INFO REPORT==== 31-Mar-2015::14:53:20 ===
msg_store_persistent: using rabbit_msg_store_ets_index to provide index

=WARNING REPORT==== 31-Mar-2015::14:53:20 ===
msg_store_persistent: rebuilding indices from scratch

=INFO REPORT==== 31-Mar-2015::14:53:20 ===
Adding vhost '/'

=INFO REPORT==== 31-Mar-2015::14:53:20 ===
Creating user 'guest'

=INFO REPORT==== 31-Mar-2015::14:53:20 ===
Setting user tags for user 'guest' to [administrator]

=INFO REPORT==== 31-Mar-2015::14:53:20 ===
Setting permissions for 'guest' in '/' to '.*', '.*', '.*'

=INFO REPORT==== 31-Mar-2015::14:53:20 ===
started TCP Listener on 0.0.0.0:29010

=INFO REPORT==== 31-Mar-2015::14:53:20 ===
Management plugin started. Port: 29031

=INFO REPORT==== 31-Mar-2015::14:53:20 ===
Statistics database started.

=INFO REPORT==== 31-Mar-2015::14:53:21 ===
Server startup complete; 6 plugins started.
* rabbitmq_management
* rabbitmq_web_dispatch
* webmachine
* mochiweb
* rabbitmq_management_agent
* amqp_client

Thanks for you help!

-----Original Message-----
From: Michael Klishin [mailto:mkli...@pivotal.io]
Sent: 31 March 2015 14:40
To: rabbitm...@googlegroups.com; Skrzypek, Jonathan [Tech]
Subject: RE: [rabbitmq-users] RE: Auto-Clustering Not working

[Michael Klishin]

I've just tried with two nodes, alice and bob, alice started first:
https://gist.github.com/michaelklishin/1c6fbafc7da243247ec4

Config file (the same on both nodes) and environment configurations:

https://gist.github.com/michaelklishin/4d0009d492f364bfbc17
https://gist.github.com/michaelklishin/f6bd51929d2eb5ff6e22

This is with 3.5.0.

Would it be possible to have gists of your configuration (in a similar way to mine)
side by side with logs?  

> --
> You received this message because you are subscribed to the Google Groups "rabbitmq-users"
> group.
> To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
> To post to this group, send an email to rabbitm...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

[Skrzypek, Jonathan]

alice always starts first :-)

I can't push things to gist, apologies.
Could I send files over ?

I checked your traces and as you said there is logging around auto clustering that I don't get in my case.

[Michael Klishin]

 On 31 March 2015 at 19:18:06, Skrzypek, Jonathan (jonathan...@gs.com) wrote:
> I can't push things to gist, apologies.
> Could I send files over ?
>
> I checked your traces and as you said there is logging around auto
> clustering that I don't get in my case.

Sure, feel free to send the files, including off list.

[Michael Klishin]

On 31 March 2015 at 20:11:22, Michael Klishin (mkli...@pivotal.io) wrote:
> Sure, feel free to send the files, including off list.

I've received Jonathan's env information off-list.

The rest of this week for us seems to be taken by putting our more IT-induced fires and doing
a release (3.5.1), so I may be able to take a look at that early next week.

 Hopefully my provided example was helpful in some way.

[Skrzypek, Jonathan]

Hello,

Have you been able to take a look at this issue ?

-----Original Message-----
From: Michael Klishin [mailto:mkli...@pivotal.io]
Sent: 01 April 2015 13:15
To: Skrzypek, Jonathan [Tech]
Cc: rabbitm...@googlegroups.com
Subject: RE: [rabbitmq-users] RE: Auto-Clustering Not working

[alessandro meyer]

Hey List

I have the very same problem and i've been pointed to this very thread from my github issue (https://github.com/rabbitmq/rabbitmq-server/issues/102).

I'm on version 3.4.3 and as far as I can see upgrading doesn't really help. 

I'm eagerly awaiting news now. :-)

Cheers
Alessandro

[Skrzypek, Jonathan]

Hi,

 

Any news on this issue now 3.5.1 is out ?
Michael, have you been able to reproduce the issue or is it always working first hand in your case ?

Regards

[Michael Klishin]

On 13 April 2015 at 15:25:29, Skrzypek, Jonathan (jonathan...@gs.com) wrote:
> Any news on this issue now 3.5.1 is out ?
> Michael, have you been able to reproduce the issue or is it always
> working first hand in your case ?

We weren't able to reproduce this. I've posted my gists.

[Skrzypek, Jonathan]

Yes I went through your files, but you probably noticed that the config I sent over is pretty much the same.

My problem is that I don't have much ways of troubleshooting this, there's not a single line about auto-clustering in the logs, even when it's the first time
the brokers start or after a reset.
And rabbitmqctl report indicates that the brokers know about their config and the fact that they should be clustered, but nothing is happening.

-----Original Message-----
From: Michael Klishin [mailto:mkli...@pivotal.io]

Sent: 13 April 2015 13:44
To: rabbitm...@googlegroups.com; Skrzypek, Jonathan [Tech]

Subject: RE: [rabbitmq-users] RE: Auto-Clustering Not working

[Jean-Sébastien Pédron]

On 13.04.2015 14:49, Skrzypek, Jonathan wrote:
> My problem is that I don't have much ways of troubleshooting this,
> there's not a single line about auto-clustering in the logs, even
> when it's the first time the brokers start or after a reset.

Hi!

Wether auto-clustering succeed or not is logged.

Here is what I get when trying to reproduce this with 3.5.0 on RHEL 6.5:

1) rabbit@host2 is started first:

=WARNING REPORT==== 14-Apr-2015::11:12:45 ===
Could not auto-cluster with rabbit@host1: {badrpc,nodedown}

=WARNING REPORT==== 14-Apr-2015::11:12:45 ===
Could not auto-cluster with rabbit@host3: {badrpc,nodedown}

=WARNING REPORT==== 14-Apr-2015::11:12:45 ===
Could not find any node for auto-clustering from: ['rabbit@host1',
'rabbit@host2',
'rabbit@host3']
Starting blank node...

2) rabbit@host3 is started:

=WARNING REPORT==== 14-Apr-2015::11:14:14 ===
Could not auto-cluster with rabbit@host1: {badrpc,nodedown}

=INFO REPORT==== 14-Apr-2015::11:14:14 ===
Node ''rabbit@host2'' selected for auto-clustering

If there is something wrong with the cluster_nodes configuration,
RabbitMQ logs something too.

Could you please:
1. remove all log files and ~rabbitmq/mnesia content
2. start the first node
3. run the following command on this same host and post the output:

rabbitmqctl eval "application:get_env(rabbit, cluster_nodes)."

[Jean-Sébastien Pédron]

On 14.04.2015 11:44, Jean-Sébastien Pédron wrote:
> Could you please:
> 1. remove all log files and ~rabbitmq/mnesia content
> 2. start the first node
> 3. run the following command on this same host and post the output:
> rabbitmqctl eval "application:get_env(rabbit, cluster_nodes)."

I'm also interested in the output from the following commands:

1. rabbitmqctl eval "mnesia:system_info(directory)."

2. Reset your cluster as if you wanted to reproduce the problem, then:
ls -la <directory returned by command #1>

[Skrzypek, Jonathan]

Ok,

Starting first node for the first time, clustering didn't happen, but still nothing in the logs.
Eval :

rabbitmqctl eval "application:get_env(rabbit,cluster_nodes)."

{ok,{['test_dev_1@host1','test_dev_2@host2',
'test_dev_3@host3'],
disc}}

rabbitmqctl test_dev_1 eval "mnesia:system_info(directory)."
"/local/data/rabbitmq/test_dev_1/mnesia"

rabbitmqctl test_dev_1 reset
resetting node 'test_dev_1@host1' ...

ls -lisa /local/data/rabbitmq/test_dev_1/mnesia
total 16
2237569 4 drwxrwxr-x 2 rabbitadm rabbitadm 4096 Apr 16 16:26 .
2237565 4 drwxrwxr-x 5 rabbitadm rabbitadm 4096 Apr 7 11:35 ..
2237570 4 -rw-rw-r-- 1 rabbitadm rabbitadm 57 Apr 16 16:26 cluster_nodes.config
2237574 4 -rw-rw-r-- 1 rabbitadm rabbitadm 28 Apr 16 16:26 nodes_running_at_shutdown

cat /local/data/rabbitmq/test_dev_1/mnesia/cluster_nodes.config
{['test_dev_1@host1'],['test_dev_1@host1']}.

Interesting that it lists twice the same node.

cat /local/data/rabbitmq/test_dev_1/mnesia/nodes_running_at_shutdown
['test_dev_1@host1'].

Thanks for your help

[Norberto Meijome]

Hi!
Going back to the basics ( apologies if I missed this in earlier emails),
- you have epmd running on all nodes
- same Erlang cookie in all nodes
- all nodes can talk to each other via the port defined for epmd ( I don't recall whether TCP/5672 is also needed intra node)

Re. Same node listed twice, that's OK (expected nodes in cluster, current nodes in cluster)

Cheers,
Beto

[Jean-Sébastien Pédron]

On 16.04.2015 17:33, Skrzypek, Jonathan wrote:
> Starting first node for the first time, clustering didn't happen, but
> still nothing in the logs.

I went through your logs again and I saw that the PID file is created in
the Mnesia directory:
+ RABBITMQ_PID_FILE=/local/data/rabbitmq//test_dev_3/mnesia/.pid

Do you set this variable in the environment?

Could you please choose a location outside of the Mnesia directory and
try again?

[Jean-Sébastien Pédron]

If you do not set this variable, could you please check the exact value
of RABBITMQ_MNESIA_DIR in your configuration? In particular, does it
have a trailing '/'? Your logs say it doesn't but just to be sure.

Likewise, if there is a trailing slash, remove it and try again.

[Skrzypek, Jonathan]

Ok,

There is something very weird going on with those directories.
I managed to have the autoclustering trigger (and I see the output in the logs).

We do set RABBITMQ_MNESIA_DIR, without trailing /.

I tried a couple combinations, and sometimes the pid file doesn't get created at all.

A couple tests with a simple script :

export RABBITMQ_PID_FILE=/local/data/rabbitmq/.pid
/gns/mw/lang/erlang/Erlang-R16B01-11/bin/erl -pa /gns/software/infra/messaging/rabbitmq/rabbitmq-server-qa/ebin -noinput -s rabbit boot -sname test_dev_1@host1 -boot start_sasl -config /local/data/rabbitmq/test_dev_1/conf/rabbitmq.config +W w +K true +A30 +P 1048576 -kernel inet_default_connect_options '[{nodelay,true}]' -setcookie ABCDEFGHIJKLMNOPQRST -sasl errlog_type error -sasl sasl_error_logger false -rabbit error_logger '{file,"/local/data/rabbitmq/test_dev_1/logs/rabbitmq.log"}' -rabbit sasl_error_logger '{file,"/local/data/rabbitmq/test_dev_1/logs/rabbitmq-sasl.log"}' -rabbit enabled_plugins_file '"/local/data/rabbitmq/test_dev_1/conf/enabled_plugins"' -rabbit plugins_dir '"/gns/software/infra/messaging/rabbitmq/rabbitmq-server-qa/plugins"' -rabbit plugins_expand_dir '"/local/data/rabbitmq/test_dev_1/mnesia/plugin-expand"' -os_mon start_cpu_sup false -os_mon start_disksup false -os_mon start_memsup false -mnesia dir '"/tmp/mnesia"'

Works but no pid file got created either in RABBITMQ_PID_FILE nor in /tmp/mnesia


export RABBITMQ_PID_FILE=/local/data/rabbitmq/.pid
/gns/mw/lang/erlang/Erlang-R16B01-11/bin/erl -pa /gns/software/infra/messaging/rabbitmq/rabbitmq-server-qa/ebin -noinput -s rabbit boot -sname test_dev_1@host1 -boot start_sasl -config /local/data/rabbitmq/test_dev_1/conf/rabbitmq.config +W w +K true +A30 +P 1048576 -kernel inet_default_connect_options '[{nodelay,true}]' -setcookie ABCDEFGHIJKLMNOPQRST -sasl errlog_type error -sasl sasl_error_logger false -rabbit error_logger '{file,"/local/data/rabbitmq/test_dev_1/logs/rabbitmq.log"}' -rabbit sasl_error_logger '{file,"/local/data/rabbitmq/test_dev_1/logs/rabbitmq-sasl.log"}' -rabbit enabled_plugins_file '"/local/data/rabbitmq/test_dev_1/conf/enabled_plugins"' -rabbit plugins_dir '"/gns/software/infra/messaging/rabbitmq/rabbitmq-server-qa/plugins"' -rabbit plugins_expand_dir '"/local/data/rabbitmq/test_dev_1/mnesia/plugin-expand"' -os_mon start_cpu_sup false -os_mon start_disksup false -os_mon start_memsup false -mnesia dir '"/local/data/rabbitmq/test_dev_1/mnesia"'

Doesn't work, no pid file got created

export RABBITMQ_PID_FILE=/local/data/rabbitmq/.pid
/gns/mw/lang/erlang/Erlang-R16B01-11/bin/erl -pa /gns/software/infra/messaging/rabbitmq/rabbitmq-server-qa/ebin -noinput -s rabbit boot -sname test_dev_1@host1 -boot start_sasl -config /local/data/rabbitmq/test_dev_1/conf/rabbitmq.config +W w +K true +A30 +P 1048576 -kernel inet_default_connect_options '[{nodelay,true}]' -setcookie ABCDEFGHIJKLMNOPQRST -sasl errlog_type error -sasl sasl_error_logger false -rabbit error_logger '{file,"/local/data/rabbitmq/test_dev_1/logs/rabbitmq.log"}' -rabbit sasl_error_logger '{file,"/local/data/rabbitmq/test_dev_1/logs/rabbitmq-sasl.log"}' -rabbit enabled_plugins_file '"/local/data/rabbitmq/test_dev_1/conf/enabled_plugins"' -rabbit plugins_dir '"/gns/software/infra/messaging/rabbitmq/rabbitmq-server-qa/plugins"' -rabbit plugins_expand_dir '"/local/data/rabbitmq/test_dev_1/mnesia/plugin-expand"' -os_mon start_cpu_sup false -os_mon start_disksup false -os_mon start_memsup false -mnesia dir '"/local/data/rabbitmq/test_dev_1/mnesia2"'

Works, but no pid file got created

export RABBITMQ_PID_FILE=/local/data/rabbitmq/.pid
/gns/mw/lang/erlang/Erlang-R16B01-11/bin/erl -pa /gns/software/infra/messaging/rabbitmq/rabbitmq-server-qa/ebin -noinput -s rabbit boot -sname test_dev_1@host1 -boot start_sasl -config /local/data/rabbitmq/test_dev_1/conf/rabbitmq.config +W w +K true +A30 +P 1048576 -kernel inet_default_connect_options '[{nodelay,true}]' -setcookie ABCDEFGHIJKLMNOPQRST -sasl errlog_type error -sasl sasl_error_logger false -rabbit error_logger '{file,"/local/data/rabbitmq/test_dev_1/logs/rabbitmq.log"}' -rabbit sasl_error_logger '{file,"/local/data/rabbitmq/test_dev_1/logs/rabbitmq-sasl.log"}' -rabbit enabled_plugins_file '"/local/data/rabbitmq/test_dev_1/conf/enabled_plugins"' -rabbit plugins_dir '"/gns/software/infra/messaging/rabbitmq/rabbitmq-server-qa/plugins"' -rabbit plugins_expand_dir '"/local/data/rabbitmq/test_dev_1/mnesiaplugins/plugin-expand"' -os_mon start_cpu_sup false -os_mon start_disksup false -os_mon start_memsup false -mnesia dir '"/local/data/rabbitmq/test_dev_1/mnesia"'

Works, but no pid file got created

Changing the mnesia directory made it work, which had me thinking it was the root cause, however, it worked because the directory was different than plugins_expand_dir.
If directories are different for plugins_expand_dir and mnesia dir, it works fine.
I don't know how those directories are checked or taken into account internally, but there's definitely something fishy here

Thanks

-----Original Message-----
From: rabbitm...@googlegroups.com [mailto:rabbitm...@googlegroups.com] On Behalf Of Jean-Sébastien Pédron
Sent: 17 April 2015 10:46
To: rabbitm...@googlegroups.com
Subject: Re: [rabbitmq-users] RE: Auto-Clustering Not working

[Michael Klishin]

On 17 April 2015 at 15:49:46, Skrzypek, Jonathan (jonathan...@gs.com) wrote:
> Changing the mnesia directory made it work, which had me thinking
> it was the root cause, however, it worked because the directory
> was different than plugins_expand_dir.
> If directories are different for plugins_expand_dir and mnesia
> dir, it works fine.
> I don't know how those directories are checked or taken into account
> internally, but there's definitely something fishy here

Jonathan,

So, just to clarify: autoclustering does not work for you when Mnesia dir is set to a specific value
but does work when you relocate it somewhere else?

Can you send us the contents of that directory and `ls -lha` (or similar) output that demonstrates permissions
on it off-list? Please also mention if rabbitmq may be running under a non-standard user.

[Jean-Sébastien Pédron]

On 17.04.2015 14:49, Skrzypek, Jonathan wrote:
> A couple tests with a simple script :
>
> export RABBITMQ_PID_FILE=/local/data/rabbitmq/.pid

> /gns/mw/lang/erlang/Erlang-R16B01-11/bin/erl (..)

The PID file is created by the "rabbitmq-server" startup script, not
RabbitMQ itself. If you execute RabbitMQ by using erl(1) directlry, it's
expected.

RabbitMQ itself doesn't need the PID file. It's useful to init scripts
and tools such as "rabbitmqctl wait" (itself used by init scripts).

> Changing the mnesia directory made it work, which had me thinking it
> was the root cause, however, it worked because the directory was
> different than plugins_expand_dir. If directories are different for
> plugins_expand_dir and mnesia dir, it works fine. I don't know how
> those directories are checked or taken into account internally, but
> there's definitely something fishy here

Auto-clustering is triggered only when the node is "pristine". To
determine this, RabbitMQ checks the content of the Mnesia directory. If
it finds anything beside "cluster_nodes.config" and
"nodes_running_at_shutdown", it considers the node to be already
initialized and skip auto-clustering.

So you can't share the Mnesia directory with any other purpose.

We should add checks for this.

[Jean-Sébastien Pédron]

On 17.04.2015 15:10, Jean-Sébastien Pédron wrote:
> On 17.04.2015 14:49, Skrzypek, Jonathan wrote:
>> A couple tests with a simple script :
>>
>> export RABBITMQ_PID_FILE=/local/data/rabbitmq/.pid
>> /gns/mw/lang/erlang/Erlang-R16B01-11/bin/erl (..)
>
> The PID file is created by the "rabbitmq-server" startup script, not
> RabbitMQ itself. If you execute RabbitMQ by using erl(1) directlry, it's
> expected.

... that the PID file is not created (just to be clear :).

> So you can't share the Mnesia directory with any other purpose.

Ie. no PID file inside it or shared plugins_expand_dir.

[Skrzypek, Jonathan]

Ok it makes sense now, thanks.
Will change our environment to not have those in the mnesia dir.

However, the scripts you ship have the following ;

[ "x" = "x$RABBITMQ_PID_FILE" ] && RABBITMQ_PID_FILE=${RABBITMQ_MNESIA_DIR}.pid

[ "x" = "x$RABBITMQ_PLUGINS_EXPAND_DIR" ] && RABBITMQ_PLUGINS_EXPAND_DIR=${RABBITMQ_MNESIA_BASE}/${RABBITMQ_NODENAME}-plugins-expand

https://github.com/rabbitmq/rabbitmq-server/blob/master/scripts/rabbitmq-env

Thanks for the leads around mnesia dir, helped me to find the issue !

-----Original Message-----
From: Jean-Sébastien Pédron [mailto:jean-se...@rabbitmq.com]
Sent: 17 April 2015 14:11
To: Skrzypek, Jonathan [Tech]; rabbitm...@googlegroups.com
Subject: Re: [rabbitmq-users] RE: Auto-Clustering Not working

[Skrzypek, Jonathan]

Hi,

I've come across another issue regarding auto-clustering.
Auto-clustering on my clusters doesn't work when they are first started, but if I perform a stop_app/reset/start_app, it works fine.
Which makes no sense to me as I thought a start after a reset was the same as first starting.

We are running 3.5.0
My mnesia directory isn't shared with other files, I checked to see which files I should look after from the list here ;
https://github.com/rabbitmq/rabbitmq-server/commit/d12654a320ffd80a394ad4fac6720231bfadabc9
If it was the case I would expect auto-clustering to not work at all.

A rabbitmqctl report tells me the following (grepping only the lines about clustering) ;

{cluster_keepalive_interval,10000},
{cluster_nodes,
{['clusteringTest_1@host-004','clusteringTest_2@host-005',
'clusteringTest_3@host-009'],
disc}},
{cluster_partition_handling,pause_minority},

Cluster status of node 'clusteringTest_1@host-004' ...
[{nodes,[{disc,['clusteringTest_1@host-004']}]},
{running_nodes,['clusteringTest_1@host-004']},
{cluster_name,<<"clusteri...@host-004.gs.com">>},

There is nothing in the logs, I only get clustering related logs after a reset ;

=WARNING REPORT==== 12-May-2015::17:35:04 ===
Could not auto-cluster with clusteringTest_2@host-005: {badrpc,nodedown}

=WARNING REPORT==== 12-May-2015::17:35:04 ===
Could not auto-cluster with clusteringTest_3@host-009: {badrpc,nodedown}

=WARNING REPORT==== 12-May-2015::17:35:04 ===
Could not find any node for auto-clustering from: ['clusteringTest_1@host-004',
'clusteringTest_2@host-005',
'clusteringTest_3@host-009']

Sorry to come back again on this issue, any ideas ?

-----Original Message-----
From: rabbitm...@googlegroups.com [mailto:rabbitm...@googlegroups.com] On Behalf Of Jean-Sébastien Pédron
Sent: 17 April 2015 14:18
To: rabbitm...@googlegroups.com
Subject: Re: [rabbitmq-users] RE: Auto-Clustering Not working

[Jean-Sébastien Pédron]

On 12.05.2015 18:36, Skrzypek, Jonathan wrote:
> Hi,

Hi!

> Auto-clustering on my clusters doesn't work when they are first
> started, but if I perform a stop_app/reset/start_app, it works fine.
> Which makes no sense to me as I thought a start after a reset was the
> same as first starting.

Are you using the Debian package?

[Skrzypek, Jonathan]

No I'm not, we build rabbit from sources.

-----Original Message-----
From: rabbitm...@googlegroups.com [mailto:rabbitm...@googlegroups.com] On Behalf Of Jean-Sébastien Pédron
Sent: 12 May 2015 17:41
To: rabbitm...@googlegroups.com
Subject: Re: [rabbitmq-users] RE: Auto-Clustering Not working

[Jean-Sébastien Pédron]

On 13.05.2015 10:43, Skrzypek, Jonathan wrote:
> No I'm not, we build rabbit from sources.

Ok. Could you please post the output of "tree -apugsF /path/to/mnesia"
before you start the node for the first time?

You may have to install tree(1).