Cluster Recover When All Nodes Down

Skip to first unread message

Murray McCulligh

unread,
Jun 22, 2018, 10:56:31 AM6/22/18
to rabbitmq-users
If all nodes in a 2 node cluster are stopped and the second node is started without the first it seems to be waiting for the first and then crashes.  If the first node is started alone everything is fine.

Is this the expected behaviour?  

Murray

 

  ##  ##

  ##  ##      RabbitMQ 3.7.6. Copyright (C) 2007-2018 Pivotal Software, Inc.

  ##########  Licensed under the MPL.  See http://www.rabbitmq.com/

  ######  ##

  ##########  Logs: <stdout>

 

              Starting broker...

2018-06-22 14:15:30.965 [info] <0.247.0> 

 node           : rab...@coreang2.acme2013.local

 home dir       : /var/lib/rabbitmq

 config file(s) : /etc/rabbitmq/rabbitmq.conf

 cookie hash    : p3hzX8GwGIS06SdVKHrLWg==

 log(s)         : <stdout>

 database dir   : /var/lib/rabbitmq/mnesia/rab...@coreang2.acme2013.local

2018-06-22 14:15:31.414 [info] <0.253.0> Memory high watermark set to 735 MiB (771280076 bytes) of 1838 MiB (1928200192 bytes) total

2018-06-22 14:15:31.419 [info] <0.255.0> Enabling free disk space monitoring

2018-06-22 14:15:31.419 [info] <0.255.0> Disk free limit set to 50MB

2018-06-22 14:15:31.425 [info] <0.257.0> Limiting to approx 65436 file handles (58890 sockets)

2018-06-22 14:15:31.425 [info] <0.258.0> FHC read buffering:  OFF

2018-06-22 14:15:31.425 [info] <0.258.0> FHC write buffering: ON

2018-06-22 14:15:42.957 [info] <0.247.0> Waiting for Mnesia tables for 30000 ms, 9 retries left

2018-06-22 14:16:12.958 [warning] <0.247.0> Error while waiting for Mnesia tables: {timeout_waiting_for_tables,[rabbit_user,rabbit_user_permission,rabbit_topic_permission,rabbit_vhost,rabbit_durable_route,rabbit_durable_exchange,rabbit_runtime_parameters,rabbit_durable_queue]}

2018-06-22 14:16:12.958 [info] <0.247.0> Waiting for Mnesia tables for 30000 ms, 8 retries left

2018-06-22 14:16:42.959 [warning] <0.247.0> Error while waiting for Mnesia tables: {timeout_waiting_for_tables,[rabbit_user,rabbit_user_permission,rabbit_topic_permission,rabbit_vhost,rabbit_durable_route,rabbit_durable_exchange,rabbit_runtime_parameters,rabbit_durable_queue]}

2018-06-22 14:16:42.959 [info] <0.247.0> Waiting for Mnesia tables for 30000 ms, 7 retries left

2018-06-22 14:17:12.960 [warning] <0.247.0> Error while waiting for Mnesia tables: {timeout_waiting_for_tables,[rabbit_user,rabbit_user_permission,rabbit_topic_permission,rabbit_vhost,rabbit_durable_route,rabbit_durable_exchange,rabbit_runtime_parameters,rabbit_durable_queue]}

2018-06-22 14:17:12.960 [info] <0.247.0> Waiting for Mnesia tables for 30000 ms, 6 retries left

2018-06-22 14:17:42.961 [warning] <0.247.0> Error while waiting for Mnesia tables: {timeout_waiting_for_tables,[rabbit_user,rabbit_user_permission,rabbit_topic_permission,rabbit_vhost,rabbit_durable_route,rabbit_durable_exchange,rabbit_runtime_parameters,rabbit_durable_queue]}

2018-06-22 14:17:42.961 [info] <0.247.0> Waiting for Mnesia tables for 30000 ms, 5 retries left

2018-06-22 14:18:12.962 [warning] <0.247.0> Error while waiting for Mnesia tables: {timeout_waiting_for_tables,[rabbit_user,rabbit_user_permission,rabbit_topic_permission,rabbit_vhost,rabbit_durable_route,rabbit_durable_exchange,rabbit_runtime_parameters,rabbit_durable_queue]}

2018-06-22 14:18:12.962 [info] <0.247.0> Waiting for Mnesia tables for 30000 ms, 4 retries left

2018-06-22 14:18:42.963 [warning] <0.247.0> Error while waiting for Mnesia tables: {timeout_waiting_for_tables,[rabbit_user,rabbit_user_permission,rabbit_topic_permission,rabbit_vhost,rabbit_durable_route,rabbit_durable_exchange,rabbit_runtime_parameters,rabbit_durable_queue]}

2018-06-22 14:18:42.963 [info] <0.247.0> Waiting for Mnesia tables for 30000 ms, 3 retries left

2018-06-22 14:19:12.964 [warning] <0.247.0> Error while waiting for Mnesia tables: {timeout_waiting_for_tables,[rabbit_user,rabbit_user_permission,rabbit_topic_permission,rabbit_vhost,rabbit_durable_route,rabbit_durable_exchange,rabbit_runtime_parameters,rabbit_durable_queue]}

2018-06-22 14:19:12.964 [info] <0.247.0> Waiting for Mnesia tables for 30000 ms, 2 retries left

2018-06-22 14:19:42.965 [warning] <0.247.0> Error while waiting for Mnesia tables: {timeout_waiting_for_tables,[rabbit_user,rabbit_user_permission,rabbit_topic_permission,rabbit_vhost,rabbit_durable_route,rabbit_durable_exchange,rabbit_runtime_parameters,rabbit_durable_queue]}

2018-06-22 14:19:42.965 [info] <0.247.0> Waiting for Mnesia tables for 30000 ms, 1 retries left

2018-06-22 14:20:12.966 [warning] <0.247.0> Error while waiting for Mnesia tables: {timeout_waiting_for_tables,[rabbit_user,rabbit_user_permission,rabbit_topic_permission,rabbit_vhost,rabbit_durable_route,rabbit_durable_exchange,rabbit_runtime_parameters,rabbit_durable_queue]}

2018-06-22 14:20:12.966 [info] <0.247.0> Waiting for Mnesia tables for 30000 ms, 0 retries left

2018-06-22 14:20:42.967 [error] <0.246.0> CRASH REPORT Process <0.246.0> with 0 neighbours exited with reason: {{timeout_waiting_for_tables,[rabbit_user,rabbit_user_permission,rabbit_topic_permission,rabbit_vhost,rabbit_durable_route,rabbit_durable_exchange,rabbit_runtime_parameters,rabbit_durable_queue]},{rabbit,start,[normal,[]]}} in application_master:init/4 line 134

2018-06-22 14:20:42.967 [info] <0.33.0> Application rabbit exited with reason: {{timeout_waiting_for_tables,[rabbit_user,rabbit_user_permission,rabbit_topic_permission,rabbit_vhost,rabbit_durable_route,rabbit_durable_exchange,rabbit_runtime_parameters,rabbit_durable_queue]},{rabbit,start,[normal,[]]}}

{"Kernel pid terminated",application_controller,"{application_start_failure,rabbit,{{timeout_waiting_for_tables,[rabbit_user,rabbit_user_permission,rabbit_topic_permission,rabbit_vhost,rabbit_durable_route,rabbit_durable_exchange,rabbit_runtime_parameters,rabbit_durable_queue]},{rabbit,start,[normal,[]]}}}"}

Kernel pid terminated (application_controller) ({application_start_failure,rabbit,{{timeout_waiting_for_tables,[rabbit_user,rabbit_user_permission,rabbit_topic_permission,rabbit_vhost,rabbit_durable_r

 

Crash dump is being written to: /var/log/rabbitmq/erl_crash.dump...done

Michael Klishin

unread,
Jun 22, 2018, 11:02:05 AM6/22/18
to rabbitm...@googlegroups.com
The log is pretty clear on the fact that the waiting is repeated N times. The window is 5 minutes
by default: 10 attempts with a 30s timeout each. It can be increased if peers cannot all start in that amount of time.

Please take a closer look at the Clustering guide and specifically the section on node restarts [1] for details.


--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.



--
MK

Staff Software Engineer, Pivotal/RabbitMQ

Michael Klishin

unread,
Jun 22, 2018, 11:08:36 AM6/22/18
to rabbitm...@googlegroups.com
A few specific paragraphs from that section:

«It is important to understand the process node go though when they are stopped and restarted.
A stopping node picks an online cluster member (only disc nodes will be considered) to sync with after restart.
Upon restart the node will try to contact that peer 10 times by default, with 30 second response timeouts.
In case the peer becomes available in that time interval, the node successfully starts, syncs what it needs from the peer and keeps going.
When a node has no online peers during shutdown, it will start without attempts to sync with any known peers. It does not start as a standalone node, however, and peers will be able to rejoin it.
When the entire cluster is brought down therefore, the last node to go down must be the first node to be brought online.
However, since nodes will try to contact a known peer for up to 5 minutes (by default), nodes can be restarted in any order in that period of time.
In this case they will rejoin each other one by one successfully. This window of time can be adjusted using two configuration settings…»

Prior to 3.6.7 nodes had to be restarted in the reverse order from their shutdown. As of 3.6.7, thanks to the retries, there is a configurable window
of time in which all nodes have to come back (or all but N, if it happens to be in the reverse order mentioned earlier). This eliminates the order
requirement in practice for most deployment tools and scenarios.

On Fri, Jun 22, 2018 at 6:01 PM, Michael Klishin <mkli...@pivotal.io> wrote:
The log is pretty clear on the fact that the waiting is repeated N times. The window is 5 minutes
by default: 10 attempts with a 30s timeout each. It can be increased if peers cannot all start in that amount of time.

Please take a closer look at the Clustering guide and specifically the section on node restarts [1] for details.

On Fri, Jun 22, 2018 at 5:56 PM, Murray McCulligh <mmccu...@gmail.com> wrote:
If all nodes in a 2 node cluster are stopped and the second node is started without the first it seems to be waiting for the first and then crashes.  If the first node is started alone everything is fine.

Is this the expected behaviour?  

Murray

 

  ##  ##

  ##  ##      RabbitMQ 3.7.6. Copyright (C) 2007-2018 Pivotal Software, Inc.

  ##########  Licensed under the MPL.  See http://www.rabbitmq.com/

  ######  ##

  ##########  Logs: <stdout>

 

              Starting broker...

2018-06-22 14:15:30.965 [info] <0.247.0> 

 node           : rab...@coreang2.acme2013.local

 home dir       : /var/lib/rabbitmq

 config file(s) : /etc/rabbitmq/rabbitmq.conf

 cookie hash    : p3hzX8GwGIS06SdVKHrLWg==

 log(s)         : <stdout>

To post to this group, send email to rabbitm...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.
--
MK

Staff Software Engineer, Pivotal/RabbitMQ

Murray McCulligh

unread,
Jun 22, 2018, 11:37:02 AM6/22/18
to rabbitmq-users
Thanks.  Thought I had read everything on that page but missed the part about needing to bring the last one down up first.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.

To post to this group, send email to rabbitm...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
MK

Staff Software Engineer, Pivotal/RabbitMQ

Michael Klishin

unread,
Jun 22, 2018, 12:06:16 PM6/22/18
to rabbitm...@googlegroups.com
It is no longer really a requirement during rolling restarts. It is still a requirement
for feature release upgrades (e.g. 3.6.16 to 3.7.6), as the upgrade doc explains.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Michael Klishin

unread,
Jun 22, 2018, 12:06:57 PM6/22/18
to rabbitm...@googlegroups.com
…because internal database schema upgrade by design happens on one node first, then
is replicated to its peers.
Reply all
Reply to author
Forward
0 new messages