How do I restart a cluster after an all-node failure?

[Alexander]

Hello!

I have a cluster of 4 nodes, in disc mode, on 4 virtual machines. 

When I crash all 4 of them (as if there was an electrical blackout), on restart they don't reunion in a cluster again, beacuse no node knows what the last running node was.

So I have to delete /var/lib/rabbitmq/mnesia directory on each of them and manually recluster.

But then all the messages that can be stored in mnesia will be lost! 

Is there any way to set a cluster up after such a failure without mnesia deletion?

Maybe change some entries in mnesia on restart?

Configuration: rabbitmq 2.8.2 (due to corporate requirements), ubuntu 14.04.

Thanks! 

[Michael Klishin]

The docs describe how to do rolling restarts. If a node cannot be recovered it should be removed from the cluster.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Alexander]

Can you give a link please? I haven't found it in docs

среда, 6 июля 2016 г., 17:22:46 UTC+3 пользователь Michael Klishin написал:

[Michael Klishin]

"Breaking up a cluster", "Upgrading clusters" on

http://www.rabbitmq.com/clustering.html.

[V Z]

Would the answer to this question be to use 'rabbitmqctl force_boot' (per http://www.rabbitmq.com/clustering.html) on one of the nodes, and then start the rest?

[Michael Klishin]

That, too.  

On 7 July 2016 at 09:03:12, V Z (uvzu...@gmail.com) wrote:
> Would the answer to this question be to use 'rabbitmqctl force_boot' (per http://www.rabbitmq.com/clustering.html)
> on one of the nodes, and then start the rest?
>

> --
> You received this message because you are subscribed to the Google Groups "rabbitmq-users"
> group.
> To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.

> To post to this group, send an email to rabbitm...@googlegroups.com.

> For more options, visit https://groups.google.com/d/optout.
>

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

[V Z]

Is one option better than another?

[Joseph Casale]

On Thu, Jul 7, 2016 at 3:28 PM, V Z <uvzu...@gmail.com> wrote:
> Is one option better than another?

Check the logs and infer who was the master and thus the holder of the
likely most up to date queue, force it up and reset the rest.

[Rabbit Mq]

Is either way to recovery less destructive?

[Alexander]

Thank you guys, but that's all for new versions. There is no command "force_boot" in rabbitmqctl 2.8.2, nor there are commands like 'forget_cluster_node' and similar. When upgrading you still need to know what disc node was the last one and make it an upgrader.

Breaking cluster with "reset" and "force_reset" doesn't take effect if all nodes have crashed at once and no node knows which one was the last. Moreover, if I'm not mistaken, the reset/force_reset command returns mnesia db to it's virgin state, meaning, all persistent messages will be lost.

четверг, 7 июля 2016 г., 11:27:49 UTC+3 пользователь Michael Klishin написал:

[V Z]

Does force_boot reset amnesia?