Recovering from mnesia error

[Vasanthi Ramasamy]

Hi,

I'm studying RabbitMQ cluster node failures and its impact. I have a 2 node cluster. As part of this testing, I stop RabbitMQ on one node (stop_app) and then start it (start_app). Broker gets stuck when I try start it once in few times and I see the error below in RabbitMQ log. I have given it enough time to try coming up. It doesn't.  How can I recover the node from this point ?  One suggestion was to delete everything under /var/lib/rabbitmq/mnesia/. If I do that, all vhosts and user configurations are lost. 

=WARNING REPORT==== 8-Jun-2018::15:03:32 ===

Error while waiting for Mnesia tables: {timeout_waiting_for_tables,

                                        [rabbit_user,rabbit_user_permission,

                                         rabbit_vhost,rabbit_durable_route,

                                         rabbit_durable_exchange,

                                         rabbit_runtime_parameters,

                                         rabbit_durable_queue]}

=INFO REPORT==== 8-Jun-2018::15:03:32 ===

Waiting for Mnesia tables for 30000 ms, 8 retries left

Thanks,

Vasanthi

[Michael Klishin]

That suggestion comes from someone who is misinformed and has no idea what is really going on.

This documentation section covers it: http://www.rabbitmq.com/clustering.html#restarting.

Your node *waits for a peer to come online* and retries to contact it periodically. The docs mention that

if the peer is known to never be coming back, `rabbitmqctl force_boot` can be used but it is NOT meant to be used

in any other scenario.

As for deleting node's data directory… do we really need to explain what effects that will have and how wise this suggestion is?

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

[Vasanthi Ramasamy]

This is enough. Thank you ! 

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[spici...@gmail.com]

I am not able to run rabbitmqctl stop_app , rabbitmqctl force_boot 

Everytime the kubernetes pod restarts I login to the pod and execute the above commands I get the below message

RabbitMQ hasn't finished starting yet. Waiting for startup to finish before stopping...

2018-12-28 04:32:24.987 [info] <0.248.0> Waiting for Mnesia tables for 30000 ms, 9 retries left

2018-12-28 04:32:43.221 [info] <0.276.0> RabbitMQ hasn't finished starting yet. Waiting for startup to finish before stopping...

2018-12-28 04:32:54.987 [warning] <0.248.0> Error while waiting for Mnesia tables: {timeout_waiting_for_tables,[rabbit_user,rabbit_user_permission,rabbit_topic_permission,rabbit_vhost,rabbit_durable_route,rabbit_durable_exchange,rabbit_runtime_parameters,rabbit_durable_queue]}

2018-12-28 04:32:54.988 [info] <0.248.0> Waiting for Mnesia tables for 30000 ms, 8 retries left

[Michael Klishin]

This list uses a "one question per thread" policy. Please start a new thread and see [1]. Your node cannot

contact a previously known peer so it fails to start, and it refuses to be stopped before it boots successfully.

1. http://www.rabbitmq.com/clustering.html#restarting

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.