No recovery from network partition (caused by VM pause) with autoheal mode

[Павел Полушин]

Hi there.

In our environment all RMQ servers are virtualized. And there might be a situation when VM is paused.

I know that vm pause is not good for RabbitMQ cluster and leads to network partition.

I set autoheal mode in cluster. Cluster has 3 nodes.

When I pause and then resume VM I see message like this in log

2019-01-17 11:41:28.430 [warning] <0.1636.0> Autoheal: we were selected to restart; winner is rabbit...

And then application goes down and that's all.

According to documentation node should get up again and join cluster:

In autoheal mode RabbitMQ will automatically decide on a winning partition if a partition is deemed to have occurred, and will restart all nodes that are not in the winning partition.

But it's not. Application on node is not starting automatically. Please advice what is wrong? 

[Павел Полушин]

Find out that sometimes autoheal is working. But it happens one time from ten.

Attached the log of the process when VM comes bask from pause and RabbitMQ trying to autoheal.

[Павел Полушин]

Hello again. Does someone has any suggestions about this case?

[Luke Bakken]

Hello,

Please don't reply to your own threads. The RabbitMQ team and other people who help out on this list do so when they have time.

Thanks,

Luke

[Luke Bakken]

Hello,

Suspend and resume is documented here:

https://www.rabbitmq.com/partitions.html#suspend

You will have to resolve partitions caused by suspend/resume manually when autoheal fails to do so.

Please see that document for procedures on how to resolve partitions - https://www.rabbitmq.com/partitions.html#recovering

Thanks,

Luke

[Павел Полушин]

Hello Luke and thank you for your answer.

I posted a bump to be sure that my thread is not missed by other. Sorry for that.

Am I understand things right, none of existing partition recover methods provides a guarantee for recover when partition caused by VM suspend?

If yes, I think it should be included in guides explicitly.

[Michael Klishin]

A node that is "resumed" (after a VM suspension) has missed every single event that happened in the cluster since its VM has been hibernated.

Perhaps a predominantly Raft-based (schema store and quorum queues all the way) version of RabbitMQ 4.0 might be able to catch up reasonably well

since Raft has a way of detection of stale log information but currently you are basically correct, the node won't really know the state of the cluster

once it wakes up from hibernation and that's problematic and confusing (both to the node and to the operator) in many ways.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

[Michael Klishin]

Welcome to the world of distributed systems and different planes of control. A node that has been suspended and then resumed on a different

host has no idea what has just happened. This is irrelevant for systems that are not clustered and stateful and can be critically important for those

that are, such as RabbitMQ.

We have some specific ideas for 4.0 that would make suspension less problematic but the fundamental problem remains. If you have a solution in mind

for it and not just want to vent and insult developers of software you very likely get for free), we welcome specific actionable ideas on this list.

If you just want to vent and insult, let me know and I will relieve this list from your presence.

On Tue, Feb 25, 2020 at 10:02 AM 0815 163 <rsevc...@gmail.com> wrote:

"While it's fine to run RabbitMQ clusters in virtualised environments, you should make sure that VMs are not suspended while running. Note that some virtualisation features such as migration of a VM from one host to another will tend to involve the VM being suspended."

https://www.rabbitmq.com/partitions.html#suspend

They are so funny. It is fine to run RabbitMQ clusters in virtualised environments, but you cannot use any of the features that makes the virtualised environment useful (vMotion, etc.). So basically they are saying that you need to reserve memory, cpu and pin the VMs to specific ESX hosts and don't do image level backups. So a physical machine it is. Fantastic. We're back in the 20th century again. Bought, but not owned by VMware...

On Tuesday, February 5, 2019 at 8:23:40 PM UTC+1, Michael Klishin wrote:

A node that is "resumed" (after a VM suspension) has missed every single event that happened in the cluster since its VM has been hibernated.

Perhaps a predominantly Raft-based (schema store and quorum queues all the way) version of RabbitMQ 4.0 might be able to catch up reasonably well

since Raft has a way of detection of stale log information but currently you are basically correct, the node won't really know the state of the cluster

once it wakes up from hibernation and that's problematic and confusing (both to the node and to the operator) in many ways.

On Sun, Jan 27, 2019 at 2:03 PM Павел Полушин <pawe...@gmail.com> wrote:

Hello Luke and thank you for your answer.

I posted a bump to be sure that my thread is not missed by other. Sorry for that.

Am I understand things right, none of existing partition recover methods provides a guarantee for recover when partition caused by VM suspend?

If yes, I think it should be included in guides explicitly.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitm...@googlegroups.com.

To post to this group, send email to rabbitm...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/9f9a53c5-d92d-42f9-893f-b86fd9bbd619%40googlegroups.com.

[Michael Klishin]

To be more specific, pausing and suspension are particularly problematic for the partition handling strategies. Once they are gone

in 4.0 thanks to the new Raft-based schema database (previously announced as Mnevis), there would be fewer scenarios

where suspension and resume of the VM can meaningfully matter to RabbitMQ nodes. However, we can never be sure that the number is zero,

since e.g. time can jump in curious ways that can create otherwise impossible scenarios and system states.

Unfortunately not all software is your typical Web app which outsources its entire state to a PostgreSQL somewhere and can be suspended, taken down, replaced

at any moment. Some software has to have its own distributed state, and VM management can produce impossible to detect situations for nodes.

Hopefully in RabbitMQ 4.0 there will be fewer such problematic cases. Of course, by then we can have a new shiny runtime/virtualization system

that introduces all kinds of problems that were not present before it (see all the Docker-specific issues for data services).

In practice, a data service such as RabbitMQ will have to have some resources preallocated one way or another. Sharing memory, disk and CPU resources

with other tools will produce contention, violated expectations and hard to diagnose issues. Sharing resources between resource intensive systems

is a great idea in theory only. But those who are willing to contribute to open source and can articulate their ideas without insults are welcome to

suggest them on this list.