Sentinel failover minimum available nodes

[Benjamin Vetter]

Hi,

does anyone know if it's possible to configure sentinel to only make a master failover if at least N nodes are available to choose from?

Use case:

i have a 3 node sentinel setup and i'm using WAIT to guarantee that my write reaches at least 2 nodes before returning success.

If i could force sentinel to only make a failover if at least 2 nodes are up, it's no longer possible to loose writes, or is it?

Thanks in advance

  Benjamin

[hva...@gmail.com]

I guess I don't understand.  Sentinel will fail over a master when the master seems to be unavailable.

If the master becomes unavailable, and there's only one slave available, you want Sentinel to do nothing?  Then the situation becomes no master is available to receive writes, only a slave for reads.

To me that seems worse than failing over to the remaining slave to handle both reads and writes.

  -Greg

[Benjamin Vetter]

yes, i want exactly that - and it's not uncommon.

if i have N machines i can build on the assumption that N / 2 + 1 machines are up - and if they're not up i don't want failover to happen such that no writes will be lost, because loosing writes is worse.

i simple don't see a feature in sentinel to specify a minimum available nodes.

[Benjamin Vetter]

let me clarify the scenario: i have a 3 node sentinel setup + 3 redis server nodes and i'm using the redis WAIT command to guarantee that my write reaches at least 2 redis server nodes before returning success. I want sentinel to only make a failover if at least 2 Redis server nodes are up ... such that no matter what every write will be available on the newly elected master.

The docs regarding the WAIT command say:

both Sentinel and Redis Cluster will do a best-effort attempt to promote the best slave among the set of available slaves. However this is just a best-effort attempt so it is possible to still lose a write synchronously replicated to multiple slaves.

but i'm not sure what the "best-effort" is. I think restriting the failover process, such that a minimum number of nodes is up, should be sufficient to remove the "best-effort" approach in my scenario, isn't it? 

[hva...@gmail.com]

Yes, but your example doesn't address the rule you want Sentinel to apply.

Your example is:  You have one master and more three slaves.  Now the master goes down, and Sentinel is allowed to fail over (transfer the master role to another server) because there are three remaining nodes.  One can become the master, and there will still be two slaves to satisfy your WAIT command requirement.

My example is different:  You used to have one master and three slaves, but for some reason one of the slaves is not available.  Now the master goes down.  You are saying Sentinel should not fail over because there are only two remaining nodes - if one becomes master there aren't enough slaves to satisfy your WAIT command requirement.

I'm pointing out that in my example, if Sentinel is not allowed to fail over (transfer the master role to another server), you will have a major outage.  Writes will be lost because the client cannot connect to the master to send them.  Also, if the event that caused the master to be unavailable also makes the master lose data (corrupted persistence file, or the most recent writes weren't written to disk yet), then the resurrected master could have less data than the slaves who stayed alive.  When those slaves perform a full resync with the resurrected master, they would drop the same data.

I'm not sure if Sentinel can be configured to do as you ask.  I'm just asking if, in the scenario where you're asking Sentinel to refrain from failing over, the potential loss of data is okay.

  -Greg

[Benjamin Vetter]

Am Mittwoch, 28. Dezember 2016 02:03:11 UTC+1 schrieb hva...@gmail.com:

Yes, but your example doesn't address the rule you want Sentinel to apply.

Your example is:  You have one master and more three slaves.  Now the master goes down, and Sentinel is allowed to fail over (transfer the master role to another server) because there are three remaining nodes.  One can become the master, and there will still be two slaves to satisfy your WAIT command requirement.

no, once again: 3 nodes in total, one is master. If the master goes down there are 2 remaining slaves, one can become master, but only if both 2 slaves are up and available.

 

My example is different:  You used to have one master and three slaves, but for some reason one of the slaves is not available.  Now the master goes down.  You are saying Sentinel should not fail over because there are only two remaining nodes - if one becomes master there aren't enough slaves to satisfy your WAIT command requirement.

I'm pointing out that in my example, if Sentinel is not allowed to fail over (transfer the master role to another server), you will have a major outage.  Writes will be lost because the client cannot connect to the master to send them.  Also, if the event that caused the master to be unavailable also makes the master lose data (corrupted persistence file, or the most recent writes weren't written to disk yet), then the resurrected master could have less data than the slaves who stayed alive.  When those slaves perform a full resync with the resurrected master, they would drop the same data. 

I'm not sure if Sentinel can be configured to do as you ask. I'm just asking if, in the scenario where you're asking Sentinel to refrain from failing over, the potential loss of data is okay.

you're talking about an even less likely scenario (that results in data loss) than i did ?!

... in your scenario a) the master node must crash (to have a corrupt/out of date persistence file) plus b) a network outage must occur at the same time (such that no failover happens due to the policy i was talking about) plus c) crashed master node comes back online with corrupt persistence file.

that doesn't make much sense, as the initial scenario (master no longer available (network outage OR crash) -> failover to out of sync slave -> data loss) that i want to protect against is of course more likely.

[hva...@gmail.com]

Benjamin Vetter wrote:

Am Mittwoch, 28. Dezember 2016 02:03:11 UTC+1 schrieb hva...@gmail.com:

Yes, but your example doesn't address the rule you want Sentinel to apply.

Your example is:  You have one master and more three slaves.  Now the master goes down, and Sentinel is allowed to fail over (transfer the master role to another server) because there are three remaining nodes.  One can become the master, and there will still be two slaves to satisfy your WAIT command requirement.

no, once again: 3 nodes in total, one is master. If the master goes down there are 2 remaining slaves, one can become master, but only if both 2 slaves are up and available.

But this use case is what I was talking about.  After failover there will be only one master and one slave.  Won't your WAIT fail because there is only one slave to accept the replicated write command?  Not the minimum of two slaves?
 

[The Real Bill]

In your described scenario the failover does absolutely nothing for you. If you insist on two slaves accepting the related command before the write is allowed, then a three Redis node setup has no availability after a single failure of any of the three nodes.

If a slave fails, you can't accept writes because you have only one slave instead of two. If the master fails sentinel promotes one of your two slaves. At that point you have one master and one slave, and therefore can't accept the writes when requiring two slaves.

Thus with a wait specifying two slaves, and three total Redis nodes (1m, 2s) you have zero ability to tolerate a failure of any Redis node in the system as far writes are concerned.

For such a write condition you need a minimum slave count of three to handle only a single failure. After that it wouldn't matter from a write perspective what sentinel does because you can't tolerate a second failure anyway.

This is a case when adding more conditions decreases the integrity of the system as opposed to increasing it.

Cheers,
Bill

[Benjamin Vetter]

"i have a 3 node sentinel setup and i'm using WAIT to guarantee that my write reaches at least 2 nodes before returning success."

Sorry if you misunderstood, but the master is a node as well (2 nodes = master node + 1 slave).

i was talking about WAIT requiring 1 slave to get the update, so i can loose 1 machine.

And in terms of failover: if the failover requires 2 machines to be up (3 / 2 + 1 = 2), the slave having the update will be elected the new master.

[Bill Anderson]

On Wed, Jan 4, 2017 at 2:51 AM, 'Benjamin Vetter' via Redis DB <redi...@googlegroups.com> wrote:

"i have a 3 node sentinel setup and i'm using WAIT to guarantee that my write reaches at least 2 nodes before returning success."

Sorry if you misunderstood, but the master is a node as well (2 nodes = master node + 1 slave).

No. It is a node, but WAIT specifically waits for slaves, and the master doesn't count. As per the documentation on WAIT:

"This command blocks the current client until all the previous write commands are successfully transferred and acknowledged by at least the specified number of slaves"

Emphasis added to ensure it is noted. It does not say the specified number of nodes, but the number of slaves. Therefore your assumption that the master is counted among them is invalid and unsupported. If you think through it, there is no reason to expect WAIT to count the master since it runs on the master and WAIT is about ensuring slaves have the write.

i was talking about WAIT requiring 1 slave to get the update, so i can loose 1 machine.

Then you need to use the correct terminology and parameters. If you use, as you specified, a WAIT of 2, you can not lose a single node. 

If your goal is actually for only one slave to be waited on, then you need to use a wait of 1, and you configure Sentinel normally nice you only need one slave. With that setup you can maintain writability after the failure of one node in your three node system and need to do nothing special with Sentinel. After the first failover it will operate normally. After a second failure you have no slaves and thus no WAITs will be successful. You will need to do far more work on them client side than you realize. You can't simply say "WAIT for N slaves" and assume it works, you have to also either:

1) use a timeout and check the return value to see how many slaves it wrote to before determining it a success or retry, or

2) set a timeout of zero, then add code to detect the connection getting closed (such as during a failover or network blip, etc) and handle retries

In your proposed scenario of Sentinel not doing a failover, you would also then have to add code to detect replication mode in addition to sentinel lookups. In the case of using WAIT with a zero timeout, you would do well to read up on blocking commands and failover at http://objectrocket.com/blog/how-to/reliable-pubsub-and-blocking-commands-during-redis-failovers as much of the underpinnings will be the same. Finally, remember that WAIT is a client command, not a configuration. Sentinel has no part in client commands, and thus in order to maintain a correct pod configuration all it needs is a single viable slave for failover to occur. With WAIT being a client command and choice, it is up to the client to handle the scenario of not enough slaves, not the server or Sentinel.

Cheers,

Bill

[Benjamin Vetter]

... i know that WAIT doesn't count the master, but i do count the master, such that with using "WAIT 1" i can be sure that 2 nodes have my write,

if "WAIT 1" returns 1. i was describing the scenario, not the parameters of WAIT.

however let's stop here, ... thx