high availability and Redis Sentinel on two nodes

[S.Kras]

The idea behind Redis Sentinel is a high availability approach. A configured quorum is taken as the number of Sentinel processes, which have to detect that the Master Node is not available anymore. During an election process the majority (or the configured quorum if its value is higher than the majority) of Sentinel processes is required to vote for a new master.

This idea seems to work for installations with at least 3 Sentinels.

What is with small installations on two nodes that like to be high available?

A setup could be to run a Redis MASTER on one box and a Redis SLAVE on the other box. The high availability results by running Redis Sentinel on both boxes, setting the quorum to 1 and monitoring both Redis instances via each sentinel instance. When the master box goes down (of course including the sentinel process) the other box could be able to decide on its own that the master is not available anymore and could elect (without any other sentinel) itself as the new master. What does argue against such behavior?

This approach does not work because Redis sentinel insists on the majority for an election. In our case we need 2 sentinel instances, but have only one up and running. In result you get an “-failover-abort-not-elected master” error and the not available MASTER stays master. Your application stops to work. Bye Bye high availability with Redis :(

I did not get the point why Redis sentinel makes a difference between the configured quorum and the majority that is required for a successful reelection. Especially in case of two Sentinel instances this behavior seems strange to me.

Is there no possibility to run Redis on two nodes with a high availability configuration?

[Matt Stancliff]

> On Feb 11, 2015, at 10:25, S.Kras <sylwest...@googlemail.com> wrote:
>
> Is there no possibility to run Redis on two nodes with a high availability configuration?

You’re right: two Sentinels can’t perform a majority election during a failure because, if one Sentinel is down or unreachable, you can’t have a majority of 50%+1 with 2 nodes because 50%+1 of 2 is 2, and 2 - 1 != 2.

That’s also the reason why Redis Cluster requires a minimum of 3 master nodes. In Redis Cluster, only the masters vote for failover, and you can’t have a majority vote with less than 3 masters.

Redis takes the approach of “everything in unreliable and will fail randomly” so it does’t allow one process to make failover decisions. A single process/server could have a bias view of the network. The theoretically-ideal situation for Sentinel deployments is to give each Sentinel a different view of your Redis servers by placing each Sentinel in a different rack or in a different core network. In practice, that’s not always realistic these days, so you just have to add enough Sentinels to give yourself a good enough feeling of “this entire group of monitoring servers has enough coverage to reliably detect failures and recover across everything they are monitoring without any majority of monitoring servers having a biased view of their monitoring targets."


-Matt

[Anton Mironenko]

Thanks for raising this requirement - running Sentinel on two hosts only.
This is very popular HA topology in telecom to have a cluster of two hosts.

A single process/server could have a bias view of the network.

Regarding this argument, we can use bonded network interface, sitting on top of two different Ethernet ports, which go to different switches. Having this, there will be no "bias view of the network", i.e. no split brain syndrome.
Why not to add a flexible quorum configuration, so if we want to use Sentinel on two hosts for HA, we configure a quorum to "1" and use it.
And if we are afraid of “everything in unreliable and will fail randomly”, then will configure a quorum to "majority" and provide 2N+1 hosts, N > 0.

[The Baldguy]

Telecom has a very different set of requirements and often hardware. It is thus natural they will have different solutions. Multiple routes from points a to b are a different case than "where is the authoritative source of data?".

Btw even in your proposed network arrangement you can actually still get a biased view. All it takes is a simple misconfiguration of a switch port, or a faulty bonding config, mismatched config between switch and nic, etc. and you are back to a split brain scenario.

Even with a crossover cable to provide a connection direct from mic to nic it can happen.

Now if you add in the fact that these direct style connections don't reflect the availability that matters - the connectivity of the clients, it gets even tricker.

[V*i*n]

Hi,

I have another question about the same subject :

having a cluster of N master with replicas = 1 (so 1 slave per master).

I have a network configuration with 2 availability zones (A.Z.), so each one hosts a server that have the same key slots.

each A.Z. also have my redis client application. 

I would add to this setup 3 sentinels, one in each availability zone, and one outside each of them... symbolizing the outside world access/internet.

now if I understood correctly, each slave will kick up and become master just when it should but a master isolated will not step down. So the client connected to it (on the same A.Z.) will continue to use the isolated master (case which I have to avoid).

now there is this min-slaves-to-write 1 server feature for it to step down, but I can't use it because I only have 2 servers.

- alternate idea 1) (if it exists) : Does it exists a min-sentinel-to-write  parameter to use so I could set it to 2 and tell the isolated server to step down because it cannot contact at least 2 sentinels ?

- alternate idea 2) (if it exists) : can we put a slave server "on pause" which would mean that replication of data from master to sync would be stopped ?

If such a thing exists I would set up a 3rd server where my 3rd sentinel is (on the internet - outside both A.Z.) , let it on pause forever then put min-slaves-to-write 2. Now my isolated server would step down, and the other one (which has access to the internet) will step to master.

NOTE that I don't want replication traffic to go somewhere else (...the internet) than between the 2 A.Z. because of bandwidth problem.

Do you have any other alternate idea or a solution to my "simple" problem ?

Thank you.

The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient is prohibited. If you received this in error, please contact the sender and delete the material from any computer.

[Tuco]

If you are using cluster, you are not using sentinels

Also, i think you should create a separate thread for your question.