Automatic restoration of a failed master

[Pieter du Preez]

Hi,

I have a question about what a Redis Cluster does with a master that failed and returns to normal operation as a slave.

My topology is as follows:

(I specified the replica-priority per redis server)

Node A

------
M1 (replica-priority = 7)
S2 (replica-priority = 4)
S3 (replica-priority = 1)

Node B
------
M2 (replica-priority = 8)
S1 (replica-priority = 5)
S3 (replica-priority = 2)

Node C
------
M3 (replica-priority = 9)
S1 (replica-priority = 6)
S2 (replica-priority = 3)

If I now fail (stop) M3, the following happens, as expected:

Node A

------
M1 (ok)
S2 (ok)
M3 (ok, failed-over from Node C)

Node B
------
M2 (ok)
S1 (ok)
S3 (ok)

Node C
------
M3 (fail)
S1 (ok)
S2 (ok)

If I now start the stopped M3 on Node C, it comes back as slave of M3, which now resides on Node A.

Node A

------
M1 (ok)
S2 (ok)
M3 (ok)

Node B
------
M2 (ok)
S1 (ok)
S3 (ok)

Node C
------
S3 (ok)
S1 (ok)
S2 (ok)

S3 on Node C never becomes a master. In this situation the cluster will fail if the entire Node A goes down, as it has 2 masters.

As I would like to make sure that my cluster is fit for the next problem, I'd like to fail S3 on Node C over to be a master. Like this I'll have a master on every node again, which means any one of the nodes can go down without breaking the cluster.

Is there a way to automatically ensure that the number of masters are always balanced across all cluster nodes? i.e. In my case: Make sure that there's one master per node, if possible.

So far my solution to this problem was to run Sentinal on all nodes. I then have a process that subscribes to all +try-failover, -sdown, -odown and +reboot messages. If a potential master appears in these messages, the process tests if it is a slave, and if so, a "cluster failover" is done on it. This has worked up to now, but it seems to be a huge amount of overhead for such a small task.

Thanks