Stopping two masters puts cluster in bad state

[hokie...@gmail.com]

Hi Everyone,

Redis cluster is working well in my testing (it's super cool!), but I've found one scenario where it fails to restart the terminated instances and return the cluster to correct state.  Specifically, I am running a cluster with 3 masters and 2x slaves for a total of 9 redis servers, and when I stop two of the masters the cluster remains in a fail state and the replacement masters never start.

Has anyone else encountered this?

--John

[Jan-Erik Rediger]

This is to be expected. You need a majority of master servers to agree
on a failover. With 2 out of 3 master servers being down, there is no
majority left to initiate a failover.

> --
> You received this message because you are subscribed to the Google Groups "Redis DB" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to redis-db+u...@googlegroups.com.
> To post to this group, send email to redi...@googlegroups.com.
> Visit this group at https://groups.google.com/group/redis-db.
> For more options, visit https://groups.google.com/d/optout.

[hokie...@gmail.com]

Ah, cool, makes sense, thanks!  Driving home I was wondering if something like this was the case. In my experience Redis is rock-solid and performs as expected, so I figured that there was some misconfiguration on my part.

Thanks again!

--John

[AlexanderB]

If you're a bit slower, and don't stop the second master till the first slave has been promoted you should be fine. 

The cluster can keep itself going in a ship of Theseus sort of way, but only as long as the majority of masters are around to have quorum to perform promotions.