Sentinels in Multi Region configuration

[Dvir Volk]

Let's say I have to Data Centers, A and B.

A is the "master DC", and B is the "slave" DC where you should generally not do any writes.

So the configuration is 

DC A: master =====> failover slave 

             || 

             ||

             ||

             \/

DC B: "pseudo master" =====> { slave 1, slave 2 }

"B master" is a slave of "A master", and has slaves of its own (this is in order to keep just one replication pipeline between datacenters).

Now, I want to have 2 separate sentinel clusters:

In A - handler failover properly, BUT never failing over to "B master".

In B - handling local failover, and making sure the new "B master" is now a slave of "A master"

I see (at least) two potential problems here:

1. The smaller one, is that a failover in B won't make B master slave of A master. That can be solved with extra monitoring in B.

2. Knowledge of DC A will propagate to B via the sentinel gossip-over-master, and create a whole mess.

So, is this possible at all?

And if not, what can be changed in the sentinel protocol to make this feasible?

[Yiftach Shoolman]

Hi Dvir,

A few comments/questions:

1. I assume 'A' and 'B' are located in different geographical regions, is it correct ? my 3rd point is only relevant if this is your case

2. In each DC, why do you use only one replication ? or in other words why don't you make sure you have a quorum ? from past threads I remember you were saying you've never experienced a network split event inside a DC, but if that happens (we had it many times) it can lead to a complete data loss situation that I guess you really want to avoid

3. Practically speaking, I would NOT recommend to anyone to implement a multi-region auto-failover mechanism for the following reasons: (a) like inside a DC you need to have a quorum, i.e. 3 DCs, 3 regions, and it becomes complex and expensive; (b) I can run a list with so many corner cases where a false auto-failover decision can break your entire app; (c) this is exactly one of the cases where things need to be done manually and with a careful look. How many complete data-center outages you had during the last 12 months ? develop such a complex and risky mechanism for an event that might happen once a year just doesn't deserve the effort IMO

--
You received this message because you are subscribed to the Google Groups "Redis DB" group.
To unsubscribe from this group and stop receiving emails from it, send an email to redis-db+u...@googlegroups.com.
To post to this group, send email to redi...@googlegroups.com.
Visit this group at http://groups.google.com/group/redis-db.
For more options, visit https://groups.google.com/d/optout.

--

Yiftach Shoolman
+972-54-7634621

[Dvir Volk]

My needs are simpler than you describe.

Only DC A is being written to. 

If I have DC B and C they are only slaves of A.

In each DC I'll want to reach a quorum on the local master.

And let's say I'll do the B->A failover manually, that's easy indeed.

But the question is more technical - say I have a sentinel cluster for each DC. They shouldn't know each other. But the sentinel gossip protocol over the masters will make them know each other, and do crazy stuff if they can speak to each other. Am I right about this assumption?

And if so, what besides firewall rules (which I don't want to get into and not sure they'll even help) can prevent this? 

[Rhommel Lamas]

We have a similar use case than yours and what we did was basically:

DC A:

On this DC we have a 3 node cluster of one master with 2 slaves working with sentinel with a quorum for this DC.

DC B:

On DC B we have a 3 node cluster but in this case the Master instance of Redis is a slave of the Master on DC A, al instances in this DC have 'priority 0` this way we ensure that these instances won't failover unless we do it manually.

Also on this DC we have a sentinel monitoring the cluster on DC B and with a quorum bigger than the amount of Sentinels (this sentinel will be able to see sentinels on DC A so be careful) so we ensure that Sentinel won't trigger notifications.

[Salvatore Sanfilippo]

On Fri, Jun 27, 2014 at 9:33 AM, Dvir Volk <dv...@everything.me> wrote:
> But the question is more technical - say I have a sentinel cluster for each
> DC. They shouldn't know each other. But the sentinel gossip protocol over
> the masters will make them know each other, and do crazy stuff if they can
> speak to each other. Am I right about this assumption?

Hello Dvir,

this assumption is not correct actually, assuming I understand it
correctly. For example if you have two different groups of Sentinels
monitoring the same master, but using two different names for the
master, they'll not discover / updated each other.

However this is not enough to avoid problems, the first set of
Sentinels may failover the master and the other set will sense the
switch of the promoted slave into a master as a misconfiguration and
will reconfigure the slave again to replicate from the old master and
so forth.

Aside from this problems, there is also the problem that for Sentinel
the role of an instance is the one published in INFO by the instance.
This means that you can't failover a slave that happens to be the
master of other chained instances. Also the chained instances will not
be detected, and if detected because of a temporary role change for
some reason, they'll be reconfigured to replicate with what Sentinel
believe to be the master.
Basically you can find your ways to make it working but currently the
support for chained replication in Sentinel is near zero.

Salvatore

--
Salvatore 'antirez' Sanfilippo
open source developer - GoPivotal
http://invece.org

To "attack a straw man" is to create the illusion of having refuted a
proposition by replacing it with a superficially similar yet
unequivalent proposition (the "straw man"), and to refute it
— Wikipedia (Straw man page)

[Rhommel Lamas]

@antirez

Agree with that, what we are doing is in order to have a passive failover.

this way we ensure that our failover cluster will be in sync at all time and in 

case of any failover on the active region this cluster will continue to be in-sync.

We also ensure that whenever we activate the failover region due to a 

disaster recovery plan (last resort) we will have our data correctly replicated.

regards.