[feature request] sticky slave whose slaveof config won't be changed by Redis sentinel [WAS: questions about cross datacenter replication]

[Liu Yubao]

Just made a patch to resolve it,  https://github.com/Dieken/redis/tree/sticky-slave.

The patch isn't perfect,  in sentinelRefreshInstanceInfo() Sentinel collects all slaves and prints "+slave" messages,  actually those special slaves are ignored during slaveof change and they are not slaves of new master at all:

(1) initial state

A1 <---- B1, A2, A3

A2 <-----B2
A3 <-----B3

(2) kill A1,  elect A2 to be new master

A1 <--- B1  (serve stale data or reject read)

A2 <--- B2, A3

A3 <--- B3

Now sentinel will think B1 is slave of A2 but actually the slaveof command to B1 is dropped.  If we kill A2, then A3 becomes new master and sentinel will think B1/B2 are slaves of A3,  the new master likes a magnet,  at last all slaves will be regarded to synchronized from the single master,  this false information breaks Sentinel API "SENTINEL masters" and "SENTINEL slaves".

I planned to change sentinelRefreshInstanceInfo() to not add sticky slaves into sentinelRedisInstance.slaves but Redis "INFO" command doesn't show slave_priority for "slaveN" lines (I use "slave_priority -1" to mark sticky slave),  it looks like a race condition if I remove a slave from sentinelRedisInstance.slaves after checking Redis "INFO" for that slave later.

Ideal solution is to change implementation of setting "slaveof" and "slave-priority" to tell master the slave_priority value(or a sticky_slave flag) of a slave,  so Sentinel can just ignore sticky slaves according to INFO response of the Redis master.  But this looks a not so small change if sticky_slave flag can dynamically change at runtime.

Regards,
Yubao Liu

On Fri, Nov 29, 2013 at 2:41 AM, Liu Yubao <yuba...@gmail.com> wrote:

I wouldn't like to enable dynamic slaveof change in data center B, because sooner or later Redis sentinel will make all slaves follows same master, in my use case, 5 slaves follow single master,  not maximum 3 slaves follow single master,  that may introduce high load on the Redis master.

Regards,
Yubao Liu

On Fri, Nov 29, 2013 at 2:37 AM, Liu Yubao <yuba...@gmail.com> wrote:

Got help from huangz1990@github,  my use case can be achieved by those two settings in three Redis instances in datacenter B:

slave-priority 0
rename-command slaveof ""

With this setting, suppose A1 is master,  after kill A1 Redis sentinel will elect A2 or A3 as new master, won't elect B1/B2/B3,  and the slaveof setting of B1/B2/B3 won't be changed(this is expected, I want mirror a Redis cluster into another datacenter),  but Redis sentinel continuously reports this error message every 10 seconds:

+fix-slave-config slave 127.0.0.1:7379 127.0.0.1 7379 @ mymaster 127.0.0.1 6399

This is annoying because it will eat up my disk, any solution except patching Redis to allow this special kind of slave?

The configuration for the six Redis instances are almost same with default settings, except 6 different ports: A1 6379, A2 6389, A3 6399,  B1 7379, B2 7389, B3, 7399.  The master-slave relation is A2/A3/B1 follows A1,  B2 follows A2, B3 follows A3.

Regards,
Yubao Liu

On Thu, Nov 28, 2013 at 9:17 AM, Liu Yubao <yuba...@gmail.com> wrote:

Hi all,

I want to setup two Redis clusters in two datacenters A and B,  A is master DC which all writes happen,
B is slave DC which just synchronize whole data set from A.

A1  <-----------stunnel--------- B1
A2  <-----------stunnel--------- B2
A3  <-----------stunnel--------- B3

In datacenter A I plan to use Redis sentinel for the three Redis instances A1/A2/A3,  one of them is Redis master,
two of them are Redis slaves.   In datacenter B all Redis instances are slaves.

I haven't verified it, according to Redis documentation,  when a failover in DC A happens,  Redis sentinel will
elect a slave to become master,  so it possibly selects a slave in DC B,  this is unexpected,  I expect
the slaveof relationship is fixed between A and B,  that's to say,  B_i  always tracks A_i.

How can I avoid Redis sentinel selecting slave in B?  by disabling slaveof command in B?

Another related question,  seems I can't use the "Redis Cluster" solution in DC A,  because B1/B2/B3 don't form
a "Redis Cluster"(they sync from A), the clients in DC B can't know the dynamically changing partition mapping,
is my guess right?

Regards,
Yubao Liu

--
You received this message because you are subscribed to the Google Groups "Redis DB" group.
To unsubscribe from this group and stop receiving emails from it, send an email to redis-db+u...@googlegroups.com.
To post to this group, send email to redi...@googlegroups.com.
Visit this group at http://groups.google.com/group/redis-db.
For more options, visit https://groups.google.com/groups/opt_out.

[Liu Yubao]

Seems I got it with extra one line patch:

@@ -1181,6 +1183,7 @@ int sentinelResetMasterAndChangeAddress(sentinelRedisInstance *master, char *ip,
         sentinelRedisInstance *slave = dictGetVal(de);

         if (sentinelAddrIsEqual(slave->addr,newaddr)) continue;
+        if (sentinelIsStickySlave(slave)) continue;
         slaves = zrealloc(slaves,sizeof(sentinelAddr*)*(numslaves+1));
         slaves[numslaves++] = createSentinelAddr(slave->addr->ip,
                                                  slave->addr->port);

So the sticky slaves following old master are not inserted to new master's slave list. 
And if we obey the rule slave-priority won't change when it's a slave, there is no race condition

in sentinel to accidentally send "slaveof" command to sticky slave.

I have sent a pull request, please review it and merge to unstable and 2.8.x, thanks!
https://github.com/antirez/redis/pull/1413

Regards,

Yubao Liu