Redis Sentinel master not downgraded to slave immediately

[Damian Blazejewski]

down votfavorite

I have an architecture with three Redis instances (one master and two slaves) and three Sentinel instances. In front of it there is a HaProxy. All works well until the master Redis instance goes down. The new master is properly chosen by Sentinel. However, the old master (which is now down) is not reconfigured to be a slave. As a result, when that instance is up again I have two masters for a short period of time (about 11 seconds). After that time that instance which was brought up is properly downgraded to slave. Shouldn't it work that way, that when the master goes down it is downgraded to slave straight away? Having that, when it was up again, it would be slave immediately. I know that (since Redis 2.8?) there is that CONFIG REWRITE functionality so the config cannot be modified when the Redis instance is down. Having two masters for some time is a problem for me because the HaProxy for that short period of time instead of sending requests to one master Redis, it does the load balancing between those two masters. Is there any way to downgrade the failed master to slave immediately? Obviously, I changed the Sentinel timeouts. Here are some logs from Sentinel and Redis instances after the master goes down: Sentinel 81358:X 23 Jan 22:12:03.088 # +sdown master redis-ha 127.0.0.1 63797.0.0.1 26381 @ redis-ha 127.0.0.1 6379 81358:X 23 Jan 22:12:03.149 # +new-epoch 1 81358:X 23 Jan 22:12:03.149 # +vote-for-leader 6b5b5882443a1d738ab6849ecf4bc6b9b32ec142 1 81358:X 23 Jan 22:12:03.174 # +odown master redis-ha 127.0.0.1 6379 #quorum 3/2 81358:X 23 Jan 22:12:03.174 # Next failover delay: I will not start a failover before Sat Jan 23 22:12:09 2016 81358:X 23 Jan 22:12:04.265 # +config-update-from sentinel 127.0.0.1:26381 127.0.0.1 26381 @ redis-ha 127.0.0.1 6379 81358:X 23 Jan 22:12:04.265 # +switch-master redis-ha 127.0.0.1 6379 127.0.0.1 6381 81358:X 23 Jan 22:12:04.266 * +slave slave 127.0.0.1:6380 127.0.0.1 6380 @ redis-ha 127.0.0.1 6381 81358:X 23 Jan 22:12:04.266 * +slave slave 127.0.0.1:6379 127.0.0.1 6379 @ redis-ha 127.0.0.1 6381 81358:X 23 Jan 22:12:06.297 # +sdown slave 127.0.0.1:6379 127.0.0.1 6379 @ redis-ha 127.0.0.1 6381 Redis 81354:S 23 Jan 22:12:03.341 * MASTER <-> SLAVE sync started 81354:S 23 Jan 22:12:03.341 # Error condition on socket for SYNC: Connection refused 81354:S 23 Jan 22:12:04.265 * Discarding previously cached master state. 81354:S 23 Jan 22:12:04.265 * SLAVE OF 127.0.0.1:6381 enabled (user request from 'id=7 addr=127.0.0.1:57784 fd=10 name=sentinel-6b5b5882-cmd age=425 idle=0 flags=x db=0 sub=0 psub=0 multi=3 qbuf=14 qbuf-free=32754 obl=36 oll=0 omem=0 events=rw cmd=exec') 81354:S 23 Jan 22:12:04.265 # CONFIG REWRITE executed with success. 81354:S 23 Jan 22:12:04.371 * Connecting to MASTER 127.0.0.1:6381 81354:S 23 Jan 22:12:04.371 * MASTER <-> SLAVE sync started 81354:S 23 Jan 22:12:04.371 * Non blocking connect for SYNC fired the event. 81354:S 23 Jan 22:12:04.371 * Master replied to PING, replication can continue... 81354:S 23 Jan 22:12:04.371 * Partial resynchronization not possible (no cached master) 81354:S 23 Jan 22:12:04.372 * Full resync from master: 07b3c8f64bbb9076d7e97799a53b8b290ecf470b:1 81354:S 23 Jan 22:12:04.467 * MASTER <-> SLAVE sync: receiving 860 bytes from master 81354:S 23 Jan 22:12:04.467 * MASTER <-> SLAVE sync: Flushing old data 81354:S 23 Jan 22:12:04.467 * MASTER <-> SLAVE sync: Loading DB in memory 81354:S 23 Jan 22:12:04.467 * MASTER <-> SLAVE sync: Finished with success

[Greg Andrews]

Having two masters for some time is a problem for me because the HaProxy for that short period of time instead of sending requests to one master Redis, it does the load balancing between those two masters.

That's not behavior I would expect.  Since Redis slaves can only replicate from one master, it doesn't make sense for haproxy to behave as if there are two masters in the same cluster.  There should only be one.  Is there an haproxy config option that will prevent it from sending traffic to two masters like that?

How would Sentinel win the race with haproxy to reach the revived master first and configure it as a slave before haproxy sees it?  Should Sentinel try several times per second to make a connection?

I've said this in a different thread a few weeks ago, but I don't see how mixing haproxy and Sentinel will work well when each of them has been developed to work independently of the other rather than in cooperation.  Sentinel thinks it doesn't need to reconfigure the revived master within milliseconds, because the clients are supposed to ask Sentinel which servers to use, and Sentinel will point them away from that master until its ready as a slave.  Haproxy doesn't seem to realize Sentinel is part of the picture and, as you point out, sends write traffic to the wrong Redis instance because it started again.

  -Greg

[The Real Bill]

On Tuesday, January 26, 2016 at 4:34:37 PM UTC-6, Damian Blazejewski wrote:

down votfavorite

I have an architecture with three Redis instances (one master and two slaves) and three Sentinel instances. In front of it there is a HaProxy. All works well until the master Redis instance goes down. The new master is properly chosen by Sentinel. However, the old master (which is now down) is not reconfigured to be a slave. As a result, when that instance is up again I have two masters for a short period of time (about 11 seconds). After that time that instance which was brought up is properly downgraded to slave. Shouldn't it work that way, that when the master goes down it is downgraded to slave straight away?

How? The master is DOWN, which means you can not talk to it to reconfigure it. You have to wait until it is available again and responsive before you can reconfigure it. But that isn't the problem here. 

The problem here is the way you configure HAProxy. Though you've left that key description out we can surmise that you configure HAProxy to talk to both instances at the same time. This is the path to dark days. The proper way to use HAProxy in front of a Redis pod is to configure it to talk to the master *only*. Then you use the sentinel client-reconfigure-script setting in Sentinel to update your HAProxy config to point to the new master on failover - removing the old master from the configuration.

It is in the way that you can easily and accurately keep HAProxy aimed at *only* the master. There are some "are you master" checks for HAProxy floating around out there and they are terrible in that they lead to the problem you see here. You have to think of HAProxy as a *client* to Redis, not merely a proxy, and do what a client would do. Since HAProxy is ignorant of the Redis protocol and Sentinel it won't do it for you. I've seen some TCP checks which take to Sentinel but they are problematic in that they do not fully prevent this situation from occurring and there are issues with the way HAProxy handles connections when failing a backend which prevent the setup from working as it should.

Cheers,

Bill