Redis Sentinel Automatic Failover not working, manual failover does (3.0.6)

[Alexander Cook]

Hi, 

I am new to setting up Redis Clusters, but I seem to have been able to set one up that works. I have a single master with two slaves as follows: 

10.100.0.000:7000> cluster nodes

b3a5363d1117e5e01eace29733eb0a111b1636a0 10.100.0.000:7000 myself,master - 0 0 4 connected 0-16383

e848b14114612e211492f16746b7e6c9bba8a3d6 10.100.0.000:7002 slave b3a5363d1117e5e01eace29733eb0a111b1636a0 0 1475527977448 4 connected

e82bee8308533e7ff227701be737b7f80d004955 10.100.0.000:7001 slave b3a5363d1117e5e01eace29733eb0a111b1636a0 0 1475527978450 4 connected

I'm able to do a manual failover using $ redis-cli -c -p 7001 cluster failover <node> 

However, when I try to have a sentinel do the failover, I always get a  -failover-abort-slave-timeout master mymaster 10.100.0.000 7000 error. 

Could someone please point me in the right direction? Here is my sentinel config before the system modifies it: 

sentinel monitor mymaster 10.100.0.000 7000 1 # just 1, so that I only need one sentinel

sentinel down-after-milliseconds mymaster 6000

sentinel failover-timeout mymaster 18000

sentinel config-epoch mymaster 0

Here are the log messages I get from the sentinel: 

23068:X 03 Oct 16:55:38.958 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.

23068:X 03 Oct 16:55:38.958 # Sentinel runid is e29e3e1ba19e44c7de4845bcace050aba6393cf4

23068:X 03 Oct 16:55:38.958 # +monitor master mymaster 10.100.0.000 7000 quorum 1

23068:X 03 Oct 16:55:53.268 # +sdown master mymaster 10.100.0.000 7000

23068:X 03 Oct 16:55:53.268 # +odown master mymaster 10.100.0.000 7000 #quorum 1/1

23068:X 03 Oct 16:55:53.268 # +new-epoch 1

23068:X 03 Oct 16:55:53.268 # +try-failover master mymaster 10.100.0.000 7000

23068:X 03 Oct 16:55:53.276 # +vote-for-leader e29e3e1ba19e44c7de4845bcace050aba6393cf4 1

23068:X 03 Oct 16:55:53.276 # +elected-leader master mymaster 10.100.0.000 7000

23068:X 03 Oct 16:55:53.276 # +failover-state-select-slave master mymaster 10.100.0.000 7000

23068:X 03 Oct 16:55:53.339 # +selected-slave slave 10.100.0.000:7001 10.100.0.000 7001 @ mymaster 10.100.0.000 7000

23068:X 03 Oct 16:55:53.339 * +failover-state-send-slaveof-noone slave 10.100.0.000:7001 10.100.0.000 7001 @ mymaster 10.100.0.000 7000

23068:X 03 Oct 16:55:53.422 * +failover-state-wait-promotion slave 10.100.0.000:7001 10.100.0.000 7001 @ mymaster 10.100.0.000 7000

23068:X 03 Oct 16:56:11.451 # -failover-abort-slave-timeout master mymaster 10.100.0.000 7000

23068:X 03 Oct 16:56:11.535 # Next failover delay: I will not start a failover before Mon Oct  3 16:56:29 2016

23068:X 03 Oct 16:56:29.982 # +new-epoch 2

23068:X 03 Oct 16:56:29.982 # +try-failover master mymaster 10.100.0.000 7000

23068:X 03 Oct 16:56:29.983 # +vote-for-leader e29e3e1ba19e44c7de4845bcace050aba6393cf4 2

23068:X 03 Oct 16:56:29.983 # +elected-leader master mymaster 10.100.0.000 7000

23068:X 03 Oct 16:56:29.983 # +failover-state-select-slave master mymaster 10.100.0.000 7000

23068:X 03 Oct 16:56:30.055 # +selected-slave slave 10.100.0.000:7001 10.100.0.000 7001 @ mymaster 10.100.0.000 7000

23068:X 03 Oct 16:56:30.055 * +failover-state-send-slaveof-noone slave 10.100.0.000:7001 10.100.0.000 7001 @ mymaster 10.100.0.000 7000

23068:X 03 Oct 16:56:30.113 * +failover-state-wait-promotion slave 10.100.0.000:7001 10.100.0.000 7001 @ mymaster 10.100.0.000 7000

23068:X 03 Oct 16:56:47.190 # -sdown master mymaster 10.100.0.000 7000

23068:X 03 Oct 16:56:47.190 # -odown master mymaster 10.100.0.000 7000

23068:X 03 Oct 16:56:48.130 # -failover-abort-slave-timeout master mymaster 10.100.0.000 7000

This is in Redis 3.0.6. Thank you in advance for any help or hints! 

Thanks, 

Alex (this was resubmitted cleaned up) 

[Tuco]

Sentinels are used for failover of normal master-slaves redis instances, and not in a cluster. 

A cluster can failover manually(by the failover command you mentioned) or automatically(if one master goes down).