Redis-Sentinel start a loop when the master instance is down

[Raul Jimenez]

Hello,

I have been testing Redis with HA (Redis + Sentinel) and have been following several articles available.

This is the configuration I have so far

Redis HA:
3 nodes, 1 Master 2 Slaves, one slave writing to disk with the higher slave value

Master IP: 10.155.3.77

Master Sentinel.conf:

Port 10000
sentinel monitor secure 10.115.3.77 6379 2
sentinel down-after-milliseconds secure 3000
sentinel failover-timeout secure 60000
sentinel auth-pass secure RSW_SS,8FKB1<8Fs:?
sentinel parallel-syncs secure 1

Slave 1 IP: 10.155.3.76

Slave 1 sentinel.conf:
Port 10001
sentinel monitor secure 10.115.3.77 6379 2
sentinel down-after-milliseconds secure 3000
sentinel failover-timeout secure 60000
sentinel auth-pass secure RSW_SS,8FKB1<8Fs:?
sentinel parallel-syncs secure 1

Slave 2 IP: 10.155.2.50

Slave 2 sentinel.conf:
Port 10002
sentinel monitor secure 10.115.3.77 6379 2
sentinel down-after-milliseconds secure 3000
sentinel failover-timeout secure 60000
sentinel auth-pass secure RSW_SS,8FKB1<8Fs:?
sentinel parallel-syncs secure 1

However when I stop Redis in the Master Instance, Sentinel start a loop between the slaves and never finish migrating to the new Master.

Any guidance on this will be appreciate it

Thanks

[Jan-Erik Rediger]

Could you attach the log from your sentinels?

> --
> You received this message because you are subscribed to the Google Groups "Redis DB" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to redis-db+u...@googlegroups.com.
> To post to this group, send email to redi...@googlegroups.com.
> Visit this group at http://groups.google.com/group/redis-db.
> For more options, visit https://groups.google.com/d/optout.

[Raul Jimenez]

Thanks Janerik. This log is the Master Sentinel
_._
_.-``__ ''-._
_.-`` `. `_. ''-._ Redis 2.8.17 (00000000/0) 64 bit
.-`` .-```. ```\/ _.,_ ''-._
( ' , .-` | `, ) Running in sentinel mode
|`-._`-...-` __...-.``-._|'` _.-'| Port: 10000
| `-._ `._ / _.-' | PID: 4296
`-._ `-._ `-./ _.-' _.-'
|`-._`-._ `-.__.-' _.-'_.-'|
| `-._`-._ _.-'_.-' | http://redis.io
`-._ `-._`-.__.-'_.-' _.-'
|`-._`-._ `-.__.-' _.-'_.-'|
| `-._`-._ _.-'_.-' |
`-._ `-._`-.__.-'_.-' _.-'
`-._ `-.__.-' _.-'
`-._ _.-'
`-.__.-'

[4296] 31 Jul 14:03:02.236 # Sentinel runid is 270b8b320393ce2d1ac64db3ffd4f6c0615ba998
[4296] 31 Jul 14:03:02.236 # +monitor master secure 10.115.3.77 6379 quorum 2
[4296] 31 Jul 14:03:02.251 * +slave slave 10.115.3.76:6380 10.115.3.76 6380 @ secure 10.115.3.77 6379
[4296] 31 Jul 14:03:02.251 * +slave slave 10.115.2.50:6381 10.115.2.50 6381 @ secure 10.115.3.77 6379
[4296] 31 Jul 14:10:13.392 * +sentinel sentinel 10.115.2.50:10002 10.115.2.50 10002 @ secure 10.115.3.77 6379
[4296] 31 Jul 14:12:39.001 * +sentinel sentinel 10.115.3.76:10001 10.115.3.76 10001 @ secure 10.115.3.77 6379
[4296] 31 Jul 14:57:05.751 # Accepting client connection: accept: Unknown error
_._
_.-``__ ''-._
_.-`` `. `_. ''-._ Redis 2.8.17 (00000000/0) 64 bit
.-`` .-```. ```\/ _.,_ ''-._
( ' , .-` | `, ) Running in sentinel mode
|`-._`-...-` __...-.``-._|'` _.-'| Port: 10000
| `-._ `._ / _.-' | PID: 4844
`-._ `-._ `-./ _.-' _.-'
|`-._`-._ `-.__.-' _.-'_.-'|
| `-._`-._ _.-'_.-' | http://redis.io
`-._ `-._`-.__.-'_.-' _.-'
|`-._`-._ `-.__.-' _.-'_.-'|
| `-._`-._ _.-'_.-' |
`-._ `-._`-.__.-'_.-' _.-'
`-._ `-.__.-' _.-'
`-._ _.-'
`-.__.-'

[4844] 31 Jul 15:08:48.939 # Sentinel runid is 560599b4ccb8fd1cbe68907728f3f56f6961a19e
[4844] 31 Jul 15:08:48.939 # +monitor master secure 10.115.3.77 6379 quorum 2
[4844] 31 Jul 15:08:49.001 # +new-epoch 12
[4844] 31 Jul 15:08:51.954 # +sdown master secure 10.115.3.77 6379
[4844] 31 Jul 15:08:52.017 # +odown master secure 10.115.3.77 6379 #quorum 3/2
[4844] 31 Jul 15:08:52.017 # +new-epoch 13
[4844] 31 Jul 15:08:52.017 # +try-failover master secure 10.115.3.77 6379
[4844] 31 Jul 15:08:52.017 # +vote-for-leader 560599b4ccb8fd1cbe68907728f3f56f6961a19e 13
[4844] 31 Jul 15:08:52.033 # 10.115.2.50:10002 voted for 560599b4ccb8fd1cbe68907728f3f56f6961a19e 13
[4844] 31 Jul 15:08:52.033 # 10.115.3.76:10001 voted for 560599b4ccb8fd1cbe68907728f3f56f6961a19e 13
[4844] 31 Jul 15:08:52.111 # +elected-leader master secure 10.115.3.77 6379
[4844] 31 Jul 15:08:52.111 # +failover-state-select-slave master secure 10.115.3.77 6379
[4844] 31 Jul 15:08:52.173 # +selected-slave slave 10.115.3.76:6380 10.115.3.76 6380 @ secure 10.115.3.77 6379
[4844] 31 Jul 15:08:52.173 * +failover-state-send-slaveof-noone slave 10.115.3.76:6380 10.115.3.76 6380 @ secure 10.115.3.77 6379
[4844] 31 Jul 15:08:52.236 * +failover-state-wait-promotion slave 10.115.3.76:6380 10.115.3.76 6380 @ secure 10.115.3.77 6379
[4844] 31 Jul 15:08:55.111 # -odown master secure 10.115.3.77 6379
[4844] 31 Jul 15:09:04.111 # -sdown master secure 10.115.3.77 6379
[4844] 31 Jul 15:09:52.283 # -failover-abort-slave-timeout master secure 10.115.3.77 6379
[4844] 31 Jul 15:12:57.626 # +sdown sentinel 10.115.2.50:10002 10.115.2.50 10002 @ secure 10.115.3.77 6379
[4844] 31 Jul 15:13:00.939 # +sdown sentinel 10.115.3.76:10001 10.115.3.76 10001 @ secure 10.115.3.77 6379
[4844] 31 Jul 15:13:01.079 # Accepting client connection: accept: Unknown error