Sentinel Failover version 3.2.9

[Itamar Melamed]

Hi All,

We wanted to upgrade our production environment from Redis version 3.0.7 to 3.2.9
(Version 3.0.7 is currently working perfect for us, the upgrade is just for being up-to-date)
Our Redis environment  runs 3 instances = 1 master + 2 slaves synced and running sentinel on Ubuntu servers
The sentinel master election in fail-over works perfect in this version - 3.0.7

I guess I have missed something or doing something wrong but in the last version 3.2.9 same sentinel configuration I was using in 3.0.7 is not working.
I can see the known-sentinel objects and I do see PING and HELO in "redis-cli monitor" but the fail over isn't working :(
actually  I am getting sdown for sentinel object even-though they send HELOs and PINGs to quorum just in the right time.
when master is down, sdown master is received but no election process starts.

Please Advise.

Thanks,
Itamar


Sentinel Configuration we are using:

ita@imredis01:~$ grep ^[^#] /etc/redis/sentinel.conf
daemonize yes
pidfile "/var/run/redis-sentinel.pid"
loglevel notice
logfile "/var/log/redis-sentinel.log"
port 26379
dir "/tmp"
sentinel myid 6ef01249bc50e4ec2cae048719843664342a7844
sentinel monitor mymaster 192.168.13.11 6379 2
sentinel config-epoch mymaster 0
sentinel leader-epoch mymaster 0
sentinel known-slave mymaster 192.168.13.13 6379
sentinel known-slave mymaster 192.168.13.12 6379
sentinel known-sentinel mymaster 192.168.13.13 26379 65402554357ce078e9716c3c10b484da8c4ad426
sentinel known-sentinel mymaster 192.168.13.12 26379 5612b895e35d91089676d482188dae2a95d97202
sentinel current-epoch 0
ita@imredis01:~$
______________________________________________________________________________________________________________

root@imredis02:/usr/local/bin#  grep ^[^#] /etc/redis/sentinel.conf
daemonize yes
pidfile "/var/run/redis-sentinel.pid"
loglevel notice
logfile "/var/log/redis-sentinel.log"
port 26379
dir "/tmp"
sentinel myid 5612b895e35d91089676d482188dae2a95d97202
sentinel monitor mymaster 192.168.13.11 6379 2
sentinel config-epoch mymaster 0
sentinel leader-epoch mymaster 0
sentinel known-slave mymaster 192.168.13.13 6379
sentinel known-slave mymaster 192.168.13.12 6379
sentinel known-sentinel mymaster 192.168.13.13 26379 65402554357ce078e9716c3c10b484da8c4ad426
sentinel known-sentinel mymaster 192.168.13.11 26379 6ef01249bc50e4ec2cae048719843664342a7844
sentinel current-epoch 0
______________________________________________________________________________________________________________

root@imredis03:~# grep ^[^#] /etc/redis/sentinel.conf
daemonize yes
pidfile "/var/run/redis-sentinel.pid"
loglevel notice
logfile "/var/log/redis-sentinel.log"
port 26379
dir "/tmp"
sentinel myid 65402554357ce078e9716c3c10b484da8c4ad426
sentinel monitor mymaster 192.168.13.11 6379 2
sentinel config-epoch mymaster 0
sentinel leader-epoch mymaster 0
sentinel known-slave mymaster 192.168.13.12 6379
sentinel known-slave mymaster 192.168.13.13 6379
sentinel known-sentinel mymaster 192.168.13.12 26379 5612b895e35d91089676d482188dae2a95d97202
sentinel known-sentinel mymaster 192.168.13.11 26379 6ef01249bc50e4ec2cae048719843664342a7844
sentinel current-epoch 0
root@imredis03:~#

root@imredis03:~# more /var/log/redis-sentinel.log
1381:X 06 Jun 15:48:18.812 * Increased maximum number of open files to 10032 (it was originally set to 1024).
                _._
           _.-``__ ''-._
      _.-``    `.  `_.  ''-._           Redis 3.2.9 (00000000/0) 64 bit
  .-`` .-```.  ```\/    _.,_ ''-._
 (    '      ,       .-`  | `,    )     Running in sentinel mode
 |`-._`-...-` __...-.``-._|'` _.-'|     Port: 26379
 |    `-._   `._    /     _.-'    |     PID: 1381
  `-._    `-._  `-./  _.-'    _.-'
 |`-._`-._    `-.__.-'    _.-'_.-'|
 |    `-._`-._        _.-'_.-'    |           http://redis.io
  `-._    `-._`-.__.-'_.-'    _.-'
 |`-._`-._    `-.__.-'    _.-'_.-'|
 |    `-._`-._        _.-'_.-'    |
  `-._    `-._`-.__.-'_.-'    _.-'
      `-._    `-.__.-'    _.-'
          `-._        _.-'
              `-.__.-'

1381:X 06 Jun 15:48:18.814 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the
lower value of 128.
1381:X 06 Jun 15:48:18.818 # Sentinel ID is 65402554357ce078e9716c3c10b484da8c4ad426
1381:X 06 Jun 15:48:18.818 # +monitor master mymaster 192.168.13.11 6379 quorum 2
1381:X 06 Jun 15:48:18.819 * +slave slave 192.168.13.13:6379 192.168.13.13 6379 @ mymaster 192.168.13.11 6379
1381:X 06 Jun 15:48:18.823 * +slave slave 192.168.13.12:6379 192.168.13.12 6379 @ mymaster 192.168.13.11 6379
1381:X 06 Jun 15:48:19.133 * +sentinel sentinel 6ef01249bc50e4ec2cae048719843664342a7844 192.168.13.11 26379 @ mymaster 192.168.13.11 6379
1381:X 06 Jun 15:48:20.047 * +sentinel sentinel 5612b895e35d91089676d482188dae2a95d97202 192.168.13.12 26379 @ mymaster 192.168.13.11 6379
1381:X 06 Jun 15:48:49.162 # +sdown sentinel 6ef01249bc50e4ec2cae048719843664342a7844 192.168.13.11 26379 @ mymaster 192.168.13.11 6379
1381:X 06 Jun 15:48:50.058 # +sdown sentinel 5612b895e35d91089676d482188dae2a95d97202 192.168.13.12 26379 @ mymaster 192.168.13.11 6379
1381:X 06 Jun 16:10:28.954 # +sdown master mymaster 192.168.13.11 6379

[Jay Rolette]

Haven't tested against 3.2.9 yet, but sentinel failover is working fine for us on 3.2.8.

You may be running into problems with "protected mode", which was added in 3.2. See the comments in the stock version of sentinel.conf.

Jay

--
You received this message because you are subscribed to the Google Groups "Redis DB" group.
To unsubscribe from this group and stop receiving emails from it, send an email to redis-db+unsubscribe@googlegroups.com.
To post to this group, send email to redi...@googlegroups.com.
Visit this group at https://groups.google.com/group/redis-db.
For more options, visit https://groups.google.com/d/optout.

[Itamar Melamed]

That's it removing the comment mark from "protection mode no" in sentinel.config file

Thanks! 

To unsubscribe from this group and stop receiving emails from it, send an email to redis-db+u...@googlegroups.com.