Redis-cluster failover not working.

[kuka TeRRaN]

Hello community!

In recent weeks, studying the work of the new version of redis-cluster 3.0 and can not understand why the failover does not work from the slave servers.

Cluster configuration is as follows:

192.168.1.231:6379> CLUSTER NODES
ada13a7cff7daa339c5d07cb5e0197d1ea33a484 192.168.1.233:6379 slave 1de4a0ba5cdbb865221ef30e46069d8800b2730f 1430224802404 0 3 connected
1de4a0ba5cdbb865221ef30e46069d8800b2730f 192.168.1.231:6379 myself, master - 0 0 1 connected 0-16383
e00065feff6cf707b138ccc23396de084498d63e 192.168.1.232:6379 slave 1de4a0ba5cdbb865221ef30e46069d8800b2730f 0 1430224801404 2 connected
192.168.1.231:6379> CLUSTER INFO
cluster_state: ok
cluster_slots_assigned: 16384
cluster_slots_ok: 16384
cluster_slots_pfail: 0
cluster_slots_fail: 0
cluster_known_nodes: 3
cluster_size: 1
cluster_current_epoch: 7
cluster_my_epoch: 1
cluster_stats_messages_sent: 3906
cluster_stats_messages_received: 3906

When we disable the master node (service redis-server stop), Slave server does not go into the status of the Master:

192.168.1.232:6379> CLUSTER NODES
1de4a0ba5cdbb865221ef30e46069d8800b2730f 192.168.1.231:6379 master, fail? - 1430225104021 1430225103921 1 disconnected 0-16383
e00065feff6cf707b138ccc23396de084498d63e 192.168.1.232:6379 myself, slave 1de4a0ba5cdbb865221ef30e46069d8800b2730f 0 0 2 connected
ada13a7cff7daa339c5d07cb5e0197d1ea33a484 192.168.1.233:6379 slave 1de4a0ba5cdbb865221ef30e46069d8800b2730f 0 1 1430225131625 connected
192.168.1.232:6379> CLUSTER INFO
cluster_state: fail
cluster_slots_assigned: 16384
cluster_slots_ok: 0
cluster_slots_pfail: 16384
cluster_slots_fail: 0
cluster_known_nodes: 3
cluster_size: 1
cluster_current_epoch: 7
cluster_my_epoch: 1
cluster_stats_messages_sent: 2291
cluster_stats_messages_received: 1630

Configuration of servers in the cluster of the same and is as follows:

cluster-enabled yes
cluster-config-file nodes.conf
cluster-node-timeout 15000
cluster-slave-validity-factor 0
cluster-migration-barrier 1

Can you help us understand this issue? Why Slave server does not want to take over the work of the master server?

If you will need additional data or settings of the problem - we will be happy to provide them.

[Salvatore Sanfilippo]

Hello, you need at least 3 masters for it to work. Cheers.

> --
> You received this message because you are subscribed to the Google Groups
> "Redis DB" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to redis-db+u...@googlegroups.com.
> To post to this group, send email to redi...@googlegroups.com.
> Visit this group at http://groups.google.com/group/redis-db.
> For more options, visit https://groups.google.com/d/optout.



--
Salvatore 'antirez' Sanfilippo
open source developer - Pivotal http://pivotal.io

"If a system is to have conceptual integrity, someone must control the
concepts."
— Fred Brooks, "The Mythical Man-Month", 1975.

[Amolak Gill]

Have you configured redis-sentinels? if yes, please attach the sentinel configs.

if no, you need sentinels which monitor the health of the redis nodes. they are responsible for triggering failovers.

[Salvatore Sanfilippo]

No Sentinels needed for Redis Cluster. Cheers, Salvatore.

[kuka TeRRaN]

Thanks for the quick response!

With 3 master servers, failover is working correctly!

Then focus on this scheme, thanks again!

Redis-sentinel we do not use.

вторник, 28 апреля 2015 г., 19:22:05 UTC+6 пользователь Salvatore Sanfilippo написал:

[Sonal Kumar]

Hi

I am facing the same issue.
I tried the same with 4 master and 2 slaves.
making 1 master fail to check if slave of that master becomes master but it did not become master.
Could you please help how you achieved this..?

172.31.35.35:6384> cluster nodes
71fecd6da1be74dde8f5b5334e712f9649a2dcb0 172.31.35.35:6381@16381 master - 0 1682434563876 3 connected 10923-16383
378d750792214a99d1bbe147cb4f84cba1644890 172.31.35.35:6382@16382 slave 71fecd6da1be74dde8f5b5334e712f9649a2dcb0 1682434562874 1682434560870 3 connected
d0f38c3199d1d10be4fc1eaa1cab52b1058bfd85 172.31.35.35:6380@16380 master - 0 1682434563000 1 connected 5461-10922
5772da3f3050592c4a4447e9940abc00b50cdc41 172.31.35.35:6384@16384 myself,slave 087e8a88a64977a11d935dff25d53e4d9d343793 0 1682434560000 80 connected
087e8a88a64977a11d935dff25d53e4d9d343793 172.31.35.35:6383@16383 master,fail - 1682430188234 1682430188234 80 disconnected
51b8787d044772a52b3e8e7889ebf604eeff9e45 172.31.35.35:6379@16379 master - 0 1682434563000 79 connected 0-5460

I failed master port:6383 and its slave port:6384 should have become master but it did not.
could you please help me by pointing out what am I missing as according to above discussion it looks like its possible.

Configuration of servers in the cluster 

port 6382
dir /var/lib/redis/6382
appendonly no
protected-mode no
cluster-enabled yes
cluster-node-timeout 5000
cluster-config-file /etc/redis/cluster/6382/nodes_6382.conf
pidfile /var/run/redis/redis_6382.pid
logfile /var/log/redis/redis_6382.log
loglevel notice

Any help is appreciated.

[Sonal Kumar]

I resolved this.
In case anyone comes across this -same error - the issue was .conf files and dump files had some users as root instead of redis.
after making al users as redis it resolved the issue and I was able to test redis cluster automatic failover
Ex:
drwxr-xr-x  2 root  root  4096 Apr 26 10:44 file_1/