adding a new node to cluster via CLI is not forming a cluster

[Allada lakshmi sowjanya]

Hi All,

I am trying to scale the cluster by adding an addition node as recommended in the doc. Cluster meet command is working fine. But wanted to understand why redis-cli command is failing as redis official doc is recommending to use redis-cli to add a node.

Steps followed:

Added an empty node to the cluster on port 7008

redis-cli --cluster add-node 127.0.0.1:7008 127.0.0.1:7000

On running the above command

vagrant@vagrant:~/redis-stable$ redis-cli --cluster add-node 127.0.0.1:7008 127.0.0.1:7000
>>> Adding node 127.0.0.1:7008 to cluster 127.0.0.1:7000
>>> Performing Cluster Check (using node 127.0.0.1:7000)
M: 96d8dce113386caa1388f96df2ea5ff96f9d3dee 127.0.0.1:7000
   slots:[0-5460] (5461 slots) master
   2 additional replica(s)
S: 84c9a3c36688cb6dfc4f1876ea24478e5c0a6552 127.0.0.1:20000
   slots: (0 slots) slave
   replicates 96d8dce113386caa1388f96df2ea5ff96f9d3dee
S: 99f77c114d2badd5542d6cb64c8011f4cc49462b 127.0.0.1:7006
   slots: (0 slots) slave
   replicates 96d8dce113386caa1388f96df2ea5ff96f9d3dee
S: dc9ab96a1cbf689c16344b9bd6765529b68a3d14 127.0.0.1:7001
   slots: (0 slots) slave
   replicates 2bd1531ea0937b5a00076baeb5ae0d0e96a9162e
M: 2bd1531ea0937b5a00076baeb5ae0d0e96a9162e 127.0.0.1:7002
   slots:[5461-10922] (5462 slots) master
   1 additional replica(s)
M: d7849b792208433e8f2e4d90999aa629fd6de652 127.0.0.1:7003
   slots:[10923-16383] (5461 slots) master
   1 additional replica(s)
S: 3bd809889e8afea60271c18c47e717b316539add 127.0.0.1:7005
   slots: (0 slots) slave
   replicates d7849b792208433e8f2e4d90999aa629fd6de652
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
[WARNING] Node 127.0.0.1:7000 has slots in importing state 12789,14553.
[WARNING] Node 127.0.0.1:7002 has slots in importing state 1684,12789,14553.
[WARNING] Node 127.0.0.1:7003 has slots in importing state 1684.
[WARNING] The following slots are open: 1684,12789,14553.
>>> Check slots coverage...
[OK] All 16384 slots covered.

But when I look into the cluster, they have not formed the cluster at all. And also observe that there are few slots which 7008 port is holding from the cluster which is running. 

vagrant@vagrant:~/redis-stable$ redis-cli -c -p  7008
127.0.0.1:7008> cluster nodes
030105952740cbb648322d12dc93e53a47a90687 127.0.0.1:7008@17008 myself,master - 0 0 0 connected 1684 12789 14553
127.0.0.1:7008> cluster info
cluster_state:fail
cluster_slots_assigned:3
cluster_slots_ok:3
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:1
cluster_size:1
cluster_current_epoch:0
cluster_my_epoch:0
cluster_stats_messages_sent:0
cluster_stats_messages_received:0
127.0.0.1:7008> cluster slots
1) 1) (integer) 1684
   2) (integer) 1684
   3) 1) "127.0.0.1"
      2) (integer) 7008
      3) "030105952740cbb648322d12dc93e53a47a90687"
2) 1) (integer) 12789
   2) (integer) 12789
   3) 1) "127.0.0.1"
      2) (integer) 7008
      3) "030105952740cbb648322d12dc93e53a47a90687"
3) 1) (integer) 14553
   2) (integer) 14553
   3) 1) "127.0.0.1"
      2) (integer) 7008
      3) "030105952740cbb648322d12dc93e53a47a90687"
127.0.0.1:7008> cluster slots
1) 1) (integer) 1684
   2) (integer) 1684
   3) 1) "127.0.0.1"
      2) (integer) 7008
      3) "030105952740cbb648322d12dc93e53a47a90687"
2) 1) (integer) 12789
   2) (integer) 12789
   3) 1) "127.0.0.1"
      2) (integer) 7008
      3) "030105952740cbb648322d12dc93e53a47a90687"
3) 1) (integer) 14553
   2) (integer) 14553
   3) 1) "127.0.0.1"
      2) (integer) 7008
      3) "030105952740cbb648322d12dc93e53a47a90687"
127.0.0.1:7008> 

vagrant@vagrant:~/redis-stable$ redis-cli -c -p  7000
127.0.0.1:7000> cluster nodes
84c9a3c36688cb6dfc4f1876ea24478e5c0a6552 127.0.0.1:20000@30000 slave 96d8dce113386caa1388f96df2ea5ff96f9d3dee 0 1576646683000 1 connected
98b89c9bc230678764b7ed7ac1a9990d9f2b6d77 :0@0 slave,fail,noaddr 2bd1531ea0937b5a00076baeb5ae0d0e96a9162e 1576645274981 1576645274981 2 disconnected
99f77c114d2badd5542d6cb64c8011f4cc49462b 127.0.0.1:7006@17006 slave 96d8dce113386caa1388f96df2ea5ff96f9d3dee 0 1576646684000 6 connected
dc9ab96a1cbf689c16344b9bd6765529b68a3d14 127.0.0.1:7001@17001 slave 2bd1531ea0937b5a00076baeb5ae0d0e96a9162e 0 1576646684000 4 connected
2bd1531ea0937b5a00076baeb5ae0d0e96a9162e 127.0.0.1:7002@17002 master - 0 1576646685337 2 connected 5461-10922
d7849b792208433e8f2e4d90999aa629fd6de652 127.0.0.1:7003@17003 master - 0 1576646683310 3 connected 10923-16383
96d8dce113386caa1388f96df2ea5ff96f9d3dee 127.0.0.1:7000@17000 myself,master - 0 1576646680000 1 connected 0-5460 [12789-<-d7849b792208433e8f2e4d90999aa629fd6de652] [14553-<-d7849b792208433e8f2e4d90999aa629fd6de652]
3bd809889e8afea60271c18c47e717b316539add 127.0.0.1:7005@17005 slave d7849b792208433e8f2e4d90999aa629fd6de652 0 1576646684324 5 connected
127.0.0.1:7000> cluster info
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:8
cluster_size:3
cluster_current_epoch:6
cluster_my_epoch:1
cluster_stats_messages_ping_sent:4582
cluster_stats_messages_pong_sent:1088
cluster_stats_messages_sent:5670
cluster_stats_messages_ping_received:1087
cluster_stats_messages_pong_received:1152
cluster_stats_messages_meet_received:1
cluster_stats_messages_received:2240
127.0.0.1:7000> cluster slots
1) 1) (integer) 5461
   2) (integer) 10922
   3) 1) "127.0.0.1"
      2) (integer) 7002
      3) "2bd1531ea0937b5a00076baeb5ae0d0e96a9162e"
   4) 1) "127.0.0.1"
      2) (integer) 7001
      3) "dc9ab96a1cbf689c16344b9bd6765529b68a3d14"
2) 1) (integer) 10923
   2) (integer) 16383
   3) 1) "127.0.0.1"
      2) (integer) 7003
      3) "d7849b792208433e8f2e4d90999aa629fd6de652"
   4) 1) "127.0.0.1"
      2) (integer) 7005
      3) "3bd809889e8afea60271c18c47e717b316539add"
3) 1) (integer) 0
   2) (integer) 5460
   3) 1) "127.0.0.1"
      2) (integer) 7000
      3) "96d8dce113386caa1388f96df2ea5ff96f9d3dee"
   4) 1) "127.0.0.1"
      2) (integer) 7006
      3) "99f77c114d2badd5542d6cb64c8011f4cc49462b"
   5) 1) "127.0.0.1"
      2) (integer) 20000
      3) "84c9a3c36688cb6dfc4f1876ea24478e5c0a6552"
127.0.0.1:7000> 

Can someone help me understand the issue here. But when I use cluster meet command it is forming a cluster

vagrant@vagrant:~/redis-stable$ redis-cli -c -p  7000
127.0.0.1:7000> cluster info
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:9
cluster_size:3
cluster_current_epoch:6
cluster_my_epoch:1
cluster_stats_messages_ping_sent:5559
cluster_stats_messages_pong_sent:2080
cluster_stats_messages_sent:7639
cluster_stats_messages_ping_received:2078
cluster_stats_messages_pong_received:2129
cluster_stats_messages_meet_received:2
cluster_stats_messages_received:4209
127.0.0.1:7000> cluster nodes
030105952740cbb648322d12dc93e53a47a90687 127.0.0.1:7008@17008 slave 96d8dce113386caa1388f96df2ea5ff96f9d3dee 0 1576647435994 1 connected
84c9a3c36688cb6dfc4f1876ea24478e5c0a6552 127.0.0.1:20000@30000 slave 96d8dce113386caa1388f96df2ea5ff96f9d3dee 0 1576647436000 1 connected
2bd1531ea0937b5a00076baeb5ae0d0e96a9162e 127.0.0.1:7002@17002 master - 0 1576647434575 2 connected 5461-10922
dc9ab96a1cbf689c16344b9bd6765529b68a3d14 127.0.0.1:7001@17001 slave 2bd1531ea0937b5a00076baeb5ae0d0e96a9162e 0 1576647436000 4 connected
d7849b792208433e8f2e4d90999aa629fd6de652 127.0.0.1:7003@17003 master - 0 1576647437008 3 connected 10923-16383
99f77c114d2badd5542d6cb64c8011f4cc49462b 127.0.0.1:7006@17006 slave 96d8dce113386caa1388f96df2ea5ff96f9d3dee 0 1576647436000 6 connected
98b89c9bc230678764b7ed7ac1a9990d9f2b6d77 :0@0 slave,fail,noaddr 2bd1531ea0937b5a00076baeb5ae0d0e96a9162e 1576645274981 1576645274981 2 disconnected
3bd809889e8afea60271c18c47e717b316539add 127.0.0.1:7005@17005 slave d7849b792208433e8f2e4d90999aa629fd6de652 0 1576647438016 5 connected
96d8dce113386caa1388f96df2ea5ff96f9d3dee 127.0.0.1:7000@17000 myself,master - 0 1576647433000 1 connected 0-5460 [12789-<-d7849b792208433e8f2e4d90999aa629fd6de652] [14553-<-d7849b792208433e8f2e4d90999aa629fd6de652]
127.0.0.1:7000> 

Thanks ,

Sowjanya 

[Tuco]

I think your cluster is in a broken state, that's why it is showing 3 nodes as in importing state etc.

You may want to fix it by running.

./redis-trib.rb fix 127.0.0.1:7000

Make sure that you have this script and ruby.
This used to work atleast upto version 4 of redis
Not sure whether the cluster fix script was included in the redis-cli of redis-5.