Can't reshard my node. Slots not all assigned error. Cluster 'fail'.

[Eric Greer]

Somewhere when adding a slave using redis-trib.rb, I ended up putting my cluster in the 'fail' state.  I'm trying to rescue it.

127.0.0.1:6379> cluster info
cluster_state:fail
cluster_slots_assigned:0
cluster_slots_ok:0
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:3
cluster_size:0
cluster_current_epoch:2
cluster_stats_messages_sent:471058
cluster_stats_messages_received:471058

127.0.0.1:6379> cluster nodes
ea99c3d1bd69a592001111bbf64009441ce82376 192.168.1.89:6379 master - 0 1400555634672 0 connected
6964fa8f83c071bd8f3c1bf5d9adedd5784d79a6 192.168.1.110:6379 master - 0 1400555635674 1 connected
e1c9faa3b9c2875dca336e11b3cdb752f8b3da7a :0 myself,master - 0 0 2 connected

So I tried a fix...

./redis-trib.rb fix 127.0.0.1:6379

At the top of my output, I get this:

[root@redis8 ~]# ./redis-trib.rb fix 127.0.0.1:6379
Connecting to node 127.0.0.1:6379: OK
Connecting to node 192.168.1.89:6379: OK
Connecting to node 192.168.1.110:6379: OK
>>> Performing Cluster Check (using node 127.0.0.1:6379)
M: e1c9faa3b9c2875dca336e11b3cdb752f8b3da7a 127.0.0.1:6379
slots: (0 slots) master
0 additional replica(s)
M: ea99c3d1bd69a592001111bbf64009441ce82376 192.168.1.89:6379
slots: (0 slots) master
0 additional replica(s)
M: 6964fa8f83c071bd8f3c1bf5d9adedd5784d79a6 192.168.1.110:6379
slots: (0 slots) master
0 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[ERR] Not all 16384 slots are covered by nodes.
>>> Fixing slots coverage...
List of not covered slots: 0,1,2,3,4,5,6,7,8... (continues to 16383)

At the bottom of my output, I get this:

The following uncovered slots have no keys across the cluster:
./redis-trib.rb:410:in `fix_slots_coverage': undefined method `keys' for #<Array:0xb76d1610> (NoMethodError)
from ./redis-trib.rb:352:in `check_slots_coverage'
from ./redis-trib.rb:331:in `check_cluster'
from ./redis-trib.rb:749:in `fix_cluster_cmd'
from ./redis-trib.rb:1075:in `send'
from ./redis-trib.rb:1075

So that didn't work.  I figured I would try a reshard:

[root@redis8 ~]# ./redis-trib.rb reshard 127.0.0.1:6379
Connecting to node 127.0.0.1:6379: OK
Connecting to node 192.168.1.89:6379: OK
Connecting to node 192.168.1.110:6379: OK
>>> Performing Cluster Check (using node 127.0.0.1:6379)
M: e1c9faa3b9c2875dca336e11b3cdb752f8b3da7a 127.0.0.1:6379
slots: (0 slots) master
0 additional replica(s)
M: ea99c3d1bd69a592001111bbf64009441ce82376 192.168.1.89:6379
slots: (0 slots) master
0 additional replica(s)
M: 6964fa8f83c071bd8f3c1bf5d9adedd5784d79a6 192.168.1.110:6379
slots: (0 slots) master
0 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[ERR] Not all 16384 slots are covered by nodes.
*** Please fix your cluster problems before resharding

No luck.  I'm stuck!  My cluster is in a 'fail' state and I don't know what to do to fix it.  Help please?!?!

[Balaji Rao]

Did you find a resolution to this ? I'm seeing similar problems

[Salvatore Sanfilippo]

Please, try redis-trib "check" and "fix" commands to get more idea and
fix the issue. Thanks.

> --
> You received this message because you are subscribed to the Google Groups
> "Redis DB" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to redis-db+u...@googlegroups.com.
> To post to this group, send email to redi...@googlegroups.com.
> Visit this group at http://groups.google.com/group/redis-db.
> For more options, visit https://groups.google.com/d/optout.



--
Salvatore 'antirez' Sanfilippo
open source developer - GoPivotal
http://invece.org

"Fear makes the wolf bigger than he is."
— German proverb