Can not get redis 3.0.6 cluster to "meet"

[Chris Lewis]

Hello,

I have 3x Debian 8 machines running redis 3.0.6 (from the dotdeb repo) on which I would like to make a 6 node cluster, 3 masters and 3 slaves, where the slaves are on different physical machines to the masters.

I followed the cluster-tutorial on the web site, and can set up a 3 node or 6 node cluster on a single host no problem.

However when I try to run redis-trib.rb involving remote machines, things start not to work. This is what happens when I try to create a 3 master cluster using 3 separate machines (10.10.150.90 & 91 & 92).

#./redis-trib.rb create 10.10.150.90:7000 10.10.150.91:7000 10.10.150.92:7000
>>> Creating cluster
>>> Performing hash slots allocation on 3 nodes...
Using 3 masters:
10.10.150.90:7000
10.10.150.91:7000
10.10.150.92:7000
M: fcfbcfb925d67a7c96b8f065faa0ec1f73a408ee 10.10.150.90:7000
   slots:0-5460 (5461 slots) master
M: 07a286e7d0d1ac18158dd217e37909049d4dc254 10.10.150.91:7000
   slots:5461-10922 (5462 slots) master
M: 0548b81601bf92e13b76a2855a5633eaa4fcbdcb 10.10.150.92:7000
   slots:10923-16383 (5461 slots) master
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join........................................


The nodes in question are not firewalled i.e. all nodes can reach all others on ports 7000 and 17000.

Here is some more info from redis-cli after the above was run.

10.10.150.90:7000> cluster info
cluster_state:fail
cluster_slots_assigned:5461
cluster_slots_ok:5461
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:1
cluster_size:1
cluster_current_epoch:1
cluster_my_epoch:1
cluster_stats_messages_sent:0
cluster_stats_messages_received:0
10.10.150.90:7000> cluster nodes
fcfbcfb925d67a7c96b8f065faa0ec1f73a408ee :7000 myself,master - 0 0 1 connected 0-5460
10.10.150.90:7000> quit

redis@redis1:/mnt/redis/7000$ redis-cli  -h 10.10.150.91 -p 7000
10.10.150.91:7000> cluster nodes
07a286e7d0d1ac18158dd217e37909049d4dc254 :7000 myself,master - 0 0 2 connected 5461-10922
10.10.150.91:7000> quit

redis@redis1:/mnt/redis/7000$ redis-cli  -h 10.10.150.92 -p 7000
10.10.150.92:7000> cluster nodes
0548b81601bf92e13b76a2855a5633eaa4fcbdcb :7000 myself,master - 0 0 3 connected 10923-16383
10.10.150.92:7000>

I can see the cluster addslots has worked as the slots are correclty distibuted. However the cluster nodes are unaware of each other.

Manually trying to run cluster meet:

10.10.150.90:7000> CLUSTER MEET 10.10.150.91 7000
OK

A handshake state is acknowledged:
10.10.150.90:7000> CLUSTER nodes
d08f8044e762b8e4f3b05c32f768436650f5921d 10.10.150.91:7000 handshake - 0 0 0 connected
fcfbcfb925d67a7c96b8f065faa0ec1f73a408ee :7000 myself,master - 0 0 1 connected 0-5460
But the other node soon disappears from the list (after 5s):
10.10.150.90:7000> CLUSTER nodes
fcfbcfb925d67a7c96b8f065faa0ec1f73a408ee :7000 myself,master - 0 0 1 connected 0-5460
10.10.150.90:7000>

Can anyone advise if I am doing something wrong?

Thanks in advance

Chris

[Chris Lewis]

Hello

I have discovered the problem is caused by the network interface MTU being set to 9000.  After setting the interfaces to 1500, things worked as expected

AFAIK this is not an unusual setting to support network jumbo frames. Therefore this might well be a bug?

Cheers

Chris