redis cluster failure in swarm/overlay network

[Kesava Vunnava]

Hi All,

Able to bring up redis cluster with three masters (no slaves) using "redis-trib.rb create" command in normal environment. 

Then, want to achieve the same via docker/swarm .

Steps :

1] Initiated docker swarm on a node and joined other two nodes to swam as managers. (in total now 3 nodes and all are managers).

2] Created a Overlay network with encryption using the command :

    "docker network create redis-network --driver overlay --opt encrypted"

3] Deployed redis service using the command : 

"docker service create --publish=6381:6381 --publish=16381:16381 --mode=global --network=redis-network --detach=false --name=redis redis:3.2.8 --cluster-enabled yes --cluster-node-timeout 15000 --cluster-require-full-coverage no  --protected-mode no --port 6381 --appendonly yes --appendfsync everysec --logfile redis1.log --loglevel debug".

Can see that on three nodes redis service was started in cluster mode  & confirmed the same using "docker ps" command.

4] Got in to the container ("docker exec -it <container ID /bin/bash>") of one of the nodes and executed command "redis-trib.rb create <IP1>:6381 <IP2>:6381 <IP3>:6381"

some times command never returned & some times it returned with err some thing like "slot coverage not OK"

I cant get enough information from redis logs too. How to proceed on this further ?? Interestingly, "redis-cli cluster meet" command responds with "OK" from all nodes. Redis version being used was 3.2.8. 

##################

root@f1a5bdf47900:/data# redis-trib.rb create 10.77.123.171:6381 10.77.123.172:6381 10.77.123.176:6381

>>> Creating cluster

>>> Performing hash slots allocation on 3 nodes...

Using 3 masters:

10.77.123.171:6381

10.77.123.172:6381

10.77.123.176:6381

M: 9c50c88f5a92a14d687ec4f5ef84916e96dd25c8 10.77.123.171:6381

   slots:0-5460 (5461 slots) master

M: 48e9949e92277819b2dd3ba162769d127dc1e10d 10.77.123.172:6381

   slots:5461-10922 (5462 slots) master

M: 87f1b72b71260ceb0506ab681aa6cb8821c3960f 10.77.123.176:6381

   slots:10923-16383 (5461 slots) master

Can I set the above configuration? (type 'yes' to accept): yes

>>> Nodes configuration updated

>>> Assign a different config epoch to each node

>>> Sending CLUSTER MEET messages to join the cluster

Waiting for the cluster to join.......................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................^C/usr/local/bin/redis-trib.rb:653:in `sleep': Interrupt

        from /usr/local/bin/redis-trib.rb:653:in `wait_cluster_join'

        from /usr/local/bin/redis-trib.rb:1306:in `create_cluster_cmd'

        from /usr/local/bin/redis-trib.rb:1701:in `<main>'

#########################

[hva...@gmail.com]

Have you seen the Cluster Tutorial page on redis.io, especially the section titled "Redis Cluster and Docker"?    https://redis.io/topics/cluster-tutorial#redis-cluster-and-docker

[Kesava Vunnava]

Yes. It is mentioned that in order to make docker compatible with redis cluster need to use the host networking mode of docker. That's what I did and able to build cluster inside docker which I mentioned in my previous mail (first line itself) . 

But, later challenge was to deploy redis as a service inside swarm and use the overlay network to encrypt communication happening between cluster nodes . This is where I was facing issues. 

As per docker documentation (https://success.docker.com/Architecture/Docker_Reference_Architecture%3A_Designing_Scalable%2C_Portable_Docker_Container_Networks ) ., it was mentioned that 

"Docker supports IPSec encryption for overlay networks out-of-the-box. The Swarm & UCP managed IPSec tunnels encrypt network traffic as it leaves the source container and decrypts it as it enters the destination container. This ensures that your application traffic is highly secure when it's in transit regardless of the underlying networks." . 

Was trying to test this for redis application.

-Thanks,

Kesav.

[sathish kumar]

Kesava,

Redis Cluster deployments on Docker, the new NAT/port-forwarding features (Redis 4.0) just mean that it is now possible to change from host-mode networking to a bridge or overlay network.

Refer: https://get-reddie.com/blog/redis4-cluster-docker-compose/