Redis cluster and docker don't play well

[Brian Picciano]

This is kind of similar to this thread: https://groups.google.com/forum/#!searchin/redis-db/docker/redis-db/midrbfgxBrQ/o8Egs60y-cgJ except that it applies to redis cluster, and doesn't have a nice solution as far as I can see.

Essentially, I have a set of docker containers running on a host, each running a single redis cluster instance. The host's ip is 172.17.8.101 and it is reachable from other hosts. The containers, however, see only an ip subnet in the 10.0.0.0 range. They can communicate with each other over this range, but outside boxes cannot.

When I start up a cluster I do a:

redis-trib.rb create 172.17.8.101:6479 172.17.8.101:6480 172.17.8.101:6481

Since I've mapped the ports in the containers to their corresponding ports on the host this works, and I get the following output at the end:

>>> Performing Cluster Check (using node 172.17.8.101:6479)

M: e14cb8715db04e2d0d0c9084d4512532e0425276 172.17.8.101:6479

   slots:0-5460 (5461 slots) master

M: 300a7f463c398f8c614fe739e26a75c8166a1977 172.17.8.101:6480

   slots:5461-10922 (5462 slots) master

M: 931e7e6242b79baf2ab95baed497c6bf18f4e050 172.17.8.101:6481

   slots:10923-16383 (5461 slots) master

[OK] All nodes agree about slots configuration.

But when I do a "cluster nodes" on any node in the cluster I get the following:

e14cb8715db04e2d0d0c9084d4512532e0425276 10.1.0.246:6479 myself,master - 0 0 1 connected 0-5460

931e7e6242b79baf2ab95baed497c6bf18f4e050 10.1.42.1:6481 master - 0 1422752155029 3 connected 10923-16383

300a7f463c398f8c614fe739e26a75c8166a1977 10.1.42.1:6480 master - 0 1422752156033 2 connected 5461-10922

So the cluster nodes are advertising the ips they think they have within the docker containers, but these ips aren't actually reachable from outside the docker host, so when outside clients call "cluster slots" they get a bunch of unusable ips. I tried setting the bind parameter to 172.17.8.101, but the instances would then fail to start up since within the container that ip isn't actually assigned, only the 10.0.0.0 one.

I think the ideal solution would be to somehow allow for users to specify the advertised ip:port of a cluster instance. That's probably a rather big change to make this late in the rc game, so I'm definitely open to other solutions (or just putting the issue off all-together). Thanks for reading this far, and I'm happy to answer any questions about my setup that might also be contributing to the problem.

[Joshua Rubin]

I just wanted to post that I am in the exact same situation as Brian. The generated nodes.conf files differ subtly though, in that he gets the docker subnet range for all nodes listed, whereas I get the external ip and port for the non-myself nodes. For the myself node, some nodes get no ip address, just the port (e.g. :7000) and some (about 2/3) get the docker 10.0.0.0 IP address in the myself node.

Manually editing the nodes.conf file and restarting redis seems to work.

I posted a lot of detail in this GitHub issue: https://github.com/fzzy/radix/issues/52 but I have retained the most pertinent info here as well.

However I have some questions that may be better answered here:

1. Is manually editing nodes.conf as I described is a reliable fix?
2. Do we have to restart redis after making the change?
3. Is there any redis cluster command to make the change to the advertised IP (once per node) so that a restart is not required?
4. Is this an issue with redis-trib.rb or the redis server?
5. What is the consequence of listing only a port and not an IP address for the myself line in nodes.conf?

Thanks for the help!

[Matt Stancliff]

> On Jan 31, 2015, at 20:04, Brian Picciano <mediocr...@gmail.com> wrote:
>
> This is kind of similar to this thread: https://groups.google.com/forum/#!searchin/redis-db/docker/redis-db/midrbfgxBrQ/o8Egs60y-cgJ except that it applies to redis cluster, and doesn't have a nice solution as far as I can see.
>
> Essentially, I have a set of docker containers running on a host, each running a single redis cluster instance. The host's ip is 172.17.8.101 and it is reachable from other hosts. The containers, however, see only an ip subnet in the 10.0.0.0 range. They can communicate with each other over this range, but outside boxes cannot.
>

> e14cb8715db04e2d0d0c9084d4512532e0425276 10.1.0.246:6479 myself,master - 0 0 1 connected 0-5460
> 931e7e6242b79baf2ab95baed497c6bf18f4e050 10.1.42.1:6481 master - 0 1422752155029 3 connected 10923-16383
> 300a7f463c398f8c614fe739e26a75c8166a1977 10.1.42.1:6480 master - 0 1422752156033 2 connected 5461-10922
>
> So the cluster nodes are advertising the ips they think they have within the docker containers, but these ips aren't actually reachable from outside the docker host, so when outside clients call "cluster slots" they get a bunch of unusable ips. I tried setting the bind parameter to 172.17.8.101, but the instances would then fail to start up since within the container that ip isn't actually assigned, only the 10.0.0.0 one.

Correct. Redis components aren’t really designed to work behind layers of NAT since each node self-discovers IP addresses. Running every individual application in its own limited, non-routable subnet is really dumb anyway, but that seems to be the way the world is moving.

You can try to trick Redis to use different IP addresses though. Redis uses the first ‘bind’ IP address as the address of the entire instance. So, make sure Redis isn’t listening on ‘*’ — force the first bind IP to be the IP address of the _external_ reachable IP address then use your local IPs for the rest of the bind addresses. You may need to play with kernel parameters like “allow non local bind” to let Redis listen to an IP address your local kernel network namespace can’t see. You’ll also have to match your local listening port to the external forwarding port number so the cluster gossip fields get populated correctly there too.


-Matt

[Brian Picciano]

Thanks for the reply Matt! At this moment it is possible to map the ports correctly (although with some hassle), but annoyingly sysctl settings cannot be changed inside docker unless it is run with --privileged, which I'd really rather avoid when trying to run a datastore.

As I mentioned in the OP I totally understand that this probably won't make it into 3.0.0, but could a discussion be started about having a "cluster-advertise-as <host:port>" parameter in redis.conf at some point in the future? It would vastly simplify both this and I'm sure other people's strange use-cases for cluster involving NATs.

On Wednesday, February 18, 2015 at 4:03:40 PM UTC-5, Matt Stancliff w

[Tomas Kramar]

Something analogous to sentinel's announce-ip and announce-port would be really helpful here. It's basically the same problem.

[Tomas Kramar]

I've created an issue to address this here: https://github.com/antirez/redis/issues/2527

On Sunday, February 1, 2015 at 2:04:02 AM UTC+1, Brian Picciano wrote:

[Tomas Kramar]

I tried the recommended workaround with non-local bind address but that doesn't seem to work either. Did any of you manage to get redis cluster and docker working together?

For the record, I explicitly specified bind addresses, starting with the host ip first and followed by the docker internal ip. I allowed non-local bind in sysctl and added NET_ADMIN capability to the docker container. I can connect to any of the containerized cluster nodes using the host ip, but when I try to meet two nodes, redis fails to establish connection to the other node and fails with "Unable to connect to Cluster Node [x.x.x.x]:18001 -> creating socket: Invalid argument".

On Sunday, February 1, 2015 at 2:04:02 AM UTC+1, Brian Picciano wrote:

[Brian Picciano]

I was never able to get a working cluster

--
You received this message because you are subscribed to a topic in the Google Groups "Redis DB" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/redis-db/ZLOKATn6vZc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to redis-db+u...@googlegroups.com.
To post to this group, send email to redi...@googlegroups.com.
Visit this group at http://groups.google.com/group/redis-db.
For more options, visit https://groups.google.com/d/optout.

[David Quarles]

Hey Brian!

I'm running into this exact same issue.  I'm about to throw in the towel and use host networking mode, but I'm curious if anyone was ever able to get this working or if there is some sort of workaround in 3.0?

Thanks,

David

[Brian Picciano]

David! Fancy seeing you here :P I never did get this working, I think I kinda just gave up. Perhaps someone else in the thread found a better interim solution? Also according to https://github.com/antirez/redis/issues/2527 the announce-ip/port configuration options will be going into the next RC in "a couple weeks", and backported into 3.0.x, so if you can wait for that that might be the best option.