Kong Cluster using Docker containers in different hosts

[Felipe Esteves]

Hi,

I'm facing a situation in Kong that seems to be a tough one. I use Kong in a docker container, and have the need to use the docker random port mapping to mount a cluster in diferent hosts.

If I have just 1 Kong instance, it works well, no problems. But when I try to build a cluster, the nodes can't communicate with each other because the exposed cluster listen port is a random one. And, as it is different hosts, the nodes also can't reach each other using the internal Docker IPs (like 172.17.0.12:7946)

######

## Address and port used by the node to communicate with other Kong nodes in the cluster with both UDP and

## TCP messages. All the nodes in the cluster must be able to communicate with this node on this address.

## Only IPv4 addresses are allowed (no hostnames).

# cluster_listen: "0.0.0.0:7946"

Does anyone has any experience in a multi-hosts Kong cluster running in docker containers? 

Felipe Esteves

Tecnologia

felipe....@b2wdigital.com

Tel.: (21) 3504-7162 ramal 57162

Skype: felipe2esteves

[Felipe Esteves]

Hi,

I've managed to successfully configure the Kong node, but now have a different problem:

As the Dockerfile exposes by default the 7946 TCP and 7946 UDP. it results in two different mapped ports.

What's the real need for UDP in Kong?

Felipe Esteves

Tecnologia

felipe....@b2wdigital.com

Tel.: (21) 3504-7162 ramal 57162

Skype: felipe2esteves

[Marco]

Both TCP and UDP traffic on 7946 is required for intra-cluster communication. 

 Marco Palladino | CTO @ Mashape | mashape.com | +1 (415) 361-3858

--
You received this message because you are subscribed to the Google Groups "Kong" group.
To unsubscribe from this group and stop receiving emails from it, send an email to konglayer+...@googlegroups.com.
To post to this group, send email to kong...@googlegroups.com.
Visit this group at https://groups.google.com/group/konglayer.
To view this discussion on the web visit https://groups.google.com/d/msgid/konglayer/CAE0%2BNNXuY0BSXa%3DKErfWXPC0UrJpZBQvdpamUq4u_BjmrBqXyA%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

[Felipe Esteves]

Hi Marco,

We've managed to configure the cluster with just TCP and it seems normally.

The output of "kong cluster reachabilty" is showing no acks because of the UDP lack connectivity, but the members are listed well and we've used apis and plugins in all nodes ok.

I'm trying to find what errors the UDP connectivity problem can cause, but as far seems there's no one.

Felipe Esteves

Tecnologia

felipe....@b2wdigital.com

Tel.: (21) 3504-7162 ramal 57162

Skype: felipe2esteves

[Marco]

TCP is being used only when connecting the first time to the other nodes, while UDP is being used to actually propagate the events. So right now the nodes cannot talk with each other, and if you wait a minute or so and check their status with "kong cluster members" you will see that it will be "failed".

Having UDP available is a requirement. If you make any change to APIs, Consumers or Plugins or Plugin's entites on one node, it won't propagate to the other nodes, creating inconsistencies. I would recommend reading this https://getkong.org/docs/0.7.x/clustering/#why-do-we-need-kong-clustering

 Marco Palladino | CTO @ Mashape | mashape.com | +1 (415) 361-3858

[Felipe Esteves]

Hi Marco,

Right now, I have a cluster created 4 hours ago. The three nodes are alive in Cassandra and in "/cluster" in each node.

I've made a test of updating an api, changing the upstream URL.

At first it worked only on the local node, but after less than a minute it propagated to other nodes correctly.

Felipe Esteves

Tecnologia

felipe....@b2wdigital.com

Tel.: (21) 3504-7162 ramal 57162

Skype: felipe2esteves

[vladimi...@oneflow.io]

I am facing the same issue, whereas I opened the ports on the host and mapped them to the containers. 
Could you share the cluster config settings?

[al...@qburst.com]

Could you guyz share the configuration used for this?

[nicolas.l...@gmail.com]

Hi Felipe,

Can you publish your Kong configuration file ? I tried to create a Kong cluster with Docker. I can connect mys Kong nodes on the Cassandra database but I didn't see my nodes with the command "kong cluster members".
If you can push your conf, it will be nice.

Nicolas.

[Marco Palladino]

Hello,

If you are using Docker Compose: https://github.com/Mashape/docker-kong/tree/docker-compose/autopilot

 Marco Palladino | CTO @ Mashape | mashape.com | +1 (415) 361-3858

[nicolas.l...@gmail.com]

Hi Marco,

I don't use docker-compose. I tried to create my cluster with the different docker command for docker like :

docker run -d --name kong-database \
              -p 9042:9042 \
              cassandra:2.2

docker run -d --name kong \
              --link kong-database:kong-database \
              -e "DATABASE=cassandra" \
              -p 8000:8000 \
              -p 8443:8443 \
              -p 8001:8001 \
              -p 7946:7946 \
              -p 7946:7946/udp \
              --security-opt seccomp:unconfined \
              mashape/kong

I just change use a specific kong.yml file to define cassandra database connection. 
My docker nodes communicated with the database, but I didn't view all my nodes with "kong cluster members"
I think the kong nodes registred with docker container IP. They couldn't communicate with thoses. 

Thanks,
Nicolas.

[Marco]

Back in the days I wrote a Dockerfile that automatically retrieves the right IP address to be advertised to the other nodes, at https://github.com/Mashape/docker-kong/blob/cluster/Dockerfile

It's been a while since the last time I played with it, but I remember it working: https://github.com/Mashape/docker-kong/tree/cluster

Maybe it's a good start.

 Marco Palladino | CTO @ Mashape | mashape.com | +1 (415) 361-3858

To view this discussion on the web visit https://groups.google.com/d/msgid/konglayer/f64d002b-90b5-49e5-a279-d88b3550f0f7%40googlegroups.com.

[nicolas.l...@gmail.com]

Hi Marco,

I tested the autopilot docker-compose solution but kong container didn't start. It staid at "Restarting" status whan I used "docker ps".
Then, how can I connect another docker kong container on an another host ?

Thanks for your help.

Nicolas.

Le mardi 12 juillet 2016 17:41:01 UTC+2, Marco a écrit :

[Marco]

Does this other approach work? https://github.com/Mashape/docker-kong/tree/docker-compose/compose

 Marco Palladino | CTO @ Mashape | mashape.com | +1 (415) 361-3858

To view this discussion on the web visit https://groups.google.com/d/msgid/konglayer/2e1a76d2-5be2-4cc0-a6db-c3057a0a0c6a%40googlegroups.com.