Docker swarm mode mesh routing is not working at all

883 views
Skip to first unread message

Tim Ecklund

unread,
Sep 5, 2016, 2:36:33 PM9/5/16
to docker-dev


I have docker swarm mode set up with 3 ubuntu 16.04 machines with vagrant. I don't think mesh routing is working at all.

If I set up a service like

docker service create --name helloworld --replicas 1 -p 8888:80 nginx

I can see my service with
docker service ls
ID NAME REPLICAS IMAGE COMMAND
evbp2spkjn50 helloworld 1/1 nginx

I can curl to the ip of the machine that the actual container is running on:
curl 172.28.100.101:8888
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
....

But if I go to any other machine in the swarm, I am not routed properly:
curl 172.28.100.102:8888
curl: (7) Failed to connect to 172.28.100.102 port 8888: Connection refused

Now, if I scale the service such that a container is running on all of the machines like this:
docker service scale helloworld=3 helloworld scaled to 3

All of a sudden I can curl to it.
curl 172.28.100.102:8888
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
....

All ports are open between these machines (they are set up with vm.network :private_network, :ip => "172.28.100.10X", :netmask => "255.255.0.0" in vagrant).
I have tried this with a dedicated network with no change. I think mesh routing is just not working at all.

docker --version
Docker version 1.12.1, build 23cf638


Tim Ecklund

unread,
Sep 5, 2016, 4:16:41 PM9/5/16
to docker-dev
On the advice of James Mills I switched from Ubuntu to RancherOS and the bad behavior disappears. There must be something about stock Ubuntu 16.04 that isn't working. 

Nicholas Maietta

unread,
Oct 26, 2016, 9:50:56 PM10/26/16
to docker-dev
Unfortunately I cannot do this. Mesh routing worked in 1.12 "experimental" but not 1.12.1 and stock Ubuntu 16.04.

To pour salt in the wound, after being forced to go the way of "docker machine", I learned that the flag for switching docker engines during provisioning of machine is not working. I cannot switch out versions as it is documented in Docker's official Docker Machine pages.

Andrea Luzzardi

unread,
Oct 27, 2016, 10:54:39 AM10/27/16
to Nicholas Maietta, docker-dev
Have you tried 1.12.2?

It contains *lots* of mesh networking fixes, specifically to work on different kernel/environments.
--
You received this message because you are subscribed to the Google Groups "docker-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to docker-dev+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Madhu Venugopal

unread,
Oct 27, 2016, 11:07:22 AM10/27/16
to Andrea Luzzardi, Santhosh Ram, Nicholas Maietta, docker-dev
Yes. 1.12.2 fixed a few routing-mesh issues. Also Santhosh (copied here) was looking at 16.04 specific issues as well. He may have some insights.

-Madhu

To unsubscribe from this group and stop receiving emails from it, send an email to docker-dev+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "docker-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to docker-dev+unsubscribe@googlegroups.com.

Nicholas Maietta

unread,
Oct 27, 2016, 4:48:20 PM10/27/16
to Madhu Venugopal, Andrea Luzzardi, Santhosh Ram, docker-dev

I have confirmed that the Routing Mesh in Swarm Mode Ubuntu 16.04 and Docker Engine 1.12.3 works perfectly.


Unfortunately, I couldn't easily use docker-machine to upgrade the machines. (I have physical machines spread out across the city).


For each machine I couldn't upgrade with "docker machine upgrade <nodeX>", i ran the following:


$  docker-machine ssh <nodeX>

$  curl -sSL https://get.docker.com/ | docker_version=1.12.3 sh


Running "docker-machine ls" revealed that the machine does reflect it's definitely running 1.12.3 :) (finally)


The other issue I had was that Ubuntu refused to upgrade with "sudo apt-get upgrade", but was able to after running "sudo do-release-upgrade".


I hope this helps others!!!


After months of hair pulling, I now have Swarm Mode + Routing Mesh. Now to figure out converting my existing docker-compose stuff to "service" ready methods. The docker bundle thing really doesn't cut it yet, because i do health checking against Apache weberserver by adding a hostname "healthcheck" to the /etc/hosts in my container so that I can configure apache httpd.conf to not write to logs for those requests. To add a host, i must do so at the creation of a running container like so:


docker service create --name webtest1 --publish 80:80 --add-host='healthcheck:127.0.0.1' commnetivity/webserver


Except that "docker service" does not currently support the "--add-host" option like I need. I don't know of another way to write to hosts file in my image yet.


But at least I can say a milestone has been reached today!!! I have Docker "native swarm" and routing mesh working. FINALLY.

From: Madhu Venugopal <ma...@docker.com>
Sent: Thursday, October 27, 2016 8:07:20 AM
To: Andrea Luzzardi; Santhosh Ram
Cc: Nicholas Maietta; docker-dev
Subject: Re: [docker-dev] Re: Docker swarm mode mesh routing is not working at all
 

Andrea Luzzardi

unread,
Oct 28, 2016, 1:37:35 AM10/28/16
to Arnaud Porterie, Daniel Nephin, Madhu Venugopal, Mike Goelzer, Nicholas Maietta, Santhosh Ram, docker-dev
+Daniel +Arnaud +Mike

Is --add-host something we can squeeze in for 1.13?

To unsubscribe from this group and stop receiving emails from it, send an email to docker-dev+...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "docker-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to docker-dev+...@googlegroups.com.
Reply all
Reply to author
Forward
0 new messages