docker consul + ui on OSX

Ernest Mueller

unread,

Dec 19, 2016, 3:03:35 PM12/19/16

to Consul

Hey all, new to consul. I'm just trying to set up a simple server cluster on my Mac and then connect an agent to it with the UI running. I have it mostly working but it's having some issues and it's clear I don't understand everything going on at the networking level. Thoughts from the more experienced would be welcome.

I'm using docker for mac (no docker-machine) and the official consul docker image from dockerhub.

I start up a 3-node server cluster like:

docker run -d --name node1 -h node1 consul agent -server -bootstrap-expect 3
JOIN_IP="$(docker inspect -f '{{.NetworkSettings.IPAddress}}' node1)"
docker run -d --name node2 -h node2 consul agent -server -join $JOIN_IP
docker run -d --name node3 -h node3 consul agent -server -join $JOIN_IP

And this works fine. docker exec consul members shows them all in there, docker logs shows a leader's elected and everyone's in the cluster.

So now I want to run a client agent, both to mimic my mac joining up as a client and also to look at the UI. What is getting me the closest to working is:

docker run -d -p 8500:8500 --name node4 -h node4 consul agent -join $JOIN_IP  -ui -client=0.0.0.0 -bind=127.0.0.1

This starts up, and I can hit the Web UI from localhost:8500/ui. But node4's logs are full of

2016/12/19 18:47:06 [ERR] memberlist: Failed to send ping: write udp 127.0.0.1:8301->172.17.0.3:8301: sendto: invalid argument

consul members:

Node Address Status Type Build Protocol DC

node1 172.17.0.2:8301 alive server 0.7.1 2 dc1

node2 172.17.0.3:8301 alive server 0.7.1 2 dc1

node3 172.17.0.4:8301 alive server 0.7.1 2 dc1

node4 127.0.0.1:8301 alive client 0.7.1 2 dc1

In the UI, nodes 1-3 look fine but node4 is in error state, the Serf Health Status says "Agent not live or unreachable".

"moby" (the docker for mac host VM) also shows up in the node list now, no services, also in error state.

Just for kicks I ran registrator; it added in a service entry for consul:8500 on node4 and when I ran a redis container it added it in to node4 (also in failing state).

--net=host doesn't work as you might expect on docker for mac, it only exposes into the VM host not the actual Mac itself. I tried adding -p <everything else>m no real difference except that registrator shows a bunch more ports worth of service failing health check.

In the UI node4 seems to go green then orange and back again. In e.g. node1's docker log it is thrashing on it:

2016/12/19 20:00:43 [ERR] memberlist: Failed TCP fallback ping: EOF

2016/12/19 20:00:44 [WARN] memberlist: Got ping for unexpected node 'node4' from=127.0.0.1:8301

2016/12/19 20:00:46 [INFO] serf: EventMemberFailed: node4 127.0.0.1

2016/12/19 20:00:46 [INFO] consul: member 'node4' failed, marking health critical

2016/12/19 20:00:46 [INFO] memberlist: Suspect node4 has failed, no acks received

2016/12/19 20:01:09 [INFO] serf: attempting reconnect to moby 127.0.0.1:8301

2016/12/19 20:01:09 [INFO] serf: EventMemberJoin: node4 127.0.0.1

2016/12/19 20:01:09 [INFO] consul: member 'node4' joined, marking health alive

2016/12/19 20:01:09 [WARN] memberlist: Got ping for unexpected node 'node4' from=127.0.0.1:8301

2016/12/19 20:01:10 [WARN] memberlist: Got ping for unexpected node 'node4' from=127.0.0.1:8301

2016/12/19 20:01:11 [WARN] memberlist: Got ping for unexpected node node4 from=127.0.0.1:58984

2016/12/19 20:01:11 [ERR] memberlist: Failed TCP fallback ping: EOF

2016/12/19 20:01:11 [INFO] memberlist: Suspect node4 has failed, no acks received

2016/12/19 20:01:13 [WARN] memberlist: Got ping for unexpected node 'node4' from=127.0.0.1:8301

2016/12/19 20:01:15 [INFO] serf: EventMemberFailed: node4 127.0.0.1

2016/12/19 20:01:15 [INFO] consul: member 'node4' failed, marking health critical

2016/12/19 20:01:39 [INFO] serf: attempting reconnect to node4 127.0.0.1:8301

2016/12/19 20:01:39 [INFO] serf: EventMemberJoin: node4 127.0.0.1

2016/12/19 20:01:39 [INFO] consul: member 'node4' joined, marking health alive

2016/12/19 20:01:40 [WARN] memberlist: Got ping for unexpected node 'node4' from=127.0.0.1:8301

2016/12/19 20:01:41 [WARN] memberlist: Got ping for unexpected node node4 from=127.0.0.1:59000

2016/12/19 20:01:41 [ERR] memberlist: Failed TCP fallback ping: EOF

2016/12/19 20:01:41 [INFO] memberlist: Suspect node4 has failed, no acks received

2016/12/19 20:01:41 [WARN] memberlist: Got ping for unexpected node 'node4' from=127.0.0.1:8301

2016/12/19 20:01:44 [WARN] memberlist: Got ping for unexpected node 'node4' from=127.0.0.1:8301

2016/12/19 20:01:45 [WARN] memberlist: Got ping for unexpected node 'node4' from=127.0.0.1:8301

2016/12/19 20:01:46 [WARN] memberlist: Got ping for unexpected node node4 from=127.0.0.1:59010

2016/12/19 20:01:46 [ERR] memberlist: Failed TCP fallback ping: EOF

2016/12/19 20:01:46 [INFO] memberlist: Marking node4 as failed, suspect timeout reached (0 peer confirmations)

2016/12/19 20:01:46 [INFO] memberlist: Suspect node4 has failed, no acks received

2016/12/19 20:01:46 [INFO] serf: EventMemberFailed: node4 127.0.0.1

2016/12/19 20:01:46 [INFO] consul: member 'node4' failed, marking health critical

Ideas? My end goal is to prototype out a consul/registrator/consul-template cluster locally for evaluation and, ideally, larger scale prod deployment on AWS/ECS, primarily for config store but eventually for service discovery too. But I need to get it running locally enough to figure it out...

Thanks,

Ernest

James Phillips

unread,

Dec 19, 2016, 11:34:33 PM12/19/16

to consu...@googlegroups.com

Hi Ernest,

On the Mac, the -p argument should be enough to get the ports mapped
to localhost. Something like this should work (dropped the -bond
argument):

docker run -d -p 8500:8500 --name node4 -h node4 consul agent -join
$JOIN_IP -ui -client=0.0.0.0

This'll still expose that containers TCP port 8500 on localhost:8500.

-- James

> --
> This mailing list is governed under the HashiCorp Community Guidelines -
> https://www.hashicorp.com/community-guidelines.html. Behavior in violation
> of those guidelines may result in your removal from this mailing list.
>
> GitHub Issues: https://github.com/hashicorp/consul/issues
> IRC: #consul on Freenode
> ---
> You received this message because you are subscribed to the Google Groups
> "Consul" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to consul-tool...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/consul-tool/be17ea58-c754-46af-a65d-901ef9b08241%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

Ernest Mueller

unread,

Dec 20, 2016, 9:57:06 AM12/20/16

to Consul

OK thanks! I guess I figured that if the node4 consul wasn't bound more to the actual node I'd lose something, and while it doesn't report the node's IP it runs, I can see the UI, and registrator can connect to it and register other containers. I was trying to follow a mix of the official consul docker docs that talk about splitting with client/bind (https://hub.docker.com/_/consul/), which say to use -bind=<external_ip> in client mode and then bridge over with -client, and Jeff's older progrium/consul docs which make more sense (at least to me) but are slightly different for both the newer container and Docker for Mac (--net=host doesn't punch ports all the way through). Thanks much!

Just to enhance my understanding, when/how/why would I use -bind? And using it, the health checks seemed to alternately fail/pass, while I would have expected either constant pass or constant fail... There's something about the networking here I'm clearly not getting.

Thanks,

Ernest

Ernest Mueller

unread,

Dec 22, 2016, 4:41:00 PM12/22/16

to Consul

Oh by the way I got a tip from "Sean" on SO about some new 0.7.2 functionality that makes this work fully the way I want:

Try using the 0.7.2 release of Consul and start the agent using the following (beta as of 0.7.2, final by 0.8.0) syntax:
$ docker run -d -p 8500:8500 --name node4 -h node4 consul agent -join $JOIN_IP -ui -client=0.0.0.0 -bind='{{ GetPrivateIP }}'
The change being the argument to -bind where Consul will now render out the IP address of a private IP address. The other template parameters are documented in the hashicorp/go-sockaddr.