Running Consul on the Docker host

[Chris Stevens]

I have been trying to move some consul-enabled applications into containers (docker) that will eventually run on Nomad.

I want to run a Consul agent on the docker host (not in a container) and access it from within the containers since this is how I plan to run it with Nomad, but there is so very little information out there about how to make this work.

My specific setup is:

* a VM running consul, dnsmasq, and docker (and later, with nomad),

* consul APIs and dns requests to *.service.consul work as expected on the VM,

* apps running directly on the VM access currently the consul agent via 127.0.0.1:8500,

* a simple containerized app started via docker-compose (for now).

Does anybody have recommended configurations or examples or suggested reading for a setup like this?

I've tried setting "dns" in docker / compose.

I've tried creating a "local" network that uses the new docker embedded dns server per this article:

https://medium.com/@leshik/a-little-trick-with-docker-to-simplify-access-to-consul-from-inside-your-containers-12686df15d58#.s0pi58loj

I've tried using 0.0.0.0 for consul client_addr and bind_addr.

I have seen recommendations on this forum to avoid most of the more elaborate docker networking modes and use net=host (or network_mode=host). This seems like good advice and will try that next.

Thanks!

- Chris

[Michael Fischer]

Our trick is to configure a dummy interface (dummy0) on the host with the IP address 169.254.1.1 (this is a link-local, non-routable IP).  We then bind Consul and dnsmasq to this address (as well as the loopback address), but we don't change Consul's advertising address (it remains that of the routable interface).  

Containers can then access both dnsmasq and Consul via 169.254.1.1.  

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/consul/issues
IRC: #consul on Freenode
---
You received this message because you are subscribed to the Google Groups "Consul" group.
To unsubscribe from this group and stop receiving emails from it, send an email to consul-tool+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/consul-tool/6410a28a-d0cf-4fe0-a11d-92ceb9788e63%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Chris Stevens]

That sounds fantastic. I'll work on that today.

BTW, I think you get the hat-tip for the net=host recommendation that I mentioned. Are you still running that way and recommend it?

[Michael Fischer]

That's only if you're running Consul in a container, which we do not -- I don't really see the point, as it's a single static binary.

On Thu, Aug 4, 2016 at 8:27 AM, Chris Stevens <chris....@traxo.com> wrote:

That sounds fantastic. I'll work on that today.

BTW, I think you get the hat-tip for the net=host recommendation that I mentioned. Are you still running that way and recommend it?

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/consul/issues
IRC: #consul on Freenode
---
You received this message because you are subscribed to the Google Groups "Consul" group.
To unsubscribe from this group and stop receiving emails from it, send an email to consul-tool+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/consul-tool/b5c7eee8-699f-4b42-83c4-ca555afcaa5d%40googlegroups.com.

[Chris Stevens]

I thought net=host was for all the containers and not just for the consul agent since it talked about the performance boost vs bridged networking. I'll have to go re-read that post.

For the consul agent though, I totally agree.

On Thursday, August 4, 2016 at 10:30:37 AM UTC-5, Michael Fischer wrote:

That's only if you're running Consul in a container, which we do not -- I don't really see the point, as it's a single static binary.

On Thu, Aug 4, 2016 at 8:27 AM, Chris Stevens <chris....@traxo.com> wrote:

That sounds fantastic. I'll work on that today.

BTW, I think you get the hat-tip for the net=host recommendation that I mentioned. Are you still running that way and recommend it?

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/consul/issues
IRC: #consul on Freenode
---
You received this message because you are subscribed to the Google Groups "Consul" group.

To unsubscribe from this group and stop receiving emails from it, send an email to consul-tool...@googlegroups.com.

[Adam Greene]

Hi Chris,

I ended up moving consul to run outside of docker for two reasons:

* if docker went down, it took the consul-server down, which caused flutter to propegate beyond the host

* docker logging kept causing memory errors (it was unbounded until 1.11 I believe)

* I wanted to use consul exec

so what I ended up doing was:

* still use consul docker but only as a packaging mechanism; my start script just copies consul and all its configs to /opt/consul and I run from there

* I bind consul to the host's IP address; in CoreOS this is `-bind ${COREOS_PRIVATE_IPV4}`

* I run dnsmasq (within docker) and my final cmd ends up like this: `/usr/bin/docker run --rm --name dnsmasq -e SERVICE_NAME=dnsmasq -h ip-10-0-0-182.us-west-2.compute.internal -p 53:53/tcp -p 53:53/udp --cap-add=NET_ADMIN andyshinn/dnsmasq --cache-size=1024 -S /consul/10.0.0.182#8600 -S 10.0.0.2`  How I set this up is:

```

ExecStartPre=-/usr/bin/mkdir -p /etc/systemd/resolved.conf.d

ExecStartPre=-/bin/sh -c 'rm /etc/systemd/resolved.conf.d/00-consul-dns.conf && systemctl restart systemd-resolved'

ExecStartPre=-/usr/bin/docker kill %p

ExecStartPre=-/usr/bin/docker rm -f %p

ExecStartPre=/usr/bin/flock /tmp/docker_pull.lock /usr/bin/docker pull andyshinn/dnsmasq

ExecStart=/usr/bin/docker run \

  --rm \

  --name %p \

  -e "SERVICE_NAME=dnsmasq" \

  -h %H \

  -p 53:53/tcp \

  -p 53:53/udp \

  --cap-add=NET_ADMIN \

  andyshinn/dnsmasq \

    --cache-size=1024 \

    -S /consul/${COREOS_PRIVATE_IPV4}#8600 \

    -S 10.0.0.2

ExecStartPost=/bin/sh -c '\

        echo -e "[Resolve]\nDNS=${COREOS_PRIVATE_IPV4}" > /etc/systemd/resolved.conf.d/00-consul-dns.conf && \

        systemctl restart systemd-resolved;'

ExecStop=/usr/bin/flock /tmp/docker_pull.lock /usr/bin/docker pull andyshinn/dnsmasq

ExecStop=-/usr/bin/docker stop -t 25 %p

ExecStop=-/usr/bin/docker rm %p

ExecStopPost=-/bin/sh -c 'rm /etc/systemd/resolved.conf.d/00-consul-dns.conf && systemctl restart systemd-resolved'

```

now I haven't touched this in awhile, and I'm not 100% sure you need to set the resolved.conf like I do, but this is what I have and it works like a charm (FYI: -S 10.0.0.2 is the AWS dns service)

Let me know if you have any questions!

Adam

[Michael Fischer]

Adam,

Which IP do the other containers on the host use to connect to Consul and dnsmasq?  The host IP?  If so, how do they obtain it?  It will obviously vary per host.

Thanks,

--Michael

--

This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/consul/issues
IRC: #consul on Freenode
---
You received this message because you are subscribed to the Google Groups "Consul" group.

To unsubscribe from this group and stop receiving emails from it, send an email to consul-tool+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/consul-tool/267ce643-7665-4e47-b2d5-7a502ee7a301%40googlegroups.com.

[Adam Greene]

within the containers, I use the dns service names consul provides.  So to hit vault, I call `active.vault.service.consul`, or `consul.service.consul`, or `elasticsearch.service.consul`

I'm on AWS, so I have tight firewalls, and I don't open up port `:8500`, so I can't call `http://consul.service.consul:8500`; I always hit the local consul agent, such as:

source /etc/environment

curl http://$COREOS_PRIVATE_IPV4:8500/v1/catalog/services?pretty=1

curl http://0.0.0.0:8500/v1/catalog/services?pretty=1

from within a docker container, there are three strategies I use:

* in almost *all* cases, I use the dns name of the service that I need to hit; I rarely call consul directly from a service (for example, each app stores all its configs in vault and not directly in consul k/v store)

* if I do need to call consul directly from within a container, I either pass in the host IP as an ENV (so for docker, it could be `-e HOST_IP=${COREOS_PRIVATE_IPV4}`), and then call `curl http://${HOST_IP}:8500/v1/...`

* or I reference the docker bridge, which defaults to `172.17.0.1`; now I would be *very* careful as this used to be hard-coded but my understanding is that it no longer is... so you have to either hard-code it when you boot docker on the host (`--bip 172.17.0.1/16`) or go inspect the docker bridge and get its IP before calling... I ended up just hardcoding the bridge, but I *rarely* use this approach and would rather setup a DNS address via a consul-service or pass in the host IP.  This call would look like `curl http://172.17.0.1:8500/v1/...`

the eco system is changing so quickly this data about how to access the host IP from within docker may no longer apply (though: https://github.com/docker/docker/issues/1143 ; I still don't think you can get this in an automated, docker-supplied way)

 

[Adam Greene]

I should have add that I almost exclusively interface with consul from outside of docker and directly on the host.  Then I just use http://0.0.0.0:8500

[Chris Stevens]

Thanks all for the input so far. I'll summarize what I have so far and then try to do a real write-up later on.

Generally following Michael's suggestions:

1) Created "dummy0" device/interface on host with the 169.254.1.1.

2) Configure dnsmasq

* My dnsmasq config only enabled "interface=lo".

* Enabling "interface=dummy0" didn't work.

* I found that I could ping from a container to 169.254.1.1, but the request would hang for dns requests to that address.

* Watching the dnsmasq query logs, I removed the interface restrictions and saw the docker0 interface was making the requests.

* I configured dnsmasq to listen on "interface=docker0" and was able to get lookups to the host.

3) Configure consul

My config looks like this right now:

  "client_addr": "0.0.0.0",

  "bind_addr": "0.0.0.0",

  "addresses" : {

   "dns": "0.0.0.0",

   "http": "0.0.0.0",

   "https": "0.0.0.0",

   "rpc": "0.0.0.0"

 },

 "advertise_addr": "127.0.0.1",

I just need to change the advertise_addr to my vagrant IP (in this case) or a routable IP for a server installation, but I can see that working just fine.

4) Containers

I tend to use Alpine based containers and the lookup from /etc/resolv.conf is random and/or in parallel.

I saw some inconsistent behavior in this if both the 169. address and the 10. address (from my host) were both present.

To avoid any issues, my plan is to always run docker with --dns=169.254.1.1 and go through the dnsmasq instance.

CentOS containers seemed to work reliably in the order given in /etc/resolv.conf FWIW.

Thanks!

- Chris

[Michael Fischer]

On Fri, Aug 5, 2016 at 10:09 AM, Chris Stevens <chris....@traxo.com> wrote:

2) Configure dnsmasq

* My dnsmasq config only enabled "interface=lo".

* Enabling "interface=dummy0" didn't work.

* I found that I could ping from a container to 169.254.1.1, but the request would hang for dns requests to that address.

* Watching the dnsmasq query logs, I removed the interface restrictions and saw the docker0 interface was making the requests.

* I configured dnsmasq to listen on "interface=docker0" and was able to get lookups to the host.

Try adding "listen-address=169.254.1.1" to the dnsmasq configuration, instead of "interface=docker0" or "interface=dummy0".  That should work.

--Michael

[Michael Fischer]

Oh, also "listen-address=127.0.0.1" and "listen-address=::1"

[Chris Stevens]

Michael:

Worked like a champ, thanks! 

[ja...@fpcomplete.com]

I would fit into that camp, and at least for my use, this has worked out well. I have not found enough justification to put a docker networking plugin or tool to use - I would rather have my docker containers skip docker's networking and just go out the host as usual. For my containers accessing consul on the host, the containers are usually in either localhost or net=host mode, and they can see consul when I want them to.