Best practices for DNS-based service discovery?

[Joe Shaw]

Hi,

I'm a new Nomad user and struggling a bit to figure out how best to do DNS-based service discovery within services started by Nomad.  I'm running Consul on my machines alongside Nomad.

I've read https://www.nomadproject.io/docs/jobspec/servicediscovery.html and while it does a good job of describing how services defined in Nomad are represented with Consul, it doesn't really go into how to use it in conjunction with Nomad services.

DNS is important to me because I am running some "legacy" services which can't talk to the Consul API or do DNS SRV requests.  I've used static port allocations, which is fine, but I want to make sure that Consul's DNS is available to Nomad's services so they can find each other.

Is running dnsmasq to merge the system resolver with Consul's still considered the best way to do things?  How do I pass that into Nomad services?  The Docker driver has settings for dns_servers, but (a) I don't want to have to set those for every service and (b) it's not clear to me what value I would pass in exactly -- 127.0.0.1 clearly isn't right inside the container, but the IP address of the docker bridge interface isn't static.

Any guidance would be appreciated.

Thanks,

Joe

[Mathias Lafeldt]

Hi Joe,

The other day, I stumbled upon this Terraform code which shows how to deploy Nomad together with Consul and dnsmasq:

https://github.com/hashicorp/atlas-examples/blob/master/nomad-cluster/aws-google-nomad-cluster/aws-instances.tf

Hope this helps.

-Mathias

https://tinyletter.com/production-ready

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/nomad/issues
IRC: #nomad-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Nomad" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nomad-tool+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/nomad-tool/85a6c02d-9308-4abd-acb1-5c46865c5e19%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Samit Pal]

I'm new to nomad as well, but it seems like configuring dns forwarding as explained https://www.consul.io/docs/guides/forwarding.html is one way to achieve.

Perhaps you could run dnsmasq within the the container itself and pass 127.0.0.1 as value for the 'dns_servers' option. Configure dnsmasq to forward .consul TLD following link above.

The other way could be running dnsmasq separately and configured appropriately. Then pass that ip(s) to the 'dns_servers'  option for your containers.

To view this discussion on the web visit https://groups.google.com/d/msgid/nomad-tool/CAMFa-2hHSZSx2ho00K0_otKGwkoigmCtVo9yTycPsJXEssJ4QA%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Regards

Samit

[Diptanu Choudhury]

Hi,

It depends on the driver you are using. If you are using the raw_exec or the exec drivers, Nomad's services are not any different than any other process running on the server. So if you have dnsmasq setup properly, things should just work.

If you are using the docker driver, then things get complicated if you are using the bridge/default networking mode as the docker container's network would be isolated from the host, and your dnsmasq running on the host won't work properly.

You have a valid point about not needing to specify the dns servers in every job, so I think we can support the dns server as a client config which the operator can set for all containers.

--

This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/nomad/issues
IRC: #nomad-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Nomad" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nomad-tool+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/nomad-tool/85a6c02d-9308-4abd-acb1-5c46865c5e19%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Thanks,
Diptanu Choudhury

Web - www.linkedin.com/in/diptanu

Twitter - @diptanu

[Joe Shaw]

Hi,

Thanks for the reply.  I should have made it clear in my original email: I'm using the Docker driver.

Joe

[Sam Barham]

I recently got this working, after some head scratching.  My solution was to add the file "/etc/systemd/system/docker.service.d/start.conf" containing the following:

[Service]

ExecStart=

ExecStart=/usr/bin/docker daemon --dns=172.17.0.1 --dns=<<Downstream DNS>> -H fd://

If you're on SysV instead, edit /etc/default/docker, add a line like:

DOCKER_OPTS="--dns=172.17.0.1 --dns=<<Downstream DNS>>"

<<Downstream DNS>> is whatever is listed as a nameserver in /etc/resolv.conf

[Joe Shaw]

Hi,

I assume that 172.17.0.1 is the IP address assigned in the bridge network by Docker.  How stable is that address?

Joe

You received this message because you are subscribed to a topic in the Google Groups "Nomad" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/nomad-tool/w2fZ2M2cRVw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to nomad-tool+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/nomad-tool/5133aa5c-5447-4884-b08a-7f4f75a56a63%40googlegroups.com.