Basic Consul DNS on AWS

[Chris Forbes]

Hi everyone,

I've installed a Consul  cluster (v 0.8.1) on CentOS 7 in AWS and all is well. I'm now trying to configure my Agents to talk to Consul with `vault.service.consul`. I've followed the documentation on DNS Forwarding, trying with both BIND and DNSMasq. In both cases, I was able to `dig @localhost vault.service.consul` which returned:

[centos@ip-10-32-4-223 ~]$ dig @localhost vault.service.consul

; <<>> DiG 9.9.4-RedHat-9.9.4-38.el7_3.3 <<>> @localhost vault.service.consul

; (2 servers found)

;; global options: +cmd

;; Got answer:

;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 49752

;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:

;vault.service.consul.          IN      A

;; ANSWER SECTION:

vault.service.consul.   0       IN      A       10.32.4.223

;; Query time: 13 msec

;; SERVER: 127.0.0.1#53(127.0.0.1)

;; WHEN: Fri May 26 09:42:32 UTC 2017

;; MSG SIZE  rcvd: 54

However, neither option was able to work without specifying `@localhost`

[centos@ip-10-32-4-223 ~]$ dig vault.service.consul

; <<>> DiG 9.9.4-RedHat-9.9.4-38.el7_3.3 <<>> vault.service.consul

;; global options: +cmd

;; Got answer:

;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 34383

;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION:

; EDNS: version: 0, flags:; udp: 4096

;; QUESTION SECTION:

;vault.service.consul.          IN      A

;; AUTHORITY SECTION:

.                       60      IN      SOA     a.root-servers.net. nstld.verisign-grs.com. 2017052600 1800 900 604800 86400

;; Query time: 20 msec

;; SERVER: 10.32.0.2#53(10.32.0.2)

;; WHEN: Fri May 26 09:42:39 UTC 2017

;; MSG SIZE  rcvd: 124

I've tried adding `"recursor": "169.254.169.253",` to my Consul configuration, which I have seen suggested in another topic here, but that made no difference. I've also done a lot of Googling, and everyone suggests just adding DNSMasq and the single config file `echo "server=/consul/127.0.0.1#8600" > "/etc/dnsmasq.d/10-consul". I'm quite frustrated now.

My networking knowledge is very limited, so this has been quite difficult. I really wish there were some complete guides on the HashiCorp sites. If anyone could help me out, I'd really appreciate it.

[Brian Lalor]

Your /etc/resolv.conf needs to point to 127.0.0.1 or a local IP of the host.  And dnsmasq needs to be configured with a DNS server (recursor) to forward requests to.  dig, host, and most other applications (I’m looking at you, nginx) use /etc/resolv.conf to determine which DNS server to contact.

Logically, the order would go dig → /etc/resolv.conf → dnsmasq → recursor.  When looking up something under the .consul domain, it would go dig → /etc/resolv.conf → dnsmasq → consul agent.

I’m also running centos 7, consul, and dnsmasq.  My dnsmasq looks like:

/etc/dnsmasq.conf:

conf-dir=/etc/dnsmasq.d

/etc/dnsmasq.d/000-base.conf:

# Never forward plain names (without a dot or domain part)

domain-needed

# Never forward addresses in the non-routed address spaces.

bogus-priv

# disable negative caching

no-negcache

# read from resolv.conf generated via NetworkManager

resolv-file=/etc/resolv.conf.dnsmasq

/etc/dnsmasq.d/010-consul.conf:

# forward queries for .consul TLD to the consul agent

server=/consul/127.0.0.1#8600

/etc/resolv.conf has the local IP of the instance:

# generated by /etc/NetworkManager/dispatcher.d/50-update-dnsmasq-resolv-conf; do not edit

nameserver 10.112.16.146

search node.consul us-east-1.gen.aws.dev.example.com

/etc/resolv.conf.dnsmasq contains the nameserver from the DHCP config in the VPC:

nameserver 10.112.0.2

/etc/NetworkManager/dispatcher.d/50-update-dnsmasq-resolv-conf is a script I wrote to manage /etc/resolv.conf.dnsmasq:

#!/bin/bash

## script to be executed when dhcp changes are made; maintains /etc/resolv.conf

## man NetworkManager

interface="${1}"

action="${2}"

/bin/logger -t "${0}" \

    "invoked for interface ${interface:-<not_provided>} and action '${action}' with nameservers '${IP4_NAMESERVERS}', domains '${IP4_DOMAINS}'"

if [ -n "${IP4_NAMESERVERS}" ]; then

    echo "# generated by ${0}; do not edit" >| /etc/resolv.conf.dnsmasq

    

    for ns in ${IP4_NAMESERVERS}; do

        echo "nameserver ${ns}" >> /etc/resolv.conf.dnsmasq

    done

    ## use IP of eth0, so we can use the same resolv.conf and mount it in

    ## containers

    local_ip=$( ip addr show eth0 | awk '/inet / {print substr($2, 0, index($2, "/") - 1)}' )

    echo "# generated by ${0}; do not edit" >| /etc/resolv.conf

    echo "nameserver ${local_ip:-127.0.0.1}" >> /etc/resolv.conf

    

    if [ -n "${IP4_DOMAINS}" ]; then

        echo "search node.consul ${IP4_DOMAINS}" >> /etc/resolv.conf

    fi

fi

The final piece is /etc/NetworkManager/conf.d/000-hands-off-my-resolv.conf, which tells NetworkManager to not manage /etc/resolv.conf:

[main]

## configure NetworkManager to not manage /etc/resolv.conf, as we'll bring our own

## man NetworkManager.conf

dns=none

I hope that helps, Chris.

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/consul/issues
IRC: #consul on Freenode
---
You received this message because you are subscribed to the Google Groups "Consul" group.
To unsubscribe from this group and stop receiving emails from it, send an email to consul-tool...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/consul-tool/3f97ef88-6481-4ddf-a869-a82b31dec156%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

— 

Brian Lalor

bla...@bravo5.org

[Chris Forbes]

Thank you for the quick response, Brian. I'll give it a try and let you know how it goes.

[Chris Forbes]

So, we finally got it working! Thank you for the advice. I didn't take the exact same approach, but used some bits.

I changed the IP of the "recursor" in the Consul config file from the suggested value I copied from somewhere else to be the AWS VPC cidr + 2 (The built in DNS server).

/etc/consul.d/config.json

"recursor": "10.32.0.2",

I then added the two DNSMasq config files with the additions from yours.

/etc/dnsmasq.d/00-base.conf

# Never forward plain names (without a dot or domain part)

domain-needed

# Never forward addresses in the non-routed address spaces.

bogus-priv

# Disable negative caching.

no-negcache

/etc/dnsmasq.d/10-consul.conf

# Forward queries for ".consul" TLD to the Consul Agent.

server=/consul/127.0.0.1#8600

And rather than make changes to /etc/resolv.conf with the NetworkManager, I prepended a nameserver with a command I found elsewhere by doing the following:

/etc/dhcp/dhclient.conf

prepend domain-name-servers 127.0.0.1;

so that /etc/resolv.conf looks like this with the additions from AWS's DHCP options (so that they can update in future if needed).

/etc/resolv.conf

; generated by /usr/sbin/dhclient-script

search eu-west-1.compute.internal

nameserver 127.0.0.1

nameserver 10.32.0.2

There may be a few scenarios where this will not be sufficient - and hopefully people will tell me if they know of them - but for my immediate needs, this does the job.

Thanks again for your help, Brian!