CONSUL DNS - Dig from external servers dont work , but telnet to the remote server works

[Om]

Hello All,

 I have a problem with CONSUL DNS queries through DIG.  So requesting for guidance.

Dig to a external Consul Server fails, but telnet to that external server succeeds.

Here is my setup:

Consul Server 1 - x.x.x.1

Consul Server 2 - x.x.x.2

Consul Server 3 - x.x.x.3

Remote Node  - x.x.x.4

Consul is running DNS server on 8600 on all of these . Firewall is also opened to all these servers to port 8600 

I am also running DNSMASQ to route the DNS queries on port 53 to the local consul server on port 8600, config as below from the remote node(x.x.x.4)

vi /etc/dnsmasq.d/10-consul

server=/consul/127.0.0.1#8600

vi /etc/resolv.conf

nameserver 127.0.0.1

nameserver a.b.c.d

nameserver w.x.y.z

On Node x.x.x.3, the below dig command all works fine.

dig @localhost -p 8600 consul.service.consul 

dig @x.x.x.3 -p 53 consul.service.consul

dig @x.x.x.3 -p 8600 consul.service.consul

dig consul.service.consul

On Remote Node: But when I try to hit this DNS service from a remote node, then dig fails with the below error code. 

dig x.x.x.3 -p 8600 consul.service.service

;; connection timed out; no servers could be reached

But telnet x.x.x.3 8600 - works fine

(I really want to do this, so I can configure the DNSMASQ on my local node to route the .consul TLD requests to remote consul server ) 

The same works fine on my local(MAC) machine, so I am suspecting this could be something do with network policies , but not sure.

I have done hours of research and tried many things, all in vain. 

Kindly see if any one of you can guide me here.

Thanks in advance,

Om

[Pierre Souchay]

Hello Om,

your example "On Remote Node" is not correct:

It should be:

dig @x.x.x.3 -p 8600 consul.service.consul

@ in order to specify the target DNS server (missing)
consul.service.consul (not consul.service.service) to request the correct name

Regards

[Om]

sorry that was a typo in my original post.

I was actually issuing dig @x.x.x.3 -p 8600 consul.service.consul only.

thanks for pointing it out.

[Pierre Souchay]

I am doing this kind of requests daily on our cluster, this is weird.

Try using +tcp option of dig to work in same conditions as telnet and put dig in verbose mode.

Are you sure your traffic is not simply blocked by a firewall or iptables?

Regards

[Om]

Hi Pierre,

First off thanks for your time. 

 Here is what is happening I do a dig @x.x.x.3 - p 8600 consul.service.consul +tcp.

it returns the list of service instances, (glad it works).

But it crashes my remote Consul server x.x.x.3.

Here is the error log.

panic: sync: negative WaitGroup counter

goroutine 101 [running]:

sync.(*WaitGroup).Add(0xc420250990, 0xffffffffffffffff)

    /goroot/src/sync/waitgroup.go:75 +0x134

sync.(*WaitGroup).Done(0xc420250990)

    /goroot/src/sync/waitgroup.go:100 +0x34

github.com/hashicorp/consul/vendor/github.com/miekg/dns.(*Server).serve(0xc4202508f0, 0x1db8860, 0xc420620e70, 0x1db0da0, 0xc4203f6a10, 0xc42037f2c0, 0x32, 0x32, 0x0, 0x0, ...)

    /gopath/src/github.com/hashicorp/consul/vendor/github.com/miekg/dns/server.go:611 +0x3ae

github.com/hashicorp/consul/vendor/github.com/miekg/dns.(*Server).serveTCP.func1(0xc4203f6a50, 0x1dc3e00, 0xc420138370, 0x77359400, 0xc4202508f0, 0xc4203f6a60)

    /gopath/src/github.com/hashicorp/consul/vendor/github.com/miekg/dns/server.go:498 +0x141

created by github.com/hashicorp/consul/vendor/github.com/miekg/dns.(*Server).serveTCP

    /gopath/src/github.com/hashicorp/consul/vendor/github.com/miekg/dns/server.go:492 +0x2e3 

Thanks,

Om.

[Kyle Havlovitz]

Hi Om, it looks like you're hitting this error in 1.0.5: https://github.com/hashicorp/consul/issues/3877. Is that the version you're running? If it is, upgrading to 1.0.6 should fix the problem.

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/consul/issues
Community chat: https://gitter.im/hashicorp-consul/Lobby
---
You received this message because you are subscribed to the Google Groups "Consul" group.
To unsubscribe from this group and stop receiving emails from it, send an email to consul-tool+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/consul-tool/f284ef8a-1272-4847-931c-b8ec6c09d90d%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Om]

Hi Kyle, 

  Exactly we are in 1.0.5. I will upgrade to 1.0.6. 

Thanks,

Om

To unsubscribe from this group and stop receiving emails from it, send an email to consul-tool...@googlegroups.com.

[Om]

Hi Pierre,

 I was able to do a packet trace on the dig over udp and found that there was a softfirewall on the server which was preventing UDP on 8600. Fixing that solved the problem. 

Your guidance helped me in looking out for the correct root cause. 

Thanks a bunch.

Om.

[Pierre Souchay]

Great, I was glad to help