DNS queries doesn't resolve randomly

k1.hed...@gmail.com

unread,

Jan 8, 2018, 6:57:40 AM1/8/18

to Kubernetes user discussion and Q&A

Hello,
I've set up a cluster in Azure using Kubespray, VMs are Ubuntu 16.04, and Kubespray configs are pretty much default, after installation when I try to resolve DNS name I get NXDOMAIN randomly.
I've tried instructions in this link:
https://kubernetes.io/docs/tasks/administer-cluster/dns-custom-nameservers/#debugging-dns-resolution
This is the response:

k1@azurekube1:~$ kubectl exec -it busybox -- nslookup kubernetes.default
Server: 10.233.0.3
Address 1: 10.233.0.3

Name: kubernetes.default
Address 1: 10.233.0.1 kubernetes.default.svc.azurekube.pinsvc.net

k1@azurekube1:~$ kubectl exec -it busybox -- time nslookup kubernetes.default
Server: 10.233.0.3
Address 1: 10.233.0.3 kube-dns.kube-system.svc.azurekube.pinsvc.net

nslookup: can't resolve 'kubernetes.default'
command terminated with exit code 1
real 0m 40.03s
user 0m 0.00s
sys 0m 0.00s

First query gets resolved but second does not. What's interesting is when I don't get response it takes like 20, 30 or 40 seconds to respond. And I've tried to increase logging in kube-dns, I don't see any warning or errors in kubedns container, but dnsmasq has something like this:

I0108 11:48:16.711820 dnsmasq[14]: query[A] kubernetes.default.svc.azurekube.pinsvc.net from 127.0.0.1
I0108 11:48:16.711913 dnsmasq[14]: cached kubernetes.default.svc.azurekube.pinsvc.net is 10.233.0.1
I0108 11:48:19.686014 dnsmasq[14]: reply 10.233.0.3 is kube-dns.kube-system.svc.azurekube.pinsvc.net
I0108 11:48:21.712354 dnsmasq[14]: query[A] kubernetes.default.svc.azurekube.pinsvc.net from 127.0.0.1
I0108 11:48:21.712437 dnsmasq[14]: cached kubernetes.default.svc.azurekube.pinsvc.net is 10.233.0.1
I0108 11:48:24.691391 dnsmasq[14]: query[AAAA] kubernetes.default from 10.233.124.134
I0108 11:48:24.691457 dnsmasq[14]: cached kubernetes.default is NXDOMAIN
I0108 11:48:25.462845 processUpdate {Version: Data:map[]}
I0108 11:48:25.462895 Config was unchanged (version )
I0108 11:48:26.712929 dnsmasq[14]: query[A] kubernetes.default.svc.azurekube.pinsvc.net from 127.0.0.1
I0108 11:48:26.712986 dnsmasq[14]: cached kubernetes.default.svc.azurekube.pinsvc.net is 10.233.0.1
I0108 11:48:29.691979 dnsmasq[14]: query[AAAA] kubernetes.default from 10.233.124.134
I0108 11:48:29.692043 dnsmasq[14]: cached kubernetes.default is NXDOMAIN
I0108 11:48:29.692204 dnsmasq[14]: query[AAAA] kubernetes.default.default.svc.azurekube.pinsvc.net from 10.233.124.134
I0108 11:48:29.692532 dnsmasq[14]: forwarded kubernetes.default.default.svc.azurekube.pinsvc.net to 127.0.0.1
I0108 11:48:29.692832 dnsmasq[14]: reply kubernetes.default.default.svc.azurekube.pinsvc.net is NXDOMAIN
I0108 11:48:29.692999 dnsmasq[14]: query[AAAA] kubernetes.default.svc.azurekube.pinsvc.net from 10.233.124.134
I0108 11:48:29.693076 dnsmasq[14]: cached kubernetes.default.svc.azurekube.pinsvc.net is NODATA-IPv6
I0108 11:48:31.713324 dnsmasq[14]: query[A] kubernetes.default.svc.azurekube.pinsvc.net from 127.0.0.1
I0108 11:48:31.713365 dnsmasq[14]: cached kubernetes.default.svc.azurekube.pinsvc.net is 10.233.0.1

I've tried setting up cluster multiple times with Kubernetes 1.9 and 1.8.

I'm out leads to debug this situation. Why this happens randomly?

John Belamaric

unread,

Jan 9, 2018, 3:02:53 PM1/9/18

to kubernet...@googlegroups.com

It is probably this:

https://github.com/kubernetes/dns/issues/186

You’ll see this less with CoreDNS but it looks like we are having a similar problem but less often:

https://github.com/coredns/coredns/issues/1365

John

--
You received this message because you are subscribed to the Google Groups "Kubernetes user discussion and Q&A" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-use...@googlegroups.com.
To post to this group, send email to kubernet...@googlegroups.com.
Visit this group at https://groups.google.com/group/kubernetes-users.
For more options, visit https://groups.google.com/d/optout.

Keyvan Hedayati

unread,

Jan 13, 2018, 1:28:16 AM1/13/18

to kubernet...@googlegroups.com

Issue is that I've another cluster installed with same kubespray version and it doesn't have this issue and my services are not headless.

Anyway I was in rush to setup cluster so I used AKS in azure instead of setting it up with kubespray

To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-users+unsubscribe@googlegroups.com.
To post to this group, send email to kubernetes-users@googlegroups.com.

Visit this group at https://groups.google.com/group/kubernetes-users.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "Kubernetes user discussion and Q&A" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/kubernetes-users/9HYfOfkizvo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to kubernetes-users+unsubscribe@googlegroups.com.
To post to this group, send email to kubernetes-users@googlegroups.com.

Visit this group at https://groups.google.com/group/kubernetes-users.
For more options, visit https://groups.google.com/d/optout.

--

Live long and prosper,

K1.H

Reply all

Reply to author

Forward