kube-dns stops working after random amount of time

[Brent P]

Every time I initialize a new cluster, everything works perfectly for anywhere from 3 days to around a month. Then kube-dns simply stops functioning. I can shell into the kubedns container, and it seems to be running fine, although I don't really know what to look for. I can ping a hostname, it resolves and is reachable, so kubedns container itself still has dns service. It's just not providing it for other containers in the cluster. And the failure happens in both containers that have been running since before it started (so they used to be able to resolve+ping a hostname, but now cannot resolve it, but can still ping with IP), and new containers that are created.

I'm not sure if it's related to time, or the number of jobs or pods that have been created. The most recent incident happened after 32 pods had been created, and 20 jobs.

If I delete the kube-dns pod with:

kubectl delete pod --namespace kube-system kube-dns-<pod_id>

A new kube-dns pod is created and things go back to normal (DNS works for all containers, new and old).

I have one master node and two worker nodes. They are all CentOS 7 machines.

To setup the cluster, on the master, I run:

systemctl start etcd
etcdctl mkdir /kube-centos/network
etcdctl mk /kube-centos/network/config "{ "Network": "172.30.0.0/16", "SubnetLen": 24, "Backend": { "Type": "vxlan" } }"
systemctl disable etcd && systemctl stop etcd
systemctl enable docker && systemctl start docker
systemctl enable kubelet && systemctl start kubelet
kubeadm init --kubernetes-version v1.10.0 --apiserver-advertise-address=$(hostname) --ignore-preflight-errors=DirAvailable--var-lib-etcd
kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=1.10&env.WEAVE_NO_FASTDP=1&env.CHECKPOINT_DISABLE=1"
kubectl --kubeconfig=/etc/kubernetes/admin.conf proxy -p 80 --accept-hosts='.*' --address=<master_ip> &

And on the two workers, I run:

kubeadm join --token <K8S_MASTER_HOST>:6443 --discovery-token-ca-cert-hash sha256: --ignore-preflight-errors cri

Here's some shell commands+output that I've run that could be useful:

Before the failure starts happening, this is a container that's running on one of the workers:

bash-4.4# env
PACKAGES= dumb-init musl libc6-compat linux-headers build-base bash git ca-certificates python3 python3-dev
HOSTNAME=network-test
KUBERNETES_PORT_443_TCP_PROTO=tcp
KUBERNETES_PORT_443_TCP_ADDR=10.96.0.1
KUBERNETES_PORT=tcp://10.96.0.1:443
PWD=/
HOME=/root
KUBERNETES_SERVICE_PORT_HTTPS=443
KUBERNETES_PORT_443_TCP_PORT=443
ALPINE_VERSION=3.7
KUBERNETES_PORT_443_TCP=tcp://10.96.0.1:443
TERM=xterm
SHLVL=1
KUBERNETES_SERVICE_PORT=443
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
KUBERNETES_SERVICE_HOST=10.96.0.1
_=/usr/bin/env

 

bash-4.4# ifconfig
eth0 Link encap:Ethernet HWaddr 8A:DD:6E:E8:C4:E3
inet addr:10.44.0.1 Bcast:10.47.255.255 Mask:255.240.0.0
UP BROADCAST RUNNING MULTICAST MTU:65535 Metric:1
RX packets:1 errors:0 dropped:0 overruns:0 frame:0
TX packets:1 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:42 (42.0 B) TX bytes:42 (42.0 B) 

lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)

 

bash-4.4# ip route
default via 10.44.0.0 dev eth0
10.32.0.0/12 dev eth0 scope link src 10.44.0.1

 

bash-4.4# cat /etc/resolv.conf
nameserver 10.96.0.10
search default.svc.cluster.local svc.cluster.local cluster.local dc1.int.company.com dc2.int.company.com dc3.int.company.com
options ndots:5

 

-- Note that even when working, the DNS IP can't be pinged. --
bash-4.4# ping 10.96.0.10
PING 10.96.0.10 (10.96.0.10): 56 data bytes
-- Never unblocks. So we know it's fine that the container can't ping the DNS IP. --

 

bash-4.4# ping 10.44.0.0
PING 10.44.0.0 (10.44.0.0): 56 data bytes
64 bytes from 10.44.0.0: seq=0 ttl=64 time=0.139 ms
64 bytes from 10.44.0.0: seq=1 ttl=64 time=0.124 ms
--- 10.44.0.0 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 0.124/0.131/0.139 ms

bash-4.4# ping somehost.env.dc1.int.company.com
PING somehost.env.dc1.int.company.com (10.112.17.2): 56 data bytes
64 bytes from 10.112.17.2: seq=0 ttl=63 time=0.467 ms
64 bytes from 10.112.17.2: seq=1 ttl=63 time=0.271 ms
64 bytes from 10.112.17.2: seq=2 ttl=63 time=0.214 ms
64 bytes from 10.112.17.2: seq=3 ttl=63 time=0.241 ms
64 bytes from 10.112.17.2: seq=4 ttl=63 time=0.350 ms
--- somehost.env.dc1.int.company.com ping statistics ---
5 packets transmitted, 5 packets received, 0% packet loss
round-trip min/avg/max = 0.214/0.308/0.467 ms

bash-4.4# ping 10.112.17.2
PING 10.112.17.2 (10.112.17.2): 56 data bytes
64 bytes from 10.112.17.2: seq=0 ttl=63 time=0.474 ms
64 bytes from 10.112.17.2: seq=1 ttl=63 time=0.404 ms
--- 10.112.17.2 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 0.404/0.439/0.474 ms

bash-4.4# ping worker1.env
PING worker1.env (10.112.5.50): 56 data bytes
64 bytes from 10.112.5.50: seq=0 ttl=64 time=0.051 ms
64 bytes from 10.112.5.50: seq=1 ttl=64 time=0.102 ms
--- worker1.env ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 0.051/0.076/0.102 ms

After failure starts, same container that's been running the whole time:

bash-4.4# env

PACKAGES= dumb-init musl libc6-compat linux-headers build-base bash git ca-certificates python3 python3-dev

HOSTNAME=vda-test

KUBERNETES_PORT_443_TCP_PROTO=tcp

KUBERNETES_PORT_443_TCP_ADDR=10.96.0.1

KUBERNETES_PORT=tcp://10.96.0.1:443

PWD=/

HOME=/root

KUBERNETES_SERVICE_PORT_HTTPS=443

KUBERNETES_PORT_443_TCP_PORT=443

ALPINE_VERSION=3.7

KUBERNETES_PORT_443_TCP=tcp://10.96.0.1:443

TERM=xterm

SHLVL=1

KUBERNETES_SERVICE_PORT=443

PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

KUBERNETES_SERVICE_HOST=10.96.0.1

OLDPWD=/root

_=/usr/bin/env

bash-4.4# ifconfig

eth0 Link encap:Ethernet HWaddr 22:5E:D5:72:97:98

inet addr:10.44.0.2 Bcast:10.47.255.255 Mask:255.240.0.0

UP BROADCAST RUNNING MULTICAST MTU:65535 Metric:1

RX packets:1645 errors:0 dropped:0 overruns:0 frame:0

TX packets:1574 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:0

RX bytes:718909 (702.0 KiB) TX bytes:150313 (146.7 KiB)

lo Link encap:Local Loopback

inet addr:127.0.0.1 Mask:255.0.0.0

UP LOOPBACK RUNNING MTU:65536 Metric:1

RX packets:0 errors:0 dropped:0 overruns:0 frame:0

TX packets:0 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:1

RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)

bash-4.4# ip route

default via 10.44.0.0 dev eth0

10.32.0.0/12 dev eth0 scope link src 10.44.0.2

bash-4.4# cat /etc/resolv.conf

nameserver 10.96.0.10

search default.svc.cluster.local svc.cluster.local cluster.local dc1.int.company.com dc2.int.company.com dc3.int.company.com

options ndots:5

bash-4.4# ping 10.44.0.0

PING 10.44.0.0 (10.44.0.0): 56 data bytes

64 bytes from 10.44.0.0: seq=0 ttl=64 time=0.130 ms

64 bytes from 10.44.0.0: seq=1 ttl=64 time=0.097 ms

64 bytes from 10.44.0.0: seq=2 ttl=64 time=0.072 ms

64 bytes from 10.44.0.0: seq=3 ttl=64 time=0.102 ms

64 bytes from 10.44.0.0: seq=4 ttl=64 time=0.116 ms

64 bytes from 10.44.0.0: seq=5 ttl=64 time=0.099 ms

ping somehost.env.dc1.int.company.com64 bytes from 10.44.0.0: seq=6 ttl=64 time=0.167 ms

64 bytes from 10.44.0.0: seq=7 ttl=64 time=0.086 ms

--- 10.44.0.0 ping statistics ---

8 packets transmitted, 8 packets received, 0% packet loss

round-trip min/avg/max = 0.072/0.108/0.167 ms

bash-4.4# ping somehost.env.dc1.int.company.com

ping: bad address 'somehost.env.dc1.int.company.com'

bash-4.4# ping 10.112.17.2

PING 10.112.17.2 (10.112.17.2): 56 data bytes

64 bytes from 10.112.17.2: seq=0 ttl=63 time=0.523 ms

64 bytes from 10.112.17.2: seq=1 ttl=63 time=0.319 ms

64 bytes from 10.112.17.2: seq=2 ttl=63 time=0.304 ms

--- 10.112.17.2 ping statistics ---

3 packets transmitted, 3 packets received, 0% packet loss

round-trip min/avg/max = 0.304/0.382/0.523 ms

bash-4.4# ping worker1.env

ping: bad address 'worker1.env'

bash-4.4# ping 10.112.5.50

PING 10.112.5.50 (10.112.5.50): 56 data bytes

64 bytes from 10.112.5.50: seq=0 ttl=64 time=0.095 ms

64 bytes from 10.112.5.50: seq=1 ttl=64 time=0.073 ms

64 bytes from 10.112.5.50: seq=2 ttl=64 time=0.083 ms

--- 10.112.5.50 ping statistics ---

3 packets transmitted, 3 packets received, 0% packet loss

round-trip min/avg/max = 0.073/0.083/0.095 ms

And here are some commands in the kube-dns container:

/ # ifconfig

eth0 Link encap:Ethernet HWaddr 9A:24:59:D1:09:52

inet addr:10.32.0.2 Bcast:10.47.255.255 Mask:255.240.0.0

UP BROADCAST RUNNING MULTICAST MTU:65535 Metric:1

RX packets:4387680 errors:0 dropped:0 overruns:0 frame:0

TX packets:4124267 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:0

RX bytes:1047398761 (998.8 MiB) TX bytes:1038950587 (990.8 MiB)

lo Link encap:Local Loopback

inet addr:127.0.0.1 Mask:255.0.0.0

UP LOOPBACK RUNNING MTU:65536 Metric:1

RX packets:4352618 errors:0 dropped:0 overruns:0 frame:0

TX packets:4352618 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:1

RX bytes:359275782 (342.6 MiB) TX bytes:359275782 (342.6 MiB)

/ # ping somehost.env.dc1.int.company.com

PING somehost.env.dc1.int.company.com (10.112.17.2): 56 data bytes

64 bytes from 10.112.17.2: seq=0 ttl=63 time=0.430 ms

64 bytes from 10.112.17.2: seq=1 ttl=63 time=0.252 ms

--- somehost.env.dc1.int.company.com ping statistics ---

2 packets transmitted, 5 packets received, 0% packet loss

round-trip min/avg/max = 0.208/0.274/0.430 ms

/ # netstat

Active Internet connections (w/o servers)

Proto Recv-Q Send-Q Local Address Foreign Address State

tcp 0 0 kube-dns-86f4d74b45-2kxdr:10054 10.32.0.1:53152 TIME_WAIT

tcp 0 0 kube-dns-86f4d74b45-2kxdr:sunproxyadmin 10.32.0.1:58424 TIME_WAIT

tcp 0 0 kube-dns-86f4d74b45-2kxdr:10054 10.32.0.1:53174 TIME_WAIT

tcp 0 0 kube-dns-86f4d74b45-2kxdr:sunproxyadmin 10.32.0.1:58468 TIME_WAIT

tcp 0 0 kube-dns-86f4d74b45-2kxdr:sunproxyadmin 10.32.0.1:58446 TIME_WAIT

tcp 0 0 kube-dns-86f4d74b45-2kxdr:10054 10.32.0.1:53096 TIME_WAIT

tcp 0 0 kube-dns-86f4d74b45-2kxdr:sunproxyadmin 10.32.0.1:58490 TIME_WAIT

tcp 0 0 kube-dns-86f4d74b45-2kxdr:10054 10.32.0.1:53218 TIME_WAIT

tcp 0 0 kube-dns-86f4d74b45-2kxdr:10054 10.32.0.1:53100 TIME_WAIT

tcp 0 0 kube-dns-86f4d74b45-2kxdr:10054 10.32.0.1:53158 TIME_WAIT

tcp 0 0 kube-dns-86f4d74b45-2kxdr:10054 10.32.0.1:53180 TIME_WAIT

tcp 0 0 kube-dns-86f4d74b45-2kxdr:sunproxyadmin 10.32.0.1:58402 TIME_WAIT

tcp 0 0 kube-dns-86f4d74b45-2kxdr:10054 10.32.0.1:53202 TIME_WAIT

tcp 0 0 kube-dns-86f4d74b45-2kxdr:10054 10.32.0.1:53178 TIME_WAIT

tcp 0 0 kube-dns-86f4d74b45-2kxdr:sunproxyadmin 10.32.0.1:58368 TIME_WAIT

tcp 0 0 kube-dns-86f4d74b45-2kxdr:10054 10.32.0.1:53134 TIME_WAIT

tcp 0 0 kube-dns-86f4d74b45-2kxdr:10054 10.32.0.1:53200 TIME_WAIT

tcp 0 0 kube-dns-86f4d74b45-2kxdr:10054 10.32.0.1:53136 TIME_WAIT

tcp 0 0 kube-dns-86f4d74b45-2kxdr:10054 10.32.0.1:53130 TIME_WAIT

tcp 0 0 kube-dns-86f4d74b45-2kxdr:10054 10.32.0.1:53222 TIME_WAIT

tcp 0 0 kube-dns-86f4d74b45-2kxdr:10054 10.32.0.1:53196 TIME_WAIT

tcp 0 0 kube-dns-86f4d74b45-2kxdr:48230 10.96.0.1:https ESTABLISHED

tcp 0 0 kube-dns-86f4d74b45-2kxdr:10054 10.32.0.1:53102 TIME_WAIT

netstat: /proc/net/tcp6: No such file or directory

netstat: /proc/net/udp6: No such file or directory

netstat: /proc/net/raw6: No such file or directory

Active UNIX domain sockets (w/o servers)

Proto RefCnt Flags Type State I-Node Path

Version/OS info on master+worker nodes:

kubectl version

Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.0", GitCommit:"fc32d2f3698e36b93322a3465f63a14e9f0eaead", GitTreeState:"clean", BuildDate:"2018-03-26T16:55:54Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}

Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.0", GitCommit:"fc32d2f3698e36b93322a3465f63a14e9f0eaead", GitTreeState:"clean", BuildDate:"2018-03-26T16:44:10Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}

cat /etc/os-release

NAME="CentOS Linux"

VERSION="7 (Core)"

ID="centos"

ID_LIKE="rhel fedora"

VERSION_ID="7"

PRETTY_NAME="CentOS Linux 7 (Core)"

ANSI_COLOR="0;31"

CPE_NAME="cpe:/o:centos:centos:7"

HOME_URL="https://www.centos.org/"

BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"

CENTOS_MANTISBT_PROJECT_VERSION="7"

REDHAT_SUPPORT_PRODUCT="centos"

REDHAT_SUPPORT_PRODUCT_VERSION="7"

uname -a

Linux master1.env.dc1.int.company.com 3.10.0-514.el7.x86_64 #1 SMP Tue Nov 22 16:42:41 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

[Brent P]

I've discovered that in a working environment, I can shell into the kube-dns pod --> kubedns container, and ping external addresses, either with a domain name or IP. In an environment where the DNS has started failing (but connectivity still functions when using IPs), if I shell into the kubedns container, I can't ping external addresses, even if using IPs.

[Jay Vyas]

The first thing to check is how many total pods are running on your cluster, and the kubectl logs for dns pods.

Iptables can start to misbehave once you get lots of services. 

Is your  cluster  over subscribed ?

Do actual pod IP addresses shown from kubectl get endpoints respond to pings ? 

On Dec 6, 2018, at 8:44 PM, Brent P <brent....@gmail.com> wrote:

I've discovered that in a working environment, I can shell into the kube-dns pod --> kubedns container, and ping external addresses, either with a domain name or IP. In an environment where the DNS has started failing (but connectivity still functions when using IPs), if I shell into the kubedns container, I can't ping external addresses, even if using IPs.

--
You received this message because you are subscribed to the Google Groups "Kubernetes developer/contributor discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-de...@googlegroups.com.
To post to this group, send email to kuberne...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-dev/5311d8dc-16a3-45d2-8837-e02e9c4eaa78%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Brent P]

In a functioning cluster the logs are all empty. In a failed cluster, dnsmasq and sidecar are empty, BUT kubedns has this:

kubectl -n kube-system log kube-dns-86f4d74b45-rndqn kubedns I1206 23:42:41.555778       1 logs.go:41] skydns: failure to forward request "read udp 10.32.0.4:52820->1.2.3.4:53: i/o timeout" I1206 23:42:43.556240       1 logs.go:41] skydns: failure to forward request "read udp 10.32.0.4:37910->1.2.3.4:53: i/o timeout" I1206 23:42:46.561815       1 logs.go:41] skydns: failure to forward request "read udp 10.32.0.4:48967->1.2.3.4:53: i/o timeout" I1206 23:42:48.562266       1 logs.go:41] skydns: failure to forward request "read udp 10.32.0.4:40959->1.2.3.4:53: i/o timeout"

Note that 10.32.0.4 is the kube-dns pod's IP. 1.2.3.4 (not the actual IP, just not sure I should share company IPs even if they are internal) is dns.my.company.com, and is in the host machine's /etc/resolve.conf.

Any non-kube-dns container in the failed cluster can still ping 1.2.3.4. But it can't resolve its name because it has the kube-dns pod's IP as the nameserver in its /etc/resolve.conf.

I'm not sure what over subscribed means. 

There are 13 running pods, 12 of which are in the kube-system namespace. But there are 1247 pods with "Terminated: Completed" status. I'm wondering if the issue is caused by IPs not being made available to use when a pod terminates, and being limited to 256 IPs. Although I can create a new pod right now that has the ability to ping external servers using IPs, just not domain names, I assume because the kubedns container can't talk to anything (even with IPs). 

The pod IP address do respond to pings. I can create a new pod and ping its pod IP from the host machine or from another pod. I can even do this with the kube-dns pod IP.

[John Belamaric]

Check for conntrack failures in the nodes. Sometimes the table fills up if you have lots of outbound DNS queries, because UDP has to timeout rather than get removed immediately when the connection closes.

This sort of thing is the reason for https://github.com/kubernetes/enhancements/blob/master/keps/sig-network/0030-nodelocal-dns-cache.md

--
You received this message because you are subscribed to the Google Groups "Kubernetes developer/contributor discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-de...@googlegroups.com.
To post to this group, send email to kuberne...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-dev/a51516b1-d468-4048-bfb4-230d7a6af11a%40googlegroups.com.

[Duffie Cooley]

This is also worth a read.

https://blog.quentin-machu.fr/2018/06/24/5-15s-dns-lookups-on-kubernetes/

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-dev/CAC_RkjwZanK8EabS-4ZM5ZH5pPN9W1uPr625_-3uaLRcxc1DMg%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Duffie Cooley

415.237.0082

Field Engineer 

Heptio.com

[Brent P]

"Check for conntrack failures in the nodes"

Sorry for my ignorance, but how do I do that?

Do I do it on the actual K8s (master + worker) nodes, or from within the containers that can't use DNS?

Thanks!

-Brent

[John Belamaric]

Depends on your linux distro, probably they'll be in /var/log/messages or journalctl or dmesg, something like "nf_conntrack: table full". See https://security.stackexchange.com/questions/43205/nf-conntrack-table-full-dropping-packet and other links.

Check the nodes having DNS problems, and also the nodes running kube-dns.

--
You received this message because you are subscribed to the Google Groups "Kubernetes developer/contributor discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-de...@googlegroups.com.
To post to this group, send email to kuberne...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-dev/0456dcf6-4cb8-4158-befe-43672f808a7a%40googlegroups.com.