Broken nodes on custom node pools: kube-proxy-gke Init:ImagePullBackOff

[paolo.m...@sparkfabrik.com]

We are experiencing networking problems over an entire pool of nodes, they seems to have network issues causing also the kube-proxy to fail, the nodes are freshly provisioned as they are configured to autoscale, i've also tried to delete the entire pool but the problem persists, also for new pools.

Kubectl output:

kube-system kube-proxy-gke-spark-op-services-gitlab-ci-build-fb120c5e-q331 0/1 Init:ImagePullBackOff 0 18m

The issue is quite urgent.

[paolo.m...@sparkfabrik.com]

What happens is that the docker containers provisioned on the nodes (by gitlab in this case) suffer of networking issues, i suppose because the kube-proxy is missing.

[paolo.m...@sparkfabrik.com]

The kubectl output

spark-k8s-services git:dev ❯ kubectl logs -f kube-proxy-gke-spark-op-services-gitlab-ci-build-fb120c5e-tct7 -nkube-system
Error from server (BadRequest): container "kube-proxy" in pod "kube-proxy-gke-spark-op-services-gitlab-ci-build-fb120c5e-tct7" is waiting to start: PodInitializing


spark-k8s-services git:dev ❯ kubectl describe pod kube-proxy-gke-spark-op-services-gitlab-ci-build-fb120c5e-tct7 -nkube-system
Name: kube-proxy-gke-spark-op-services-gitlab-ci-build-fb120c5e-tct7
Namespace: kube-system
Node: gke-spark-op-services-gitlab-ci-build-fb120c5e-tct7/10.132.0.8
Start Time: Thu, 12 Oct 2017 18:52:22 +0200
Labels: component=kube-proxy
tier=node
Annotations: kubernetes.io/config.hash=e1aba4d0cdf8ee0f1b89b21ea6b46704
kubernetes.io/config.mirror=e1aba4d0cdf8ee0f1b89b21ea6b46704
kubernetes.io/config.seen=2017-10-12T16:52:17.72294518Z
kubernetes.io/config.source=file
scheduler.alpha.kubernetes.io/critical-pod=
Status: Pending
IP: 10.132.0.8
Init Containers:
touch-lock:
Container ID:
Image: busybox
Image ID:
Port: <none>
Command:
/bin/touch
/run/xtables.lock
State: Waiting
Reason: ImagePullBackOff
Ready: False
Restart Count: 0
Environment: <none>
Mounts:
/run from run (rw)
Containers:
kube-proxy:
Container ID:
Image: gcr.io/google_containers/kube-proxy:v1.7.6
Image ID:
Port: <none>
Command:
/bin/sh
-c
echo -998 > /proc/$$$/oom_score_adj && kube-proxy --master=https://104.199.47.251 --kubeconfig=/var/lib/kube-proxy/kubeconfig --cluster-cidr=10.0.0.0/14 --resource-container="" --v=2 --feature-gates=ExperimentalCriticalPodAnnotation=true --iptables-sync-period=1m --iptables-min-sync-period=10s 1>>/var/log/kube-proxy.log 2>&1
State: Waiting
Reason: PodInitializing
Ready: False
Restart Count: 0
Requests:
cpu: 100m
Environment: <none>
Mounts:
/etc/ssl/certs from etc-ssl-certs (ro)
/run/xtables.lock from iptableslock (rw)
/usr/share/ca-certificates from usr-ca-certs (ro)
/var/lib/kube-proxy/kubeconfig from kubeconfig (rw)
/var/log from varlog (rw)
Conditions:
Type Status
Initialized False
Ready False
PodScheduled True
Volumes:
usr-ca-certs:
Type: HostPath (bare host directory volume)
Path: /usr/share/ca-certificates
etc-ssl-certs:
Type: HostPath (bare host directory volume)
Path: /etc/ssl/certs
kubeconfig:
Type: HostPath (bare host directory volume)
Path: /var/lib/kube-proxy/kubeconfig
varlog:
Type: HostPath (bare host directory volume)
Path: /var/log
run:
Type: HostPath (bare host directory volume)
Path: /run
iptableslock:
Type: HostPath (bare host directory volume)
Path: /run/xtables.lock
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: :NoExecute
Events: <none>

[paolo.m...@sparkfabrik.com]

A dump of k8s events:

```
kube-system 2017-10-12 18:58:11 +0200 CEST 2017-10-12 18:52:24 +0200 CEST 6 kube-proxy-gke-spark-op-services-gitlab-ci-build-fb120c5e-tct7 Pod spec.initContainers{touch-lock} Normal Pulling kubelet, gke-spark-op-services-gitlab-ci-build-fb120c5e-tct7 pulling image "busybox"
kube-system 2017-10-12 18:58:11 +0200 CEST 2017-10-12 18:52:31 +0200 CEST 6 kube-proxy-gke-spark-op-services-gitlab-ci-build-fb120c5e-tct7 Pod spec.initContainers{touch-lock} Warning Failed kubelet, gke-spark-op-services-gitlab-ci-build-fb120c5e-tct7 Failed to pull image "busybox": rpc error: code = 2 desc = error pulling image configuration: unknown blob
kube-system 2017-10-12 18:58:11 +0200 CEST 2017-10-12 18:52:31 +0200 CEST 36 kube-proxy-gke-spark-op-services-gitlab-ci-build-fb120c5e-tct7 Pod Warning FailedSync kubelet, gke-spark-op-services-gitlab-ci-build-fb120c5e-tct7 Error syncing pod
```