Docker networking broken on GKE GCI image

[Wil Reichert]

I've got a GKE cluster that was create with a command something like

# gcloud container clusters create "cluster" --machine-type "n1-standard-1" --image-type=GCI --num-nodes "3" --network "default"

The resulting vm image os-release file is

BUILD_ID=8530.71.0
NAME="Google Container-VM Image"
GOOGLE_CRASH_ID=Lakitu
VERSION_ID=53
BUG_REPORT_URL=https://crbug.com/new
PRETTY_NAME="Google Container-VM Image"
VERSION=53
GOOGLE_METRICS_PRODUCT_ID=26
HOME_URL="https://cloud.google.com/compute/docs/containers/vm-image/"
ID=gci

GKE is configured with kubenet & a cbr0 bridge. Docker bridge networking is setup to use the docker0 bridge. Since the docker0 bridge does not exist any attempt by docker to use this network fails like:

# docker run --rm -i -t busybox sh

docker: Error response from daemon: failed to create endpoint pedantic_lalande on network bridge: adding interface vethdc5e518 to bridge docker0 failed: could not find bridge docker0: route ip+net: no such network interface.

Adding a --net=Host to the above run statement works fine. The primary problem is docker does not accept the --net parameter for builds. The relevant Docker issue has been open for a year & a half, a PR fix for 9 months.

My primary use case is running Docker builds on Jenkins slave pods which is a complete no go with the above configuration.

Wil

[Amey Deshpande]

From what I can tell, the use of kubenet and cbr0 is mutually exclusive: https://github.com/kubernetes/kubernetes/blob/master/cluster/gce/gci/configure-helper.sh#L372

When cbr0 is being used, Docker daemon is restarted with "--bridge=cbr0" on commandline to use the correct bridge device.  However, when cni is being used, I do not see any evidence of deleting docker0 in the code.  I'll let K8s/GKE networking experts comment on it.

--
You received this message because you are subscribed to the Google Groups "Kubernetes user discussion and Q&A" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-users+unsubscribe@googlegroups.com.
To post to this group, send email to kubernetes-users@googlegroups.com.
Visit this group at https://groups.google.com/group/kubernetes-users.
For more options, visit https://groups.google.com/d/optout.

[Wil Reichert]

1.3.6 is slightly different but the result should be the same - https://github.com/kubernetes/kubernetes/blob/v1.3.6/cluster/gce/gci/configure-helper.sh#L322

The logic in the code seems contrary to the kubenet docs - 'Kubenet plugin: implements basic cbr0 using the bridge and host-local CNI plugins' i.e. 'cbr0 exists when cni is being used' but perhaps I'm just misinterpreting something.

To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-use...@googlegroups.com.
To post to this group, send email to kubernet...@googlegroups.com.

[Amey Deshpande]

Turns out in 1.3.6 we actually delete docker0: https://github.com/kubernetes/kubernetes/blob/v1.3.6/cluster/gce/gci/configure-helper.sh#L458

It was fixed recently https://github.com/kubernetes/kubernetes/pull/31637.  I think v1.4.0 will have the fix.  I don't know if there are plans to fix it in 1.3 branch.

To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-users+unsubscribe@googlegroups.com.
To post to this group, send email to kubernetes-users@googlegroups.com.