Kubernetes Security Announcement: dnsmasq vulnerability CVE-2017-14491 requires upgrade to patched version

[CJ Cullen]

Kubernetes Security Announcement

dnsmasq vulnerability CVE-2017-14491 requires upgrade to patched version.

A critical vulnerability has been disclosed in dnsmasq, which runs as a part of the k8s-dns-dnsmasq-nanny container in the kube-dns pod. The vulnerability has been assigned CVE-2017-14491, and is described further here.

We have released version 1.14.5 of all kube-dns containers to gcr.io/google_containers/k8s-dns-*, which includes k8s-dns-dnsmasq-nanny:1.14.5, fixing the vulnerability.

The kube-dns version string has been updated to 1.14.5 in kubernetes/cluster/addons/dns and kubernetes/cmd/kubeadm/app/phases/addons/dns/versions.go.

We have released Kubernetes versions 1.5.8, 1.6.11, and 1.7.7 containing the updated version string. Kubernetes version 1.8.0 also includes the updated kube-dns version.

Who is affected?

All Kubernetes clusters running k8s-dns-dnsmasq-nanny:1.14.4 and earlier are affected.

To check which version of kube-dns is running in your cluster, run:

kubectl get deployment -n kube-system kube-dns -o jsonpath='{.spec.template.spec.containers[?(@.name == "dnsmasq")].image}'

If you run with the default kube-dns installation, we recommend you upgrade your Kubernetes master to a patched version as soon as possible, which will update your kube-dns installation to the patched version.

If you manage a non-default installation of kube-dns, we recommend you update your kube-dns deployment to the patched version.

What is the impact?

This vulnerability is potentially remotely-exploitable. See the post on the Google Security Blog for more details.

Thank You,

CJ Cullen

On-behalf of the Kubernetes Product Security Team

[Brandon Philips]

Instructions for CoreOS Tectonic users on this post: DNS security vulnerability patched in Kubernetes and Tectonic.

If you have specific upgrade instructions for a setup that you maintain please reply and post for users to find.

Brandon

[CJ Cullen]

All GKE clusters have already been patched, so no action is necessary for GKE users.

Users will see a "-gke.1" version for the MASTER_VERSION of their GKE clusters, which contains the patch.

On Monday, October 2, 2017 at 6:15:36 AM UTC-7, CJ Cullen wrote:

[Chris Love]

Do we have a backward compatibility list?  In other words, I am running k8s 1.7.2 in production, do I need to update to the latest 1.7.x release to guarantee compatibility?  I am guessing that we are we are good 1.5.x and up with running kube-dns 1.14.5 but what regression tests have been run?

Thanks

[Varsha]

We are running our production in 1.5.5 version Kubernetes. Is there a patch available for this? Or how should I fix for this secuirty vulnerablity.
Our DNSMasq docker images shows up as gcr.io/google_containers/kube-dnsmasq-amd64:1.4

[Tim Hockin]

1.5.8 is the patched version of the 1.5 series.  You should probably upgrade to that, though you might want to consider advancing to a more recent series soon, since 1.8.0 is out now.

--
You received this message because you are subscribed to the Google Groups "Kubernetes developer/contributor discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-dev+unsubscribe@googlegroups.com.
To post to this group, send email to kubernetes-dev@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-dev/934b9e63-9b61-4055-be60-1913e4e02eac%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Varsha]

Thanks for the reply. We are deployed in AWS and I read posts saying the kube-push is broken. Can you tell me how I can upgrade the nodes/ master with 1.5.8 patch or upgrade from 1.5.5 to stable 1.7 or 1.8 release.

To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-de...@googlegroups.com.
To post to this group, send email to kuberne...@googlegroups.com.

[Justin Santa Barbara]

kops is the upgrade path I recommend for AWS kube-up clusters: https://github.com/kubernetes/kops/blob/master/docs/upgrade_from_kubeup.md

Given the likely renewed interest in this I'll be revalidating this upgrade over the next few days, so you may want to hold off on a full cluster upgrade for a few days.  The kops community is collating a repsonse, including a hotfix: https://github.com/kubernetes/kops/issues/3512

We believe it suffices simply to patch the image in your kube-dns deployment (can someone from the security team please confirm):

kubectl set image deployment/kube-dns -n kube-system dnsmasq=gcr.io/google_containers/k8s-dns-dnsmasq-amd64:1.14.5

And that this can be verified using:

kubectl get deployment -n kube-system kube-dns -o jsonpath='{.spec.template.spec.containers[?(@.name == "dnsmasq")].image}'




Justin

[Varsha]

Hi Justin,

Thanks for the reply. Will try out updating the kube-dns part alone.

[Aaron Levy]

(disclaimer: not on the security team):

For clarity, I believe it's slightly different commands depending on version:

v1.5.x:

$ kubectl set image deployment/kube-dns dnsmasq=gcr.io/google_containers/k8s-dns-dnsmasq-amd64:1.14.5 --namespace=kube-system

v1.6.x/v1.7.x:

$ kubectl set image deployment/kube-dns dnsmasq=gcr.io/google_containers/k8s-dns-dnsmasq-nanny-amd64:1.14.5 kubedns=gcr.io/google_containers/k8s-dns-kube-dns-amd64:1.14.5 sidecar=gcr.io/google_containers/k8s-dns-sidecar-amd64:1.14.5 --namespace=kube-system

To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-dev+unsubscribe@googlegroups.com.
To post to this group, send email to kubernetes-dev@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-dev/acb92ad6-8e7a-483a-a616-7a6a94ca7bab%40googlegroups.com.

[Chris Love]

Can someone on the security team please validate Aaron's comment. In 1.5.x which containers need to be updated with which versions?

Does anyone have an answer about testing? Have we once done burn in on the latest released versions?

Next time we come out with CVE can we work out a instructions for ahotfix? Happy to help out with this process, but support matrix and instructions are key. Have a lot of users asking those exact questions.

Aaron thanks my friend, will updates kops' open issue.

Chris

You received this message because you are subscribed to a topic in the Google Groups "Kubernetes developer/contributor discussion" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/kubernetes-dev/QWIzhD3JhhE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to kubernetes-de...@googlegroups.com.

To post to this group, send email to kuberne...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-dev/CAGuNsh6CjweD1XhvHvjQ8FhmKB0gKYWTDRn7G-CtsrxL%3D9KVuw%40mail.gmail.com.

[Mike Splain]

Agreed to Chris' comments, it'd be really helpful for operators if there were more details on hotfixing at release time.  Many kops users have a relatively easy upgrade path for 1.6 and 1.7 but 1.5 appears harder but we're trying to work through it. 

Thanks for everyones hard work on this.

Mike

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-dev/acb92ad6-8e7a-483a-a616-7a6a94ca7bab%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to a topic in the Google Groups "Kubernetes developer/contributor discussion" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/kubernetes-dev/QWIzhD3JhhE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to kubernetes-de...@googlegroups.com.

To post to this group, send email to kuberne...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-dev/CAGuNsh6CjweD1XhvHvjQ8FhmKB0gKYWTDRn7G-CtsrxL%3D9KVuw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "Kubernetes developer/contributor discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-de...@googlegroups.com.
To post to this group, send email to kuberne...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-dev/FD711AC9-AAB1-4686-9F68-94CBD5D7FE96%40cnmconsulting.net.

[Roberto Rojas]

@Aaron,

    

 Thanks for providing the commands. Very helpful! 

In my case, for a v1.6.2 cluster I have,  the v1.6.x command complained that it was unable to find the sidecar container and the status stayed on 'CrashLoopBackOff'

The command that did work for my v1.6.2 was:

kubectl set image deployment/kube-dns dnsmasq=gcr.io/google_containers/k8s-dns-dnsmasq-nanny-amd64:1.14.5 --namespace=kube-system

I tried the v1.6.x/v1.7.x command and that worked on a v1.7.5 cluster.

[Canh Ngo]

@Aaron: thank you. Your command upgrades my k8s 1.7 cluster without any problem.

[Chris Love]

FYI - We have opened an issue in kops tracking patching this CVE.  Users have encountered issues upgrading the container when running k8s 1.5 and 1.4.  Installations of 1.4 and 1.5 often did not have the kube-dns confimap that is required with kube-dns 1.14.5.  See here for more information: https://github.com/kubernetes/kops/issues/3512

On Monday, October 2, 2017 at 7:15:36 AM UTC-6, CJ Cullen wrote:

[CJ Cullen]

Hi all,

I've put together a draft retrospective on how we handled the disclosure/announcement of this CVE. Right now it is a google doc, but I could also send it out as a PR in the community repo if that would be easier. The purpose of this retrospective is to improve our Security Release Process, so please leave any comments you think will help that process get better.

Thanks,

-CJ