Thoughts on separating a CRI for docker from Kubelet

[Davanum Srinivas]

Folks,

The recent runc container breakout vulnerability (CVE-2019-5736 [1]) sparked a discussion in WG-LTS[1] which led to exploring the option of separating out docker CRI into a separate binary. 

Please see issue [1] about a long standing TODO and a prototype for a new cri-dockerd binary [2]. 

The pros are

- cri-dockerd can be maintained independently

- over time we can remove vendored docker dependendencies in kubelet

- docker is not special and just a CRI just like every other CRI in our ecosystem

The cons are

- migration pain with a new binary in addition to kubelet

- updating all the eco-system tools to support the new binary

WDYT? what other pros and cons do you see? 

Thanks,

Dims

[1] https://www.openwall.com/lists/oss-security/2019/02/11/2

[2] https://kubernetes.slack.com/archives/CDTMPM75W/p1550073633009200

[3] https://github.com/kubernetes/kubernetes/issues/73933

[4] https://github.com/kubernetes/kubernetes/pull/74051

--

Davanum Srinivas :: https://twitter.com/dims

[Eric Ernst]

TL;DR: sgtm

CRI has been out for a while, so at the very least, I don't think it makes sense to have it built-in at this point.  

More controversially (sorry), with the maturity of other CRI implementations like containerd and CRI-O, I've wondered and questioned why dockershim is still default (in particular since it won't support other runtimes aside from runc).

I understand the pain points, but perhaps by starting the process now we can make this feasible in a near-future release?  Many users already need to use a new binary in addition to kubelet already (containerd/crio).

-Eric

Aside from that, there’s no reason to have dockershim in Kubelet; CRI has been out for a while now...

--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-node" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-...@googlegroups.com.
To post to this group, send email to kubernete...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-node/CANw6fcFKtY5u5gJfXW3vJ11Vr9vuUSZwbZWY7US9KSiskG19sA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[Davanum Srinivas]

Agree Eric.

The other angle to this is ... Do we really need yet another CRI? given we have a bunch of them already. 

How about we just remove the docker shim and adopt one of the existing CRI implementations for use in upstream CI?

If we want to do that, then How do we make people comfortable with the fact that they may not have docker at all in their kubernetes environments? 

Thanks,

Dims

[Brendan Burns]

One thing to consider is that many people use the built in Docker for in-cluster image build. While that may not be something we recommend for a variety of reasons, it will be a breaking change for many users.

--brendan

---

From: kubernetes-si...@googlegroups.com <kubernetes-si...@googlegroups.com> on behalf of Davanum Srinivas <dav...@gmail.com>
Sent: Saturday, February 16, 2019 6:47:46 AM
To: Eric Ernst
Cc: kubernete...@googlegroups.com; kubernetes-si...@googlegroups.com; kubernet...@googlegroups.com
Subject: Re: Thoughts on separating a CRI for docker from Kubelet

 

You received this message because you are subscribed to the Google Groups "kubernetes-sig-architecture" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-arch...@googlegroups.com.
To post to this group, send email to kubernetes-si...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-architecture/CANw6fcGcEk_fCgSo3wLBOv%2BKryBtGsWC6P6SoKZBv_dRErFFNQ%40mail.gmail.com.

[Ricardo Aravena]

I think longer term ideally there would be just one CRI, I believe that was the original rationale behind it(?)

IMO, the main advantage of having a docker-cri would be keeping docker compatibility while allowing the kubelet to eventually phase out/deprecate docker.  Tons of people are using docker in prod today. One thing to keep in mind when creating a docker-cri would be to set a timeline for phase out/deprecation and create a path for docker prod environments to be migrated to a single CRI.

So short term makes sense to create a docker-cri and long term (i.e 6-months-2years) makes sense to eventually phase out that same docker-cri.

My two cents.

Rico.

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-node/CY4PR21MB05048D89FB89800517C9380BDB610%40CY4PR21MB0504.namprd21.prod.outlook.com.

[Brian Grant]

+kubernetes-sig-cluster-lifecycle 

Hi, Dims.

In addition to thinking about pros and cons, have you lined up the developers and reviewers/approvers necessary to make this happen, or is this at the "should this be an eventual goal" stage?

How would this compare to SIG Network and SIG Storage plans for CNI and CSI plugins for in-tree/in-binary implementations that predated the plugin mechanisms?

On Fri, Feb 15, 2019 at 8:08 AM Davanum Srinivas <dav...@gmail.com> wrote:

--

You received this message because you are subscribed to the Google Groups "kubernetes-sig-architecture" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-arch...@googlegroups.com.
To post to this group, send email to kubernetes-si...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-architecture/CANw6fcFKtY5u5gJfXW3vJ11Vr9vuUSZwbZWY7US9KSiskG19sA%40mail.gmail.com.

[Davanum Srinivas]

Brian,

right, we are at the exploratory "should this be an eventual goal" stage.

Next step would be a KEP. @resouer indicated that he will help with starting one. We have the previous patterns from CNI/CSI/Cloud-provider etc to help with transitioning to look for guidance from. 

Thanks,

Dims

[Rostislav Georgiev]

Folks,

 

For several versions now, Docker ships with containerd. In theory we can use containerd’s CRI plugin to connect to it directly (instead of going through docker shim, dockerd and then reach containerd).

My proposition is to connect to containerd on newer Docker versions and keep the shim as long as we support old Docker versions.

I don’t have a strong opinion on how the shim is going to be nursed until it’s removed (could be a separate binary or remain in kubelet).

 

Pros:

• We get to remove a couple of hops until we reach containerd (which we are going to do either way) and thus reduce latency and chances for error.
• We can drop the shim earlier and even avoid moving it out of kubelet.

 

Cons:

• containerd is relatively separate and working with standard socket and config file paths since Docker 18.09.
• The CRI plugin of containerd is disabled by default, when it’s bundled with Docker. This requires one extra step by the user – to enable it in the config file and to restart the containerd service.
• Docker and the containerd CRI plugins work in different containerd namespaces. Therefore resources cannot be shared between them (“docker ps” and “crictl ps” will show different results). This may be inconvenient for some users, but it may actually be perceived as a plus by others.

 

Have a wonderful day!

--

Rostislav (Ross) M. Georgiev

VMware Open Source Technology Center

 

From: kubernetes-sig-c...@googlegroups.com <kubernetes-sig-c...@googlegroups.com> On Behalf Of Davanum Srinivas
Sent: Thursday, February 21, 2019 12:43 AM
To: Brian Grant <brian...@google.com>
Cc: kubernetes-sig-cluster-lifecycle <kubernetes-sig-c...@googlegroups.com>; kubernetes-sig-node <kubernete...@googlegroups.com>; kubernetes-sig-architecture <kubernetes-si...@googlegroups.com>; kubernet...@googlegroups.com
Subject: Re: Thoughts on separating a CRI for docker from Kubelet

 

Brian,

--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-cluster-lifecycle" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-cluster...@googlegroups.com.
To post to this group, send email to kubernetes-sig-c...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-cluster-lifecycle/CANw6fcEYZewq%3DSdRwAmx7tyhaAncnbrNiEKu1-j%2BsEyyRWs9oQ%40mail.gmail.com.

[Fox, Kevin M]

Might be interesting to get Docker Inc's view too, as they are supporting Kubernetes on Docker. If they plan on supporting Kubernetes using the containerd route, then the plan below might be a better path. If they would like it to go through dockerd still, then maybe splitting out the code to its own cri module would be best? They would have incentive to help with the migration in that case.

Thanks,
Kevin

---

From: 'Rostislav Georgiev' via kubernetes-sig-cluster-lifecycle [kubernetes-sig-c...@googlegroups.com]
Sent: Thursday, February 21, 2019 5:22 AM
To: Davanum Srinivas; Brian Grant
Cc: kubernetes-sig-cluster-lifecycle; kubernetes-sig-node; kubernetes-sig-architecture; kubernet...@googlegroups.com
Subject: RE: Thoughts on separating a CRI for docker from Kubelet

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-cluster-lifecycle/MN2PR05MB6094F5A69E63103FBFD7B47AB37E0%40MN2PR05MB6094.namprd05.prod.outlook.com.

[Deep Debroy]

From the SIG Windows perspective, I wanted to point out that in the immediate timeframe, Windows support on Kubernetes is going to be depend on dockershim => docker engine with built-in support for Windows HCS. The containerd.exe => runhcs.exe route for Windows is still maturing as certain features like GMSA support is not plumbed in through the containerd.exe => runhcs.exe path yet. 

As Windows support on Kubernetes marches towards GA, in terms of specific timelines, feature parity between dockershim => docker engine and CRI => containerd => HCS is worth considering in the context of this discussion.

Thanks,

Deep

You received this message because you are subscribed to the Google Groups "kubernetes-sig-node" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-...@googlegroups.com.
To post to this group, send email to kubernete...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-node/MN2PR05MB6094F5A69E63103FBFD7B47AB37E0%40MN2PR05MB6094.namprd05.prod.outlook.com.

[Dawn Chen]

Thanks everyone for participating in this thread !

SIG Node is going to discuss this topic at tomorrow's meeting (Tues 10:00am PST). I invited folks from Docker Inc. to attend tomorrow meeting too. In the past, the folks from Docker Inc. also express the strong interest to maintain docker-shim for Kubernetes users too. 

Please note that we are at the exploratory "should this be an eventual goal" stage, no decision and action items yet. We understand that many production users are still depending on docker-shim, and WIP Windows Containers are depending on it too. We also worked very closely with containerd / dockerd folks to come 

up a smooth transition plan for K8s users, for example, merging CRI-containerd to containerd, and having dockerd (since 18.06 release) built on the top of (cri-)containerd, etc. 

See you tomorrow !

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-node/CAMK2eKJhqJhg%2BspV7yQj0uMDEKUkaTvUfCQnG-Z9xRnz-ibr3Q%40mail.gmail.com.

[Dawn Chen]

As a follow-up, we spent half hour at Tue's SIG meeting on this topic, and the detail meeting note is at here. Thanks our note taker: Haiyan and Lantao.

Davanum, unfortunately you didn't attend the meeting. Hope the meeting notes can answer some of your questions. We also recorded the meeting and would upload them soon. 

In a summary, many productions are depending on docker-shim today even both containerd and cri-o are productionized; SIG Node would like to kick out the process to define dockershim deprecation criteria, and Harry Zhang (@resour) agreed to take the action of drafting the KEP. We plan to come back to this topic one month after the KEP was uploaded.

Thanks,

Dawn

[Davanum Srinivas]

Dawn,

Sounds great! 

Thanks,

Dims