Container runtimes and Kubernetes

[Sergey Kanzhelev]

Hi,

This is a follow up from discussions we had before in sig node and this comment: https://github.com/kubernetes/enhancements/pull/4895#discussion_r1791268767

The main question is whether we should continue promoting KEPs which have container runtime dependencies and what can be a good policy for k8s in terms of CRI APIs real-world validation. Before 2.0, we always had at least CRI-O and Containerd support, which was covering a majority of use cases and deployments. The question is whether we can assume 2.0 will happen before 1.32 and whether we should add a buffer for Containerd 2.0 adoption after it is released. I will add this as a topic for the SIG Node meeting.

Last couple of k8s releases we have operated on the assumption that Containerd 2.0 is right around the corner. However it is still not released. In fact, it has 3 items officially tracked as release blocking and 13 open items in the milestone. Moreover, once released, we will not have good feedback from production environments while people will be adopting the 2.0, which has many breaking changes.

I looked over a few recent KEPs to see where we are.

We have accumulated two KEPS in beta with only 2.0 support:

• Recursive Read-only (RRO) mounts · Issue #3857 · kubernetes/enhancements · GitHub

• https://github.com/kubernetes/enhancements/issues/4033

This release two KEPs that are targeting beta in 1.32 are:

• VolumeSource: OCI Artifact and/or Image · Issue #4639 · kubernetes/enhancements · GitHub

• Fine-grained SupplementalGroups control · Issue #3619 · kubernetes/enhancements · GitHub

Moreover we have alpha KEPs that are not supported on Containerd for a long time:

• Forensic Container Checkpointing · Issue #2008 · kubernetes/enhancements · GitHub

• Kubelet support for Split Image Filesystem · Issue #4191 · kubernetes/enhancements · GitHub

And any new KEP that will need CRI API change, will only be supported on Containerd 2.0 based on a current Containerd policy.

/Sergey

[Davanum Srinivas]

Sergey,

Thanks for bringing this up! We are on RC5 for containerd 2.0 and the last thing that needs to land is the following:
https://github.com/containerd/errdefs/pull/21

Containerd has a tradition of releasing during kubecon, so currently unless something else pops up, i'd expect the same for 2.0 as well. Regarding breaking changes, there is definitely an effort to document them, so any help there from folks here would be appreciated.

I think as long as k8s works with 1.7.x (and corresponding runc) degrading gracefully when the newer CRI api is not present, we should be ok. For future KEPs we can be more stringent during the review process itself. The alpha KEPs are definitely worrisome if there is no work going on ... on the containerd side. We should probably be more conditional on work being done in both runtimes going forward.

thanks,

Dims

--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-architecture" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-arch...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-architecture/CA%2Bsr0%2BAS%3DJau3hOxoz4OLWAxB8Mpc82N-k9RgFix4MhAMU%3DTDQ%40mail.gmail.com.

--

Davanum Srinivas :: https://twitter.com/dims

[Adrian Reber]

Just for completeness, Forensic Container Checkpointing is part of
containerd RC releases for almost half a year now. We are also waiting
for the containerd release for people to use it more easily (without
using a containerd RC release).

The container restore functionality is currently only merged in CRI-O
and the containerd PR is open for a couple of months.

Adrian

> > <https://github.com/orgs/containerd/projects/9?query=sort%3Aupdated-desc+is%3Aopen>

> > as release blocking and 13 open items in the milestone

> > <https://github.com/containerd/containerd/issues?q=is%3Aissue%20state%3Aopen%20milestone%3A2.0>.

> > Moreover, once released, we will not have good feedback from production
> > environments while people will be adopting the 2.0, which has many breaking
> > changes.
> >
> > I looked over a few recent KEPs to see where we are.
> >
> > We have accumulated two KEPS in beta with only 2.0 support:
> >

> > -

> >
> > Recursive Read-only (RRO) mounts · Issue #3857 ·
> > kubernetes/enhancements · GitHub

> > <https://github.com/kubernetes/enhancements/issues/3857>
> > -

> >
> > https://github.com/kubernetes/enhancements/issues/4033
> >
> >
> > This release two KEPs that are targeting beta in 1.32 are:
> >

> > -

> >
> > VolumeSource: OCI Artifact and/or Image · Issue #4639 ·
> > kubernetes/enhancements · GitHub

> > <https://github.com/kubernetes/enhancements/issues/4639>
> > -

> >
> > Fine-grained SupplementalGroups control · Issue #3619 ·
> > kubernetes/enhancements · GitHub

> > <https://github.com/kubernetes/enhancements/issues/3619>

> >
> >
> > Moreover we have alpha KEPs that are not supported on Containerd for a
> > long time:
> >

> > -

> >
> > Forensic Container Checkpointing · Issue #2008 ·
> > kubernetes/enhancements · GitHub

> > <https://github.com/kubernetes/enhancements/issues/2008>
> >
> >
> > -

> >
> > Kubelet support for Split Image Filesystem · Issue #4191 ·
> > kubernetes/enhancements · GitHub

> > <https://github.com/kubernetes/enhancements/issues/4191>

[Peter Hunt]

Thanks for starting this discussion, I'm looking forward to chatting today.

CRI stats is another KEP that has languished from slow containerd support. It would be genuinely useful for CRI-O. My ideal is we move towards a world where the kubelet is resilient to CRI not supporting things

I think now that we have both the RuntimeHandlerFeatures and RuntimeFeatures in the CRI spec, we can decouple kubernetes features from runtime support. We can have features on by default in kube that aren't actually usable and the kubelet can gracefully detect and report that. We even could have example scheduler plugins that read these values from the node object and fail to schedule them.

--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-node" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-node/ZwUafimkrnnpMpak%40dcbz.redhat.com.

[Sergey Kanzhelev]

Hi,

Picking up this thread before the 1.33 release opened, here is the document we discussed yesterday at SIG Node that describes how we want to approach feature development that requires container runtime changes: https://docs.google.com/document/d/1y42XrUPrm-6DZby1RQjexYYoNn822IRR6igsOiy_62c/edit?tab=t.0

I also added it as an agenda topic for tomorrow's SIG Architecture meeting in case there will be some in-person feedback. 

/Sergey

On Mon, Oct 14, 2024 at 4:10 AM Tim Bannister <t...@scalefactory.com> wrote:

The IETF looks for at least two independent implementations of a thing before moving any RFCs to an internet standard. The two implementations can be open or proprietary but shouldn't have a common codebase / heritage.

I think that's a good approach; we shouldn't specify what the names of those implementations are or how popular they are.

When a new or changed behavior depends on an external component, we shouldn't enable the new thing by default until there are two external implementations that have their support for it available in a public release.

We should also be cautious about moving to beta, but the crucial point is about when we change default behavior.

I like the idea of bumping the tooling default (eg, what you get from kubeadm) a minor release ahead of changing the component default (eg, how kube-apiserver behaves unless overridden). We should decide which of those 2 promotion points is gated on having at least n external implementations.

Tim Bannister

Senior lead consultant

The Scale Factory

On Sunday 13 October 2024 at 23:19:30 UTC+1 tho...@google.com wrote:

For things that manifest as APIs in k/k and implementations elsewhere,
we DEPEND on the implementations to provide feedback. I think
"adoption" here means "by implementors" not end users.

--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-architecture" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-arch...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-architecture/85892617-76b5-4060-9862-3088d59979f2n%40googlegroups.com.

[Sergey Kanzhelev]

Hi,

The document is now a PR: https://github.com/kubernetes/website/pull/49455 

/Sergey