KEP: Class-based resources

[Lehtonen, Markus]

Hi folks,

 

I'd like to propose to add support for class-based resources in Kubernetes. Such resources would have a fixed set of classes to which containers (or Pods) can be assigned to. An existing analogy in K8s could be Pod QoS: we have three classes, Guaranteed, Burstable and BestEffort which containers belong to. In this proposal these resources would be virtually opaque to Kubernetes/kubelet and configuration and management of them would be handled by the container runtime.

 

The initial motivation for this was Intel RDT (Resource Director Technology) which is a class-based mechanism for controlling the cache and memory bandwidth allocation. RDT has been supported (for Linux) in OCI runtime spec quite a while and we recently implemented support for that in CRI-O and containerd runtimes. We also implemented a class-based approach for configuring the blockio cgroup controller that would fit in this scheme. In the future this could be used e.g. for network qos or memory type prioritization.

 

I have already created KEP-3008 that describes everything in more detail. This KEP now takes a minimalistic approach and captures what I think of as a "minimum viable implementation", with future steps (KEPs) laid out to enhance and complete the functionality.

 

I'd like to hear your thoughts about this.

 

Thanks,

  Markus

[Peter Hunt]

+1 I'd like to move from the annotations based approach to a more kubernetes-first approach. Does anyone have any reservations about moving forward with this KEP for 1.25?

[Lehtonen, Markus]

Thanks for the feedback and questions on this KEP in the SIG Node Meeting (April 12^th). There was good discussion but also some misconceptions.

 

First, probably the naming is somewhat confusing and overloaded. For example, the goal is not try to implement anything like the Resource Classes a few years back.

 

In short, I think class resources (as in this KEP) could be described as "opaque non-integer resources". They could be thought as extended resources for non-accountable resource types. Opaque in the sense (like the existing extended resources) that basically all management would be done by the container runtime – Kubernetes would not handle the discovery or configuration of the resources, for example. Non-accountable as they would enable QoS type resources that cannot be managed via existing mechanisms.

 

There was also some misunderstanding from my side regarding PodSpec changes vs. Pod annotations and I didn't respond properly. In the meeting there was a suggestion to nix PodSpec changes at this point but use Pod annotations as the "K8s user API" at this point. An approach that was used for seccomp support, for example. Annotations is mentioned under "Alternatives" in the KEP. We actually had annotations as the main proposal in the first draft of the KEP. However, when I presented that in SIG Node (October 19^th 2021) the feedback was "annotations should generally be avoided for new KEPs as they are very difficult to manage with version skew, use alpha fields instead" that (copy-paste from the meeting notes). Thus, the KEP was modified to include PodSpec, too.

 

Cheers,

  Markus

--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-node" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-node/B961D48D-13D0-4DF3-9CEF-2E80B695AF0D%40intel.com.