Topology Aware Scheduling: NodeResourceTopology CRD API definition

[Swati Sehgal]

Hi All,

Yesterday (27th October 2020) we had a discussion in SIG Node to gain consensus on the the location of the NodeResourceTopology CRD API, the API that is needed by the following two proposed components:

1. Topology exporter daemon which is to be implemented as part of Node feature Discovery (NFD)
2. Topology aware scheduler plugin proposed as an in-tree plugin as part of kube-scheduler

The problem statement

Both the Topology aware scheduler plugin and Node Feature Discovery need to import the CRD API definition (clientset, informers, etc). The question is, where to locate this API?

Possible Options:

1. In Node feature Discovery: This cannot be done as it leads to circular dependency between NFD and k8s (NFD imports K8s and scheduler plugin in kube-scheduler needs to import NFD to access the CRD API).
2. In an external repository ( e.g. kubernetes-sigs): Similar challenge as above as the external repo defining the CRD API will need to import k8s and  (topology aware scheduler plugin in) k8s will need the CRD.
3. In staging: This is the only viable option currently so we propose having the NodeResourceTopology CRD API definition be under staging at path staging/src/k8s.io/api/node in k/k.

Recommended Solution/Discussion in SIG Node:  As per the discussion in SIG node, we would place the API under staging/src/k8s.io/api/node/noderesourcetopologyapi. If the mechanism of supporting CRDs changes in the future we would adapt and make updates based on that.

In addition this, we will include a code-generation script for maintainability which will be placed in the k/k hack directory.

The following two KEPs have been updated based on the above decision and are ready for a final round of review. We are targeting these for kubernetes 1.21.

• KEP: Topology aware resource provisioning daemon #1870:
• KEP: Simplified version of topology manager in kube-scheduler #1858 

Regards
Swati Sehgal

[lig...@google.com]

> we would place the API under staging/src/k8s.io/api/node/noderesourcetopologyapi

staging/src/k8s.io/api/node is already in use for node.k8s.io APIs.

Definitions for a new API group should get their own folder following the k8s.io/api/<group-alias>/<version> structure, e.g. staging/src/k8s.io/api/topology

Are you proposing having just the go types live there, or also the generated CRD manifest?

[Swati Sehgal]

On Wed, Oct 28, 2020 at 3:02 PM 'lig...@google.com' via kubernetes-sig-node <kubernete...@googlegroups.com> wrote:

staging/src/k8s.io/api/node is already in use for node.k8s.io APIs.

Definitions for a new API group should get their own folder following the k8s.io/api/<group-alias>/<version> structure, e.g. staging/src/k8s.io/api/topology

Since NodeResourceTopology is a CRD, an instance of which would store resource topology information of a node in the cluster, we thought it would be appropriate to place it under staging/src/k8s.io/api/node but thanks for making us aware of the conventions. We noticed the metric API under staging/src/k8s.io/metrics here and would like to clarify if it would be appropriate to have it in staging/src/k8s.io/api/noderesourcetopology-api  or in staging/src/k8s.io/noderesourcestopology-api?

 

Are you proposing having just the go types live there, or also the generated CRD manifest?

We are proposing API definition in terms of go types along with the code generated by code-generator i.e. the clientset, informers and listeners. Please take a look here for an example. Here we also have some CRD manifests for ease of use of the API as well.

Regards

Swati

[Alexey Perevalov]

@dims and @nikhita hi! I have a question about root cause of the problem with circular dependency, the code we want to use in kube-scheduler plugin and in NodeFeatureDiscovery is a CRD types.go/clientset/informer/lister
It has usual for such project dependencies:
       k8s.io/api v0.18.4
k8s.io/apimachinery v0.18.4
k8s.io/client-go v11.0.0+incompatible // indirect
k8s.io/code-generator v0.18.4
k8s.io/gengo v0.0.0-20200428234225-8167cfdcfc14 // indirect
k8s.io/klog v1.0.0 // indirect
k8s.io/kube-openapi v0.0.0-20200427153329-656914f816f9 // indirect
k8s.io/sample-controller v0.18.4 // indirect
k8s.io/utils v0.0.0-20200603063816-c1c6865ac451 // indirectthe projects we can now find in staging/vendor which has separate repo like https://github.com/kubernetes-sigs/apiserver-network-proxy, https://github.com/kubernetes/metrics they also depends on k8s.io/apimachinery, k8s.io/client-go, k8s.io/klog.

ср, 28 окт. 2020 г. в 20:31, Swati Sehgal <swse...@redhat.com>:

--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-node" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-node/CALXDz50bGvuOg0dkOyGWv1eLpKJohWaU_Mt9LVoD8JpuxHPNCA%40mail.gmail.com.

[Swati Sehgal]

Thanks Alexey for sharing  your findings with us.

Based on Alexey's findings it appears that external repos like https://github.com/kubernetes-sigs/apiserver-network-proxy, https://github.com/kubernetes/metrics are mirrored in kubernetes staging. These repos depend on k8s.io/apimachinery, k8s.io/client-go, k8s.io/klog etc. just like the repo we are proposing.

To conform to the way these repos are being used in kubernetes currently, looks like we should setup an external repo https://github.com/kubernetes-sigs/noderesourcestopology-api ( which is used for  CRD API definition and generated clientset, informers and listeners) and mirror that in staging/src/k8s.io/noderesourcestopology-api.  What does the sig-node community think about this?

If there is consensus on this, we would like to start the process of setting up a separate repo in kubernetes-sigs.

Feedback on this would be really appreciated,

Thanks,

Kind Regards

Swati

[Jordan Liggitt]

https://github.com/kubernetes/metrics is not vendored into k8s.io/kubernetes, it is published from k8s.io/kubernetes. The canonical source is https://github.com/kubernetes/kubernetes/tree/master/staging/src/k8s.io/metrics

The only reason bits of https://github.com/kubernetes-sigs/apiserver-network-proxy can be vendored into k8s.io/kubernetes is because it defines a submodule (https://github.com/kubernetes-sigs/apiserver-network-proxy/tree/master/konnectivity-client) that has no dependencies on anything in k8s.io/kubernetes/staging. Critically, k8s.io/kubernetes only vendors that submodule (https://github.com/kubernetes/kubernetes/blob/release-1.19/go.mod#L498)

[Swati Sehgal]

Thanks Jordan for the detailed information,

This certainly throws light on dependency management in kubernetes.

I had an offline conversation with Dims to get more clarity on the process that we need to follow and clarified some of the misconceptions. In order to go ahead we need approval from the SIGs to request for repo creation in staging/src/k8s.io/noderesourcestopology-api.

Based on the conversation we had in the last SIG node meeting, can we go ahead with the CRD API repo in staging? 

If everyone is onboard we will create an issue requesting a repo creation in kubernetes/org followed by a PR in k/k.

Thanks,

Kind Regards

Swati

[Davanum Srinivas]

+1 from me!

You received this message because you are subscribed to the Google Groups "kubernetes-sig-scheduling" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-sch...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-scheduling/CALXDz52LUUu0kaeigqUiHgKa1w21EBNodhOogqSA7p-ZHigYqA%40mail.gmail.com.

--

Davanum Srinivas :: https://twitter.com/dims

[Wei Huang]

+1 from me for placing NodeResourceTopology API into staging, and hence build in-tree scheduling plugin based on that API.

[Dawn Chen]

+1 for placing NodeResourceTopology API into staging as a temporary solution before we figure out an alternative path. 

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-node/23dc2d71-021a-4eb1-9e39-eba5d3d9b853n%40googlegroups.com.

[Swati Sehgal]

Thanks everyone for feedback and +1s on this!

We have requested for a repo to be created in staging/src/k8s.io/noderesourcestopology-api here and have a repo created https://github.com/kubernetes/noderesourcetopology-api now. Thanks Dims and Nikhita for helping out in this process!

 The following two PRs need to be reviewed to complete this process:

1. kubernetes/test-infra#19814 - for branch protection rules
2. kubernetes/community#5290 - to add the repo to sigs.yaml

Also, the Topology aware scheduling KEPs have been updated now and are ready for a final round of review, we are targeting these KEPs for 1.21 release.

• KEP: Topology aware resource provisioning daemon #1870:
• KEP: Simplified version of topology manager in kube-scheduler #1858 

Thanks

Kind Regards

Swati Sehgal

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-node/CAJo%3DGk7ctXCw4jmBmBDJ6ZevPZeh_eXC%2B-UH5EcRRYeDPEPEQw%40mail.gmail.com.