What is a correct way to add a plugin argument struct in out of tree scheduler-plugins ?

415 views
Skip to first unread message

Seong Moon

unread,
Dec 18, 2020, 3:52:39 AM12/18/20
to kubernetes-sig-scheduling
Hi There !

I am using the out-of-tree scheduler-plugins source tree to add my custom scheduler plugin.
It implemented my PreFilter, Filter and Score function, and build the my-scheduler images successfully.
It works fine as what I want to do.

Finally, I'd like to add myPluginArgs for my custom plugin, which will be used to pass some customized configuration information to my scheduler plugin.

So, I added myPluginArgs struct and registering codes to
- pkg/apis/config/register.go
- pkg/apis/config/types.go
- pkg/apis/config/v1beta1/register.go
- pkg/apis/config/v1beta1/types.go
- pkg/myplugin/myplugin.go <-- of course, this code uses myPluginArgs.

and, I run './hack/update-codegen.sh'  and run 'make'
then, I could make my-scheduler image successfully.

But, When I run my-scheduler image, I got this error.
couldn't create scheduler using provider "DefaultProvider": initializing profiles: creating profile for scheduler name my-scheduler: error initializing plugin "myPluginName": want args to be of type myPluginNameArgs, got *runtime.Unknown%!

Would u let me know how I can add myPluginArgs related code correctly ?
I'd like to know exact procedure for adding myPluginArgs in out-of-tree scheduler source tree.

Thanks in advance.

Adhityaa Chandrasekar

unread,
Dec 21, 2020, 3:24:27 PM12/21/20
to Seong Moon, kubernetes-sig-scheduling
It's difficult to say exactly what's going wrong without seeing the code changes, but you can follow the doc we have on adding args to scheduler plugins: https://github.com/kubernetes/kubernetes/tree/master/pkg/scheduler/framework/plugins#adding-plugin-configuration-parameters-through-kubeschedulerconfiguration

If you're looking for an example PR adding args to a scheduler plugin, see https://github.com/kubernetes/kubernetes/pull/96202

--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-scheduling" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-sch...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-scheduling/7fe54fed-ccf9-4abb-a8c9-3890070759bbn%40googlegroups.com.

Aldo Culquicondor

unread,
Dec 22, 2020, 9:42:22 AM12/22/20
to Adhityaa Chandrasekar, Seong Moon, kubernetes-sig-scheduling
It sounds like you might have missed registering your plugin args type:


Aldo


To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-scheduling/CABWScxJpQfT9KybB-5tMWSGJm4DYhq4bzGFFB0BGmv0FGN%3DKiQ%40mail.gmail.com.

Seong Moon

unread,
Dec 23, 2020, 4:26:38 AM12/23/20
to kubernetes-sig-scheduling
Hi. 
I still can't find the way to add PluginArgs in my plugin.
I absolutely need the community help.

I did all things what I understand from the following guides.
But my-scheduler still occurs error as belows :
bandwidth_fit.go:117] NewBandwidthFit called
couldn't create scheduler using provider "DefaultProvider": initializing profiles: creating profile for scheduler name my-scheduler: error initializing plugin "NodeBandwidthFit": want args to be of type BandwidthFitArgs, got *runtime.Unknown%!(EXTRA bool=false)

I attached scheduler-config.yaml and my code and diffs from the out-of-tree scheduler.
Would you let me know how I can add PluginArgs for my plugin ?

My plugin's source tree is :
I attached the bandwidth_fit.go file on the last section.

My scheduler-config.yaml file is :
=======================================
kind: KubeSchedulerConfiguration
leaderElection:
  leaderElect: false
clientConnection:
  kubeconfig: "/etc/kubernetes/scheduler.conf"
profiles:
- schedulerName: my-scheduler
  plugins:
    preFilter:
      enabled:
      - name: NodeBandwidthFit
    filter:
      enabled:
      - name: NodeBandwidthFit
    score:
      enabled:
      - name: NodeBandwidthFit
  pluginConfig:
  - name: NodeBandwidthFit
    args:
      nodeBandwidth: "1Gi"
      reservedBandwidthForSystem: "100Mi"
=======================================


Belows are diffs of source codes.

===================================================================
--- cmd/scheduler/main.go       (revision 248)
+++ cmd/scheduler/main.go       (working copy)
@@ -29,6 +29,7 @@

        // Ensure scheme package is initialized.
@@ -47,7 +48,8 @@
                app.WithPlugin(crossnodepreemption.Name, crossnodepreemption.New),
                app.WithPlugin(podstate.Name, podstate.New),
                app.WithPlugin(qos.Name, qos.New),
-               app.WithPlugin(noderesources.BandwidthFitName, noderesources.NewBandwidthFit),
+               app.WithPlugin(bandwidth.BandwidthFitName, bandwidth.NewBandwidthFit),
+               //app.WithPlugin(noderesources.BandwidthFitName, noderesources.NewBandwidthFit),
        )
===================================================================
--- pkg/apis/config/register.go (revision 248)
+++ pkg/apis/config/register.go (working copy)
@@ -40,6 +40,7 @@
                &CoschedulingArgs{},
                &NodeResourcesAllocatableArgs{},
                &CapacitySchedulingArgs{},
+               &BandwidthFitArgs{},
        )
        return nil
 }
===================================================================
--- pkg/apis/config/types.go    (revision 248)
+++ pkg/apis/config/types.go    (working copy)
@@ -73,3 +73,13 @@
        // KubeConfigPath is the path of kubeconfig.
        KubeConfigPath string
 }
+
+// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object
+
+// BandwidthFitArgs defines the scheduling parameters for BandwidthFit plugin.
+type BandwidthFitArgs struct {
+       metav1.TypeMeta
+
+       NodeBandwidth string `json:"nodeBandwidth,omitempty"`
+       ReservedBandwidthForSystem string `json:"reservedBandwidthForSystem,omitempty"`
+}
===================================================================
--- pkg/apis/config/v1beta1/defaults.go (revision 248)
+++ pkg/apis/config/v1beta1/defaults.go (working copy)
@@ -38,6 +38,9 @@
        }

        defaultKubeConfigPath string = "/etc/kubernetes/scheduler.conf"
+
+       defaultNodeBandwidth string ="1Gi"
+       defaultReservedBandwidthForSystem string="100Mi"
 )

 // SetDefaultsCoschedulingArgs sets the default parameters for Coscheduling plugin.
@@ -69,3 +72,13 @@
                obj.KubeConfigPath = &defaultKubeConfigPath
        }
 }
+
+// SetDefaultsBandwidthFitArgs sets the default parameters for BandwidthFit plugin.
+func SetDefaultsBandwidthFitArgs(obj *BandwidthFitArgs) {
+       if obj.NodeBandwidth == nil {
+               obj.NodeBandwidth = &defaultNodeBandwidth
+       }
+       if obj.ReservedBandwidthForSystem == nil {
+               obj.ReservedBandwidthForSystem = &defaultReservedBandwidthForSystem
+       }
+}
===================================================================
--- pkg/apis/config/v1beta1/register.go (revision 248)
+++ pkg/apis/config/v1beta1/register.go (working copy)
@@ -40,6 +40,7 @@
                &CoschedulingArgs{},
                &NodeResourcesAllocatableArgs{},
                &CapacitySchedulingArgs{},
+               &BandwidthFitArgs{},
        )
        return nil
 }
===================================================================
--- pkg/apis/config/v1beta1/types.go    (revision 248)
+++ pkg/apis/config/v1beta1/types.go    (working copy)
@@ -73,3 +73,13 @@
        // KubeConfigPath is the path of kubeconfig.
        KubeConfigPath *string `json:"kubeConfigPath,omitempty"`
 }
+
+// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object
+
+// BandwidthFitArgs defines the scheduling parameters for NodeBandwidthFit plugin.
+type BandwidthFitArgs struct {
+       metav1.TypeMeta `json:",inline"`
+
+       NodeBandwidth *string `json:"nodeBandwidth,omitempty"`
+       ReservedBandwidthForSystem *string `json:"reservedBandwidthForSystem,omitempty"`
+}
===================================================================
--- pkg/apis/config/v1beta1/zz_generated.conversion.go  (revision 248)
+++ pkg/apis/config/v1beta1/zz_generated.conversion.go  (working copy)
@@ -37,6 +37,16 @@
 // RegisterConversions adds conversion functions to the given scheme.
 // Public to allow building arbitrary schemes.
 func RegisterConversions(s *runtime.Scheme) error {
+       if err := s.AddGeneratedConversionFunc((*BandwidthFitArgs)(nil), (*config.BandwidthFitArgs)(nil), func(a, b interface{}, scope conversion.Scope) error {
+               return Convert_v1beta1_BandwidthFitArgs_To_config_BandwidthFitArgs(a.(*BandwidthFitArgs), b.(*config.BandwidthFitArgs), scope)
+       }); err != nil {
+               return err
+       }
+       if err := s.AddGeneratedConversionFunc((*config.BandwidthFitArgs)(nil), (*BandwidthFitArgs)(nil), func(a, b interface{}, scope conversion.Scope) error {
+               return Convert_config_BandwidthFitArgs_To_v1beta1_BandwidthFitArgs(a.(*config.BandwidthFitArgs), b.(*BandwidthFitArgs), scope)
+       }); err != nil {
+               return err
+       }
        if err := s.AddGeneratedConversionFunc((*CapacitySchedulingArgs)(nil), (*config.CapacitySchedulingArgs)(nil), func(a, b interface{}, scope conversion.Scope) error {
                return Convert_v1beta1_CapacitySchedulingArgs_To_config_CapacitySchedulingArgs(a.(*CapacitySchedulingArgs), b.(*config.CapacitySchedulingArgs), scope)
        }); err != nil {
@@ -70,6 +80,36 @@
        return nil
 }

+func autoConvert_v1beta1_BandwidthFitArgs_To_config_BandwidthFitArgs(in *BandwidthFitArgs, out *config.BandwidthFitArgs, s conversion.Scope) error {
+       if err := v1.Convert_Pointer_string_To_string(&in.NodeBandwidth, &out.NodeBandwidth, s); err != nil {
+               return err
+       }
+       if err := v1.Convert_Pointer_string_To_string(&in.ReservedBandwidthForSystem, &out.ReservedBandwidthForSystem, s); err != nil {
+               return err
+       }
+       return nil
+}
+
+// Convert_v1beta1_BandwidthFitArgs_To_config_BandwidthFitArgs is an autogenerated conversion function.
+func Convert_v1beta1_BandwidthFitArgs_To_config_BandwidthFitArgs(in *BandwidthFitArgs, out *config.BandwidthFitArgs, s conversion.Scope) error {
+       return autoConvert_v1beta1_BandwidthFitArgs_To_config_BandwidthFitArgs(in, out, s)
+}
+
+func autoConvert_config_BandwidthFitArgs_To_v1beta1_BandwidthFitArgs(in *config.BandwidthFitArgs, out *BandwidthFitArgs, s conversion.Scope) error {
+       if err := v1.Convert_string_To_Pointer_string(&in.NodeBandwidth, &out.NodeBandwidth, s); err != nil {
+               return err
+       }
+       if err := v1.Convert_string_To_Pointer_string(&in.ReservedBandwidthForSystem, &out.ReservedBandwidthForSystem, s); err != nil {
+               return err
+       }
+       return nil
+}
+
+// Convert_config_BandwidthFitArgs_To_v1beta1_BandwidthFitArgs is an autogenerated conversion function.
+func Convert_config_BandwidthFitArgs_To_v1beta1_BandwidthFitArgs(in *config.BandwidthFitArgs, out *BandwidthFitArgs, s conversion.Scope) error {
+       return autoConvert_config_BandwidthFitArgs_To_v1beta1_BandwidthFitArgs(in, out, s)
+}
+
 func autoConvert_v1beta1_CapacitySchedulingArgs_To_config_CapacitySchedulingArgs(in *CapacitySchedulingArgs, out *config.CapacitySchedulingArgs, s conversion.Scope) error {
        if err := v1.Convert_Pointer_string_To_string(&in.KubeConfigPath, &out.KubeConfigPath, s); err != nil {
                return err
===================================================================
--- pkg/apis/config/v1beta1/zz_generated.deepcopy.go    (revision 248)
+++ pkg/apis/config/v1beta1/zz_generated.deepcopy.go    (working copy)
@@ -26,6 +26,41 @@
 )

 // DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *BandwidthFitArgs) DeepCopyInto(out *BandwidthFitArgs) {
+       *out = *in
+       out.TypeMeta = in.TypeMeta
+       if in.NodeBandwidth != nil {
+               in, out := &in.NodeBandwidth, &out.NodeBandwidth
+               *out = new(string)
+               **out = **in
+       }
+       if in.ReservedBandwidthForSystem != nil {
+               in, out := &in.ReservedBandwidthForSystem, &out.ReservedBandwidthForSystem
+               *out = new(string)
+               **out = **in
+       }
+       return
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new BandwidthFitArgs.
+func (in *BandwidthFitArgs) DeepCopy() *BandwidthFitArgs {
+       if in == nil {
+               return nil
+       }
+       out := new(BandwidthFitArgs)
+       in.DeepCopyInto(out)
+       return out
+}
+
+// DeepCopyObject is an autogenerated deepcopy function, copying the receiver, creating a new runtime.Object.
+func (in *BandwidthFitArgs) DeepCopyObject() runtime.Object {
+       if c := in.DeepCopy(); c != nil {
+               return c
+       }
+       return nil
+}
+
+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
 func (in *CapacitySchedulingArgs) DeepCopyInto(out *CapacitySchedulingArgs) {
        *out = *in
        out.TypeMeta = in.TypeMeta
@@ -69,6 +104,16 @@
                *out = new(int64)
                **out = **in
        }
+       if in.KubeMaster != nil {
+               in, out := &in.KubeMaster, &out.KubeMaster
+               *out = new(string)
+               **out = **in
+       }
+       if in.KubeConfigPath != nil {
+               in, out := &in.KubeConfigPath, &out.KubeConfigPath
+               *out = new(string)
+               **out = **in
+       }
        return
 }

Index: pkg/apis/config/v1beta1/zz_generated.defaults.go
===================================================================
--- pkg/apis/config/v1beta1/zz_generated.defaults.go    (revision 248)
+++ pkg/apis/config/v1beta1/zz_generated.defaults.go    (working copy)
@@ -28,22 +28,5 @@
 // Public to allow building arbitrary schemes.
 // All generated defaulters are covering - they call all nested defaulters.
 func RegisterDefaults(scheme *runtime.Scheme) error {
-       scheme.AddTypeDefaultingFunc(&CapacitySchedulingArgs{}, func(obj interface{}) { SetObjectDefaultsCapacitySchedulingArgs(obj.(*CapacitySchedulingArgs)) })
-       scheme.AddTypeDefaultingFunc(&CoschedulingArgs{}, func(obj interface{}) { SetObjectDefaultsCoschedulingArgs(obj.(*CoschedulingArgs)) })
-       scheme.AddTypeDefaultingFunc(&NodeResourcesAllocatableArgs{}, func(obj interface{}) {
-               SetObjectDefaultsNodeResourcesAllocatableArgs(obj.(*NodeResourcesAllocatableArgs))
-       })
        return nil
 }
-
-func SetObjectDefaultsCapacitySchedulingArgs(in *CapacitySchedulingArgs) {
-       SetDefaultsCapacitySchedulingArgs(in)
-}
-
-func SetObjectDefaultsCoschedulingArgs(in *CoschedulingArgs) {
-       SetDefaultsCoschedulingArgs(in)
-}
-
-func SetObjectDefaultsNodeResourcesAllocatableArgs(in *NodeResourcesAllocatableArgs) {
-       SetDefaultsNodeResourcesAllocatableArgs(in)
-}
Index: pkg/apis/config/zz_generated.deepcopy.go
===================================================================
--- pkg/apis/config/zz_generated.deepcopy.go    (revision 248)
+++ pkg/apis/config/zz_generated.deepcopy.go    (working copy)
@@ -26,6 +26,31 @@
 )

 // DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *BandwidthFitArgs) DeepCopyInto(out *BandwidthFitArgs) {
+       *out = *in
+       out.TypeMeta = in.TypeMeta
+       return
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new BandwidthFitArgs.
+func (in *BandwidthFitArgs) DeepCopy() *BandwidthFitArgs {
+       if in == nil {
+               return nil
+       }
+       out := new(BandwidthFitArgs)
+       in.DeepCopyInto(out)
+       return out
+}
+
+// DeepCopyObject is an autogenerated deepcopy function, copying the receiver, creating a new runtime.Object.
+func (in *BandwidthFitArgs) DeepCopyObject() runtime.Object {
+       if c := in.DeepCopy(); c != nil {
+               return c
+       }
+       return nil
+}
+
+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
 func (in *CapacitySchedulingArgs) DeepCopyInto(out *CapacitySchedulingArgs) {
        *out = *in
        out.TypeMeta = in.TypeMeta


--------------------------------------------------------------------------------
This is the bandwidth_fit.go !

root@sched-master:~/working/golang/src/sigs.k8s.io/scheduler-plugins# cat pkg/bandwidth/bandwidth_fit.go
/*
Copyright 2019 The Kubernetes Authors.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at


Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/

package bandwidth

import (
        "context"

        v1 "k8s.io/api/core/v1"
        "k8s.io/klog/v2"
        "fmt"

/*
        "strings"

        utilfeature "k8s.io/apiserver/pkg/util/feature"
*/

)

var _ framework.PreFilterPlugin = &BandwidthFit{}
var _ framework.FilterPlugin = &BandwidthFit{}
var _ framework.ScorePlugin = &BandwidthFit{}

const (
        // BandwidthFitName is the name of the plugin used in the plugin registry and configurations.
        BandwidthFitName = "NodeBandwidthFit"
        BandwidthRequestKey = "etri.io/egress-bw-reqeust"
        BandwidthLimitKey = "etri.io/egress-bw-limit"

        DefaultNodeBandwidth int64 = 1*1024*1024*1024 // 1Gbps : Default Node Bandwidth
        DefaultReservedBandwidthForSystem int64 = 100*1024*1024 // 100Mbps

        preFilterStateKey = "Filter" + BandwidthFitName
)

// BandwidthFit is a plugin that checks if a node has sufficient egress bandwidth resources.
type BandwidthFit struct {
        handle framework.FrameworkHandle
        nodeBandwidth int64
        reservedBandwidth int64
}

// preFilterState computed at preFilter and used at Filter
// nodeAllocatedBandwidth[] computed at Filter and used at Score
type preFilterState struct {
        podName string // pod Name
        requestedBandwidth int64 // bandwidth reqeust of the pod
        qosClass PodQOSClass
        nodeAllocatedBandwidth map[string]int64 // Allocated Bandwidth sum of each node, map[key] : nodeName
}

// Clone the prefilter state
func (s *preFilterState) Clone() framework.StateData {
        return s
}

// Name returns name of the plugin. It is used in logs, etc.
func (f *BandwidthFit) Name() string {
        return BandwidthFitName
}

func validateBandwidthArgs(args *config.BandwidthFitArgs) (nodeBandwidth int64, reservedBandwidth int64, e error) {

        return 0, 0, nil

        nodeBW, err := resource.ParseQuantity(args.NodeBandwidth)
        if err != nil {
                return 0, 0, fmt.Errorf("cannot parse nodeBandwidth %v", args.NodeBandwidth)
        }

        if err := validateBandwidthIsReasonable(&nodeBW); err != nil {
                return 0, 0, fmt.Errorf("invalid range nodeBandwidth %v", nodeBW)
        }
        nodeBandwidth = int64(nodeBW.Value())

        reservedBW, err := resource.ParseQuantity(args.ReservedBandwidthForSystem)
        if err != nil {
                return 0, 0, fmt.Errorf("cannot parse ReservedBandwidthForSystem %v", args.ReservedBandwidthForSystem)
        }

        if err := validateBandwidthIsReasonable(&reservedBW); err != nil {
                return 0, 0, fmt.Errorf("invalid range ReservedBandwidthForSystem %v", reservedBW)
        }
        reservedBandwidth = int64(reservedBW.Value())

        return nodeBandwidth, reservedBandwidth, nil
}

// NewBandwidthFit initializes a new plugin and returns it.
func NewBandwidthFit(plArgs runtime.Object, h framework.FrameworkHandle) (framework.Plugin, error) {
        klog.V(2).Infof("NewBandwidthFit called")

        var nodeBandwidth int64 = 0
        var reservedBandwidth int64 = 0
        var err error

        if plArgs != nil {
                args, ok := plArgs.(*config.BandwidthFitArgs)
                if !ok {
                        return nil, fmt.Errorf("want args to be of type BandwidthFitArgs, got %T", plArgs, ok)
                }

                // validating arguments
                nodeBandwidth, reservedBandwidth, err = validateBandwidthArgs(args)
                if err != nil {
                        return nil, err
                }
        } else {
                klog.V(2).Infof("NewBandwidthFit plArgs is nil")
        }


        if nodeBandwidth == 0 { nodeBandwidth = DefaultNodeBandwidth }
        if reservedBandwidth == 0 { reservedBandwidth = DefaultReservedBandwidthForSystem }

        klog.V(2).Infof("NewBandwidthFit: nodeBW(%d), reservedBW(%d)", nodeBandwidth, reservedBandwidth)

        return &BandwidthFit{
                handle: h,
                nodeBandwidth: nodeBandwidth,
                reservedBandwidth: reservedBandwidth,
                }, nil
}

// PreFilter invoked at the prefilter extension point.
func (f *BandwidthFit) PreFilter(ctx context.Context, cycleState *framework.CycleState, pod *v1.Pod) *framework.Status {
        //klog.V(2).Infof("PreFilter called")

        state := &preFilterState{
                podName: pod.ObjectMeta.Name,
                nodeAllocatedBandwidth: make(map[string]int64),
        }

        egressRequest, egressLimit, err := ExtractPodBandwidthResources(pod.ObjectMeta.Annotations)
        if err != nil {
        return framework.NewStatus(framework.Error, fmt.Sprintf("getting pod %q bandwidth request %v", pod.ObjectMeta.Name, err))
        }

        var requestedBandwidth int64 = 0
        if egressLimit != 0 {
                requestedBandwidth = egressLimit
        } else if egressRequest != 0 {
                requestedBandwidth = egressRequest
        }
        state.requestedBandwidth = requestedBandwidth

        var qosClass PodQOSClass
        if egressRequest == 0 && egressLimit == 0 {
                qosClass = PodQOSBestEffort
        } else if egressRequest == egressLimit {
                qosClass = PodQOSGuaranteed
        } else {
                qosClass = PodQOSBurstable
        }
        state.qosClass = qosClass

        klog.V(2).Infof("PreFilter: pod %q, egressRequest: %d, egressLimit: %d, requestedBW: %d, qosClass: %q", pod.ObjectMeta.Name, egressRequest, egressLimit, requestedBandwidth, qosClass)

        cycleState.Write(preFilterStateKey, state)
        return nil
}

func getPreFilterState(cycleState *framework.CycleState) (*preFilterState, error) {
        c, err := cycleState.Read(preFilterStateKey)
        if err != nil {
                // preFilterState doesn't exist, likely PreFilter wasn't invoked.
                return nil, fmt.Errorf("error reading %q from cycleState: %v", preFilterStateKey, err)
        }

        s, ok := c.(*preFilterState)
        if !ok {
                return nil, fmt.Errorf("%+v  convert to BandwidthFit.preFilterState error", c)
        }
        return s, nil
}

// PreFilterExtensions returns prefilter extensions, pod add and remove.
func (f *BandwidthFit) PreFilterExtensions() framework.PreFilterExtensions {
        return nil
}

// Score invoked at the score extension point.
func (f *BandwidthFit) Score(ctx context.Context, cycleState *framework.CycleState, pod *v1.Pod, nodeName string) (int64, *framework.Status) {
        /*
    nodeInfo, err := f.handle.SnapshotSharedLister().NodeInfos().Get(nodeName)
    if err != nil {
        return 0, framework.NewStatus(framework.Error, fmt.Sprintf("getting node %q from Snapshot: %v", nodeName, err))
    }
        */

        state, err := getPreFilterState(cycleState)
        if err != nil {
                return 0, framework.NewStatus(framework.Error, err.Error())
        }

        // we favor the node which the bandwidth will be used less.
        var score int64
        score = ( f.nodeBandwidth - (state.nodeAllocatedBandwidth[nodeName] + state.requestedBandwidth + f.reservedBandwidth) ) * framework.MaxNodeScore / f.nodeBandwidth

        klog.V(2).Infof("Score: Score of Node %q for Pod %q : %d(DefaultBW(%d), reqeustedBW(%d), allocatedBW(%d)", nodeName, pod.ObjectMeta.Name, score, f.nodeBandwidth, state.requestedBandwidth, state.nodeAllocatedBandwidth[nodeName])

    return score, nil
}

// ScoreExtensions of the Score plugin.
func (f *BandwidthFit) ScoreExtensions() framework.ScoreExtensions {
    return f
}

// NormalizeScore invoked after scoring all nodes.
func (f *BandwidthFit) NormalizeScore(ctx context.Context, state *framework.CycleState, pod *v1.Pod, scores framework.NodeScoreList) *framework.Status {
    return pluginhelper.DefaultNormalizeScore(framework.MaxNodeScore, false, scores)
}

// Filter invoked at the filter extension point.
// Checks if a node has sufficient egress bandwidth resource to run a pod.
// It returns a list of insufficient resources, if empty, then the node has all the resources requested by the pod.
func (f *BandwidthFit) Filter(ctx context.Context, cycleState *framework.CycleState, pod *v1.Pod, nodeInfo *framework.NodeInfo) *framework.Status {
        klog.V(2).Infof("BandwidthFit Filter called")

        state, err := getPreFilterState(cycleState)
        if err != nil {
                return framework.NewStatus(framework.Error, err.Error())
        }

        insufficientResources := f.fitsRequest(state, cycleState, pod, nodeInfo)

        if len(insufficientResources) != 0 {
                // We will keep all failure reasons.
                failureReasons := make([]string, 0, len(insufficientResources))
                for _, r := range insufficientResources {
                        failureReasons = append(failureReasons, r.Reason)
                }
                return framework.NewStatus(framework.Unschedulable, failureReasons...)
        }

        return nil
}

// InsufficientResource describes what kind of resource limit is hit and caused the pod to not fit the node.
type InsufficientResource struct {
        ResourceName v1.ResourceName
        // We explicitly have a parameter for reason to avoid formatting a message on the fly
        // for common resources, which is expensive for cluster autoscaler simulations.
        Reason    string
        Requested int64
        Used      int64
        Capacity  int64
}

// PodQOSClass defines the supported qos classes of Pods.
type PodQOSClass string

// These are valid values for PodQOSClass
const (
    // PodQOSGuaranteed is the Guaranteed qos class.
    PodQOSGuaranteed PodQOSClass = "Guaranteed"
    // PodQOSBurstable is the Burstable qos class.
    PodQOSBurstable PodQOSClass = "Burstable"
    // PodQOSBestEffort is the BestEffort qos class.
    PodQOSBestEffort PodQOSClass = "BestEffort"
)

func (f *BandwidthFit) fitsRequest(state *preFilterState, cycleState *framework.CycleState, pod *v1.Pod, nodeInfo *framework.NodeInfo) []InsufficientResource {
        insufficientResources := make([]InsufficientResource, 0, 1)

        if state.requestedBandwidth == 0 {
                return insufficientResources
        }

        var bandwidthSum int64 = 0
        for _, pod := range nodeInfo.Pods {
                request, limit, err := ExtractPodBandwidthResources(pod.Pod.ObjectMeta.Annotations)
                if err != nil {
                        klog.V(2).Infof("fitsRequest: pod %q Bandwidth Parsing Error", pod.Pod.ObjectMeta.Name)
                        continue
                }
                if limit != 0 {
                        bandwidthSum += limit
                } else if request != 0 {
                        bandwidthSum += request
                }
        }

        klog.V(2).Infof("fitsReqeust: pod %q, requestedBW(%d), qosClass(%q), node %q, DefaultNodeBW(%d), allocatedBW(%d)", pod.ObjectMeta.Name, state.requestedBandwidth, state.qosClass, nodeInfo.Node().ObjectMeta.Name, f.nodeBandwidth, bandwidthSum)

        if (state.requestedBandwidth + bandwidthSum + f.reservedBandwidth) > f.nodeBandwidth &&
                state.qosClass == PodQOSGuaranteed {
                insufficientResources = append(insufficientResources, InsufficientResource{
                        "nodeBandwidth",
                        "Insufficient bandwidth",
                        state.requestedBandwidth,
                        bandwidthSum,
                        f.nodeBandwidth - bandwidthSum - f.reservedBandwidth,
                })
        }

        state.nodeAllocatedBandwidth[nodeInfo.Node().ObjectMeta.Name] = bandwidthSum
        cycleState.Write(preFilterStateKey, state)

        return insufficientResources
}

var minRsrc = resource.MustParse("1k")
var maxRsrc = resource.MustParse("1P")

func validateBandwidthIsReasonable(rsrc *resource.Quantity) error {
        if rsrc.Value() < minRsrc.Value() {
                return fmt.Errorf("resource is unreasonably small (< 1kbit)")
        }
        if rsrc.Value() > maxRsrc.Value() {
                return fmt.Errorf("resoruce is unreasonably large (> 1Pbit)")
        }
        return nil
}

// ExtractPodBandwidthResources extracts the egress-request and egress-limit from the given pod annotations
func ExtractPodBandwidthResources(podAnnotations map[string]string) (request, limit int64, err error) {
        var egressRequest, egressLimit *resource.Quantity

        request = 0
        limit = 0

        if podAnnotations == nil {
                return request, limit, nil
        }
        str, found := podAnnotations[BandwidthRequestKey]
        if found {
                egressRequestValue, err := resource.ParseQuantity(str)
                if err != nil {
                        return request, limit, err
                }
                egressRequest = &egressRequestValue
                if err := validateBandwidthIsReasonable(egressRequest); err != nil {
                        return request, limit, err
                }
                request = int64(egressRequest.Value())
        }
        str, found = podAnnotations[BandwidthLimitKey]
        if found {
                egressLimitValue, err := resource.ParseQuantity(str)
                if err != nil {
                        return request, limit, err
                }
                egressLimit = &egressLimitValue
                if err := validateBandwidthIsReasonable(egressLimit); err != nil {
                        return request, limit, err
                }
                limit = int64(egressLimit.Value())
        }
        return request, limit, nil
}
root@sched-master:~/working/golang/src/sigs.k8s.io/scheduler-plugins#


2020년 12월 22일 화요일 오후 11시 42분 22초 UTC+9에 aco...@google.com님이 작성:

Aldo Culquicondor

unread,
Dec 23, 2020, 10:23:45 AM12/23/20
to Seong Moon, kubernetes-sig-scheduling
The name of your plugin NodeBandwidthFit doesn't match the name of the args BandwithFitArgs

It should be named NodeBandwithFitArgs

Aldo


To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-scheduling/168b3c2a-7530-49d7-9629-b05c26857c18n%40googlegroups.com.

Seong Moon

unread,
Dec 23, 2020, 7:44:06 PM12/23/20
to kubernetes-sig-scheduling
It works correctly after changing the struct name BandwidthFitArgs to NodeBandwidthFitArgs.

I am just curious why the name of PluginArgs struct should be same as the name of Plugin.
It seems like there is some hidden rules.

Anyway, Thank you very much of your advice.
Regards, seong.

2020년 12월 24일 목요일 오전 12시 23분 45초 UTC+9에 aco...@google.com님이 작성:

Aldo Culquicondor

unread,
Dec 29, 2020, 9:32:09 AM12/29/20
to Seong Moon, kubernetes-sig-scheduling
We use the name of the plugin + "Args" to find the args type.

Aldo


To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-scheduling/25c367c6-ce34-49b7-a10b-7cd4c0cedfe8n%40googlegroups.com.
Reply all
Reply to author
Forward
0 new messages