Using schedulingGates feature in volcano scheduler

[sunil kalva]

Subject: We want to gate all the pods and approve by an external quota manager for team’s capacity, only then pick by volcano and schedule them.

We've implemented an external quota manager responsible for reviewing all incoming POD requests. Only once these requests receive approval from the quota manager are they considered eligible for scheduling. I had intended to utilize the feature described in the following documentation: Kubernetes Pod Scheduling and Readiness. However, it seems that Volcano is not adhering to this feature. It proceeds to schedule pods without respecting the schedulingGates. This behavior is in contrast to the default Kubernetes kube-scheduler, which functions as expected.

Even though these pods have set the schedulingGates, and were assigned to a node by the Volcano scheduler, it appears that they are subsequently rejected by the Kubernetes API server. I believe this error is occurring after the execution of the Volcano scheduler.

Error:

Failed to bind pod <namsepace/pod-name> to node <node> : &errors.StatusError{ErrStatus:v1.Status{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ListMeta:v1.ListMeta{SelfLink:"", ResourceVersion:"", Continue:"", RemainingItemCount:(*int64)(nil)}, Status:"Failure", Message:"Operation cannot be fulfilled on pods/binding \”app-name\”: pod pod-name has non-empty .spec.schedulingGates", Reason:"Conflict", Details:(*v1.StatusDetails)(0x140002188a0), Code:409}}

Reference : https://kubernetes.io/blog/2022/12/26/pod-scheduling-readiness-alpha/