[kubernetes/kubernetes] FindPodVolumes is returning success for assumed volumes on other nodes (#56596)
Michelle Au
Is this a BUG REPORT or FEATURE REQUEST?:
@kubernetes/sig-storage-bugs
/assign
What happened:
Volume got bound to PV on node "kubernetes-minion-group-82c8"
I1130 00:52:54.196214 5 scheduler_binder.go:181] AssumePodVolumes for pod "default/mysql-faster-0", node "kubernetes-minion-group-82c8"
I1130 00:52:54.196241 5 scheduler_binder.go:280] PVC "default/data-mysql-faster-0" is not bound
I1130 00:52:54.196289 5 scheduler_binder.go:194] AssumePodVolumes: getBindVolumeToClaim for PV "local-pv-b5cf5b33", PVC "data-mysql-faster-0". newPV 0xc420dae500, dirty true, err: <nil>
I1130 00:52:54.196306 5 scheduler_assume_cache.go:260] Assumed v1.PersistentVolume "local-pv-b5cf5b33", version 814
I1130 00:52:54.196331 5 scheduler.go:325] Trying to bind volumes for pod "default/mysql-faster-0"
I1130 00:52:54.196337 5 scheduler_binder.go:228] BindPodVolumes for pod "default/mysql-faster-0"
I1130 00:52:54.196344 5 pv_controller.go:804] claim "default/data-mysql-faster-0" bound to volume "local-pv-b5cf5b33"
I1130 00:52:54.203157 5 pv_controller.go:817] updating PersistentVolume[local-pv-b5cf5b33]: bound to "default/data-mysql-faster-0"
I1130 00:52:54.203196 5 scheduler.go:334] Successfully bound volumes for pod "default/mysql-faster-0"
When the pod went back through scheduling, the predicate returned success for other nodes even though we should have updated the PV cache to point to the PVC:
I1130 00:52:54.222461 5 scheduler_binder.go:133] FindPodVolumes for pod "default/mysql-faster-0", node "kubernetes-minion-group-q9pw"
I1130 00:52:54.222477 5 scheduler_binder.go:280] PVC "default/data-mysql-faster-0" is not bound
I1130 00:52:54.229299 5 scheduler_binder.go:373] Found matching volumes on node "kubernetes-minion-group-q9pw"
I1130 00:52:54.229352 5 predicates.go:1492] All PVCs found matches for pod %v/%v, node %qdefaultmysql-faster-0kubernetes-minion-group-q9pw
I1130 00:52:54.229388 5 scheduler_binder.go:133] FindPodVolumes for pod "default/mysql-faster-0", node "kubernetes-minion-group-82c8"
I1130 00:52:54.229405 5 scheduler_binder.go:280] PVC "default/data-mysql-faster-0" is not bound
I1130 00:52:54.229573 5 scheduler_binder.go:373] Found matching volumes on node "kubernetes-minion-group-82c8"
I1130 00:52:54.229618 5 predicates.go:1492] All PVCs found matches for pod %v/%v, node %qdefaultmysql-faster-0kubernetes-minion-group-82c8
I1130 00:52:54.234484 5 scheduler_binder.go:133] FindPodVolumes for pod "default/mysql-faster-0", node "kubernetes-minion-group-nrfq"
I1130 00:52:54.234522 5 scheduler_binder.go:280] PVC "default/data-mysql-faster-0" is not bound
I1130 00:52:54.235880 5 scheduler_binder.go:373] Found matching volumes on node "kubernetes-minion-group-nrfq"
I1130 00:52:54.235903 5 predicates.go:1492] All PVCs found matches for pod %v/%v, node %qdefaultmysql-faster-0kubernetes-minion-group-nrfq
I think in findMatchingVolumes, instead of continuing if the node affinity doesn't match, it should fail if the PV is prebound.
—
You are receiving this because you are on a team that was mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.
Michelle Au
This causes two issues:
- On the second pass, the scheduler may choose a different node. Then in Assume/Bind volumes, we'll also prebind a second set of PVs.
- If the scheduler chose a different node on the second pass, and the PVCs became fully bound in between the predicates and assume volumes, then assume volumes will return success and the pod will get scheduled to the wrong node.
b0b0haha
I'm confused why after the AssumePodVolumes make Volume got bound to PV on node "kubernetes-minion-group-82c8", the FindPodVolumes still show pod "default/mysql-faster-0 's pvc "default/data-mysql-faster-0" is not bound?
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you are on a team that was mentioned.