Running E2E tests with ResourceQuota admission plugin enabled

[Eric Hernandez]

Hello, we are trying to get an answer on if conformance tests should pass in a cluster that has an admission plugin enabled, specifically talking about ResourceQuota. With this admission plugin enabled, all namespaces are expected to have a ResourceQuota mentioning the Pod PriorityClass requested on test pods. Luckily, a PriorityClass will be set by default if not specified, but a ResourceQuota mentioning this PriorityClass in a ScopeSelector does not get created by default for E2E test namespaces. As a result, all E2E tests immediately fail on test pod creation. As a workaround, I have added the creation of an empty ResourceQuota object on test namespaces via framework/util.go, which allows test pods to be created and majority of E2Es to pass in clusters where this admission plugin is enabled.

We are trying to determine if E2E tests should be updated to support this use case, or how the community recommend running E2E tests with this admission plugin enabled.

https://kubernetes.io/docs/concepts/policy/resource-quotas/#limit-priority-class-consumption-by-default

https://github.com/kubernetes/kubernetes/issues/114006

Slack discussion in #sig-testing: https://kubernetes.slack.com/archives/C09QZ4DQB/p1668796862379909

[John Belamaric]

Short answer: no, there is no expectation that conformance tests can run with ResourceQuota admission plugin enabled

Long answer: Conformance applies only to non-optional features. For example, RBAC is not required for conformance. Furthermore, conformance applies to a distribution, not a cluster. If a distribution is *able to produce a cluster* that can pass the conformance tests, then the distribution is conforming and can use the Kubernetes name. *Every* cluster produced by the distribution need not be able to pass the conformance tests.

John

--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-architecture" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-arch...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-architecture/72865965-a27c-44d8-8c1e-69432bf2d930n%40googlegroups.com.

[Jordan Liggitt]

Conformance is certainly able to pass with the ResourceQuota admission plugin on (it's on in all the CI runs we do in Kubernetes). In fact, there are conformance tests that require ResourceQuota admission to be enabled (e.g. "ResourceQuota should create a ResourceQuota and capture the life of a configMap")

What is not guaranteed is that conformance can pass with arbitrary resource quota configuration. In Eric's case, they had a cluster-wide configuration set up to forbid pods with priorityclass by default, unless quota was re-granted within particular namespaces. There was a discussion in slack about some methods for running conformance tests on a cluster with restrictions like that in place.

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-architecture/CAC_Rkjw7CNhfq6ai9699zW1rTABK7zxsU043BFM8Zz9w61paqA%40mail.gmail.com.

[John Belamaric]

Thanks Jordan, I stand corrected on that :)

[Antonio Ojea]

I did the same mistake as John, glad we have Jordan to correct us :)

I also think that Jordan's suggestion to tag namespaces created by the e2e test, so external controllers can detect them and react,  is the way to go https://github.com/kubernetes/kubernetes/issues/114006#issuecomment-1363966629

[Eric Hernandez]

Thanks everyone for your responses. We will be trying to use a mutating web hook to a set a priorityClass on a list of namespace prefixes, to capture the ones used in the E2E's. We will set a priorityClass which is not set in the ResourceQuota Admission Plugin, so it will not require a ResourceQuota object to exist. OPA can then help us limit who can use that new pod priorityClass.