[Action Required] e2e Test Improvements in 1.34
Benjamin Elder
Hello Kubernetes contributors! Late in 1.33 we landed improvements to the e2e framework that you should adopt for your 1.33+ tests, Action Required:
If your e2e test is related to feature gated functionality pass the k8s.io/kubernetes/pkg/features featuregate like framework.WithFeatureGate(features.DeclarativeValidation) to the test. The featuregate details including Alpha/Beta/GA/Deprecated and OffByDefault will be communicated to the test framework and runners as ginkgo labels.
If your test is related to multiple featuregates, pass each of them.
If your test depends on a cluster e2e “feature” such as a “LoadBalancer”, continue passing features.LoadBalancer (k8s.io/kubernetes/test/e2e/feature). This includes any special configuration (e.g. IPv6 or DualStack) or additional components (LoadBalancer) that we cannot assume all clusters have available/configured by default that are not merely enabling feature gates.
If you previously had an e2e “feature” that was merely a stand-in for feature gate enablement, please migrate your tests to 1) and remove the e2e “feature” in favor of WithFeatureGate.
Some tests may require both, if they are linked to a feature gate AND require special configuration / additional components, in which case you should pass both WithFeatureGate and the e2e framework “feature”. For example DRA e2e tests require both DRA feature gates and the DRA e2e “feature” for container runtime support in the test cluster. Example:
Setting these appropriately will ensure we can correctly run them in the right CI jobs.
In particular, if your test only depends on feature gates and not e2e “features” and you adopt 1) & 3), we can now automatically run them by default in shared alpha / beta feature test jobs without any action on your part (you should still verify that your tests are running).
When reviewing / approving feature promotions, please ask for evidence of healthy running tests. Changing the featuregate state will change the test labels (e.g. from Alpha to Beta or Beta to GA) and therefore “promote” which CI jobs we run them in for any tests using WithFeatureGate associated with the featuregate.
For more details / tracking see: https://github.com/kubernetes/kubernetes/issues/131040
Also, I’d like to announce that we have finally achieved zero hard-coded test skips in pull-kubernetes-e2e-kind and related jobs, running roughly --ginkgo.label-filter=’Feature: isEmpty && !Slow && !Disruptive && !Flaky’. This is near parity with pull-kubernetes-e2e-gce (1056 tests vs 1080) in approximately half the runtime (~30m vs ~1h). Thank you to everyone who helped debug and fix these.
We’ll have more updates as we iterate on #131040 and the alpha/beta CI jobs in particular.
If you have any questions, feel free to reach out to me directly or to #sig-testing.
Thanks!
- Ben
Rodrigo Campos
> Hello Kubernetes contributors! Late in 1.33 we landed improvements
>
>
> If your e2e test is related to feature gated functionality pass the
> kubernetes/blob/master/pkg/features/kube_features.go>featuregate
> like framework.WithFeatureGate(features.DeclarativeValidation)to the
> OffByDefault will be communicated to the test framework and runners
> as ginkgo labels.
> If your test is related to multiple featuregates, pass each of them.
For example, I have this test:
https://github.com/kubernetes/kubernetes/blob/893486dfd16ff8c628c6f33bb2bea869ad86115f/test/e2e/common/node/security_context.go#L93
How shall it be adapted?
The `feature.UserNamespacesSupport` comes from:
https://github.com/kubernetes/kubernetes/blob/893486dfd16ff8c628c6f33bb2bea869ad86115f/test/e2e/feature/feature.go#L531
So basically, it is already expanded to:
UserNamespacesSupport =
framework.WithFeature(framework.ValidFeatures.Add("UserNamespacesSupport"))
So shall I add a:
framework.WithFeatureGate(feature.UserNamespacesSupport)
? That is all, I guess?
> 2.
>
> If your test depends on a cluster e2e “feature” <https://github.com/
> kubernetes/kubernetes/blob/master/test/e2e/feature/feature.go>such
> feature>). This includes any special configuration (e.g. IPv6 or
> assume all clusters have available/configured by default that are
> not merely enabling feature gates.
features.UserNamespacesSupport) to run them in nodes that the stack has
all that is required for userns. So I _think_ that part is probably okay.
However, we do have one node with special configuration to test the case
that the node is configured to use a specific mappings (instead of the
default) for userns. Tests on that node are scheduled to run
periodically. We can run manually using: /test
pull-kubernetes-node-crio-cgrpv2-userns-e2e-serial
What we are doing currently is skip these specific tests if the node
isn't properly configured with a custom kubelet userns mapping
configuration:
https://github.com/kubernetes/kubernetes/blob/893486dfd16ff8c628c6f33bb2bea869ad86115f/test/e2e/common/node/security_context.go#L132-L137
Shall this be migrated to cluster e2e features? In that case, is this
planned already? Or do you have any pointers on how that should be done?
Benjamin Elder
cc also sig-testing for specific testing questions.
> So shall I add a:
>
>framework.WithFeatureGate(feature.UserNamespacesSupport)
>? That is all, I guess?
This test seems to require host settings that are not universally available, and won't benefit as much from this as feature work that only requires featuregates.
> What we are doing currently is skip these specific tests if the node
isn't properly configured with a custom kubelet userns mapping
configuration:
https://github.com/kubernetes/kubernetes/blob/893486dfd16ff8c628c6f33bb2bea869ad86115f/test/e2e/common/node/security_context.go#L132-L137
>
> Shall this be migrated to cluster e2e features? In that case, is this
> planned already? Or do you have any pointers on how that should be done?
Hope that helps, feel free to ping me in slack or github as well for follow-ups.