Hello!
With Ginkgo v1, test selection in jobs or manual E2E testing had to be
done by embedding special [] tags like [Slow] into the test name
and then using carefully grafted regular expressions with `ginkgo
-focus` and/or `-skip`:
config/jobs/kubernetes/sig-node/containerd.yaml: - --test_args=--nodes=1 --timeout=4h --focus="\[Serial\]" --skip="\[Flaky\]|\[Benchmark\]|\[NodeSpecialFeature:.+\]|\[NodeSpecialFeature\]|\[NodeAlphaFeature:.+\]|\[NodeAlphaFeature\]|\[NodeFeature:Eviction\]"
Ginkgo v2 added support for test labels and a single command line flag
--label-filter to select tests based on those labels [1]. The above
`--focus/--skip` could be replaced with:
--label-filter="Serial && !(Flaky || Benchmark || /NodeSpecialFeature(:.+)?/ || /NodeAlphaFeature(:.+)?/ || NodeFeature:Eviction)
But first we need to ensure that tests are registered with labels. While
we transition, we still have to insert the traditional tags because jobs
depend on them. Therefore I proposed some helper functions in
test/e2e/framework which take care of registering tests with inline tags
*and* labels [2, 3]. Here's how usage of those looks like in E2E test
source code:
ginkgo.It("PVC should be recreated when pod is pending due to missing PVC [Disruptive][Serial]"
->
f.It("PVC should be recreated when pod is pending due to missing PVC", f.WithDisruptive(), f.WithSerial()
If you want to know which tests exist and what labels are available for
filtering, use:
go test -v ./test/e2e -list-tests -list-labels
The following labels can be used with 'gingko run --label-filter':
Beta
Conformance
Disruptive
Environment:NotInUserNS
Feature:APIServerIdentity
...
sig-storage
sig-windows
The following spec names can be used with 'ginkgo run --focus/skip':
apimachinery/watchlist.go:41: [sig-api-machinery] API Streaming (aka. WatchList) [Serial] [Feature:WatchList] should be requested when ENABLE_CLIENT_GO_WATCH_LIST_ALPHA is set
...
windows/reboot_node.go:46: [sig-windows] [Feature:Windows] [Excluded:WindowsDocker] [MinimumKubeletVersion:1.22] RebootHost containers [Serial] [Disruptive] [Slow] should run as a reboot process on the host/node
There is also a KEP pending in provisional state about cleaning up which
tags should be used and removing some of the current ambiguities and
redundancy [4]. In preparation for that PR, features, node features, and
environments are defined in test/e2e/[feature|nodefeature|environment]
and tests are prevented from inventing their own. Ideally, new entries
there should come with comments that properly explain what the entry
means - we currently don't have that.
Dependencies on feature gates can be declared via
`framework.WithFeatureGate("SomeFeature")`. That will also automatically
add `Alpha` or `Beta`. In contrast to `framework.WithFeature`, such a
feature gate label must be removed together with removing the feature
gate (won't compile otherwise!). As with the other helper functions, a
framework instance has methods which have the same effect, so often
`f.WithFeatureGate` and `f.WithFeature` can be used instead.
I currently have one PR ready [5] which converts tests to use the
standard labels. To keep its size manageable, it does not yet convert
all of the other arbitrary tags to `framework.WithLabel`. That can
follow later. Please have a look if you are interested:
https://github.com/kubernetes/kubernetes/pull/121394
The goal was to change test names as little as possible, but spaces are
now used consistently around tags and thus some test names have spaces
where they had none before.
Once that PR is merged, verify-e2e-suites.sh will check that labels are
used as intended and E2E suites will refuse to start if there are errors
like using the traditional `[Slow]` instead of the new
`framework.WithSlow()`.
Note that ginkgo must not be called directly when using these labels, it
would fail during test suite startup with:
ginkgo.It("should support sysctls with slashes as separator [MinimumKubeletVersion:1.23]", environment.NotInUserNS, func(ctx context.Context) {
/nvme/gopath/src/
k8s.io/kubernetes/test/e2e/common/node/sysctl.go:188
[It] node was passed an unknown decorator:
'framework.label{parts:[]string{"Environment", "NotInUserNS"}, extra:""}'
[1]
https://onsi.github.io/ginkgo/#filtering-specs
[2]
https://github.com/kubernetes/kubernetes/pull/112894
[3]
https://github.com/kubernetes/kubernetes/pull/121358
[4]
https://github.com/kubernetes/enhancements/tree/master/keps/sig-testing/3041-node-conformance-and-features
[5]
https://github.com/kubernetes/kubernetes/pull/121394
--
Best Regards
Patrick Ohly
Cloud Software Architect