enabling E2E test selection via Ginkgo labels

[Patrick Ohly]

Hello!

With Ginkgo v1, test selection in jobs or manual E2E testing had to be
done by embedding special [] tags like [Slow] into the test name
and then using carefully grafted regular expressions with `ginkgo
-focus` and/or `-skip`:

config/jobs/kubernetes/sig-node/containerd.yaml: - --test_args=--nodes=1 --timeout=4h --focus="\[Serial\]" --skip="\[Flaky\]|\[Benchmark\]|\[NodeSpecialFeature:.+\]|\[NodeSpecialFeature\]|\[NodeAlphaFeature:.+\]|\[NodeAlphaFeature\]|\[NodeFeature:Eviction\]"

Ginkgo v2 added support for test labels and a single command line flag
--label-filter to select tests based on those labels [1]. The above
`--focus/--skip` could be replaced with:

--label-filter="Serial && !(Flaky || Benchmark || /NodeSpecialFeature(:.+)?/ || /NodeAlphaFeature(:.+)?/ || NodeFeature:Eviction)

But first we need to ensure that tests are registered with labels. While
we transition, we still have to insert the traditional tags because jobs
depend on them. Therefore I proposed some helper functions in
test/e2e/framework which take care of registering tests with inline tags
*and* labels [2, 3]. Here's how usage of those looks like in E2E test
source code:

ginkgo.It("PVC should be recreated when pod is pending due to missing PVC [Disruptive][Serial]"
->
f.It("PVC should be recreated when pod is pending due to missing PVC", f.WithDisruptive(), f.WithSerial()

If you want to know which tests exist and what labels are available for
filtering, use:

go test -v ./test/e2e -list-tests -list-labels

The following labels can be used with 'gingko run --label-filter':
Beta
Conformance
Disruptive
Environment:NotInUserNS
Feature:APIServerIdentity
...
sig-storage
sig-windows

The following spec names can be used with 'ginkgo run --focus/skip':
apimachinery/watchlist.go:41: [sig-api-machinery] API Streaming (aka. WatchList) [Serial] [Feature:WatchList] should be requested when ENABLE_CLIENT_GO_WATCH_LIST_ALPHA is set
...
windows/reboot_node.go:46: [sig-windows] [Feature:Windows] [Excluded:WindowsDocker] [MinimumKubeletVersion:1.22] RebootHost containers [Serial] [Disruptive] [Slow] should run as a reboot process on the host/node

There is also a KEP pending in provisional state about cleaning up which
tags should be used and removing some of the current ambiguities and
redundancy [4]. In preparation for that PR, features, node features, and
environments are defined in test/e2e/[feature|nodefeature|environment]
and tests are prevented from inventing their own. Ideally, new entries
there should come with comments that properly explain what the entry
means - we currently don't have that.

Dependencies on feature gates can be declared via
`framework.WithFeatureGate("SomeFeature")`. That will also automatically
add `Alpha` or `Beta`. In contrast to `framework.WithFeature`, such a
feature gate label must be removed together with removing the feature
gate (won't compile otherwise!). As with the other helper functions, a
framework instance has methods which have the same effect, so often
`f.WithFeatureGate` and `f.WithFeature` can be used instead.

I currently have one PR ready [5] which converts tests to use the
standard labels. To keep its size manageable, it does not yet convert
all of the other arbitrary tags to `framework.WithLabel`. That can
follow later. Please have a look if you are interested:

https://github.com/kubernetes/kubernetes/pull/121394

The goal was to change test names as little as possible, but spaces are
now used consistently around tags and thus some test names have spaces
where they had none before.

Once that PR is merged, verify-e2e-suites.sh will check that labels are
used as intended and E2E suites will refuse to start if there are errors
like using the traditional `[Slow]` instead of the new
`framework.WithSlow()`.

Note that ginkgo must not be called directly when using these labels, it
would fail during test suite startup with:

ginkgo.It("should support sysctls with slashes as separator [MinimumKubeletVersion:1.23]", environment.NotInUserNS, func(ctx context.Context) {
/nvme/gopath/src/k8s.io/kubernetes/test/e2e/common/node/sysctl.go:188
[It] node was passed an unknown decorator:
'framework.label{parts:[]string{"Environment", "NotInUserNS"}, extra:""}'

[1] https://onsi.github.io/ginkgo/#filtering-specs
[2] https://github.com/kubernetes/kubernetes/pull/112894
[3] https://github.com/kubernetes/kubernetes/pull/121358
[4] https://github.com/kubernetes/enhancements/tree/master/keps/sig-testing/3041-node-conformance-and-features
[5] https://github.com/kubernetes/kubernetes/pull/121394

--
Best Regards

Patrick Ohly
Cloud Software Architect

[Antonio Ojea]

I think that is a nice feature that will simplify the jobs definitions to run e2e tests and provide normalization on the tests tags spaces, avoiding to have complex regexs on our job definitions.

Thanks for working on this Patrick

--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-testing" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-te...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-testing/yrjhmsw8m1hr.fsf%40pohly-mobl1.fritz.box.

[Patrick Ohly]

"Patrick Ohly" <patric...@intel.com> writes:
> But first we need to ensure that tests are registered with labels. While
> we transition, we still have to insert the traditional tags because jobs
> depend on them.

Let me clarify the "transition" part: the goal right now is only to
enable some common labels. More labels, in particular for all the various
free-form tags that were added over time, might come next. There is no
plan to remove the inline tags (yet), so there's no rush to rewrite jobs.

The KEP will be used to discuss and plan removal of redundant or useless
tags. That might be a good place to also decide about the future of
inline tags:

https://github.com/kubernetes/enhancements/tree/master/keps/sig-testing/3041-node-conformance-and-features

--
Best Regards

Patrick Ohly
Cloud Software Architect

Intel GmbH
System Software Engineering/Cloud Native
Usenerstr. 5a Phone: +49-228-2493652
53129 Bonn
Germany