[PATCH] Documentation: kunit: Clarify test filter format

[Brendan Jackman]

It seems obvious once you know, but at first I didn't realise that the
suite name is part of this format. Document it and add example.

Signed-off-by: Brendan Jackman <jack...@google.com>
---
Documentation/dev-tools/kunit/run_wrapper.rst | 9 ++++++++-
1 file changed, 8 insertions(+), 1 deletion(-)

diff --git a/Documentation/dev-tools/kunit/run_wrapper.rst b/Documentation/dev-tools/kunit/run_wrapper.rst
index 19ddf5e07013..e75a5fc05814 100644
--- a/Documentation/dev-tools/kunit/run_wrapper.rst
+++ b/Documentation/dev-tools/kunit/run_wrapper.rst
@@ -156,13 +156,20 @@ Filtering tests
===============

By passing a bash style glob filter to the ``exec`` or ``run``
-commands, we can run a subset of the tests built into a kernel . For
+commands, we can run a subset of the tests built into a kernel,
+identified by a string like ``$suite_name.$test_name``. For
example: if we only want to run KUnit resource tests, use:

.. code-block::

./tools/testing/kunit/kunit.py run 'kunit-resource*'

+Or to run just one specific test from that suite:
+
+.. code-block::
+
+ ./tools/testing/kunit/kunit.py run 'kunit-resource-test.kunit_resource_test_init_resources'
+
This uses the standard glob format with wildcard characters.

.. _kunit-on-qemu:
--
2.44.0.396.g6e790dbe36-goog

[Daniel Latypov]

On Thu, Mar 28, 2024 at 7:20 AM 'Brendan Jackman' via KUnit
Development <kuni...@googlegroups.com> wrote:
>
> It seems obvious once you know, but at first I didn't realise that the
> suite name is part of this format. Document it and add example.
>
> Signed-off-by: Brendan Jackman <jack...@google.com>
> ---
> Documentation/dev-tools/kunit/run_wrapper.rst | 9 ++++++++-
> 1 file changed, 8 insertions(+), 1 deletion(-)
>
> diff --git a/Documentation/dev-tools/kunit/run_wrapper.rst b/Documentation/dev-tools/kunit/run_wrapper.rst
> index 19ddf5e07013..e75a5fc05814 100644
> --- a/Documentation/dev-tools/kunit/run_wrapper.rst
> +++ b/Documentation/dev-tools/kunit/run_wrapper.rst
> @@ -156,13 +156,20 @@ Filtering tests
> ===============
>
> By passing a bash style glob filter to the ``exec`` or ``run``
> -commands, we can run a subset of the tests built into a kernel . For
> +commands, we can run a subset of the tests built into a kernel,
> +identified by a string like ``$suite_name.$test_name``. For

Apologies for the overly terse docs, that's my fault :)
I'm wondering if we can further improve it while we're here.

Note, the format for the glob is: $suite_name[.$test_name].

This current wording and examples (before and after this change) might
make the user think otherwise, i.e. that it works like
effective_name = suite_name + '.' + test_name
return glob_matches(effective_name, filter_glob)

E.g. given a test name like `suite.test_name` and glob='suite*name'
they might expect it to match, but it does *not*.

The logic actually works like:
suite_glob, test_glob = split(filter_glob)
if not_glob_matches(suite_name, suite_glob):
return False
if test_glob and not glob_matches(test_name, test_glob):
return False
return True

Perhaps expanding the list of examples to cover more of the edge cases
could help get the right intuition?

E.g. perhaps these:
kunit.py run <suite_name> # runs all tests in a specific suite
kunit.py run <suite_name>.<test_name> # run a specific test

kunit.py run suite_prefix* # what the current example shows
kunit.py run *.*test_suffix # matches all suites, only tests w/ a
certain suffix
kunit.py run suite_prefix*.*test_suffix # combined version of above

Thoughts?

Thanks,
Daniel

[Brendan Jackman]

On Thu, 28 Mar 2024 at 19:27, Daniel Latypov <dlat...@google.com> wrote:
> This current wording and examples (before and after this change) might
> make the user think otherwise, i.e. that it works like
> effective_name = suite_name + '.' + test_name
> return glob_matches(effective_name, filter_glob)
>
> E.g. given a test name like `suite.test_name` and glob='suite*name'
> they might expect it to match, but it does *not*.
>
> The logic actually works like:
> suite_glob, test_glob = split(filter_glob)
> if not_glob_matches(suite_name, suite_glob):
> return False
> if test_glob and not glob_matches(test_name, test_glob):
> return False
> return True
>
> Perhaps expanding the list of examples to cover more of the edge cases
> could help get the right intuition?
>
> E.g. perhaps these:
> kunit.py run <suite_name> # runs all tests in a specific suite
> kunit.py run <suite_name>.<test_name> # run a specific test
>
> kunit.py run suite_prefix* # what the current example shows
> kunit.py run *.*test_suffix # matches all suites, only tests w/ a
> certain suffix
> kunit.py run suite_prefix*.*test_suffix # combined version of above
>
> Thoughts?

Thanks yeah, good point. The result is pretty verbose but it doesn't
create much cognitive load for the reader so might as well just be
really explicit. v2 incoming if `make htmldocs` ever finishes....

[Brendan Jackman]

It seems obvious once you know, but at first I didn't realise that the

suite name is part of this format. Document it and add some examples.

Signed-off-by: Brendan Jackman <jack...@google.com>
---

v1->v2: Expanded to clarify that suite_glob and test_glob are two separate
patterns. Also made some other trivial changes to formatting etc.

Documentation/dev-tools/kunit/run_wrapper.rst | 33 +++++++++++++++++--
1 file changed, 30 insertions(+), 3 deletions(-)

diff --git a/Documentation/dev-tools/kunit/run_wrapper.rst b/Documentation/dev-tools/kunit/run_wrapper.rst
index 19ddf5e07013..b07252d3fa9d 100644
--- a/Documentation/dev-tools/kunit/run_wrapper.rst
+++ b/Documentation/dev-tools/kunit/run_wrapper.rst
@@ -156,12 +156,39 @@ Filtering tests

===============

By passing a bash style glob filter to the ``exec`` or ``run``
-commands, we can run a subset of the tests built into a kernel . For

-example: if we only want to run KUnit resource tests, use:

+commands, we can run a subset of the tests built into a kernel,

+identified by a string like ``<suite_glob>[.<test_glob>]``.
+
+For example, to run the ``kunit-resource-test`` suite:
+
+.. code-block::
+
+ ./tools/testing/kunit/kunit.py run kunit-resource-test
+
+To run a specific test from that suite:
+
+.. code-block::
+
+ ./tools/testing/kunit/kunit.py run kunit-resource-test.kunit_resource_test
+
+To run all tests from suites whose names start with ``kunit``:
+
+.. code-block::
+
+ ./tools/testing/kunit/kunit.py run 'kunit*'
+
+To run all tests whose name ends with ``remove_resource``:
+
+.. code-block::
+
+ ./tools/testing/kunit/kunit.py run '*.*remove_resource'
+
+To run all tests whose name ends with ``remove_resource``, from suites whose
+names start with ``kunit``:

.. code-block::

- ./tools/testing/kunit/kunit.py run 'kunit-resource*'
+ ./tools/testing/kunit/kunit.py run 'kunit*.*remove_resource'

This uses the standard glob format with wildcard characters.

--
2.44.0.478.gd926399ef9-goog

[Daniel Latypov]

On Tue, Apr 2, 2024 at 5:51 AM Brendan Jackman <jack...@google.com> wrote:
>
> It seems obvious once you know, but at first I didn't realise that the
> suite name is part of this format. Document it and add some examples.
>
> Signed-off-by: Brendan Jackman <jack...@google.com>

Reviewed-by: Daniel Latypov <dlat...@google.com>

Thanks!

I agree with your comment on v1, I think the extra verbosity is fine.
It's still easy to read and this should hopefully eliminate the
ambiguity for most readers.

> ---
> v1->v2: Expanded to clarify that suite_glob and test_glob are two separate
> patterns. Also made some other trivial changes to formatting etc.

<snip>

[Brendan Jackman]

On Wed, Apr 03, 2024 at 02:59:43PM -0700, Daniel Latypov wrote:
> Reviewed-by: Daniel Latypov <dlat...@google.com>

Hi Jonathan, I think this is ready to be applied?

Thanks,
Brendan

[Jonathan Corbet]

I'm happy to take this, but normally these patches go through the kunit
tree, so I've not been paying much attention. Let me know please if I
should pick it up.

Thanks,

jon

[David Gow]

On Thu, 28 Mar 2024 at 22:20, Brendan Jackman <jack...@google.com> wrote:
>
> It seems obvious once you know, but at first I didn't realise that the
> suite name is part of this format. Document it and add example.
>
> Signed-off-by: Brendan Jackman <jack...@google.com>
> ---

We can take this via KUnit -- sorry for the delay.

Reviewed-by: David Gow <davi...@google.com>

Cheers,
-- David