Reducing the friction to run kunit tests

[Theodore Ts'o]

Recently, in ext4 we had a few test bugs which showed up because
people weren't running the ext4 Kunit tests as part of their
development workflow, and when they made changes to the code, they
didn't notice that the Kunit tests were failing.

When I was working the bug report, I realized that I had forgotten
how to run the kunit tests, which is probably why I didn't notice the
bug when I was reviewing the code. I can never remember that the path
to "kunit_tool" is "./tools/testing/kunit/kunit.py" so I end up having
to look it up each time it occurs to me that I should try running
ext4's kunit tests.

I've fixed this for myself and those ext4 developers using the
gce-xfstests/kvm-xfstests test infrastructure by making "kbuild --test"
running the kunit tests, so I don't have to look it up each
time. See:

https://github.com/tytso/xfstests-bld/commit/9fa435ce1add7846a5345c90817852ad6632c6c2

But it caused me to wonder; perhaps it would be useful if we added a
top-level make target, so that something like "make kunit", or "make
test" in the kernel tree would run the kunit tests. For people who
use the "bazel" tool, or Google's internal "blaze" tool, being able to
run unit tests really easily using "bazel test" probably makes it much
more likely that people will actually run the tests that everyone has
been investing a lot of effort to create.

What do people think? I'm happy to create a patch doing this, if
people think this would be a good idea, and would be supportive of it.

Cheers,

- Ted

[David Gow]

I'm definitely in favour of anything which makes it easier to run tests. :-)

The biggest question I'd have around having "make kunit" / "make test"
would be which configuration should be used. The simple answer of the
default KUnit configuration (kunit.py run) is probably right, but UML
only works on x86{,-64} and ignores the current config. Something more
interesting which uses ARCH and/or the current .config to better match
the developer's intent might be better, but has the downsides of being
more work to implement, and being less reliable as a consistent check.

There was a push to have something more general than KUnit, too, which
catalogued per-subsystem test requirements (either in MAINTAINERS, or
as a part of something like
https://github.com/dzickusrh/test-catalog/tree/main ), which would
solve the problem of different subsystems having different test
requirements, but consequently introducing more complexity for the
average user. (And I think those efforts are largely morphing into
more coverage-focused things, than per-maintainer/subsystem tests
requirements, too.)

I'd certainly get behind just adding "make kunit" as a wrapper around
"kunit.py run" for now, though, with the ability to build on it with
more options later as we need them. And I'm even more in favour of
people adding a call out to KUnit (either via make test, or manual
calls to kunit.py, possibly with subsystem-specific options) which
test the things they care about. (I, for instance, have a script which
runs kunit.py repeatedly with different options to test KUnit patches,
which is hacky, but works well).

Cheers,
-- David

[Theodore Ts'o]

On Sat, Jul 26, 2025 at 05:27:11PM +0800, 'David Gow' via KUnit Development wrote:
>
> The biggest question I'd have around having "make kunit" / "make test"
> would be which configuration should be used. The simple answer of the
> default KUnit configuration (kunit.py run) is probably right, but UML
> only works on x86{,-64} and ignores the current config.

At least for now, what I'd suggest is to have "make kunit" use the
default config for those architectures that support UML. At the
moment the default Kunit configuration runs all of the unit tests, and
the unit tests are so fast that I think it's fine.

On my development system, running all of the tests takes 4.8s of wall
clock time. Running just the ext4 kunit tests only takes 2.5s. Now,
this *is* a pretty fast development system with 48 cores, but even if
I force only one CPU to be used with --jobs=1, it's still 6.8 seconds
for al tests versus 3.9 seconds for just the ext4 kunit tests. So
using the default configuration should be relatively acceptable.

> There was a push to have something more general than KUnit, too, which
> catalogued per-subsystem test requirements (either in MAINTAINERS, or
> as a part of something like
> https://github.com/dzickusrh/test-catalog/tree/main ), which would
> solve the problem of different subsystems having different test
> requirements, but consequently introducing more complexity for the
> average user. (And I think those efforts are largely morphing into
> more coverage-focused things, than per-maintainer/subsystem tests
> requirements, too.)

Yeah, that's a much harder problem. It's going to vary depending on
problem scoping; are you someone who is writing a drive-by patch and
you want to run the minimum set of tests demanding by the maintainer?
Are you a subsystem developer who wants to run a more expansive set of
testS? Or are you the subsystem maintainer running the full set of
tests before sending a pull request to Linus?

And it might depend on the test resources that you have available. If
you are a subsystem developer who is on the road and only has their
laptop, the set of test you might run might be different if you are at
home with your 48-core threadripper development system.

So yeah, let's just have "make kunit" be a simpler way of kicking off
all of the kunit tests. That's probably going to be much less
controversial, and a lot easier to get upstream.

Cheers,

- Ted

[David Gow]

On Sat, 26 Jul 2025 at 23:47, Theodore Ts'o <ty...@mit.edu> wrote:
>
> On Sat, Jul 26, 2025 at 05:27:11PM +0800, 'David Gow' via KUnit Development wrote:
> >
> > The biggest question I'd have around having "make kunit" / "make test"
> > would be which configuration should be used. The simple answer of the
> > default KUnit configuration (kunit.py run) is probably right, but UML
> > only works on x86{,-64} and ignores the current config.
>
> At least for now, what I'd suggest is to have "make kunit" use the
> default config for those architectures that support UML. At the
> moment the default Kunit configuration runs all of the unit tests, and
> the unit tests are so fast that I think it's fine.
>
> On my development system, running all of the tests takes 4.8s of wall
> clock time. Running just the ext4 kunit tests only takes 2.5s. Now,
> this *is* a pretty fast development system with 48 cores, but even if
> I force only one CPU to be used with --jobs=1, it's still 6.8 seconds
> for al tests versus 3.9 seconds for just the ext4 kunit tests. So
> using the default configuration should be relatively acceptable.
>

I'm less worried about performance here than compatibility (make kunit
won't work at all on non-x86 systems) and the need to test things
which aren't in the default config at all.

But I agree, this is a problem for a follow-up patch (or at least some
way of passing arguments through): having `make kunit` run the default
config is still the way to go.

> > There was a push to have something more general than KUnit, too, which
> > catalogued per-subsystem test requirements (either in MAINTAINERS, or
> > as a part of something like
> > https://github.com/dzickusrh/test-catalog/tree/main ), which would
> > solve the problem of different subsystems having different test
> > requirements, but consequently introducing more complexity for the
> > average user. (And I think those efforts are largely morphing into
> > more coverage-focused things, than per-maintainer/subsystem tests
> > requirements, too.)
>
> Yeah, that's a much harder problem. It's going to vary depending on
> problem scoping; are you someone who is writing a drive-by patch and
> you want to run the minimum set of tests demanding by the maintainer?
> Are you a subsystem developer who wants to run a more expansive set of
> testS? Or are you the subsystem maintainer running the full set of
> tests before sending a pull request to Linus?
>
> And it might depend on the test resources that you have available. If
> you are a subsystem developer who is on the road and only has their
> laptop, the set of test you might run might be different if you are at
> home with your 48-core threadripper development system.
>
> So yeah, let's just have "make kunit" be a simpler way of kicking off
> all of the kunit tests. That's probably going to be much less
> controversial, and a lot easier to get upstream.

Sounds good to me. If you've got a patch ready to go, send it out and
I'll take a look. Otherwise, I'll have a go at putting something
together.

Cheers,
-- David