Issue in parsing of tests that use KUNIT_CASE_PARAM

[Ojaswin Mujoo]

Hello,

While writing some Kunit tests for ext4 filesystem, I'm encountering an
issue in the way we display the diagnostic logs upon failures, when
using KUNIT_CASE_PARAM() to write the tests.

This can be observed by patching fs/ext4/mballoc-test.c to fail
and print one of the params:

--- a/fs/ext4/mballoc-test.c
+++ b/fs/ext4/mballoc-test.c
@@ -350,6 +350,8 @@ static int mbt_kunit_init(struct kunit *test)
struct super_block *sb;
int ret;

+ KUNIT_FAIL(test, "Failed: blocksize_bits=%d", layout->blocksize_bits);
+
sb = mbt_ext4_alloc_super_block();
if (sb == NULL)
return -ENOMEM;

With the above change, we can observe the following output (snipped):

[18:50:25] ============== ext4_mballoc_test (7 subtests) ==============
[18:50:25] ================= test_new_blocks_simple ==================
[18:50:25] [FAILED] block_bits=10 cluster_bits=3 blocks_per_group=8192 group_count=4 desc_size=64
[18:50:25] # test_new_blocks_simple: EXPECTATION FAILED at fs/ext4/mballoc-test.c:364
[18:50:25] Failed: blocksize_bits=12
[18:50:25] [FAILED] block_bits=12 cluster_bits=3 blocks_per_group=8192 group_count=4 desc_size=64
[18:50:25] # test_new_blocks_simple: EXPECTATION FAILED at fs/ext4/mballoc-test.c:364
[18:50:25] Failed: blocksize_bits=16
[18:50:25] [FAILED] block_bits=16 cluster_bits=3 blocks_per_group=8192 group_count=4 desc_size=64
[18:50:25] # test_new_blocks_simple: EXPECTATION FAILED at fs/ext4/mballoc-test.c:364
[18:50:25] Failed: blocksize_bits=10
[18:50:25] # test_new_blocks_simple: pass:0 fail:3 skip:0 total:3
[18:50:25] ============= [FAILED] test_new_blocks_simple ==============
<snip>

Note that the diagnostic logs don't show up correctly. Ideally they
should be before test result but here the first [FAILED] test has no
logs printed above whereas the last "Failed: blocksize_bits=10" print
comes after the last subtest, when it actually corresponds to the first
subtest.

The KTAP file itself seems to have diagnostic logs in the right place:

KTAP version 1
1..2
KTAP version 1
# Subtest: ext4_mballoc_test
# module: ext4
1..7
KTAP version 1
# Subtest: test_new_blocks_simple
# test_new_blocks_simple: EXPECTATION FAILED at fs/ext4/mballoc-test.c:364
Failed: blocksize_bits=10
not ok 1 block_bits=10 cluster_bits=3 blocks_per_group=8192 group_count=4 desc_size=64
# test_new_blocks_simple: EXPECTATION FAILED at fs/ext4/mballoc-test.c:364
Failed: blocksize_bits=12
not ok 2 block_bits=12 cluster_bits=3 blocks_per_group=8192 group_count=4 desc_size=64
# test_new_blocks_simple: EXPECTATION FAILED at fs/ext4/mballoc-test.c:364
Failed: blocksize_bits=16
not ok 3 block_bits=16 cluster_bits=3 blocks_per_group=8192 group_count=4 desc_size=64
# test_new_blocks_simple: pass:0 fail:3 skip:0 total:3
not ok 1 test_new_blocks_simple
<snip>

By tracing kunit_parser.py script, I could see the issue here is in the
parsing of the "Subtest: test_new_blocks_simple". We end up associating
everything below the subtest till "not ok 1 block_bits=10..." as
diagnostic logs of the subtest, while these lons actually belong to the
first of the 3 subtests under this test.

I tired to figure out a way to fix the parsing but fixing one thing
broke something else. Im starting to think that the issue is that there
are 3 subtests under test_new_block_simple (array of 3 params passed to
KUNIT_CASE_PARAM), but instead of creating 3 structured subtests, the
KTAP output dumps the results of all 3 directly under
subtest:test_new_blocks_simple. Which makes it tricky to determine where
the diagnostic log/attributes of test_new_blocks_simple ends and that of its
children begins.

I'm not very familiar with KUnit framework so I though I'd reach out
here for some pointers. I can dedicate some time fixing this but I'd
like to know if this is something we need to somehow fix in parsing or
during generation of the KTAP file itself? Any pointers would be
appreciated.

Thanks,
Ojaswin

[David Gow]

On Thu, 18 Dec 2025 at 15:20, Ojaswin Mujoo <oja...@linux.ibm.com> wrote:

> So looking into this a bit more and comparing the parameterized output
> with non parameterized output, I'm seeing that the difference is that
> output via KUNIT_CASE_PARAM is not printing the test plan line right
> here. This plan sort of serves as divider between the parent and the 3
> children's logs and without it our parsing logic gets confused. When I
> manually added a "1..3" test plan I could see the parsing work correctly
> without any changes to kunit_parser.py.
>

Thanks for looking into this!

There's been a bit of back-and-forth on how to include the test plan
line for the parameterised tests: it's not always possible to know how
many times a test will run in advance if the gen_params function is
particularly complicated.

We did have a workaround where array parameters would record the array
size, but there were a couple of tests which were wrapping the
gen_params function to skip / add entries which weren't in the array.

One "fix" would be to use KUNIT_CASE_PARAM_WITH_INIT() and have an
init function which calls kunit_register_params_array(), and then use
kunit_array_gen_params() as the generator function: this has an escape
hatch which will print the test plan.

Otherwise, as a hack, you could effectively revert
https://lore.kernel.org/linux-kselftest/20250821135447.1...@google.com/
— which would fix the issue (but break some other tests).

Going through and fixing this properly has been on my to-do list; with
some combination of fixing tests which modify the gen_params function
and improving the parsing to better handle cases without the test
plan.

Cheers,
-- David

[Ojaswin Mujoo]

Hi David,

Thanks for the workaround, KUNIT_CASE_PARAM_WITH_INIT() did the trick!

So I'm just wondering if it makes sense to still have a placeholder test
plan line in cases we can't determine the number of tests. I think something
like 1..X should be enough to not throw off the parsing. (Although I
think this might not be exactly compliant to KTAP).

Regards,
ojaswin

[David Gow]

Hmm… that could be a good idea as something to add to KTAPv2.

One other option might be to use the proposed KTAP metadata's
:ktap_test: line as a way of delimiting tests in the parser:
https://lwn.net/ml/all/20251107052926....@google.com/

In the meantime, I'm going to look into if we can update all of the
tests using KUNIT_ARRAY_PARAM() with modified gen_params, so we can
get the correct test plan in most cases.

Cheers,
-- David

[Ojaswin Mujoo]

Ohh, nice that can also be a good idea.

>
> In the meantime, I'm going to look into if we can update all of the
> tests using KUNIT_ARRAY_PARAM() with modified gen_params, so we can
> get the correct test plan in most cases.

Sure, thanks for looking into this issue and providing a quick
workaround!

Regards,
ojaswin

>
> Cheers,
> -- David