[PATCH 0/2] kunit: tool: Don't overwrite test status based on subtest counts

[Thomas Weißschuh]

If a subtest itself reports success, but the outer testcase fails,
the whole testcase should be reported as a failure. However the status
is recalculated based on the test counts, overwriting the outer test
result. Synthesize a failed test in this case to make sure the failure
is not swallowed.

These patches where previously part of my KUnit UAPI series [0].
I split them out, as they are useful on their own and the KUnit UAPI
series does not actually have a hard dependency on them.

[0] https://lore.kernel.org/lkml/20250717-kunit-kselft...@linutronix.de/

Signed-off-by: Thomas Weißschuh <thomas.w...@linutronix.de>
---
Thomas Weißschuh (2):
kunit: tool: Add test for nested test result reporting
kunit: tool: Don't overwrite test status based on subtest counts

tools/testing/kunit/kunit_parser.py | 3 +++
tools/testing/kunit/kunit_tool_test.py | 11 +++++++++++
.../kunit/test_data/test_is_test_passed-failure-nested.log | 10 ++++++++++
3 files changed, 24 insertions(+)
---
base-commit: a77b19b747c8a53ac16289685b0f77085a2ad553
change-id: 20251230-kunit-nested-failure-a3a56ec999ba

Best regards,
--
Thomas Weißschuh <thomas.w...@linutronix.de>

[Thomas Weißschuh]

Currently there is a lack of tests validating the result reporting from
nested tests. Add one, it will also be used to validate upcoming changes
to the nested test parsing.

Signed-off-by: Thomas Weißschuh <thomas.w...@linutronix.de>
Reviewed-by: David Gow <davi...@google.com>
Reviewed-by: Rae Moar <rm...@google.com>
---
tools/testing/kunit/kunit_tool_test.py | 10 ++++++++++
.../kunit/test_data/test_is_test_passed-failure-nested.log | 7 +++++++
2 files changed, 17 insertions(+)

diff --git a/tools/testing/kunit/kunit_tool_test.py b/tools/testing/kunit/kunit_tool_test.py
index bbba921e0eac..b74dc05fc2fe 100755
--- a/tools/testing/kunit/kunit_tool_test.py
+++ b/tools/testing/kunit/kunit_tool_test.py
@@ -165,6 +165,16 @@ class KUnitParserTest(unittest.TestCase):
self.assertEqual(kunit_parser.TestStatus.FAILURE, result.status)
self.assertEqual(result.counts.errors, 0)

+ def test_parse_failed_nested_tests_log(self):
+ nested_log = test_data_path('test_is_test_passed-failure-nested.log')
+ with open(nested_log) as file:
+ result = kunit_parser.parse_run_tests(file.readlines(), stdout)
+ self.assertEqual(kunit_parser.TestStatus.FAILURE, result.status)
+ self.assertEqual(result.counts.failed, 2)
+ self.assertEqual(kunit_parser.TestStatus.FAILURE, result.subtests[0].status)
+ self.assertEqual(kunit_parser.TestStatus.FAILURE, result.subtests[1].status)
+ self.assertEqual(kunit_parser.TestStatus.FAILURE, result.subtests[1].subtests[0].status)
+
def test_no_header(self):
empty_log = test_data_path('test_is_test_passed-no_tests_run_no_header.log')
with open(empty_log) as file:
diff --git a/tools/testing/kunit/test_data/test_is_test_passed-failure-nested.log b/tools/testing/kunit/test_data/test_is_test_passed-failure-nested.log
new file mode 100644
index 000000000000..2e528da39ab5
--- /dev/null
+++ b/tools/testing/kunit/test_data/test_is_test_passed-failure-nested.log
@@ -0,0 +1,7 @@
+KTAP version 1
+1..2
+not ok 1 subtest 1
+ KTAP version 1
+ 1..1
+ not ok 1 subsubtest 1
+not ok 2 subtest 2

--
2.52.0

[Thomas Weißschuh]

If a subtest itself reports success, but the outer testcase fails,
the whole testcase should be reported as a failure. However the status
is recalculated based on the test counts, overwriting the outer test
result. Synthesize a failed test in this case to make sure the failure
is not swallowed.

Signed-off-by: Thomas Weißschuh <thomas.w...@linutronix.de>
Reviewed-by: David Gow <davi...@google.com>

---
tools/testing/kunit/kunit_parser.py | 3 +++
tools/testing/kunit/kunit_tool_test.py | 1 +
tools/testing/kunit/test_data/test_is_test_passed-failure-nested.log | 3 +++
3 files changed, 7 insertions(+)

diff --git a/tools/testing/kunit/kunit_parser.py b/tools/testing/kunit/kunit_parser.py
index 333cd3a4a56b..5338489dcbe4 100644
--- a/tools/testing/kunit/kunit_parser.py
+++ b/tools/testing/kunit/kunit_parser.py
@@ -689,6 +689,9 @@ def bubble_up_test_results(test: Test) -> None:
elif test.counts.get_status() == TestStatus.TEST_CRASHED:
test.status = TestStatus.TEST_CRASHED

+ if status == TestStatus.FAILURE and test.counts.get_status() == TestStatus.SUCCESS:
+ counts.add_status(status)
+
def parse_test(lines: LineStream, expected_num: int, log: List[str], is_subtest: bool, printer: Printer) -> Test:
"""
Finds next test to parse in LineStream, creates new Test object,
diff --git a/tools/testing/kunit/kunit_tool_test.py b/tools/testing/kunit/kunit_tool_test.py
index b74dc05fc2fe..d5bbcb95ab6a 100755
--- a/tools/testing/kunit/kunit_tool_test.py
+++ b/tools/testing/kunit/kunit_tool_test.py
@@ -172,6 +172,7 @@ class KUnitParserTest(unittest.TestCase):
self.assertEqual(kunit_parser.TestStatus.FAILURE, result.status)
self.assertEqual(result.counts.failed, 2)
self.assertEqual(kunit_parser.TestStatus.FAILURE, result.subtests[0].status)
+ self.assertEqual(kunit_parser.TestStatus.SUCCESS, result.subtests[0].subtests[0].status)
self.assertEqual(kunit_parser.TestStatus.FAILURE, result.subtests[1].status)

self.assertEqual(kunit_parser.TestStatus.FAILURE, result.subtests[1].subtests[0].status)

diff --git a/tools/testing/kunit/test_data/test_is_test_passed-failure-nested.log b/tools/testing/kunit/test_data/test_is_test_passed-failure-nested.log
index 2e528da39ab5..5498dfd0b0db 100644
--- a/tools/testing/kunit/test_data/test_is_test_passed-failure-nested.log
+++ b/tools/testing/kunit/test_data/test_is_test_passed-failure-nested.log
@@ -1,5 +1,8 @@
KTAP version 1
1..2

+ KTAP version 1
+ 1..1

+ ok 1 test 1
not ok 1 subtest 1
KTAP version 1
1..1

--
2.52.0

[David Gow]

On Tue, 30 Dec 2025 at 20:26, Thomas Weißschuh
<thomas.w...@linutronix.de> wrote:
>
> Currently there is a lack of tests validating the result reporting from
> nested tests. Add one, it will also be used to validate upcoming changes
> to the nested test parsing.
>
> Signed-off-by: Thomas Weißschuh <thomas.w...@linutronix.de>
> Reviewed-by: David Gow <davi...@google.com>
> Reviewed-by: Rae Moar <rm...@google.com>
> ---

Thanks for splitting these out.

This is still:
Reviewed-by: David Gow <davi...@google.com>

(Note, though, that it introduces another call to test_data_path(),
which conflicts with your pytest patches.)

Cheers,
-- David

[David Gow]

On Tue, 30 Dec 2025 at 20:26, Thomas Weißschuh
<thomas.w...@linutronix.de> wrote:
>

> If a subtest itself reports success, but the outer testcase fails,
> the whole testcase should be reported as a failure. However the status
> is recalculated based on the test counts, overwriting the outer test
> result. Synthesize a failed test in this case to make sure the failure
> is not swallowed.
>
> Signed-off-by: Thomas Weißschuh <thomas.w...@linutronix.de>
> Reviewed-by: David Gow <davi...@google.com>
> ---

Thanks. Still looking good!

Reviewed-by: David Gow <davi...@google.com>

Cheers,
-- David