Parameterized Test names in JUnit Jupiter

[Kenneth Wayman]

TL;DR When running JUnit Jupiter parameterized tests, we would like the display name, not the indexed method name (i.e. "testMethod[1]"), to appear in the results on the "Tests" tab. 

We are using GoCD to run API tests against our compiled product. These are implemented as a series of JUnit Jupiter Parameterized tests, each of which runs with several hundred inputs. To give more descriptive names than "testTheAPI[1...235]", we modify the DisplayName of each test method based on the input. This approach works fine in the Eclipse IDE and from the command line. However, the JUnit XML results that GoCD is processing do not populate the "name" attribute of the "test-case" element with this value, so GoCD is still displaying "testTheApi[1]" etc. on the "Tests" tab. The DisplayName IS included in the result XML, it is just NOT in the attribute the UnitTestReportGenerator expects. Short of either rewriting the reports from JUnit 5 or modifying and patching unittests.xsl, is there any way around this? 

[Chad Wilson]

Hi Kenneth

Which tool are you using to generate the "JUnit XML"? I don't believe there is actually a proper schema for JUnit XML, and that it is a de-facto "standard" that emerged from Ant originally, and then incorporated into Maven/Gradle etc in their own XML report outputs. I am aware of an attempt here to build a schema, but not actually sure which attribute might be used for this, and whether this is also part of some de-facto standard.

My inference is that GoCD seemed to move away from a focus on processing test results like this (there are just so many different approaches across language and tool ecosystems to keep up with), which is maybe why there are missing pieces like this, and also evidenced by the relatively ugly, unformatted view GoCD currently displays for the Tests tab.

Personally, I think there are better tools for generating HTML reports available via standard build automation tools (such as Gradle has), which can then be artifacted and a custom tab added (similar to this) to display that report. If GoCD did something with the XML (like tracking repeatedly failing tests over time, slow tests etc to give you more insight) it'd have more utility, but just for reporting on an individual build, I prefer to use reports generated by the build tool/plugin. This is the approach used by the GoCD team to build GoCD - based on Gradle-generated HTML reports. :-)

Contributions are welcome though - perhaps you can raise a GitHub issue with examples of the JUnit XML, and/or submit a PR alongside some of the research into extensions to the "schema" and any standardisation?

-Chad

On Tue, Mar 1, 2022 at 2:35 AM Kenneth Wayman <c00...@gmail.com> wrote:

TL;DR When running JUnit Jupiter parameterized tests, we would like the display name, not the indexed method name (i.e. "testMethod[1]"), to appear in the results on the "Tests" tab. 

We are using GoCD to run API tests against our compiled product. These are implemented as a series of JUnit Jupiter Parameterized tests, each of which runs with several hundred inputs. To give more descriptive names than "testTheAPI[1...235]", we modify the DisplayName of each test method based on the input. This approach works fine in the Eclipse IDE and from the command line. However, the JUnit XML results that GoCD is processing do not populate the "name" attribute of the "test-case" element with this value, so GoCD is still displaying "testTheApi[1]" etc. on the "Tests" tab. The DisplayName IS included in the result XML, it is just NOT in the attribute the UnitTestReportGenerator expects. Short of either rewriting the reports from JUnit 5 or modifying and patching unittests.xsl, is there any way around this? 

--
You received this message because you are subscribed to the Google Groups "go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email to go-cd+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/go-cd/9aed64b7-16d8-42d6-8775-c68fc338b4b5n%40googlegroups.com.

[Kenneth Wayman]

Hey Chad,

Thanks for your response! 

Which tool are you using to generate the "JUnit XML"? I don't believe there is actually a proper schema for JUnit XML, and that it is a de-facto "standard" that emerged from Ant originally, and then incorporated into Maven/Gradle etc in their own XML report outputs. I am aware of an attempt here to build a schema, but not actually sure which attribute might be used for this, and whether this is also part of some de-facto standard.

 

The "tool" we are using to generate the XML is JUnit5's LegacyXmlReportGeneratingListener (we've implemented our own test runner in order to get JUnit-style testing on the running system). According to the JUnit Jupiter docs, that class generates a report in an "XML format...compatible with the de facto standard for JUnit 4 based test reports that was made popular by the Ant build system". I do not know whether they are following a schema, but they aren't doing any programmatic validation. There's obviously some interest in defining an up-to-date reporting schema to support new features of JUnit5 (JUnit5 issue 373 on GitHub) but to the best of my knowledge there has been no movement on that. 

Contributions are welcome though - perhaps you can raise a GitHub issue with examples of the JUnit XML, and/or submit a PR alongside some of the research into extensions to the "schema" and any standardisation?

 

We considered submitting a patch to unittests.xsl. However, the behavior we want is not a reasonable default and I don't see a way to configure the process that creates the results page. I will open a GitHub issue just to surface the problem, though. 

For the time being we've just implemented an additional TestExecutionListener which outputs a more useful format to the console and advised the development team to ignore the "Tests" tab. 

Thanks again for your help,

~Kenneth

 

You received this message because you are subscribed to a topic in the Google Groups "go-cd" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/go-cd/PqvJrhNTeag/unsubscribe.
To unsubscribe from this group and all its topics, send an email to go-cd+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/go-cd/CAA1RwH9MGfCtx_uCo7wnoYTrO7aLCNwNpbTVnKXjWtiTCx2eGQ%40mail.gmail.com.