Lax or slack unit tests

20 views
Skip to first unread message

Florin Semenescu

unread,
Feb 2, 2023, 3:44:03 AM2/2/23
to PIT Users
Hello,

Can I somehow get a list of lax/slack unit tests?
I mean those unit tests which might have killed some of the generated mutants, but they didn't?
Test strength metric give us a clue of how strong the existing unit tests are in killing generated mutants, but can I pinpoint somehow the unit tests that didn't do such a great job in killing mutants?
I would like something like a metric, out of 1000-unit tests, 250 didn't kill mutants and they could have done it and here's the list of them. 

Thank you,
Florin


Henry Coles

unread,
Feb 2, 2023, 5:21:46 AM2/2/23
to pitu...@googlegroups.com
The issue with this sort of analysis is that it is incredibly expensive. Knowing if each test "could have" killed a mutant requires pitest to run all tests covering a mutant against each mutant, which makes it impractically slow for day to day development. For this reason it is not something that will ever be implemented out of the box in pitest.

Less expensive information that could be captured is a list of all killing tests. A metric of "104 out of the 1000003 available tests were required to kill these mutants" could be produced from this.

With a bit of work, your original lax/slack tests metric can be calculated though. Pitest has a "not really supported or documented" `fullMutationMatrix`mode which was added for academic use. When activated, pitest will keep running tests after a killing one is found. Details of the other tests run are available in the XML output (but not in other formats). It should be possible to extract the data you need from this.

Henry



--
You received this message because you are subscribed to the Google Groups "PIT Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pitusers+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pitusers/945ce7f4-ed3a-4fc7-98df-a92c1e109773n%40googlegroups.com.

Florin Semenescu

unread,
Feb 3, 2023, 6:07:35 AM2/3/23
to PIT Users
Thank you!
I tried with "fullMutationMatrix" but I always get:
"org.apache.maven.plugin.PluginExecutionException: Execution default-cli of goal org.pitest:pitest-maven:1.7.5:mutationCoverage failed: Full mutation matrix is only supported in the output format XML."
even if I use <sourceDataFormats><sourceDataFormat>XML.
Any idea? Am I doing something wrong? Thanks!

Henry Coles

unread,
Feb 3, 2023, 6:10:34 AM2/3/23
to pitu...@googlegroups.com
The parameter that controls the output format is `outputFormats`.

See




Reply all
Reply to author
Forward
0 new messages