Mutant generation how it really work?

[mark caste]

Good morning everyone,
I was inquiring about some theoretical aspects of PITest and I had these doubts about mutant generation.
From what I understand Pit generates mutants based on the rows my tests cover, so the more tests I have the more mutants.
So I expect that if I create classes and do no tests on those classes, pit will not generate mutants for these classes.
However I found two inconsistencies:
1)the first is on the web report where in the mutation coverage is 0/50 for example NotUsedClassed, so that means it created 50 mutants (the image is in attachment)

2) however running PItest with --version I get in the logs the mutants that are created and there is no reference on the classes: NotUsedClass,Player, team

How is this possible?
How then does the generation of mutants actually work in relation to the lines of code

Thank you in advance

[henry]

On Wednesday 2 October 2024 at 09:47:25 UTC+1 markc...@gmail.com wrote:

Good morning everyone,
I was inquiring about some theoretical aspects of PITest and I had these doubts about mutant generation.
From what I understand Pit generates mutants based on the rows my tests cover, so the more tests I have the more mutants.
So I expect that if I create classes and do no tests on those classes, pit will not generate mutants for these classes.

This is incorrect. Pitest generates mutants for any code matching the filters supplied to it. Line coverage information is used only to select tests to challenge the mutants with. If you have a class with no test coverage, pitest will still generate mutants. They will all be instantly marked with the special status of NO_COVERAGE, to indicate that they survived.

 

2) however running PItest with --version I get in the logs the mutants that are created and there is no reference on the classes: NotUsedClass,Player, team

I do not follow what you are trying to do here. --version is not a flag that pitest supports. The various structured outputs that pitest provides (html, xml, csv) give a definitive view of the mutants generated and assessed.

Henry

[mark caste]

For point 2) I apologise for the confusion but I meant with the flag --verbose

For point 1) I have another question regarding point 1) Does Pitest perform a mutant generation reduction technique (such as sampling, selective mutation, or mutant clustering) to choose the mutants that make sense for the application i.e., does it randomly select a representative subset of the mutants, limiting those to be examined?

Mark

[henry]

On Wednesday 2 October 2024 at 10:27:46 UTC+1 markc...@gmail.com wrote:

For point 1) I have another question regarding point 1) Does Pitest perform a mutant generation reduction technique (such as sampling, selective mutation, or mutant clustering) to choose the mutants that make sense for the application i.e., does it randomly select a representative subset of the mutants, limiting those to be examined?

Pitest does not provide random sampling. It would be easy to add via a plugin, but it has few use cases in industry. It's an idea that seems to have grown in academia to work around the limits of the early mutation testing tools which were impractically inefficient. It only really makes sense if you are trying to produce a single number metric for a codebase instead of providing actionable feedback. There is perhaps a useful version of this where the selection is based in which mutants seem "interesting" rather than representative (I believe Google's in house system does something like this), but that would be more challenging to implement.

Pitest will produce the same mutants for the same inputs each time and, for most codebases, is efficient enough to asses them all in a reasonable timeframe. When the computational cost is too much, the preferred approach is limit the number of operators or areas of code to which it is applied with human guidance.

Henry

[mark caste]

I was reading some scientific publications reporting that Kim et al. were the first to introduce Java-specific mutation operators, proposing 20 operators based on HAZOP (Hazard and Operability Studies), divided into six groups: Types/Variables, Names, Classes/Interfaces, Blocks, Expressions and others.
Next, they introduced Class Mutation, a technique for OO programmes, with 15 Java-specific operators.
I was therefore wondering if there is a theoretical correlation between the mutations applied by Pitest and these operators.
Pitest implements operators such as unary operator insertion (UOI), arithmetic operator replacement (AOR) ,... which derive from the first Mothra mutational testing software used for Fortran, but are the other mutations it applies related to some study or are they designed by Pitest?

I apologise for the purely theoretical question but it is essential for me to understand this concept for my thesis.
Thank you :)
Best regards!

[Henry Coles]

Pitest's operators were initially based on those from an earlier bytecode based mutation system - jumble. Since then more have been added (and some removed), largely without reference to any academic papers. Most/all papers suggest operators based on source code constructs. These are often difficult or impossible to replicate when working at the bytecode level

--
You received this message because you are subscribed to the Google Groups "PIT Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pitusers+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pitusers/46cdd7de-a6c3-4b32-b852-c8b8ae75fcf0n%40googlegroups.com.

[mark caste]

Good morning everyone,

from the various theoretical articles I have consulted, I have learnt that there are various mutation methods aimed at reducing the computational cost:

- Strong Mutation: this is the traditional technique, in which a mutant is considered to be ‘killed’ if the output differs from that of the original programme at the end of full execution.

- Weak Mutation: in this case, the mutant is ‘killed’ if a single component of the mutated program produces an output that differs from that of the original component, without having to execute the entire program.

- Firm Mutation: represents a compromise between the two previous modes.

I would like to know which of these approaches is used by Pitest.

Thank you in advance for your support.

Best regards,

Mark

[Jakub W]

Hello Mark,

I hope I understood your consideration correctly.

IMHO, it is impossible to classify PiTest as either heavy or weak mutation. PiTest runs all the tests in your project, just like mvn test does. So, depending on the types of tests you have, PiTest will check mutations at different levels of granularity:

1. When you have integration or E2E tests, you can classify them as heavy mutation tests.
2. If you have integration tests with a smaller scope (e.g., Service -> Repository), they can be classified as firm mutation tests.
3. When you have unit tests with a small scope, PiTest performs weak mutation tests.

I hope this helps :)

Best regards,
Jakub

[Henry Coles]

Your definitions of strong, weak and firm mutation do not match the ones I'm familiar with. Under the standard definition, pitest is a strong mutation testing system (as is pretty much every modern tool I'm aware of).

To view this discussion visit https://groups.google.com/d/msgid/pitusers/c34f391b-c59a-4725-b68c-993ea269404cn%40googlegroups.com.