Mutants are killed in PIT, but they are not killed in manual mutation testing.

[podr...@gmail.com]

Hi,

I used PIT with a simple test case and a program. PIT reports some mutants are killed and it seems they should not be. I made a mutant myself, build and ran JUnit with the test case. Then, it is not killed contrary to PIT.
(First, I tried with the really simple test case which contains no assertion at all.)

Target source contains
...
lastBounds = new Rectangle(0, 0, 0, 0);
...

Following mutants are generated
1. removed call to org/eclipse/swt/graphics/Rectangle::<init> → KILLED
2. Substituted 0 with 1 → KILLED
3. Substituted 0 with 1 → KILLED
4. Substituted 0 with 1 → KILLED
5. Substituted 0 with 1 → KILLED
6. Removed assignment to member variable lastBounds → KILLED


I doubt that mutant 2 is killed and I modified the target source with
...
lastBounds = new Rectangle(1, 0, 0, 0);
...

After execution, JUnit reports the test case has passed. I used JUnit 4.10 and PIT 1.1.3.

Thank you.

[Jarkko Rantavuori]

Hi,

could you link us a sample project you used so we could examine this behaviour?

Br,

-Jarkko

--
You received this message because you are subscribed to the Google Groups "PIT Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pitusers+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Henry Coles]

Have you checked the report to see which test case is killing the mutant? Is it the same one you are editing /  running through junit?

If there answer is yes then the standard answer to this is that there are two likely causes of falsely killed mutants.

1) Non repeatable unit tests such as hidden test order dependencies

If one of your tests fails when run multiple times or in an unexpected orders then it will appear to kill any mutant it is used to challenge. There's not a lot a mutation testing framework can do about this beyond try to detect such tests or forking a jvm for each test to increase isolation (very very expensive).

2) JVM "poisoning" by a mutant

If a mutant causes some state within the JVM to become corrupt then that state may cause test failures/errors for unrelated mutants inserted into the same JVM. The most common case is state held in static variables, but other types are also possible (e.g if the codebase performs bytecode manipulation it may create load a "poisoned" class).

The only 100% reliable way to prevent this is to use a separate JVM for each mutant, which unfortunately is very expensive. PIT's default behaviour is therefore to guarantee that mutants from class A are never assessed in a JVM that has contained a mutant from a class other than A.

Mutants can therefore only be affected by other mutants from the same class, but analysis cost remains relatively low.

You can enable strong mutant isolation (i.e. a jvm per mutant) by setting mutationUnitSize to 1.

If you can provide a minimal (but complete) project that reproduces the behaviour I can take a look.

Thanks

Henry

--

[henry]

Did you manage to resolve this? If so what was the issue?

Thanks

Henry

[podr...@gmail.com]

I am sorry for late response.

I found that "--mutationUnitSize 1" solves the issue. The argument is described in http://pitest.org/quickstart/maven/. However, I was executing pitest through command-line interface, and I missed it.

I was looking for how to create strong isolation between test classes as well as mutations, so I replied lately. (The issue is solved when only one test case is used. I need few more test classes to be applied.)

Thank you for your help.

2015년 1월 12일 월요일 오후 9시 8분 30초 UTC+9, henry 님의 말: