Pitest sonar plugin

[Alexandre Victoor]

Hello all,
I have started to implement a very simple sonar plugin for pitest.
There is no release yet but the plugin is already usable if you have
the soul of a beta tester.
The documentation is available in sonar wiki : http://docs.codehaus.org/display/SONAR/Pitest
Let me know if you have any question on this or sonar in general.
Look forward to get your feedback
Regards

Alex

[henry]

Thanks Alex - I might finally get around to trying Sonar.

[Olivier Demeijer]

Hi Alex,
I'm using Sonar for more than 2 years now, and it's one of the most
promising plugin i've seen recently :)
It's now part of our Sonar Sandbox (a sonar instance where everyone
can evaluate pet projects, branches, ...). I'll keep you informed if
we encounter bugs, glitches, ... and share the feedback I receive of
my (sometimes really conservative) fellow devs :) Btw, i ran Pit on
two well tested (90+% coverage) : one was 100% future proof (hurray
for me :)), the other ... Well, Pit show me 5 obvious problems I
should I've seen before, so ... Hurray for Pit :)
Side note : By default, Pit violation was desactivate on the default
(customized) Sonar configuration. Don't know if it's intended or not,
buf if so, I guess that people may benefit to know it : may be you
should mention it on Pitest's Codehaus page ?
Regards,
Olivier

[Alexandre Victoor]

Hello Olivier
Thanks a lot for your message.
I know that by default pitest is deactivate in sonar. Indeed I do not
think there is a way to make it enable by default in sonar.
I have tried to stress in the wiki page the mandatory configuration
steps but I guess I still have some work to do ;)
Let me know if you have any other remark
Regards

Alex

[Olivier Demeijer]

Hello Alex,
Next step for me is to configure my maven builds in order to have PIT
report refreshed when i run a sonar:sonar ... because I'm lazy :)
Another discovery, regarding PIT, is that it's a very bad idea to run
against ... integration tests.
But so far so good, PIT *and* sonar pit plugin look really promising !
Regards,
Olivier

[henry]

I should probably add that to the FAQ.

Just to give you a heads up of a few other issues you are likely to
encounter.

1) Mutation testing does not scale well. Analysis times will increase
sharply as the size of your code base increases.

For the moment the best way to deal with this is to reduce the scope of
the analysis using one or all of the following.

a) Target only specific portions of your codebase (using the class
filters)
b) Limit the mutation operators used
c) Limit the number of mutations per class

Tweaking the number of threads used may also help.

2) Equivalent mutants. In my experience using PIT the prevalence of
these has been greatly over stated in the academic literature. I'm not
sure if this is is due to PIT's mutation operators (which are designed
to try and reduce the number of equivalent mutations generated), or I've
just been lucky with the code bases I've mutated. You will definitely
encounter some however.

3) Issue 24 - http://code.google.com/p/pitestrunner/issues/detail?id=24

As PIT is developed we'll be looking to make improvements in all of
these areas.

Cheers

Henry

[henry]

Hi Alexandre,

The documentation page says there is an issue with maven 3

http://docs.codehaus.org/display/SONAR/Pitest

Can you provide a bit more detail about what the problem is? I have a
friend that is looking to roll out pit with sonar for a team using maven
3.

Thanks

Henry

[Alexandre Victoor]

Hello Hery

Sorry for the delay, I am just back from vacations.

The issue with maven3 is that you need to run PIT by yourself prior to sonar. This is not a big deal since you can sill use the "reuseReport" mode descibe in the wiki section "PIT launched before Sonar".

The problem is that the sonar plugin API does not provide (yet) something similar to maven expression "${plugin.artifactMap}", hence I am not able to build the classpath needed for test execution.

Let me know if you have any other question

Alex

[Teodor Stoev]

Hi Alexandre,

Thanks for the nice Sonar plugin that you have developed. Unfortunately, I can't get it to work in my case, even though I completed all the steps described here: http://docs.codehaus.org/display/SONAR/Pitest. When I activate the plugin and start the sonar:sonar maven goal, I always get the following error message as soon as mutation testing is done:

[ERROR] Failed to execute goal org.codehaus.mojo:sonar-maven-plugin:2.1:sonar (default-cli) on project see: Can not execute SonarQube analysis: Unable to execute Sonar: Parameter 'directory' is not a directory -> [Help 1]

With the plugin disabled, the build is successful. In case it is relevant to the problem, my Maven project is multi-module. I also tried using Maven 2 instead of Maven 3 but I got the same error. I would be very grateful if you could give me a hint how to fix that.

Thanks,

Teodor

[Teodor Stoev]

Hi Alexandre,

I already figured it out. The sonar.pitest.reportsDirectory parameter has to be set explicitly for each project module in case target structure diverges from the default one (which is target/pit-reports). It works like a charm now!

Cheers,

Teodor

[Leonid Rozenblyum]

Hi Alexandre!

Thanks for great job you are doing by introducing mutation testing into Sonar world!

I have a question about not-a-green-suite.

Suppose our project has a failing test, then when I run Sonar analyzing with pitest plugin integrated it fails 

[ERROR] BUILD ERROR
[INFO] ------------------------------------------------------------------------
[INFO] Can not execute Sonar

Embedded error: fail
All tests did not pass without mutation when calculating line coverage. Mutation testing requires a green suite.
See http://pitest.org for more details.

What's an intended way to overcome this?
Sometimes we have not-green suite but we:

1) don't want to avoid Sonar analyzing of the project - the most critical...

2) in the ideal case it would be cool to have it possible pitest plugin itself working in not-a-green-suite (so only for green tests probably).

Thank you!

[henry]

Sometimes we have not-green suite but we:

1) don't want to avoid Sonar analyzing of the project - the most critical...

2) in the ideal case it would be cool to have it possible pitest plugin itself working in not-a-green-suite (so only for green tests probably).

This has been requested a few times now, so it's probably about time it made it onto the backlog. It is unlikely to make it into the next release, but might make it into 0.33.

Cheers

Henry

[Alexandre Victoor]

Hello all

Sorry (again) for the late response. I will try to figure out what happened with the jira. 

I will to improve asap the behavior of the plugin for multi modules projects on reuse report mode.

Thanks a lot again for your feedback!

Regards

Alex

--
You received this message because you are subscribed to the Google Groups "PIT Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pitusers+u...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Leonid Rozenblyum]

Hello Alexandre!
Just to make my case more clear: we're not using reuse reports mode.

Our Sonar analyzing is invoked by 2-step Hudson job:

mvn -X clean install -DskipTests=true 

mvn sonar:sonar

In our pom.xml

we have   <sonar.pitest.mode>active</sonar.pitest.mode>

[Artem Khvastunov]

Hi there,

Thank you for the plugin, it's awesome!

But I have a little problem when I try to use it with my project. My project contains module that doesn't have any source file and test as well. When PitestSensor tries to find report for this project it gets IllegalArgumentException("Parameter 'directory' is not a directory"). It happens because target folder of this module doesn't contains pit-reports directory at all.

So I suggest that in such case report directory just should be skipped. You can find patch with implementation of this workground in attachment. What do you think about it?

Regards,

Artem

[Dan Ignat]

Hi Teodor.

 

Can you please elaborate a bit on how you got "reuseReport" mode working with a multi-module Maven project?

 

I don't see how you can specify a different value for the sonar.pitest.reportsDirectory parameter for each child module, because that parameter can only be set once for the entire build.  There is no way to set it separately for each child module.

 

Or did you mean that you created a separate build for each child module?  (That would not be a great solution for us, because we have a lot of projects, and it would increase the number of build jobs we have to create and maintain by a factor of 5 or so.)

 

For the time being, we can't upgrade to Sonar 3.7, which means we are stuck on the 0.2 version of Alexandre's plugin.

 

Alexandre, are you aware of any way to make "reuseReport" mode work with a multi-module Maven project, while on version 0.2 of your plugin?

 

Also, out of curiosity how do later versions of your plugin handle this issue?  Do they aggregate the child module reports somehow?

 

Many thanks,

-Dan

[Alexandre Victoor]

Hello Dan

You need to use pitest maven plugin to generate xml reports and then run a sonar analysis. 

On the project I am working during my "day job", I have done this using a "multi steps" jenkins job.

For each module on which I want to run PIT, there is a maven step that runs pit.

Then there is a last step where sonar analysis is done. 

I have not changed "sonar.pitest.directory" property because default behavior (i.e. get latest generated report from module/target directory) works pretty well. 

However if you need/want to specify a different value for each submodule, it is possible! You just need to add a maven property block in each pom file, setting property "sonar.pitest.directory".

Let me know if you need more help

Alex

--
You received this message because you are subscribed to the Google Groups "PIT Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pitusers+u...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Maxime Schneider-Dufeutrelle]

Hi Alex,

I'm having an issue with multi-module projects using maven analysis (plugin version is 0.2 as 0.3 gives the weird message "Mutation in an unknown resource: org.sonar.plugins.pitest.Mutant@a2324be8" and we are using SonarQube 3.7.4). Here is the project structure :

Project-Parent (with pom)

 |---SubProject1

 |---SubProject2

 |---SubProject3

Every SubProject has his own pom too.

mvn org.pitest:pitest-maven:mutationCoverage is executing correctly on Project-Parent and produce the reports in every SubProject except in Project-Parent.

When i try to execute mvn sonar:sonar -Dsonar.pitest.mode=reuseReport the subProjects seem to work fine but the goal stop when coming to the Project-Parent : [ERROR] Failed to execute goal org.codehaus.mojo:sonar-maven-plugin:2.4:sonar (default-cli) on project Project-Parent: Can not execute SonarQube analysis: Parameter 'directory' is not a directory -> [Help 1]

As far as I understand, it's because it can't find the report in Project-Parent/target/pit-reports which is normal because it doesn't exist...

Does someone have any idea on how to solve the problem ? Is there a way to configure the fact that there is no report for the ProjectParent but only in his SubProjects ?

Thanks for your time,

Regards,

Maxime

[Alexandre Victoor]

Hello
Have you tried  version 0.4 . It should fix the issue you have encountered.
Regards

Alex

[Maxime Schneider-Dufeutrelle]

Thanks Alex for your answer.

It's working fine in 0.4 BUT the Mutation Coverage tab doesn't show up anymore in sonar when you look at the sources. It still appears in the violations though. (cf attached file SonarPit.png). Is it normal ?

Regards,

Max

[Dan Ignat]

For anyone who's stuck on a pre-3.7 version of Sonar and therefore is stuck on v0.2 of Alex's plugin, I found a fairly simple workaround.  When using "-Dsonar.pitest.mode=reuseReport" with a multi-module Maven project, the plugin expects the "target/pit-reports" directory to exist in the root module, and fails the build when it can't find it.

 

So all you need to do is explicitly create that directory somehow, and then the plugin won't fail the build anymore.  You can do this either in Jenkins as a Post-Build Step (Execute Shell > mkdir target/pit-reports), or in Maven pom.xml:

 

<plugin>
 <groupId>org.apache.maven.plugins</groupId>
 <artifactId>maven-antrun-plugin</artifactId>
 <version>1.7</version>
 <executions>
  <execution>
   <id>fixPITSonarPlugin0_2Bug</id>
   <phase>verify</phase>
   <goals>
    <goal>run</goal>
   </goals>
   <configuration>
    <target>
     <mkdir dir="target/pit-reports" />
    </target>
   </configuration>
  </execution>
 </executions>
</plugin>

-Dan

[Alexandre Victoor]

Hello

This is a known issue. I have never had explanations fron sonarsource folks about this problem. 

Since everything else is working properly (survived mutants violations, mutation coverage and related metrics...), I have released this version 0.4.

Regards

Alex

[Annushree Satpathy]

Hi Team,

We are facing resource crunch issue in the continous integration pipe after configuring pitest. Pitest build is running for a longer duration (almost 16hrs) and other builds are getting slow due to lack of resource.

Is there any possible solutions available to make the pitest build faster.

We need this help urgently.

Regards,

Annushree

[henry]

On Thursday, 2 July 2015 06:00:51 UTC+1, Annushree Satpathy wrote:

We are facing resource crunch issue in the continous integration pipe after configuring pitest. Pitest build is running for a longer duration (almost 16hrs) and other builds are getting slow due to lack of resource.

Is there any possible solutions available to make the pitest build faster.

I'm afraid I can't offer any help specific to the sonar plugin, but this is the general advice on scaling mutation testing :-

The most effective point to apply mutation testing is while coding. i.e developers should use it to support their work and apply it locally.

When working with mutation testing locally there is no need to mutate the entire codebase, just the section of code you are working on. This can be done either by using the filters, or using the version control integration to only analyse modified classes.

Mutation testing can be applied to a codebase of any size using this approach.

Scaling on a CI server is harder. If your codebase is very large you will need to make some compromises, but there are some things that can be checked/tuned first

1) Make sure you are using an appropriate number of threads - usually equal to the number of CPUs/cores on your CI server.
2) Make sure you are filtering out any logging or assertion libraries you are using
3) Make sure any exceptionally slow tests are being excluded
4) Try reducing the timeout constant

Once these have been optimised the next thing to consider is using a history file. This can be difficult to setup on a CI server, but will significantly reduce the mutation analysis time after the first analysis.

If you've done all of the above the only option is to mutate less. This means either reducing the number mutation operators, or limiting the analysis to the parts of the code you really care about, or both. You can also limit the maximum number of mutations per class.

Apart from the above the only other advice I can give is to make sure your code and tests are clean. Re-factoring imperative loops to a more functional style makes code more readable and less mutable. Fast unit tests that run in milliseconds will kill mutants far faster than slower tests tied to frameworks like spring.  Time spent mutating unused code that could be deleted is time entirely wasted.

Henry