Pitest sonar plugin

Showing 1-22 of 22 messages
Pitest sonar plugin Alexandre Victoor 3/17/12 2:07 AM
Hello all,
I have started to implement a very simple sonar plugin for pitest.
There is no release yet but the plugin is already usable if you have
the soul of a beta tester.
The documentation is available in sonar wiki : http://docs.codehaus.org/display/SONAR/Pitest
Let me know if you have any question on this or sonar in general.
Look forward to get your feedback
Regards

Alex
Re: Pitest sonar plugin henry 3/17/12 12:33 PM
Thanks Alex - I might finally get around to trying Sonar.
Re: Pitest sonar plugin Olivier Demeijer 3/19/12 4:07 AM
Hi Alex,
I'm using Sonar for more than 2 years now, and it's one of the most
promising plugin i've seen recently :)
It's now part of our Sonar Sandbox (a sonar instance where everyone
can evaluate pet projects, branches, ...). I'll keep you informed if
we encounter bugs, glitches, ... and share the feedback I receive of
my (sometimes really conservative) fellow devs :) Btw, i ran Pit on
two well tested (90+% coverage) : one was 100% future proof (hurray
for me :)), the other ... Well, Pit show me 5 obvious problems I
should I've seen before, so ... Hurray for Pit :)
Side note : By default, Pit violation was desactivate on the default
(customized) Sonar configuration. Don't know if it's intended or not,
buf if so, I guess that people may benefit to know it : may be you
should mention it on Pitest's Codehaus page ?
Regards,
Olivier
Re: Pitest sonar plugin Alexandre Victoor 3/19/12 1:59 PM
Hello Olivier
Thanks a lot for your message.
I know that by default pitest is deactivate in sonar. Indeed I do not
think there is a way to make it enable by default in sonar.
I have tried to stress in the wiki page the mandatory configuration
steps but I guess I still have some work to do ;)
Let me know if you have any other remark
Regards

Alex
Re: Pitest sonar plugin Olivier Demeijer 3/20/12 2:29 AM
Hello Alex,
Next step for me is to configure my maven builds in order to have PIT
report refreshed when i run a sonar:sonar ... because I'm lazy :)
Another discovery, regarding PIT, is that it's a very bad idea to run
against ... integration tests.
But so far so good, PIT *and* sonar pit plugin look really promising !
Regards,
Olivier
Re: Pitest sonar plugin henry 3/20/12 5:33 AM
I should probably add that to the FAQ.

Just to give you a heads up of a few other issues you are likely to
encounter.

1) Mutation testing does not scale well. Analysis times will increase
sharply as the size of your code base increases.

For the moment the best way to deal with this is to reduce the scope of
the analysis using one or all of the following.

a) Target only specific portions of your codebase (using the class
filters)
b) Limit the mutation operators used
c) Limit the number of mutations per class

Tweaking the number of threads used may also help.
 
2) Equivalent mutants. In my experience using PIT the prevalence of
these has been greatly over stated in the academic literature. I'm not
sure if this is is due to PIT's mutation operators (which are designed
to try and reduce the number of equivalent mutations generated), or I've
just been lucky with the code bases I've mutated. You will definitely
encounter some however.

3) Issue 24 - http://code.google.com/p/pitestrunner/issues/detail?id=24

As PIT is developed we'll be looking to make improvements in all of
these areas.

Cheers


Henry

Re: Pitest sonar plugin henry 7/26/12 4:47 AM
Hi Alexandre,

The documentation page says there is an issue with maven 3

http://docs.codehaus.org/display/SONAR/Pitest

Can you provide a bit more detail about what the problem is? I have a
friend that is looking to roll out pit with sonar for a team using maven
3.

Thanks

Henry

Re: Pitest sonar plugin Alexandre Victoor 7/29/12 12:07 PM
Hello Hery
Sorry for the delay, I am just back from vacations.
The issue with maven3 is that you need to run PIT by yourself prior to sonar. This is not a big deal since you can sill use the "reuseReport" mode descibe in the wiki section "PIT launched before Sonar".

The problem is that the sonar plugin API does not provide (yet) something similar to maven expression "${plugin.artifactMap}", hence I am not able to build the classpath needed for test execution.
Let me know if you have any other question

Alex
Re: Pitest sonar plugin Teodor Stoev 10/9/13 5:07 AM
Hi Alexandre,

Thanks for the nice Sonar plugin that you have developed. Unfortunately, I can't get it to work in my case, even though I completed all the steps described here: http://docs.codehaus.org/display/SONAR/Pitest. When I activate the plugin and start the sonar:sonar maven goal, I always get the following error message as soon as mutation testing is done:

[ERROR] Failed to execute goal org.codehaus.mojo:sonar-maven-plugin:2.1:sonar (default-cli) on project see: Can not execute SonarQube analysis: Unable to execute Sonar: Parameter 'directory' is not a directory -> [Help 1]

With the plugin disabled, the build is successful. In case it is relevant to the problem, my Maven project is multi-module. I also tried using Maven 2 instead of Maven 3 but I got the same error. I would be very grateful if you could give me a hint how to fix that.

Thanks,
Teodor


29 юли 2012, неделя, 21:07:06 UTC+2, Alexandre Victoor написа:
Re: Pitest sonar plugin Teodor Stoev 10/17/13 6:17 AM
Hi Alexandre,

I already figured it out. The sonar.pitest.reportsDirectory parameter has to be set explicitly for each project module in case target structure diverges from the default one (which is target/pit-reports). It works like a charm now!

Cheers,
Teodor

09 октомври 2013, сряда, 14:07:55 UTC+2, Teodor Stoev написа:
Re: Pitest sonar plugin Leonid Rozenblyum 10/31/13 12:43 AM
Hi Alexandre!
Thanks for great job you are doing by introducing mutation testing into Sonar world!

I have a question about not-a-green-suite.

Suppose our project has a failing test, then when I run Sonar analyzing with pitest plugin integrated it fails 
[ERROR] BUILD ERROR
[INFO] ------------------------------------------------------------------------
[INFO] Can not execute Sonar

Embedded error: fail
All tests did not pass without mutation when calculating line coverage. Mutation testing requires a green suite.
See http://pitest.org for more details.


What's an intended way to overcome this?
Sometimes we have not-green suite but we:
1) don't want to avoid Sonar analyzing of the project - the most critical...
2) in the ideal case it would be cool to have it possible pitest plugin itself working in not-a-green-suite (so only for green tests probably).

Thank you!
Re: Pitest sonar plugin henry 11/4/13 10:13 AM

Sometimes we have not-green suite but we:
1) don't want to avoid Sonar analyzing of the project - the most critical...
2) in the ideal case it would be cool to have it possible pitest plugin itself working in not-a-green-suite (so only for green tests probably).


This has been requested a few times now, so it's probably about time it made it onto the backlog. It is unlikely to make it into the next release, but might make it into 0.33.

Cheers

Henry
Re: Pitest sonar plugin Alexandre Victoor 11/4/13 12:24 PM
Hello all
Sorry (again) for the late response. I will try to figure out what happened with the jira. 
I will to improve asap the behavior of the plugin for multi modules projects on reuse report mode.
Thanks a lot again for your feedback!
Regards

Alex


--
You received this message because you are subscribed to the Google Groups "PIT Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pitusers+u...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Pitest sonar plugin Leonid Rozenblyum 11/7/13 12:19 PM
Hello Alexandre!
Just to make my case more clear: we're not using reuse reports mode.

Our Sonar analyzing is invoked by 2-step Hudson job:
mvn -X clean install -DskipTests=true 
mvn sonar:sonar

In our pom.xml
we have   <sonar.pitest.mode>active</sonar.pitest.mode>
Re: Pitest sonar plugin Artem Khvastunov 2/13/14 4:59 AM
Hi there,

Thank you for the plugin, it's awesome!
But I have a little problem when I try to use it with my project. My project contains module that doesn't have any source file and test as well. When PitestSensor tries to find report for this project it gets IllegalArgumentException("Parameter 'directory' is not a directory"). It happens because target folder of this module doesn't contains pit-reports directory at all.
So I suggest that in such case report directory just should be skipped. You can find patch with implementation of this workground in attachment. What do you think about it?

Regards,
Artem
Re: Pitest sonar plugin Dan Ignat 8/21/14 10:23 AM
Hi Teodor.
 
Can you please elaborate a bit on how you got "reuseReport" mode working with a multi-module Maven project?
 
I don't see how you can specify a different value for the sonar.pitest.reportsDirectory parameter for each child module, because that parameter can only be set once for the entire build.  There is no way to set it separately for each child module.
 
Or did you mean that you created a separate build for each child module?  (That would not be a great solution for us, because we have a lot of projects, and it would increase the number of build jobs we have to create and maintain by a factor of 5 or so.)
 
For the time being, we can't upgrade to Sonar 3.7, which means we are stuck on the 0.2 version of Alexandre's plugin.
 
Alexandre, are you aware of any way to make "reuseReport" mode work with a multi-module Maven project, while on version 0.2 of your plugin?
 
Also, out of curiosity how do later versions of your plugin handle this issue?  Do they aggregate the child module reports somehow?
 
Many thanks,
-Dan
Re: Pitest sonar plugin Alexandre Victoor 8/22/14 12:46 PM
Hello Dan
You need to use pitest maven plugin to generate xml reports and then run a sonar analysis. 

On the project I am working during my "day job", I have done this using a "multi steps" jenkins job.
For each module on which I want to run PIT, there is a maven step that runs pit.
Then there is a last step where sonar analysis is done. 

I have not changed "sonar.pitest.directory" property because default behavior (i.e. get latest generated report from module/target directory) works pretty well. 
However if you need/want to specify a different value for each submodule, it is possible! You just need to add a maven property block in each pom file, setting property "sonar.pitest.directory".
Let me know if you need more help

Alex



--
You received this message because you are subscribed to the Google Groups "PIT Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pitusers+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Pitest sonar plugin Maxime Schneider-Dufeutrelle 9/4/14 1:33 AM
Hi Alex,

I'm having an issue with multi-module projects using maven analysis (plugin version is 0.2 as 0.3 gives the weird message "Mutation in an unknown resource: org.sonar.plugins.pitest.Mutant@a2324be8" and we are using SonarQube 3.7.4). Here is the project structure :

Project-Parent (with pom)
 |---SubProject1
 |---SubProject2
 |---SubProject3

Every SubProject has his own pom too.
mvn org.pitest:pitest-maven:mutationCoverage is executing correctly on Project-Parent and produce the reports in every SubProject except in Project-Parent.

When i try to execute mvn sonar:sonar -Dsonar.pitest.mode=reuseReport the subProjects seem to work fine but the goal stop when coming to the Project-Parent : [ERROR] Failed to execute goal org.codehaus.mojo:sonar-maven-plugin:2.4:sonar (default-cli) on project Project-Parent: Can not execute SonarQube analysis: Parameter 'directory' is not a directory -> [Help 1]

As far as I understand, it's because it can't find the report in Project-Parent/target/pit-reports which is normal because it doesn't exist...

Does someone have any idea on how to solve the problem ? Is there a way to configure the fact that there is no report for the ProjectParent but only in his SubProjects ?

Thanks for your time,
Regards,

Maxime


Le vendredi 22 août 2014 21:46:57 UTC+2, Alexandre Victoor a écrit :
Re: Pitest sonar plugin Alexandre Victoor 9/4/14 5:19 AM

Hello
Have you tried  version 0.4 . It should fix the issue you have encountered.
Regards

Alex

Le 4 sept. 2014 10:33, "Maxime Schneider-Dufeutrelle" <sd.m...@gmail.com> a écrit :
Re: Pitest sonar plugin Maxime Schneider-Dufeutrelle 9/4/14 7:26 AM
Thanks Alex for your answer.

It's working fine in 0.4 BUT the Mutation Coverage tab doesn't show up anymore in sonar when you look at the sources. It still appears in the violations though. (cf attached file SonarPit.png). Is it normal ?

Regards,
Max

Le jeudi 4 septembre 2014 14:19:39 UTC+2, Alexandre Victoor a écrit :
Re: Pitest sonar plugin Dan Ignat 9/4/14 8:57 AM
For anyone who's stuck on a pre-3.7 version of Sonar and therefore is stuck on v0.2 of Alex's plugin, I found a fairly simple workaround.  When using "-Dsonar.pitest.mode=reuseReport" with a multi-module Maven project, the plugin expects the "target/pit-reports" directory to exist in the root module, and fails the build when it can't find it.
 
So all you need to do is explicitly create that directory somehow, and then the plugin won't fail the build anymore.  You can do this either in Jenkins as a Post-Build Step (Execute Shell > mkdir target/pit-reports), or in Maven pom.xml:
 
<plugin>
 <groupId>org.apache.maven.plugins</groupId>
 <artifactId>maven-antrun-plugin</artifactId>
 <version>1.7</version>
 <executions>
  <execution>
   <id>fixPITSonarPlugin0_2Bug</id>
   <phase>verify</phase>
   <goals>
    <goal>run</goal>
   </goals>
   <configuration>
    <target>
     <mkdir dir="target/pit-reports" />
    </target>
   </configuration>
  </execution>
 </executions>
</plugin>
-Dan
Re: Pitest sonar plugin Alexandre Victoor 9/4/14 12:39 PM
Hello
This is a known issue. I have never had explanations fron sonarsource folks about this problem. 
Since everything else is working properly (survived mutants violations, mutation coverage and related metrics...), I have released this version 0.4.
Regards

Alex

More topics »