PIT and Sonar: Mutation in an unknown resource

[Gregor Jakob]

Hello everybody,

first of all I want to say that I am very happy to have found PIT which allows me to use the great concept mutation testing of our code base!

We have a Maven multimodule build and use Maven 3.0.5 and Sonar 3.7.4 for our builds (running on Jenkins but I can reproduce the problem with a local maven build and sonar run so I'll leave this out).

I successfully set up the Maven build with the pitest-maven-plugin and get nice reports :)

However I didn't succeed yet to have those results collected by Sonar:
The PitestSensor obviously finds the mutations.xml files of the the varoius Maven modules and alayzes them, but the pitest wdiget in Sonar shows "no data"

A hint for the cause of the problem might be that during the Sonar run I can see that for every mutant recorded in the XML report a warning is generated

Mutation in an unknown resource: org.sonar.plugins.pitest.Mutant@<hashcode>

From what I understand from the sourcecode of sonar-pitest the problem seems to be that the call against the sonar context resturns null (in org.sonar.plugins.pitest.PitestSensor.collectMetrics(Collection<Mutant>, SensorContext, List<ActiveRule>)

resource = context.getResource(new org.sonar.api.resources.File(mutant.getSonarJavaFileKey()));

As a consequence the mutant is processed with no metrics, wich explains why the sonar report shows no data.

Here is what I did for my set up :

Maven

Using version 0.33 I configured the Pitest plugin to generate XML output format

            <plugin>
                <groupId>org.pitest</groupId>
                <artifactId>pitest-maven</artifactId>
                <version>0.33</version>
                <configuration>                    
                    <targetClasses>
                        <param>kam.*</param>
                    </targetClasses>
                    <targetTests>
                        <param>kam.*</param>
                    </targetTests>
                    <failWhenNoMutations>false</failWhenNoMutations>
                    <outputFormats>
                        <outputFormat>XML</outputFormat>
                        <outputFormat>HTML</outputFormat>                      
                    </outputFormats>                    
                </configuration>
            </plugin>

Sonar

I followed the instructions from http://docs.codehaus.org/display/SONAR/Pitest
* I set "resuseReport" in the configuration of the Pitest plugin
* I activated the "mutant rule"

I'm greatful for any input what is going wrong and happy to provide further details if needed!

Thank you,
Gregor

[ste...@cyphoria.net]

Hi Gregor, Hi Alexandre,

 

I ran into the same issue and could resolve the issue to the same line. We are using SonarQube 4.1 with sonar-pitest plugin 0.3 - Compatibility matrix at http://docs.codehaus.org/display/SONAR/Plugin+version+matrix says plugin version 0.3 works with SonarQube 4.1 so I didn't expect any issues.

The line was changed some days prior to the 0.3 release (see https://github.com/SonarCommunity/sonar-pitest/commit/e9e9ef4fcdee7271024565bbe1fd369add4f695d).

@Gregor maybe you can switch to sonar-pitest plugin version 0.2 which should work with Sonar 3.7 according to the compatibility matrix.

 

I'll ask my admin if he can update SonarQube to 4.3 but I don't expect him to resolve this before thursday. Does anyone have any other ideas?

 

Regards

Stefan

--
You received this message because you are subscribed to the Google Groups "PIT Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pitusers+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

 

 

[Gregor Jakob]

Stop forget everything that I wrote: It concerns the version of the maven plugin instead of the sonar plugin, sorry for the confusion. Obviously it is already too late for my brain...
I'll try the old sonar plugin version tomorrow!

Thank you,
Gregor

On 19 May 2014 17:28:53 CEST, Gregor Jakob <jesu...@googlemail.com> wrote:

You are right, Stefan, Sonar 3.7.x should work with pitest 0.2x.!
I tested but the issue remains:

With versions 0.25 to 0.29 I can reproduce the issue.
With versions 0.20 to 0.24 the Maven build with pitest fails (various reasons, "Cannot construct org.pitest.mutationtest.MutationCoverageReport as it does not have a no-args constructor", Test suite fails, ...)

I also tested with the remaining 0.3x versions, also no luck here:
With 0.30 and 0.32 I can reproduce the issue
Version 0.31 lets the pitest build fail (claims test suite would be not green)

I'm happy for any futher input!

Thank you,
Gregor

--
Sent from my Android device with K-9 Mail. Please excuse my brevity.

[Gregor Jakob]

Good morning!

I tried now with the version 0.2 of the Sonar plugin and I got it to work, but only with a single module.
But before I plunge into the details let me congratulate you, Alexandre, for your awesome work with this plugin! the results look really great, that's exactly what I was looking for!

So now to my issues:

As I said I managed to run successfully a sonar analysis on a single module project. However when I try to repeat that on a multi module project the sonar run breaks with the error

Can not execute SonarQube analysis: Parameter 'directory' is not a directory

This happens already when building the root module (the one with the pom packaging)
I tried setting in sonar "sonar.pit.fail.when.no.mutations" to false, this didn't help. But I managed to successfully trick it by copying a pit-report folder in the target folder of the root module :) After this the analysis ran through all submodules and I got a look at the nice reports in Sonar :)

I am still confused about the version concerning the sonar-plugin:
Like Stefan pointed out, Sonar 3.7.4 is according to the Plugin version matrix http://docs.codehaus.org/display/SONAR/Plugin+version+matrix compatible with sonar plugin 0.2. But on http://docs.codehaus.org/display/SONAR/Pitest it is stated that version 0.3 would be "Compatible with LTS version (SonarQube 3.7.4)"

To sum up (on my machine ;)
Sonar plugin 0.2 and sonar 3.7.4 works but only for single module projects
Sonar plugin 0.3 and sonar 3.7.4 has the issue with "Mutation in an unknown resource"

I am happy for any further input :)

Thank you,
Gregor

[Alexandre Victoor]

Hello all,

Thanks for your feedback.

You are right there is an issue with version 0.3 . The good news is I have already received a PR about it. I will try to integrate it this week.

About version 0.2, there is an easy workaround for the multi modules project bug: just set property "sonar.pitest.mode" to "skip" in modules where pit should not run. 

Anyway I will try to launch a vote for a 0.4 release. I will send a mail to this mailing list in order to gather your feedback. 

Regards

Alex

[Alexandre Victoor]

Hello again,

I have integrated a fix. If you are in a hurry you can test it already with a snapshot version:

https://sonarplugins.ci.cloudbees.com/job/pitest/ws/target/sonar-pitest-plugin-0.4-SNAPSHOT.jar

Let me know if it works ok for you

Regards

Alex

[Gregor Jakob]

Hello Alex,

thank you for your quick reaction!

I have good news: The 04.-SNAPSHOT version works fine and I also managed to get the 0.2 version working for multi module with your workaround tip :)

Now the details:

Version 0.2
-----------------

Since the module that is crashing is my root (POM packaging) module it's a bit tricky: To avoid the crash I set the "sonar.pitest.mode" to "skip", but since this property is inherited by all child modules all pitest report collection by sonar are skipped X-D. What I had to do was to overwrite the property with "reuseReport" in the POMs of the child modules. Not very convenient, but, hey, it's a workaround ;)

Version 0.4-SNAPSHOT
------------------------------------

Works fine now, also with multi module :)

Just two things caught my eye In the drilldown view of sonar:
* I miss the link to "Mutations Coverage" that blends so nicely the source code with the mutants information (to be clear what I mean I pasted a screenshot of version 0.2 below)
* in the navigation columns on the top of the drilldown view "root" is shown instead of the module/package names

I'm aware that this is a snapshot version, just wanted to give you the feedback in case it is useful to you ;)

Thank you again for a nice work Alexandre!

I'm looking forward to version 0.4 :)

Greetings,
Gregor

[Gregor Jakob]

Well copy and paste doesn't work with images I see...

Hope this time it works :)

[Alexandre Victoor]

Hello

Gregor, I do not get the first point, the one about your screenshot. If you enable the Pitest widget, you will have the very same "mutation coverage" link...

About the second point, I guess this is relate to this new sonarqube version and to the changes that have been made in order to support multi languages projects.

Regards

Alex

[Gregor Jakob]

Hello Alex,

I have the Pitest widget activated and the drilldown view iI was talking about is the one that can be reached by clicking on any link in the widget.
The link I was referring to is one of those under the classname, next to "Coverage", "Duplications", "Issues" and "Source". With version 0.2 I had also one labelled "Mutant Coverage", that I don't see when using plugin verison 0.4 (see first screen shot below). By clicking on it you got this pretty view of the source code showing wich line was mutated how and if this mutant was killed or not (see second screen shot taken from result of version 0.2).

Any idea what is wrong with my setup? Or this there another way to reach this source code view with the mutant coverage information?

Thank you!

Gregor

[Stefan Penndorf]

Hi Alex, hi Gregor,

had the chance to test pitest plugin version 0.4-SNAPSHOT with Sonar 4.3 (my admin had some time to update from 4.1.7 to 4.3 but complained it was a nightmare - unit test coverage does not work yet...)
Nevertheless he also installed pitest-plugin 0.4-SNAPSHOT and it works!

Maybe the "Mutant Coverage" tab is a sonar version thing (3.7 vs 4.3) because the tab is shown in sonar 4.3 and works fine.

Alex, thanks for your work and immediate bugfix!


Stefan

--

[Alexandre Victoor]

Hello all

Thanks Stefan for these explanations. I guess you are right, there seems to be a compatibility issue with sonarqube 3.7 .

In latest versions of sonarqube the plugin API has changed a lot, this might be the root cause. Nevertheless I will try to find a workaround with sonarqube folks.

Regards

Alex

[Gregor Jakob]

Hello all,

I can confirm that this is a compatibility issue with sonarqube 3.7. I just tried the version 4.3 in a local installation and it is like Stefan says, everything works fine (also the "root" issue in the navigation columns seems to be a 3.7 thing)

I will see if it is an option for us to upgrade to sonar 4.3 :)

Thanks again for your reactivity and your work, Alex!

Good night,

Gregor

You received this message because you are subscribed to a topic in the Google Groups "PIT Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/pitusers/xqnyWpgMjDA/unsubscribe.
To unsubscribe from this group and all its topics, send an email to pitusers+u...@googlegroups.com.

[Stefan Penndorf]

Hi Gregor,

be carefull when updating. My admin told me there was some manual database migration stuff to do. But our biggest problem is the unit test coverage: Until 4.3 (or better Java Plugin 2.2) sonar did start coverage calculation and execution of unit tests on it's own. Since Java Plugin 2.2 you have to integrate the coverage analysis into your build (and provide the results). Sonar only reads the (xml) statistics. This causes us to integrate Jacoco into all our maven projects. See Upgrade Notes at http://docs.codehaus.org/display/SONAR/Java+Ecosystem

Stefan

[Gregor Jakob]

Hi Stefan,

thank you for the warning! The coverage shouldn't be a problem since we use clover and already let sonar collect the reports generated by it, but I will still check this carefully :)

Cheers,
Gregor

[Gregor Jakob]

Hello everybody!

I just wanted to let you know, that after upgrading to SonarQube 4.3 the pitest sonar plugin works perfectly :)

@Stefan: In fact the upgrade was quite easy. Since we let already, like mentioned above, Sonar collect the coverage reports of clover, we didn't have to do anything here (just install the clover plugin of course). The database migration was done automatically by sonar and worked smoothly.

@Alexandre: Do you still plan to make a relase 0.4.? Working with a snapshot version has always kind of a unstable feeling... ;)

Cheers,
Gregor

[Alexandre Victoor]

Hello

Gregor, yes I am still willing to release a version 0.4 . I am pretty busy right now, hence this (very) late response, but I will work on it as soon as I can.

Regards

Alex

--