generic test data and puppet plugin

[Johan De Wit]

Hi,

Just confused how to get this working :

• Consolidated "coverage" features
• SonarQube merges the unit, integration and overall coverage metrics into a single set of "coverage" metrics to have a unified experience across the different types of projects, whatever the language(s), and whatever the test types
• Following that direction, SonarQube has now built-in support for its own test and coverage report format - which was previously brought by the Generic Coverage plugin. When coverage tools implement that format natively, integrating their reports will be easier than ever

Reading this I should think it should be working out of the box using the right format ....

I have created an rspec custom formatter that generates the generic test data format for unit tests.  But when I try to import it, is seems to get ignored, but parsed.

Which you can find here https://github.com/witjoh/rspec_sonar_formatter

From the logs :

09:37:01.125 DEBUG: Sensors : Generic Test Executions Report -> Puppet Squid Sensor -> SonarJavaXmlFileSensor -> Analyzer for "php.ini" files -> Zero Coverage Sensor -> CPD Block Indexer
09:37:01.125 INFO: Sensor Generic Test Executions Report
09:37:01.125 WARN: Property 'sonar.genericcoverage.unitTestReportPaths' is deprecated. Please use 'sonar.testExecutionReportPaths' instead.
09:37:01.125 INFO: Parsing /vagrant/playground/workflow/sonar_demo/puppetlabs-ntp/junit/reports_sonar.xml
09:37:01.184 INFO: Imported test execution data for 0 files
09:37:01.184 INFO: Test execution data ignored for 2 unknown files, including:
./spec/classes/contains_spec.rb
./spec/classes/ntp_spec.rb
09:37:01.184 INFO: Sensor Generic Test Executions Report (done) | time=59ms

Puppet code does have *.pp, and the test files have *spec.rb extensions ....  

Could that be the reason the reports are ignored ? 

And what do we need to do to connect the right test (_spec.rb) file to the right manifest (.pp) file ?

The docs about generic test data are really sparse .... for me ...

I'm really lost now, and  if someone could put me on the right track to make this work, I will be very thankfull ...

Grts

Johan

[nicolas...@sonarsource.com]

Hi Johan,

The first area I suggest you check on your side is based on the following aspect (quote from documentation):

(warning) Unlike for coverage reports, the files present in the report must be test file names, not source code files covered by tests.

What you need to understand at this stage is how SonarQube actually treats your test files during scanner execution. You can check for the 'Test paths' log in your scanner output, and/or also simply check the path to test files in SonarQube.

Once you've got that, then correlate it to the paths that you're setting in your Generic Test Data XML , and see whether it matches. If any remaining concern, then sharing an actual example XML with corresponding scanner output might help you get further inputs.

Best regards,

Nicolas

[Johan De Wit]

Thanks for the reply.

Here is a shortened version of the xml:
<testExecutions version="1">
  <file path="./spec/classes/contains_spec.rb">
    <testCase name="should compile into a catalogue without dependency cycles" duration="2.837896645"/>
--
    <testCase name="should contain File[foo.rb] that notifies Service[ntp]" duration="0.004804526"/>
  </file>
  <file path="./spec/classes/ntp_spec.rb">
    <testCase name="should compile into a catalogue without dependency cycles" duration="0.092275344"/>
--
    <testCase name="disallows large clock skews" duration="0.081455687"/>
  </file>
</testExecutions>

And those are indeed the test files used.

I prepared a demo project at https://github.com/witjoh/puppetlabs-ntp/tree/sonarqube

At least the sonar specific settings can be reviewed ...

Grts

[nicolas...@sonarsource.com]

Hi Johan,

Your sonar-project.properties shows that you're only defining sonar.sources , which means that all code in there will be treated by SonarQube as production source code (including these rb test files). Per previous post, you need to get these test files to be treated by SonarQube as test code, and for that you should have a look at the sonar.tests analysis parameter. Let me know how that goes.

Best regards,

Nicolas

[Nicolas Bontoux]

Hi Johan,

Please make sure to keep the group included in your responses, that way the overall community can benefit from this discussion and related findings.

I confirm that the test files must be mapped to a specific language analyzer, you'll notice in fact that with v6.7 the analysis currently fails with: Line 2 of report refers to a file with an unknown language: ./spec/classes/contains_spec.rb

Sounds like you need to follow-up with Puppet/Ruby Community Plugin maintainers to discuss how this could all play together.

Best regards,

Nicolas

On Thu, 8 Feb 2018 at 14:27 Johan De Wit <jozo...@gmail.com> wrote:

Still no results ...

Could the cause be thet puppet only parses *.pp files, and the test files are *.rb, sonarqube cannot match tests files against source files ?

Thanks for helping

--
You received this message because you are subscribed to a topic in the Google Groups "SonarQube" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/sonarqube/hON5Z4dfWIA/unsubscribe.
To unsubscribe from this group and all its topics, send an email to sonarqube+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/sonarqube/8a93b706-4a00-41f0-8eb4-85f18a082625%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Nicolas Bontoux | SonarSource

Support Engineer

http://sonarsource.com

[Johan De Wit]

Ok, I will try to get in touch with them ... and keep this thread updated ..

thanks

On Fri, Feb 9, 2018 at 1:08 PM, Nicolas Bontoux <nicolas...@sonarsource.com> wrote:

Hi Johan,

Please make sure to keep the group included in your responses, that way the overall community can benefit from this discussion and related findings.

I confirm that the test files must be mapped to a specific language analyzer, you'll notice in fact that with v6.7 the analysis currently fails with: Line 2 of report refers to a file with an unknown language: ./spec/classes/contains_spec.rb

Sounds like you need to follow-up with Puppet/Ruby Community Plugin maintainers to discuss how this could all play together.

Best regards,

Nicolas

On Thu, 8 Feb 2018 at 14:27 Johan De Wit <jozo...@gmail.com> wrote:

Still no results ...

Could the cause be thet puppet only parses *.pp files, and the test files are *.rb, sonarqube cannot match tests files against source files ?

Thanks for helping

To unsubscribe from this group and all its topics, send an email to sonarqube+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/sonarqube/8a93b706-4a00-41f0-8eb4-85f18a082625%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Venkata Nandagiri]

Please let me know, if it get resolved. I have been facing same issue and could not show the junit reports in sonar qube.

On Friday, 9 February 2018 06:20:36 UTC-6, Johan De Wit wrote:

Ok, I will try to get in touch with them ... and keep this thread updated ..

thanks

On Fri, Feb 9, 2018 at 1:08 PM, Nicolas Bontoux <nicolas...@sonarsource.com> wrote:

Hi Johan,

Please make sure to keep the group included in your responses, that way the overall community can benefit from this discussion and related findings.

I confirm that the test files must be mapped to a specific language analyzer, you'll notice in fact that with v6.7 the analysis currently fails with: Line 2 of report refers to a file with an unknown language: ./spec/classes/contains_spec.rb

Sounds like you need to follow-up with Puppet/Ruby Community Plugin maintainers to discuss how this could all play together.

Best regards,

Nicolas

On Thu, 8 Feb 2018 at 14:27 Johan De Wit <jozo...@gmail.com> wrote:

Still no results ...

Could the cause be thet puppet only parses *.pp files, and the test files are *.rb, sonarqube cannot match tests files against source files ?

Thanks for helping

To unsubscribe from this group and all its topics, send an email to sonarqube+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/sonarqube/8a93b706-4a00-41f0-8eb4-85f18a082625%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.