Differences between sonarlint and sonar server - how to scan Maven main and test properly

[Damon Jebb]

I'm looking at implementing sonarlint command line as the primary developer analysis tool, but there is a problem with differences between the results that sonarlint gives and those from sonar.  We're using sonar 5.4 currently, and I'm evaluating sonarlint 2.0 for a Maven java 7 project (I'm using java 1.8 to run sonarlint though).  I'm running sonarlint with a connection to the sonar server, which is working and I see output in the debug logging that tells me that it is using the same quality profile as the project uses on the server.  We run sonar analysis through a Jenkins job that uses the Jenkins plugin.  I have seen a number of posts that suggest that this could be due to the use of non-SSRI rules on the server, but I do not believe this to be the only factor involved.

When running sonar the rules seem to treat test code differently to that for core source.  Almost all the test case classes use the standard approach to defining a test method;

public void testMethod Throws Exception {}

On the server this does not result in the message 'Generic Exceptions should never be thrown' yet sonarlint reports this issue for these methods.

Similarly, on the sonar server we see issues reported for 'Tests should include assertions' and 'Assertions should be complete' but sonarlint does not report these.  As far as I can tell these are both standard sonarqube rules (but I may be wrong, I can't see a great deal of information that tells me the difference between non-standard rules and standard ones).  perhaps the unit test rules are not standard, which may explain the lack of issues reported for them, but the fact that sonarlint is reporting issues for the 'Never throw generic exceptions' error means that it is doing something differently as well.

I have tried narrowing the scope of what sonarlint considers to be source - I started by pointing at the Maven **/src/ folders, picking up both main and test.  Setting the value of --src to **/src/main reduces the number of issues reported, as would be expected because the test code is not being scanned, but adding --tests **/src/test does not report anything about the testing code.

Some of my differences in overall reporting of errors is likely to be due to the issue of missing dependency binaries; I have attempted to resolve this by pointing sonarlint at my .m2/repository folder, which works, but extends the time it takes to run the scan from < 1 minute to > 20 minutes (probably to be expected, it's a big repository), and does not impact the reporting of the issues discussed above, so I'm looking to understand the behaviour of sonarlint with respect to main vs test and these gross differences before examining the detail that is affected by the missing dependency binaries.

Can anyone offer any insight into how I can run a sonarlint such that it treats my project source in the same way as the server does?  That is it allows test code to throw Exception and reports unit test issues such as missing assertions?

Thanks
Damon

[Julien HENRY]

Hi Damon,

See this other thread. You should tell SonarLint for CLI what are test files using --tests command line argument:

https://groups.google.com/forum/#!topic/sonarlint/JY6Moc_mUeQ

I will update the documentation.

++

Julien

--
You received this message because you are subscribed to the Google Groups "SonarLint" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sonarlint+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/sonarlint/e69b4e7f-d61e-4d84-bc98-0687e090e546%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Julien HENRY]

I read you post a bit too fast, but still the other thread give the solution. --tests allow to specify what are the test files inside the source set. So this should be a subset of what is given to --src.

Taking your example, you should use:

--src src/** --tests src/test/**

++

Julien

[Damon Jebb]

Hi Julien,

As I mentioned in my post, I have set the --tests parameter for CLI to **/src/test but this does not result in any of the test source being scanned.  I can tell this because the number of files that sonarlint reports as having been scanned is lower than that when I use --src = **/src and there are no issues reported for the test source.

Thanks
Damon

To unsubscribe from this group and stop receiving emails from it, send an email to sonarlint+...@googlegroups.com.

[Damon Jebb]

Hi Julien,

Yes, thanks, I realised that from reading the other thread that you linked.

So, I have run using --src src/** and --tests src/test/** and this results in eliminating the issues about never thrown a generic exception, but does not raise the issues about missing assertions.  Does this mean that the assertion rules are not supported by sonarlint?

Is there a simple way in which I can tell whether a rule  in sonar is supported by sonarlint?

Many thanks
Damon

[Duarte Meneses]

Hi Damon,

There are a few things that could explain the difference:

• Files analyzed as main or test. As Julien explained, the way the patterns are used is different - in SonarLint the test pattern is used to indicate which subset of the main sources are test sources. To check if the number of sources being analyzed in SonarLint is consistent with the scanner analysis, you can see in the logs how many "main" and "test" files are being analyzed. This is more effective than comparing the number of issues. Note that many rules don't apply to test files.
  
• Third party plugins. SonarLint only supports SonarSource analyzers, which means that rules from plugins like Findbugs, Checkstyle, etc. are not used by SonarLint and therefore no issue is created for those rules.
• Resolved issues. The latest version of SonarLint CLI doesn't support hiding issues that are marked in SonarQube server as resolved (won't fix, false positive, etc.). These issues will appear in the SonarLint report.
• Classpath. As you pointed out, Scanner for Maven will automatically collect all classpath used during the compilation of the project. In SonarLint, you would need to configure this yourself. The classpath needs to include the bytecode of the project itself (that you are analyzing). If some elements of the classpath are missing, there will be issues missing.
• Exclusions. I think the latest version of the SonarLint CLI doesn't support yet loading exclusions defined in the SonarQube server. So some files or issues that are configured in the SonarQube server to be excluded from the Scanner analysis might still be analyzed by SonarLint.
• Custom rules plugins. Once again, SonarLint CLI doesn't support yet custom rules plugins.

SonarLint for Eclipse and IntelliJ have fixes for many of these limitations. May I ask why you are using SonarLint CLI instead of using the SonarLint in the IDE?

To unsubscribe from this group and stop receiving emails from it, send an email to sonarlint+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/sonarlint/bfdc9679-b7f1-4a6a-8612-fdeed050335b%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Duarte MENESES | SonarSource

http://sonarsource.com

[Damon Jebb]

Hi Duarte.

Thanks for the detailed info here.  I have been able to identify the majority of differences as due to non-supported plugins such as checkstyle, so am reasonably happy now with the results.  I have one or two oddities where an issue reported by the sonar scanner is not reported by sonarlint in one file, but the same issue (same rule) is reported for another file.  The file that is missed is not in the exclusions I'm using, indeed it's not reported even when I do not use any exclusions.  I'll post a question if I can isolate a simple demonstration of this.

I'm looking at CLI because I wanted to use this as a means of running quick CI build checks during our jenkins jobs and for developers to run the same checks locally.  Whilst IDE is useful for developers that like to use them I have a tendency to spend most of my time at the command line when it comes to building.  This is because IDE builds do not adequately reflect every aspect of a CI server build using Maven, etc.  I have been looking at using checkstyle, findbugs, PMD, etc. and was doing an exercise to make the rules used by those tools match the ones used by sonar when I decided that the effort required to maintain and synchronise the rules between the various tools was wasteful and so I looked for sonarlint.  It's a great tool, I hope that the support for non-SonarSource analysers may be forthcoming because that would obviously make the report matching with sonar much simpler.

Thanks again to you and Julien for the prompt responses.

Damon