SonarQube vs FindBugs, CheckStyle, PMD

[Brian Sperlongano]

Hello!

I was wondering what the differences are between the SonarQube Java analyzer versus FindBugs/CheckStyle/PMD.  I'm a long-time SonarQube user and I always thought that the Java analyzer included those 3 analyzers - but I see here in this group plugin updates for them, which implies they are separate.  What are the differences / overlap between them?  My team is trying to come up with a "most comprehensive" quality analyzer and it's not clear what the "right" collection of tools are for best quality.

Thanks,

Brian

[G. Ann Campbell]

Hi Brian,

On the assumption that you're not just troll-baiting... :-)

At the dawn of time, or at least the dawn of Sonar(Qube) there was no Java analyzer (which is now known as SonarJava). Instead, the founders incorporated the output of the three external tools you mentioned: FindBugs, PMD, and Checkstyle.

Then bug reports and requests for changes started to roll in - for those tools. And we couldn't respond adequately to those requests because they weren't for code we controlled. So the work on our analyzers started. 

Fast forward to today, and we've replaced all the valuable Checkstyle and PMD rules and a large majority of the FindBugs rules, in addition to creating many, many rules not seen in any of those tools.

In short, we feel that SonarJava is the only rule engine you need.

Ann

[Nicolas Peru]

Hi, 

Just to add a few things to what Ann clearly described and for eventual further readers

• The linked blog post is beginning to get old (almost 2 years old now...) and we are doing way more than what FindBugs is doing now. (and if you think I am wrong I am more than happy to hear about it). 
• To clarify : every rule starting with squid:SXXX comes from the SonarJava analyzer. All the code of SonarJava is on github and the checks implementation are available for you to check them.
• We claim to be better than those tools but we are happy to not be taken for granted and get challenged. And we are definitely not done on how we can improve SonarJava (Cross files analysis for instance is most probably gonna be a hot topic for the upcoming year for us). 

HTH, 

Cheers

--
You received this message because you are subscribed to the Google Groups "SonarQube" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sonarqube+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/sonarqube/413ce2ff-7d99-4245-a7dc-1907fd34f47e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Nicolas PERU | SonarSource

Senior Developer
http://sonarsource.com

https://twitter.com/benzonico

[Brian Sperlongano]

Thank you Ann and Nicolas for your replies.  Very helpful responses.

[Brian Sperlongano]

Hi,

Just to expand a bit on the question and rationale; we've started using the SQ FindBugs plugin as a complementary addition to the SQ Java analyzer, and we've found that FB occasionally picks up on some pretty legitimate issues that aren't found in the SQ Java analyzer.  Is the SQ team interested in these examples in order to improve the java analyzer?

Thanks,

Brian

[Richard Kettelerij]

I'd like to chime in on this. In our codebase we make heavy use of the FindBugs @DefaultAnnotation(NonNull.class) annotation along with @CheckForNull (available in the edu.umd.cs.findbugs.annotations and/or javax.annotation package). FindBugs uses these annotations in its static code analysis to warn about potential NPE's. This works pretty well. It prevents *a lot* of accidental NPE's from happening. You're saying SQ has gone beyond FindBugs in terms of capabilities but I haven't been able to find this feature yet. It would be great if SonarQube/SonarLint was able to parse these annotations and provide similair functionality.

[Nicolas Peru]

Hi Richard, 

Thanks for asking !

If I understand correctly, SonarJava (the analyzer used by both SQ and SonarLint) is already doing what you ask for. We already support some of the annotations defined by the JSR-305 that was driven by FindBugs at the time (and so  we rely on javax.annotation rather than vendor specific annotations). 

(side note : defaultAnnotation is deprecated btw). 

What is supported today in the symbolic execution engine (and thus in the rule about null pointer dereference S2259) are 

javax.annotation.ParametersAreNonnullByDefault (at package level)

javax.annotation.ParametersAreNullableByDefault (at package level)

javax.annotation.CheckForNull

javax.annotation.Nullable

And we plan to support more with the following : https://jira.sonarsource.com/browse/MMF-664

I do hope that this shed a bit of light. 

Cheers, 

To view this discussion on the web visit https://groups.google.com/d/msgid/sonarqube/604397e1-a7f8-4397-86a3-92d6282b5221%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Roman Ivanov]

Hi ALL,

by default usage of only sonarqube is ok, better then no analysis at all.

But sonarqube lacks a lot of rules, not all cases are detected, a lot of false-positives, customization of rules is hard and sometime impossible.

If user want to go further, extra tools need to be used. 

I hope in many years Sonar will cover all cases, but it is not how it is now, and I doubt it ever happen.

I am using Sonar for many many years (almost from the begging) and I love it. In same time I am developing Checkstyle. 

From my long experience of fighting with technical dept, I could say - Use all tools !!! 

There are even more tools beyong FB, PMD and Checkstyle.

In checkstyle development we use all of them (FB, PMD, Checkstyle, finbiddenapi, IntelijIdea inspections, nondex,  .......). 

If you got complain from all tools about the same - it is even better , as you sure that this need to be fixed. By fix one code you remove few violations.

All this tools are written by humans, so all of them could miss a problematic case. 

Non of them are ideal. All of them have unique features/rules.

If you use all of them, chance to miss is very small. 

thanks,

Roman Ivanov

[Tibor Blenessy]

Hello Roman,

I would like to point out that your statements are highly opinion based. If you have concrete examples of "lot of false-positives", reporting them is appreciated. 

SonarJava analyzer (shipped by default with SonarQube platform) currently covers industry recognized standards like CWE, CERT, ... Details can be found here https://www.sonarsource.com/why-us/products/languages/java.html

This is of course continuing effort and improving with every release.

I would be also careful with using all the possible tools. What we try to achieve with SonarQube is to have good noise-to-signal ratio, so developer is not overwhelmed and can effectively act on accumulation of technical debt.

Best regards 

--

You received this message because you are subscribed to the Google Groups "SonarQube" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sonarqube+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/sonarqube/71e42124-d317-43c1-abad-8107d19127aa%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Tibor BLENESSY | SonarSource

Software Engineer

http://sonarsource.com

[climba...@gmail.com]

I don't believe Sonarqube is difficult to implement rules in.

Using the basic XML rules our own teams have added 30 using XPath with no effort.

I myself found it quite easy to create a Java rule for detecting XXE vulnerabilities.

[jordan....@gmail.com]

Hi Tibor,

Noob question here. In SonarQube we can see a great difference between the amount of rules proposed by the Sonar Way Quality Profile (154) compared to FindBugs (374) and FindBugs+FB-Contrib (709).

Are there any concerns about that or do you consider this is still part of the noise-to-signal ratio thing ?

Thanks for your help

[Tibor Blenessy]

Hello Jordan,

I am obviously bit biased to answer that question, but lot of effort was put in last few years to match Findbugs, and I think results we have now are comparable if not better. However you should try for yourself (and for sure let us know if you think we could do better, we are listening to feedback!).

Best regards

To view this discussion on the web visit https://groups.google.com/d/msgid/sonarqube/8a669392-c90e-44a8-8cc8-a67393986b64%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Tibor Blenessy | SonarSource

SonarJava Developer

http://sonarsource.com 

[Nicolas Peru]

Hi, 

Regarding your question about number of rules between findbugs and Sonar Way profile : sonar way profile is the "editor pick" of the rules that should be activated by default because we think that those rule are applicable regardless of your specific context. So these are not all the rules implemented by SonarJava (the other rules might makes sense only in some context and are to be activated depending on your preferences). 

Regarding number of rules : we have a count of 423 as of latest release : https://sonarcloud.io/organizations/default/rules#languages=java 

but comparing this number with the number you quote from FindBugs and fb contrib is comparing apples and carrots : the mapping is far from one rule to one rule between the tools (it always depend on how you decide to group or not patterns and our philosophy is a bit different from FindBugs). 

When getting some inspiration from FindBugs (two years ago now... ) we grouped quite some detectors under one rule (for instance https://sonarcloud.io/organizations/default/rules#rule_key=squid%3AS2275 covers more than 10 Findbugs rules). 

So, I am a lot biased here ;) but as of today I think we are doing way better than Findbugs in terms of detection, false positive ratio and even on number of rules in the end (we will detect more precisely more patterns, and this without mentioning precision in reporting them and documenting those issues). Regarding fb-contrib, I know we have some overlap but I have no idea how much from the top of my head, I'll be happy to know if you miss something from fb-contrib in SonarJava. 

Hope that helps.

Cheers, 

--

Nicolas Peru | SonarSource

Senior Developer

http://twitter.com/benzonico

http://sonarsource.com