Pull Request Analysis plugin is analyzing the entire codebase, and not just the PR files

[serc...@gmail.com]

plugin version: 1.4.2.1027

SonarQube version: 5.6.7


I have a ridiculously large codebase, 24000 java files. I added the github plugin as instructed here: https://docs.sonarqube.org/display/PLUG/GitHub+Plugin

But every time I make a change (even a 1 line change in just 1 file), the pull request analysis goes through the entire codebase, which takes 25 minutes. And then, when it completes, it only comments on the line of code I changed. 

This is the jenkins script that starts the analysis:

mvn sonar:sonar \

        -Dsonar.host.url=http://code-metrics.blahblah.net:9000 \

        -Dsonar.login=blahblahblahblahblahblahblahblahblahblah \

        -Dsonar.analysis.mode=preview \

        -Dsonar.github.pullRequest=${ghprbPullId} \

        -Dsonar.github.repository=blahblah/blahblah-commerce-modules \

        -Dsonar.github.oauth=blahblahblahblahblahblahblahblahblahblah \

        -Dsonar.findbugs.timeout=1800000 \

        -Dsonar.github.endpoint=https://github.blahblah.net/api/v3/

And then it adds the comment on the pull request github page for the 1 line I added as a test.

Why does it run a full analysis of my 24000 java files if I only changed 1 of them, and then it only comments on that only file?

[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 42:26 min
[INFO] Finished at: 2017-11-24T00:50:48+00:00
[INFO] Final Memory: 261M/3823M
[INFO] ------------------------------------------------------------------------
POST BUILD TASK : SUCCESS

[Julien HENRY]

Hi,

The preview mode only analyze changed files (comparing file checksum with the previous analysis on SonarQube). Look at the logs of the SonarJava analyzer to know the number of analyzed files (this is different from the number of indexed files).

Note that this optimization of analyzing only changed files has some drawback, like not being able to detect cross file issues. That's why with the new PR analysis feature of SonarQube 6.7, we decided to analyze all files, even if only a few files are updated in the P/R.

++

Julien

[serc...@gmail.com]

Thanks for your answer Julien. 

Sorry to be a pain in the ass, but I'm stuck and out of ideas. 

Is there an option where I can limit the analysis to scan only the project (root folder) where the PR files have changed, and ignore the rest of the projects? The whole analysis to all files on the codebase is taking 2 hours, and people are not very happy about waiting that long for their PR's to be approved.

Thanks again

[Julien HENRY]

Hi,

Can you clarify: is your full analysis taking 25 minutes or 2 hours?

For comparison (even if analysis duration is not linearly correlated with project size), the full analysis of the SonarQube codebase (4700 files, 250k loc) takes around 6 min on Travis-ci.org.

To better understand where is the bottleneck, could you run your analysis with parameter -Dsonar.showProfiling=true and send me (privately) the logs + all the *-profiler.properties files.

Thanks

Julien

[Sergio Castiñeyras]

Thanks for your reply

mvn sonar:sonar \

        -Dsonar.host.url=http://code-metrics.digitalriverws.net:9000 \

        -Dsonar.login=$SonarLogin \

        -Dsonar.analysis.mode=preview \

        -Dsonar.github.pullRequest=${ghprbPullId} \

        -Dsonar.github.repository=GlobalCommerce/global-commerce-modules \

        -Dsonar.github.oauth=$GitHubToken \

        -Dsonar.findbugs.timeout=1800000 \

        -Dsonar.github.endpoint=https://github.digitalriverws.net/api/v3/ \

        -Dsonar.showProfiling=true

Console log with profiling info: https://goo.gl/xcLRKL

*-profiler.properties files attached.

What bothers me is that I don't see any hashcodes on fileDataByModuleAndPath when I query http://code-metrics.digitalriverws.net:9000/batch/project?key=com.digitalriver.pacific:pacific. I have the feeling that the missing hashcodes are related to how long the job is taking.

Thanks again!

--
You received this message because you are subscribed to a topic in the Google Groups "SonarQube" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/sonarqube/NXhOvSmz4io/unsubscribe.
To unsubscribe from this group and all its topics, send an email to sonarqube+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/sonarqube/b267c098-5c2e-492c-843c-f88902c83afd%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Julien HENRY]

Hi Sergio,

Not having file hashes returned by the batch/project WS is indeed surprising. Are you using sonar.branch parameter for your regular analysis?

The duration of your build is almost 50% due to SonarJava and 50% due to Findbugs. SonarJava and Findbugs both have their own way to parse code, so you are basically doing most of the work of parsing twice. Most Findbugs rules have been implemented in SonarJava, so I would encourage you to migrate.  That would ideally allows you to divide your analysis duration by 2.

I have also reported your case to the developers of SonarJava, so that they can tell if numbers of SonarJava could be improved.

Julien Henry | SonarSource

Developer

https://sonarsource.com

[Payne.C]

Hi Julien,

I am a little confused by the message "Scanning only changed files" when using “preview” mode, but it looks still to analyze all source files in JAVA sensor step.

Could you share one more piece about "That's why with the new PR analysis feature of SonarQube 6.7"? such as which ticket to indicate this change details?

Thanks

Payne