Multiple issues in Sonarqube Scanner for Gradle for Android projects
m...@thomaskeller.biz
apply plugin: 'org.sonarqube'
allprojects {
group = libGroupId
version = libVersionName
}
sonarqube {
properties {
property 'sonar.scm.provider', 'git'
property 'sonar.scm.url', 'scm:git:path/to/git/repo'
}
}
\:demo-app.sonar.java.test.binaries=path/to/demo-app/build/intermediates/classes/test/debug
\:player-exo.sonar.java.test.binaries=path/to/player-exo/build/intermediates/classes/test/debug
\:player-ui.sonar.java.test.binaries=path/to/player-ui/build/intermediates/classes/test/debug
\:player.sonar.java.test.binaries=path/to/player/build/intermediates/classes/test/debug
player-ui and player-exo actually have Android Instrumentation tests, but their binary path is "$MODULE/build/intermediates/classes/androidTest/debug". Furthermore, neither "demo-app" nor "player" have JVM-based unit tests, so this configuration is completely bogus.
We're using Retrolambda and the plugin seems to pick this up, but again incomplete:
$ grep retrolambda dump.properties
\:demo-app.sonar.binaries=/Users/thomas/Entwicklung/Kunden/maxdome/android-videoplayer/demo-app/build/retrolambda/debug
\:demo-app.sonar.java.binaries=/Users/thomas/Entwicklung/Kunden/maxdome/android-videoplayer/demo-app/build/retrolambda/debug
It only finds retrolambda for the "demo-app" module, but actually there is such a directory for every module:
$ find . -name "retrolambda"
./demo-app/build/retrolambda
./player/build/retrolambda
./player-exo/build/retrolambda
./player-ui/build/retrolambda
What the plugin also completely misses is player-{ui,exo}/build/retrolambda/debugAndroidTest, i.e. retrolambda output for android tests.
Android projects also often run lint on their sources, so do we. The Sonarqube scanner however does not find these files automatically:
$ find . -name "lint-results*.xml"
./demo-app/build/outputs/lint-results-debug.xml
./player/build/outputs/lint-results-debug.xml
./player-exo/build/outputs/lint-results-debug.xml
./player-ui/build/outputs/lint-results-debug.xml
and consequently does not configure them for "sonar.android.lint.report". For flavored builds these files carry the flavor name as well, i.e. "build/outputs/lint-results-<camelCasedFlavorName>{Debug,Release}.xml
Furthermore, neither Android-Test results nor Code Coverage files of those tests are found automatically. These usually reside in "build/outputs/androidTest-results/connected" (for non-flavored builds, otherwise in "build/outputs/androidTest-results/connected/flavors/<UPPERCASE_FLAVOR_NAME>") and "build/outputs/code-coverage/connected/<DEVICE_OR_EMULATOR_NAME>-coverage.ec" (for non-flavored builds, otherwise "build/outputs/code-coverage/connected/flavors/<UPPERCASE_FLAVOR_NAME>/<DEVICE_OR_EMULATOR_NAME>-coverage.ec").
Regarding code coverage: Modern Android projects (and we're talking about nothing else) can only use Jacoco as code coverage tool, as this is the only coverage tool that is supported by Android's instrumentation and Unit tests, so this plugin should be configured as default for Android projects:
sonar.core.codeCoveragePlugin=jacoco
Here another feature request: Since https://jira.sonarsource.com/browse/SONAR-4101 is still not solved (having the ability to report JVM test and instrumentation test results at the same time), you might want to do some magic within the plugin and copy the test results of both tests (JVM and Instrumentation tests) to a common place (say "build/sonarqube/combined-test-results") that is then set as "sonar.junit.reportsPath".
When using modules in sub-directories (i.e. notated through ":libs/foo", ":libs/bar" in settings.gradle), the plugin creates weird paths in the resulting properties file:
\:libs.\:libs\:foo.<some-sonar-property>
\:libs.\:libs\:bar.<some-sonar-property>
I'd have expected something along these lines:
\:libs\:foo.<some-sonar-property>
\:libs\:bar.<some-sonar-property>
or even just
\:foo.<some-sonar-property>
\:bar.<some-sonar-property>
So, in the current state the plugin is not very helpful for Android analysis, because too many things are missing or even wrong and one should rather reside to manual configuration of analysis parameter.
@Sonarsource: If you need help figuring out more of the nitty-gritty details, contact me via PM. I really want to see first-class Android support in Sonarqube Gradle analysis and am here to help.
Thomas.
Julien HENRY
Le mardi 27 septembre 2016 09:45:28 UTC+2, m...@thomaskeller.biz a écrit :
I'm trying to analyze an Android multi-module project with the Sonarqube Gradle Scanner, version 2.1. There are four Gradle modules on the top level, "player", "player-ui", "player-exo" and "demo-app":
The analysis parameters that result from such a configuration however are largely incomplete or even wrong:$ ./gradlew sonarqube -DsonarRunner.dumpToFile=dump.properties$ grep test.binaries dump.properties\:demo-app.sonar.java.test.binaries=path/to/demo-app/build/intermediates/classes/test/debug
\:player-exo.sonar.java.test.binaries=path/to/player-exo/build/intermediates/classes/test/debug
\:player-ui.sonar.java.test.binaries=path/to/player-ui/build/intermediates/classes/test/debug
\:player.sonar.java.test.binaries=path/to/player/build/intermediates/classes/test/debug
player-ui and player-exo actually have Android Instrumentation tests, but their binary path is "$MODULE/build/intermediates/classes/androidTest/debug".
public interface TestedVariant {
/**
* Returns the build variant that will test this build variant.
*
* <p>The android test variant exists only for one build type, by default "debug". This is
* controlled by {@link TestedAndroidConfig#getTestBuildType}.
*/
@Nullable
TestVariant getTestVariant();
/**
* Returns the build variant that contains the unit tests for this variant.
*/
@Nullable
UnitTestVariant getUnitTestVariant();
}
Furthermore, neither "demo-app" nor "player" have JVM-based unit tests, so this configuration is completely bogus.
We're using Retrolambda and the plugin seems to pick this up, but again incomplete:
$ grep retrolambda dump.properties
\:demo-app.sonar.binaries=/Users/thomas/Entwicklung/Kunden/maxdome/android-videoplayer/demo-app/build/retrolambda/debug
\:demo-app.sonar.java.binaries=/Users/thomas/Entwicklung/Kunden/maxdome/android-videoplayer/demo-app/build/retrolambda/debug
It only finds retrolambda for the "demo-app" module, but actually there is such a directory for every module:[...]
What the plugin also completely misses is player-{ui,exo}/build/retrolambda/debugAndroidTest, i.e. retrolambda output for android tests.
Android projects also often run lint on their sources, so do we. The Sonarqube scanner however does not find these files automatically:
and consequently does not configure them for "sonar.android.lint.report". For flavored builds these files carry the flavor name as well, i.e. "build/outputs/lint-results-<camelCasedFlavorName>{Debug,Release}.xml
Furthermore, neither Android-Test results nor Code Coverage files of those tests are found automatically. These usually reside in "build/outputs/androidTest-results/connected" (for non-flavored builds, otherwise in "build/outputs/androidTest-results/connected/flavors/<UPPERCASE_FLAVOR_NAME>") and "build/outputs/code-coverage/connected/<DEVICE_OR_EMULATOR_NAME>-coverage.ec" (for non-flavored builds, otherwise "build/outputs/code-coverage/connected/flavors/<UPPERCASE_FLAVOR_NAME>/<DEVICE_OR_EMULATOR_NAME>-coverage.ec").
Regarding code coverage: Modern Android projects (and we're talking about nothing else) can only use Jacoco as code coverage tool, as this is the only coverage tool that is supported by Android's instrumentation and Unit tests, so this plugin should be configured as default for Android projects:
sonar.core.codeCoveragePlugin=jacoco
Here another feature request: Since https://jira.sonarsource.com/browse/SONAR-4101 is still not solved (having the ability to report JVM test and instrumentation test results at the same time), you might want to do some magic within the plugin and copy the test results of both tests (JVM and Instrumentation tests) to a common place (say "build/sonarqube/combined-test-results") that is then set as "sonar.junit.reportsPath".
When using modules in sub-directories (i.e. notated through ":libs/foo", ":libs/bar" in settings.gradle), the plugin creates weird paths in the resulting properties file:
\:libs.\:libs\:foo.<some-sonar-property>\:libs.\:libs\:bar.<some-sonar-property>
So, in the current state the plugin is not very helpful for Android analysis, because too many things are missing or even wrong and one should rather reside to manual configuration of analysis parameter.
@Sonarsource: If you need help figuring out more of the nitty-gritty details, contact me via PM. I really want to see first-class Android support in Sonarqube Gradle analysis and am here to help.
Thomas.
Thomas Keller
Hi Julien!
Julien HENRY schrieb:
> [snip]
>
>
> player-ui and player-exo actually have Android Instrumentation
> tests, but their binary path is
> "$MODULE/build/intermediates/classes/androidTest/debug".
>
>
> Here is how I'm collecting those properties. Maybe you would find
> something wrong. For any module, I look for the first variant having
> "debug" buildType
> API), I get the "UnitTestVariant" from it
> sure about the difference. the Javadoc is not clear to me:
>
> public interface TestedVariant {
>
> /**
> * Returns the build variant that will test this build variant.
> *
> * <p>The android test variant exists only for one build type, by default "debug". This is
> * controlled by {@link TestedAndroidConfig#getTestBuildType}.
> */
> @Nullable
> TestVariant getTestVariant();
>
>
> /**
> * Returns the build variant that contains the unit tests for this variant.
> */
> @Nullable
> UnitTestVariant getUnitTestVariant();
> }
>
>
> Do you have an idea of what is missing to collect the correct output?
is an important difference between test and unit test variant.
Basically, Android Tooling has support for native (running on emulator /
device) and non-native (running on standard VM) tests.
The former are often referred to "instrumentation tests", since they
require "instrumenting" an actual device to run the tests, while the
latter are linked against a file called
"mockable-android-<apiVersion>.jar" which basically is an interface-only
version of the specific target Android SDK. The test developer has to
provide mocks for everything in there (e.g. by using Mockito), but often
also throw in a library called Robolectric that provides some basic
implementations / shadows for the rather huge API surface Android has.
Now source-wise, these two test types are separated in "src/androidTest"
(tests that run for every flavor, of course there is support for
additional source sets via "<camelCasedFlavorName>AndroidTest") and the
Gradle / Maven-standard "src/test" (again tests that run for every
flavor, support for additional source sets is available).
Here is the catch: Many, if not most, Android projects actually deploy
both types of tests in an Android module (be it of app or library type),
because it is of course much faster to execute pure JVM tests for
things with little to no dependency to Android internals (like business
objects, managers, etc.). For environment-dependent tests that cannot be
easily mocked out or for Espresso / UIAutomator / general UI tests
people choose to implement instrumentation-based tests. That is why you
need to support both test variants.
And here is another catch: Since all these variants exist, I think it is
not enough to just look out for the first variant with a "debug" build
type for either test type, since many such variants could exist and also
could have been executed prior to the plugin execution (assembled
binaries and test results "at the right place" would hint you on their
execution).
> Furthermore, neither "demo-app" nor "player" have JVM-based unit
> tests, so this configuration is completely bogus.
>
>
> What do you mean by bogus? Is it crashing the analysis? I'm not 100%
> sure, but passing this property is harmless even if there are no .class
> in the output dir.
plugin would have figured out that I have no "src/*test*" directories,
hence no JVM unit tests and therefor skip this configuration.
> We're using Retrolambda and the plugin seems to pick this up, but
> again incomplete:
>
>
> $ grep retrolambda dump.properties
>
> \:demo-app.sonar.binaries=/Users/thomas/Entwicklung/Kunden/maxdome/android-videoplayer/demo-app/build/retrolambda/debug
>
> \:demo-app.sonar.java.binaries=/Users/thomas/Entwicklung/Kunden/maxdome/android-videoplayer/demo-app/build/retrolambda/debug
>
>
> It only finds retrolambda for the "demo-app" module, but actually
> there is such a directory for every module:[...]
>
>
> Again, I'm using variant.getJavaCompile().getDestinationDir() to get the
> output dir containing .class. I know getJavaCompile() is deprecated
> since it not applicable when Jack toolchain is used. Anyway we currently
> rely on bytecode for the analysis, so we need the project to be compiled
> with javac, at least when doing a SQ analysis.
other destination dirs as well, because the "retrolambda" one is really
only one very special, 3rd party library that brings back some of the
Java 8 features to the dreaded Android Java-kind-of-7 world. It is
completely optional, but often included in projects nonetheless.
> What the plugin also completely misses
> is player-{ui,exo}/build/retrolambda/debugAndroidTest, i.e.
> retrolambda output for android tests.
>
>
> For the previous issues, the best would be to share a reproducer so that
> I can simply play with APIs to find the best way to get the correct
> values. It would also serve to improve integration tests to avoid later
> regressions.
* separate sources by flavor
* separate unit / instrumentation tests by flavors
* referenced library modules
* source, binary, lint, code coverage and test result configuration
based on "best guesses" in a separated sonar-project.properties file
I hope this will help you a little.
> Android projects also often run lint on their sources, so do we. The
> Sonarqube scanner however does not find these files automatically:
>
> and consequently does not configure them
> for "sonar.android.lint.report". For flavored builds these files
> carry the flavor name as well, i.e.
> "build/outputs/lint-results-<camelCasedFlavorName>{Debug,Release}.xml
>
>
> This is a known issue. I quickly tried but I was not able to find an API
> (even internal) allowing to get the Android Lint report path. I would
> like to avoid hardcoding in the plugin what is currently the default,
> with the potential risk to have a different behavior with different
> Android plugin versions, or even the user having manually changed the
> report path (if that's possible).
contents (name, description, url, author, ...) for each listed
dependency in an Android application. What I learned there was to start
from the active application variants the Android build model gives you
(you should not build this list of variants up yourself, because
developers have the possibility to remove "non-existing" variants of
multi-dimensional configurations from this list that makes no sense) and
then build up the paths and things yourself.
Yes, this is tedious and might not be needed for everything, but with
some internal understanding you quickly have something workable and
robust. And whether the build tools change some API (which they do quite
often) or change paths of things in the filesystem is then neglectable,
you'd have to adapt your plugin anyways.
> sonar.core.codeCoveragePlugin=jacoco
>
>
> This property is deprecated an useless. Passing sonar.jacoco.reportPath
> is all you need.
> Here another feature request: Since
> https://jira.sonarsource.com/browse/SONAR-4101
> <https://jira.sonarsource.com/browse/SONAR-4101> is still not solved
> (having the ability to report JVM test and instrumentation test
> results at the same time), you might want to do some magic within
> the plugin and copy the test results of both tests (JVM and
> Instrumentation tests) to a common place (say
> "build/sonarqube/combined-test-results") that is then set as
> "sonar.junit.reportsPath".
>
>
> Can you please be a bit more specific about what is a JVM test versus
> instrumentation test? Is it the difference between UnitTestVariant and
> TestVariant?
> If your need is to support several test reports in different location,
> we could instead enhance the sonar.junit.reportsPath property to accept
> a list of paths. Again having a sample project would help me to convince
> my colleagues working on the Java plugin than this is a real use case :)
support for jacoco.itReportPath and jacoco.reportPath in the past and
even jacoco merging, and I'd imagine a similar thing for
junit.reportsPath (i.e. a new "junit.itReportsPath").
To support multi-variant runs that drop multiple *.ec files after
execution, I'd also say jacoco.itReportPath and jacoco.reportPath should
be extended to allow globs like their junit counterparts.
In any way, files that originate from instrumentation test runs on
devices will be dynamically named, because the name of the connected
device(s) is not available at the time Gradle completed its
configuration phase, so you'll always either have to scan the suspected
output directories or work with globs (if this becomes available).
> So, in the current state the plugin is not very helpful for Android
> analysis, because too many things are missing or even wrong and one
> should rather reside to manual configuration of analysis parameter.
>
>
> I'm sorry you couldn't provide this feedback earlier. Anyway let's try
> to iterate and improve the different issues.
dubbed to replace the tedious manual configuration via
sonar-project.properties, but then again, if you have no Android experts
on your team it is hard to get all this information. I'm using Sonarqube
for Android project analysis since a couple of years now and grew up
with all kinds of nastiness on either side - either some path of
artifacts changed due to Android Tooling changes, or Sonarsource decided
to expand / deprecate / replace analysis parameter :)
Thomas.
--
GPG-Key 0xCD45F2FD | Skype: realdeadfish | https://www.thomaskeller.biz
Please note that according to the EU law on data retention, information
on every electronic information exchange might be retained for a period
of six months or longer: http://www.vorratsdatenspeicherung.de/?lang=en
m...@thomaskeller.biz
Julien HENRY
I don't know much about the Gradle internals, but I can tell you there
is an important difference between test and unit test variant.
Basically, Android Tooling has support for native (running on emulator /
device) and non-native (running on standard VM) tests. [...]
And here is another catch: Since all these variants exist, I think it is
not enough to just look out for the first variant with a "debug" build
type for either test type, since many such variants could exist and also
could have been executed prior to the plugin execution (assembled
binaries and test results "at the right place" would hint you on their
execution).
Well, no, it is not crashing the analysis, but I'd have thought the
plugin would have figured out that I have no "src/*test*" directories,
hence no JVM unit tests and therefor skip this configuration.
> We're using Retrolambda and the plugin seems to pick this up, but
> again incomplete:
> [...]
> It only finds retrolambda for the "demo-app" module, but actually
> there is such a directory for every module:[...]
Again, I don't know much of the internals here, but there have to be
other destination dirs as well, because the "retrolambda" one is really
only one very special, 3rd party library that brings back some of the
Java 8 features to the dreaded Android Java-kind-of-7 world. It is
completely optional, but often included in projects nonetheless.
> What the plugin also completely misses
> is player-{ui,exo}/build/retrolambda/debugAndroidTest, i.e.
> retrolambda output for android tests.
Yes, I figured as much. I'll try and make a small example project that shows
* separate sources by flavor
* separate unit / instrumentation tests by flavors
* referenced library modules
* source, binary, lint, code coverage and test result configuration
based on "best guesses" in a separated sonar-project.properties file
I hope this will help you a little.
Some time back I hacked together a Gradle plugin to retrieve POM
contents (name, description, url, author, ...) for each listed
dependency in an Android application. What I learned there was to start
from the active application variants the Android build model gives you
(you should not build this list of variants up yourself, because
developers have the possibility to remove "non-existing" variants of
multi-dimensional configurations from this list that makes no sense) and
then build up the paths and things yourself.
Yes, this is tedious and might not be needed for everything, but with
some internal understanding you quickly have something workable and
robust. And whether the build tools change some API (which they do quite
often) or change paths of things in the filesystem is then neglectable,
you'd have to adapt your plugin anyways.
To support multi-variant runs that drop multiple *.ec files after
execution, I'd also say jacoco.itReportPath and jacoco.reportPath should
be extended to allow globs like their junit counterparts.
In any way, files that originate from instrumentation test runs on
devices will be dynamically named, because the name of the connected
device(s) is not available at the time Gradle completed its
configuration phase, so you'll always either have to scan the suspected
output directories or work with globs (if this becomes available).
Actually I was shocked that it was so little useful, though it was
dubbed to replace the tedious manual configuration via
sonar-project.properties, but then again, if you have no Android experts
on your team it is hard to get all this information.
Thomas Keller
> test to SQ unit test and instrumentation tests as ITs (at least
> regarding test execution and jacoco report). The "problem" is we are
> considering to drop this difference in some upcoming SQ version, and
> merge everything in just "tests" so I'm not sure it is worth to spend
> time on that. I'll think about it. I have
> updated https://jira.sonarsource.com/browse/SONARGRADL-25 accordingly
> and also created https://jira.sonarsource.com/browse/SONARGRADL-26.
for both types, if they could rather easily be managed by merging XML
files into one common directory.
> My understanding is that you can have a source file with the same name
> (e.g. MyClass.java) in two different variants with a different content.
> That use case could not be supported by our Java analyzer. That's why I
> consider we can only analyze a single variant. If you really want to
> analyze several variants I think the best is to run different analysis
> for each (using for example sonar.branch to distinguish them).
> If you are really sure that two variants are not overlapping and want to
> analyze the "merge" of them then I guess using manual configuration
> would be the solution, since the plugin can't really decide that.
> So apart if you have a better proposal, I will keep this behavior: take
> the first debug variant (that was not disabled by user), and use it to
> configure "main" properties. From this variant, get the associated test
> and unit test variants, and configure "test" properties. There is also
> the possibility to manually configure the variant that should be considered.
instead of automatically picking one, because the first variant really
might not be the one that is of most interest (or even has any tests, so
effectively nothing would be reported). IMHO in multi-variant builds
this automatic picking would be most of the time wrong.
You might even error out during analysis if you find more than one debug
variant and tell the user that (s)he needs to pick a specific variant
through the said configuration value.
> I was expecting the Android Gradle API to not return a UnitTestVariant
> when there is no "src/*test*" directory. But I can easily check for
> existence of directories and filter them out if needed.
> https://jira.sonarsource.com/browse/SONARGRADL-27
should be named or exist, but obviously the contents / paths are then
only existant if the particular task actually ran.
> > What the plugin also completely misses
> > is player-{ui,exo}/build/retrolambda/debugAndroidTest, i.e.
> > retrolambda output for android tests.
>
>
> Yep, it is all due to the fact I'm completely ignoring instrumentation
> tests. Ticket created:
> https://jira.sonarsource.com/browse/SONARGRADL-28
> Some time back I hacked together a Gradle plugin to retrieve POM
> contents (name, description, url, author, ...) for each listed
> dependency in an Android application. What I learned there was to start
> from the active application variants the Android build model gives you
>
>
> Any code snippet to be sure I understand what you are talking about? How
> to get "active application variants"?
http://pastebin.com/mxu8gNXM
Remember, totally different use case, I wanted to retrieve all
"compileXXX" configurations to look for its dependencies.
> Actually I was shocked that it was so little useful, though it was
> dubbed to replace the tedious manual configuration via
> sonar-project.properties, but then again, if you have no Android
> experts
> on your team it is hard to get all this information.
>
>
> We indeed have no Android developer in the team. For such integration we
> really need community feedback. I'll have a look at your sample project
> (please add coverage if possible) and let you know.
coverage is already active. I'll try to add the missing piece tonight.
m...@thomaskeller.biz
android {
...
testBuildType "staging"
}