Proposal: First-Class Support for Unit Tests in the Gradle Plugin

[Jake Wharton]

In my personal projects and at my job unit testing is an essential aspect of our development. I know that many other developers from professional to hobbyist join me in this.

On-device testing has been around since API 1. It's a bit sorely documented, requires the presence of a device, and is slow to run. The most common variant of this type of testing is instrumentation testing whereby you write an app to test your app. This black/white box hybrid is extremely prone to breakage due to changes in the app flow, view hierarchy, or resources of your app. It is no doubt the slowest of slow. Unit tests are actually supported by this test solution but they suffer most of the same problems as instrumentation but to a slightly lesser degree.

Unit tests are fast to run, fast to develop (assuming you scope them properly), and are friendly to running on both CI and local machines for every change. They're fundamental downside is that they run on the JVM and therefore cannot pass through Android code. In practice unit tests shouldn't really be relying on Android's code anyways. They're around to test isolated, local behavior of your code. Mocks or Robolectric just provided help when you're forced to touch an Android type.

Ant, the Eclipse ADT, and the new Gradle plugin have never supported unit tests officially. Users of Maven, sbt, and other Gradle solutions have been afforded unit tests since the build tools were built on existing Java infrastructure and have leaned on it heavily. Nobody argues these are a replacement for instrumentation tests. They are a complement for a proverbial one-two testing punch.

Ideally the infrastructure for accomplishing this will mostly be a duplicate of that in place for instrumentation tests (e.g., build types and flavors/flavor group support) and hopefully a lot of that code can be shared.

Please, Xav and team, bring us this blessing for the first time as a first-party solution!

---
Jake Wharton
http://about.me/jakewharton

[Boris Burtin]

I agree.  Tests should be split into two categories: unit tests and integration tests.  The unit tests that we have now are actually integration tests, due to the dependency on an Android device.  There should be a way to quickly run unit tests that test application code and don't require a device or emulator.

[Tristan Waddington]

Agreed. Real unit testing would be a huge boon to our development. I remember when I was writing web apps built on Django we had very good test coverage simply because tests were very easy to write. Not the case with Android at the moment.

-Tristan

[Xavier Ducrohet]

Hey Jake.

Yes. I agree 100% we need to have good unit test support.

How we get there is more complicated. Until we have a better solution, i'll help supporting robolectric. For instance, I need to update the model so that your plugin can have IDE support.

I'm sad it's taking so long to make really good progress on the build system (I'm hiring!), but do know that testing is high on my list of things where I want to bring massive improvements to the current situation.

Xav

--
You received this message because you are subscribed to the Google Groups "adt-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to adt-dev+u...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Xavier Ducrohet
Android SDK Tech Lead
Google Inc.
http://developer.android.com | http://tools.android.com

Please do not send me questions directly. Thanks!

[Jake Wharton]

Yeah the whole reason I write is because I hate writing and maintaining this standalone plugin. It's basically mirroring the setup for instrumentation tests with less ease (and my flaky knowledge of Groovy and the plugin internals). I'd rather have something like this built in (eliminating the need for IDE hooks, at least in this case) and just figure out how to teach libraries like Roboelctric to play nice.

---
Jake Wharton
http://about.me/jakewharton

[Matthew Fremont]

I've been experimenting with the 0.8 version of the plugin and Android Studio 0.4.4, and I hit upon an interim solution that achieves a better integration with the IDE than the other work-arounds I've seen.

Because the Android plugin has its own concepts of a source set and the build lifecycle that are not really interoperable with the Java plugin and the gradle Test task, I created a separate submodule for the unit tests.

I called this module "unitTest", and unit test code is all under unitTest/src/test/java. The build.gradle for this submodule applies the java plugin and makes use of the core Test task. It's a bit of a hack to resolve the dependencies between the unit test module and the app module, but I've been able to get it work enough to achieve the following:

1. Run unit tests that use Robolectric both from the shell and with the JUnit runner in the IDE (to support debug and a TDD workflow).

2. Android Studio will automatically recognizes the src/test subtree as test code.

3. Android Studio will resolve symbols from the app code and its declared dependencies while editing the test code and completion works.

4. No need to duplicate the declared dependencies of the app within the test module.

Here is an example build.gradle: https://gist.github.com/mfremont/9073002

Does this seem reasonable?

I think it highlights some desirable additions to the Android plugin API:

1. a better way to reference the compile SDK used by the app/library module as a dependency

2. a better way to reference the compiled app/library classes that provides proper support for flavors and build variants

3. a better way to reference the AndroidManifest.xml and resources tree

Exposing these aspects of the app/library module with an official API gives the developer the flexibility to choose the test framework and tools without requiring explicitly support of any specific framework or runtime within the Android plugin. This might even make it easier to implement automated BDD-style testing of an Android app at the controller layer.

Thoughts? Suggestions?

Thanks,

Matthew Fremont

[Przemek Jakubczyk]

Well a time ago I started to add support for unit testing in more automated way. The whole idea is based of course on robolectric but blessed with Spock. You can check http://robospock.org/ for sample configuration.

It's my night project and it'd be great if I have more data how to improve the library and gradle plugin.

Hope this helps

[Matthew Fremont]

I was experimenting with spock last week and happened upon the RoboSpock plugin while trying to figure out how to combine Robolectric and Spock. The plugin made it easier for me to get going with spock on my Android project. Thanks!

I'm only just getting started with spock, but I like the way in which it enables expressive, concise tests. IMHO this is very desirable property in a test suite, especially if you use it to highlight the domain behavior that is being tested.

This is an example of the kind of choice I had in mind when I suggested that it would be very desirable for the Android plugin to expose an API that makes it straightforward to express the dependency of unit test code on the Android SDK, the compiled classes from the Android app/lib, and the directory that contains the AndroidManifest.xml. This approach gives developers the freedom to choose a testing framework/library that is most appropriate for their needs.

As it stands with version 0.3.1 of RoboSpock, it resolves the SDK dependency with  com.google.android:android:4.1.1.4. This may be fine for some projects, but will it work properly if the code under test or the unit tests have dependencies on features introduced more recently that 4.1?

The particular unit test I started with also highlighted another issue that occurs when app or test code uses libraries bundled with the Android SDK. My particular case makes use of org.json.JSONObject. The stubbed version of this class is present in android.jar. In order to actually execute my tests and avoid the "stub" exception, it was necessary to exclude the transitive dependency on com.google.android:android, add an explicit dependency on org.json:json:20080701, and adjust the classpath so that at test runtime the non-stubbed version was loaded instead of the stubs bundled in android.jar. I imaging a similar situation applies for the other XML and other APIs that are bundled with android.jar.

It seems that core gradle does not provide a way to control the order of dependencies other than by manipulating the classpath property. Until this is changed, then it is desirable for the Android gradle plugin to expose an API that allows an app to obtain a FileCollection (or File) reference to android.jar that can be appended to classpath.

My work-around in build.gradle uses android.plugin.sdkDirectory to construct the path to android.jar. Based on comments that Xavier made in another thread, I'm guessing that the sdkDirectory property is not part of the public API. I would, of course, prefer that my unit tests only make use of the public API of the Android gradle plugin.

Matthew