Multiple test tasks for a single build

[Seth Goldenberg]

Right now, it only seems possible to run one set of instrumentation tests for a build. I was hoping to add a new test task to run automation tests in addition to the instrumentation tests. That seems like a tall order given the current state of the build tools. I'd like to propose two options.

1) The ideal option - Devs can create a new test task that can test at least one build flavor independent of the supported "instrumentTest" directory. In your build.gradle file, you would define a test task with a name (ex. "automationTest") that points to a directory of the same name in the src folder. These tests would only be run when a given Gradle task is called, separately from the "connnectedDeviceCheck" but maybe folded into the "check" task.

2) The less-than-ideal option - The user can create a new task that runs an existing task (probably "connectedDeviceCheck") but pulls in extra classes from another directory before running the tests. This could probably be done with the existing tools.

I'm open to other workarounds, but these are the ideas I've been considering.

I brought this up on the ADT Google+ page.

https://plus.google.com/114026192098014497501/posts/bokGD6aN66n

Thanks,

Seth

[Xavier Ducrohet]

The issue is that this doesn't support build variants.

It would be interesting to have a helper of sort to create tasks for all variants, but when you start running into setting source folders and dependency configuration that's a bit complicated.

I think it is useful though to allow more than one type of tests, so something specific to that might work.

Something like:

android.createTestTask(baseName, Closure)

would create, for each variant:

• a task called "${baseName}${variantName}" or some other combination of both.
• a dependency configuration object called "${variantName}${baseName}"
• a new source set object called "${baseName}${variantName}" in the android.sourceSets container

Your closure would do the work. We might also want to automatically generate a compile task for it, so that it's separate from the task running the actual test...

There are other things to figure out, like the actual classpath beyond the specific Configuration object (does it extend the app classpath? probably, but there might be options).

After that you're probably on your own to run it, but if you want to run it on the device, you're going to want to have access to whatever we do for the extension and it becomes non simple.

There's a lot of details to figure out.

--
You received this message because you are subscribed to the Google Groups "adt-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to adt-dev+u...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
Xavier Ducrohet
Android SDK Tech Lead
Google Inc.
http://developer.android.com | http://tools.android.com

Please do not send me questions directly. Thanks!

[Xavier Ducrohet]

Actually we were having an internal talk about allowing different type of tests and giving control about how to run them.

Can you tell me how your automation test are setup compared to your instrumentation test?

[Seth Goldenberg]

Thanks for the response, Xavier. That is a lot of factors to consider. Our use case is much simpler. We have a set of Robotium automation tests that run like instrumentation tests. We'd just like to keep them separate because 1) they're slow to run and we want to keep our continuous build times short and 2) our QA engineer who writes and maintains the tests likes to use a separate manifest files where he can fiddle with permissions (this one is less important).

We used to keep the automation separate in Maven by having a separate project that was included when building with a given profile (ex. mvn clean install -P automation). Given that the automation tests don't need any extra dependencies besides Robotium, I was thinking of writing a workaround that copied the automation test classes to "instrumentTest" folder, ran the tests, and then deleted the classes. I imagine that other developers' use cases won't be as simple though.

[Xavier Ducrohet]

Technically you can annotate tests with @Small/Medium/Large and then only run certain types.

This is not supported yet, but this is something we want to provide. Write all your test in the same place (if they are all run the same way through the instrumentation runner). Annotate them differently, and then create different run configs and run them separately.

We also want to expand the current annotations.

[Seth Goldenberg]

That would actually be great for us. I see that it's available on the existing instrumentation test runner (http://developer.android.com/reference/android/test/InstrumentationTestRunner.html), even though there isn't support in the build tools yet. Do you have any idea where annotation support would fall on the roadmap?

[Philippe Breault]

I just stumbled on this problem and was wondering if there is a solution now.

It would be great to be able to specify the default annotions (@small, @medium, @large) and/or be able to define some of our own (e.g. @Functional, @Acceptation).

It would also be very interesting to have options to include or exclude some annotated tests from a test run which would be configurable in a task. (e.g. run everything but classes having @Large).