Ability to exclude some tests from execution via command line

1,534 views
Skip to first unread message

Denis Karpenko

unread,
Feb 9, 2016, 8:39:20 AM2/9/16
to gradle-dev
Hello,

There is a nice functionality which allows to execute only specified tests in Gradle

But we have thousands of tests and would like to exclude a some of them from execution and it would be great if it’ll be as easy as passing command line arguments. There is no such a functionality in Gradle yet but there are a few workarounds for classes, etc (see Related links section below).

High level implementation plan is following: 
  • rename “--tests” option to “--tests-include"
    • or leave “--tests” as an alias for “tests-include” and print warning message about future deprecation
  • introduce “--tests-exclude” option
    • apply include/exclude rules in the same order (priority) as it’s done for “copy" tasks
    • use the same mechanism for filtering as currently used for inclusion
    • make implementation for TestNG and JUnit
  • update tests and documentation 

Basically, I have following questions:
  • Does “tests-exclude” command line parameter look sane?
  • Are there any common rules in Gradle for include/exclude application priorities?
  • Do you have any other suggestions for me?

Please take a look at early/draft/wip implementation in Pool Request (dkarpenko/gradle #1).


Related links:

Rene Groeschke

unread,
Feb 9, 2016, 11:19:22 AM2/9/16
to gradl...@googlegroups.com
Hello Denis

this list is for discussions about the development of Gradle itself. For questions and assistance regarding Gradle usage, please direct your enquiry to http://discuss.gradle.org

Best regards,
Rene

--
Rene Groeschke
Principal Engineer,
Gradle Inc. - Gradle Training, Support, Consulting
re...@gradle.com
http://gradle.org


9 February 2016 at 14:39
--
You received this message because you are subscribed to the Google Groups "gradle-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gradle-dev+...@googlegroups.com.
To post to this group, send email to gradl...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/gradle-dev/953536e9-a9b0-4f43-85ce-680a133e481f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Rene Groeschke

unread,
Feb 9, 2016, 11:24:05 AM2/9/16
to gradl...@googlegroups.com
Hey Denis,

sorry for my preemptive email about you being on the wrong list. I was
wrong sorry again.

Rene Groeschke wrote:
> Hello Denis
>
> this list is for discussions about the development of Gradle itself. For
> questions and assistance regarding Gradle usage, please direct your
> enquiry to http://discuss.gradle.org
>
> Best regards,
> Rene
>
> --
> Rene Groeschke
> Principal Engineer,
> Gradle Inc. - Gradle Training, Support, Consulting
> re...@gradle.com
> http://gradle.org
>
>
>> Denis Karpenko <mailto:de...@karpenko.me>
>> 9 February 2016 at 14:39
>> Hello,
>>
>> There is a nice functionality which allows to execute only specified
>> tests
>> <https://docs.gradle.org/current/userguide/java_plugin.html#test_filtering>
>> in Gradle
>>
>> But we have thousands of tests and would like to exclude a some of
>> them from execution and it would be great if it’ll be as easy as
>> passing command line arguments. There is no such a functionality in
>> Gradle yet but there are a few workarounds for classes, etc (see
>> Related links section below).
>>
>> High level implementation plan is following:
>>
>> * rename “--tests” option to “--tests-include"
>> o or leave “--tests” as an alias for “tests-include” and print
>> warning message about future deprecation
>> * introduce “--tests-exclude” option
>> o apply include/exclude rules in the same order (priority) as
>> it’s done for “copy" tasks
>> o use the same mechanism for filtering as currently used for
>> inclusion
>> <https://github.com/gradle/gradle/blob/cf6ffbe913ae8599d01227a717fda008648786bc/subprojects/testing-jvm/src/main/java/org/gradle/api/internal/tasks/testing/junit/JUnitTestClassExecuter.java#L135>
>> o make implementation for TestNG and JUnit
>> * update tests and documentation
>>
>>
>> Basically, I have following questions:
>>
>> * Does “tests-exclude” command line parameter look sane?
>> * Are there any common rules in Gradle for include/exclude
>> application priorities?
>> * Do you have any other suggestions for me?
>>
>>
>> Please take a look at /early/draft/wip/ implementation in Pool Request
>> (dkarpenko/gradle #1) <https://github.com/dkarpenko/gradle/pull/1>.
>>
>>
>> Related links:
>>
>> * https://issues.gradle.org/browse/GRADLE-1484
>> * https://discuss.gradle.org/t/excluding-tests-from-command-line/4766
>> * http://stackoverflow.com/questions/26809485/gradlle-test-unit-tests-run-all-except-one-or-few-command-line
>> * https://docs.gradle.org/current/userguide/java_plugin.html#test_filtering
>>
>>
>> Best regards,
>> Denis Karpenko
>> --
>> You received this message because you are subscribed to the Google
>> Groups "gradle-dev" group.
>> To unsubscribe from this group and stop receiving emails from it, send
>> an email to gradle-dev+...@googlegroups.com
>> <mailto:gradle-dev+...@googlegroups.com>.
>> To post to this group, send email to gradl...@googlegroups.com
>> <mailto:gradl...@googlegroups.com>.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/gradle-dev/953536e9-a9b0-4f43-85ce-680a133e481f%40googlegroups.com
>> <https://groups.google.com/d/msgid/gradle-dev/953536e9-a9b0-4f43-85ce-680a133e481f%40googlegroups.com?utm_medium=email&utm_source=footer>.
>> For more options, visit https://groups.google.com/d/optout.
>
> --
> You received this message because you are subscribed to the Google
> Groups "gradle-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to gradle-dev+...@googlegroups.com
> <mailto:gradle-dev+...@googlegroups.com>.
> To post to this group, send email to gradl...@googlegroups.com
> <mailto:gradl...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/gradle-dev/56BA1185.7090000%40breskeby.com
> <https://groups.google.com/d/msgid/gradle-dev/56BA1185.7090000%40breskeby.com?utm_medium=email&utm_source=footer>.

Pepper Lebeck-Jobe

unread,
Feb 9, 2016, 11:48:50 AM2/9/16
to gradl...@googlegroups.com
Denis,

I am happy to help you with your planned contribution to the Gradle codebase.  By the way, thanks for contacting us early in your process to make sure we align what you are doing with the broader scope of development on Gradle.  You have presented your plans and justification very well.

I want to start with just a little bit of due diligence on my part to make sure the use-case actually warrants the change you are describing (I suspect it very-well may.)

Motivation
Why do you sometimes want to exclude tests from the command line?

A) Is it that some tests take longer than others to run?  If so, you might consider making a separate source-set or test task just to hold your expensive tests.  That way, the whole development team of your project gets the benefits of being able to scale down the tests they run regularly.

B) Is it because you know the test to be broken, and just want to see what other tests may also be broken?  If that is the case, you might be able to just use --continue to see everything that is broken.

C) Is there a compelling other reason to want to exclude tests on an ad-hoc basis from the command line?

All further comments are in-line and pending a solid use-case emerges which really shouldn't be handled through other extant mechanisms.

On Tue, Feb 9, 2016 at 2:39 PM Denis Karpenko <de...@karpenko.me> wrote:
Hello,

There is a nice functionality which allows to execute only specified tests in Gradle

But we have thousands of tests and would like to exclude a some of them from execution and it would be great if it’ll be as easy as passing command line arguments. There is no such a functionality in Gradle yet but there are a few workarounds for classes, etc (see Related links section below).

High level implementation plan is following: 
  • rename “--tests” option to “--tests-include"
    • or leave “--tests” as an alias for “tests-include” and print warning message about future deprecation
I like the idea of adding an alias and deprecation of --tests.
  • introduce “--tests-exclude” option
    • apply include/exclude rules in the same order (priority) as it’s done for “copy" tasks
    • use the same mechanism for filtering as currently used for inclusion
    • make implementation for TestNG and JUnit
  • update tests and documentation 

Basically, I have following questions:
  • Does “tests-exclude” command line parameter look sane?
Actually, personal preference here would be to lead with the more descriptive part of the option. --include-tests and --exclude-tests (but those are mouthfulls.) Maybe we should entertain simply --include and --exclude as these options will only be supported by the test task anyway.  I'm not sure of this. Maybe other Gradle devs can weigh in on this. 
  • Are there any common rules in Gradle for include/exclude application priorities?
There are. Basically, you always first filter down to only what has been included, and then exclude after that.  For example:
Starting Set: a, b, c, d, e, f, g
--include a --include d --include f --exclude d --exclude g
Result Set: a, f
  • Do you have any other suggestions for me?
Be sure to sync. There have been some changes recently to how --tests works see: https://github.com/gradle/gradle/pull/561 (You'll want to make sure your code also works in the context of Suites and Parameterized tests.)
 
--
You received this message because you are subscribed to the Google Groups "gradle-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gradle-dev+...@googlegroups.com.
To post to this group, send email to gradl...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/gradle-dev/953536e9-a9b0-4f43-85ce-680a133e481f%40googlegroups.com.

Pepper Lebeck-Jobe

unread,
Feb 9, 2016, 11:56:16 AM2/9/16
to gradl...@googlegroups.com
Denis,

One more thing. There is a (now out of date) design doc about test selection. https://github.com/gradle/gradle/blob/master/design-docs/test-selection.md  Please make sure you read through that to get a sense of the original plans around test selection and considering updating it to more correctly reflect the current state of Gradle (including your enhancement) if we decide to go ahead with this work.

Thanks,
Pepper

Denis Karpenko

unread,
Feb 10, 2016, 7:54:17 AM2/10/16
to gradle-dev
Pepper, 
Thanks for your comments and useful links

Motivation
The answer is C. In more details we have a large set of tests in a project and some of the tests are flaky. We can easily detect them, but disabling them manually became a pain. So we would like our system to exclude them from build pipeline without modifying codebase (with appropriate notifications, tickets, etc, but it’s another story).

Best regards,
Denis Karpenko

Pepper Lebeck-Jobe

unread,
Feb 10, 2016, 8:20:33 AM2/10/16
to gradl...@googlegroups.com
I think I understand your use-case, but I want to say it back to you in more detail so you can correct me and elaborate just a little more. Especially, because I will argue that this use-case is better served through extant features of gradle.

I think your use-case works like this:
1. You have an automated build (maybe a CI server) that is running your build and tests.
2. Occasionally, you find that a test has become flakey.
3. Now, you want to be able to exclude this test from the tests which are run as part of your automated builds (while notifying owners, etc, that this is flakey.)

You state clearly that you don't want to have to modify the codebase for the pipeline to be able to exclude the tests.  However, I don't understand yet the motivation for this. Why not modify the codebase?

The main argument for going ahead and modifying your build files or the annotations on the flakey tests, or something in the codebase is that you keep all of the configuration of your automation in a single, portable place where the entire development team can keep track of what changed and why (in the commit records).

If you only exclude the flakey tests in the configuration of your build pipeline by modifying the arguments you pass to gradle, then your collaborators may end up checking out the code and running tests which are known by you to be flaky (because you excluded them in the pipeline configs,) and becoming frustrated when they investigate the failures only to find out that they were known to be flakey.

One of the design principles which leads us to be reluctant to add lots of command-line options to Gradle is that we want to avoid the situation where institutional knowledge is captured outside of the project's build configuration.  We are aware each time we add a command-line option we are inviting our users to encode yet another variation on their build process in configuration files separate from the codebase.

All three of the previous paragraphs of this mail are intended just to explain why we think you should be very reluctant to desire this feature. If you've read them and remain convinced that you still need this feature, I'm happy to review your work as you progress toward the enhancement.

Best regards,
Denis Karpenko

--
You received this message because you are subscribed to the Google Groups "gradle-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gradle-dev+...@googlegroups.com.
To post to this group, send email to gradl...@googlegroups.com.

Denis Karpenko

unread,
Feb 11, 2016, 8:58:56 AM2/11/16
to gradle-dev
Why not modify the codebase?

The main reason is that when tests are marked as @Ignored the only way to verify that they are still flaky is to remove this annotation in the code base. (Yes, there are other techniques can be used here, like conditional test runners, but they all require me to implement this "run/not run decision" logic in all test runners or custom gradle task but they shouldn't actually know about it).

From my side, having a capability of filtering (excluding) on Gradle level looks pretty straight forward.

I'm planing to add possibility to exclude tests in scope of one PR and make it possible to pass multiple rules in scope of https://groups.google.com/forum/#!topic/gradle-dev/sYZ6p_e-LRs.  
And, if I understand correctly, https://github.com/gradle/gradle/pull/561 allows us to remove exclude/include categories functionality, doesn't it?
All this chagnes will leave us with only two options (include/exclude) which looks like more than enough for filtering. What do you think about it?

Pepper, should I use https://issues.gradle.org/browse/GRADLE-1484 for my changes?

I'm going to make my changes available for review soon.

Best regards,
Denis Karpenko
Reply all
Reply to author
Forward
0 new messages