run multiple tests in one java_test rule

[Sam Farid]

Hello Bazel Team,

We're in the midst of porting our Maven project over to Bazel, and so far I can't tell from the docs what the best way is for us to run all the tests on a classpath with one java_test rule. I imagine Google wants to maintain as much concurrency when running tests but it seems that adding each test separately to the BUILD files is rather tedious. Is there a feature that I'm missing? What do you recommend?

Best,
Sam

[Damien Martin-guillerez]

Hi Sam,

We are using a ClasspathSuite runner to do that in Bazel, see https://github.com/bazelbuild/bazel/blob/master/src/test/java/com/google/devtools/common/options/AllTests.java

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/CAEi%3DzJNe95J8LOmB7q8_qXoZO4tg5ksdJHdPumgO6W8WpJq_KA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[ittai zeidman]

Damien,

Looking through the code it seems there are several instances of the AllTests class. Any specific reason for this?

Maybe bazel should offer AllTests as a first class citizen and remove the duplication?

I'd be happy to contribute this although it will be little use to me since that uses TestSuiteBuilder which mandates JUnit4 tests to have a @RunWith annotation (we just use *Test name pattern).

[Damien Martin-guillerez]

Agreed that it should be a first class citizen but we are thinking of removing the need for it.

Historically we shipped bazel without the test runner so without a common library for doing the ClasspathSuite and duplicated that work all over the place.

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/8dfdf22c-07c6-447b-b412-bbd4ce678b3e%40googlegroups.com.

[Sam Farid]

Ittai,

I don't know if this is any help to you but we avoided the AllTests duplication by specifying the package in the java_test args field and then parsing that while building the suite in the java code. This way, all of the test_class fields can point to one AllTests file, but you can use it anywhere.

[ittai zeidman]

Thanks, I'll look into it.
I'll need to finalize our requirements and see.

Damien,
Can you elaborate some more on the plans for removing the need?
Spent quite a bit of time on understanding how this currently works and would rather not build elaborate constructs if the basis is under consideration

[Damien Martin-guillerez]

On Tue, Jun 7, 2016 at 7:31 PM ittai zeidman <itt...@gmail.com> wrote:

Thanks, I'll look into it.
I'll need to finalize our requirements and see.

Damien,
Can you elaborate some more on the plans for removing the need?
Spent quite a bit of time on understanding how this currently works and would rather not build elaborate constructs if the basis is under consideration

IIRC +Ulf Adams was saying that it would be easy to include the same mechanisms as the ClasspathSuite directly in the Java runner. +Ulf Adams can complete.

[ittai zeidman]

Thanks for the clarification.
Unfortunately for us we don't add @RunWith on every test class but rather we use filename pattern (*Test.java, *IT.java, *E2E.java) so it doesn't seem like it will help us.

[Ulf Adams]

That's right - my proposal only works with Junit4-style tests using @RunWith annotations.

[ittai zeidman]

WDYT about the filename pattern approach? AFAIK it's a pretty prevalent approach in the maven ecosystem

[Damien Martin-guillerez]

Can't you use globs? Something like:

[java_test(name = f[:-5], srcs = [f], deps = ...) for f in glob(["**Test.java"])]

Of course this will create a lot of java_test.

[ittai zeidman]

That will solve most issues but why is that preferred to having my own Filename runner?

AFAIU your solution will also mandate a macro (so people don't repeat the loop everywhere) and they will share the deps (since I want to save that management overhead) so that sounds very similar.

My current plan, if I don't find one better, is to write a macro that will call java_test with the main_class set to the new runner. Sounds very similar.

Actually come to think of it, your solution will enable parallelization of the targets, right?

It doesn't solve caching (since they share the deps) but enables parallelization.
Am I correct? If so it sounds like you're correct though I don't know the "costs" of a lot of java_test

[Ulf Adams]

On Thu, Jun 9, 2016 at 11:46 AM, ittai zeidman <itt...@gmail.com> wrote:

That will solve most issues but why is that preferred to having my own Filename runner?

AFAIU your solution will also mandate a macro (so people don't repeat the loop everywhere) and they will share the deps (since I want to save that management overhead) so that sounds very similar.

My current plan, if I don't find one better, is to write a macro that will call java_test with the main_class set to the new runner. Sounds very similar.

Actually come to think of it, your solution will enable parallelization of the targets, right?

It doesn't solve caching (since they share the deps) but enables parallelization.

Am I correct? If so it sounds like you're correct though I don't know the "costs" of a lot of java_test

Yes, it'll allow the tests to run in parallel. But note that the Jvm startup overhead is significant, so that may actually slow down the end-to-end test time because it could waste a lot of CPU on Jvm startup for every individual test.

[Damien Martin-guillerez]

I don't think it is preferred over a runner that support the sharding protocol. It is just easier to do. It might be easier to use a variation of the ClasspathSuite that looks for *Test classes instead of the RunWith annotation.

[Paul Gross]

I don't quite understand the resolution of this thread. What's the preferred way to run a set of tests in a single java_test target? It seems like there are a few options, each with different tradeoffs:

1. Write a skylark rule that will generate a suite based on files.

2. Use a custom RunWith runner that will generate a suite on the fly.

It seems like bazel does #2, but the code is all internal. We'd have to write our own, or extract it somehow. It looks like there are some similar open source libraries, but I couldn't get them working in my short spikes (and they look fairly dead with no updates in over a year).

Is the bazel team planning on adding this functionality directly to bazel? I liked the old behavior, where I could just specify a glob in srcs and bazel would run everything it found. I think this matches how other java build tools tend to do it. I tried to switch back to the legacy behavior, but it does not seem to work with bazel 0.3.0.

Thanks,

Paul

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/CAN3hOS_5i84WK2t2wA4VEw6PnEKGq7oXW_gvXXCgj3e3bkWqBQ%40mail.gmail.com.

[Ulf Adams]

On Tue, Jul 12, 2016 at 10:42 PM, Paul Gross <pgr...@gmail.com> wrote:

I don't quite understand the resolution of this thread. What's the preferred way to run a set of tests in a single java_test target? It seems like there are a few options, each with different tradeoffs:

1. Write a skylark rule that will generate a suite based on files.

2. Use a custom RunWith runner that will generate a suite on the fly.

It seems like bazel does #2, but the code is all internal. We'd have to write our own, or extract it somehow. It looks like there are some similar open source libraries, but I couldn't get them working in my short spikes (and they look fairly dead with no updates in over a year).

The Bazel code should be fairly self-contained, IIRC.

 

Is the bazel team planning on adding this functionality directly to bazel? I liked the old behavior, where I could just specify a glob in srcs and bazel would run everything it found. I think this matches how other java build tools tend to do it. I tried to switch back to the legacy behavior, but it does not seem to work with bazel 0.3.0.

I did a prototype with Ittai (when he visted us in MUC last week), which uses an annotation processor, and a corresponding test suite. If there's general agreement, we could move forward with that. I think it should work for Bazel itself as well, though we haven't tried that yet. Code is here:

https://github.com/ulfjack/bazel/tree/auto-java-test

[Justine Tunney]

The best practice way to test Java in Bazel is to have a single java_test() rule for every single FooTest.java file. Each rule should be named the same as the file (without the extension.) Each rule should specify the full set of statically-linked deps within that test file.

For example:

# //javatests/io/bazel/project/package/BUILD

java_test(

    name = "FooTest",

    size = "small",

    srcs = ["FooTest.java"],
    deps = [
        "//java/io/bazel/project/package",

        "//third_party/java/junit",
        "//third_party/java/guava",

        "//third_party/java/truth",

    ],

)

java_test(

    name = "BarTest",

    size = "small",

    srcs = ["BarTest.java"],

    deps = [

        "//java/io/bazel/project/package",

        "//third_party/java/junit",
        "//third_party/java/mockito",

        "//third_party/java/truth",

    ],

)

Then you run:

bazel test //javatests/... --test_output=errors

This is the approach that allows Bazel to work best. Because Bazel is an abstract build system. It wants you to define the relationship between your source files as a directed acyclic graph. The DAG is what allows Bazel to figure out how to make things fast, distributed, and scalable. The more effort you put into defining that DAG, the smarter Bazel will be. And the benefits flow over into any tool (like CI systems) that utilize Bazel's build graph.

On the other hand, if you had a single test rule in the base directory of your project that just did glob(["**/*Test.java"]) on everything, then you wouldn't really have much of a DAG at all. So Bazel can't be smart. So at that point, Bazel honestly isn't that much better than just writing a shell script which invokes javac and friends. So using Bazel that way is the opposite of ideal. The ideal we strive towards is a fine-grained build graph. Even though we're not always able to achieve our ideals.

I recognize that the best practice approach is onerous. But look on the bright side. That just means there's an opportunity for a member of the Bazel community to write a command line tool in Go (just like Buildifier) which can automate writing best practice java_test() and java_library() rules, by scanning the imports in Java files. The simplest possible design of such a tool would require that the user to write a configuration file mapping Java classes to Bazel deps.

Since no such tool currently exists, and the world is imperfect, the next best thing is to follow what Pants calls the 1:1:1 rule and use a macro to generate java_test rules. Which is what Damien recommended earlier. (I'm not sure why classpath scanning or any of that other voodoo would be necessary.)

Many Googlers use the GenTestRules macro, which I've open sourced just for you guys. If you use this macro, your test packages would look like the following:

# //javatests/io/bazel/project/package/BUILD

load("//somewhere:GenTestRules.bzl", "GenTestRules")

java_library(
    name = "package",

    srcs = glob(["*.java"]),

    deps = [

        "//java/io/bazel/project/package",

        "//third_party/java/junit",

        "//third_party/java/guava",

        "//third_party/java/mockito",

        "//third_party/java/truth",

    ],

)

GenTestRules(

    name = "GeneratedTestRules",

    test_files = glob(["*Test.java"]),

    deps = [":package"],

)

This approach saves the developer on time maintaining BUILD files, at the expense of build specificity and performance. For example, the :package rule above creates a bottleneck in the DAG, because all generated java_test() rules depend on it. So for example, if Mockito gets upgraded, then FooTest needs to be recompiled and rerun, even though it wasn't actually affected. Although this isn't a problem in practice if your projects has lots of Java packages and each is of a reasonable size.

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/CAMXKpGCFeT52DQsY0cO%2BdEY%2BDU6%2B%3DJ269wiSZALnpfm8UVecow%40mail.gmail.com.

[Paul Gross]

Justine,

Thank you for the detailed and thorough response.

For context, we have a monorepo with a top level directory per library/application. And we generally have one or a small number of test suites per library. With the old behavior, we had a single test target per library, which seemed simple and made the test output nice:

//foo:test   PASSED in 8.9s

//bar:test   PASSED in 2.0s

//baz:test   PASSED in 5.2s

I was able to preserve this existing behavior using the ClasspathSuite from https://github.com/takari/takari-cpsuite.

Given your response, I also gave your approach a try and got it working. Thanks for open sourcing the macro.

I see the advantages, but in practice, I'm not sure I like it better. In my limited testing, it actually seems slower than our previous approach, even if nothing changes (bazel test //... --nocache_test_results). Given that it's running more tests in parallel, I'm not sure why it would be slower. I haven't investigated further. This approach also makes the test output incredibly long, albeit more targeted since you can see exactly which test failed.

I'm glad there are multiple viable options, and we're discussing internally which approach we'd like to take.

Thanks,

Paul

[Ming Zhao]

Either you can go with classpath suite, or you can write things like
SmallTests.java which lists all tests explicitly, like this:

import org.junit.runner.RunWith;
import org.junit.runners.Suite;


@RunWith(Suite.class)
@Suite.SuiteClasses({ATest.class, BTest.class})
public class SmallTests {
}

It's a little tedious, but really affordable I think, :)

> https://groups.google.com/d/msgid/bazel-discuss/CAMXKpGD_4gg39Q0OJ4S8bkWiVdZyLcduRGc48DK0Md12xNnMiQ%40mail.gmail.com.

[Ulf Adams]

Depending on the runtime of the tests, the overhead of starting up a Jvm for each test class can completely negate the benefits of the finer-grained triggering and caching, and parallelism is limited if you don't have remote execution. As such, I perfectly understand if not everyone wants to have one java_test target per class. Me personally, I'm not trying to push Bazel users into a specific model, though some models may be better supported than others.

The code at https://github.com/ulfjack/bazel/tree/auto-java-test is one way to automate the multiple test classes per java_test rule such that a) you don't have to manually construct test hierarchies, and b) you don't have the runtime cost of scanning the classpath for test classes. It isn't strictly necessary. I think everything can be achieved today without Bazel changes. But it makes it work out of the box, with no additional configuration or macros required.

[Justine Tunney]

The JVM startup penalty problem is easily solved for Java compilation using --strategy=Javac=worker.

Is java_test() able to run as a worker too? If not, I hope Bazel will support that in the near future.

[Ulf Adams]

On Thu, Jul 14, 2016 at 4:25 PM, Justine Tunney <ja...@google.com> wrote:

The JVM startup penalty problem is easily solved for Java compilation using --strategy=Javac=worker.

Is java_test() able to run as a worker too? If not, I hope Bazel will support that in the near future.

No. We've talked about it, though, and would like to make it possible.

[Justine Tunney]

In that case, would you recommend users still write finer-grained test rules? It'll require tolerating a performance penalty on small tests in the near term, until this feature is added. But is beneficial to codebases in the long run, because the rich build graph grants Bazel more flexibility to parallelize, distribute, and report metrics. So once all these nice tools are developed for the open source community, users will be able to reap their benefits without having to rewrite their build files.

[Paul Gross]

Hi Ulf,

I like the implementation in auto-java-test, and if it gets merged, we'll probably switch to it.

Thanks,

Paul

[ittai zeidman]

I'll add one more note which is that we often have auxiliary classes to our tests (either some case classes which are examples or some utilities shared between a few tests).

Following the 1:1:1 when the packages are fine grained minimizes dramatically the BUILD file management overhead since in Justine's suggestion, AFAIU, I would also need to create a java_library for my auxiliary test code and have that added as a dep to all of the java_test targets. While this is fine if someone feels they need this gain I think that out of the box I'd try to steer our devs into finding a sweet point between optimization and ease of maintenance.  

[nikita.g...@cxense.com]

Half a year has passed. Maybe some new convenient way of running multiple tests appeared?

[ittai zeidman]

I've actually coded a small integration which basically lists the resulting classes which end with a suffix, writes it to a file and then pass a custom test and runner which reads this file and runs all tests listed in it. It's rough around the edges and is internal to the scala_rules but I might be able to split it out if people will want it.

On Mon, 6 Mar 2017 at 18:56 <nikita.g...@cxense.com> wrote:

Half a year has passed. Maybe some new convenient way of running multiple tests appeared?

--
You received this message because you are subscribed to a topic in the Google Groups "bazel-discuss" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/bazel-discuss/MFYYkJk1tFs/unsubscribe.
To unsubscribe from this group and all its topics, send an email to bazel-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/05dce314-46f7-4343-885a-e1ec297377ab%40googlegroups.com.

[gdon...@etsy.com]

Etsy is getting started on its migration to Bazel and we're interested in what the current best practice is for java_test.

1) One java_test per JUnit test -- "The best practice way to test Java in Bazel is to have a single java_test() rule for every single FooTest.java file"

2) One java_test per directory using the AllTests suite approach, auto-java-test, or GenTestRules

3) something else

If java_test per test class is the best practice, the overhead of generating the rules via a script is not too bad -- I did that to migrate our Gradle build to a set of java_library rules per package.

Thanks!

[ittai zeidman]

I can't say what the bazel best practice is but I can tell you from our side migrating from maven to bazel.

#1 Has some appeal but it's not 100% clear the overhead of target per test is worth it in general and specifically now that java_test doesn't use a worker (as per Ulf's comment above). Additionally people have to maintain these targets and to me they feel too fine grain. I personally like the trade off of 1:1:1.

You can take a look at the suite implementation I coded in rules_scala which I've discussed with Damien about donating back to bazel and exposing it as part of @java_tools or something.

It's essentially very very similar to how maven failsafe/surefire plugin works.

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/0d332b3f-51cf-4a7f-87a3-6910fce4e2c6%40googlegroups.com.

[Rob Figueiredo]

Current docs say it is possible to have a globbed junit target:

https://bazel.build/versions/master/docs/be/java.html#java_test

The java_test example target is missing the "test_class" attribute. Adding that and using a ClasspathSuite, I can get it to run. However, I could not get it to successfully find the test classes using the ClasspathSuites available. Any tips?  Here's what I tried

https://gist.github.com/robfig/b2ae93b90dd93382dbf7e225e91b3659

It looks like the ClasspathSuite in the bazel testutil package might be useful, but it seemed quite overkill to import a huge jar for that functionality, nor did it look easy to transplant just the classes involved.

https://github.com/bazelbuild/bazel/blob/master/src/test/java/com/google/devtools/build/lib/testutil/ClasspathSuite.java

Alternatively, it looks like this skylark rule might be the easiest thing, but I was hoping to stick to official rules as much as possible.

https://github.com/bazelbuild/bazel/issues/2539

Any advice?

Thanks,
Rob

[Justine Tunney]

Here's one really good way to have multiple tests in one java_test() rule:

# BUILD

java_test(

    name = "FooTestSuite",

    size = "small",

    srcs = glob(["*.java"]),

    deps = ["@junit"],

)

// FooTestSuite.java

/** Test suite for foo package. */

@RunWith(Suite.class)

@SuiteClasses({

  FooTest.class,

  BarTest.class,

  // etc.

})

public class FooTestSuite {}

You basically just have a test suite with the same name as the rule and then statically list all your test classes.

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discuss+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/8fc84dfa-d855-461c-9605-5ab15f0da602%40googlegroups.com.

[Rob Figueiredo]

Thanks for that option. I was really hoping to avoid having to list all tests individually though. Developers here don't have to explicitly list tests today, adding a new *Test.java target in a directory is sufficient to have it run. Besides the extra book-keeping, it would also lead to scenarios where tests are accidentally not being run in continuous integration (they may have developed and run it only in Eclipse, for example)

[ittai zeidman]

Hi Rob,
You can take a look at rules_scala to see how I've solved it. I built a mechanism very similar to maven surefire.
I've started breaking this off to its own separate repo so people can use it without scala.
There is one rough edge related to getting a short_path on a macro but I hope to figure something out with the Bazel people very soon.
If you're interested that would definitely add motivation.

On Wed, 17 May 2017 at 18:25 Rob Figueiredo <rob...@yext.com> wrote:

Thanks for that option. I was really hoping to avoid having to list all tests individually though. Developers here don't have to explicitly list tests today, adding a new *Test.java target in a directory is sufficient to have it run. Besides the extra book-keeping, it would also lead to scenarios where tests are accidentally not being run in continuous integration (they may have developed and run it only in Eclipse, for example)

--
You received this message because you are subscribed to a topic in the Google Groups "bazel-discuss" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/bazel-discuss/MFYYkJk1tFs/unsubscribe.
To unsubscribe from this group and all its topics, send an email to bazel-discus...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/0f2a00d4-e766-46db-992b-3bd524817488%40googlegroups.com.

[Justine Tunney]

If you don't want to do write the test suites, then there's always this. (I haven't actually run the code.)

# foo.bzl

def jart_java_test(name, srcs, package, **kwargs):

  cmd = ["(",

         "echo 'package %s;'" % package,

         "echo 'import org.junit.runner.RunWith;'",

         "echo 'import org.junit.runners.Suite;'",

         "echo 'import org.junit.runners.Suite.SuiteClasses;'",

         "echo '@RunWith(Suite.class)'",

         "echo '@SuiteClasses({'"]

  for src in srcs:

    if src.endswith("Test.java"):

      cmd.append("echo '  %s.class,'" % src)

  cmd.append("echo 'public class %s {}'" % name)

  cmd.append(") >$@")

  native.genrule(name=name + "_testsuite",

                 outs=[name + ".java"],

                 cmd="\n".join(cmd),

                 testonly=1,

                 visibility=["//visibility:private"])

  native.java_test(name=name, srcs=srcs + [name + ".java"], **kwargs)

# BUILD

load("//:foo.bzl", "jart_java_test")

jart_java_test(

    name = "FooTestSuite",

    size = "small",

    package = "com.doodle",

    srcs = glob(["*.java"]),

    deps = ["@junit"],

)

On Wed, May 17, 2017 at 8:25 AM, Rob Figueiredo <rob...@yext.com> wrote:

Thanks for that option. I was really hoping to avoid having to list all tests individually though. Developers here don't have to explicitly list tests today, adding a new *Test.java target in a directory is sufficient to have it run. Besides the extra book-keeping, it would also lead to scenarios where tests are accidentally not being run in continuous integration (they may have developed and run it only in Eclipse, for example)

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discuss+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/0f2a00d4-e766-46db-992b-3bd524817488%40googlegroups.com.

[Rob Figueiredo]

Very cool! Thank you.

[Sándor Korotkevics]

I tried most of solutions discussed in the thread while learning Bazel, and then found this, what do you think about it?

https://github.com/GerritCodeReview/bazlets/blob/master/tools/junit.bzl

To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.

[ittai zeidman]

For us this wasn’t good enough since in Scala the fileName and classnames can differ (you can also have multiple classes in a file) so we needed to dynamically discover the relevant classes on runtime.
You can take a look at rules_scala for the implementation.
We’ve been meaning to take it out to its own repo and be jvm agnostic but never got around to it.
Think of it as maven surefire test discovery for Bazel

You received this message because you are subscribed to a topic in the Google Groups "bazel-discuss" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/bazel-discuss/MFYYkJk1tFs/unsubscribe.
To unsubscribe from this group and all its topics, send an email to bazel-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/816d5019-4f2c-4c7b-99cc-18d027f15206%40googlegroups.com.

[Igor Gatis]

Any one tried using

java_test(

   ...

   test_class = "com.google.devtools.build.lib.AllTests",

)

The missing bit is the dependency that contains AllTests

On Wednesday, April 25, 2018 at 5:22:52 PM UTC-3, ittai zeidman wrote:

For us this wasn’t good enough since in Scala the fileName and classnames can differ (you can also have multiple classes in a file) so we needed to dynamically discover the relevant classes on runtime.
You can take a look at rules_scala for the implementation.
We’ve been meaning to take it out to its own repo and be jvm agnostic but never got around to it.
Think of it as maven surefire test discovery for Bazel

To unsubscribe from this group and all its topics, send an email to bazel-...@googlegroups.com.