Implementing a scala_test rule

[Dino Wernli]

Hi bazel-dev

TL;DR: I have an implementation of a scala_test rule that I don't know how to upstream.

I have recently been looking into adopting Bazel for a Scala project and, so far, the experimentation has been going well. One thing I stumbled upon was the lack of a scala_test rule analogous to its java_test counterpart.

After a bit of fiddling, I was able to put together a scala_test rule which, just like the Java rule, shares the same attributes as scala_binary and reuses the scala_binary_impl as its implementation. One road block I came across though is that, unlike the JUnit test runner used for the Java rule, its scalatest counterpart seems to require explicitly passing the requested test suites on the command line.

Since it was hard to tell which source files are actually test suites a priori, I had to write a custom BazelTestRunner which does the following:

- It opens up the *last* jar on its classpath.
- In there, it looks for subtypes of org.scala.test.Suite .
- It then executes the scalatest runner under the hood, enumerating the test suites it found.

The purpose of this email is to discuss the options around upstreaming this scala_test rule. The big question I have is around Scala versions. My solution requires Bazel to depend on the scalatest library jar (similarly to the situation with JUnit). However, there are differences between the scalatest library for Scala 2.10 and the one for Scala 2.11, which begs the question: what is the policy on picking a Scala version? Ideally, Bazel would pick the “right” one corresponding to the jar at /usr/share/java/scala-library.jar, but I’m not sure how to pull that off.

Also, I have a bunch of smaller questions:

1) My scala_test build rule comes with a custom BazelTestRunner, written in Scala, which wraps the runner provided by scalatest. Would it be possible to include that with the Bazel distribution? If shipping Scala code in Bazel is a problem, it might be possible to investigate writing the runner in Java instead, but it's not clear whether that would work.

2) The approach also relies on the last jar in the classpath being the jar containing the compiled classes from the "srcs" attribute of the scala_test rule. Is it ok to assume this is not going to change, or might there be a better way?

Cheers
Dino

[Ulf Adams]

The Java test runner actually also requires passing the class-to-test to the runner.

For java_test, we should use main_class and test_class attributes, the first to set the test runner (and which could be set to a useful default value), and the test_class to set the class-to-test, which could also be guessed from the srcs attribute (which is what we do right now).

--
You received this message because you are subscribed to the Google Groups "bazel-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-dev+...@googlegroups.com.
To post to this group, send email to baze...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-dev/CAGnkpeKhv8wsXUOTHK8Kfff_kikhJDDpoHUQYdsZVTijVp6SwQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[Dino Wernli]

Understood. My approach would be inline with the "guessing from the sources" idea.

As far as I can tell, setting a sensible default for the runner would still require shipping a version of scalatest as part of Bazel. Any suggestions regarding whether to select between the version compatible with Scala 2.10 and the one compatible with 2.11?

[er...@xperiel.com]

FWIW groovy_test has very similar issues. Choosing which classes to pass as arguments to JUnitCore is the hard part. My solution sounds less sophisticated than yours - I take everything in srcs, strip src/test/groovy or src/test/java from the prefix, drop the .groovy from the suffix, and change the slashes to dots, passing the results to JUnit. There are obviously a lot of ways this can break. It would be cool to find a robust solution that works for all three languages.

Regarding the versioning question, for Groovy I left it up to the user to bind whatever targets they want to //external:groovy, so they can use any version. The example uses 2.4.4, but anything would work if the WORKSPACE and BUILD files were set up correctly. Scala might be able to do something similar.

[Ulf Adams]

Technically, we only need the list of classes when we actually run the test runner.

My proposal for Java is to use an annotation processor that understands JUnit4, and can output a list of classes, or can generate a test suite class with a specific name (say, based on the name of the rule). That would completely automate most use cases, and also make some use cases faster (right now, bazel's test iterate over the classpath at runtime, which can be slow if the classpath is big).

I'm not sure how that would work for scala or groovy.

--

You received this message because you are subscribed to the Google Groups "bazel-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-dev+...@googlegroups.com.
To post to this group, send email to baze...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-dev/14f44f28-d247-4e73-b616-59853c080cbd%40googlegroups.com.

[Dino Wernli]

@erik: interesting approach with the groovy version. Does this mean that users are expected to define artifact targets with specific names in their WORKSPACE if they want to use the rules? I think that might be a viable approach for Scala as well, but I was not sure if that's acceptable.

[Erik Kuefler]

Users are expected to define bind() rules in their WORKSPACE with specific names, which leads to targets like "//external:groovy" that the groovy rules implicitly depend on. They can point those bindings at whatever artifact they want, so there's a bit of flexiblity and users can swap to different artifacts just by changing the binding. I talked this over with the Bazel core people at the time and they thought it was a good approach; appengine and I think some of the other rules work this way too. You can look at the setup section of the Groovy README for an exact description of what users need to provide. The pattern with groovy and the other build defs so far has been to provide a sample WORKSPACE file that users could copy/paste into their WORKSPACE, modifying if needed.

[Dino Wernli]

Ah, excellent, I think this solves the problem of the Scala version, thanks!

Regarding test suite discovery: the scalatest runner requires specifying the canonical class names (qualified by package) of all the suites to run. Unlike in Java though, the package name is not tied to the filesystem, so replacing "/" with "." would probably not work in all cases.

Are there any strong opinions on whether it is a problem to ship a custom runner written in Scala? It only scans the *last* jar in its classpath, so it's pretty efficient and can reliably find all suites.

[Dino Wernli]

Thanks for you input, everyone. I've created PR: https://github.com/bazelbuild/bazel/pull/657

It add a simple test rule which, for now, requires manually specifying the test suites. That way, we can discuss/agree on a suite discovery strategy separately.

[o...@wix.com]

I've been experimenting with doing something similar to what Bazel itself does with the AllTests class and ClasspathSuite. The basic idea is to use java_test with a single argument which @RunsWith a custom Suite, which only runs the tests found in the specific test jar (we don't want to run *all* the tests in the classpath). Benefits are that we don't rely on source code/package names, and reuse the java_test implementation.

I've got a working example of this here:

https://github.com/orrsella/bazel-example/blob/master/tools/build_defs/scala/scala-test.bzl

https://github.com/orrsella/bazel-example/tree/master/tools/src/main/java/testing/junit

Much of the java source code was lifted from bazel itself for the time being.

Would love to hear your thoughts.

[P. Oscar Boykin]

This @RunsWith approach is how pants currently does things, it's a bit ugly if you ask me. I guess I prefer it to listing each test fixture in the build file (I prefer to add code/annotations to source rather than BUILD given the choice).

I'm not sure how sbt does it, but it seems to manage without any special markers other than extending certain test classes.

[Dino Wernli]

As mentioned in the original post of this thread, there is also the option of using a custom test runner to find all test suites using a subtype check. It has the advantage that it doesn't require annotating the tests in any special way, but it would require shipping a custom test runner (and my implementation is currently about 50 lines of Scala).

Would it make sense to let users simply choose? They can decide to either specify suites, or annotate them, or have them auto-discovered. The way they would choose would be by simply overriding the runner. What do you think?

[P. Oscar Boykin]

I think shipping a custom runner, if it can make it very easy to run the tests, would be totally fine. Users don't care how it is implemented.

PS: I'm hoping this gets merged soon: https://bazel-review.googlesource.com/#/c/2410 and we can merge this test target on top of that.

[o...@wix.com]

 I agree that a custom runner is probably fine, but would make sure it handles ScalaTest/Specs2/JUnit, at the very least (Dino only referenced ScalaTest in his original post).