Parallel test groups & specs2

[Emre Çelikten]

Hello,

Is there a way to specify groups for test parallelization? We have a program that distributes tests into groups according to their completion times and we would like to use it with Jumi.

Also, is specs2 supported? I wasn't able to make Jumi run any tests, perhaps it was a problem on my end. Here is the configuration for sbt-jumi:

Seq(
      jumiTestThreadsCount := 12,
      jumiJvmOptions := Seq("-ea", "-Xmx512M"),
      jumiIncludedTestsPattern := "glob:**Spec.class"
)

This is what I get:

[root] $ jumiTest
[-----=====-----=====-----=====-----=====-----=====]
Pass: 0, Fail: 0
[success] Total time: 1 s, completed Apr 29, 2014 9:29:08 AM

Thanks in advance,

Emre

[Esko Luontola]

Emre Çelikten wrote on 29.4.2014 9:30:
> Hello,
>
> Is there a way to specify groups for test parallelization? We have a
> program that distributes tests into groups according to their completion
> times and we would like to use it with Jumi.

Hi,

What do you mean by "groups for test parallelization"? What would those
groups be and in what way would they affect how the tests are executed?

Currently Jumi runs all tests using a single thread pool, which has by
default as many threads as CPU cores. I have plans for making it
possible to disable test parallelization (e.g. by annotating the test
class) for tests that would not work in parallel.

Please tell me more about your use case, so that I can make sure whether
the current plans for parallelization configuration will fit it, or
whether they need rethinking.

> Also, is specs2 supported? I wasn't able to make Jumi run any tests,
> perhaps it was a problem on my end. Here is the configuration for sbt-jumi:

You will need to configure the specs2 classes to be runnable via JUnit:
http://etorreborre.github.io/specs2/guide/org.specs2.guide.Runners.html#Via+JUnit

Jumi has support for running JUnit tests, and through that support it
can run all testing frameworks that JUnit can run.

--
Esko Luontola
www.orfjackal.net

[Emre Çelikten]

Hello Esko,

Thanks very much for your answer. Sorry, I should have been more clear.

We have tests that take different amounts of time, say:

ASpec: 10 seconds
BSpec: 10 seconds
CSpec: 20 seconds

Assume that I am running tests in 2 threads. Then it would make sense to run ASpec and BSpec in a thread and CSpec in another, so they all finish at approximately the same time. Is there a way to specify this in Jumi? Something like this in Build.scala for instance:

val testGroups = List(List("ASpec", "BSpec"), List("CSpec"))

jumiTestGroupingScheme := testGroups

We have hundreds of tests that we would like to run in parallel using a distribution scheme like above. The goal is to cut down build times in our continuous integration server.

Best,

Emre

[Esko Luontola]

Emre Çelikten wrote on 2.5.2014 18:15:
> We have tests that take different amounts of time, say:
>
> ASpec: 10 seconds
> BSpec: 10 seconds
> CSpec: 20 seconds
>
> Assume that I am running tests in 2 threads. Then it would make sense to
> run ASpec and BSpec in a thread and CSpec in another, so they all finish
> at approximately the same time. Is there a way to specify this in Jumi?

I see. Currently there is no such thing, but test order priorization is
surely one of the things that I'll look into later. My idea is that Jumi
will collect statistics that how long it takes to run different tests,
what classes they use etc. and then use that information to
automatically figure out the best order for running the tests.

Here are some excerpts from the roadmap:

- Statistics (required by many other features)

- Test priorization, run first those which will most probably fail, run
fast tests first etc.

- Look for the fastest order of running tests from class loading point
of view (best-effort thread affinity at test class level, to avoid all
threads being blocked by loading the same classes?)

- IO/CPU boundness checking (to choose how many threads to use)

https://github.com/orfjackal/jumi/blob/master/ROADMAP.txt

--
Esko Luontola
www.orfjackal.net

[Emre Çelikten]

Hello,

Thanks for your answer.

Interesting idea, I didn't think about class loading times before.

I think it would still be a good idea to allow the users to specify test
configuration themselves. Distributing tests according to their time
statistics can be made a non-trivial task (e.g. fit bimodal Gaussians
into each test, then try to minimize moments of Gaussian mixtures while
making means the same) and every minute counts when you have a large
number of Selenium tests.

It might also be good to specify a target path for collected data too,
as CI servers generally download the source from a repository and build
from scratch in a clean environment.

Best,

Emre


03/05/14 08:28 tarihinde, Esko Luontola yazdı: