Mixing sequential and parallel tests.

[Dmitry Makhno]

Hello,

I'm a bit stub with mixing parallel and sequential tests.

Let's say I have:

class mixParallelAndSequentialTest extends FreeSpec with Logging{

  def sequential_test(alias:String) {

    "for %s".format(alias) - {

      "Phase 1" in {

        for (j <- 0 until 10) {

          logger.info("working %s     Phase-1 %s".format(alias,j))

          Thread.sleep(50)

        }

      }

      "Phase 2" in { //should be tested only after Phase 1, and it would be nice to mix

        for (j <- 0 until 10) {

          logger.info("working %s     Phase-2 %s".format(alias,j))

          Thread.sleep(50)

        }

      }

    }

  }

  "Configuration testing" - {

    val configurations = List("Config1", "Config2") //naturally this is DataDriven and the length is dynamic and unknown

    configurations.foreach(sequential_test(_)) //need to be parallel

  }

}

Thus most important:

I cannot get it how to make Configuration testing itself parallel, and keep specific tests sequential. How I should redesign this?

Really desired:

And for sequential_test it would be nice to mix CancelAfterFailure. Meaning sequence canceled, but other configurations keep going. How to achieve this also?

I don't want to invent bicycle and sure that scalatest have required abilities. But I don't know how to get them out.

Best regards,

   Dmytro

[Bill Venners]

Hi Dmitry,

What I'd suggest is that you create two FreeSpec classes (assuming
that's what you're using), Phase1Tests and Phase2Tests, to contain the
relevant tests for each that you want to run in parallel. Mix
ParallelTestExecution into those two classes. Then create a suite that
contains returns instance of each of those two classes, in the order
you want them to run, from its nestedSuites method. Then mix
SequentialNestedSuites execution into that outer suite. Then run the
whole shebang with parallel enabled and you're good to go.

If you are using some kind of discovery, you'll want the nested sites
to not be discovered. If you're using a 2.0 milestone, you can
annotate them with a @DoNotDiscover annotation. Another way is to give
them a primary constructor that takes a parameter. A good way to do
that is to put it in as a nested class, because nested classes will
get a constructor with a reference to the enclosing class instance.
So, something like:

class PhasesSpec extends FlatSpec with SequentialNestedSuitesExecution {

class Phase1Tests extends FlatSpec with ParallelTestExecution {
// ...
}

class Phase2Tests extends FlatSpec with ParallelTestExecution {
// ...
}

override val nestedSuites = Vector(new Phase1Tests, new Phase2Tests)
}

Please let me know if you have any trouble getting this working.

Bill

> --
> --
> You received this message because you are subscribed to the Google
> Groups "scalatest-users" group.
> To post to this group, send email to scalate...@googlegroups.com
> To unsubscribe from this group, send email to
> scalatest-use...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/scalatest-users?hl=en
> ScalaTest itself, and documentation, is available here:
> http://www.artima.com/scalatest
> ---
> You received this message because you are subscribed to the Google Groups
> "scalatest-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to scalatest-use...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>



--
Bill Venners
Artima, Inc.
http://www.artima.com

[Dmitry Makhno]

Oh, Thanks Bill. I didn't know about this abilities. And will try to use soon. 

Unfortunately this is not what I need. I was not clear enough on what I need... Bellow are timeline:

Using your suggestion I tried to reformat into following:

class mixParallelAndSequentialTest extends FreeSpec with ParallelTestExecution with Logging{

  class SequentialTest(alias:String) extends FreeSpec with CancelAfterFailure{

    "Configuration for %s".format(alias) - {

      "Phase 1" in {

        for (j <- 0 until 10) {

          logger.info("working %s     Phase-1 %s".format(alias,j))

          Thread.sleep(50)

        }

      }

      "Phase 2" in {

        for (j <- 0 until 10) {

          logger.info("working %s     Phase-2 %s".format(alias,j))

          Thread.sleep(50)

        }

      }

    }

  }

  override val nestedSuites = Vector("Config1", "Config2", "Config3").map(new SequentialTest(_))  //this are not parallel still

}

What I have, If it is still not clear I will do an image

---------------->t

Config1[Phase1->Phase2] -> Config2[Phase1->Phase2] -> ... -> ConfigN[Phase1->Phase2]

What I really want

---------------->t

Config1[Phase1->Phase2]

Config2[Phase1->Phase2]

ConfigN[Phase1->Phase2]

And It is important that I don't know counts of config. 

After your snippet I tired to do:

- Make parrent with ParallelTestExecution

- Create test method as nested class with CancelAfterFailure. This is nice that this part is sequential, I don't need, at least now, parallel here.

- Override nestedSuite in FreeSpec

And I still see, that Config2 only executed after Config1, and I expect that execution of Config1 and for example Config3 will be almost at the same time.

Sincerelly,

   Dmytro

[Bill Venners]

Hi Dmitry,

I am about to leave for the airport, but I printed this out and will
read it in the cab. Should be able to respond from airport wireless
before my flight.

Bill

[Bill Venners]

Hi Dmitry,

Cab was late so answering via my phone. It sounds like you have multiple batches of tests, want to run each batch of tests sequentially,  but run the batches themselves in parallel. If that's correct you're in luck, because that's ScalaTest's default. Just put each batch of tests into its own FreeSpec class then run them with parallel enabled (-P in Runner). The FreeSpecs will execute in parallel, but the tests within each FreeSpec will run sequentially. 

Bill

[Dmitry Makhno]

Hi Bill,

I use this functionality a lot, and sbt+scalatest are good in parallel run. BUT.

The reason why I want to mix is because configs are stored separately. And some people, that are not related to test-automation and coding, still able to change that config.

This config drives test and implement so called "data driven testing".

I have a bunch of integration tests that holds integration with some third party vendors. The tests keeps the same and it is only few configuration changes, I don't want to put this configuration into scala code and keep it outside...

The good example is compatibility.

Using IaaS I'm able to run tests on different VMs. And I have requirement to keep compatibility with some parts for Ubuntu, Centos, Redhat. Generally this is _one_ test. But with different parameters to run with.

Now I have FreeSpec Tests like SomethingUbuntuTest, SomethingCentosTest... it was convenient before there were few of them... And more and more, I have a need that different team mate were able to change configuration file and rerun same test (compiled once) but with different configs.

I'm thinking on it in this way... when I'll have "new Linux", I get new row in my compatibility matrix, and NOT in my test suite.

And I have several dimensions of this compatibility: different IaaS APIs, VM images, OSs, vendors, cookbooks, etc. To control this is from code, is not a right place in process that I have.

Is it possible to get what I need? 

Sincerely,

  Dmytro

[Bill Venners]

Hi Dmitry,

You get a nice play by play of my evening! Cab was 20 minutes late but once I got to the airport I realized I'd misread my boarding pass and had an extra half hour. Which means I both have time for a pizza and to code up an example for you. Yes you can have what you want quite simple by creating the nested FreeSpecs dynamically with youe for loop. I'll code up an example for you post-pizza.

Bill

[Dmitry Makhno]

Bill,

Have a good meal, thanks in advance.

Dynamic FreeSpecs - sounds amazing... and should fit my need.

Dmity

[Bill Venners]

Hi Dmitry,

Actually in looking at your second round of code, it looks about
right. Not sure why it wouldn't run in parallel, but didn't have time
to run it. Here's one that seems to:

import org.scalatest._

class PhasesSpec(override val suiteName: String) extends FreeSpec with
CancelAfterFailure {
"Configuration for %s".format(suiteName) - {

"Phase 1" in {
for (j <- 0 until 10) {

println("working %s Phase-1 %s".format(suiteName, j))

Thread.sleep(50)
}
}
"Phase 2" in {
for (j <- 0 until 10) {

println("working %s Phase-2 %s".format(suiteName, j))
Thread.sleep(50)
}
}
}
}

class DmitrySpec extends Suites(
(for (alias <- Vector("Config1", "Config2", "Config2"))
yield new PhasesSpec(alias)): _*
)

With this command line:

scala -cp target/jar_contents/ org.scalatest.tools.Runner -s
DmitrySpec -P -o -R .

The output looks like:

Run starting. Expected test count is: 6
DmitrySpec:
Config1:
Config2:
Config2:
Unfortunately Suite has been deprecated as a style trait. Please use
trait Spec instead.
Configuration for Config2
Configuration for Config1
Configuration for Config2
working Config2 Phase-1 0
working Config1 Phase-1 0
working Config2 Phase-1 0
working Config1 Phase-1 1
working Config2 Phase-1 1
working Config2 Phase-1 1
working Config1 Phase-1 2
working Config2 Phase-1 2
working Config2 Phase-1 2
working Config1 Phase-1 3
working Config2 Phase-1 3
working Config2 Phase-1 3
working Config2 Phase-1 4
working Config1 Phase-1 4
working Config2 Phase-1 4
working Config1 Phase-1 5
working Config2 Phase-1 5
working Config2 Phase-1 5
working Config1 Phase-1 6
working Config2 Phase-1 6
working Config2 Phase-1 6
working Config2 Phase-1 7
working Config2 Phase-1 7
working Config1 Phase-1 7
working Config1 Phase-1 8
working Config2 Phase-1 8
working Config2 Phase-1 8
working Config1 Phase-1 9
working Config2 Phase-1 9
working Config2 Phase-1 9
- Phase 1
working Config2 Phase-2 0
working Config2 Phase-2 0
working Config1 Phase-2 0
- Phase 1
- Phase 1
working Config1 Phase-2 1
working Config2 Phase-2 1
working Config2 Phase-2 1
working Config1 Phase-2 2
working Config2 Phase-2 2
working Config2 Phase-2 2
working Config1 Phase-2 3
working Config2 Phase-2 3
working Config2 Phase-2 3
working Config1 Phase-2 4
working Config2 Phase-2 4
working Config2 Phase-2 4
working Config2 Phase-2 5
working Config2 Phase-2 5
working Config1 Phase-2 5
working Config2 Phase-2 6
working Config2 Phase-2 6
working Config1 Phase-2 6
working Config2 Phase-2 7
working Config2 Phase-2 7
working Config1 Phase-2 7
working Config2 Phase-2 8
working Config1 Phase-2 8
working Config2 Phase-2 8
working Config2 Phase-2 9
working Config2 Phase-2 9
working Config1 Phase-2 9
- Phase 2
- Phase 2
- Phase 2
Run completed in 1 second, 136 milliseconds.
Total number of tests run: 6
Suites: completed 4, aborted 0
Tests: succeeded 6, failed 0, canceled 0, ignored 0, pending 0
All tests passed.

Is that what you're after? I'm in my seat now, so before long they'll
ask me to stop compiling Scala. After that it'll be several hours
before I can get on again.

Bill

[Dmitry Makhno]

Bill, thank you very much, the log exacly what I need.

Seems I'm stub a bit with sbt things, seems it is not very friendly to scalatest parallel execution. 

[Bill Venners]

Hi Dmitry,

You're use case is timely. We're going to try and ensure the new
Framework API gets into sbt 0.13. When sbt runs the show, it has the
thread pool, so anything done in parallel should really be done with
threads from that pool, not a pool inside ScalaTest. I think the new
Framework API should already allow it, but I'll make sure we can do
what you are trying to do from sbt when we implement this. It is here
by the way:

https://github.com/bvenners/test-interface/tree/master/src/org/scalasbt/testing

It has been languishing there for looks like almost a year now.

Bill

[Dmitry Makhno]

Hi Bill,

Thanks for information, we don't yet use sbt 0.13 yet... And if it will be valuable for testing sbt+scalatest I'd like to add some cases.

I tried on 

SBT: 0.12.1

Scala: 2.9.2

ScalaTest: 2.0.M3

... they are not the latest.

Project settings:

    .configs( IntegrationTest )

    .settings( Defaults.itSettings : _*)

    .settings(

      parallelExecution in IntegrationTest := true,

      concurrentRestrictions in Global := Seq(

        Tags.limit(Tags.Test, 6)))

Tests:

https://gist.github.com/dmakhno4gd/5029402

Sbt commands

1. it:test-only parallel*

Actual: All 20 tests were executed in parallel.

Expected: Only 6 tests are executed in parallel, inheriting concurentRestriction.

Note: I found it important to parallel more that cpu-cores count. (I have 4 cores, and for testing using restriction as 6), parallel execution used to limit it on default.

2. it:test-only dynamicParallelTests

Actual: dynamic tests are executed one by one - in a sequence

Expected: All generated tests executed in parallel, according to sbt restrictions

3. it:test-only dynamicParallelTests2

... Same as 2.

Please let me know if I should re-post it into sbt group. I still feel frustrated among sbt and scalatest thread pools, and not sure whom should be addressed. Seems for me this is integration issues and both teams should be aware.

Looking forward to hearing from you,

Sincerely,

   Dmytro

[Bill Venners]

Hi Dmitry.

sbt 0.13 isn't due to be rc'd until April. The current release is
0.12.x. The real problem is the Framework API which sbt uses to talk
to test frameworks. It provides no way to allow the test framework to
give sbt more jobs to run in parallel. This is possible in the new
Framework API, but this has not been integrated into sbt as yet. After
six months of waiting for it to get integrated I finally realized we'd
better do it outselves, so that's the plan. We are going to try to get
2.0.M6 out in the next couple of weeks, after which we'll focus on
enhancing both sbt and ScalaTest with the new Framework API. The test
case you provided here will be very useful at that time. I'll post to
this thread when there's something to try.

Bill

[Bill Venners]

Hi Dmitry,

One other explanation that may help you understand the situation is
that what will happen with the existing, original Framework API is
that only test classes that sbt discovers will be run in parallel.
Since sbt doesn't discover the PhasesSpec, because it does not have a
no-arg constructor, that one won't get executed in parallel. What
ScalaTest could do is pass its own distributor down, but I think
that's wrong because sbt should have the thread pool, and besides we
have no way of knowing whether the user asked for sequential execution
in the build file. So it should really be sbt, and this requires the
new Framework API.

I did want to give you one more tweak to the code, though, which is this:

import org.scalatest._

class PhasesSpec(override val suiteName: String) extends FreeSpec with
CancelAfterFailure {

"Configuration for %s".format(suiteName) - {

"Phase 1" in {
for (j <- 0 until 10) {

println("working %s Phase-1 %s".format(suiteName, j))
Thread.sleep(50)
}
}

"Phase 2" in {

for (j <- 0 until 10) {

println("working %s Phase-2 %s".format(suiteName, j))
Thread.sleep(50)
}
}
}

override def suiteId = suiteName
}

class DmitrySpec extends Suites(
(for (alias <- Vector("Config1", "Config2", "Config3"))

yield new PhasesSpec(alias)): _*
)

I fixed a Config2 => Config3 typo in my earlier at-airport-written
code, but main thing is I added an override of suiteId. This enables
sorting of events with -PS. If you run with just plain -P, you get no
special sorting of events:

Run starting. Expected test count is: 6
DmitrySpec:
Config1:
Config2:

Config3:
Configuration for Config3

Configuration for Config1
Configuration for Config2

working Config3 Phase-1 0

working Config1 Phase-1 0
working Config2 Phase-1 0

working Config3 Phase-1 1
working Config2 Phase-1 1
working Config1 Phase-1 1
working Config1 Phase-1 2
working Config3 Phase-1 2
working Config2 Phase-1 2
working Config3 Phase-1 3

working Config1 Phase-1 3
working Config2 Phase-1 3

working Config3 Phase-1 4
working Config2 Phase-1 4
working Config1 Phase-1 4
working Config1 Phase-1 5
working Config3 Phase-1 5

working Config2 Phase-1 5
working Config1 Phase-1 6
working Config2 Phase-1 6

working Config3 Phase-1 6
working Config3 Phase-1 7

working Config2 Phase-1 7
working Config1 Phase-1 7

working Config3 Phase-1 8
working Config2 Phase-1 8
working Config1 Phase-1 8
working Config3 Phase-1 9

working Config1 Phase-1 9
working Config2 Phase-1 9

working Config2 Phase-2 0
working Config1 Phase-2 0

working Config3 Phase-2 0

- Phase 1
- Phase 1
- Phase 1
working Config1 Phase-2 1
working Config2 Phase-2 1

working Config3 Phase-2 1
working Config1 Phase-2 2
working Config3 Phase-2 2
working Config2 Phase-2 2
working Config3 Phase-2 3
working Config2 Phase-2 3
working Config1 Phase-2 3

working Config1 Phase-2 4
working Config2 Phase-2 4

working Config3 Phase-2 4
working Config3 Phase-2 5
working Config1 Phase-2 5
working Config2 Phase-2 5
working Config3 Phase-2 6
working Config1 Phase-2 6
working Config2 Phase-2 6
working Config1 Phase-2 7
working Config3 Phase-2 7
working Config2 Phase-2 7

working Config1 Phase-2 8
working Config2 Phase-2 8

working Config3 Phase-2 8
working Config2 Phase-2 9
working Config1 Phase-2 9
working Config3 Phase-2 9

- Phase 2
- Phase 2
- Phase 2

Run completed in 1 second, 243 milliseconds.

Total number of tests run: 6
Suites: completed 4, aborted 0
Tests: succeeded 6, failed 0, canceled 0, ignored 0, pending 0
All tests passed.

That was as before. If you say -PS, though, it will sort the events.
Even though it goes in parallel, you get sorted output as if you ran
sequentially. You can see it is running in parallel because the
printlns are still coming out in parallel:

$ scala -cp target/jar_contents/ org.scalatest.tools.Runner -s
DmitrySpec -PS -o -R .

Run starting. Expected test count is: 6
DmitrySpec:
Config1:

Configuration for Config1
working Config1 Phase-1 0
working Config3 Phase-1 0
working Config2 Phase-1 0
working Config3 Phase-1 1
working Config2 Phase-1 1
working Config1 Phase-1 1

working Config1 Phase-1 2
working Config2 Phase-1 2

working Config3 Phase-1 2

working Config1 Phase-1 3
working Config2 Phase-1 3

working Config3 Phase-1 3
working Config3 Phase-1 4

working Config1 Phase-1 4
working Config2 Phase-1 4

working Config3 Phase-1 5

working Config1 Phase-1 5
working Config2 Phase-1 5

working Config2 Phase-1 6
working Config1 Phase-1 6
working Config3 Phase-1 6
working Config2 Phase-1 7
working Config3 Phase-1 7
working Config1 Phase-1 7
working Config2 Phase-1 8
working Config3 Phase-1 8
working Config1 Phase-1 8
working Config2 Phase-1 9
working Config3 Phase-1 9
working Config1 Phase-1 9
- Phase 1
working Config1 Phase-2 0
working Config3 Phase-2 0
working Config2 Phase-2 0
working Config1 Phase-2 1
working Config3 Phase-2 1

working Config2 Phase-2 1
working Config1 Phase-2 2
working Config2 Phase-2 2

working Config3 Phase-2 2
working Config1 Phase-2 3
working Config3 Phase-2 3

working Config2 Phase-2 3
working Config1 Phase-2 4
working Config2 Phase-2 4

working Config3 Phase-2 4
working Config1 Phase-2 5
working Config2 Phase-2 5
working Config3 Phase-2 5
working Config1 Phase-2 6
working Config2 Phase-2 6
working Config3 Phase-2 6
working Config1 Phase-2 7
working Config3 Phase-2 7
working Config2 Phase-2 7

working Config1 Phase-2 8
working Config2 Phase-2 8

working Config3 Phase-2 8
working Config1 Phase-2 9
working Config2 Phase-2 9
working Config3 Phase-2 9
- Phase 2
Config2:
Configuration for Config2
- Phase 1
- Phase 2
Config3:
Configuration for Config3
- Phase 1
- Phase 2
Run completed in 1 second, 178 milliseconds.

Total number of tests run: 6
Suites: completed 4, aborted 0
Tests: succeeded 6, failed 0, canceled 0, ignored 0, pending 0
All tests passed.

Once you're convinced it is running in parallel, you can change the
println to info calls, which result in events that will also get
sorted. Now the whole thing is in sequential order at the output, even
though it was run in parallel:

import org.scalatest._

class PhasesSpec(override val suiteName: String) extends FreeSpec with
CancelAfterFailure {

"Configuration for %s".format(suiteName) - {

"Phase 1" in {
for (j <- 0 until 10) {

info("working %s Phase-1 %s".format(suiteName, j))
Thread.sleep(50)
}
}
"Phase 2" in {

for (j <- 0 until 10) {

info("working %s Phase-2 %s".format(suiteName, j))
Thread.sleep(50)
}
}
}
override def suiteId = suiteName
}

class DmitrySpec extends Suites(
(for (alias <- Vector("Config1", "Config2", "Config3"))

yield new PhasesSpec(alias)): _*
)

$ scala -cp target/jar_contents/ org.scalatest.tools.Runner -s
DmitrySpec -PS -o -R .

Run starting. Expected test count is: 6
DmitrySpec:
Config1:

Configuration for Config1
- Phase 1
+ working Config1 Phase-1 0
+ working Config1 Phase-1 1
+ working Config1 Phase-1 2
+ working Config1 Phase-1 3
+ working Config1 Phase-1 4
+ working Config1 Phase-1 5
+ working Config1 Phase-1 6
+ working Config1 Phase-1 7
+ working Config1 Phase-1 8
+ working Config1 Phase-1 9
- Phase 2
+ working Config1 Phase-2 0
+ working Config1 Phase-2 1
+ working Config1 Phase-2 2
+ working Config1 Phase-2 3
+ working Config1 Phase-2 4
+ working Config1 Phase-2 5
+ working Config1 Phase-2 6
+ working Config1 Phase-2 7
+ working Config1 Phase-2 8
+ working Config1 Phase-2 9
Config2:
Configuration for Config2
- Phase 1
+ working Config2 Phase-1 0
+ working Config2 Phase-1 1
+ working Config2 Phase-1 2
+ working Config2 Phase-1 3
+ working Config2 Phase-1 4
+ working Config2 Phase-1 5
+ working Config2 Phase-1 6
+ working Config2 Phase-1 7
+ working Config2 Phase-1 8
+ working Config2 Phase-1 9
- Phase 2
+ working Config2 Phase-2 0
+ working Config2 Phase-2 1
+ working Config2 Phase-2 2
+ working Config2 Phase-2 3
+ working Config2 Phase-2 4
+ working Config2 Phase-2 5
+ working Config2 Phase-2 6
+ working Config2 Phase-2 7
+ working Config2 Phase-2 8
+ working Config2 Phase-2 9
Config3:
Configuration for Config3
- Phase 1
+ working Config3 Phase-1 0
+ working Config3 Phase-1 1
+ working Config3 Phase-1 2
+ working Config3 Phase-1 3
+ working Config3 Phase-1 4
+ working Config3 Phase-1 5
+ working Config3 Phase-1 6
+ working Config3 Phase-1 7
+ working Config3 Phase-1 8
+ working Config3 Phase-1 9
- Phase 2
+ working Config3 Phase-2 0
+ working Config3 Phase-2 1
+ working Config3 Phase-2 2
+ working Config3 Phase-2 3
+ working Config3 Phase-2 4
+ working Config3 Phase-2 5
+ working Config3 Phase-2 6
+ working Config3 Phase-2 7
+ working Config3 Phase-2 8
+ working Config3 Phase-2 9
Run completed in 1 second, 167 milliseconds.

Total number of tests run: 6
Suites: completed 4, aborted 0
Tests: succeeded 6, failed 0, canceled 0, ignored 0, pending 0
All tests passed.

Bill

[Dmitry Makhno]

Hi Bill,

Thanks again, for some new stuff.

I had several days to think about:

What 
ScalaTest could do is pass its own distributor down, but I think 
that's wrong because sbt should have the thread pool, and besides we 
have no way of knowing whether the user asked for sequential execution 
in the build file. So it should really be sbt, and this requires the 
new Framework API. 

Imho, scalatest should have ability to pass in its thread pool.

I can now only imagine scenario when in general sbt put into sequential run, but some tests requires parallel execution, thus they should be forced to use scalaTest thread pool. If it can be controlled by annotation or something like this. Other possible scenario when test is required to run for example in 10 threads in parallel, and what if sbt never gives a such pool size.

The only reasonable example seems for me is multi-jvm. I didn't play with it yet, but seems sbt+scalatest+multi-jvm can be also confusing understanding and ensuring how it is paralleled.

Regards, 

   Dmytro