Generating tests with FlatSpec

[David Capwell]

I am new to scalatest and trying to get data driven testing to work with FlatSpec and not getting it to work.  I have the following.

class FlatTests extends FlatSpec with Matchers {

  // test marked as fail

  "a for loop" should "do logic" in {

    println("hi")

    true should be (false)

  }

  // test marked as fail

  "a for loop" should "do logic and still move on" in {

    println("hi")

    true should be (false)

  }

  // all tests pass, nothing gets logged to stdout

  for {i <- 1 to 10} {

    s"inline for loop: $i" should "do logic" in {

      println(s"hi $i")

      true should be (false)

    }

  }

  // no tests run

  // An exception or error caused a run to abort: true was not false

  // org.scalatest.exceptions.TestFailedException: true was not false

  for {i <- 1 to 10} {

    s"inline for loop: $i" should behave like {

      println(s"hi $i")

      true should be (false)

    }

  }

}

I tried the same with FunSuite and it works how i thought it would

class FunTests extends FunSuite with ShouldMatchers {

  for {i <- 1 to 10} {

    test(s"test failure: $i") {

      true should be (false)

    }

  }

}

Any reason that this works with FunSuite but not FlatSpec?

Using scala 2.10 and scalatest 2.0

[David Capwell]

One other thing I am seeing is that when I try the other specs that let me nest behaviors, it seems that only the in clause can do work.  I have the following

class ClustersResourceTest extends WordSpec with Matchers {

  "create new cluster" should {

    val inputCluster = Cluster(name = "newCluster", description = "random content")

    val rsp: ClientResponse = createCluster(inputCluster)

    "have response status of 200" in {

      rsp.getStatus should be(200)

    }

    "have content-type of application/json" in {

      rsp.getHeaders.getFirst(HttpHeaders.CONTENT_TYPE) should be(MediaType.JSON_UTF_8.`type`())

    }

    "return created cluster" should {

      val outputCluster = rsp.getEntity(classOf[ClusterWithMeta])

      "have matching timestamps for createTS and modTS" in {

        outputCluster.createTS should be(outputCluster.modTS)

      }

    }

  }

}

createCluster is currently implemented as ??? since I am working on the tests before I work on the code.  When I run I see that no tests run and the engine crashes with the following:

An exception or error caused a run to abort: an implementation is missing 

scala.NotImplementedError: an implementation is missing

...

at org.scalatest.SuperEngine.registerNestedBranch(Engine.scala:613)

at org.scalatest.WordSpecLike$class.org$scalatest$WordSpecLike$$registerBranch(WordSpecLike.scala:139)

at org.scalatest.WordSpecLike$$anon$2.apply(WordSpecLike.scala:870)

...

Is there any spec that lets me nest behaviors where different levels do work?  Going over the docs I don't see any docs talking about this, and it wasn't clear if any of the default specs would allow this.

[Bill Venners]

Hi David,

An "in" clause is what registers a test at suite construction time. It takes a by-name that is executed later when run is called. The scopes surrounding the in clauses, like "should" in FlatSpec, are executed at construction time. So if you have an exception that occurs in a scope, it will abort the suite, because it is unable to actually construct the suite. The reason the scopes aren'te evaluated lazily is so a suite can know up front how many tests it contains. It returns this number from expectedTestCount, and that's used to make a nice progress bar, such as you see in IntelliJ or Eclipse.

The trick if you are calling something from within a scope that will blow up because of a ??? is to make it lazy. That will delay the execution of that ??? until the tests are actually run. Here's what it would look like in your case:

class ClustersResourceTest extends WordSpec with Matchers {

  "create new cluster" should {

    lazy val inputCluster = Cluster(name = "newCluster", description = "random content")

    lazy val rsp: ClientResponse = createCluster(inputCluster)

    "have response status of 200" in {

      rsp.getStatus should be(200)

    }

    "have content-type of application/json" in {

      rsp.getHeaders.getFirst(HttpHeaders.CONTENT_TYPE) should be(MediaType.JSON_UTF_8.`type`())

    }

    "return created cluster" should {

      lazy val outputCluster = rsp.getEntity(classOf[ClusterWithMeta])

      "have matching timestamps for createTS and modTS" in {

        outputCluster.createTS should be(outputCluster.modTS)

      }

    }

  }

}

Sorry I was doing a build and couldn't try the example in your earlier email yesterday, then forgot to go back to it. I'll look at that now.

Bill

--
You received this message because you are subscribed to the Google
Groups "scalatest-users" group.
To post to this group, send email to scalate...@googlegroups.com
To unsubscribe from this group, send email to
scalatest-use...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/scalatest-users?hl=en
ScalaTest itself, and documentation, is available here:
http://www.artima.com/scalatest
---
You received this message because you are subscribed to the Google Groups "scalatest-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scalatest-use...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Bill Venners
Artima, Inc.
http://www.artima.com

[David Capwell]

Thanks for your reply.  As you said, adding lazy to anything that could fail will get a similar behavior to what i want (the tests to all fail).  One problem there is that since the val will fail later on, each test will get the same error, that ??? was used.  If the issue is a ephemeral issue (random network issue), the first test could fail, the rest would cause a retry and pass (cause this api throws exception rather than returning a monad; which is a different issue).  It would be nice that if a suite and nested suite had an exception at construction time, that it would be marked as failed with the reason rather than things halting all together.  

With lazy I get the following output

// all red and failed

create new cluster

  have response status of 200

  have content-type of application/json

  should return created cluster

    have matching timestamp for createTS and modTS

what would be ideal for me is to get just the suite and not the sub tests (since that may cause me to try to debug why the sub-tests failed, when it was really the suite)

// marked red and failed

create new cluster

[Bill Venners]

Hi David,

I think you had something else going on in the first example that gave you trouble. It looks fine, and it also works fine:

This code:

import org.scalatest._

class FlatTests extends FlatSpec with Matchers {

  // test marked as fail
  "a for loop" should "do logic" in {
    println("hi")
    true should be (false)
  }

  // test marked as fail
  "a for loop" should "do logic and still move on" in {
    println("hi")
    true should be (false)
  }

  // all tests pass, nothing gets logged to stdout
  for {i <- 1 to 10} {
    s"inline for loop: $i" should "do logic" in {
      println(s"hi $i")
      true should be (false)
    }
  }

/*

  // no tests run
  // An exception or error caused a run to abort: true was not false
  // org.scalatest.exceptions.TestFailedException: true was not false
  for {i <- 1 to 10} {
    s"inline for loop: $i" should behave like {
      println(s"hi $i")
      true should be (false)
    }
  }

*/
}

Gives this result:

$ scala -cp target/jar_contents/ org.scalatest.tools.Runner -o -s FlatTests -R .
Run starting. Expected test count is: 12
FlatTests:
a for loop
hi
hi
- should do logic *** FAILED ***
  true was not false (delme.scala:8)
hi 1
a for loop
- should do logic and still move on *** FAILED ***
  true was not false (delme.scala:14)
hi 2
inline for loop: 1
- should do logic *** FAILED ***
  true was not false (delme.scala:21)
hi 3
inline for loop: 2
- should do logic *** FAILED ***
  true was not false (delme.scala:21)
inline for loop: 3
hi 4
- should do logic *** FAILED ***
  true was not false (delme.scala:21)
inline for loop: 4
hi 5
- should do logic *** FAILED ***
  true was not false (delme.scala:21)
inline for loop: 5
hi 6
- should do logic *** FAILED ***
  true was not false (delme.scala:21)
inline for loop: 6
hi 7
- should do logic *** FAILED ***
  true was not false (delme.scala:21)
inline for loop: 7
hi 8
- should do logic *** FAILED ***
  true was not false (delme.scala:21)
inline for loop: 8
hi 9
- should do logic *** FAILED ***
  true was not false (delme.scala:21)
inline for loop: 9
hi 10
- should do logic *** FAILED ***
  true was not false (delme.scala:21)
inline for loop: 10
- should do logic *** FAILED ***
  true was not false (delme.scala:21)
Run completed in 259 milliseconds.
Total number of tests run: 12
Suites: completed 1, aborted 0
Tests: succeeded 0, failed 12, canceled 0, ignored 0, pending 0
*** 12 TESTS FAILED ***

The second example you gave is actually working as it is supposed to. This code will indeed abort the suite:

  // no tests run
  // An exception or error caused a run to abort: true was not false
  // org.scalatest.exceptions.TestFailedException: true was not false
  for {i <- 1 to 10} {
    s"inline for loop: $i" should behave like {
      println(s"hi $i")
      true should be (false)
    }
  }

The reason is the "should behave like" syntax is just syntax sugar for registering shared tests. But you don't actually have an tests in there. What you want to say is something like:

import org.scalatest._

class FlatTests extends FlatSpec with Matchers {

/*

  // no tests run
  // An exception or error caused a run to abort: true was not false
  // org.scalatest.exceptions.TestFailedException: true was not false
  for {i <- 1 to 10} {
    s"inline for loop: $i" should behave like {
      println(s"hi $i")
      true should be (false)
    }
  }

*/

  def aGoodLoop(i: Int): Unit = {
    it should "do logic: " + i in {

      println(s"hi $i")
      true should be (false)
    }
  }

  behavior of "A loop"

  for (i <- 1 to 10) {
    it should behave like aGoodLoop(i)
  }
}

When run this gives:

$ scala -cp target/jar_contents/ org.scalatest.tools.Runner -o -s FlatTests -R .
Run starting. Expected test count is: 10
FlatTests:
A loop
hi 1
hi 2
hi 3
hi 4
hi 5
hi 6
hi 7
- should do logic: 1 *** FAILED ***
  true was not false (delme.scala:20)
hi 8
- should do logic: 2 *** FAILED ***
  true was not false (delme.scala:20)
hi 9
- should do logic: 3 *** FAILED ***
  true was not false (delme.scala:20)
hi 10
- should do logic: 4 *** FAILED ***
  true was not false (delme.scala:20)
- should do logic: 5 *** FAILED ***
  true was not false (delme.scala:20)
- should do logic: 6 *** FAILED ***
  true was not false (delme.scala:20)
- should do logic: 7 *** FAILED ***
  true was not false (delme.scala:20)
- should do logic: 8 *** FAILED ***
  true was not false (delme.scala:20)
- should do logic: 9 *** FAILED ***
  true was not false (delme.scala:20)
- should do logic: 10 *** FAILED ***
  true was not false (delme.scala:20)
Run completed in 247 milliseconds.
Total number of tests run: 10
Suites: completed 1, aborted 0
Tests: succeeded 0, failed 10, canceled 0, ignored 0, pending 0
*** 10 TESTS FAILED ***

Let me know if you have other questions. I'll try and find some way to elaborate in the documentation on when things execute. In short, only tests themselves are not executed at construction time. Scopes (describe clauses, should/must/can clauses, etc.) and everything else in the constructor are executed at construction time. If an exception happens during construction for whatever reason, a SuiteAborted event is fired and no tests will ever run. If you want to a test failure instead, then you need to delay execution of that code until tests are run. The usual way to do that is make variables lazy.

Bill

--

You received this message because you are subscribed to the Google
Groups "scalatest-users" group.
To post to this group, send email to scalate...@googlegroups.com
To unsubscribe from this group, send email to
scalatest-use...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/scalatest-users?hl=en
ScalaTest itself, and documentation, is available here:
http://www.artima.com/scalatest
---
You received this message because you are subscribed to the Google Groups "scalatest-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scalatest-use...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[David Capwell]

ok, i think i found whats up.  The example I gave, I had removed the before/after calls that I had.  When I run the example above everything fails as expected.  When I add before/after in thats when everything passes (running in intellij not cli)

class FlatTests extends FlatSpec with Matchers with BeforeAndAfter with BeforeAndAfterAll {

  override def beforeAll() {

    println("before all")

  }

  // Delete the temp file

  override def afterAll() {

    println("after all")

  }

  before {

    println("before")

  }

  after {

    println("after")

  }

  // test marked as fail

  "a for loop" should "do logic" in {

    println("hi")

    true should be(false)

  }

  // test marked as fail

  "a for loop" should "do logic and still move on" in {

    println("hi")

    true should be(false)

  }

  // all tests pass, nothing gets logged to stdout

  for {i <- 1 to 10} {

    s"inline for loop: $i" should "do logic" in {

      println(s"hi $i")

      true should be(false)

    }

  }

  // no tests run

  // An exception or error caused a run to abort: true was not false

  // org.scalatest.exceptions.TestFailedException: true was not false

  //  for {i <- 1 to 10} {

  //    s"inline for loop: $i" should behave like {

  //      println(s"hi $i")

  //      true should be (false)

  //    }

  //  }