Multiple before blocks in Spec with BeforeAndAfter

[Rich Apodaca]

I'm interested in using multiple before blocks like other BDD
frameworks allow. For example:

class NestedSpec extends Spec with BeforeAndAfter {
describe("First level") {
before {
// initialize first level
}
describe("Second level") {
before {
// initialize second level
}
}
}
}

Unfortunately doing so with ScalaTest-1.6.1 gives this runtime error:

org.scalatest.NotAllowedException: You are only allowed to call before
once in each Suite that mixes in BeforeAndAfter.

Poking around the source for BeforeAndAfter:

http://code.google.com/p/scalatest/source/browse/trunk/src/main/scala/org/scalatest/BeforeAndAfter.scala

I notice that the BeforeAndAfter trait uses a single reference to the
before function.

Is there any reason why BeforeAndAfter couldn't support multiple
before and after blocks, each nested in its own describe block?

Thanks,
Rich

[Bill Venners]

Hi Rich,

Interesting request. Yes, before is just designed to be used at the
top level. A before invocation registers code you want to run before
each test in the suite (or spec). I have had two other people suggest
an enhancement for before, but not the one you are talking about.
Others wanted to be able to stack before invocations like you can
stack beforeEach in BeforeAndAfterEach. BeforeAndAfter is really just
a replacement for JUnit's setup and teardown that has minimal
boilerplate, but that can't (currently at least) be stacked. If you
don't know what I mean by "stacking," it is described here:

http://www.scalatest.org/user_guide/sharing_fixtures#composingFixtures

Can I ask whether you'd want each test to see fresh copies of all the
mutations made in the before clauses? For example:

class StackSpec extends Spec with MustMatchers {

val stack = new scala.collection.mutable.Stack[String]

describe("A stack") {

describe("when empty") {

it("is empty") {
stack must be ('empty)
}

it("complains on pop") {
evaluating { stack.pop() } must produce [NoSuchElementException]
}
}

describe("when two objects have been pushed onto a stack") {

before {
stack.push("pushed first")
stack.push("pushed last")
}

it("should pop the last value pushed first") {
val poppedFirst = stack.pop()
poppedFirst must be ("pushed last")
}

// For this test to work, it must not see side effects of the previous test
it("should pop the first object pushed second") {
stack.pop()
val poppedLast = stack.pop()
poppedLast must be ("pushed first")
}

// For this test to work, it must not see side effects of the
previous two tests
it("should again be empty after popping both pushed objects") {
stack.pop()
stack.pop()
stack must be 'empty
}
}
}

Would you expect it to work the way shown above, with that kind of
isolation, or would you not expect or prefer that? By the way, what
other test frameworks are you talking about when you say "other BDD
frameworks." Does RSpec do this perhaps? Do they isolate variables
like I described?

Thanks.

Bill

> --
> You received this message because you are subscribed to the Google
> Groups "scalatest-users" group.
> To post to this group, send email to scalate...@googlegroups.com
> To unsubscribe from this group, send email to
> scalatest-use...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/scalatest-users?hl=en
> ScalaTest itself, and documentation, is available here:
> http://www.artima.com/scalatest

--
Bill Venners
Artima, Inc.
http://www.artima.com

[Richard Apodaca]

Hello Bill,

> Would you expect it to work the way shown above, with that kind of

isolation, or would you not expect or prefer that?

Yes, exactly.  Every 'it' block should start with a clean scope built up from successively applying any 'before' blocks defined above it.

> By the way, what
other test frameworks are you talking about when you say "other BDD
frameworks." Does RSpec do this perhaps? Do they isolate variables
like I described?

Both RSpec and Jasmine (JavaScript), allow unlimited 'before' blocks - each one adding state to the ones defined above it hierarchically, but acting independently of 'before' blocks defined at the same level.

I've found this useful when defining specs that branch over various possible scenarios. Rather than build up the context for every single test (as I need to do in JUnit), I can build it up progressively and test at each stage, testing branches along the way.

This is another take on the feature:

http://blog.codefront.net/2007/12/30/why-i-love-rspec-nested-example-groups/

Thanks,

Rich

Richard L. Apodaca 

Founder and CEO 

Metamolecular, LLC 

8070 La Jolla Shores Drive #464 

La Jolla, CA 92037 

858.658.0860 

 

http://metamolecular.com 

[Bill Venners]

Hi Richard,

Well then you're the second official request for that sort of
behavior. The first one came in about a year ago. I did a lot of
thinking about it, my main concern being it might be too non-obvious
what was going on behind the scenes. Ultimately decided it was
worthwhile to support so long as I could come up with a way that makes
the execution model easy to understand. What I came up with is three
separate style classes (not traits) in a new package,
org.scalatest.path, that exhibited this behavior. The three would
correspond to the three style traits that allow nesting: Spec,
FreeSpec, and WordSpec.

The special way these would execute is that they by definition would
run each test in its own instance of the class (extending
OneInstancePerTest), but would only execute the "path" to the test and
nothing else. This is like your statement, "Every 'it' block should

start with a clean scope built up from successively applying any

'before' blocks defined above it," except in this case, you wouldn't
need to say "before," and "after" would work as well. If you place a
statement textually before a test, it will execute before. If you
place a statement textually after a test, it will execute after.
Anything that does not appear textually before or after a test will
*not* be executed in the instance that executes that test.

For this to work, the initial instance of the class must know to
record all the paths, and the subsequent instances (one per test) must
execute just one path (so they need to know which path to execute).
The most obvious way to get that path to the instance is to pass it
into a constructor, which is why these would be style *classes*
instead of traits.

The code would look like:

import org.scalatest.path
import org.scalatest.matchers.MustMatchers

class StackSpec(p: Path = All) extends path.Spec(p) with MustMatchers {

describe("A stack") {

val stack = new scala.collection.mutable.Stack[String]

describe("when two objects have been pushed onto a stack") {

stack.push("pushed first")
stack.push("pushed last")

it("should pop the last value pushed first") {
val poppedFirst = stack.pop()
poppedFirst must be ("pushed last")
}

it("should pop the first object pushed second") {

stack.pop()
val poppedLast = stack.pop()
poppedLast must be ("pushed first")
}

it("should again be empty after popping both pushed objects") {

stack.pop()
stack.pop()
stack must be 'empty
}
}

describe("when empty") {

it("is empty") {
stack must be ('empty)
}

it("complains on pop") {
evaluating { stack.pop() } must produce [NoSuchElementException]
}
}
}
}

Would this approach work for you?

Bill

[Richard Apodaca]

Hello Bill,

I hadn't thought of eliminating 'before' blocks... now that I have, I kind of like it.

And yes, the rest of the approach you've outlined is spot-on with how I'd want to use ScalaTest.

I'm not sure about the beginning of the class declaration:

class StackSpec(p: Path = All)

I'm new to Scala, but it looks like 'p' is not used in StackSpec. Is there any way a ScalaTest user would use the Path when writing a test? If not, it should ideally not be seen in StackSpec.

Thanks,

Rich

[Bill Venners]

Hi Richard,

On Thu, Dec 1, 2011 at 10:37 AM, Richard Apodaca
<rapo...@metamolecular.com> wrote:
> Hello Bill,
>

> I hadn't thought of eliminating 'before' blocks... now that I have, I kind
> of like it.
>

As far as I know, I think this approach was first suggested for Scala
testing by Esko Luontola. He created the specsy framework to do
something like this. You may want to take a look at that as well:

https://github.com/orfjackal/specsy

> And yes, the rest of the approach you've outlined is spot-on with how I'd
> want to use ScalaTest.
>
> I'm not sure about the beginning of the class declaration:
>
> class StackSpec(p: Path = All)
>
> I'm new to Scala, but it looks like 'p' is not used in StackSpec. Is there
> any way a ScalaTest user would use the Path when writing a test? If not, it
> should ideally not be seen in StackSpec.
>

The p is used only to pass the Path up to the superclass Spec:

class StackSpec(p: Path = All) extends Spec(p) { ...

It is relatively ugly boilerplate. There's a way to get rid of it,
which would also allow me to use a trait not a class, but it would
mean that there would be an exception to the rule that "if you want it
to happen before, put it before; if you want it to happen after, put
it after". I'd have to tack on, "*unless* you put it at the top level,
in which case it will happen before the test, even if you place it
after." At least I couldn't figure out a way to do it without that
caveat. So choosing between boilerplate and caveat, I thought it
better to go with boilerplate, because it makes the behavior more
consistent and I think may help people understand the execution model.
They can see a path is being passed up to the supertype, with the
default of All. That means the first time it is instantiated, it will
execute all paths to discover them, then it will pass a specific path
to each instance created to run a test. That was my thought anyway.
And there may be some way to do it that gives the best of both worlds
that I haven't thought of yet. Once I got to implement it, I may learn
more. If anyone has some other ideas let me know.

Thanks.

Bill

[Esko Luontola]

On Thursday, December 1, 2011 8:58:51 PM UTC+2, Bill Venners wrote:

Hi Richard,

On Thu, Dec 1, 2011 at 10:37 AM, Richard Apodaca
<rapo...@metamolecular.com> wrote:
> Hello Bill,
>
> I hadn't thought of eliminating 'before' blocks... now that I have, I kind
> of like it.
>
As far as I know, I think this approach was first suggested for Scala
testing by Esko Luontola. He created the specsy framework to do
something like this. You may want to take a look at that as well:

https://github.com/orfjackal/specsy

Yes. :)

In the documentation you can see the aforementioned stack example: http://specsy.org/documentation.html#isolated_execution_model

In Specsy the "before" blocks are implicit, in addition to which there are multiple "after" blocks which are executed in LIFO order as described in http://specsy.org/documentation.html#before_and_after_blocks

Nowadays there is also a non-isolated execution mode, which can be useful as a performance optimization or for writing tests for multi-step processes (if you wish to avoid deep nesting).

http://specsy.org/documentation.html#nonisolated_execution_model

BTW, adding that non-isolated execution mode was surprisingly simple, just modifying two lines and adding a couple more: https://github.com/orfjackal/specsy/commit/c31f8508969098b05f7bedac0a9fe9a1b6fe833a#diff-3

 

The p is used only to pass the Path up to the superclass Spec:

class StackSpec(p: Path = All) extends Spec(p) { ...

It is relatively ugly boilerplate.

In Specy I pass it using a thread local [1] where the value is stored right before the test class instance is created [2] and then the value is taken from there in the test class' constructor [3].

[1] https://github.com/orfjackal/specsy/blob/master/src/main/scala/net/orfjackal/specsy/core/ContextDealer.scala

[2] https://github.com/orfjackal/specsy/blob/master/src/main/scala/net/orfjackal/specsy/core/SpecClassRunner.scala

[3] https://github.com/orfjackal/specsy/blob/master/src/main/scala/net/orfjackal/specsy/core/ContextDealer.scala

 

In the Go language there are no thread locals, so with GoSpec the path/context is passed as a parameter to the test function [4].

[4] https://github.com/orfjackal/gospec/blob/release/examples/stack_test.go

[Esko Luontola]

On Thursday, December 1, 2011 8:58:51 PM UTC+2, Bill Venners wrote:

The p is used only to pass the Path up to the superclass Spec:

class StackSpec(p: Path = All) extends Spec(p) { ...

It is relatively ugly boilerplate. There's a way to get rid of it,
which would also allow me to use a trait not a class, but it would
mean that there would be an exception to the rule that "if you want it
to happen before, put it before; if you want it to happen after, put
it after". I'd have to tack on, "*unless* you put it at the top level,
in which case it will happen before the test, even if you place it
after." At least I couldn't figure out a way to do it without that
caveat. 

In Specsy it works without exceptions to that rule. The top level is the same as all nested levels, with the only exception that in the top level it's not a compile error to use a variable before it is initialized (because it's a field and the top level is a constructor), so it's good to be careful with helper methods which use the fields.

For example, this test passes:

import org.junit.runner.RunWith

import net.orfjackal.specsy._

import org.hamcrest.Matchers._

import org.hamcrest.MatcherAssert.assertThat

@RunWith(classOf[Specsy])

class Test extends Spec {

  var i = 0

  "this child spec is executed before whatever code comes after it" >> {

    i += 1

  }

  i += 2

  assertThat(i, is(3))

}

(Though I prefer to not write code like that, because the defer statement is much more useful and it keeps the setup and teardown next to each other. For example in [1] you will see that networkActor.start() opens a server socket, two clients connect to the socket when they are created with new ClientRunner(port), and when the test ends they all will be closed in the reverse order: client2.disconnect(), client1.disconnect(), networkActor.stop())

[1] https://github.com/orfjackal/dimdwarf/blob/master/dimdwarf-core/src/test/scala/net/orfjackal/dimdwarf/net/NetworkActorSpec.scala#L25-32

I think the most important idea in achieving that execution model is to stop thinking in terms of a test class which keeps track of its own state. Instead think of constructing the class to be a function call or a closure, which takes a context as parameter - the context is in control of what happens. If you look at the test cases for Specsy [2] , you will see that they don't need any test classes as guinea pigs [3], because the "test classes" are just closures and they are easily declared inside a test method.

[2] https://github.com/orfjackal/specsy/blob/master/src/test/scala/net/orfjackal/specsy/ExecutionModelTest.scala

[3] Guinea pig is a test double which is used as a specimen for testing reflection and other generic code. I wonder if the term is in common use?

Since I wrote GoSpec [4] first and there functions are the most natural way to write test suites, it was easy to apply the same in Scala after seeing how the Scala testing frameworks used the constructor.

[4] https://github.com/orfjackal/gospec/tree/gospec-1.3.8/examples/

 

[Richard Apodaca]

Esko,

> In the documentation you can see the aforementioned stack example: http://specsy.orgdocumentation.html#isolated_execution_model

Isolated execution mode is exactly the behavior I'm looking for. Thanks for the link. Bill, is there any chance of implementing this in ScalaTest?

Best,

Rich

--

You received this message because you are subscribed to the Google
Groups "scalatest-users" group.
To post to this group, send email to scalate...@googlegroups.com
To unsubscribe from this group, send email to
scalatest-use...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/scalatest-users?hl=en
ScalaTest itself, and documentation, is available here:
http://www.artima.com/scalatest

[Bill Venners]

Hi Richard and Esko,

Yes, Richard, I decided a while back to add it. The main thing I want to do is make the code as easy to reason about as possible. The more magic the framework does behind the scenes the more difficult users can have figuring out their own code.

I'm heading to the airport and will be flying most of the day. When I get back online I will follow Esko's links. My guess (as I'm unable to look at the code yet) Esko is that you are using a "global" thread local variable that your Spec traits look at to determine the path. Is that correct?

Thanks.

Bill

----

Bill Venners

Artima, Inc.

http://www.artima.com

[Esko Luontola]

On Saturday, December 3, 2011 7:38:26 PM UTC+2, Bill Venners wrote:

I'm heading to the airport and will be flying most of the day. When I get back online I will follow Esko's links. My guess (as I'm unable to look at the code yet) Esko is that you are using a "global" thread local variable that your Spec traits look at to determine the path. Is that correct?

The context [1] (which contains the target path) is passed to the base class of test using a thread local, to avoid the need for the user to declare a constructor parameter and pass it to the base class.

The context is not global in the sense that unrelated tests would share some data structures. Each isolated execution path has its own context and its own instance of the test class. The context records all newly discovered paths which are not yet executed, and the test runner will create and execute a new context for each of them until all paths have been covered.

If a shared data structure would be used, then it would prevent parallel execution, but the implementation would be much simpler. In NanoSpec.go [2] I used a shared mutable tree for the paths and execution state of all nested specs [3]. The tree keeps track of whether all nodes in a subtree have been executed, and the logic of which path to execute can be implemented in just a few lines of code (see the usages of ShouldExecute() in [3]).

In Specsy and GoSpec I use a more complex method of giving as parameter a target path which has not yet been executed (the initial value is the root spec's path), and then some complex conditionals [4] to check whether a path should be executed now, should be postponed, or has already been executed. The algorithm is that the first child spec is executed when it is first seen, and all other children are postponed for later execution. (Originally I arrived to those complex conditionals in GoSpec using TDD, so that at first I made it working with a cryptic method of 10-15 lines and multiple conditionals, after which I refactored it until it made sense - the result was some 50 lines spread over many small methods. When writing Specsy I just translated the conditionals from Go into Scala.)

[1] https://github.com/orfjackal/specsy/blob/master/src/main/scala/net/orfjackal/specsy/core/Context.scala

[2] https://github.com/orfjackal/nanospec.go

[3] https://github.com/orfjackal/nanospec.go/blob/release/src/context.go#L77-123

[4] https://github.com/orfjackal/specsy/blob/master/src/main/scala/net/orfjackal/specsy/core/SpecDeclaration.scala#L42-50

[Bill Venners]

Hi Esko,

Thanks for the clarification. Looks like I'll be offline possibly until tomorrow night, so I can't post code until later. But I just coded up a proof of concept of using a thread local to "pass data from one instance of a trait to another," and it works as advertised. So I'm pretty sure I could use that approach to avoid the constructor boilerplate. Thanks very much for the suggestion!

The downside is that the execution model is more magical compared to the constructor approach, but I think I may be able to explain it in the Scaladoc. I will give that a try, and unless I hit problems once I go to implement, will plan to include the three path *traits* (not classes with constructors) in ScalaTest 2.0, the next release.

Bill

----

Bill Venners

Artima, Inc.

http://www.artima.com

--

[akbac...@gmail.com]

Did anything ever come of this? I really love scalatest, but not having nested before/after has made porting rspecs from a ruby framework pretty painful.

[Bill Venners]

Hi,

Yes, look at org.scalatest.path.FunSpec. That's the best way to port rspec tests:

http://doc.scalatest.org/3.0.0/index.html#org.scalatest.path.FunSpec

Bill

To post to this group, send email to scalatest-users@googlegroups.com

To unsubscribe from this group, send email to

scalatest-users+unsubscribe@googlegroups.com

For more options, visit this group at
http://groups.google.com/group/scalatest-users?hl=en
ScalaTest itself, and documentation, is available here:
http://www.artima.com/scalatest

---

You received this message because you are subscribed to the Google Groups "scalatest-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to scalatest-users+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

[akbac...@gmail.com]

Based on that approach it would appear I cannot perform before/after each/all. In the absence of such things I would be unable to keep code dry. Am I missing something?

[Bill Venners]

One of us is missing something. If you want something to happen before, put it before, if after, put it after. You don't need "before" or "after" to appear in the code. Just put it before or after and it will work essentially like RSpec, but without the explicit "before" and "after" clauses.

Bill

On Mon, Oct 31, 2016 at 10:30 PM, <akbac...@gmail.com> wrote:

Based on that approach it would appear I cannot perform before/after each/all. In the absence of such things I would be unable to keep code dry. Am I missing something?

--

You received this message because you are subscribed to the Google
Groups "scalatest-users" group.
To post to this group, send email to scalatest-users@googlegroups.com
To unsubscribe from this group, send email to
scalatest-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/scalatest-users?hl=en
ScalaTest itself, and documentation, is available here:
http://www.artima.com/scalatest
---
You received this message because you are subscribed to the Google Groups "scalatest-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scalatest-users+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Patrick Boatner]

Thank you so much for this Bill! I'm a new developer using Scala test. I've used RSpec a lot, and I've come to tremendously appreciate being able to make a describe/context block, and then immediately override certain values to fulfill what I just described. Being able to nest these and add setup code inside each layer is huge, IMO. I've joined a codebase that was using org.scalatest.FunSpec as the base class, and this was just totally broken (perhaps by design). Now that I switched over to org.scalatest.path.FunSpec, this all works as expected:

var x = 3

describe("some other things that just need an appropriate value for x") {
  ...
}

describe("when x is huge") {
  x = 1000

  it("does asdf") { .... }

  it("does qwer") { .... }

  describe("when y is small") {
    y = 1
   
    it("...") { ...}
  }

  describe("when y is huge") {
    y = 1000
    ...
  }
}

I think this would be really helpful to add to the FunSpec documentation. The current behavior with org.scalatest.FunSpec was very jarring. There was no compilation/runtime error when I made arbitrary assignments inside describe blocks, but they just didn't take effect (the value was wrong, like the value was set only at the top level, or maybe by another statement, not sure which). I'm biased by my RSpec experience, but I would almost expect path.FunSpec to be the default behavior.

Many thanks for the link again! This will make ScalaTest way more pleasant to work with.

scalate...@googlegroups.com

For more options, visit this group at
http://groups.google.com/group/scalatest-users?hl=en
ScalaTest itself, and documentation, is available here:
http://www.artima.com/scalatest

---
You received this message because you are subscribed to the Google Groups "scalatest-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to scalate...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.