Side-effect free specs?

[Esko Luontola]

I'm considering writing the my tests in the future using Scala even in
my Java projects, because of the more readable syntax (now I'm using
Java and JDave). I'm now to Scala and right now I'm trying to
understand how Specs exactly works. I would need to know exactly that
when and in which order the test objects are created and the test
methods are run, or else I will probably run into hard-to-find bugs,
because already the production code that I write is complex and uses
much concurrency.

It appears that Specs carries state from one test to another, because
the following spec fails and prints "a 0, b 1, c 2". Is there some way
to execute the following spec so that it will pass and print "a 0, b
1, a 0, c 1"?

class ExampleSpec extends Specification {
"a" should {
var x = 0
println("a " + x)

"b" in {
x = x + 1
println("b " + x)
x must_== 1
}

"c" in {
x = x + 1
println("c " + x)
x must_== 1
}
}
}

Also, http://code.google.com/p/specs/wiki/DeclareSpecifications says
under "Sequential execution" something about executing the examples
concurrently. How are they exactly executed concurrently? Where is the
code that runs the specs, so that I could have a look at it more
closely?

[etorreborre]

Hi Esko,

A Specification is composed of System(s) under specification and
Example(s).

When you write something like:

"my system" should {
"do this" in {...}
}

there is an implicit definition creating a "System" object with 'my
system' as a description.
Then, everything inside the "should" brackets is evaluated to create
examples. When you have

"do this" in {...}

another implicit definition creates an Example object with "do this"
as the example description and what's inside the brackets as the
Example body. However nothing is executed until you ask for the report
of a Specification to get the number of failures and errors.

Now, the side-effects you see in your specification are because of the
way Scala's closure "capture" their surrounding state. When you write:

var x = 0

"b" in {
x = x + 1

x must_== 1
}

the "x" variable inside the bracket get a value at the time it is
evaluated and not at the time it is declared. Not sure this is
clear,... This means that when you evaluate the block in brackets
you'll get the current value of "x" which may not be 0 if it has been
modified by some other code.

So I would say that you're currently seeing side-effect because you're
indeed sharing state by sharing a variable.

There are 2 work-arounds for this.

First of all, you could use before/after methods (http://
code.google.com/p/specs/wiki/
DeclareSpecifications#Call_a_function_before_or_after_each_example) to
reset the "x" variable before each example. But this will not work if
the examples are executed concurrently. This is not the case anyway by
default in specs, the examples are executed one by one when requesting
for their status. However, a specs Specification can be also executed
as a ScalaTest TestSuite and ScalaTest provides a concurrent executor.

The other workaround for shared state is to use "System contexts":
http://code.google.com/p/specs/wiki/DeclareSpecifications#System_contexts.
This will require a bit more work on your side to define the context
you want to act on and modify but this will guarantee a clean state
before each example is executed, even in the case of concurrent
execution of the examples.

>Also, http://code.google.com/p/specs/wiki/DeclareSpecifications says
>under "Sequential execution" something about executing the examples
>concurrently. How are they exactly executed concurrently?

"Sequential execution" is probably not correctly named. This actually
means that examples are evaluated as soon as they are declared. So
this is more like "eager evaluation".

> Where is the code that runs the specs, so that I could have a look at it more
closely?

You can get the code here: http://code.google.com/p/specs/source/checkout

Don't hesitate to ask questions as the code may not be so easy to
follow for example building and execution.

Eric.

[Esko Luontola]

In the message http://groups.google.com/group/specs-users/msg/02f64e47cc4483fd?hl=en&
you said "I don't like very much the idea of sharing variables anyway
between examples. With JUnit, many of us made the mistake to use
static variables to initialize some persistent context between tests,
as a workaround for the 'fresh instance for each test'. And the result
was usually painful,... I don't think that any framework can prevent
you from doing this. Yet, there is no current solution in specs to
avoid variables sharing."

If I understand you there correctly, you would like to have examples
without side-effects, if there just would be a good solution for it.
In that case we are like-minded.


I would like to propose a way for executing the examples so, that they
would have no side-effects and could be executed in parallel quite
easily. Having no side-effects is important for writing isolated and
reliable tests/code. Being able to execute the tests in parallel is
important, because it is the only way to take advantage of newer CPUs,
and especially with TDD the speed of running the tests is closely
linked with programmer productivity (I even overclocked my CPU just to
be able to run my tests in 1.6 seconds instead of 2.0 seconds).

Also, the framework should encourage writing tests without side-
effects so, that the easiest way to write the tests is to avoid side-
effects. Now it seems that Specs requires the use of system contexts
to avoid side-effects. But because using system contexts requires
typing more test code, the framework punishes those who want to write
well isolated tests, which in turn leads to the programmers being more
inclined to write dirty non-isolated tests. That's why the recommended
way to write the code should also be the simplest way to write the
code. And defaulting to immutability and no side-effects is
recommendable.


I will explain my ideas in Java, because I don't yet know Scala well
enough. Given the previous spec:

class ExampleSpec extends Specification {
"a" should {
var x = 0

"b" in {
x = x + 1
x must_== 1
}

"c" in {
x = x + 1

x must_== 1
}
}
}

The genererated bytecode will be similar to the Java code below (I
decompiled the .class files and read the bytecode to understand Specs
and Scala). I have renamed the variables to be more readable and
removed the unrelated generated code.

public class ExampleSpec extends Specification
implements ScalaObject
{
public ExampleSpec() {
specify("a").should(new FuncA(this));
}
}

public final class FuncA
implements Function0, ScalaObject
{
ExampleSpec outer;

public FuncA(ExampleSpec outer) {
this.outer = outer;
}

public Example apply() {
IntRef x = new IntRef(0);
outer.forExample("b").in(new FuncB(this, x));
return outer.forExample("c").in(new FuncC(this, x));
}
}

public final class FuncB
implements Function0, ScalaObject
{
FuncA outer;
IntRef x;

public FuncB(FuncA outer, IntRef x) {
this.outer = outer;
this.x = x;
}

public Result<Integer> apply() {
x.elem += 1;
return /* "x must_== 1" */;
}
}

And FuncC will be otherwise the same as FuncB, but with a different
class name.


We see that the examples form a tree structure. The root node is
ExampleSpec and it has one child node, FuncA. The FuncA node has two
child nodes, FuncB and FuncC. The FuncB and FuncC nodes do not have
child nodes, so we say that FuncB and FuncC are leaf nodes.

As all the nodes are Java objects generated at compile-time, it is
possible to call the Object.getClass() method on any of them and use
the Class object to identify each node. Any node in the tree can be
identified by the path that leads to it in the tree. The path of the
root node is a single-element list [ExampleSpec.class]. The paths of
the leaf nodes are lists [ExampleSpec.class, FuncA.class, FuncB.class]
and [ExampleSpec.class, FuncA.class, FuncC.class].

To be able to isolate unwanted side-effects, we want to execute the
examples so that each path to a leaf node is executed once, and none
of the objects created during one such execution is shared with any
other execution. So each example node will see only the side-effects
from its parents, but not from its siblings. The parent is always
executed fully before its children are executed (it should also be
possible to execute the children inside its parents, inside the
Example.in() method, but then an exception in the first child might
prevent the latter children from being found by the framework - what
do you think makes most sense?).

I will now describe the proposed algorithm for executing the examples.


We need a queue containing the paths of examples which have not yet
been executed. Let's call it 'examplesToBeExecuted'.

1. The test runner adds [ExampleSpec.class] to 'examplesToBeExecuted'.


2.1. A worker thread gets [ExampleSpec.class] from
'examplesToBeExecuted' and saves it as 'currentPath'. It constructs a
new instance of ExampleSpec, which executes its constructor.
ExampleSpec's constructor registers an instance of FuncA.

2.2. Since FuncA.class is a node deeper than 'currentPath', it is a
new node that has not been executed. FuncA.apply() is called on the
instance, which registers instances of FuncB and FuncC.

2.3. Since FuncB.class is deeper than 'currentPath', it is a new node
that has not been executed. FuncB.apply() is called on the instance,
but it does not register any nodes. So we know that FuncB.class is a
leaf node. We save the information that the current execution has
completed until a leaf node, which we mark by setting
'leafExampleExecuted = true'.

2.4. Because FuncB has no more children, we step up in the stack.

2.5. Since FuncC.class is deeper than 'currentPath', it is a new node
that has not been executed. But because 'leafExampleExecuted == true',
we can not execute any more examples during this execution. So we
schedule FuncC to be executed later, i.e. we add the path
[ExampleSpec.class, FuncA.class, FuncC.class] to
'examplesToBeExecuted'.

2.6. Because FuncA has no more children, we step up in the stack.

2.7. Because ExampleSpec has no more children, we step up in the
stack.


3.1. A worker thread gets [ExampleSpec.class, FuncA.class,
FuncC.class] from 'examplesToBeExecuted' and saves it as
'currentPath'. It constructs a new instance of ExampleSpec, which
executes its constructor. ExampleSpec's constructor registers an
instance of FuncA.

3.2. Since FuncA.class is part of 'currentPath', we need to execute it
to reach our target subtree (FuncC). FuncA.apply() is called on the
instance, which registers instances of FuncB and FuncC.

3.3. Since FuncB.class is not deeper than 'currentPath' and it is not
part of 'currentPath', it must be a node that has already been found
before, and thus it has either been executed or been scheduled for
execution. So we ignore the FuncB instance and do not execute it.

3.4. Since FuncC.class is part of 'currentPath', we need to execute it
to reach our target subtree (FuncC). FuncC.apply() is called on the
instance, but it does not register any nodes. So we know that
FuncC.class is a leaf node. We save the information that the current
execution has completed until a leaf node, which we mark by setting
'leafExampleExecuted = true'.

3.5. Because FuncC has no more children, we step up in the stack.

3.6. Because FuncA has no more children, we step up in the stack.

3.7. Because ExampleSpec has no more children, we step up in the
stack.


4. The test runner recieves the test results from the executions of
[ExampleSpec.class, FuncA.class, FuncB.class] and [ExampleSpec.class,
FuncA.class, FuncC.class]. Since all worker threads have finished
execution and 'examplesToBeExecuted' is empty, we know that all
examples have been executed and the test run has ended.


To my knowledge, this way of executing the tests would avoid unwanted
side-effects, would not require the programmer to write any additional
code to manage side-effects, and would allow the execution of the
tests in parallel with relative ease.

If for some reason side-effects from sibling examples are needed, such
behaviour can be enabled with the setSequential() method, in which
case the node containing the call to setSequential() and all its child
nodes (but not its siblings) would be executed sequentially and they
would see each other's side-effects. (I'm not sure that should also
the children's children be executed sequentially.)

What do you think, would it make sense to change Specs to execute its
tests this way? It seems to me that the necessary changes would need
to be made in Example.scala and the code that originally starts the
tests (BTW, which method starts the tests of a Specification class?).

[Bill Venners]

Hi All,

Eric mentioned yesterday that you can use ScalaTest to run
Specifications, and ScalaTest has a concurrent mode, but he implied
that ScalaTest might execute the examples of one Specification
concurrently. I wanted to clarify that the model in ScalaTest would
not require that Specifications be thread-safe. I can't remember for
sure how Eric has mapped Specs Specifications and Examples to
ScalaTests's notion of "suite" and "test." But in ScalaTest a "test"
is something that has a name, can be started and either succeeds or
fails. A "suite" is a collection of tests. So the mapping should be
that a Specification appears to ScalaTest as a Suite, and each example
inside a Specification appears as a "test." ScalaTest can execute
suites concurrently, but it will always execute the tests inside one
suite sequentially. Mapped to Specs, this means that ScalaTest can
execute Specs Specifications concurrently, but the examples in one
Specification will always be executed sequentially.

The reason I chose this model for concurrency is so that people would
never have to worry about making their suites (Specifications in the
Specs case), thread safe. Because that's error prone and hard. What
Esko described here for running examples in parallel sounds a bit like
what ScalaTest already does to run suites in parallel. The mechanism
used is called a Distributor, and it is documented here:

http://www.artima.com/scalatest/doc-0.9.5/org/scalatest/Distributor.html

There's one more aspect to this problem, though, which I don't think
has been mentioned yet in this discussion. Executing Specifications
concurrently means that the output being generated for the run will be
arrive in a mixed up order. Since one of the goals of BDD is to
generate these nice, readable outputs, it would be nice if these could
be reordered into an expected order. ScalaTest 0.9.5 does not do this
yet, but 0.9.6 will contain something called an Ordinal that will be
used to do this. I try to deprecate for two releases before I remove
stuff, and getting this aspect of concurrent execution fully working
will require that I remove some things. So it won't fully work until
ScalaTest 0.9.8, which I expect will be out sometime this summer.

Eric and I will eventually complete our integration. We've just been
busy. But the next release of ScalaTest (0.9.6) will also include
something called IndentedText, which I put in there in part so that
Specs output can appear nested when you run it through ScalaTest.

So anyway, you can run Specifications concurrently today using the
ScalaTest Runner's -c option, but the output will be out of order. So
for the time being, when you're just looking for failures, you can run
concurrently, but when you want to generate a pretty output to show
others, then you must execute sequentially. But after 0.9.8 you'll be
able to have your cake and eat it too.

Bill

[Esko Luontola]

On May 18, 2:29 am, Bill Venners <b...@artima.com> wrote:
> The reason I chose this model for concurrency is so that people would
> never have to worry about making their suites (Specifications in the
> Specs case), thread safe. Because that's error prone and hard.

Makes sense.

The algorithm that I described can also be executed single-threadedly.
My main point is avoiding side-effects between examples, even when
executing them with a single thread. Added concurrency would be a
secondary benefit.

> Executing Specifications
> concurrently means that the output being generated for the run will be
> arrive in a mixed up order. Since one of the goals of BDD is to
> generate these nice, readable outputs, it would be nice if these could
> be reordered into an expected order. ScalaTest 0.9.5 does not do this
> yet, but 0.9.6 will contain something called an Ordinal that will be
> used to do this.

Will you order also the stdout and stderr? For example, by setting
java.lang.System.setOut/setErr to a custom PrintStream which keeps
track of what was printed in each thread (maybe it could be backed by
a ThreadLocal that holds one PrintStream buffer for each thread, and
the buffer is flushed synchronously to the actual stdout/stderr when
the test ends).

[Esko Luontola]

On May 18, 2:29 am, Bill Venners <b...@artima.com> wrote:

> What Esko described here for running examples in parallel sounds a bit like
> what ScalaTest already does to run suites in parallel. The mechanism
> used is called a Distributor, and it is documented here:
>
> http://www.artima.com/scalatest/doc-0.9.5/org/scalatest/Distributor.html

I had a look at the Distributor and it appears to fit very nicely into
the algorithm that I described. Just replace "add the path [X, Y, Z]
to 'examplesToBeExecuted'" with "add a Suite for executing the path
[X, Y, Z] to Distributor" in my description. So using Distributor it
would be possible to parallelize the execution of every leaf example
in Specs, possibly improving the CPU efficiency (when the tasks are
short, they can be divided more evenly between all CPU cores, and the
total time for running all tests will be shorter).

The key to being able to divide the execution of a Specification is
being able to identify each example in some other way than comparing
the identity of Example instances. The Object.getClass() approach that
I described earlier should do this at least on JVM and the current
Scala implementation. Another way would be to give ordinal numbers for
each child example (then you could specify the path to FuncB as
[ExampleSpec.class, 1st child, 1st child] and FuncC as
[ExampleSpec.class, 1st child, 2nd child]).

[etorreborre]

Hi Esko and Bill,

As Bill said, we have in mind to provide more elaborate executions of
Specifications through ScalaTest.

I think that this would make sense for specs to allow concurrency to
happen at different levels: Specification, System under specification,
Examples.

Still I find it a bit strange that you would write a Specification
with "shared" variables, and have them being actually shared or not
between examples depending on the way the Specification is executed. I
think that in that case, it may be better to actively specify Contexts
to make it clear on what you're working on. Given that Scala allow you
to nest classes and that the class declaration syntax is minimal, this
reduces the burden.

Anyway, we'll go on working with Bill on that and make some
announcement when ready.

Eric.

[Esko Luontola]

On May 19, 4:10 am, etorreborre <etorrebo...@gmail.com> wrote:
> Still I find it a bit strange that you would write a Specification
> with "shared" variables, and have them being actually shared or not
> between examples depending on the way the Specification is executed.

I'm not advocating that the Specification should have many ways to
execute them. I'm saying that they should always be executed so that
there are no side-effects. And if for some rare reason side-effects
between examples are needed, that should require the programmer to
write extra code, so that the framework would discourage writing tests
that rely on the state left by previous tests.

The parent examples function as common setup code for their nested
child examples. Nothing is shared between sibling examples. This is
the safe way and should be the default. It won't be possible to write
isolated and repeatable tests (http://agileinaflash.blogspot.com/
2009/02/first.html) if there are some side-effects leaking.

For example have a look at the specification at http://www.jdave.org/examples.html
- In it EmptyStack, FullStack and StackWhichIsNeitherEmptyNorFull all
set up a different context for the individual tests to work on (in the
create() method). Out of those individual tests,
EmptyStack.isNoLongerEmptyAfterPush, FullStack.complainsOnPush,
FullStack.doesNotContainRemovedItem,
FullStack.containsAllButRemovedItem and
StackWhichIsNeitherEmptyNorFull.addsToTheTopWhenSentPush modify the
state of the context. But since none of the objects is shared between
tests - a new StackSpec instance is created for each test - there are
no side-effects between tests.

I use a similar approach in many of my tests. Often times there are
two levels of setup code - one that is common code for all examples in
the specification, and another that is common for one subtree of the
examples. Here is one example of such a class with complex setup code
at two levels:
http://github.com/orfjackal/dimdwarf/blob/67e661c1d8e90c1359c857c584bccd1ca6a06832/dimdwarf-core/src/test/java/net/orfjackal/dimdwarf/scheduler/TaskThreadPoolSpec.java

> it may be better to actively specify Contexts
> to make it clear on what you're working on. Given that Scala allow you
> to nest classes and that the class declaration syntax is minimal, this
> reduces the burden.

Anything that requires you to type more to write safer code, is
inherently bad. As I said earlier, because using system contexts
require you to type more than when not using system context, the
programmers will avoid using system context whenever possible - the
framework does not encourage writing good tests, but instead punishes
those who want to write isolated and repeatable tests.

[Esko Luontola]

Here is my proposal that how the Specifications should be run. The
following spec should pass all its tests.


class ProposalForIsolatedTestsSpec extends Specification {
var x = 0

"A" should {
x += 1
x must_== 1
var y = 0

// "AA" does see the side-effects of "A", because "A" is its
parent
"AA" in {
x += 1
x must_== 2
y += 1
y must_== 1
}

// "AB" does _not_ see the side-effects of "AA", because "AA" is
its sibling
"AB" in {
x += 1
x must_== 2
y += 1
y must_== 1
}
}

// "B" does _not_ see the side-effects of "A", because "A" is its
sibling
"B" should {
enableSideEffectsBetweenChildExamples()
x += 1
x must_== 1
var z = 0

"BA" in {
x += 1
x must_== 2
z += 1
z must_== 1

// 'enableSideEffectsBetweenChildExamples' does not affect child-
child examples
"BAA" in {
z += 1
z must_== 2
}
"BAB" in {
z += 1
z must_== 2
}
}

// "BB" _sees_ the side-effects of "BA", even though "BA" is its
sibling,
// because 'enableSideEffectsBetweenChildExamples' was used in
their common parent "B".
// But "BB" does _not_ see the side-effects of "BAA" and "BAB",
because
// 'enableSideEffectsBetweenChildExamples' is not recursive.
"BB" in {
x += 1
x must_== 3
z += 1
z must_== 2
}
}
}

[etorreborre]

Hi Esko,

Sorry for the latency in my answers, but I've just relocated to
another country and don't have a lot of "open-source time" on my hands
at the moment.

Thanks for the specification, I'll certainly give it some thoughts and
see if I can prototype something just to get the feel of it (maybe
something a bit simpler to start with, like complete isolation of all
examples).

I agree with you on the objective of isolating examples but I think I
have an afterthought that this is at the same time working against the
language semantic where variables are shared when in the same scope.
So it has to be very clear from the beginning what it means to write
"var x= ..." in a specification.

Writing this, I'm thinking that one possibility of achieving the
objective, not with zero code, but with a small amount of it, would be
to use enhanced properties (http://code.google.com/p/specs/source/
browse/trunk/src/main/scala/org/specs/util/Property.scala):

object spec extends Specification {

val x: Int = 0
val y: Property[Int] = isolated(0) // creates a Property[Int]
object, tied to the specification

"do this with x and y" in {
// update y
y(3)
y() must_== 3
}
"do that with x and y" in {
// y is resetted to its initial value before each example by the
specification
y() must_== 0
}
}

That could be a compromise between the way Scala works (sharing
variables) and the isolation we want to achieve when setting up
contexts.

I'll try to see if I can make some progress on both ideas.

Eric.

[Esko Luontola]

On May 21, 3:15 am, etorreborre <etorrebo...@gmail.com> wrote:
> Thanks for the specification, I'll certainly give it some thoughts and
> see if I can prototype something just to get the feel of it (maybe
> something a bit simpler to start with, like complete isolation of all
> examples).

Thanks. :)

> I agree with you on the objective of isolating examples but I think I
> have an afterthought that this is at the same time working against the
> language semantic where variables are shared when in the same scope.

The "something" in { ... } etc. structures are not part of the Scala
language - they are part of the DSL of Specs for writing tests. As
such, Specs is in complete control of their semantics.

> So it has to be very clear from the beginning what it means to write
> "var x= ..." in a specification.

"val x=..." is equally affected. If it points to a mutable object,
then it opens up the possibility of leaking side-effects. That's why
many testing frameworks, for example JUnit, create a new instance of
the surrounding class for each test.

>   val y: Property[Int] = isolated(0)

Too much typing. Also its mental model is too complicated.

[etorreborre]

> Too much typing. Also its mental model is too complicated.

You can forget about it anyway. I started prototyping it and I
realized that even if the properties were reinitialized between
examples they would still be shared if the examples were executed
concurrently, so we could have concurrency issues.

I'll go on with your idea of instantiating a new Specification object
for each example execution then regrouping the results for reporting.

Eric.

[Bill Venners]

Hi Eric, Esko,

You know JUnit 3 created a new TestCase for each test it executed, and
I always thought it was non-intuitive. I'm pretty sure they stuck with
that approach in JUnit 4 too. The reason they did that was to allow
tests to have side effects on shared state referenced from instance
variables (which they call fixtures). This is done despite offering
setup and teardown methods that would reset the state before, and
clean it up afterwords.

I thought it was not necessary to create a separate instance per test,
because I felt it was reasonable to say that if you don't use setup
and teardown correctly such that tests can see side effects on
instance variables from other tests, then *that's a bug* in your
tests. I figure it would be rare, and when it happened, you'd get a
failed test, which could be easily fixed, after which, it wouldn't
happen anymore. So in SuiteRunner and then ScalaTest, that's the
approach I took. All tests in Suite, FunSuite, or Specs (the
org.scalatest.Spec kind) in ScalaTest are executed in the context of
one suite class instance.

Concurrently executing tests/examples does make the problem harder,
because shared fixtures could be updated concurrently. Creating a new
suite (or whatever you call a collection of tests -- in the Specs case
it is a Specification) for each individual test would isolate the
tests from each other as far as instance variables are concerned, but
not for external resources.

But I would ask: why do you need to execute examples concurrently? The
reason I wanted to enable concurrent execution of tests in ScalaTest
was to speed up the test runs, which can get long. I thought
concurrency made sense especially since the number cores is already
multiple and will keep growing. But to get tests to go faster, do you
need to run examples concurrently? I don't think so. I'd guess you'd
almost always get a satisfactory speed up by just running
Specifications concurrently, and if you take that that solves the
whole problem of worrying about making Specifications safe for
multiple threads. That's the way I went with ScalaTest, which in turn
means you can run Specs Specifications concurrently in this was
already through ScalaTest.

So I'm curious, is there some benefit you'd be able to get by by
running examples concurrently that you can't get running
Specifications concurrently?

Bill

--
Bill Venners
Artima, Inc.
http://www.artima.com

[Esko Luontola]

On May 22, 2:07 am, Bill Venners <b...@artima.com> wrote:
> The reason they did that was to allow
> tests to have side effects on shared state referenced from instance
> variables (which they call fixtures). This is done despite offering
> setup and teardown methods that would reset the state before, and
> clean it up afterwords.

No. JUnit creates a new fixture instance for each test, so that the
tests would be isolated. Fixtures exists for tests to share common
setup (and cleanup) code, so as to avoid duplication. But tests don't
share the state from other tests, because that would be dangerous.
http://junit.sourceforge.net/doc/faq/faq.htm#tests_2

JUnit has a way for running one setup code before all tests in the
class, but using that is discouraged, because "if two or more tests
must share the same test fixture state, then the tests may be trying
to tell you that the classes under test have some undesirable
dependencies."
http://junit.sourceforge.net/doc/faq/faq.htm#organize_3

> I figure it would be rare, and when it happened, you'd get a
> failed test, which could be easily fixed, after which, it wouldn't
> happen anymore.

If tests are not fully isolated, you can never be sure whether a test
failed because that test had a problem, or whether another test had a
problem. It might cause tests to fail when they should pass, or even
worse, tests to pass when they should fail. And then you will need to
waste time in debugging and can not be sure whether your tests are
actually testing what they should.

Isolation and repeatability are two of main properties of good unit
tests (http://agileinaflash.blogspot.com/2009/02/first.html).

When test the framework creates a new instance of the fixture for
every test, writing isolated and repeatable tests is easy. You just
need to avoid using static mutable state and accessing the file system
and other external resources. The garbage collector will do the
cleanup of instance variables automatically, so there won't be any
possibility of bugs in the cleanup.

> Creating a new
> suite (or whatever you call a collection of tests -- in the Specs case
> it is a Specification) for each individual test would isolate the
> tests from each other as far as instance variables are concerned, but
> not for external resources.

For tests using external resources, it would be necessary to be able
to tag a fixture as not being able to be executed concurrently.

> So I'm curious, is there some benefit you'd be able to get by by
> running examples concurrently that you can't get running
> Specifications concurrently?

If there is a project with one test fixture that takes a long time to
execute, then the total time of executing will be restricted by that
fixture.

Let's say that a project has a fixture BigTest which has a couple of
dozen tests, which in total take 10 seconds to execute, because the
tests need to set up a heavy library (a servlet container, ORM
framework or similar). In addition to that, the project has tens of
other fixtures, which in total take 5 seconds to execute. This should
be quite typical of a project - most of the application code can be
tested quickly, but there are a small number of tests which can not be
decoupled from other frameworks, and will then execute slowly.

With per-fixture concurrency, they can be executed on a 4-core machine
in a bit over 11 seconds. But with per-test concurrency, they can be
executed in 4 seconds - a huge improvement in a TDD'ing programmer's
productivity.

[Bill Venners]

Hi Esko,

On Fri, May 22, 2009 at 3:14 AM, Esko Luontola <esko.l...@gmail.com> wrote:
>
> On May 22, 2:07 am, Bill Venners <b...@artima.com> wrote:
>> The reason they did that was to allow
>> tests to have side effects on shared state referenced from instance
>> variables (which they call fixtures). This is done despite offering
>> setup and teardown methods that would reset the state before, and
>> clean it up afterwords.
>
> No. JUnit creates a new fixture instance for each test, so that the
> tests would be isolated. Fixtures exists for tests to share common
> setup (and cleanup) code, so as to avoid duplication. But tests don't
> share the state from other tests, because that would be dangerous.
> http://junit.sourceforge.net/doc/faq/faq.htm#tests_2
>

I think you misunderstood me. This creating "a new fixture instance
for each test" is what I was talking about. In JUnit 3, you had to
extend class TestCase, and this class could house test methods and a
"fixture" in the form of instance variables shared by those test
methods. If you created a MyTestCase with ten test methods, then when
JUnit ran that class it would create ten instances of MyTestCase, and
run each test method in its own instance.

> JUnit has a way for running one setup code before all tests in the
> class, but using that is discouraged, because "if two or more tests
> must share the same test fixture state, then the tests may be trying
> to tell you that the classes under test have some undesirable
> dependencies."
> http://junit.sourceforge.net/doc/faq/faq.htm#organize_3
>

I wasn't talking about running setup code before all tests. The setup
and teardown method of JUnit 3 correspond to @Before and @After of
JUnit 4. JUnit 3 didn't have anything corresponding to JUnit 4's
@BeforeClass and @AfterClass as far as I remember. My point was that
if you use setup and teardown so that before each test runs to
reinitialize fixture variables, then you don't need to run each test
method in its own instance. Or, conversely, if you're going to run
each test in its own instance, you don't need a setup method, because
you can just do everything in the constructor.

>> I figure it would be rare, and when it happened, you'd get a
>> failed test, which could be easily fixed, after which, it wouldn't
>> happen anymore.
>
> If tests are not fully isolated, you can never be sure whether a test
> failed because that test had a problem, or whether another test had a
> problem. It might cause tests to fail when they should pass, or even
> worse, tests to pass when they should fail. And then you will need to
> waste time in debugging and can not be sure whether your tests are
> actually testing what they should.
>

Tests can be fully isolated by writing them fully isolated, and I
don't think it is that difficult to do. If a test fails, you probably
aren't sure why it failed until you go look. Then you can figure out
the problem and fix it.

> Isolation and repeatability are two of main properties of good unit
> tests (http://agileinaflash.blogspot.com/2009/02/first.html).
>

Agreed, but this doesn't require creating a new class instance for
each test method.

> When test the framework creates a new instance of the fixture for
> every test, writing isolated and repeatable tests is easy. You just
> need to avoid using static mutable state and accessing the file system
> and other external resources. The garbage collector will do the
> cleanup of instance variables automatically, so there won't be any
> possibility of bugs in the cleanup.
>

Well I agree here, but in Scala at least you can pass a function down
to a withFixture method, and that method can create a fixture, execute
the function, and clean up the fixture. That's one of the ways I
recommend folks do this in ScalaTest, and prior to 1.0 I'll be
providing a set of traits that make this style even easier. There's no
need for the framework to create a new instance of the class for each
test.

That's a good point. I was thinking that in such cases, which I expect
to be a small percentage of all tests written, people could just put
these in different suites. However, I can see that some people such as
yourself might prefer to do it this way. Also, people may want to keep
those together in the same Specification class because logically they
belong together in the textual specification, and they want to see
them that way in the output. I'll look into adding a trait that can be
mixed in to get this functionality. The trait can override execute and
create a new instance of itself for each test, and if there's a
Distributor, it could pass those instances to the Distributor. So if
you mix in this trait you'd also get examples run concurrently, and
because of ScalaTest's ordinal stuff the output will still be in the
right order.

Bill

[Joakim Ohlrogge]

Just my two cents: for what they're worth.

On the whole isolation thingy I would have to agree with Esko here. On
the running tests in parallell I couldn't care less but it's cool ;)

Though I agree with you Bill that this new instance per test is a bit
surprising, even confusing (perhaps) but it only really bites you if
you expect to be able to share state among tests which is a pretty bad
idea (for the most part) and that's how you learn about the new
instance per test thing and perhaps even the reasons for that.
The upside though is that the JUnit framework really, really tries to
make it hard to mess up due to shared state. Sure, there is a setup
and teardown method that you can, well, setup and tear down your state
but I'd rather have that state disappear automatically without having
to explicitly do it myself. If I have to explicitly remove it myself
chances are pretty big that I won't until I realize potentially after
hours of bug-hunting that against my better knowledge clearing up that
state actually was important.
To be frank, when I write tests/specs I don't really want to worry
about clean up. I want to spend as much time as possible focusing on
the code and don't worry about side-effects. It's all to easy to
forget to clean up something and false positives are the worst
possible outcome from forgetting. Having a clean slate (new instance
for each test) really helps with preventing those sideeffects. And
what is really the practical downside of of a new instance for each
test? I've never saw it as a limitation in JUnit. In fact, I've come
to depend on it.

Granted, this whole state-thingy would not be much of a big deal if
all your objects in your fixture were immutable. Immutability seems to
resonate with the scala-tradition so I thought thats why scalatest and
specs did not bother with the new instance though I was a bit put off
by it when I realized this was the case.

<side-track>
I love the scala-test BDD-DSL (only weird thing was a false possitive
when I added an enter after a "be"

be
(

instead of
be (

The first case always passes :/ It would be better if it always failed.
</side-track>

[Bill Venners]

Hi Joakim,

That all makes sense. My goal with ScalaTest is to facilitate
different styles of testing, and you and Esko made me realize this is
an important style to support. The default will continue to be the
same, but getting the behavior you like will be easy: just mix in
trait OneInstancePerTest. That alone won't get you distributed tests
though. For that you'll need to also mix in trait TestDistribution,
which will cause tests to be executed concurrently when a Distributor
is passed to execute. This should work when running Specs
Specifications through ScalaTest too.

Also, I agree there's a gotcha with the be-return-paren form. I had
never thought to try that. I'll see if I can't get it to generate a
test failure with a helpful error message.

Thanks.

Bill
----

Bill Venners
Artima, Inc.
http://www.artima.com

On May 23, 2009, at 12:59 PM, Joakim Ohlrogge

[etorreborre]

Hi Esko, Bill and Joakim,

I've been thinking about this too over the week-end and I don't see
any reason why the example isolation shouldn't be the default in a
specification. I've started working on this and I am currently
checking some implementation details (like instantiating a
Specification which is declared a nested class).

I'll let you know when it's ready if you want to test it.

Eric.

On May 25, 1:08 am, Bill Venners <b...@artima.com> wrote:
> Hi Joakim,
>
> That all makes sense. My goal with ScalaTest is to facilitate  
> different styles of testing, and you and Esko made me realize this is  
> an important style to support. The default will continue to be the  
> same, but getting the behavior you like will be easy: just mix in  
> trait OneInstancePerTest. That alone won't get you distributed tests  
> though. For that you'll need to also mix in trait TestDistribution,  
> which will cause tests to be executed concurrently when a Distributor  
> is passed to execute. This should work when running Specs  
> Specifications through ScalaTest too.
>
> Also, I agree there's a gotcha with the be-return-paren form. I had  
> never thought to try that. I'll see if I can't get it to generate a  
> test failure with a helpful error message.
>
> Thanks.
>
> Bill
> ----
> Bill Venners

> Artima, Inc.http://www.artima.com

>
> On May 23, 2009, at 12:59 PM, Joakim Ohlrogge  
>

[etorreborre]

Hi,

I've just committed a new version of specs-1.5.1-SNAPSHOT (you should
be able to get it from the hudson site: http://hudson.scala-tools.org/job/specs).

Basically the default is that a new Specification object is created
for each example. If you don't want this behavior to happen, you can
switch it off by calling "shareVariables()".

Esko, this is not implementing yet your specification as what you
specified would also necessitate a brand new specification per system
under specification.

Please have a go at it and tell me what you think.

Eric.

[Esko Luontola]

Looks like a good start. When you still add isolation for "system
under specification", then I can begin using Specs in my projects (I
just need to wait for IntelliJ's Scala plugin to get better, as I'm
having unusable performance with it).

It appears that examples inside examples are not isolated. It would be
good to fix that. For example currently the "b2" example in the
following spec fails:

class Example1Spec extends org.specs.Specification {
var x = 0
"a" should {

"b" in {
x += 1
x must_== 1

"b1" in {

x += 1
x must_== 2
}

"b2" in {
x += 1
x must_== 2 // currently fails with
FailureExceptionWithResult: '3' is not equal to '2'
}
}

"c" in {
x += 1
x must_== 1
}
}
}

How deep is the effect of "shareVariables()"? Will it be like in the
specification that I gave, that it will affect only the direct
children, and will not be recursive? (Now I can't know, because even
without shareVariables() the examples inside examples are not
isolated, as I pointed out above.) It would be nice for all levels of
the specs (SUS, example, example in example etc.) to be isolated, and
that you could nest the examples unlimitedly (although usually two
levels is enough, but sometimes I have had the need for more nested
levels).

It would be good for shareVariables() to be non-recursive, because
then you can have controlled side-effects, for example when specifying
a series of actions such as in a UI:

class LoginUiSpec extends org.specs.Specification {
shareVariables()
val ui = new LoginUiModel

"when user enters the login screen" should {

"the username field is empty" in { ui.username must_== "" }
"the password field is empty" in { ui.password must_== "" }
"the password field is disabled" in { ui.passwordEnabled must_==
false }
"the login button is disabled" in { ui.loginEnabled must_==
false }
}

"and when user enters his username" should {
ui.username = "jdoe"

"the username field has the username" in { ui.username must_==
"jdoe" }
"the password field is empty" in { ui.password must_== "" }
"the password field is enabled" in { ui.passwordEnabled must_==
true }
"the login button is disabled" in { ui.loginEnabled must_==
false }
}

"and when user enters his password" should {
ui.password = "1234"

"the username field has the username" in { ui.username must_==
"jdoe" }
"the password field has the password" in { ui.password must_==
"1234" }
"the password field is enabled" in { ui.passwordEnabled must_==
true }
"the login button is enabled" in { ui.loginEnabled must_== true }
}

"and when user clicks the login button" should {
ui.clickLogin()

"the user is logged into the system" in { ... }
}
}

In the above pseudo spec, some code duplication can be avoided by
taking advantage of sharing variables at the top level. Then you don't
need to repeat 'ui.username = "jdoe"' etc. in the beginning of every
SUS. But for the deeper examples, it would be good to have them
isolated, to avoid any unwanted side-effects.

I would expect that to be executed like this, so that each line
starting with "-" has its own instance of the spec:

- "when user enters the login screen", "the username field is empty"
- "when user enters the login screen", "the password field is empty"
...
- "when user enters the login screen", "and when user enters his
username", "the username field has the username"
- "when user enters the login screen", "and when user enters his
username", "the password field is empty"
...
- "when user enters the login screen", "and when user enters his
username", "and when user enters his password", "the username field
has the username"
...
- "when user enters the login screen", "and when user enters his
username", "and when user enters his password", "the login button is
enabled"
- "when user enters the login screen", "and when user enters his
username", "and when user enters his password", "and when user clicks
the login button", "the user is logged into the system"

[Esko Luontola]

BTW, would it be possible to have a meaningless alias for the word
"should"? There are so many ways to name the tests, that having some
words always appended to the test description it not always wanted.
For example, the LoginUiSpec in my previous message is not compatible
with the word "should". It would be better to write it like this:

class LoginUiSpec extends org.specs.Specification {
...
"when user enters the login screen" >> {
"the username field is empty" >> { ... }

"the password field is empty" >> { ... }

"the password field is disabled" >> { ... }
"the login button is disabled" >> { ... }
}
...
}

[Bill Venners]

Hi Esko,

It sounds like you may want to phrase things more like you can do with
Ruby's RSpec or ScalaTest's Spec. In Spec it would look like this:

class LoginUiSpec extends org.scalatest.Spec {

describe("The login screen") {
describe("(when the user first enters it)") {
it("should have a empty username field") {}
it("should have an empty password field") {}
it("should have a disabled password field") {}
it("should have a disabled login button") {}
}
describe("(when the user enters his or her username)") {
it("should have a username field containing the user's username") {}
it("should have an empty password field") {}
it("should have an enabled password field") {}
it("should have a disabled login button") {}
}
describe("(when the user enters his or her password)") {
it("should have a username field containing the user's entered
username") {}
it("should have a password field containing the user's entered
password") {}
it("should have an enabled password field") {}
it("should have a enabled login button") {}
}
describe("(when the user clicks the login button)") {
it("should log the user into the system") {}
}
}
}

The output looks like:

scala> :load esko.scala
Loading esko.scala...
defined class LoginUiSpec

scala> (new LoginUiSpec).execute()
The login screen (when the user first enters it)
- should have a empty username field
- should have an empty password field
- should have a disabled password field
- should have a disabled login button
The login screen (when the user enters his or her username)
- should have a username field containing the user's username
- should have an empty password field
- should have an enabled password field
- should have a disabled login button
The login screen (when the user enters his or her password)
- should have a username field containing the user's entered username
- should have a password field containing the user's entered password
- should have an enabled password field
- should have a enabled login button
The login screen (when the user clicks the login button)
- should log the user into the system

If you don't like describe or it, you could mix in some implicit
conversions on strings that drops it. I actually released a trait that
does this in 0.9.3, but deprecated and dropped it later because I
didn't want to recommend it. However it still works. It would make
your code look like this:

class LoginUiSpec extends org.scalatest.Spec with SpecDasher {

"The login screen" -- {
"(when the user first enters it)"-- {
"should have a empty username field" - {}
"should have an empty password field" - {}
"should have a disabled password field" - {}
"should have a disabled login button" - {}
}
"(when the user enters his or her username)" -- {
"should have a username field containing the user's username" - {}
"should have an empty password field" - {}
"should have an enabled password field" - {}
"should have a disabled login button" - {}
}
"(when the user enters his or her password)" -- {
"should have a username field containing the user's entered username" - {}
"should have a password field containing the user's entered password" - {}
"should have an enabled password field" - {}
"should have a enabled login button" - {}
}
"(when the user clicks the login button)" -- {
"should log the user into the system" - {}
}
}
}

I used this style for several months, and it works, but in the end I
felt it wasn't a justified use of either operators or implicit
conversions. I blogged about it here:

http://www.artima.com/weblogs/viewpost.jsp?thread=245965

But users can do it if they prefer, or pick some other operators, like
>> if you like that better.

Bill

[Joakim Ohlrogge]

Hi Bill, (I really should join the scalatest-list, sorry for hijacking
this list)

> That all makes sense. My goal with ScalaTest is to facilitate
> different styles of testing, and you and Esko made me realize this is
> an important style to support. The default will continue to be the
> same, but getting the behavior you like will be easy: just mix in
> trait OneInstancePerTest.

I'm glad I managed to make some sense, that doesn't happen everytime
:) I appreciate your efforts to keep all of us happy :) Of course I
would be the happiest if a new fixture per spec/test was the default
and that you would have to mixin a trait to deviate from that. The
reason is that scalatest would be more in line with xUnit this way so
people that migrate from those frameworks would be less surprised.
Secondly the benefits of one instance per test is simplified
testing/specing though I would not have realized this myself if I did
not run into a situation where I noticed it (at least not as fast) so
having it as a default amplifies learning so to speak.

Perhaps the benefits of one instance per test would be more obvious if
the Trait was not named after the implementation. Something like:

CleanFixtureBeforeEach
CleanFixtureEachTest

or something (I realize it's hard to choose between spec/test in the
naming thats why I prefer the first suggestion but maybe neither is
good). My thoughts are that it would be easier to decide wether you
want a CleanFixtureForBeforeEach than wether you want a
NewInstanceEachTest ?

> That alone won't get you distributed tests
> though. For that you'll need to also mix in trait TestDistribution,
> which will cause tests to be executed concurrently when a Distributor
> is passed to execute. This should work when running Specs
> Specifications through ScalaTest too.
>

Sounds really cool. I'd definitely would not want test distribution to
go with CleanFixtureBeforeEach so I agree with you that it should be
another mixin to control that.

> Also, I agree there's a gotcha with the be-return-paren form. I had
> never thought to try that. I'll see if I can't get it to generate a
> test failure with a helpful error message.
>

Thats understandable. It bit me when I was testing a CSS-parser and it
made sense to format the code that way to make the CSS AST:s more
readable in the specs. I can see how you overlooked that gotcha, sure
know I would've.

Thanks!
/Joakim

--
-----------------------------------------------------
Joakim Ohlrogge
Agical AB
Västerlånggatan 79, 2 tr
111 29 Stockholm, SWEDEN

Mobile: +46-708-754004
Blog: johlrogge.wordpress.com
E-mail: joakim....@agical.se

[Joakim Ohlrogge]

You're definitely fast in honoring requests! Keep up the good work. I
am currently set on ScalaTest, it works well for me at the moment so I
won't be testing the new feature anytime soon. But if you don't mind
the occasional opinion from me I'd love to stick around to see how
specs develops "on paper". When I can't keep my mouth shut; consider
me a second class citizen in the specs community whenever my opinion
is in conflict with yours or any first class citizen of the specs
community (1st class = regular user)

I'm impressed by your speed and time to market!

[etorreborre]

Hi Esko,

> BTW, would it be possible to have a meaningless alias for the word
> "should"?

There is currently no meaningless alias for should but in your case,
you could add something like "then" with the following implicit
definition:

implicit def ToThenDefinition(s: String) = new ThenDefinition(s)
class ThenDefinition(s: String) {
def then(ex: =>org.specs.specification.Example) = {
s.should(e).verb = "then" // this creates the sus with a new
example and modifies the default verb ("should") to "then"
}
}
"when user enters the login screen" then {
....
}

However, you cannot use >> as a meaningless operator since this is
already defined for the Example class.

Note also that if necessary, there is also a way to add another word
to "should": http://code.google.com/p/specs/wiki/DeclareSpecifications#Extend_the_should_verb.

Eric.

[etorreborre]

Hi Esko,

At the moment, I only slightly modified the execution model of a
specification to either execute all examples using a new specification
instance or not (using shareVariables).

I'll try to see how I can generalize this to systems under
specification as you defined it but that may not be so easy.
The other thing which I currently don't like much is the way
subexamples work. Since subexamples are declared inside the body of an
example, they are only executed when the Example is executed. So their
execution model is a bit different. To make subexamples truly work
like examples would require a new "keyword", like:

"this is an example" where { // this block is immediately executed
when the example is defined so that subexamples can be added to their
parent example right away
"this is true" in { ... }
"this happens" in { ... }
}

I think I need more (quiet,...) time to think about all this and how I
could implement a full control over what's shared or not in a
specification. It looks like it would be very valuable to have this
kind of "scalability" in the specification.

Eric.

[etorreborre]

Thanks Joakim,

Hi Joakim,

> You're definitely fast in honoring requests

I try my best to support specs users!

> But if you don't mind the occasional opinion from me

Not at all, don't keep your mouth shut, criticism and suggestions are
always welcome to improve things :-)

Eric.

[Bill Venners]

Hi Joakim,

On Sun, May 31, 2009 at 12:10 AM, Joakim Ohlrogge
<joakim....@gmail.com> wrote:
>
> Hi Bill, (I really should join the scalatest-list, sorry for hijacking
> this list)
>

Yes, we should probably move further discussion of OneInstancePerTest
over to the scalatest-users list.

>> That all makes sense. My goal with ScalaTest is to facilitate
>> different styles of testing, and you and Esko made me realize this is
>> an important style to support. The default will continue to be the
>> same, but getting the behavior you like will be easy: just mix in
>> trait OneInstancePerTest.
>
> I'm glad I managed to make some sense, that doesn't happen everytime
> :) I appreciate your efforts to keep all of us happy :) Of course I
> would be the happiest if a new fixture per spec/test was the default
> and that you would have to mixin a trait to deviate from that. The
> reason is that scalatest would be more in line with xUnit this way so
> people that migrate from those frameworks would be less surprised.
> Secondly the benefits of one instance per test is simplified
> testing/specing though I would not have realized this myself if I did
> not run into a situation where I noticed it (at least not as fast) so
> having it as a default amplifies learning so to speak.
>
> Perhaps the benefits of one instance per test would be more obvious if
> the Trait was not named after the implementation. Something like:
>
> CleanFixtureBeforeEach
> CleanFixtureEachTest
>
> or something (I realize it's hard to choose between spec/test in the
> naming thats why I prefer the first suggestion but maybe neither is
> good). My thoughts are that it would be easier to decide wether you
> want a CleanFixtureForBeforeEach than wether you want a
> NewInstanceEachTest ?
>

Well, although I want ScalaTest to facilitate different styles of
testing, and really get out of the way and let people work they way
they feel most productive, I also want to provide a clear recommended
style. And that recommended style with regards to shared fixtures
won't be mutable data in instance variables. It will be easy to adopt
that style, for those who are used to it from their experience with
other test frameworks, or simply like it, but the ScalaTest defaults
will assume the recommended style. I haven't released yet the
recommended way to do fixtures, by the way, so stay tuned on that.
JavaOne is our biggest week at Artima, so I've been busy leading up to
that. But after next week I'll get more cycles to get the next
ScalaTest release out the door. (And it will have enhancements that
makes it nicer to run Specs Specifications through ScalaTest's Runner,
primarily the ability to control indentation in the output.)

>> Also, I agree there's a gotcha with the be-return-paren form. I had
>> never thought to try that. I'll see if I can't get it to generate a
>> test failure with a helpful error message.
>>
>
> Thats understandable. It bit me when I was testing a CSS-parser and it
> made sense to format the code that way to make the CSS AST:s more
> readable in the specs. I can see how you overlooked that gotcha, sure
> know I would've.
>

It goes to show that you're not necessarily "done when you can't think
of any more tests to write," as the XP literature puts it. Trouble is
you may not think of tests to write. I've had failures of imagination
like that many times. This is one example. I never thought to write a
test like that, so I was unaware of the problem and didn't do anything
to fix it.

Bill

> Thanks!
> /Joakim
>
>
>
> --
> -----------------------------------------------------
> Joakim Ohlrogge
> Agical AB
> Västerlånggatan 79, 2 tr
> 111 29 Stockholm, SWEDEN
>
> Mobile: +46-708-754004
> Blog: johlrogge.wordpress.com
> E-mail: joakim....@agical.se
>
> >
>

[Bill Venners]

Hi Eric, Esko,

On Sun, May 31, 2009 at 2:53 PM, etorreborre <etorr...@gmail.com> wrote:
>
> Hi Esko,
>
> At the moment, I only slightly modified the execution model of a
> specification to either execute all examples using a new specification
> instance or not (using shareVariables).
>
> I'll try to see how I can generalize this to systems under
> specification as you defined it but that may not be so easy.
> The other thing which I currently don't like much is the way
> subexamples work. Since subexamples are declared inside the body of an
> example, they are only executed when the Example is executed. So their
> execution model is a bit different. To make subexamples truly work
> like examples would require a new "keyword", like:
>
> "this is an example" where {  // this block is immediately executed
> when the example is defined so that subexamples can be added to their
> parent example right away
>  "this is true"  in { ... }
>  "this happens"  in { ... }
> }
>
> I think I need more (quiet,...) time to think about all this and how I
> could implement a full control over what's shared or not in a
> specification. It looks like it would be very valuable to have this
> kind of "scalability" in the specification.
>

I hadn't realized the subexamples were really examples inside an
example. I actually made nesting examples like that a runtime error, a
test failure, when someone tries to do that in ScalaTest:

scala> import org.scalatest.Spec
import org.scalatest.Spec

scala> class MySpec extends Spec {
| it("should blow up if I try and nest tests/examples") {
| it("this should cause a failure") {}
| }
| }
defined class MySpec

scala> (new MySpec).execute()
- should blow up if I try and nest tests/examples *** FAILED *** (<console>:7)
org.scalatest.TestFailedException: An it clause may not appear
inside another it clause.
at org.scalatest.Spec$class.it(Spec.scala:559)
at line2$object$$iw$$iw$MySpec.it(<console>:5)
at line2$object$$iw$$iw$MySpec.protected$it(<console>:7)
at line2$object$$iw$$iw$MySpec$$anonfun$1.apply(<console>:7)
at line2$object$$iw$$iw$MySpec$$anonfun$1.apply(<console>:7)

The "where" approach you suggested sounds better to me, but the word
"where" is a bit prescriptive. Since you're using operators you could
also come up with another operator to use in place of where for those
intermediate phrases.

But I also wanted to ask Esko is there a problem with doing this:

class LoginUiSpec extends org.specs.Specification {

"The login screen (when the user first enters it)" should {
"have a empty username field" in {}
"have an empty password field" in {}
"have a disabled password field" in {}
"have a disabled login button" in {}
}
"The login screen (when the user enters his or her username)" should {
"have a username field containing the user's username" in {}
"have an empty password field" in {}
"have an enabled password field" in {}
"have a disabled login button" in {}
}
"The login screen (when the user enters his or her password)" should {
"have a username field containing the user's entered username" in {}
"have a password field containing the user's entered password" in {}
"have an enabled password field" in {}
"have a enabled login button" in {}
}
"The login screen (when the user clicks the login button)" should {
"log the user into the system" in {}
}
}
}

Isn't that how you would express this in Specs? The output looks like:

acintosh-65:delus bv$ scala -classpath specs-1.5.0.jar:st/lib/junit-4.4.jar
Welcome to Scala version 2.7.4.final (Java HotSpot(TM) Client VM, Java
1.5.0_16).
Type in expressions to have them evaluated.
Type :help for more information.

scala> :load esko.scala
Loading esko.scala...
defined class LoginUiSpec

scala> (new LoginUiSpec).main(Array())
Specification "LoginUiSpec"
The login screen (when the user first enters it) should
o have a empty username field
PENDING: not yet implemented (ConsoleReporter.scala:204)
o have an empty password field
PENDING: not yet implemented (ConsoleReporter.scala:204)
o have a disabled password field
PENDING: not yet implemented (ConsoleReporter.scala:204)
o have a disabled login button
PENDING: not yet implemented (ConsoleReporter.scala:204)

Total for SUS "The login screen (when the user first enters it)":
Finished in 0 second, 0 ms
4 examples (4 skipped), 0 expectation, 0 failure, 0 error

The login screen (when the user enters his or her username) should
o have a username field containing the user's username
PENDING: not yet implemented (ConsoleReporter.scala:204)
o have an empty password field
PENDING: not yet implemented (ConsoleReporter.scala:204)
o have an enabled password field
PENDING: not yet implemented (ConsoleReporter.scala:204)
o have a disabled login button
PENDING: not yet implemented (ConsoleReporter.scala:204)

Total for SUS "The login screen (when the user enters his or her username)":
Finished in 0 second, 0 ms
4 examples (4 skipped), 0 expectation, 0 failure, 0 error

The login screen (when the user enters his or her password) should
o have a username field containing the user's entered username
PENDING: not yet implemented (ConsoleReporter.scala:204)
o have a password field containing the user's entered password
PENDING: not yet implemented (ConsoleReporter.scala:204)
o have an enabled password field
PENDING: not yet implemented (ConsoleReporter.scala:204)
o have a enabled login button
PENDING: not yet implemented (ConsoleReporter.scala:204)

Total for SUS "The login screen (when the user enters his or her password)":
Finished in 0 second, 0 ms
4 examples (4 skipped), 0 expectation, 0 failure, 0 error

The login screen (when the user clicks the login button) should
o log the user into the system
PENDING: not yet implemented (ConsoleReporter.scala:204)

Total for SUS "The login screen (when the user clicks the login button)":
Finished in 0 second, 0 ms
1 example (1 skipped), 0 expectation, 0 failure, 0 error

Total for specification "LoginUiSpec":
Finished in 0 second, 88 ms
13 examples (13 skipped), 0 expectation, 0 failure, 0 error

Thanks.

Bill

[Bill Venners]

Hi Eric,

One more idea for you to ponder during quite time. If you made
something like a prepend operator, you could maybe use it to get rid
of redundancy in the SUT part as well as in the subexamples part:

class LoginUiSpec extends org.specs.Specification {

"The login screen" ++ {

"(when the user first enters it)" should {
"have a empty username field" in {}
"have an empty password field" in {}
"have a disabled password field" in {}
"have a disabled login button" in {}
}

"(when the user enters his or her username)" should {
"have a username field containing the user's username" in {}
"have an empty password field" in {}
"have an enabled password field" in {}
"have a disabled login button" in {}
}

"(when the user enters his or her password)" should {
"have a username field containing the user's entered username" in {}
"have a password field containing the user's entered password" in {}
"have an enabled password field" in {}
"have a enabled login button" in {}
}

"(when the user clicks the login button)" should {
"log the user into the system" in {}
}
}
}

class AmpersandSpec extends org.specs.Specification {

"The Scala language" should {
"provide a && operator" ++ {
"returning true for true && true" in { true && true must beTrue }
"returning false for true && false" in { true && false must beFalse }
"returning false for false && true" in { true && false must beFalse }
"returning false for false && false" in { false && false must beFalse }
}
}
}

In both cases ++ would conceptually immediately execute the following
block, and magically prepend its string to the strings contained in
the block, whether they are followed by "should" or "in". You could
then deprecate the old nested examples approach, and disallow them
like I did in ScalaTest, which would I think solve the isolation
problem with nested examples.

Bill

On Sun, May 31, 2009 at 2:53 PM, etorreborre <etorr...@gmail.com> wrote:
>

[Esko Luontola]

On Jun 1, 12:41 am, etorreborre <etorrebo...@gmail.com> wrote:
> There is currently no meaningless alias for should but in your case,
> you could add something like "then" with the following implicit
> definition:

Thanks. I'll try different approaches and see what fits my style the
best.

[Esko Luontola]

My idea about subexamples is to have an elegant recursive solution to
unify all levels of the spec. There would be no difference between the
Specification, SUS, example and subexamples. Instead, there would be
one top-level example, which can have multiple child examples, so that
they form a tree structure. Like this:

class ASpec {
/* this is the top-level example */
}

class ASpec {
"something" x {
/* this is a second-level example */
}
}

class ASpec {
"something" x {
"something" x {
/* this is a third-level example */
}
}
}

class ASpec {
"something" x {
"something" x {
"something" x {
/* this is a fourth-level example */
}
}
}
}
class ASpec {
"something" x {
"something" x {
"something" x {
"something" x {
/* this is a fifth-level example */
}
}
}
}
}

And so on. The basic keyword (above "x") would be the same for all
example levels. It might have multiple aliases ("should", "in", ">>"
etc.), but they all provide the same behaviour (except that they
append a different word to the example description).

It should be possible to implement that so, that only the top-level
example has some special handling (because it is implemented as code
in the constructor of the Spec class). The implementation of all other
example levels would be identical, so that they are recursive.

The test output should indent the example descriptions the same way,
as a nested tree structure.

[etorreborre]

That's indeed a good idea, I keep that in mind.

Even if I want to keep the example/subexamples approach, this may help
refactoring some description text.

E.

[etorreborre]

Hi Esko,

There is currently a distinction between a SUS and an example because:

- the SUS has some functionalities to set/reset state using Context
objects or before/after methods
- the body of a SUS is evaluated as soon as it is declared to create
its belonging Example objects while the execution of the Example body
is delayed

This is also what makes difficult to have the kind of recursivity you
describe (tough I like the idea I must say). For reporting and
execution purposes (for the JUnit integration for example), it is
necessary to create the full structure of the specification, then
execute each example. So in the "tree" of examples, the inner nodes
must be executed right away, while the leaf nodes must be executed on
demand.

This is why I'm currently thinking of reserving >> always for
declaring subexamples and "in" for example bodies (that may break
existing specifications, so that would be interesting only if this
allowed greater control on the examples isolation).

I'm going to experiment a little bit with that and I'll come back to
you.

E.

[Esko Luontola]

On Jun 2, 2:41 pm, etorreborre <etorrebo...@gmail.com> wrote:
> For reporting and
> execution purposes (for the JUnit integration for example), it is
> necessary to create the full structure of the specification, then
> execute each example.

Where can I find out more information about integration with JUnit and
others? Are they so strict that they require the tests to be executed
in a specific order? Would it be possible to have some sort of facade,
so that the tests in the tree are executed in a mixed order, but the
facade reports the test results to the surrounding framework in a
normalized order?

[Esko Luontola]

I wrote a proof of concept about recursive examples: http://gist.github.com/123025

[Esko Luontola]

On Jun 3, 6:04 pm, Esko Luontola <esko.luont...@gmail.com> wrote:
> I wrote a proof of concept about recursive examples:http://gist.github.com/123025

With this, every non-leaf example acts as a setup, so the setup
methods of Specs (doBefore/doFirst/doBeforeSpec) may be removed as
redundant. A teardown method is still needed, for example to clean up
modified global state (static variables, files, database etc.). To the
above proof-of-concept, an optional doAfter/tearDown/destroy method
should be added to each example (similar to how
RecursiveSpecification.forExample is implemented).

The contexts (shared, internal, external) might also be removed as
redundant. Or is there some use case, where they are still needed? In
the above proposal, every example is a context.

Repeated examples will need some thinking, that how to best implement
them. Some concrete real-life use cases are needed to better design
it. Maybe the repeated examples could be used to implement
parameterized tests (http://junit.sourceforge.net/javadoc/org/junit/
runners/Parameterized.html). Now the parent example is run to
completion before its child example is run, but to implement repeated
examples it might be necessary to nest the execution of the child
example with its parent (in that case a teardown could also be
implemented as code after the child example declarations).

It's unclear to me how sharing examples in Specs works ("behave like"
etc.). What is their use case? Is it possible to see some code that
uses them? Is the sharing only within one Specification class, or is
sharing possible between many classes?

Sequential execution can be removed as obsolete. It could be replaced
with the sharing of side-effects within one example, as I've described
earlier in this thread, but it's not something that I would need
often. Complete isolation is much more useful.

When an example fails, none of its child examples should be executed
(even if the system would know about some child examples). A failing
non-leaf example should show up in the test output the same way as a
failing leaf example. An added benefit of this is, that if there is a
failure in setup (i.e. a non-leaf example), it is reported only once.

It should be possible to nest examples unlimitedly, and they should be
reported as a nested tree structure. It is unclear to me, that will
all test runners (JUnit, IDEs etc.) supported deeply nested tests. I
hope some there is some solution (mostly I care about integration with
IntelliJ IDEA and Maven).

[etorreborre]

To answer your questions about JUnit:

-How to do with deeply nested examples and JUnit?

Examples with nested examples are represented as TestSuites and "leaf"
examples are represented as TestCases.

-What is the execution model?

JUnit runners require that the whole tree structure of what is
supposed to be executed (TestSuites and TestCases in JUnit3 terms) is
first fleshed out, in order to be presented on screen. Then, each test
is executed and its status attached to the tree structure. This allows
a progress bar to be displayed.

In the design you sent (which I'm still having a look at, but I have
very few free time these days :-( ), I don't see how we can do this
because there's no distinction between "in" to define subexamples and
"in" to define the example body.

Any idea?

[Esko Luontola]

> In the design you sent, I don't see how we can do this

> because there's no distinction between "in" to define subexamples and
> "in" to define the example body.

Why should there be a distinction? Do you mean by "how we can do this"
that how to do the integration with JUnit?

[Esko Luontola]

If it's not possible to know beforehand that how many tests there are,
will the only problem be that JUnit is unable to show a progress bar?
I can try some experiments with JUnit to see how the integration could
work.

If nothing else works, I have an idea about reading the examples
directly from bytecode with ASM. That would bypass the limitation that
the spec needs to be executed before we can know what examples it
contains. But implementing that will be complicated (it requires
creating a simple bytecode interpreter), so at first it's better to
try other things.

[etorreborre]

> If it's not possible to know beforehand that how many tests there are,
> will the only problem be that JUnit is unable to show a progress bar?

Yes, that's exactly the problem. I had this "bug" at first where there
would be nothing displayed in the JUnit GUI runner until the full spec
was executed, then all the examples would appear.

> If nothing else works, I have an idea about reading the examples

> directly from bytecode with ASM....

That's a little bit scary to resort to bytecode manipulation here. I'd
rather have a more obvious approach, even at the expense of using a
different keyword for nesting examples.

[etorreborre]

An other solution would be to restrict the type of the "in" method:

def in(ex: Example) = {
// create a subexample, don't execute anything
}
def in(result: Result[_]) = {
// create an example body for something ending with an expectation
like a must_== b, and store it for later execution
}
implicit def anyToResult[T](t: T) = new Result(t) // this would be
used when the Example body ends with anything but an expectation, nor
an example.

I'll see where this lead goes and come back to you.

[Esko Luontola]

On Jun 4, 5:37 am, etorreborre <etorrebo...@gmail.com> wrote:
> That's a little bit scary to resort to bytecode manipulation here. I'd
> rather have a more obvious approach, even at the expense of using a
> different keyword for nesting examples.

It's not manipulation as no bytecode is changed - it's bytecode
analyzing. I have some experience about bytecode manipulation and
analyzing, so I can try if I can produce something that works reliably
enough. Unless somebody writes specs that declare examples in loops or
inside closures, it should be possible to get a list of examples
reliably without executing the examples.

[Esko Luontola]

I have started writing a bytecode interpreter which can be used to
find out what examples a spec contains without the need to execute any
test code. It's progressing step by step: http://github.com/orfjackal/bytecode-detective

I had a try at JUnit integration, but JUnit's test runner API seems
too obscure and not well enough documented, so I gave up. ScalaTest's
API seemed better organized, although I did not look at it very
closely. I hope that you can help with the test runner integration,
after I provide something that produces a tree of all examples in the
spec.

[etorreborre]

Yes of course I can help you with the runner integration. I'm going to
have a look at your code to understand how it can be plugged to specs.

[etorreborre]

Hi all (and Esko specifically),

I have finished working on a new specs version where:

- sub-examples are first-class citizen. For example, they display
properly now in the JUnit view under Eclipse
- variables are isolated, which means that this specification sent by
Esko will pass:

class isolatedExamples extends spex.Specification {

var x = 0
"a" should {

x += 1

"b" in {
x += 1

"b1" in {
x += 1

x must_== 3

}
"b2" in {
x += 1

x must_== 3

}
}
"c" in {
x += 1

x must_== 2
}
}
"d" should {
x += 1
"e" in {
x += 1
"e1" in {
x += 1
x must_== 3
}
"e2" in {
x += 1
x must_== 3
}
}
"f" in {

x += 1
x must_== 2
}
}
}

As a direct consequence:

- setup/teardown methods are not necessary with "local" variables (but
may be required for external resources). In practice, the
specifications are more concise and face less risk of dependent
failures
- System contexts have been removed. They are not necessary at all, or
can be replaced by using a regular context, when sharing examples
between systems, as in the Stack specification:
http://code.google.com/p/specs/source/browse/trunk/src/test/scala/org/specs/samples/stackSpec.scala

You'll also notice that the version number has been incremented to
1.6.0 as there are API break risks, which I hope will stay minor. You
can find the jar here: http://scala-tools.org/repo-snapshots/org/scala-tools/testing/specs/1.6.0-SNAPSHOT.

Now I would like a few users to try this version and report any issue,
so that I can release it with confidence. And,... start working on the
migration to 2.8.0,...

Thanks,

Eric.

[Esko Luontola]

The example isolation appears to work now just as desired. Thanks. Now
I can begin using it in my projects. :)

One thing that could still be improved, is how many times each piece
of code is run. The way I write tests, most of the work is done in the
parent example and the child examples contain mostly only assertions
about different aspects of the parent example's results. That's why it
would be important to not execute the parent example any more times
than is necessary.

Given the following code:

class isolatedExamples2 extends org.specs.Specification {
println("root")
var x = 0
x must_== 0
"a" in {
println("a")
x += 1
x must_== 1
"aa" in {
println("aa")

x += 1
x must_== 2

"aaa" in {
println("aaa")

x += 1
x must_== 3
}

"aab" in {
println("aab")

x += 1
x must_== 3
}
}

"ab" in {
println("ab")

x += 1
x must_== 2
}
}

"b" in {
println("b")
x += 1
x must_== 1
"ba" in {
println("ba")

x += 1
x must_== 2

"baa" in {
println("baa")

x += 1
x must_== 3
}

"bab" in {
println("bab")

x += 1
x must_== 3
}
}

"bb" in {
println("bb")

x += 1
x must_== 2
}
}
}

When it is run, right now it prints the following:

root
root
root
a
root
root
a
aa
root
root
a
root
root
a
aa
aaa
root
root
a
root
root
a
aa
aab
root
root
a
ab
root
b
root
root
b
ba
root
root
b
root
root
b
ba
baa
root
root
b
root
root
b
ba
bab
root
root
b
bb

We can see that "root" is executed 28 times. Also, "a" is executed 3
times as alone and 4 times with its child examples. It should be
enough to execute "a" in total only 3 times to cover every execution
path.

Here is the optimal execution order, so that each example is executed
at least once in isolation, but not any more times than is necessary:

root
a
aa
aaa
root
a
aa
aab
root
a
ab
root
b
ba
baa
root
b
ba
bab
root
b
bb

I have been experimenting this way of executing the examples at
http://github.com/orfjackal/misc-tools/tree/master (package
net.orfjackal.experimental.specs). Right now I'm writing a bytecode
analyzer to find out beforehand whether an example has child examples,
and how many examples there are in total, so that integration with
IDEs would work better.

Still one more thing. Now if a parent example contains assertions,
Specs creates a child example called "example 1". Would it be possible
to get rid of those implicitly created example names in test results?
Also, if no SUS is created with "should" (like in the above
isolatedExamples2), Specs adds creates implicitly a SUS called
"specifies". Could also that be removed?

[etorreborre]

Hi Esko,

I've started improving the situation:

root
a
aa
aaa
root
a
aa
aab
root
a
ab
root
b

root
b
ba

root
b
ba
baa
root
b
ba
bab
root
b
bb

But it should be possible to do better, so I'm going keep on trying.

> Still one more thing. Now if a parent example contains assertions,
> Specs creates a child example called "example 1". Would it be possible
> to get rid of those implicitly created example names in test results?

This serves 2 purposes:

- you can write specifications as a bunch of expectations only. This
is in the case where you're not interesting in a readable output

object s extends Specification {
thisThing must have size(2)
thisThing.calculate must_== 3.14
}

- "wrapping" top level expectations as an anonymous example gives a
place to report failures (and otherwise, you would also get a bad
stacktrace showing that your spec can't even be instantiated)

> Also, if no SUS is created with "should" (like in the above
> isolatedExamples2), Specs adds creates implicitly a SUS called
> "specifies". Could also that be removed?

- yes, this doesn't add any value, I'll keep the implicitly created
SUS but I'll remove it from the output.

Eric.

[etorreborre]

Hi,

I've just deployed a new version (1.6.0-SNAPSHOT) which produces the
following output:

root

root
a
aa
aaa
root
a
aa
aab
root
a
ab
root
b

ba
baa
root
b
ba
bab
root
b
bb

for the following specification:

class isolatedExamples2 extends org.specs.SpecificationWithJUnit {

println("root")
var x = 0

"it" should {

Please, all specs users have a go at this version and tell me if it
works for you.

Thanks,

Eric.

[Esko Luontola]

It appears to work correctly. Thanks.

We just need to wait for IDEA's test runner to be updated so that it
will report all the examples, but it's reasonably usable already now.

[etorreborre]

Thanks for the feedback (small victory dance,...). I'll wait a bit
before I publish it.

> We just need to wait for IDEA's test runner to be updated so that it
> will report all the examples, but it's reasonably usable already now.

The "Notifier" trait which I provide for the specs support in Intellij
doesn't model sub-examples. I need to work with the IDEA developer on
that.

Eric.