Hi Sukant,

Your HyptheticalSpecification looks similar to the SpecDasher style I
had in an earlier version of ScalaTest. I took it out because I felt
the operators made the code look ugly and it were very non-obvious.
Here's some ScalaDoc with an example:

http://www.artima.com/scalatest/doc-0.9.4/org/scalatest/SpecDasher.html

One difference is that it didn't do the isolation of nested scopes
that you like, and it also used dashes instead of the >> operator. I
think your example looks and reads very nicely, but I've seen some
pretty bad grammar and non-sentences made with the Specs >> operator.
I have tried to guide the sentence more, in the hopes of helping
people write documentation that reads well. But I've seen examples
where that failed too. For example the it(...) clause in a Spec should
be a sentence like it("should do something"), but people have just
ignored the it and said things like it("a completely unrelated,
non-sentence, stream-of-consciousness bit of text"). I found both
kinds of examples by searching the web for open source test classes.

I dropped SpecDasher quite a while ago, but had it in my mind that
someday I might add a "FreeSpec," which is like you're example,
because it is like free verse poetry. You can format your text any way
you like. (And you could write it in any language you like, which was
another use case I figured FreeSpec might have.)

I also would note that your nesting, despite having dropped "given"
and "then," is still a given/when/then kind of structure. Vigorous
writing is concise, and so it is nice to drop given and then since
they are not needed, but the good thing about having them is that they
guide the structure. So I am warming up to the idea of a GivenSpec
that does that, but not sure yet.

As far as isolation goes, what I'd like to do is make the kind of
isolation you like a trait you mix in, but that's tough to do. The
reason is that I essentially need to pass a "path" to the test to the
constructor, so that when a class is instantiated for the purpose of
running a particular test, it doesn't execute any blocks that don't
enclose the test. So I kind of need the test to have an auxiliary
constructor that takes a List[Int], and that's hard to do with a
trait. There are ways to do it, but they make the client code more
cumbersome. So far the easiest way on users that I have figured out to
do it is to just make GivenSpec a class. That burns the base class,
which is a pain, but the style you've shown is I think a style I'd
like ScalaTest to support, because some people like to write tests
that way. I'll keep thinking about it and see if I can't find some
decent way to get the benefits of traits and still be able to get the
isolation in a non-painful way.

I would ask one question, though. Why would you want the beforeContext
thing? I would think it would be better to put the list += 2 code
right in the body of the "when two is added" clause. Then it is
explicit. If you have code duplication, then you could factor that
duplicate code out into an addTwoToTheList, and call it explicitly.

Bill

--
Bill Venners
Artima, Inc.
http://www.artima.com

Hi Bill,

I'm noticing my replies are double-posting.  I apologize for that.
I'm submitting these replies through Google's web front-end, so I
suspect it's a bug on their side.  In the meantime, I'm trying to be
very careful about what I click.

On Jul 26, 12:13 pm, Bill Venners <b...@artima.com> wrote:

This is /exactly/ my experience.  No matter what, with or without
guidance from the framework, it's up to the developer to write
cohesive, comprehensible descriptions.  Personally, I suspect this is
a higher-order skill than the programming itself.  In light of this,
my biases lean towards the SpecDasher approach.

This pretty much seems like the direction I'd prefer.  So let me see
if I'm clear on how this would look.  You wouldn't have both "-" and
"--" in FreeSpec, right?  Because the differentiation between the two
seems to be more about differentiating the SUS from the context, and I
would rather manage that "freely" through text without the test
framework forcing a structure on me.

Perhaps the difference between "-" and "--" also has to do with
helping the DSL manage formatting of the test description.  My gut
feeling is that outer closure can be detected without the syntactic
demarcation, but maybe the implementation gets hacky.

I think I followed more or less the difficulties you described.  I'll
need to get my feet more wet in more Scala before the problems and
limitations are visceral.  In the meantime, I'm enthusiastic that we
both have some interest in seeing if we can pull off test isolation in
some reasonable way.  I'm working towards getting to a point where I
can make meaningful patches to Scala projects.  I certainly would
prefer to keep ScalaTest trait-oriented if possible.  For now, I'd
like to see if we can nail down what we'd like the system to look
like, then I'll have some concrete tests to make pass and I can get
down into something more low-level.

Great.  This part of our conversation seems a little bit of stylistic
nit-picking.  You're right that my strategy is ultimately Dan North's
Given-When-Then (GWT) style.  The tweaks I do are really subtle.  One
justification you might appreciate is that GWT seems to put a more
imperative spin on the context and specification of a behavior.  The
"given" is often a declarative setup of context with the "when" being
a side-effecting action.  But if I'm working in a more immutable
system.  The difference between the "given" and the "when" becomes a
lot weaker.  Both are just calculations (say, Scala val assignments).
But really, I'm not trying to convince you of anything. . . just help
explain my inclinations.

Also, from the perspective of counting keystrokes, I just didn't see
much of a difference between

    when("under some context") { ... }

and

    "when under some context" { ... }

The important part for me is that as I read a "context path" from the
outside-in I get a clear and understandable message.

Okay, you're right to call me out on what in retrospect was a hair-
brained idea.  Let me see if I can elucidate the problem I really want
to focus on.  Specs has an admittedly imperative way of managing
fixture and context setup and teardown.  With my nesting strategy, I
can manage setup, but I haven't been explicitly clear on how to manage
teardown.  I took some time to think deeply about the
org.scalatest.fixture package.  The implicits required for the
MultipleFixture variants are too verbose for my tastes.  But for the
simple Fixture traits I can definitely appreciate the way you're
trying to manage setup and teardown functionally.  The only problem I
have is that with this approach the fixture needs to be this explicit
(typed) object.  Whereas with the FreeSpec approach we've been
discussing, through the benefits of closures, I get everything I need
in scope without the type-check hassles of passing fixtures through
functions.

So ultimately, the real problem I want to focus on is teardown.
Here's a first shot at an idea:

    class TearDownsAttempt1Spec
            extends FreeSpec
            with OneInstancePerTest
            with ShouldMatchers {

        def doAsserts(sus: System, ctx: Context) = {
            String.format(
                    "then some our invariant holds for system %s,
context %s",
                    sus, ctx) -- {
                sus forContext(ctx) holdsInvariant should equal true
            }
        }

        "given a system under test" -- {
            val sus = new System
            sus.start()

            "when using our first context" -- {
                val ctx = new Context
                ctx.init(1)
                doAsserts(sys, ctx)
                ctx.teardown()
            }

            "when using our second context" -- {
                val ctx = new Context
                ctx.init(2)
                doAsserts(sys, ctx)
                ctx.teardown()
            }

            sus.end()
        }
    }

Sorry, I stuffed two concepts into that example as I typed it: a
strategy for teardowns and a way to reuse assertions.  I'm hoping this
idea for managing teardown is not suprising, given it's very
symmetrical to how setups are managed.  For the reuse of assertions,
I'm borrowing an idea from Specs.

So in my mind, what the runner will do is iteratively run through all
the code in the specification class sequentially, selectively enabling
and disabling closures to assert test isolation -- but always running
from beginning to end until the last leaf-closure is executed (using a
tree-model of nested closures).  The algorithm seems clear in my mind,
but I want to better understand how it meets with your sensibilities
and also the difficulties of implementation you described above.

Also, what I like about this "free/isolated" strategy is that I can
trust the runner to have a compact algorithm, albeit a little tricky
to assert isolation.  This gives me the freedom to design my own
strategies for context and assertion reuse independent from the test
framework, perhaps in a way that's also sensitive to my domain.

What do you think?  Am I straying too far from ScalaTest's core
philosophy?  Or do you feel like I'm pursuing an ill-advised design?

-Sukant

Hi Sukant,

I only received one copy each time.

The differentiation is that currently tests are registered during
construction, but not executed. Everything else is executed during
construction. So for example in a Spec, describe clauses get executed,
but it clauses don't. In a FeatureSpec, feature clauses get executed
by scenario clauses don't. Well the scenario method does get called of
course, as does the it method in Spec, but what those methods do is
*register* the block of test code for later execution. The tests are
executed later when run is called on the Spec or FeatureSpec object.
Similar for WordSpec, the relatives in the fixture package, etc.

So the -- and - was differentiated so that the -- ones could be
executed at construction time and the - ones not. I would keep that,
but I am thinking of continuing to use "in" for the test marker to
reduce by half the number of non-obvious operators to just one. That
comes at the end of the phrase, so I think it isn't too in the way.
Also, "in" is used in exactly the same way in several other traits,
i.e., to mark a test, so that would also sometimes help people figure
out what it likely means. I'm not sure what symbol to use. Here's what
one would look like if we use the ~ symbol:

class ListSpec extends FreeSpec with NestedScopeIsolation with MustMatchers {

  "a ListBuffer of Ints" ~ {

    val list = ListBuffer[Int]()

    "when one is added" ~ {

       list += 1

       "should have only 1 in it" in {
         list must be === ListBuffer(1)
       }

       "and still have only 1 in it" in {
         list must be === ListBuffer(1)
       }
    }

    "when two is added" ~ {

       list += 2

       "should have only 2 in it" in {
         list must be === ListBuffer(2)
       }

       "and still have only 2 in it" in {
         list must be === ListBuffer(2)
       }
    }
  }

I'm open to suggestions for what operator to use where I'm using the ~
in this example.

I think I figured out a way to do it with traits. I haven't coded it
up yet so I may still realize I didn't imagine something, but baring
that it will be a trait. I'm not sure what to call the trait, which to
me is a sign that this is hard to describe to users. Basically when
you mix in NestedScopeIsolation or whatever I call this trait, each
test would be run in its own instance, and only surrounding blocks
will be executed. It complicates the execution model, but in return
people get the option of writing tests this way.

The way things work now is there's a "registration phase" and a "ready
phase". Here's what the ScalaDoc says for Spec, for example:

A Spec's lifecycle has two phases: the registration phase and the
ready phase. It starts in registration phase and enters ready phase
the first time run is called on it. It then remains in ready phase for
the remainder of its lifetime.

That's pretty simple and easy to understand. What it will need to change to is:

At construction time, only the top level of describe and it clauses
would get executed. These describe and it clauses would be registered
for later execution. As soon as any of methods tags, testNames, run,
expectedTestCount is executed, then full registration would take
place. Then it enters the ready phase like before.

I'd create a trait NestedScopes (not sure of the name) that has a
runTestsIsolated (also not sure of the name) method in it. Traits that
have nested scopes, like WordSpec, Spec, FeatureSpec, FreeSpec,
GivenSpec, would mix in this trait, but traits that don't have nested
scopes (FunSuite, Suite, FlatSpec) would not. RunTestsIsolated would
have a self type of NestedScopes, so it can only be mixed into a trait
that has nested scopes. (It makes no sense to mix NestedScopeIsolation
into FlatSpec, for example, because the whole point of FlatSpec is to
*not* have nesting.)

NestedScopeIsolation would extend OneInstancePerTest and override
runTests, and what it would do is create a new instance for each test,
but call runTestIsolated for each test instead of plain old runTest.
This is the method that would do partial registration, using info
passed into runTestIsolated indicating the location of the test to run
(i.e., the location would indicate which describe clauses surround the
single test to execute in this instance, so that it can run only those
surrounding describe clauses and no others).

So that's more complicated to understand, but I think I can explain it
in the ScalaDoc. And you'd have the visual trigger of "with
NestedScopeIsolation" if the tests are exhibiting this useful but
possibly surprising behavior.

I'm currently thinking of adding both a GivenSpec and a FreeSpec, so
people would have a choice. GivenSpec is more verbose, but not by
much, and it is also a good guide on how to structure your test, and
pretty clear to read. Though I may hold it back because no one has
actually asked for that one yet, whereas someone (you) is asking for
FreeSpec. I can always add it later. One concern I have is that
there's a GivenWhenThen trait that lets you put that kind of
documentation *inside* tests, pretty much any kind of style, and ...

read more »

On Jul 28, 11:26 am, Bill Venners <b...@artima.com> wrote:

I need more time to digest the entirety of your response.  But in the
meantime, I am concerned about the lifecycle of "~" closures versus
"in" closures, especially with respect to the semantics of setup and
teardown we've discussed.

My worries are that what might be a non-issue for something like an in-
memory list might become awkward with something expensive like a
database operation.  So consider the following specification using
your suggested "~"/"in" notation.

    class ListSpec
            extends FreeSpec
            with NestedScopeIsolation
            with MustMatchers {

        //1 TestDatabase.clear()

        "a repository of employees" ~ {

            //2 TestDatabase.clear()
            val repo = Repository[Employee]()

            "when a new employee has been added" ~ {

                val empName = "Sukant"
                val emp = Person(name)
                repo add emp

                "has the employee that's been added" in {
                    repo findByName name must be === Some emp
                }

                "can delete the employee that's been added" in {
                    repo delete emp
                    repo findByName name must be === None
                }
            }

            //3 TestDatabase.clear()
        }

        //4 TestDatabase.clear()

    }

I quickly freehanded the test above, so pardon any grammatical
mistakes.  In comments, I've indicated four places that we might clear
out the database to assert test isolation beyond the in-memory
isolation we get from the NestedScopeIsolation trait.

What's important for me is that the clear operation only happen
exactly twice, once for each assertion (appropriately before or after
the test as specified).  I'd /really/ not like for the clear to happen
any more times.  This is something I wasn't 100% sure Specs was
doing.  I created some simple tests to better understand the closure
lifecycle, but my analysis is not yet done.

To satisfy all four suggested positions for the database clear, I have
an idea for a test runner that wouldn't construct the class until the
first test execution.  That execution with the NestedScopeIsolation
trait mixed in would potentially have the first test execution spawn
off other "context paths" to traverse (using a simple work queue of
context paths).  When I think about this approach, I'm reminded of the
kind of execution a tool like Java PathFinder employs.  Paths aren't
registered apiori, but discovered at runtime.  Also, a consequence of
this strategy is that there's no need algorithmically to differentiate
between "in" and "~" closures.

Unfortunately, this lifecycle for a specification object might be too
naive given all the use cases ScalaTest must support, or incompatible
with ScalaTest's current architecture.

Hopefully my description is clear.  I'm tempted to code up a simple
runner to illustrate the idea when I have the time.  But I certainly
don't want to maintain my own test framework; mostly it would be for
the practice of writing Scala.  My intent is really to contribute back
to either ScalaTest or Specs.  Or at the least, facilitate some
discussion.  With ScalaTest, it seems we're starting with a greener
field, which is more interesting for me.  With Specs, there's already
an implementation for most of the features I want, so I'd probably
continue to shoehorn my own testing style into what's there.

-Sukant

Hi Sukant,

I think tests written in a nested scope isolation can get confusing to
read, which is why I've been concerned about it. The problem is it is
non-obvious what's going on behind the scenes. It sounds like you are
writing little examples in specs to try and figure out the execution
model. I'm not sure what specs does with things after a test either.
My intuition is that it would execute them after the test, but I have
no idea how Eric has implemented his isolation. If it isn't describe
in the ScalaDoc, then I'd have to write little examples to discover
the execution model also.

By default in ScalaTest what you see is what you get. If it looks like
code in one closure will see the side effects by code in some other
closure that both reference the same mutable object, it does. Closures
work like closures. If you mix in OneInstancePerTest, then you get
isolation just between the tests. I think that's surprising
behaviour--I remember quite surprised when I found out JUnit worked
that way. But at least it was simple to understand. The nested scope
isolation is both surprising and confusing. But in the right hands, I
think it can make for declarative, readable tests.

In your case, basically it sounds like you want to clear the database
before and after each test. (Correct me if I misunderstood you.) Note
that if you just mixed in BeforeAndAfterEach, even though your code
would be more verbose, it would be very clear to readers what's going
on. That would look like this:

class ListSpec
           extends FreeSpec
           with BeforeAndAfterEach
           with MustMatchers {

  val repo = Repository[Employee]()

  def beforeEach() {
    TestDatabase.clear()
    val empName = "Sukant"
    val emp = Person(name)
    repo add emp
  }

  def afterEach() {
    TestDatabase.clear()
  }

  "a repository of employees" ~ {

    "when a new employee has been added" ~ {

      "has the employee that's been added" in {
        repo findByName name must be === Some emp
      }

      "can delete the employee that's been added" in {
        repo delete emp
        repo findByName name must be === None
      }
    }
  }

I know this isn't the style you prefer, but I want to point out how it
does serve the reader of the code by marking off explicitly what you
want to happen before each test and after. The nesting serves to
reduce duplication in specifying the *text* that describes the test,
not the test code itself.

With NestedScopeIsolation, you'd just put one clear before all tests
and one after. I'd put the one after in the same scope as the line of
code that first alters the database. I.e., the cleanup would be at the
same level as the "messing up" above it, like this:

class ListSpec
           extends FreeSpec
           with NestedScopeIsolation
           with MustMatchers {

  "a repository of employees" ~ {

    val repo = Repository[Employee]()

    "when a new employee has been added" ~ {

      val empName = "Sukant"
      val emp = Person(name)
      TestDatabase.clear()
      repo add emp

      "has the employee that's been added" in {
        repo findByName name must be === Some emp
      }

      "can delete the employee that's been added" in {
        repo delete emp
        repo findByName name must be === None
      }

      TestDatabase.clear()
    }
  }

Or if you felt it was clearer to just indicate you're starting with an
empty database and ending with one, you might do it this way:

class ListSpec
           extends FreeSpec
           with NestedScopeIsolation
           with MustMatchers {

  TestDatabase.clear()

  "a repository of employees" ~ {

    val repo = Repository[Employee]()

    "when a new employee has been added" ~ {

      val empName = "Sukant"
      val emp = Person(name)
      repo add emp

      "has the employee that's been added" in {
        repo findByName name must be === Some emp
      }

      "can delete the employee that's been added" in {
        repo delete emp
        repo findByName name must be === None
      }
    }
  }

  TestDatabase.clear()

On readability, I think the BeforeAndAfterEach example is easier for
readers of the code to figure out what's going on than the
NestedScopeIsolation ones. Many readers would probably need to go to
NestedScopeIsolation's ScalaDoc and figure out when things will be
executed and how they'll be isolated.

Were I to get rid of the "in", it would mean I couldn't know until
after I run a ~ clause if there were any nested ~ clauses in it. So
I'd have to run everything at construction time, basically, to find
out how many tests there are. ScalaTest has an expectedTestCount
method that needs to return how many tests should be coming. What's
nice about the "in" is that I can register tests without executing
them. Without that "in" I'd have to essentially execute tests when a
FreeSpec was constructed and in that second step of full registration,
then store the results, and send the results out when run is called.
Or just run them again when run is called. That's not impossible, but
again I think that's non-intuitive. Also, storing results would take
up memory, and on a large test suite that could be a problem. Running
tests twice could be a problem if some tests take a long time. Both of
those potential problems can be avoided if there's a specific marker
for a test like "in". Lastly, having a special marker for a test helps
communicate what block of code actually constitutes a test. That's why
I had -- and - in SpecDasher. - was the marker for tests. -- marked
descriptive clauses. But I'll think about it. I agree it does *look*
nicer to just use one symbol all the way down. In ScalaTest, at least,
if you had a problem with memory or execution time, you could switch
to something besides FreeSpec.

Bill

...

read more »

On Jul 28, 2:46 pm, Bill Venners <b...@artima.com> wrote:

I meant more that I wanted the option of either clearing the database
before the test or after.  But that's a small clarification.  Your
examples are still relevant.

On this point, I have to agree.  But when I look at what Eric's
implemented along these lines, there's doFirst, doLast, doBefore,
doAfter, doBeforeSpec, doAfterSpec, etc.  So this is the other
extreme, and I feel I loose track of what the exact semantic is for
each of these.  Ultimately, I just like having the option to do what
makes sense given the context of the domain and the specification I'm
working with.  But perhaps that flexibility comes at a cost on the
framekwork.

I'm having a hard time understanding whether all the operations
outside the "in" closure get executed (or not) while you're
registering a expectedTestCount of the "in" closures.

If they don't get executed, what Scala feature are you using to count
the "in" closures without executing anything?  As far as I can tell
from the example I've excerpted above, simply instantiating the
ListSpec will clear the database twice, even if you choose not to
execute the outer-most "~" closure.

Actualy, I'm fine with a syntactic marker to annotate assertions.  I
think my argument against "in" really isn't so strong.  Mostly, I just
want to make sure the lifecycle of the specification's closures are
sane, which I think is your primary concern too (along with
readability of the specification).

-Sukant

Hi Sukant,

Sorry for the delay in responding to  your last email. I was busy
getting a few things out the door, including teaching a Scala class.
That behind me I'd like to pick up this discussion again. I've also
CC'd Esko Luontola, who was the first person to suggest this kind of
test isolation on the specs mailing list a little over a year ago. He
recently released a specsy framework that does the isolation the way
he likes it, and also uses the same >> character all the way down with
unlimited nesting--it sounds similar to what you've been sketching
out. Esko, I wonder if you'd mind joining the scalatest-users mailing
list so we can get your opinions on this too.

At the bottom of your most recent email, Sukant, you asked about why I
wanted a special marker for a leaf node. The reason is so that those
nodes can be registered for later execution, whereas the surrounding
nodes would be executed immediately. An example from Dasher could
clarify:

"this is a descriptive clause" -- {
  "this is another descriptive clause" -- {
    "but this marks a test" - {
    }
    "and this marks another test" - {
    }
  }

This was syntax sugar for:

describe("this is a descriptive clause") {
  describe("this is another descriptive clause") {
    it("but this marks a test") {
    }
    it("and this marks another test") {
    }
  }

When the Spec is constructed all the describe (or --) clauses are
executed at that time. I.e., the code between the curly braces is
passed up as a function to describe, and describe immediately executes
it. But when you get to an it clause (or -), the code between those
curly braces is passed to it and *not* executed by the it method, but
just registered for later execution. This allows tests to be counted
up front, but not executed until later when run() is called.

When the same marker is used all the way down, the only way to find
the leaf nodes is to execute the code of all nodes. So that means all
the code either gets executed when the Spec is constructed, or perhaps
when expectedTestCount is invoked. The results of that test execution
could be stored and reported later when run is called, so it is
possible, but surprising. If tests have side effects, users might be
very surprised about when they happen. But nested scope isolation also
yields surprising timing of test execution, so maybe it isn't so bad.
That would make tests look like both you and Esko envision:

"this is a descriptive clause because it isn't a leaf node" ~ {
  "this is another descriptive clause because it isn't a leaf node" ~ {
    "but this is a test because it is a leaf node" ~ {
    }
    "and this is another test because it is a leaf node" ~ {
    }
  }

In ScalaTest, a Suite (or Spec) is a collection of tests, each of
which has a unique string name. The test name for a FreeSpec would be
composed of the concatenation of all the strings in surrounding
clauses, one for each leaf node. This allows the test to be run
through JUnit, for example. But when you run it through ScalaTest,
you'd see a nice spec-like output.

But I have one question for both of you, Sukant and Esko: would you
ever want to be able to put assertions in non-leaf nodes? I looked at
the examples both of you have written, and in none of them do I see
something like this:

"outer" ~ {
  "inner" ~ {
     assert(something here)
     "leaf 1" ~ {
       //...
     }
  }

I have only seen assertions in leaf nodes in your examples. The reason
I ask is that an assertion would need to be inside a test, not in
descriptive clauses. You could set up fixture data in the descriptive
clauses, and nested scope isolation would make things nice, but if you
put an assertion in there that failed, then there really isn't a test
to report as failed given the current way I'm thinking of mapping
FreeSpec code to tests.

In short, would it be OK with you two if there was a rule in FreeSpec
that you could only put assertions in leaf nodes, and that the
surrounding nodes should be for setting up (and tearing down) fixture
data?

Thanks.

Bill

--
Bill Venners
Artima, Inc.
http://www.artima.com

And then replies to Bill's posts...

On Jul 26, 8:13 pm, Bill Venners <b...@artima.com> wrote:
 > As far as isolation goes, what I'd like to do is make the kind of
 > isolation you like a trait you mix in, but that's tough to do. The
 > reason is that I essentially need to pass a "path" to the test to the
 > constructor, so that when a class is instantiated for the purpose of
 > running a particular test, it doesn't execute any blocks that don't
 > enclose the test. So I kind of need the test to have an auxiliary
 > constructor that takes a List[Int], and that's hard to do with a
 > trait. There are ways to do it, but they make the client code more
 > cumbersome.

The way I do it in Specsy is to pass the parameter through a
ThreadLocal. (CGLIB does it the same way for the proxies it generates.)
That way the client code is not affected.

The code which instantiates the test uses ContextDealer.prepare() to
pass the context to the test:

http://github.com/orfjackal/specsy/blob/master/src/main/scala/net/orf...
http://github.com/orfjackal/specsy/blob/master/src/main/scala/net/orf...

And the test gets the context like this:

http://github.com/orfjackal/specsy/blob/master/src/main/scala/net/orf...

On Jul 28, 7:26 pm, Bill Venners <b...@artima.com> wrote:
 > The way things work now is there's a "registration phase" and a "ready
 > phase". Here's what the ScalaDoc says for Spec, for example:
 >
 > A Spec's lifecycle has two phases: the registration phase and the
 > ready phase. It starts in registration phase and enters ready phase
 > the first time run is called on it. It then remains in ready phase for
 > the remainder of its lifetime.
 >
 > That's pretty simple and easy to understand. What it will need to
change to is:
 >
 > At construction time, only the top level of describe and it clauses
 > would get executed. These describe and it clauses would be registered
 > for later execution. As soon as any of methods tags, testNames, run,
 > expectedTestCount is executed, then full registration would take
 > place. Then it enters the ready phase like before.

Specsy has no "phases". All of it is just test execution. The test class
(or its constructor) actually behaves more like a function. This can be
better seen in GoSpec where the specs are functions and there is less
syntactic sugar:

http://github.com/orfjackal/gospec/blob/gospec-1.2.0/examples/stack_t...

Altough this means that the framework won't known which tests there are,
until it has executed all tests. It doesn't play well with JUnit's test
runner, which makes the implicit assumption that all tests are known
beforehand. So I've had to create a JUnit runner which executes all
tests the first time either run() or getDescription() is called:

http://github.com/orfjackal/specsy/blob/master/src/main/scala/net/orf...

Also what the tests print to stdout/stderr needs to be captured and
printed later when JUnit thinks the tests are being executed.

 > NestedScopeIsolation would extend OneInstancePerTest and override
 > runTests, and what it would do is create a new instance for each test,
 > but call runTestIsolated for each test instead of plain old runTest.
 > This is the method that would do partial registration, using info
 > passed into runTestIsolated indicating the location of the test to run
 > (i.e., the location would indicate which describe clauses surround the
 > single test to execute in this instance, so that it can run only those
 > surrounding describe clauses and no others).

Actually the reason why I didn't use the test runner from ScalaTest,
Specs or TestNG, is because they require the test classes to extend a
specific base class/trait, so that the testing framework cannot fully
control how the test class is instantiated. Also I require it to be
possible to use multiple test frameworks in the same project and run
them all at the same time.

JUnit was the only one that made it possible, but because it doesn't
handle well nested tests and frameworks which do not know all tests
beforehand, nor does it have built-in support for parallel execution,
I'll create a new test runner at some point. I'll contact all testing
framework, IDE, CI server and build tool developers later this year, to
get some feedback on what is required from a test runner that can
execute all testing frameworks on the JVM. What do you think, would
CTR4J (Common Test Runner for Java) be a good name, or do you have some
other ideas?

 > I don't know of any other test framework (besides
 > specs) that has done this kind of isolation. It would have to be in a
 > language with closures, but there are other languages with closures
 > besides Scala. So I'm surprised it hadn't popped up in those
 > language's test frameworks before.

The first framework I used was JDave (which in turn was probably
influenced by JUnit and RSpec), so my style of writing tests developed
to be something that runs on JDave. It has the same concepts of
isolation and nested tests, although it requires the use of create()
methods and the level of nesting is fixed.

http://www.jdave.org/examples.html

Also I think that RSpec has the same isolation model and unlimited
nesting, although it requires you to write before(:each) and describe
blocks. If you take out all the boilerplate, you will end up with
something like Specsy.

 > I think I can let you do teardown in a way that's analogous to the way
 > you'll be doing setup. I think it should work that way, because  it
 > would be symmetrical. So if you want to clear the list after each test
 > as a tear down, for example, it would look like this:
 >
 > class ListSpec extends FreeSpec with NestedScopeIsolation with
MustMatchers {
 >
 >   "a ListBuffer of Ints" ~ {
 >
 >     val list = ListBuffer[Int]()
 >
 >     "when one is added" ~ {
 > ...
 >     }
 >
 >     "when two is added" ~ {
 > ...
 >     }
 >     list.clear()
 >   }
 >
 > }
 >
 > So if you mix in NestedScopeIsolation, only code in surrounding scopes
 > will be executed for each test. Code that appears before a test in
 > surrounding scopes will be executed before the test. Code that appears
 > after the test in surrounding scopes will be executed after the test.

Specsy actually works that way, but it's encouraged to use the defer
blocks instead (http://github.com/orfjackal/specsy#readme), because
otherwise:

- The corresponding setup and teardown code are separated, so it's
harder to see whether they are in sync. When updating setup you might
forget to update teardown because the teardown is not visible.

- If there are multiple steps in the teardown process, if one of them
fails then the rest of them are not executed.

- The user needs to manually specify the order of the teardown to match
the order of the setup. Usually the teardown needs to be done in the
reverse order than it was setup.

Bill Venners wrote on 23.8.2010 9:49:

Specsy identifies the tests by their position in the tree of tests
(http://github.com/orfjackal/specsy/blob/master/src/main/scala/net/orf...).
So it's possible for different tests to have the same name.

I haven't implemented support for running a single test from a test
class. It should be possible, but I'm not sure whether it's worth the
additional complexity (now Specsy is nice and small - the source code is
570 LOC, or 320 LOC if you exclude the test runner which will be
replaced by CTR4J). I don't have need for executing single tests,
because my tests are fast enough to execute all tests at the same time.

Yes. If there is complex setup code, then I may make some assertions
about it to make sure that the test is in a known state. In
http://github.com/orfjackal/specsy#readme both FibonacciSpec and
DeferBlocksExampleSpec/DeferBlocksExample2Spec do assertions in setup.

The @Before methods in these classes also have assertions:

http://github.com/orfjackal/tdd-tetris-tutorial/blob/beyond/src/test/...
http://github.com/orfjackal/tdd-tetris-tutorial/blob/beyond/src/test/...

Most of the time I don't need assertions there - having to assert
something about the setup is a test smell that the setup is too complex
- but every now and then it is needed.

--
Esko Luontola
www.orfjackal.net

Hi Esko,

I think you should take a closer look at ScalaTest. Trait Suite
defines methods such as run, runTests, runNestedSuites, testNames,
runTest, withFixture, nestedSuites, expectedTestCount, etc., all of
which can be overridden to define new ways of expressing and running
tests. The reason I can implement nested scope isolation as a mix in
is because the NestedScopeIsolation trait can override one or more of
these methods and add that behavior. Same for a FreeSpec, which is
basically  I designed ScalaTest to facilitate different styles of
testing, and this is one it can support.

I think for better or for worse, JUnit is the common test runner. And
I don't think there's a problem getting this style to run through it.
Though, the output is ugly. But if you're looking for a more universal
runner that also understands specification-style output, please have a
look again at ScalaTest. It has a very general, well defined model for
running and reporting of tests of any style. Test events are fired
through a Reporter, which has a method infoProvided. infoProvided (and
other events) can include an optional Formatter, which has two
subtypes MotionToSuppress and IndentedText. MotionToSuppress indicates
you'd like the event to not show up in reports, and IndentedText
indicates both an indentation level/raw string combination and has a
formattedText string for text output (which would probably put spaces
in front for indentation, maybe with a bullet character, etc.) It is
very general.

That makes sense. Already in my example that I sent to Sukant I felt
it was difficult to be sure I was putting the teardown code in the
correct block. Not sure about defer yet, but having some other option
that allows users to specify teardown code in this style seems useful.

OK. What you implemented sounds about like what I imagined I'd need to
do in FreeSpec, however, there are some choices. If you have a
different marker for leaf nodes than the surrounding nodes, you can
defer the execution of those leaf nodes until the run method is
called. That's what I was pushing back on Sukant with. So instead of:

"outer" - {
  "inner" - {
     "leaf node" - {
      }
   }

It would be something like:

"outer" - {
  "inner" - {
     "leaf node" in {
      }
   }

Sukant actually said he wouldn't mind that. But then the other thing
is what if someone puts an assert outside the in. Well I could have a
rule in FreeSpec that says your assertions have to go in leaf nodes,
because those are the "tests". But I'm not sure that sounds like
"freedom." Maybe freedom with responsibility.

Another way to go is to execute all the code the first time I need to,
such as when expectedTestCount or run is invoked, store the results
(the events), and fire them to the reporter when run is called later.
(If run is called a second time I'd run them again though. And if an
assertion goes off in a non-leaf node, I'd just report the test as
having the name up to that point. So for example:

"outer" - {
  "inner" - {
     assert(false)
     "leaf node" - {
       assert(false)
      }
   }

When that assertion goes off, the name of the failed test would be
reported as "outer inner", even though there's no leaf node in it. If
you fix that one:

"outer" - {
  "inner" - {
     assert(true)
     "leaf node" - {
       assert(false)
      }
   }

You'd get a failure for test "outer inner leaf node." Now one thing
that could happen is you end up with a duplicate test exception in
such a case. If you had this:

"outer" - {
  "inner" - {
     assert(false)
     "leaf node" - {
       assert(false)
      }
   }
  "inner" - {
   }

If these tests were to all run and pass, you'd get two tests with the
names "outer inner leaf node" and "outer inner", which works fine
because they are unique names. But since the first non-leaf test that
failed will be recorded with the name "outer inner" here, the second
test will be a duplicate. My feeling here is that this would be
extremely rare, and so long as I report the first error so it can be
fixed, the second one can just be a duplicate test name exception
until the first error is fixed.

Now on the subject of defer, it is pretty to look at, but I think it i
non-obvious. You'd have to learn what defer means by looking at the
documentation of FreeSpec, and my goal is that the meaning ScalaTest
code be obvious without looking at the documentation. The other thing
is it requires reassigning a var or mutating an object to associate
the defer function with the block.

How would you feel if I don't add any feature to support this, but
simply document there are two ways to do the teardown. One is to put
it at the end of the block:

"a directory" - {
  val dir = new File("myFile.txt")
  try {
    "can be created with mkDir" - {
      dir.mkDir() should be (true)
    }
    "should exist after being created" - {
      dir.exists should be (true)
    }
  }
  finally {
    dir.delete()
  }

The Scala way to improve this situation is using the loan pattern, and
that would be my recommended alternative:

def withDir(dir: File)(f: => ()) {
  try { f } finally { dir.delete() }

"a directory" - {
  val dir = new File("myFile.txt")
  withDir(dir) {
    "can be created with mkDir" - {
      dir.mkDir() should be (true)
    }
    "should exist after being created" - {
      dir.exists should be (true)
    }
  }

Now there's no reason to go to the documentation to understand the
code. It is jut the loan pattern, and you can follow the code. It is
right in front of you. There's also no mutation needed behind the
scenes, so it is more functional, less behind-the-scenes magical, and
therefore in my opinion, easier to understand.

Though admittedly, it is more verbose to write. I'm often faced with
trading off readability and writability, and I tend to lean towards
readability. But writability is important too. Maybe I offer all three
ways so users have a choice. (And I wonder if there's a name for
defer, like afterBlock or afterScope or something that makes it more
obvious when it is being deferred until.)

"a directory" - {
  val dir = new File("myFile.txt")
  afterBlock {
    dir.delete()
  }
  "can be created with mkDir" - {
    dir.mkDir() should be (true)
  }
  "should exist after being created" - {
    dir.exists should be (true)
  }

Not as pretty as defer. Maybe just "after."

The only other thing I'd mention is that I don't think nested scope
isolation is a good default, because it is a surprising execution
model. By ...

read more »

Bill Venners wrote on 23.8.2010 20:41:

But to be able to call the methods on Suite, the test class which
extends Suite must be instantiated, right? So the custom test runner
(which is the same as the test class) cannot execute code before the
test class is instantiated.

I forgot to mention, but there is a stronger reason why anything written
in Scala cannot be used as a common test runner: Scala programs are not
binary compatible between different Scala version (IntelliJ IDEA's Specs
or ScalaTest integration has blown up on me multiple times in the past
when it was written in Scala - I think now it's all in Java). Also
scala-library.jar is a rather big dependency to be included in every
build tool, IDE, CI server etc. Writing CTR4J in Java is the only option.

One of my goals is to reduce the duplication between all build tools,
IDEs and CI servers. For example IntelliJ IDEA has its own test runner
integration for JUnit, Specs, ScalaTest, TestNG etc. and all with
varying levels of quality and features. Also there are variations
between how Eclipse and IDEA run the tests, due to IDE specific bugs and
assumptions - I want zero bugs, especially in my test infrastructure.

For those coming from a JUnit background, not isolating the tests is
surprising.

--
Esko Luontola
www.orfjackal.net

Hi Esko,

If scala-library.jar is a problem then ScalaTest is out of the
"running" for sure. But I actually don't understand what you mean
here, and would like to. Can you clarify by giving a specific example
of code you would want to "execute before the test class is
instantiated?"

True, which kind of says that no matter what the default, it will
likely surprise some users. My feeling is that the default should be
what-you-see-is-what-you-get, with minimal or no behind-the-scenes
fancy behavior.

Since my last email I realized that executing the tests on
construction or when expectedTestCount is invoked breaks the promise
of BeforeAndAfterAll methods, overriding runTests, run, etc. These are
ways to do things before and after all tests are run. So if one of
those things is setting up a database, for example, then that wouldn't
work in a FreeSpec. So I think FreeSpec will need to differentiate
between description and test clauses, perhaps like I showed Sukant:

 "a directory" - {
   val dir = new File("myFile.txt")
   after {
     dir.delete()
   }
   "can be created with mkDir" in {
     dir.mkDir() should be (true)
   }
   "should exist after being created" in {
     dir.exists should be (true)
   }
 }

Leaf nodes get marked with an "in". Surrounding nodes are marked with
"-", and can be nested indefinitely. Code inside a description ("-")
clause get executed at class instantiation time, and code inside test
("in") clauses get executed only when run is invoked.

I would probably recommend people avoid putting assertions in
description clauses, but it wouldn't be the end of the world. These
would probably be rare, and would fail even more rarely. When they
fail they will be reported as a SuiteAborted not a TestFailed, but
will give sufficient information to go fix the problem. So I think
that's fine.

I have a new question about after. Should multiple after clauses be
allowed in a block, and if so, what order are they executed on exit?
I'm guessing yes, and last registered first executed (i.e. executed in
reverse order of appearance).  Is that what you do in Specsy? This is
another example of how such code would not be obvious to casual
readers without reading the documentation, but again, it would likely
be rare and could be well documented.

Thanks.

Bill
----

--
Bill Venners
Artima, Inc.
http://www.artima.com