Test Isolation with ScalaTest?
Sukant Hajra
I'm just beginning to learn Scala and looking at test frameworks. I'm
really
interested in BDD-style tests. Here's two examples I put together for
both as
a point of comparison:
Specs:
class MutableListSpecsTest extends Specification {
"an empty list" should {
var list = ListBuffer[Int]()
"when 1 is inserted" in {
list += 1
"has only 1 in it" in {
list mustEqual ListBuffer(1)
}
}
"when 2 is inserted" in {
list += 2
"has only 2 in it" in {
list mustEqual ListBuffer(2)
}
}
}
}
ScalaTest:
class MutableListScalaTestSpec extends Spec
with ShouldMatchers with OneInstancePerTest {
describe("an empty list") {
val list = ListBuffer[Int]()
describe("when 1 is inserted") {
list += 1
it("has only 1 in it") {
list should be (ListBuffer(1))
}
}
describe("when 2 is inserted") {
list += 2
it("has only 2 in it") {
list should be (ListBuffer(2))
}
}
}
}
As you can see, the structure of the tests in both is more or less the
same.
But the one for Specs passes, whereas the one for ScalaTest fails. I
tried to
mix in "OneInstancePerTest", but that was just a shot in the dark (I
haven't
dug into the source yet to see what it actually does).
So I'm curious if there's a way to get test isolation in ScalaTest
without a
lot of extra code or falling back on a non-BDD JUnit-based runner. Or
is this
more of a fundamental paradigm difference like the one between TestNG
and
JUnit?
Thanks,
Sukant
Bill Venners
You hopefully won't need to dig into the source. I tried to make the
documentation thorough enough that you can just consult that. The doc
for OneInstancePerTest is here:
http://www.scalatest.org/scaladoc/doc-1.0/org/scalatest/OneInstancePerTest.html
By default ScalaTest does nothing to isolate tests. So if test one has
a side effect, test two will be able to see it. This enables tests to
be written such that one test depends on another, as in TestNG. If you
prefer the isolation technique given to you by JUnit, then you can
just mix in OneInstancePerTest. JUnit actually does this by default,
and you can't really change it. If you run a JUnit test class with ten
tests in it, you'll get ten instances of that test class at runtime.
That's what you get with ScalaTest if you mix in OneInstancePerTest.
Now what you're trying to do here seems more like a given/when/then
approach. Given an empty list, when you add one to it, then it should
contain one. ScalaTest has a GivenWhenThen trait that may be a better
fit than what you're doing here. Mixing GivenWhenThen in can give you
code that reads like:
given("an empty list")
val list = ListBuffer[Int]()
when("you add one to it")
list += 1
then("it should contain one")
list should contain (1)
And this documentation gets printed out in the output too.
However, what you encountered with your example does highlight a
difference between Specs and ScalaTest. Specs has a notion of a
"nested example" or "nested test," and ScalaTest does not. I feel that
"nested test" is a non-intuitive concept. I don't know what it means.
Specs also turns off closures so that they don't work like you'd think
they would work looking at the code. It does this for the purpose of
isolating tests. In your case you wanted the behavior, but were
writing a test and expecting closures to work like closures, then you
might be scratching your head for a while.
In a ScalaTest Spec, anything in the describe clauses will be executed
during the construction of the Spec instance. The intention of the
describe clauses is to describe the subject of the test. The "tests"
that run are defined inside the it clauses. The it method call happens
at construction time, but all it does is register the code inside it
for later execution as a test. So what you were doing in your nested
describe clause is adding one to the listbuffer at construction time.
This is not happening inside a test, but at construction time in a
describe clause. So really you've got test code inside a describe
clauses, which is intended to be used for describing the subject under
test, not doing the test. To get the behavior you want, you need to
move the test code inside the it clause, like this:
class MutableListScalaTestSpec extends Spec
with ShouldMatchers with OneInstancePerTest {
describe("an empty list") {
val list = ListBuffer[Int]()
describe("when 1 is inserted") {
it("has only 1 in it") {
list += 1
list should be (ListBuffer(1))
}
}
describe("when 2 is inserted") {
it("has only 2 in it") {
list += 2
list should be (ListBuffer(2))
}
}
}
}
Bill
> So I'm curious if there's a way to get test isolation in ScalaTest
> without a
> lot of extra code or falling back on a non-BDD JUnit-based runner. Or
> is this
> more of a fundamental paradigm difference like the one between TestNG
> and
> JUnit?
>
> Thanks,
> Sukant
>
> --
> You received this message because you are subscribed to the Google
> Groups "scalatest-users" group.
> To post to this group, send email to scalate...@googlegroups.com
> To unsubscribe from this group, send email to
> scalatest-use...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/scalatest-users?hl=en
> ScalaTest itself, and documentation, is available here:
> http://www.artima.com/scalatest
>
--
Bill Venners
Artima, Inc.
http://www.artima.com
Bill Venners
I've finally got a bit of time to explain the idea behind the design
and show you some alternative ways to write your tests. The main idea
is to minimize what people need to know about the framework, to make
it easy for people reading the code to guess what's going on. When you
see a closure being updated, then later being updated again farther
down the code, what you'd normally expect in Scala code is that the
second update would see the changes to the first. That's how closures
work in Scala, and that's how they work in ScalaTest by default. If
you want to isolate tests, I prefer that it be explicit in your code,
such as by mixing in OneInstancePerTest. That way when someone else
comes along to read it, they don't need to dig into the details of a
framework's execution model to understand what's going on.
That said, ScalaTest is designed to support different styles of
testing, and the style I felt you wanted is not really the style
intended by Spec. What I felt you wanted was a "nested
given/when/then" syntax like this:
given("an empty list") {
val list = ListBuffer[Int]()
when("1 is inserted") {
list += 1
then("it should have only 1 in it") {
list should be (ListBuffer(1))
}
}
when("2 is inserted") {
list += 2
then("it should have only 2 in it") {
list should be (ListBuffer(2))
}
}
}
This is a nice syntax, but it has that thing that bugs me, which is
that looking at this code you'd expect sequential execution, and under
normal Scala closure behavior you'd expect the list to have both 1 and
2 in the last then clause. On the other hand, if this is what people
want, we could add something like this to ScalaTest. I'd like to hear
people's opinions on it at least.
What Spec, WordSpec, FeatureSpec, and FlatSpec are designed to do is
let you nest the description text, but not the tests themselves. As I
pointed out in my previous email, I don't think it makes much sense to
nest tests as Specs does. ScalaTest offers several ways to isolate
tests, but not to isolate the closures that describe the test. So for
example, here's one way to write your test using a WordSpec:
import org.scalatest.matchers.ShouldMatchers
import org.scalatest.OneInstancePerTest
import org.scalatest.WordSpec
import org.scalatest.GivenWhenThen
import org.scalatest.fixture.FixtureWordSpec
import scala.collection.mutable.ListBuffer
class MutableListScalaTestWordSpec1 extends WordSpec with ShouldMatchers {
"A List" when {
"1 is inserted" should {
"have only 1 in it" in {
val list = ListBuffer[Int]()
list += 1
list should be (ListBuffer(1))
}
}
"2 is inserted" should {
"have only 2 in it" in {
val list = ListBuffer[Int]()
list += 2
list should be (ListBuffer(2))
}
}
}
}
The output looks like:
A List (when 1 is inserted)
- should have only 1 in it
A List (when 2 is inserted)
- should have only 2 in it
This has the disadvantage that the first line of each test is
duplicated, although it does have the advantage that I only need to
look at those three lines of code to understand the test. But
ScalaTest has several ways to reduce that code duplication. The more
functional way is to use the org.scalatest.fixture package. I"ll show
that last, but if what you were really after is Given/When/Then style
documentation, you could mix in GivenWhenThen and do something like
this:
class MutableListScalaTestWordSpec2 extends WordSpec with
ShouldMatchers with GivenWhenThen {
"A List" when {
"1 is inserted" should {
"have only 1 in it" in {
given("an empty list")
val list = ListBuffer[Int]()
when("1 is inserted")
list += 1
then("it should contain only 1")
list should be (ListBuffer(1))
}
}
"2 is inserted" should {
"have only 2 in it" in {
given("an empty list")
val list = ListBuffer[Int]()
when("2 is inserted")
list += 2
then("it should contain only 2")
list should be (ListBuffer(2))
}
}
}
}
Now the output would look like:
A List (when 1 is inserted)
- should have only 1 in it
+ Given an empty list
+ When 1 is inserted
+ Then it should contain only 1
A List (when 2 is inserted)
- should have only 2 in it
+ Given an empty list
+ When 2 is inserted
+ Then it should contain only 2
And here's how you'd use a FixtureWordSpec to pass an empty list into
each test, so that the line of code that creates the empty list is no
longer duplicated:
class MutableListScalaTestWordSpec3 extends FixtureWordSpec with
ShouldMatchers {
type FixtureParam = ListBuffer[Int]
def withFixture(test: OneArgTest) {
test(ListBuffer[Int]())
}
"A List" when {
"1 is inserted" should {
"have only 1 in it" in { list =>
list += 1
list should be (ListBuffer(1))
}
}
"2 is inserted" should {
"have only 2 in it" in { list =>
list += 2
list should be (ListBuffer(2))
}
}
}
}
The output of this looks the same as the first one I showed. There's
also the BeforeAndAfterEach trait, which when mixed in lets you do
fixtures in the imperative setup/tearDown way popularized by JUnit,
and the OneInstancePerTest way that I showed in my previous email,
which causes each test to be executed in its own instance of the class
(which isolates the tests, but again, not the description text
closures).
So lots of options, but the one that isn't in there now is the nested
given/when/then one I showed first. I'm curious how people feel about
the tradeoffs there. Would you like to be able to write tests that
look like this:
given("an empty list") {
val list = ListBuffer[Int]()
when("1 is inserted") {
list += 1
then("it should have only 1 in it") {
list should be (ListBuffer(1))
}
}
when("2 is inserted") {
list += 2
then("it should have only 2 in it") {
list should be (ListBuffer(2))
}
}
}
Thanks.
Bill
/*
Suite Starting - MutableListScalaTestWordSpec3
A List (when 1 is inserted)
- should have only 1 in it
A List (when 2 is inserted)
- should have only 2 in it
Suite Completed - MutableListScalaTestWordSpec3
*/
On Sun, Jul 11, 2010 at 1:04 PM, Sukant Hajra <a9h5m...@snkmail.com> wrote:
Sukant Hajra
>
> I've finally got a bit of time to explain the idea behind the design and show
> you some alternative ways to write your tests.
subscription
after your original reply. Fortunately, I Googled the thread to
revisit the
discussion.
> The main idea is to minimize what people need to know about the framework, to
> make it easy for people reading the code to guess what's going on. When you
> see a closure being updated, then later being updated again farther down the
> code, what you'd normally expect in Scala code is that the second update
> would see the changes to the first.
like
Scala internal DSLs can rapidly become dangerously expressive (rather
than
verbose like in Java). Here's some of the arguments I have for the
other side:
- Allowing for a little weirdness/non-intuitiveness in tests might
be okay
if the behavior is broadly cross-cutting.
- Test isolation is generally good, so it doesn't strike me as
inherently
bad default.
- Many times, we really can't shield end users from understanding
what a
framework does for long. A lot of frameworks try, but many
don't
succeed. And at the end of the day, it's not always so bad if a
little
bit of a framework's internals are exposed, just as long as it's
not
gushing out.
- When writing tests, we really need to know the lifecycle of
instance/closures anyway, because we're ultimately trying to
control with
some specificity the setup of our tests; nobody wants to write
broken/fragile/slow tests.
- Perhaps what's intuitive might be a bit relative. Whenever I
pass a
closure to a third-party, I make no assumption on when or under
what
conditions that closure will be called. I ultimately have to
look for
documentation or source code.
Please don't get the idea that I'm trying to be combative. Right now,
there's
things I like about both Specs and ScalaTest. ScalaTest's DSL seems
more
readable and straight-forward. With Specs, I find the default test
isolation
just suits my personal preference, but overall, the framework has way
too many
bells and whistles for my tastes.
> If you want to isolate tests, I prefer that it be explicit in your code, such
> as by mixing in OneInstancePerTest. That way when someone else comes along to
> read it, they don't need to dig into the details of a framework's execution
> model to understand what's going on.
OneInstancePerTest to get the behavior I wanted, but I misunderstood
the exact
semantic (I understand now).
> That said, ScalaTest is designed to support different styles of testing, and
> the style I felt you wanted is not really the style intended by Spec. What I
> felt you wanted was a "nested given/when/then" syntax
differentiation
between "given" and "when" to be all that interesting for most testing
situations. I'd be happy enough with just "when/then" where "given"
is just
a special case of "when" as in:
when("given an empty list") {
when("1 is inserted") {
list += 1
then("it should have only 1 in it") {
list should be (ListBuffer(1))
}
}
}
a system
to a certain state. The nesting just lets me pop in and out of state
a little
for some finer-grained testing; essentially giving me some light code
reuse in
a natural way without needing to invest in separate helper methods.
However, I make no claims that this is necessarily a style that should
be
broadly applied. The nesting introduces a complexity that might
impede
comprehension in some situations.
> This [WordSpec] has the disadvantage that the first line of each test is
> those three lines of code to understand the test. But ScalaTest has several
> ways to reduce that code duplication. The more functional way is to use the
> org.scalatest.fixture package. I"ll show that last, but if what you were
> really after is Given/When/Then style documentation, you could mix in
> GivenWhenThen
the code
duplication was a little annoying, which is part of what I'm trying to
resolve
with nesting. So let's take a look at the FixtureWordSpec approach.
> class MutableListScalaTestWordSpec3 extends FixtureWordSpec with
> ShouldMatchers {
>
> type FixtureParam = ListBuffer[Int]
>
> def withFixture(test: OneArgTest) {
> test(ListBuffer[Int]())
> }
>
> "A List" when {
>
> "1 is inserted" should {
>
> "have only 1 in it" in { list =>
>
> list += 1
> list should be (ListBuffer(1))
> }
> }
>
> "2 is inserted" should {
>
> "have only 2 in it" in { list =>
>
> list += 2
> list should be (ListBuffer(2))
> }
> }
> }
>
> }
a given
action (like adding an element above). I apologize my original list
example
didn't highlight this. Even with the reuse of a fixture (system under
test),
I'm not getting reuse of contexts. Of course, I could have multiple
"should"
asserts in a context, but then the text output of the tests loose
specificity.
So here's something Specs does that meets my needs (no duplication,
even with
multiple assertions for a context).
class ListSpec extends Specification {
val list = ListBuffer[Int]()
val twoAdded = beforeContext {list += 2}
"a list" when oneAdded should {
"have only 1 in it" in {
}
"still have only 1 in it" in {
list must_== ListBuffer(1)
}
}
"a list" when twoAdded should {
"have only 2 in it" in {
}
"still have only 2 in it" in {
list must_== ListBuffer(2)
}
}
}
I wonder how this meets with your sensibilities. One thing I like
about this
is that it uses closures in what I think is a natural way, without
introducing
the complexity of something like OneArgTest.
By the way, I definitely am not making an argument to turn ScalaTest
into Specs
piece by piece. I am using Specs currently because I figured out how
to make
it meet my needs and expectations. But I am extremely interested in
testing
patterns, so even if I continue to use Specs, I'm interested in the
discussion.
> So lots of options, but the one that isn't in there now is the nested
> given/when/then one I showed first. I'm curious how people feel about the
> tradeoffs there.
a horse
in this race. But thanks for taking the time to write back. It
really helped
me better understand your opinions on testing strategies and styles.
Best,
Sukant
Sukant Hajra
>
> you some alternative ways to write your tests.
subscription
after your original reply. Fortunately, I Googled the thread to
revisit the
discussion.
> make it easy for people reading the code to guess what's going on. When you
> see a closure being updated, then later being updated again farther down the
> code, what you'd normally expect in Scala code is that the second update
> would see the changes to the first.
> as by mixing in OneInstancePerTest. That way when someone else comes along to
> read it, they don't need to dig into the details of a framework's execution
> model to understand what's going on.
OneInstancePerTest to get the behavior I wanted, but I misunderstood
the exact
semantic (I understand now).
> the style I felt you wanted is not really the style intended by Spec. What I
> felt you wanted was a "nested given/when/then" syntax
differentiation
between "given" and "when" to be all that interesting for most testing
situations. I'd be happy enough with just "when/then" where "given"
is just
a special case of "when" as in:
when("1 is inserted") {
list += 1
then("it should have only 1 in it") {
list should be (ListBuffer(1))
}
}
}
a system
to a certain state. The nesting just lets me pop in and out of state
a little
for some finer-grained testing; essentially giving me some light code
reuse in
a natural way without needing to invest in separate helper methods.
However, I make no claims that this is necessarily a style that should
be
broadly applied. The nesting introduces a complexity that might
impede
comprehension in some situations.
> those three lines of code to understand the test. But ScalaTest has several
> ways to reduce that code duplication. The more functional way is to use the
> org.scalatest.fixture package. I"ll show that last, but if what you were
> really after is Given/When/Then style documentation, you could mix in
> GivenWhenThen
the code
duplication was a little annoying, which is part of what I'm trying to
resolve
with nesting. So let's take a look at the FixtureWordSpec approach.
> ShouldMatchers {
>
> type FixtureParam = ListBuffer[Int]
>
> def withFixture(test: OneArgTest) {
> test(ListBuffer[Int]())
> }
>
> "A List" when {
>
> "1 is inserted" should {
>
> "have only 1 in it" in { list =>
>
> list += 1
> list should be (ListBuffer(1))
> }
> }
>
> "2 is inserted" should {
>
> "have only 2 in it" in { list =>
>
> list += 2
> list should be (ListBuffer(2))
> }
> }
> }
>
> }
a given
action (like adding an element above). I apologize my original list
example
didn't highlight this. Even with the reuse of a fixture (system under
test),
I'm not getting reuse of contexts. Of course, I could have multiple
"should"
asserts in a context, but then the text output of the tests loose
specificity.
So here's something Specs does that meets my needs (no duplication,
even with
multiple assertions for a context).
class ListSpec extends Specification {
val twoAdded = beforeContext {list += 2}
"have only 1 in it" in {
}
"still have only 1 in it" in {
list must_== ListBuffer(1)
}
}
"have only 2 in it" in {
}
"still have only 2 in it" in {
list must_== ListBuffer(2)
}
}
}
I wonder how this meets with your sensibilities. One thing I like
about this
is that it uses closures in what I think is a natural way, without
introducing
the complexity of something like OneArgTest.
By the way, I definitely am not making an argument to turn ScalaTest
into Specs
piece by piece. I am using Specs currently because I figured out how
to make
it meet my needs and expectations. But I am extremely interested in
testing
patterns, so even if I continue to use Specs, I'm interested in the
discussion.
> given/when/then one I showed first. I'm curious how people feel about the
> tradeoffs there.
Bill Venners
Thanks for your thoughtful reply. I've responded at the end, so scroll
way down to see it...
One thing I'd point out first about ScalaTest's fixture package is
that it exists to give people a way to write tests in a functional
style. The traditional JUnit setup and tearDown method approach to
fixtures is by definition an imperative style. The only way those
methods can have any effect on tests is through side effects. The
beforeContext method of Specs is also the same imperative style, but
instead of the same method being called before each test, you can
essentially specify multiple setup methods as functions and specify
which one to call. There's something in the fixture package called
MultipleFixtureWordSpec (and similar traits of other test styles),
which gives you the same kind of thing, except in the functional
style. Instead of calling a function that has side effects, you pass
the different fixtures into the different tests that need them.
Another thing I'd point out is that the problem here is that you're
seeing side effects. To the extent you write your tests so they don't
have side effects, you won't see the problem. But when you are testing
mutable objects for which you want to factor our duplicate setup code,
your tests will likely cause side effects in those shared mutable
fixture objects. (This is avoided in the fixture package by passing in
a fresh batch of those mutable objects to each test. They therefore
aren't shared.)
ScalaTest does not currently include a trait that facilitates
different setups for different tests for the imperative style, though
I could imagine how to write such a trait. (ScalaTest has a very
general execution model, which can be customized through overriding
method, so pretty much any style is possible.) What pops into my mind
is to have a trait that has a var in it that holds a "setup" function,
and a way to let the test change what that function is. That's doable,
but nevertheless I wonder if the most natural, easily readable syntax
for what we're talking about isn't with given/when/then. This syntax
was proposed by Dan North who started the whole BDD thing. He also has
the token "and" in there. You can read about given/when/then on this
page:
http://blog.dannorth.net/introducing-bdd/
With "and," you have what you want in I think a very readable way:
class ListSpec extends GivenSpec with MustMatchers {
given("a ListBuffer of Ints") {
val list = ListBuffer[Int]()
when("one is added") {
list += 1
then("it should have only 1 in it") {
list must be === ListBuffer(1)
}
and("it should still have only 1 in it") {
list must be === ListBuffer(1)
}
}
when("two is added") {
list += 2
then("it should have only 2 in it") {
list must be === ListBuffer(2)
}
and("it should still have only 2 in it") {
list must be === ListBuffer(2)
}
}
}
}
I think that looks pretty nice. No duplication, and I would expect
reading it that before the second when gets run, that the given would
be reset again. How does that syntax strike you?
Bill
> Best,
Bill Venners
One thing I forgot to ask you about yesterday. With regards to your test here:
>
> class ListSpec extends Specification {
>
> val list = ListBuffer[Int]()
>
> val oneAdded = beforeContext {list += 1}
> val twoAdded = beforeContext {list += 2}
>
> "a list" when oneAdded should {
>
> "have only 1 in it" in {
> list must_== ListBuffer(1)
> }
>
> "still have only 1 in it" in {
> list must_== ListBuffer(1)
> }
> }
>
> "a list" when twoAdded should {
>
> "have only 2 in it" in {
> list must_== ListBuffer(2)
> }
>
> "still have only 2 in it" in {
> list must_== ListBuffer(2)
> }
> }
> }
>
Doesn't the output when you run this specification read something like:
a list should
- have only 1 in it
- still have only 1 in it
a list should
- have only 2 in it
- still have only 2 in it
Using this approach do you not lose the "when one added" or "when two
added" kind of text?
Bill
----
Sukant Hajra
>
> a list should
> - have only 1 in it
> - still have only 1 in it
> a list should
> - have only 2 in it
> - still have only 2 in it
>
> Using this approach do you not lose the "when one added" or "when two
> added" kind of text?
post. My work-around is to use the string to Specs' "->-" operator
and put in the string description manually.
By the way, I'm still new to both Scala and Scala test frameworks, so
I haven't gotten through all the kinks like the one above.
On a related note. In my mind, I've always specualated that the BDD
framework I'd want would look something like this:
class ListSpec extends HypotheticalSpecification {
val list = ListBuffer[Int]()
"a list" >> {
"when 1 is added" >> {
list += 1
"should have only 1 in it" >> {
list must_== ListBuffer(1)
}
"should still have only 1 in it" >> {
list must_== ListBuffer(1)
}
}
when_2_is_added >> {
"should have only 2 in it" >> {
list must_== ListBuffer(2)
}
"should still have only 2 in it" >> {
list must_== ListBuffer(2)
}
}
}
The important part is that the ">>" method doesn't inject any natural
language into the textual output ("should," "given," "when," etc),
although some reflection would be needed to get the "when_2_is_added"
part (don't know how tricky that is). I like having all the natural
language in one place within the string or method name; it seems
cleaner and easier to read. Also, it gives me the flexibility to
manage the natural language exactly how I want to. The test framework
just gets out of the way and gives me the tools to manage test
execution.
So I think what looks like a bug in your question might actually be a
feature depending on one's perspective. But in Specs, the "system
under test" concept tightly coupled to the "should" method, which
pulls me away from the strategy above.
Again, I'm happy to have some audience for these ideas. Most people I
meet are less opinionated about testing frameworks.
-Sukant
Sukant Hajra
>
> a list should
> - have only 1 in it
> - still have only 1 in it
> a list should
> - have only 2 in it
> - still have only 2 in it
>
> Using this approach do you not lose the "when one added" or "when two
> added" kind of text?
post. My work-around is to use the string to Specs' "->-" operator
and put in the string description manually.
By the way, I'm still new to both Scala and Scala test frameworks, so
I haven't gotten through all the kinks like the one above.
On a related note. In my mind, I've always specualated that the BDD
framework I'd want would look something like this:
class ListSpec extends HypotheticalSpecification {
"a list" >> {
Bill Venners
Your HyptheticalSpecification looks similar to the SpecDasher style I
had in an earlier version of ScalaTest. I took it out because I felt
the operators made the code look ugly and it were very non-obvious.
Here's some ScalaDoc with an example:
http://www.artima.com/scalatest/doc-0.9.4/org/scalatest/SpecDasher.html
One difference is that it didn't do the isolation of nested scopes
that you like, and it also used dashes instead of the >> operator. I
think your example looks and reads very nicely, but I've seen some
pretty bad grammar and non-sentences made with the Specs >> operator.
I have tried to guide the sentence more, in the hopes of helping
people write documentation that reads well. But I've seen examples
where that failed too. For example the it(...) clause in a Spec should
be a sentence like it("should do something"), but people have just
ignored the it and said things like it("a completely unrelated,
non-sentence, stream-of-consciousness bit of text"). I found both
kinds of examples by searching the web for open source test classes.
I dropped SpecDasher quite a while ago, but had it in my mind that
someday I might add a "FreeSpec," which is like you're example,
because it is like free verse poetry. You can format your text any way
you like. (And you could write it in any language you like, which was
another use case I figured FreeSpec might have.)
I also would note that your nesting, despite having dropped "given"
and "then," is still a given/when/then kind of structure. Vigorous
writing is concise, and so it is nice to drop given and then since
they are not needed, but the good thing about having them is that they
guide the structure. So I am warming up to the idea of a GivenSpec
that does that, but not sure yet.
As far as isolation goes, what I'd like to do is make the kind of
isolation you like a trait you mix in, but that's tough to do. The
reason is that I essentially need to pass a "path" to the test to the
constructor, so that when a class is instantiated for the purpose of
running a particular test, it doesn't execute any blocks that don't
enclose the test. So I kind of need the test to have an auxiliary
constructor that takes a List[Int], and that's hard to do with a
trait. There are ways to do it, but they make the client code more
cumbersome. So far the easiest way on users that I have figured out to
do it is to just make GivenSpec a class. That burns the base class,
which is a pain, but the style you've shown is I think a style I'd
like ScalaTest to support, because some people like to write tests
that way. I'll keep thinking about it and see if I can't find some
decent way to get the benefits of traits and still be able to get the
isolation in a non-painful way.
I would ask one question, though. Why would you want the beforeContext
thing? I would think it would be better to put the list += 2 code
right in the body of the "when two is added" clause. Then it is
explicit. If you have code duplication, then you could factor that
duplicate code out into an addTwoToTheList, and call it explicitly.
Bill
> --
> You received this message because you are subscribed to the Google
> Groups "scalatest-users" group.
> To post to this group, send email to scalate...@googlegroups.com
> To unsubscribe from this group, send email to
> scalatest-use...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/scalatest-users?hl=en
> ScalaTest itself, and documentation, is available here:
> http://www.artima.com/scalatest
>
--
Sukant Hajra
I'm noticing my replies are double-posting. I apologize for that.
I'm submitting these replies through Google's web front-end, so I
suspect it's a bug on their side. In the meantime, I'm trying to be
very careful about what I click.
On Jul 26, 12:13 pm, Bill Venners <b...@artima.com> wrote:
>
> Your HyptheticalSpecification looks similar to the SpecDasher style I
> had in an earlier version of ScalaTest. I took it out because I felt
> the operators made the code look ugly and it were very non-obvious.
>
> pretty bad grammar and non-sentences made with the Specs >> operator.
> I have tried to guide the sentence more, in the hopes of helping
> people write documentation that reads well. But I've seen examples
> where that failed too. For example the it(...) clause in a Spec should
> be a sentence like it("should do something"), but people have just
> ignored the it and said things like it("a completely unrelated,
> non-sentence, stream-of-consciousness bit of text"). I found both
> kinds of examples by searching the web for open source test classes.
guidance from the framework, it's up to the developer to write
cohesive, comprehensible descriptions. Personally, I suspect this is
a higher-order skill than the programming itself. In light of this,
my biases lean towards the SpecDasher approach.
> I dropped SpecDasher quite a while ago, but had it in my mind that
> someday I might add a "FreeSpec," which is like you're example,
> because it is like free verse poetry. You can format your text any way
> you like. (And you could write it in any language you like, which was
> another use case I figured FreeSpec might have.)
if I'm clear on how this would look. You wouldn't have both "-" and
"--" in FreeSpec, right? Because the differentiation between the two
seems to be more about differentiating the SUS from the context, and I
would rather manage that "freely" through text without the test
framework forcing a structure on me.
Perhaps the difference between "-" and "--" also has to do with
helping the DSL manage formatting of the test description. My gut
feeling is that outer closure can be detected without the syntactic
demarcation, but maybe the implementation gets hacky.
> As far as isolation goes, what I'd like to do is make the kind of
> isolation you like a trait you mix in, but that's tough to do.
need to get my feet more wet in more Scala before the problems and
limitations are visceral. In the meantime, I'm enthusiastic that we
both have some interest in seeing if we can pull off test isolation in
some reasonable way. I'm working towards getting to a point where I
can make meaningful patches to Scala projects. I certainly would
prefer to keep ScalaTest trait-oriented if possible. For now, I'd
like to see if we can nail down what we'd like the system to look
like, then I'll have some concrete tests to make pass and I can get
down into something more low-level.
> I also would note that your nesting, despite having dropped "given"
> and "then," is still a given/when/then kind of structure. Vigorous
> writing is concise, and so it is nice to drop given and then since
> they are not needed, but the good thing about having them is that they
> guide the structure. So I am warming up to the idea of a GivenSpec
> that does that, but not sure yet.
nit-picking. You're right that my strategy is ultimately Dan North's
Given-When-Then (GWT) style. The tweaks I do are really subtle. One
justification you might appreciate is that GWT seems to put a more
imperative spin on the context and specification of a behavior. The
"given" is often a declarative setup of context with the "when" being
a side-effecting action. But if I'm working in a more immutable
system. The difference between the "given" and the "when" becomes a
lot weaker. Both are just calculations (say, Scala val assignments).
But really, I'm not trying to convince you of anything. . . just help
explain my inclinations.
Also, from the perspective of counting keystrokes, I just didn't see
much of a difference between
when("under some context") { ... }
and
"when under some context" { ... }
The important part for me is that as I read a "context path" from the
outside-in I get a clear and understandable message.
> I would ask one question, though. Why would you want the beforeContext
> thing? I would think it would be better to put the list += 2 code
> right in the body of the "when two is added" clause. Then it is
> explicit. If you have code duplication, then you could factor that
> duplicate code out into an addTwoToTheList, and call it explicitly.
brained idea. Let me see if I can elucidate the problem I really want
to focus on. Specs has an admittedly imperative way of managing
fixture and context setup and teardown. With my nesting strategy, I
can manage setup, but I haven't been explicitly clear on how to manage
teardown. I took some time to think deeply about the
org.scalatest.fixture package. The implicits required for the
MultipleFixture variants are too verbose for my tastes. But for the
simple Fixture traits I can definitely appreciate the way you're
trying to manage setup and teardown functionally. The only problem I
have is that with this approach the fixture needs to be this explicit
(typed) object. Whereas with the FreeSpec approach we've been
discussing, through the benefits of closures, I get everything I need
in scope without the type-check hassles of passing fixtures through
functions.
So ultimately, the real problem I want to focus on is teardown.
Here's a first shot at an idea:
class TearDownsAttempt1Spec
extends FreeSpec
with OneInstancePerTest
with ShouldMatchers {
def doAsserts(sus: System, ctx: Context) = {
String.format(
"then some our invariant holds for system %s,
context %s",
sus, ctx) -- {
sus forContext(ctx) holdsInvariant should equal true
}
}
"given a system under test" -- {
val sus = new System
sus.start()
"when using our first context" -- {
val ctx = new Context
ctx.init(1)
doAsserts(sys, ctx)
ctx.teardown()
}
"when using our second context" -- {
val ctx = new Context
ctx.init(2)
doAsserts(sys, ctx)
ctx.teardown()
}
sus.end()
}
}
Sorry, I stuffed two concepts into that example as I typed it: a
strategy for teardowns and a way to reuse assertions. I'm hoping this
idea for managing teardown is not suprising, given it's very
symmetrical to how setups are managed. For the reuse of assertions,
I'm borrowing an idea from Specs.
So in my mind, what the runner will do is iteratively run through all
the code in the specification class sequentially, selectively enabling
and disabling closures to assert test isolation -- but always running
from beginning to end until the last leaf-closure is executed (using a
tree-model of nested closures). The algorithm seems clear in my mind,
but I want to better understand how it meets with your sensibilities
and also the difficulties of implementation you described above.
Also, what I like about this "free/isolated" strategy is that I can
trust the runner to have a compact algorithm, albeit a little tricky
to assert isolation. This gives me the freedom to design my own
strategies for context and assertion reuse independent from the test
framework, perhaps in a way that's also sensitive to my domain.
What do you think? Am I straying too far from ScalaTest's core
philosophy? Or do you feel like I'm pursuing an ill-advised design?
-Sukant
Bill Venners
On Tue, Jul 27, 2010 at 10:24 AM, Sukant Hajra <a9h5m...@snkmail.com> wrote:
> Hi Bill,
>
> I'm noticing my replies are double-posting. I apologize for that.
> I'm submitting these replies through Google's web front-end, so I
> suspect it's a bug on their side. In the meantime, I'm trying to be
> very careful about what I click.
>
I only received one copy each time.
The differentiation is that currently tests are registered during
construction, but not executed. Everything else is executed during
construction. So for example in a Spec, describe clauses get executed,
but it clauses don't. In a FeatureSpec, feature clauses get executed
by scenario clauses don't. Well the scenario method does get called of
course, as does the it method in Spec, but what those methods do is
*register* the block of test code for later execution. The tests are
executed later when run is called on the Spec or FeatureSpec object.
Similar for WordSpec, the relatives in the fixture package, etc.
So the -- and - was differentiated so that the -- ones could be
executed at construction time and the - ones not. I would keep that,
but I am thinking of continuing to use "in" for the test marker to
reduce by half the number of non-obvious operators to just one. That
comes at the end of the phrase, so I think it isn't too in the way.
Also, "in" is used in exactly the same way in several other traits,
i.e., to mark a test, so that would also sometimes help people figure
out what it likely means. I'm not sure what symbol to use. Here's what
one would look like if we use the ~ symbol:
class ListSpec extends FreeSpec with NestedScopeIsolation with MustMatchers {
"a ListBuffer of Ints" ~ {
val list = ListBuffer[Int]()
"when one is added" ~ {
list += 1
"should have only 1 in it" in {
list must be === ListBuffer(1)
}
"and still have only 1 in it" in {
list must be === ListBuffer(1)
}
}
"when two is added" ~ {
list += 2
"should have only 2 in it" in {
list must be === ListBuffer(2)
}
"and still have only 2 in it" in {
list must be === ListBuffer(2)
}
}
}
}
I'm open to suggestions for what operator to use where I'm using the ~
in this example.
>> As far as isolation goes, what I'd like to do is make the kind of
>> isolation you like a trait you mix in, but that's tough to do.
>
> I think I followed more or less the difficulties you described. I'll
> need to get my feet more wet in more Scala before the problems and
> limitations are visceral. In the meantime, I'm enthusiastic that we
> both have some interest in seeing if we can pull off test isolation in
> some reasonable way. I'm working towards getting to a point where I
> can make meaningful patches to Scala projects. I certainly would
> prefer to keep ScalaTest trait-oriented if possible. For now, I'd
> like to see if we can nail down what we'd like the system to look
> like, then I'll have some concrete tests to make pass and I can get
> down into something more low-level.
>
I think I figured out a way to do it with traits. I haven't coded it
up yet so I may still realize I didn't imagine something, but baring
that it will be a trait. I'm not sure what to call the trait, which to
me is a sign that this is hard to describe to users. Basically when
you mix in NestedScopeIsolation or whatever I call this trait, each
test would be run in its own instance, and only surrounding blocks
will be executed. It complicates the execution model, but in return
people get the option of writing tests this way.
The way things work now is there's a "registration phase" and a "ready
phase". Here's what the ScalaDoc says for Spec, for example:
A Spec's lifecycle has two phases: the registration phase and the
ready phase. It starts in registration phase and enters ready phase
the first time run is called on it. It then remains in ready phase for
the remainder of its lifetime.
That's pretty simple and easy to understand. What it will need to change to is:
At construction time, only the top level of describe and it clauses
would get executed. These describe and it clauses would be registered
for later execution. As soon as any of methods tags, testNames, run,
expectedTestCount is executed, then full registration would take
place. Then it enters the ready phase like before.
I'd create a trait NestedScopes (not sure of the name) that has a
runTestsIsolated (also not sure of the name) method in it. Traits that
have nested scopes, like WordSpec, Spec, FeatureSpec, FreeSpec,
GivenSpec, would mix in this trait, but traits that don't have nested
scopes (FunSuite, Suite, FlatSpec) would not. RunTestsIsolated would
have a self type of NestedScopes, so it can only be mixed into a trait
that has nested scopes. (It makes no sense to mix NestedScopeIsolation
into FlatSpec, for example, because the whole point of FlatSpec is to
*not* have nesting.)
NestedScopeIsolation would extend OneInstancePerTest and override
runTests, and what it would do is create a new instance for each test,
but call runTestIsolated for each test instead of plain old runTest.
This is the method that would do partial registration, using info
passed into runTestIsolated indicating the location of the test to run
(i.e., the location would indicate which describe clauses surround the
single test to execute in this instance, so that it can run only those
surrounding describe clauses and no others).
So that's more complicated to understand, but I think I can explain it
in the ScalaDoc. And you'd have the visual trigger of "with
NestedScopeIsolation" if the tests are exhibiting this useful but
possibly surprising behavior.
>> I also would note that your nesting, despite having dropped "given"
>> and "then," is still a given/when/then kind of structure. Vigorous
>> writing is concise, and so it is nice to drop given and then since
>> they are not needed, but the good thing about having them is that they
>> guide the structure. So I am warming up to the idea of a GivenSpec
>> that does that, but not sure yet.
>
> Great. This part of our conversation seems a little bit of stylistic
> nit-picking. You're right that my strategy is ultimately Dan North's
> Given-When-Then (GWT) style. The tweaks I do are really subtle. One
> justification you might appreciate is that GWT seems to put a more
> imperative spin on the context and specification of a behavior. The
> "given" is often a declarative setup of context with the "when" being
> a side-effecting action. But if I'm working in a more immutable
> system. The difference between the "given" and the "when" becomes a
> lot weaker. Both are just calculations (say, Scala val assignments).
> But really, I'm not trying to convince you of anything. . . just help
> explain my inclinations.
>
> Also, from the perspective of counting keystrokes, I just didn't see
> much of a difference between
>
> when("under some context") { ... }
>
> and
>
> "when under some context" { ... }
>
> The important part for me is that as I read a "context path" from the
> outside-in I get a clear and understandable message.
>
I'm currently thinking of adding both a GivenSpec and a FreeSpec, so
people would have a choice. GivenSpec is more verbose, but not by
much, and it is also a good guide on how to structure your test, and
pretty clear to read. Though I may hold it back because no one has
actually asked for that one yet, whereas someone (you) is asking for
FreeSpec. I can always add it later. One concern I have is that
there's a GivenWhenThen trait that lets you put that kind of
documentation *inside* tests, pretty much any kind of style, and the
GivenSpec kind of clashes with that trait.
On that point, the same is true for nested scope isolation. Specs
didn't do that originally either, if I remember right. But a user
asked Eric, and he felt it made sense to do, so he added it, about a
year ago maybe. No one had asked for it in ScalaTest, and I think I
wasn't convinced anyway that it's such a good idea, because it is
non-obvious when looking at the code--until your post came along. Your
example actually made sense to me as to why people might want to write
tests this way. I don't know of any other test framework (besides
specs) that has done this kind of isolation. It would have to be in a
language with closures, but there are other languages with closures
besides Scala. So I'm surprised it hadn't popped up in those
language's test frameworks before.
I think I can let you do teardown in a way that's analogous to the way
you'll be doing setup. I think it should work that way, because it
would be symmetrical. So if you want to clear the list after each test
as a tear down, for example, it would look like this:
class ListSpec extends FreeSpec with NestedScopeIsolation with MustMatchers {
"a ListBuffer of Ints" ~ {
val list = ListBuffer[Int]()
"when one is added" ~ {
list += 1
"should have only 1 in it" in {
list must be === ListBuffer(1)
}
"and still have only 1 in it" in {
list must be === ListBuffer(1)
}
}
"when two is added" ~ {
list += 2
"should have only 2 in it" in {
list must be === ListBuffer(2)
}
"and still have only 2 in it" in {
list must be === ListBuffer(2)
}
}
list.clear()
}
}
So if you mix in NestedScopeIsolation, only code in surrounding scopes
will be executed for each test. Code that appears before a test in
surrounding scopes will be executed before the test. Code that appears
after the test in surrounding scopes will be executed after the test.
Bill
Sukant Hajra
>
> > Perhaps the difference between "-" and "--" also has to do with
> > helping the DSL manage formatting of the test description. My gut
> > feeling is that outer closure can be detected without the syntactic
> > demarcation, but maybe the implementation gets hacky.
>
> The differentiation is that currently tests are registered during
> construction, but not executed. Everything else is executed during
> construction. So for example in a Spec, describe clauses get executed,
> but it clauses don't. In a FeatureSpec, feature clauses get executed
> by scenario clauses don't. Well the scenario method does get called of
> course, as does the it method in Spec, but what those methods do is
> *register* the block of test code for later execution. The tests are
> executed later when run is called on the Spec or FeatureSpec object.
> Similar for WordSpec, the relatives in the fixture package, etc.
>
> So the -- and - was differentiated so that the -- ones could be
> executed at construction time and the - ones not. I would keep that,
> but I am thinking of continuing to use "in" for the test marker to
> reduce by half the number of non-obvious operators to just one. That
> comes at the end of the phrase, so I think it isn't too in the way.
> Also, "in" is used in exactly the same way in several other traits,
> i.e., to mark a test, so that would also sometimes help people figure
> out what it likely means. I'm not sure what symbol to use.
meantime, I am concerned about the lifecycle of "~" closures versus
"in" closures, especially with respect to the semantics of setup and
teardown we've discussed.
My worries are that what might be a non-issue for something like an in-
memory list might become awkward with something expensive like a
database operation. So consider the following specification using
your suggested "~"/"in" notation.
class ListSpec
extends FreeSpec
with NestedScopeIsolation
with MustMatchers {
"a repository of employees" ~ {
//2 TestDatabase.clear()
val repo = Repository[Employee]()
"when a new employee has been added" ~ {
val empName = "Sukant"
val emp = Person(name)
repo add emp
"has the employee that's been added" in {
repo findByName name must be === Some emp
}
"can delete the employee that's been added" in {
repo delete emp
repo findByName name must be === None
}
}
//3 TestDatabase.clear()
}
//4 TestDatabase.clear()
}
I quickly freehanded the test above, so pardon any grammatical
mistakes. In comments, I've indicated four places that we might clear
out the database to assert test isolation beyond the in-memory
isolation we get from the NestedScopeIsolation trait.
What's important for me is that the clear operation only happen
exactly twice, once for each assertion (appropriately before or after
the test as specified). I'd /really/ not like for the clear to happen
any more times. This is something I wasn't 100% sure Specs was
doing. I created some simple tests to better understand the closure
lifecycle, but my analysis is not yet done.
To satisfy all four suggested positions for the database clear, I have
an idea for a test runner that wouldn't construct the class until the
first test execution. That execution with the NestedScopeIsolation
trait mixed in would potentially have the first test execution spawn
off other "context paths" to traverse (using a simple work queue of
context paths). When I think about this approach, I'm reminded of the
kind of execution a tool like Java PathFinder employs. Paths aren't
registered apiori, but discovered at runtime. Also, a consequence of
this strategy is that there's no need algorithmically to differentiate
between "in" and "~" closures.
Unfortunately, this lifecycle for a specification object might be too
naive given all the use cases ScalaTest must support, or incompatible
with ScalaTest's current architecture.
Hopefully my description is clear. I'm tempted to code up a simple
runner to illustrate the idea when I have the time. But I certainly
don't want to maintain my own test framework; mostly it would be for
the practice of writing Scala. My intent is really to contribute back
to either ScalaTest or Specs. Or at the least, facilitate some
discussion. With ScalaTest, it seems we're starting with a greener
field, which is more interesting for me. With Specs, there's already
an implementation for most of the features I want, so I'd probably
continue to shoehorn my own testing style into what's there.
-Sukant
Bill Venners
I think tests written in a nested scope isolation can get confusing to
read, which is why I've been concerned about it. The problem is it is
non-obvious what's going on behind the scenes. It sounds like you are
writing little examples in specs to try and figure out the execution
model. I'm not sure what specs does with things after a test either.
My intuition is that it would execute them after the test, but I have
no idea how Eric has implemented his isolation. If it isn't describe
in the ScalaDoc, then I'd have to write little examples to discover
the execution model also.
By default in ScalaTest what you see is what you get. If it looks like
code in one closure will see the side effects by code in some other
closure that both reference the same mutable object, it does. Closures
work like closures. If you mix in OneInstancePerTest, then you get
isolation just between the tests. I think that's surprising
behaviour--I remember quite surprised when I found out JUnit worked
that way. But at least it was simple to understand. The nested scope
isolation is both surprising and confusing. But in the right hands, I
think it can make for declarative, readable tests.
In your case, basically it sounds like you want to clear the database
before and after each test. (Correct me if I misunderstood you.) Note
that if you just mixed in BeforeAndAfterEach, even though your code
would be more verbose, it would be very clear to readers what's going
on. That would look like this:
class ListSpec
extends FreeSpec
with BeforeAndAfterEach
with MustMatchers {
val repo = Repository[Employee]()
def beforeEach() {
TestDatabase.clear()
val empName = "Sukant"
val emp = Person(name)
repo add emp
}
def afterEach() {
TestDatabase.clear()
}
"a repository of employees" ~ {
"when a new employee has been added" ~ {
"has the employee that's been added" in {
repo findByName name must be === Some emp
}
"can delete the employee that's been added" in {
repo delete emp
repo findByName name must be === None
}
}
}
}
I know this isn't the style you prefer, but I want to point out how it
does serve the reader of the code by marking off explicitly what you
want to happen before each test and after. The nesting serves to
reduce duplication in specifying the *text* that describes the test,
not the test code itself.
With NestedScopeIsolation, you'd just put one clear before all tests
and one after. I'd put the one after in the same scope as the line of
code that first alters the database. I.e., the cleanup would be at the
same level as the "messing up" above it, like this:
class ListSpec
extends FreeSpec
with NestedScopeIsolation
with MustMatchers {
"a repository of employees" ~ {
val repo = Repository[Employee]()
"when a new employee has been added" ~ {
val empName = "Sukant"
val emp = Person(name)
TestDatabase.clear()
repo add emp
"has the employee that's been added" in {
repo findByName name must be === Some emp
}
"can delete the employee that's been added" in {
repo delete emp
repo findByName name must be === None
}
TestDatabase.clear()
}
}
}
Or if you felt it was clearer to just indicate you're starting with an
empty database and ending with one, you might do it this way:
class ListSpec
extends FreeSpec
with NestedScopeIsolation
with MustMatchers {
TestDatabase.clear()
"a repository of employees" ~ {
val repo = Repository[Employee]()
"when a new employee has been added" ~ {
val empName = "Sukant"
val emp = Person(name)
repo add emp
"has the employee that's been added" in {
repo findByName name must be === Some emp
}
"can delete the employee that's been added" in {
repo delete emp
repo findByName name must be === None
}
}
}
TestDatabase.clear()
}
On readability, I think the BeforeAndAfterEach example is easier for
readers of the code to figure out what's going on than the
NestedScopeIsolation ones. Many readers would probably need to go to
NestedScopeIsolation's ScalaDoc and figure out when things will be
executed and how they'll be isolated.
Were I to get rid of the "in", it would mean I couldn't know until
after I run a ~ clause if there were any nested ~ clauses in it. So
I'd have to run everything at construction time, basically, to find
out how many tests there are. ScalaTest has an expectedTestCount
method that needs to return how many tests should be coming. What's
nice about the "in" is that I can register tests without executing
them. Without that "in" I'd have to essentially execute tests when a
FreeSpec was constructed and in that second step of full registration,
then store the results, and send the results out when run is called.
Or just run them again when run is called. That's not impossible, but
again I think that's non-intuitive. Also, storing results would take
up memory, and on a large test suite that could be a problem. Running
tests twice could be a problem if some tests take a long time. Both of
those potential problems can be avoided if there's a specific marker
for a test like "in". Lastly, having a special marker for a test helps
communicate what block of code actually constitutes a test. That's why
I had -- and - in SpecDasher. - was the marker for tests. -- marked
descriptive clauses. But I'll think about it. I agree it does *look*
nicer to just use one symbol all the way down. In ScalaTest, at least,
if you had a problem with memory or execution time, you could switch
to something besides FreeSpec.
Bill
Sukant Hajra
>
> In your case, basically it sounds like you want to clear the database
> before and after each test. (Correct me if I misunderstood you.)
before the test or after. But that's a small clarification. Your
examples are still relevant.
> Note that if you just mixed in BeforeAndAfterEach, even though your code
> would be more verbose, it would be very clear to readers what's going on.
> That would look like this:
>
> does serve the reader of the code by marking off explicitly what you
> want to happen before each test and after. The nesting serves to
> reduce duplication in specifying the *text* that describes the test,
> not the test code itself.
implemented along these lines, there's doFirst, doLast, doBefore,
doAfter, doBeforeSpec, doAfterSpec, etc. So this is the other
extreme, and I feel I loose track of what the exact semantic is for
each of these. Ultimately, I just like having the option to do what
makes sense given the context of the domain and the specification I'm
working with. But perhaps that flexibility comes at a cost on the
framekwork.
> class ListSpec
> extends FreeSpec
> with NestedScopeIsolation
> with MustMatchers {
>
> TestDatabase.clear()
>
> "a repository of employees" ~ {
>
> val repo = Repository[Employee]()
>
> "when a new employee has been added" ~ {
>
> val empName = "Sukant"
> val emp = Person(name)
> repo add emp
>
> "has the employee that's been added" in {
> repo findByName name must be === Some emp
> }
>
> "can delete the employee that's been added" in {
> repo delete emp
> repo findByName name must be === None
> }
> }
> }
>
> TestDatabase.clear()
>
> }
>
> after I run a ~ clause if there were any nested ~ clauses in it. So
> I'd have to run everything at construction time, basically, to find
> out how many tests there are. ScalaTest has an expectedTestCount
> method that needs to return how many tests should be coming. What's
> nice about the "in" is that I can register tests without executing
> them.
outside the "in" closure get executed (or not) while you're
registering a expectedTestCount of the "in" closures.
If they don't get executed, what Scala feature are you using to count
the "in" closures without executing anything? As far as I can tell
from the example I've excerpted above, simply instantiating the
ListSpec will clear the database twice, even if you choose not to
execute the outer-most "~" closure.
> Lastly, having a special marker for a test helps
> communicate what block of code actually constitutes a test. That's why
> I had -- and - in SpecDasher. - was the marker for tests. -- marked
> descriptive clauses. But I'll think about it. I agree it does *look*
> nicer to just use one symbol all the way down.
think my argument against "in" really isn't so strong. Mostly, I just
want to make sure the lifecycle of the specification's closures are
sane, which I think is your primary concern too (along with
readability of the specification).
-Sukant
Bill Venners
Sorry for the delay in responding to your last email. I was busy
getting a few things out the door, including teaching a Scala class.
That behind me I'd like to pick up this discussion again. I've also
CC'd Esko Luontola, who was the first person to suggest this kind of
test isolation on the specs mailing list a little over a year ago. He
recently released a specsy framework that does the isolation the way
he likes it, and also uses the same >> character all the way down with
unlimited nesting--it sounds similar to what you've been sketching
out. Esko, I wonder if you'd mind joining the scalatest-users mailing
list so we can get your opinions on this too.
At the bottom of your most recent email, Sukant, you asked about why I
wanted a special marker for a leaf node. The reason is so that those
nodes can be registered for later execution, whereas the surrounding
nodes would be executed immediately. An example from Dasher could
clarify:
"this is a descriptive clause" -- {
"this is another descriptive clause" -- {
"but this marks a test" - {
}
"and this marks another test" - {
}
}
}
This was syntax sugar for:
describe("this is a descriptive clause") {
describe("this is another descriptive clause") {
it("but this marks a test") {
}
it("and this marks another test") {
}
}
}
When the Spec is constructed all the describe (or --) clauses are
executed at that time. I.e., the code between the curly braces is
passed up as a function to describe, and describe immediately executes
it. But when you get to an it clause (or -), the code between those
curly braces is passed to it and *not* executed by the it method, but
just registered for later execution. This allows tests to be counted
up front, but not executed until later when run() is called.
When the same marker is used all the way down, the only way to find
the leaf nodes is to execute the code of all nodes. So that means all
the code either gets executed when the Spec is constructed, or perhaps
when expectedTestCount is invoked. The results of that test execution
could be stored and reported later when run is called, so it is
possible, but surprising. If tests have side effects, users might be
very surprised about when they happen. But nested scope isolation also
yields surprising timing of test execution, so maybe it isn't so bad.
That would make tests look like both you and Esko envision:
"this is a descriptive clause because it isn't a leaf node" ~ {
"this is another descriptive clause because it isn't a leaf node" ~ {
"but this is a test because it is a leaf node" ~ {
}
"and this is another test because it is a leaf node" ~ {
}
}
}
In ScalaTest, a Suite (or Spec) is a collection of tests, each of
which has a unique string name. The test name for a FreeSpec would be
composed of the concatenation of all the strings in surrounding
clauses, one for each leaf node. This allows the test to be run
through JUnit, for example. But when you run it through ScalaTest,
you'd see a nice spec-like output.
But I have one question for both of you, Sukant and Esko: would you
ever want to be able to put assertions in non-leaf nodes? I looked at
the examples both of you have written, and in none of them do I see
something like this:
"outer" ~ {
"inner" ~ {
assert(something here)
"leaf 1" ~ {
//...
}
}
}
I have only seen assertions in leaf nodes in your examples. The reason
I ask is that an assertion would need to be inside a test, not in
descriptive clauses. You could set up fixture data in the descriptive
clauses, and nested scope isolation would make things nice, but if you
put an assertion in there that failed, then there really isn't a test
to report as failed given the current way I'm thinking of mapping
FreeSpec code to tests.
In short, would it be OK with you two if there was a rule in FreeSpec
that you could only put assertions in leaf nodes, and that the
surrounding nodes should be for setting up (and tearing down) fixture
data?
Thanks.
Bill
Esko Luontola
On Jul 26, 8:13 pm, Bill Venners <b...@artima.com> wrote:
> As far as isolation goes, what I'd like to do is make the kind of
> isolation you like a trait you mix in, but that's tough to do. The
> reason is that I essentially need to pass a "path" to the test to the
> constructor, so that when a class is instantiated for the purpose of
> running a particular test, it doesn't execute any blocks that don't
> enclose the test. So I kind of need the test to have an auxiliary
> constructor that takes a List[Int], and that's hard to do with a
> trait. There are ways to do it, but they make the client code more
> cumbersome.
The way I do it in Specsy is to pass the parameter through a
ThreadLocal. (CGLIB does it the same way for the proxies it generates.)
That way the client code is not affected.
The code which instantiates the test uses ContextDealer.prepare() to
pass the context to the test:
http://github.com/orfjackal/specsy/blob/master/src/main/scala/net/orfjackal/specsy/core/SpecClassRunner.scala#L12-14
http://github.com/orfjackal/specsy/blob/master/src/main/scala/net/orfjackal/specsy/core/ContextDealer.scala
And the test gets the context like this:
http://github.com/orfjackal/specsy/blob/master/src/main/scala/net/orfjackal/specsy/Spec.scala#L6
On Jul 28, 7:26 pm, Bill Venners <b...@artima.com> wrote:
> The way things work now is there's a "registration phase" and a "ready
> phase". Here's what the ScalaDoc says for Spec, for example:
>
> A Spec's lifecycle has two phases: the registration phase and the
> ready phase. It starts in registration phase and enters ready phase
> the first time run is called on it. It then remains in ready phase for
> the remainder of its lifetime.
>
> That's pretty simple and easy to understand. What it will need to
change to is:
>
> At construction time, only the top level of describe and it clauses
> would get executed. These describe and it clauses would be registered
> for later execution. As soon as any of methods tags, testNames, run,
> expectedTestCount is executed, then full registration would take
> place. Then it enters the ready phase like before.
Specsy has no "phases". All of it is just test execution. The test class
(or its constructor) actually behaves more like a function. This can be
better seen in GoSpec where the specs are functions and there is less
syntactic sugar:
http://github.com/orfjackal/gospec/blob/gospec-1.2.0/examples/stack_test.go
Altough this means that the framework won't known which tests there are,
until it has executed all tests. It doesn't play well with JUnit's test
runner, which makes the implicit assumption that all tests are known
beforehand. So I've had to create a JUnit runner which executes all
tests the first time either run() or getDescription() is called:
Also what the tests print to stdout/stderr needs to be captured and
printed later when JUnit thinks the tests are being executed.
> NestedScopeIsolation would extend OneInstancePerTest and override
> runTests, and what it would do is create a new instance for each test,
> but call runTestIsolated for each test instead of plain old runTest.
> This is the method that would do partial registration, using info
> passed into runTestIsolated indicating the location of the test to run
> (i.e., the location would indicate which describe clauses surround the
> single test to execute in this instance, so that it can run only those
> surrounding describe clauses and no others).
Actually the reason why I didn't use the test runner from ScalaTest,
Specs or TestNG, is because they require the test classes to extend a
specific base class/trait, so that the testing framework cannot fully
control how the test class is instantiated. Also I require it to be
possible to use multiple test frameworks in the same project and run
them all at the same time.
JUnit was the only one that made it possible, but because it doesn't
handle well nested tests and frameworks which do not know all tests
beforehand, nor does it have built-in support for parallel execution,
I'll create a new test runner at some point. I'll contact all testing
framework, IDE, CI server and build tool developers later this year, to
get some feedback on what is required from a test runner that can
execute all testing frameworks on the JVM. What do you think, would
CTR4J (Common Test Runner for Java) be a good name, or do you have some
other ideas?
> I don't know of any other test framework (besides
> specs) that has done this kind of isolation. It would have to be in a
> language with closures, but there are other languages with closures
> besides Scala. So I'm surprised it hadn't popped up in those
> language's test frameworks before.
The first framework I used was JDave (which in turn was probably
influenced by JUnit and RSpec), so my style of writing tests developed
to be something that runs on JDave. It has the same concepts of
isolation and nested tests, although it requires the use of create()
methods and the level of nesting is fixed.
http://www.jdave.org/examples.html
Also I think that RSpec has the same isolation model and unlimited
nesting, although it requires you to write before(:each) and describe
blocks. If you take out all the boilerplate, you will end up with
something like Specsy.
> I think I can let you do teardown in a way that's analogous to the way
> you'll be doing setup. I think it should work that way, because it
> would be symmetrical. So if you want to clear the list after each test
> as a tear down, for example, it would look like this:
>
> class ListSpec extends FreeSpec with NestedScopeIsolation with
MustMatchers {
>
> "a ListBuffer of Ints" ~ {
>
> val list = ListBuffer[Int]()
>
> "when one is added" ~ {
> ...
> }
>
> "when two is added" ~ {
> ...
> }
> list.clear()
> }
>
> }
>
> So if you mix in NestedScopeIsolation, only code in surrounding scopes
> will be executed for each test. Code that appears before a test in
> surrounding scopes will be executed before the test. Code that appears
> after the test in surrounding scopes will be executed after the test.
Specsy actually works that way, but it's encouraged to use the defer
blocks instead (http://github.com/orfjackal/specsy#readme), because
otherwise:
- The corresponding setup and teardown code are separated, so it's
harder to see whether they are in sync. When updating setup you might
forget to update teardown because the teardown is not visible.
- If there are multiple steps in the teardown process, if one of them
fails then the rest of them are not executed.
- The user needs to manually specify the order of the teardown to match
the order of the setup. Usually the teardown needs to be done in the
reverse order than it was setup.
Bill Venners wrote on 23.8.2010 9:49:
> In ScalaTest, a Suite (or Spec) is a collection of tests, each of
> which has a unique string name. The test name for a FreeSpec would be
> composed of the concatenation of all the strings in surrounding
> clauses, one for each leaf node. This allows the test to be run
> through JUnit, for example. But when you run it through ScalaTest,
> you'd see a nice spec-like output.
Specsy identifies the tests by their position in the tree of tests
(http://github.com/orfjackal/specsy/blob/master/src/main/scala/net/orfjackal/specsy/core/Path.scala).
So it's possible for different tests to have the same name.
I haven't implemented support for running a single test from a test
class. It should be possible, but I'm not sure whether it's worth the
additional complexity (now Specsy is nice and small - the source code is
570 LOC, or 320 LOC if you exclude the test runner which will be
replaced by CTR4J). I don't have need for executing single tests,
because my tests are fast enough to execute all tests at the same time.
> But I have one question for both of you, Sukant and Esko: would you
> ever want to be able to put assertions in non-leaf nodes?
Yes. If there is complex setup code, then I may make some assertions
about it to make sure that the test is in a known state. In
http://github.com/orfjackal/specsy#readme both FibonacciSpec and
DeferBlocksExampleSpec/DeferBlocksExample2Spec do assertions in setup.
The @Before methods in these classes also have assertions:
http://github.com/orfjackal/tdd-tetris-tutorial/blob/beyond/src/test/java/tetris/RotatingAFallingPieceTest.java
http://github.com/orfjackal/tdd-tetris-tutorial/blob/beyond/src/test/java/tetris/MovingAFallingPieceTest.java
Most of the time I don't need assertions there - having to assert
something about the setup is a test smell that the setup is too complex
- but every now and then it is needed.
--
Esko Luontola
www.orfjackal.net
Esko Luontola
On Jul 26, 6:03 am, Sukant Hajra <a9h5mpz...@snkmail.com> wrote:
> On a related note. In my mind, I've always specualated that the BDD
> framework I'd want would look something like this:
>
> class ListSpec extends HypotheticalSpecification {
I hadn't seen your message before, but as they say, "great minds think
alike". :) You could almost copy and paste your code into Specsy and it
would work:
> The important part is that the ">>" method doesn't inject any natural
> language into the textual output ("should," "given," "when," etc),
> although some reflection would be needed to get the "when_2_is_added"
> part (don't know how tricky that is). I like having all the natural
> language in one place within the string or method name; it seems
> cleaner and easier to read. Also, it gives me the flexibility to
> manage the natural language exactly how I want to. The test framework
> just gets out of the way and gives me the tools to manage test
> execution.
I ranted about that recently:
http://blog.orfjackal.net/2010/05/choice-of-words-in-testing-frameworks.html
On Jul 27, 8:24 pm, Sukant Hajra <a9h5mpz...@snkmail.com> wrote:
> So in my mind, what the runner will do is iteratively run through all
> the code in the specification class sequentially, selectively enabling
> and disabling closures to assert test isolation -- but always running
> from beginning to end until the last leaf-closure is executed (using a
> tree-model of nested closures). The algorithm seems clear in my mind,
> but I want to better understand how it meets with your sensibilities
> and also the difficulties of implementation you described above.
Yup.
The logic for actually implementing that is somewhat complex, so when
creating Specsy I directly copied the conditionals from GoSpec where I
had implemented the same algorithm before:
http://github.com/orfjackal/gospec/blob/release/src/gospec/context.go#L79-88
http://github.com/orfjackal/specsy/blob/master/src/main/scala/net/orfjackal/specsy/core/SpecDeclaration.scala#L31-39
If you are willing to discard the possibility of parallel execution,
then it can be implemented in a very simple way by relying on a shared
data structure, which keeps track of the branches where all leaf tests
have been executed:
http://github.com/orfjackal/nanospec.go/blob/release/src/context.go
(search for "ShouldExecute")
On Jul 28, 9:46 pm, Sukant Hajra <a9h5mpz...@snkmail.com> wrote:
> To satisfy all four suggested positions for the database clear, I have
> an idea for a test runner that wouldn't construct the class until the
> first test execution. That execution with the NestedScopeIsolation
> trait mixed in would potentially have the first test execution spawn
> off other "context paths" to traverse (using a simple work queue of
> context paths). When I think about this approach, I'm reminded of the
> kind of execution a tool like Java PathFinder employs. Paths aren't
> registered apiori, but discovered at runtime. Also, a consequence of
> this strategy is that there's no need algorithmically to differentiate
> between "in" and "~" closures.
Yes. I even use the same name, "path". :)
http://github.com/orfjackal/specsy/blob/master/src/main/scala/net/orfjackal/specsy/core/Path.scala
http://github.com/orfjackal/gospec/blob/release/src/gospec/specification.go#L80
On Jul 28, 11:34 pm, Sukant Hajra <a9h5mpz...@snkmail.com> wrote:
> On this point, I have to agree. But when I look at what Eric's
> implemented along these lines, there's doFirst, doLast, doBefore,
> doAfter, doBeforeSpec, doAfterSpec, etc. So this is the other
> extreme, and I feel I loose track of what the exact semantic is for
> each of these. Ultimately, I just like having the option to do what
> makes sense given the context of the domain and the specification I'm
> working with. But perhaps that flexibility comes at a cost on the
> framekwork.
I agree. I've also encountered bugs in the doAfter/doLast/doAfterSpec
constructs, so that actually none of them was executed every time after
a test.
In Specsy I borrowed the "defer" language construct from the Go
programming language, which is even more expressive than the typical
doAfter - you can have multiple defer blocks and they are executed in
LIFO order. See "Before" and "After" Blocks at
http://github.com/orfjackal/specsy#readme
I've decided to not implement the doBeforeSpec and doAfterSpec concepts
in Specsy, because those are mainly used to speed up slow integration
tests, and I don't write many integration tests
(http://www.infoq.com/presentations/integration-tests-scam). Also it
would make the framework's test runner more complex. For those few
end-to-end tests that I write (following the advice in the GOOS book:
http://blog.orfjackal.net/2010/07/design-for-integrability.html), I can
use some other testing framework, although even then I prefer to have
every test do a full setup and teardown to avoid leaking side-effects
between tests.
--
Esko Luontola
www.orfjackal.net
Bill Venners
On Mon, Aug 23, 2010 at 5:43 AM, Esko Luontola <esko.l...@gmail.com> wrote:
> Actually the reason why I didn't use the test runner from ScalaTest, Specs
> or TestNG, is because they require the test classes to extend a specific
> base class/trait, so that the testing framework cannot fully control how the
> test class is instantiated. Also I require it to be possible to use multiple
> test frameworks in the same project and run them all at the same time.
>
I think you should take a closer look at ScalaTest. Trait Suite
defines methods such as run, runTests, runNestedSuites, testNames,
runTest, withFixture, nestedSuites, expectedTestCount, etc., all of
which can be overridden to define new ways of expressing and running
tests. The reason I can implement nested scope isolation as a mix in
is because the NestedScopeIsolation trait can override one or more of
these methods and add that behavior. Same for a FreeSpec, which is
basically I designed ScalaTest to facilitate different styles of
testing, and this is one it can support.
> JUnit was the only one that made it possible, but because it doesn't handle
> well nested tests and frameworks which do not know all tests beforehand, nor
> does it have built-in support for parallel execution, I'll create a new test
> runner at some point. I'll contact all testing framework, IDE, CI server and
> build tool developers later this year, to get some feedback on what is
> required from a test runner that can execute all testing frameworks on the
> JVM. What do you think, would CTR4J (Common Test Runner for Java) be a good
> name, or do you have some other ideas?
>
I think for better or for worse, JUnit is the common test runner. And
I don't think there's a problem getting this style to run through it.
Though, the output is ugly. But if you're looking for a more universal
runner that also understands specification-style output, please have a
look again at ScalaTest. It has a very general, well defined model for
running and reporting of tests of any style. Test events are fired
through a Reporter, which has a method infoProvided. infoProvided (and
other events) can include an optional Formatter, which has two
subtypes MotionToSuppress and IndentedText. MotionToSuppress indicates
you'd like the event to not show up in reports, and IndentedText
indicates both an indentation level/raw string combination and has a
formattedText string for text output (which would probably put spaces
in front for indentation, maybe with a bullet character, etc.) It is
very general.
That makes sense. Already in my example that I sent to Sukant I felt
it was difficult to be sure I was putting the teardown code in the
correct block. Not sure about defer yet, but having some other option
that allows users to specify teardown code in this style seems useful.
>> But I have one question for both of you, Sukant and Esko: would you
>> ever want to be able to put assertions in non-leaf nodes?
>
> Yes. If there is complex setup code, then I may make some assertions about
> it to make sure that the test is in a known state. In
> http://github.com/orfjackal/specsy#readme both FibonacciSpec and
> DeferBlocksExampleSpec/DeferBlocksExample2Spec do assertions in setup.
>
> The @Before methods in these classes also have assertions:
>
> http://github.com/orfjackal/tdd-tetris-tutorial/blob/beyond/src/test/java/tetris/RotatingAFallingPieceTest.java
> http://github.com/orfjackal/tdd-tetris-tutorial/blob/beyond/src/test/java/tetris/MovingAFallingPieceTest.java
>
> Most of the time I don't need assertions there - having to assert something
> about the setup is a test smell that the setup is too complex - but every
> now and then it is needed.
>
OK. What you implemented sounds about like what I imagined I'd need to
do in FreeSpec, however, there are some choices. If you have a
different marker for leaf nodes than the surrounding nodes, you can
defer the execution of those leaf nodes until the run method is
called. That's what I was pushing back on Sukant with. So instead of:
"outer" - {
"inner" - {
"leaf node" - {
}
}
}
It would be something like:
"outer" - {
"inner" - {
"leaf node" in {
}
}
}
Sukant actually said he wouldn't mind that. But then the other thing
is what if someone puts an assert outside the in. Well I could have a
rule in FreeSpec that says your assertions have to go in leaf nodes,
because those are the "tests". But I'm not sure that sounds like
"freedom." Maybe freedom with responsibility.
Another way to go is to execute all the code the first time I need to,
such as when expectedTestCount or run is invoked, store the results
(the events), and fire them to the reporter when run is called later.
(If run is called a second time I'd run them again though. And if an
assertion goes off in a non-leaf node, I'd just report the test as
having the name up to that point. So for example:
"outer" - {
"inner" - {
assert(false)
"leaf node" - {
assert(false)
}
}
}
When that assertion goes off, the name of the failed test would be
reported as "outer inner", even though there's no leaf node in it. If
you fix that one:
"outer" - {
"inner" - {
assert(true)
"leaf node" - {
assert(false)
}
}
}
You'd get a failure for test "outer inner leaf node." Now one thing
that could happen is you end up with a duplicate test exception in
such a case. If you had this:
"outer" - {
"inner" - {
assert(false)
"leaf node" - {
assert(false)
}
}
"inner" - {
}
}
If these tests were to all run and pass, you'd get two tests with the
names "outer inner leaf node" and "outer inner", which works fine
because they are unique names. But since the first non-leaf test that
failed will be recorded with the name "outer inner" here, the second
test will be a duplicate. My feeling here is that this would be
extremely rare, and so long as I report the first error so it can be
fixed, the second one can just be a duplicate test name exception
until the first error is fixed.
Now on the subject of defer, it is pretty to look at, but I think it i
non-obvious. You'd have to learn what defer means by looking at the
documentation of FreeSpec, and my goal is that the meaning ScalaTest
code be obvious without looking at the documentation. The other thing
is it requires reassigning a var or mutating an object to associate
the defer function with the block.
How would you feel if I don't add any feature to support this, but
simply document there are two ways to do the teardown. One is to put
it at the end of the block:
"a directory" - {
val dir = new File("myFile.txt")
try {
"can be created with mkDir" - {
dir.mkDir() should be (true)
}
"should exist after being created" - {
dir.exists should be (true)
}
}
finally {
dir.delete()
}
}
The Scala way to improve this situation is using the loan pattern, and
that would be my recommended alternative:
def withDir(dir: File)(f: => ()) {
try { f } finally { dir.delete() }
}
"a directory" - {
val dir = new File("myFile.txt")
withDir(dir) {
"can be created with mkDir" - {
dir.mkDir() should be (true)
}
"should exist after being created" - {
dir.exists should be (true)
}
}
}
Now there's no reason to go to the documentation to understand the
code. It is jut the loan pattern, and you can follow the code. It is
right in front of you. There's also no mutation needed behind the
scenes, so it is more functional, less behind-the-scenes magical, and
therefore in my opinion, easier to understand.
Though admittedly, it is more verbose to write. I'm often faced with
trading off readability and writability, and I tend to lean towards
readability. But writability is important too. Maybe I offer all three
ways so users have a choice. (And I wonder if there's a name for
defer, like afterBlock or afterScope or something that makes it more
obvious when it is being deferred until.)
"a directory" - {
val dir = new File("myFile.txt")
afterBlock {
dir.delete()
}
"can be created with mkDir" - {
dir.mkDir() should be (true)
}
"should exist after being created" - {
dir.exists should be (true)
}
}
Not as pretty as defer. Maybe just "after."
The only other thing I'd mention is that I don't think nested scope
isolation is a good default, because it is a surprising execution
model. By default ScalaTest does nothing special to isolate tests,
which means what you see is what you get. If you want nested scope
isolation, you'll need to mix in NestedScopeIsolation (though I'm not
happy with that name, so if you have a better name idea, please
suggest it). But I think you and Sukant would get what you desire
with:
class MySpec extends FreeSpec with NestedScopeIsolation {
"a directory" - {
val dir = new File("myFile.txt")
after {
dir.delete()
}
"can be created with mkDir" - {
dir.mkDir() should be (true)
}
"should exist after being created" - {
dir.exists should be (true)
}
}
}
Bill
Esko Luontola
> I think you should take a closer look at ScalaTest. Trait Suite
> defines methods such as run, runTests, runNestedSuites, testNames,
> runTest, withFixture, nestedSuites, expectedTestCount, etc., all of
> which can be overridden to define new ways of expressing and running
> tests. The reason I can implement nested scope isolation as a mix in
> is because the NestedScopeIsolation trait can override one or more of
> these methods and add that behavior. Same for a FreeSpec, which is
> basically I designed ScalaTest to facilitate different styles of
> testing, and this is one it can support.
But to be able to call the methods on Suite, the test class which
extends Suite must be instantiated, right? So the custom test runner
(which is the same as the test class) cannot execute code before the
test class is instantiated.
I forgot to mention, but there is a stronger reason why anything written
in Scala cannot be used as a common test runner: Scala programs are not
binary compatible between different Scala version (IntelliJ IDEA's Specs
or ScalaTest integration has blown up on me multiple times in the past
when it was written in Scala - I think now it's all in Java). Also
scala-library.jar is a rather big dependency to be included in every
build tool, IDE, CI server etc. Writing CTR4J in Java is the only option.
One of my goals is to reduce the duplication between all build tools,
IDEs and CI servers. For example IntelliJ IDEA has its own test runner
integration for JUnit, Specs, ScalaTest, TestNG etc. and all with
varying levels of quality and features. Also there are variations
between how Eclipse and IDEA run the tests, due to IDE specific bugs and
assumptions - I want zero bugs, especially in my test infrastructure.
> The only other thing I'd mention is that I don't think nested scope
> isolation is a good default, because it is a surprising execution
> model.
For those coming from a JUnit background, not isolating the tests is
surprising.
--
Esko Luontola
www.orfjackal.net
Bill Venners
On Mon, Aug 23, 2010 at 1:03 PM, Esko Luontola <esko.l...@gmail.com> wrote:
> Bill Venners wrote on 23.8.2010 20:41:
>>
>> I think you should take a closer look at ScalaTest. Trait Suite
>> defines methods such as run, runTests, runNestedSuites, testNames,
>> runTest, withFixture, nestedSuites, expectedTestCount, etc., all of
>> which can be overridden to define new ways of expressing and running
>> tests. The reason I can implement nested scope isolation as a mix in
>> is because the NestedScopeIsolation trait can override one or more of
>> these methods and add that behavior. Same for a FreeSpec, which is
>> basically I designed ScalaTest to facilitate different styles of
>> testing, and this is one it can support.
>
> But to be able to call the methods on Suite, the test class which extends
> Suite must be instantiated, right? So the custom test runner (which is the
> same as the test class) cannot execute code before the test class is
> instantiated.
>
If scala-library.jar is a problem then ScalaTest is out of the
"running" for sure. But I actually don't understand what you mean
here, and would like to. Can you clarify by giving a specific example
of code you would want to "execute before the test class is
instantiated?"
> I forgot to mention, but there is a stronger reason why anything written in
> Scala cannot be used as a common test runner: Scala programs are not binary
> compatible between different Scala version (IntelliJ IDEA's Specs or
> ScalaTest integration has blown up on me multiple times in the past when it
> was written in Scala - I think now it's all in Java). Also scala-library.jar
> is a rather big dependency to be included in every build tool, IDE, CI
> server etc. Writing CTR4J in Java is the only option.
>
> One of my goals is to reduce the duplication between all build tools, IDEs
> and CI servers. For example IntelliJ IDEA has its own test runner
> integration for JUnit, Specs, ScalaTest, TestNG etc. and all with varying
> levels of quality and features. Also there are variations between how
> Eclipse and IDEA run the tests, due to IDE specific bugs and assumptions - I
> want zero bugs, especially in my test infrastructure.
>
>
>> The only other thing I'd mention is that I don't think nested scope
>> isolation is a good default, because it is a surprising execution
>> model.
>
> For those coming from a JUnit background, not isolating the tests is
> surprising.
>
True, which kind of says that no matter what the default, it will
likely surprise some users. My feeling is that the default should be
what-you-see-is-what-you-get, with minimal or no behind-the-scenes
fancy behavior.
Since my last email I realized that executing the tests on
construction or when expectedTestCount is invoked breaks the promise
of BeforeAndAfterAll methods, overriding runTests, run, etc. These are
ways to do things before and after all tests are run. So if one of
those things is setting up a database, for example, then that wouldn't
work in a FreeSpec. So I think FreeSpec will need to differentiate
between description and test clauses, perhaps like I showed Sukant:
"a directory" - {
val dir = new File("myFile.txt")
after {
dir.delete()
}
"can be created with mkDir" in {
dir.mkDir() should be (true)
}
"should exist after being created" in {
dir.exists should be (true)
}
}
Leaf nodes get marked with an "in". Surrounding nodes are marked with
"-", and can be nested indefinitely. Code inside a description ("-")
clause get executed at class instantiation time, and code inside test
("in") clauses get executed only when run is invoked.
I would probably recommend people avoid putting assertions in
description clauses, but it wouldn't be the end of the world. These
would probably be rare, and would fail even more rarely. When they
fail they will be reported as a SuiteAborted not a TestFailed, but
will give sufficient information to go fix the problem. So I think
that's fine.
I have a new question about after. Should multiple after clauses be
allowed in a block, and if so, what order are they executed on exit?
I'm guessing yes, and last registered first executed (i.e. executed in
reverse order of appearance). Is that what you do in Specsy? This is
another example of how such code would not be obvious to casual
readers without reading the documentation, but again, it would likely
be rare and could be well documented.
Thanks.
Bill
----
> --
> Esko Luontola
> www.orfjackal.net
>
> --
> You received this message because you are subscribed to the Google
> Groups "scalatest-users" group.
> To post to this group, send email to scalate...@googlegroups.com
> To unsubscribe from this group, send email to
> scalatest-use...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/scalatest-users?hl=en
> ScalaTest itself, and documentation, is available here:
> http://www.artima.com/scalatest
>
--
Esko Luontola
> If scala-library.jar is a problem then ScalaTest is out of the
> "running" for sure. But I actually don't understand what you mean
> here, and would like to. Can you clarify by giving a specific example
> of code you would want to "execute before the test class is
> instantiated?"
There's this line which passes the context to the test, so that the test
will know that which tests to execute on the current test run, and the
initialization in the Context.bootstrap() method:
http://github.com/orfjackal/specsy/blob/master/src/main/scala/net/orfjackal/specsy/core/SpecClassRunner.scala#L12
It would be possible to detect when the class is instantiated the first
time by the framework (when no context has been prepared), but having to
deal with that special case would complicate the code needlessly. It
leads to a cleaner design when the test runner is separated from the
test class. Now calling the root spec (the test code in the constructor)
can be handled the same way as calling the nested specs - in
http://github.com/orfjackal/specsy/blob/master/src/main/scala/net/orfjackal/specsy/core/Context.scala#L18-42
the bootstrap() method is for the root spec and the specify() method for
the nested specs - the only differences relate to changing the running
status.
Probably also writing tests for the testing framework code would be
harder, if the test runner would be coupled with the test class. Now I
can test the framework by just passing closures to the test runner (for
example
http://github.com/orfjackal/specsy/blob/master/src/test/scala/net/orfjackal/specsy/ExecutionModelTest.scala)
instead of having to creating a dummy class whose constructor contains
the test code. There are only a couple of tests (for the JUnit runner
and the code itself which instantiates the classes) which require
concrete dummy classes as test data.
I suppose it basically boils down to this:
http://michaelfeathers.typepad.com/michael_feathers_blog/2010/08/testng-and-what-wed-like-code-to-be.html
> I have a new question about after. Should multiple after clauses be
> allowed in a block, and if so, what order are they executed on exit?
> I'm guessing yes, and last registered first executed (i.e. executed in
> reverse order of appearance). Is that what you do in Specsy?
Yes, that's how I do it. See the chapter "Before" and "After" Blocks in
http://github.com/orfjackal/specsy#readme
> This is
> another example of how such code would not be obvious to casual
> readers without reading the documentation, but again, it would likely
> be rare and could be well documented.
Specsy's documentation (http://github.com/orfjackal/specsy#readme) is
only about 4 pages long and it consists mostly of code examples (the
most valuable kind of documentation). I'm expecting everybody to read it
before using the framework.
--
Esko Luontola
www.orfjackal.net