BDD without code execution

[Sean Griffin]

We've been using FeatureSpec as our form of BDD for quite awhile within my groups, and one of the consistent pains we see is the need to generate the newest requirements even when the tests might be broken.  Sure, we can search/replace "scenario" to "ignore" throughout everything, but that's a pain and introduces risk that they get committed to SCM that way.  After a little bit of brainstorming I came up with a simple trait that makes this possible.

trait QuietModeSupport {
  protected val quietMode: Boolean = java.lang.Boolean.getBoolean("scalatest.quietMode")
  final def %> [T] (code: => T): T = if (!quietMode) code else null.asInstanceOf
}

When writing my test I'd do something like:

class MySpec extends FeatureSpec with GivenWhenThen with QuietModeSupport {
  feature("My feature") {
    scenario("My scenario") {

      given("Some prereqs")
      %> {
        // Setup your prereqs
      }

      when("I execute my functionality")
      %> {
        // Do actual execution
      }

      then("I get back the right thing")
      %> {
        // Validate results
      }
    }
  }
}

And then I could enable the quiet mode either by ensuring the "scalatest.quietMode" system property was true or by simple overriding the quietMode val to be true when executing from the REPL, like so:

(new MySpec { override val quietMode = true }).execute()

We've started implementing at my company, but I was curious if anyone else thought it would be good to include something like this directly within ScalaTest.  If so, there might be a better, more integrated way to enable than through a system property.  If nothing else it might be beneficial information to others in the community.

[Bill Venners]

Hi Sean,

Well I've never seen a type parameter being inferred in a cast before,
like is being done in your "null.asInstanceOf". Even though it's a
cast, and its a null you're casting, frankly that's pretty cool. I
never imagined that was possible.

Sounds like you want to not ignore tests, but the code inside the test
other than the given/when/thens? I.e., you want to still see the
given/when/then output? Or is it just that it is a pain to turn off
the whole tests with ignored?

If the latter, would it help if you could annotate a class with
@Ignore and have all tests therein be ignored? You can do that in 2.0
(already works in trunk). It marks all the contained tests as ignored.
Would that solve it?

Bill

> --
> You received this message because you are subscribed to the Google
> Groups "scalatest-users" group.
> To post to this group, send email to scalate...@googlegroups.com
> To unsubscribe from this group, send email to
> scalatest-use...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/scalatest-users?hl=en
> ScalaTest itself, and documentation, is available here:
> http://www.artima.com/scalatest



--
Bill Venners
Artima, Inc.
http://www.artima.com

[Sean Griffin]

Hi Bill,

I've never seen the type inferred in a type cast either!  In fact, it was by accident that I found it.  I was helping a co-worker, who's new to Scala, make a modification to his local copy of this QuietModeSupport trait and was saying that we would have to add asInstanceOf to the null to fix the parse error in Eclipse.  After he finished typing "asInstanceOf" but before he put the "[T]" the parse error went away, and a full compile succeeded as well.  I even said "I don't think that will compile", but I was wrong.  I guess it speaks to how far along the Eclipse Scala plugin has come.  In the past I would know better than whatever parser errors it was showing me, but now it's the other way around!

I'm not in love with the null response, but I needed some value to return in the "else" case and didn't want to use Option since the return value when quietMode is true should just be ignored entirely, and most importantly, I don't want the quietMode behavior to force Some unboxing everywhere.  Nothing is not a value, so I can't return that.

Anyway, regarding your question.  Yes, at a high level I want the BDD output of the specification, but I don't want to run any code that might be broken or slow.  Of course by default I want the code to be ran, but I want an option to turn it off.  Some projects we have take 3 hrs or so to run the full suite of tests, and doing that just to look at the requirements is a productivity killer.  God forbid a test fails because of an environment glitch and I have to start all over.  So I do think it's different than marking the scenarios/tests as ignored.  I want the given/when/then statements and any other embedded info statements, just not "code". The only way I've thought of to separate the "test code" from the "spec code" is to demarcate it somehow...hence the %> method.

Why %>?  I played with a couple options, but I wanted it to be very short and easy to type, so nothing long like "codeUnderTest".  As far as the syntax of %>, it's inspired by JSP where Java code is "embedded" in the HTML.  I view this similar in that we're embedding test code into the specification.  Finally, since we have a separate person who writes the specifications from the person that implements the spec, it helps to separate responsibilities; the functional person who's not a programmer can ignore anything between the %> { }.

Ideally it would work best if it was a command-line option on the shell as opposed to a system property. Likewise, having it in the ScalaTest shell would be nice so I could do "quiet.run(new MySpec)".  But, in lieu of that, the system property works for now.

Sean

> To post to this group, send email to scalatest-users@googlegroups.com

> To unsubscribe from this group, send email to

> scalatest-users+unsubscribe@googlegroups.com

[etorreborre]

Hi Sean and Bill,

FYI there is a similar command-line argument in specs2 called "plan". When you use that option, only the text of the specification is displayed and no other code is executed.

So I guess it makes sense to provide a similar option in ScalaTest.

Cheers,

Eric.

[Bill Venners]

Hi Eric & Sean,

In Sean's case something like plan mode wouldn't help because he wants
to see the given/when/then output, which requires executing the tests.
I of course don't like the non-obvious %> operator as you might
expect. How about just making your own GivenWhenThen trait so that the
given/when/then/and constructs take the by-name. Then your code would
look like:

import org.scalatest._

class MySpec extends FeatureSpec with SkipableGivenWhenThen {

feature("My feature") {
scenario("My scenario") {

given("Some prereqs") {
println("Setup your prereqs")
}

when("I execute my functionality") {
println("Do actual execution")
}

then("I get back the right thing") {
println("Validate results")
}
}
}
}

Turning off the code would just mean putting something like "skipcode"
in the configMap (which you could do from command line or in the
Shell):

scala> import org.scalatest._
import org.scalatest._

scala> run(new MySpec)
MySpec:
My feature
Setup your prereqs
Do actual execution
Validate results
Scenario: My scenario
Given Some prereqs
When I execute my functionality
Then I get back the right thing

scala> run(new MySpec, configMap = Map("skipcode" -> true))
MySpec:
My feature
Scenario: My scenario
Given Some prereqs
When I execute my functionality
Then I get back the right thing

Here's the SkipableGivenWhenThen trait I used:

import org.scalatest._

trait SkipableGivenWhenThen extends AbstractSuite { this: Suite =>

@volatile private var skipCode: Boolean = false

/**
* Forwards a message to an implicit <code>Informer</code>, preceded
by "Given."
*
* @param message the message to forward to the passed informer
* @param fun the code, which will be executed unless
<code>skipCode</code> is contained as a
* key in the <code>configMap</code> passed to <code>run</code>
*
* @param info the <code>Informer</code> to which to forward the message
*/
def given(message: String)(fun: => Unit)(implicit info: Informer) {
info("Given " + message)
if (!skipCode) fun
}

/**
* Forwards a message to an implicit <code>Informer</code>, preceded
by "When ".
*
* @param message the message to forward to the passed informer
* @param info the <code>Informer</code> to which to forward the message
*/
def when(message: String)(fun: => Unit)(implicit info: Informer) {
info("When " + message)
if (!skipCode) fun
}

/**
* Forwards a message to an implicit <code>Informer</code>, preceded
by "Then ".
*
* @param message the message to forward to the passed informer
* @param info the <code>Informer</code> to which to forward the message
*/
def then(message: String)(fun: => Unit)(implicit info: Informer) {
info("Then " + message)
if (!skipCode) fun
}

/**
* Forwards a message to an implicit <code>Informer</code>, preceded
by "And ".
*
* @param message the message to forward to the passed informer
* @param info the <code>Informer</code> to which to forward the message
*/
def and(message: String)(fun: => Unit)(implicit info: Informer) {
info("And " + message)
if (!skipCode) fun
}

abstract override def run(
testName: Option[String],
reporter: Reporter,
stopper: Stopper,
filter: Filter,
configMap: Map[String, Any],
distributor: Option[Distributor],
tracker: Tracker
) {
skipCode = configMap.contains("skipcode")
super.run(testName, reporter, stopper, filter, configMap,
distributor, tracker)
}
}

Bill

>>> > To post to this group, send email to scalate...@googlegroups.com

>>> > To unsubscribe from this group, send email to

>>> > scalatest-use...@googlegroups.com

>>> > For more options, visit this group at
>>> > http://groups.google.com/group/scalatest-users?hl=en
>>> > ScalaTest itself, and documentation, is available here:
>>> > http://www.artima.com/scalatest
>>>
>>>
>>>
>>> --
>>> Bill Venners
>>> Artima, Inc.
>>> http://www.artima.com
>

> --
> You received this message because you are subscribed to the Google
> Groups "scalatest-users" group.

> To post to this group, send email to scalate...@googlegroups.com

> To unsubscribe from this group, send email to

> scalatest-use...@googlegroups.com

[Sean Griffin]

Bill,

That is an approach that I haven't considered, but after trying it out there are a few problems it still leaves and even introduces.  First, it's specific to given/when/then/and statements.  If using something like FunSpec, where I most likely wouldn't use GivenWhenThen, the only way to get the functionality is to mark the test as ignored.  That's not the end of the world, except that I still have to modify the source to make the tests ignored where instead I'd like it to be a property of the run, and it puts an ugly "IGNORED!!!" statement into my output, which would be quite confusing to give to my service consumers; the requirement is not ignored, the code that verifies it is being skipped.

Second, it breaks the ability to not pass by-name functions to these methods.  When our spec writer is first writing the specs she'll leave off any by-names and mark the scenarios as pending.  We can go in later when implementing the scenario and change to SkipableGivenWhenThen and add the by-names, which works, but then when we have an enhancement we need to add and she wants to add new scenarios, she'll be required to pass by-names to the methods.  I tried adding versions of the methods that were identical but didn't take by-names, but that led to errors around ambiguity when you where passing by-names.

Third, the given/when/then/and methods need to return a value.  That's an easy modification to make to the signatures you provided, though.  The reason this is needed is that sometimes the code in the "given" section creates values that need to be referenced in the "when" or "then" sections, like so:

given("A user with write privileges")

val user = // create user

when("Executing service with user")

service.call(user)

so it would need to be

val user = given("As user with write privileges") {

  // create user

}

when("Executing service with user") {

  service.call(user)

}

It's for this reason that my %> method is a generic method that returns T.  My original version didn't do this either, but after trying to plug it in to real tests on two different teams they both ran into this issue.

Sean

[Bill Venners]

Hi Sean,

Makes sense. Looks like a new ScalaTest feature may be justified. Just
in time too, because it would require an additional event added to the
event model, and that's a 2.0 "event."

The event would need to be a different from TestCanceled, TestIgnored,
or TestPending, because those have specific meanings that are
different from this. I've tried to avoid "skipped" because in English
a test is indeed skipped if it is canceled, ignored, or pending, and
that's what Maven and sbt, etc., print out for any of those. The best
I have come up with so far is TestOmitted. This would mean the test
either wasn't run at all, or if it was run, the result was dropped and
TestOmitted fired instead.

We have done the RunArgs refactor already in the trunk, which
simplifies the four run lifecycle method signatures. So I could add an
omitTests: Boolean flag to RunArgs. If you want tests to be omitted,
you'd set that flag. We'd provide a way to do so via runner, the build
tools, and the ScalaTest shell. I think we'd also need a
runOmittedTests: Boolean flag, initalized to false. More on that
later.

I'd want the output to look different. It certainly shouldn't look
like everything passed. So for the standard out or file reporter,
perhaps I'd turn color off and have a (tests omitted) next to the
Suite name.

StackSpec: (tests omitted)
A Stack
when empty
- should be empty
- should complain on pop

Similar visual differences in other reporters.

If a test is not executed, then I wouldn't know if it succeeded,
failed, was pending, or canceled. But I would know if it was ignored,
because that's known in advance of executing a test. So one question I
have for you is, would you want to see ignored notifications in the
output, or would you want TestIgnored events replaced by TestOmitted
events as well?

Then for cases like yours where you want the given/when/thens inside
the tests to show up in the structure output, perhaps we could offer a
OmittedTestCode trait. It would override run, and if omitTests is
true, set a volatile Boolean omit flag declared in the trait, then
call super.run with runOmittedTests in the RunArgs set to true. That
will cause the suite to actually run the tests, but fire a TestOmitted
event regardless of the result.

Now on to omitting slow test code. I'm confident there's a satisfying
way to avoid using a non-obvious operator if we look hard enough.
Would this be too wordy? For slow side effecting things, could you
write:

if (!omit) doTheSlowSideEffectingThing()

The volatile omit flag would be set by the run method and therefore
available to the tests. For things that require a T, could you do
something like:

val user = if (!omit) createUser("Bob") else null

The ugly null is in there, but the code is obvious. Could possibly
provide a default method that takes a T:

val user = if (!omit) createUser("Bob") else default[User]

An implicit could get you something like:

val user = createUser("Bob") orOmitAs null

or:

val user = createUser("Bob") orOmitAs default

Something along those lines? We could consider something like you %>
but the obvious text is a bit verbose I think:

unlessOmitted {
doTheSlowSideEffectingThing()
}

or:

val user = unlessOmitted {
createUser("Bob")
}

Maybe not so bad, though for the latter it is not obvious what user is
getting assigned if omit is set.

Bill