stopOnFail granularity

[nahuaque]

Hi,

I'm wondering whether I can get the stopOnFail behavior to apply only
per system under specification. I.e. if an example fails, then no
further examples for that SUS will execute, but examples for the next
SUS will execute as normal.

Thanks,

Lorand

[etorreborre]

I'm a bit reluctant to complexify the current implementation with a new mechanism so perhaps you can try the following idea:

class MySpec extends Specification {

  stopIfFail("system1") {

    "example1" >> failure

    "example2" >> success

  }

  stopIfFail("system2") {

    "example1" >> success

    "example2" >> success

  }

  def stopIfFail(title: String)(fs: =>Fragments) = {

    include(new org.specs2.Specification { def is = stopOnFail ^ title ^ fs })

  }

}

You can change the names and maybe "pimp" things a little but that could do the trick.

Eric.

[stephanos]

Sorry to resurrect such an old thread but I tried this and it doesn't seem to work:

[info] Object

[info] 

[error] x example11

[error]  failure (S040.scala:39)

[info] + example12

[info] system1

[info] + example12

[info] + example21

[info] + example22

[info]   system2

[info]   + example22

[info]  

[info] Total for specification Object

[info] Finished in 53 ms

[info] 6 examples, 1 failure, 0 error

Some examples are run twice.

Here is the complete code

import org.specs2.mutable.Specification

import org.specs2.specification.Fragments

class MySpec extends Specification {

  stopIfFail("system1") {

    "example11" >> failure

    "example12" >> success

  }

  stopIfFail("system2") {

    "example21" >> success

    "example22" >> success

  }

  def stopIfFail(n: String)(fs: =>Fragments) = {

    include(new org.specs2.Specification { def is = stopOnFail ^ n ^ fs })

  }

}

Hope someone knows what's wrong?

Stephan

[stephanos]

I'm using the immutable version now - it's only executed once but it stops on the first system and skips the second:

[info] system1

[error] x example11

[error]    failure (T030.scala:17)

[info] + example12

[info] Object

[info] 

[info] system2

[info] o example21

[info] o example22

[info]  

[info] Total for specification Object

[info] Finished in 3 ms

[info] 2 examples, 0 failure, 0 error, 2 skipped

import org.specs2.Specification

import org.specs2.specification.Fragments

The code is: 

 

class MySpec extends Specification {

    def is =

  stopIfFail("system1") {

    "example11" ! failure ^

    "example12" ! success

  } ^

  stopIfFail("system2") {

    "example21" ! success ^

    "example22" ! success

[etorreborre]

You're right, what I proposed could not really work. Here's a tested alternative:

import org.specs2.mutable.{Around, Specification}

import org.specs2.execute.{Skipped, Result}

class TestSpec extends SuperSpecification {

    sequential

    "system1" >> {

      implicit val stop = WhenFail()

      "example1" >> ko

      "example2" >> ok

    }

    "system2" >> {

      implicit val stop = WhenFail()

      "example3" >> ok

      "example4" >> ok

    }

}

case class WhenFail() extends Around {

  private var mustStop = false

  def around[R <% Result](r: =>R) = {

    if (mustStop)          Skipped("one example failed")

    else if (!r.isSuccess) { mustStop = true; r }

    else                   r

  }

}

This one prints out:

[info] TestSpec

[info] 

[info] system1

[error] x example1

[error]    ko (TestSpec.scala:12)

[info] o example2

[info] one example failed

[info]  

[info] system2

[info] + example3

[info] + example4

[info]  

[info]  

[info] Total for specification TestSpec

[info] Finished in 15 ms

[info] 4 examples, 1 failure, 0 error, 1 skipped

Eric.

[stephanos]

It works as promised. Great, thanks again :-)

[stephanos]

I just came back to this - but now I need it for an immutable spec. Since I don't want to wrap each example with a new instance of StopOnFail, my attempt was to use the map function:

import org.specs2._

import execute.{Skipped, Result}

import specification.{Example, Fragments, Around}

class Spec extends Specification {

    def is = sequential ^

        "system1" ^

            withStop(

                "example 1" ! ko ^

                "example 2" ! ok

            ) ^ p ^

        "system2" ^

            withStop(

                "example 3" ! ok ^

                "example 4" ! ok

            )

    def withStop(fs: => Fragments): Fragments =

        fs.map { f =>

            val sof = new StopOnFail()

            f match {

                case e: Example => e.map(sof(_))

                case _ => f

            }

        }

    class StopOnFail() extends Around {

        private var mustStop = false

        def around[R <% Result](r: => R) = {

            if (mustStop)           Skipped("(a previous test failed)")

            else if (!r.isSuccess)  { mustStop = true; r }

            else                    r

        }

    }

}

But I would not write this if it was working: all tests are executed - but example 2 should be skipped. What am I doing wrong?

Regards

Stephan

On Sunday, August 12, 2012 1:32:20 AM UTC+2, etorreborre wrote:

[etorreborre]

Hi Stephan,

Just reading your code I guess that you should write this:

 val sof = new StopOnFail()        

 fs.map { f =>

            f match {

                case e: Example => e.map(sof(_))

                case _ => f

            }

        }

And create your sof instance outside of the map function so that it is used for each example. Otherwise you have one StopOnFail instance per example and it can not memorize failures to stop further execution (because there's only on execution).

Cheers,

Eric. 

[etorreborre]

I think that you can get something a bit more syntactically pleasing with:

import org.specs2._

import org.specs2.specification._

import execute._

trait StopOnFail extends AroundExample with Specification {

  // make sure the specification is sequential

  override def map(fs: =>Fragments) = sequential ^ fs

  private var mustStop = false

  def around[R <% Result](r: =>R) = {

    if (mustStop)          Skipped("one example failed")

    else if (!r.isSuccess) { mustStop = true; r }

    else                   r

  }

  def startBlock = Action(mustStop = false)

}

class MySpec extends Specification with StopOnFail { def is = nocolor^

  "first block"   ^ 

    "example1"    ! ok^

    "example2"    ! ko^ 

    "example3"    ! ok^ 

                  endp^ startBlock^

  "second block"  ^ 

    "example1"    ! ko^

    "example2"    ! ok

}

specs2.run(new MySpec)

specification

first block

+ example1

x example2

   ko (<console>:20)

o example3

one example failed

 

second block

x example1

   ko (<console>:20)

o example2

one example failed

 

Total for specification specification

Finished in 26 ms

5 examples, 2 failures, 0 error, 2 skipped

Eric.

[stephanos]

Awesome, thank you so much.

  }...
Show original

[stephanos]

I was just wondering if an example within a group can not only be skipped (thus appearing in the console) but even excluded completely when a failure occurred earlier.

I'm asking because I'm preparing a Scala training and the unit tests guide the participants through the exercises - the idea above might help to make sure they only take one step at a time ...

Do you think it's possible?

On Monday, December 24, 2012 1:44:05 AM UTC+1, etorreborre wrote:

  }...
Show original

[etorreborre]

You can use the "showOnly" option with the appropriate statuses:

> test-only *MySpec* -- showOnly x!+

This will only show the examples having failures "x" or errors "!' or successes "+".

Remember that you can put those test arguments into the sbt configuration file so that you don't have to enter them all the time on the command line (see "testOptions" here).

E.

    "example1" >> failure</...
Show original

[stephanos]

I just tried to use the 'showOnly' approach, and while it works without a problem in the console, the IntelliJ plugin is not playing along so nicely:

It still displays the skipped tests, thinking they are "still running" - I guess since specs2 doesn't show the result for it.

Is there any way I can completely skip the execution of the examples after the first failure - I suppose this only works by manipulating the "fragment structure" ... ?

[etorreborre]

I thought about saying "no, let the Jetbrains guys do it" but I think it's a valid request actually. I'll have a go at it this week.

Eric.

[etorreborre]

Hi Stephanos,

I've added the proper use of "showOnly" when using the Notifier interface which IntelliJ uses. You should get a better display now (using the latest 1.12.4-SNAPSHOT or 1.13.1-SNAPSHOT), excluding the skipped examples where required.

For example: 

>test-only *MySpec* -- showOnly x!+

Note that in recent versions of IDEA there are problems with passing specs2 options as command line options so you'd better use system properties instead:

-Dspecs2.showOnly=x!+

Cheers,

Eric.

[stephanos]

Hi Eric, 

it works! Cool!

Now, I assume making it work for Eclipse is more difficult. Eclipse seems to use the default JUnit test runner, but I don't know where it "hooks" into the tests. Maybe I could manipulate the results somehow, what do you say?

Regards

Stephan

[stephanos]

By the way, here is the output from the JUnit test runner:

Don't ask my where this "action" comes from ...

[etorreborre]

Yes, JUnit cannot work that way because it insists on having the full list of tests before executing anything.

[etorreborre]

This is something I need to investigate....

[stephanos]

Hm, this sucks.

Do you think I can "hack" this to run the tests before JUnit is launched and only give JUnit the tests with/before any failure? I know it sounds a little odd, but the tests all run in less than 1 second and are only used in the mentioned class room course. Do you think it's doable?

[etorreborre]

If your tests don't take too long to execute you can use 2 passes:

import org.specs2._

import reporter._

import main._

import specification._

// this reporter executes the specification but do not display anything

object silent extends DefaultReporter {

  def export(implicit args: Arguments) = (s: ExecutingSpecification) => s.execute

  def execute(fs: Fragments) = report(new Specification { def is = sequential ^ fs })(Arguments())

}

trait Filter { this: Specification =>

  override def map(fs: =>Fragments) = {

    val executed = silent.execute(fs)

    // you can use whatever logic here based on the result of the previous execution

    val filtered = executed.fs.zip(fs.fragments).collect { case (f1, f2)  if (f1.stats.result.isSuccess) => f2 }

    Fragments.createList(filtered:_*)

  }

}

class TestSpec extends Specification with Filter { def is = 

  "ex1" ! ok^

  "ex2" ! ko^

  "ex3" ! ok

}

This is a hack is the sense that the first execution has to be executed sequentially otherwise there is a deadlock somewhere (I have no idea why). If this is ok for you, this might be the best way to go.

Cheers,

Eric.

[stephanos]

Works like a charm, thank you so much!