Retry Entire Test Suite

[Jared Yarn]

I am using Scalatest with SBT to run the tests. We have a suite of about 600 selenium tests spread into about 100 suites. All of them use FeatureSpec. I am wondering if there is a way to retry an entire suite of tests if one of the tests in the suite failed.

 I have tried using retires and overriding the withFixture method but can only get this to re run single scenarios. Is there any way to set it up to rerun an entire test suite or at lease a "feature" in the featureSpec instead of just retrying a specific "scenario".

Thanks,

Jared

[Bill Venners]

Hi Jared,

ScalaTest does not have a trait for that, but luckily you can "bend
the framework" (instead of fighting the framework). The way to
accomplish this is to override one of ScalaTest's lifecycle methods.

To rerun the entire Suite you'd most likely want to override the run
method. It would call super.run then look at the Status result. That
can be async (Status is like a domain specific Future[Boolean]), so
you should call the succeeds() method that blocks waiting for the
completion. Then if it is a failure, just call super.run again,
returning the second Status. It would look something like:

trait SuiteRetries extends SuiteMixin { this: Suite =>
abstract override def run(testName: Option[String], args: Args): Status = {
val status = super.run(testName, args)
if (status.succeeds())
status
else
super.run(testName, args)
}
}

Here how that looks in the REPL:

scala> import scala.language.postfixOps
import scala.language.postfixOps

scala> class MySuite extends FunSuite with SuiteRetries {
| test("oops") { assert(1 == 2) }
| test("good") { assert(1 + 1 == 2) }
| }
defined class MySuite

scala> new MySuite execute
MySuite:
- oops *** FAILED ***
org.scalatest.exceptions.TestFailedException was thrown. (<console>:28)
- good
- oops *** FAILED ***
org.scalatest.exceptions.TestFailedException was thrown. (<console>:28)
- good

As you can see because one test failed, it reran both of the others.
Here's the success behavior:

scala> class MyOtherSuite extends FunSuite with SuiteRetries {
| test("not oops") { assert(1 != 2) }
| test("good") { assert(1 + 1 == 2) }
| }
defined class MyOtherSuite

scala> new MyOtherSuite execute
MyOtherSuite:
- not oops
- good

As you can see, when all tests passed it doesn't rerun the suite.

Bill

> --
> You received this message because you are subscribed to the Google
> Groups "scalatest-users" group.
> To post to this group, send email to scalate...@googlegroups.com
> To unsubscribe from this group, send email to
> scalatest-use...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/scalatest-users?hl=en
> ScalaTest itself, and documentation, is available here:
> http://www.artima.com/scalatest
> ---
> You received this message because you are subscribed to the Google Groups
> "scalatest-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to scalatest-use...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.



--
Bill Venners
Artima, Inc.
http://www.artima.com

[Jared Yarn]

Thank you so much for the fast reply. It works great. 

Is there anyway to configure this so we catch the testFailed events while we are in the retry. When we run the full suite of tests in SBT we get a list of all of the suites that failed at the end. It would be nice if this test passed on the retry, it would not be included in the list of failures. I have played a little bit with the custom reporter, but can only seem to get it set up to catch all the test failures, and not just the ones during the retry. 

Thanks,

Jared

[Jared Yarn]

 I was able to figure it out by using this post https://groups.google.com/forum/?hl=en&fromgroups=#!topic/scalatest-users/lYEaHqHTqPY . Got it working perfectly. Thanks so much for the help.