Running all my specifications under one JUnitRunner

[Anatoly Rabinovich]

Hi,

I'm trying to figure out if it's at all possible to aggregate all my specifications under one runner class (that is annotated with '@RunWith(classOf[JUnitRunner])') to do an action before my tests run in TeamCity. There is a solution for that in ScalaTest, by extending the 'Suite' trait and overriding the 'nestedSuites' and 'run' methods.

Any help will be appreciated.

Thank!

[etorreborre]

Hi Anatoly,

A similar question was asked a few days ago on Stackoverflow: http://stackoverflow.com/questions/37891820/alternative-of-oneapppersuite-in-specs2-scala-testing.

You will need to pass `-Dspecs2.all` as a JVM option to your JUnit runner.

E.

[Anatoly Rabinovich]

Thanks for the reference. But I'm not sure it's the most convenient way of doing that. We've solved this in ScalaTest by annotating the suites to run with a custom annotation and then loading them via reflection to the overridden 'nestedSuites'. This way when adding a new test suite, we only need to add the annotation. It's much more practical in a big project than via the way described in the stackoverflow issue.

Is this the only way to accomplish this functionality?

Thanks.

[etorreborre]

You can either use the `Files` runner to discover specifications and run them. 

Or you can use the `SpecificationsFinder` class to find the specifications and include them in a "Suite" specification as in the StackOverflow question.

Something like that:

import org.specs2._
import runner._
import control._
import specification.core._

class MySuite extends Specification { def is = s2"""

  My specifications
    ${ SpecificationsFinder.findSpecifications(glob = "**/*Spec.scala").runOption.toList.flatten.foldLeft(Fragments.empty) {
        (res, s) => res ^ br ^ link(s)
      }
    }

"""
}

E.

[Anatoly Rabinovich]

Im probably missing something, and apologies for my ignorance, but this is what I'm trying to do: 

import org.specs2._
import runner._
import control._
import specification.core._

class MySuperSuite extends Specification { 

  val someResult = <some code here>

  Try {

     val mySpecStructure = 

        s2"""

           My specifications
             ${ 

                SpecificationsFinder.findSpecifications(glob = "**/*Spec.scala").runOption.toList.flatten.foldLeft(Fragments.empty) {
                  (res, s) => res ^ br ^ link(s)
                }
              }
          """

     <code that runs 'mySpecStructure' and returns some result, even if it's only boolean>

  } match {

      case Success(testResults) => <run more of my code using 'someResult' and 'testResults'>

      case Failure(e) => <run some other code>

  }

  
}

Is there an explicit way of running the specification structure manually from my code? 

I seem to have a gap in knowledge in using the created specification structure and cannot find any good source (almost nothing is written about this) of examples.

Thanks!

[Anatoly Rabinovich]

Is the File runner a better choice here?

[etorreborre]

Can you give me a better overview of your domain?

What kind of result is "someResult"? What kind of "testResults" are you expecting? What kind of code do you want to run if there is a failure?

Thanks.

[Anatoly Rabinovich]

We're trying to determine if TC is trying to run these tests while our deployment system is trying to deploy a new version of our service.

We have a class similar to this:

abstract class ScalaTestSimulationSuite extends Suite with SimulationSuite {
  lazy val suits = {
    val reflections = new Reflections(package2Scan, new TypeAnnotationsScanner())
    val tests: Set[Class[_]] = reflections.getTypesAnnotatedWith(testAnnotation).toSet

    tests.map { testClass =>
      val ctor = testClass.getDeclaredConstructor()
      ctor.newInstance().asInstanceOf[Suite]
    } toIndexedSeq
  }

  override def nestedSuites: IndexedSeq[Suite] = suits

  override def run(testName: Option[String], args: Args): Status = {
    val beforeVersion = getVersion
    val status = Try {
      super.run(testName, args)
    }
    val afterVersion = getVersion

    if (beforeVersion != afterVersion || beforeVersion.isEmpty) {
      handleVersionsMissMatch(beforeVersion, afterVersion)
    } else {
      // Write some logs and do something else.
    }
    status.get
  }
}

We create suites by scanning for a specific annotation on our test classes. 

The important method here is 'run' which runs the test suites one by one. First we retrieve the version of our code that runs in our TC simulation servers. Then we run a test and get the current version of the code. If they're different (we're in the middle of a deployment of a new version to our servers) then we stop the TC job that is running right now.

We then extend this class by a class with the '@RunWith(classOf[JUnitRunner])' annotation, which is run by TC.

[etorreborre]

Thanks.

This is more or less the same as what you are doing now with ScalaTest:

import org.specs2._
import runner._
import control._
import specification.core._

import specification.process._
import scalaz._, Scalaz._

class MySuite(env: Env) extends Specification { def is = s2"""

  Run all the specifications
  ${run(specifications)}

"""

  def run(specs: List[SpecificationStructure]): Fragments = {
      specs.foldLeft(Fragments.empty.right[Fragments]) {
        case (\/-(res), spec) =>
          val previousVersion = getVersion
          val stats = ClassRunner.report(env)(spec).runOption.getOrElse(Stats.empty)
          val newVersion = getVersion

          if (previousVersion != newVersion)
            \/.left(res ^ br ^ handleVersionsMissMatch(previousVersion, newVersion))
          else
            \/.right(res ^ br ^ logStuff(stats))

        case (res, _) => res
      }.fold(identity _, identity _)
  }

  def specifications: List[SpecificationStructure] =
    SpecificationsFinder.findSpecifications(glob = "**/*Spec.scala").runOption.toList.flatten

  def handleVersionsMissMatch(previousVersion: Version, newVersion: Version): Fragments =
    s2"""
      There was a new version, stopping everything !!!! $previousVersion != $newVersion
    """

  def logStuff(stats: Stats): Fragments =
    s2"""
        some logging here
        The stats are ${stats.toString}
    """

  type Version = String

  def getVersion: Version =
    "current version"
}

The main difference here is that you need to iterate over the specifications yourself and explicitly say when to stop with a foldLeft and \/.

Would that work for you?

E.

[Anatoly Rabinovich]

Hi, 

Thanks for that. I'll give it a go.