running fragments and catching output

[Alex Cozzi]

Not sure whether I am trying something stupid, but here it is: I have a Spec for a record in a large dataset (on hadoop) and I would like to run a map-reduce job that iterates over the dataset, run the spec on each record and if the spec fails saves the data and the error into an output data-file. 

the whole thing runs in spark, but ignoring all the details is looks roughly like this:

object SparkSpecs extends MustMatchers {

  import SparkUtils._

  def main(args: Array[String]) {

     val data = < read file from hdfs>

     val output = data.flatMap { record => 

          val spec = new DataSpec(record)

          val (message, result) = spec.executeItNow // <- I want to execute this now and get the result

          if (result.isFailure)  {

            Some(record, message)

          } else {

             None

          }

     }

    sc.writeAsTextFile("bla") 

  }

}

class DataSpec(record: Record) extends org.specs2.Specification {

  def is = s2"""

 a record

   must have a non-empty cidd             $e1

   tt must be 2                           $e2

                                          """

  def e1 = record.ciid must beSome

  def e2 = record.tt must_== 2

}

Thank!

Alex

[etorreborre]

Hi Alex, 

You don't need full spec for this

  def checkRecord(record: Record): Option[(Record, String)] = { 

    val result = 

      (record.ciid must beSome) and 

      (record.tt must_== 2)

    if (result.isSuccess) None

    else                  Option((record, result.message))

   }

You can also adorn your messages with a bit of explanation:

https://etorreborre.github.io/specs2/guide/SPECS2-3.7.1/org.specs2.guide.ExpectationDescription.html

def checkRecord(record: Record): Option[(Record, String)] = { 

    val result = 

      "The record has a non-empty cid" ==> { 

        record.ciid must beSome

      } and 

      "tt is 2" ==> { 

        record.tt must_== 2 

      }

    if (result.isSuccess) None

    else                  Option((record, result.message))

   }

 

Cheers,

Eric

[Alex Cozzi]

Thank you Eric! that is helpful.

The reason why I wanted to use the full spec was so that I can have managers and such review the spec, so having it look less like code and more like text is a plus, but this is already very good. 

Thank you

[etorreborre]

If you want to review something looking a bit more like a spec, you can do this:

import org.specs2.specification.core.{Fragment, Env, Fragments}

val fs: Fragments =
  s2"""|
       |The record must have a non-empty cid
       | ${record.ciid must beSome}
       |
       |tt must be 2 ${record.tt must_== 2}
       |""".stripMargin

def report(f: Fragment): Option[String] = {
  val result = f.executionResult

  if (result.isSuccess) None
  else Some(s"[fail] ${f.description.show}: ${result.message}")
}

println {
  fs.examples.map(_.updateExecution(_.execute(Env()))).map(report).flatten.mkString("\n")
}

This prints:

[fail] The record must have a non-empty cid: 'None' is not Some

[fail] tt must be 2: '1' is not equal to '2'

So you should be able to better separate the text from the code execution / reporting.

Eric.

[Alex Cozzi]

Perfect! Just what I needed. Thank you so much.