Prop.exists and ScalaTest matchers

[Juan Rodríguez Hortalá]

Hi, 

I have a problem about using ScalaTest matchers inside a Prop.exists property. When the property runs and a value is generated such that the matcher fails, it throws a TestFailedException and the property fails, instead of trying with the next generated value until reaching the limit in the number of tests. I guess this is because ScalaTest is designed around TestFailedExceptions, while ScalaCheck is designed around Prop objects. I have devised the following workaround, and I wanted to you what you think about it: 

import org.scalacheck.{Properties, Gen}

import org.scalacheck.Prop.{forAll, exists, AnyOperators}

import org.scalacheck.Prop

import org.scalatest._

import org.scalatest.Matchers._

import scala.util.{Try, Success, Failure}

import org.scalacheck.util.Pretty

object ExistsBrokeTest extends Properties("Breaking Prop.exists with ScalaTest matchers") {

  property("forAll works ok with ScalaTest matchers, for successful tests") = 

    forAll { x : Int => 

      x + x should be (2 * x)

      true

    }

  

  property("forAll works ok with ScalaTest matchers, for failing tests") = 

    forAll { x : Int => 

      x + x should be (2 * x + 1)

      true

    }

  

  property("exists works ok without ScalaTest matchers") =

    exists ("x" |: Gen.choose(1, 20)) { x : Int =>

      x ?= 1

    }

  

  property("exists fails with ScalaTest matchers, for otherwise successful tests") =

    exists ("x" |: Gen.choose(0, 10)) { x : Int =>

      x should be (1)

      true

    }

  

  property("exists works ok with ScalaTest matchers wrapped by Try") = 

    exists ("x" |: Gen.choose(0, 10)) { x : Int =>

      Try(x should be (1)).isSuccess 

  }

  

  def safeProp[P <% Prop](p : => P) : Prop = {

    Try(p) match {

      case Success(_) => Prop.proved

      case Failure(t) => t match {

        case _: TestFailedException => Prop.falsified

        case _ => Prop.exception(t) 

      }

    }

  }

  

  def safeExists[A, P](g: Gen[A])(f: (A) => P)

                      (implicit pv: (P) => Prop, pp: (A) => Pretty): Prop = {    

    exists(g)((x : A) => safeProp(pv(f(x))))(identity[Prop], pp)

    // This doesn't work for reasons unknown

    // exists(g)((x : A) => Prop.secure(pv(f(x))))(identity[Prop], pp)

  }  

  property("safeExists works ok with ScalaTest matchers, for successful tests") =

   safeExists("x" |: Gen.choose(0, 10)) { x : Int =>

     x should be (1)

     true

   }

  

  property("safeExists works ok with ScalaTest matchers, for failing tests") =

   safeExists("x" |: Gen.choose(0, 10)) { x : Int =>

     x should be (1000)

     true

   }

}

+ Breaking Prop.exists with ScalaTest matchers.forAll works ok with ScalaTe

  st matchers, for successful tests: OK, passed 100 tests.

! Breaking Prop.exists with ScalaTest matchers.forAll works ok with ScalaTe

  st matchers, for failing tests: Exception raised on property evaluation.

> ARG_0: -2147483648

> Exception: org.scalatest.exceptions.TestFailedException: 0 was not equal 

  to 1

org.scalatest.MatchersHelper$.newTestFailedException(MatchersHelper.scala:1

  60)

org.scalatest.Matchers$ShouldMethodHelper$.shouldMatcher(Matchers.scala:623

  1)

org.scalatest.Matchers$AnyShouldWrapper.should(Matchers.scala:6265)

es.ucm.fdi.sscheck.ExistsBrokeTest$$anonfun$2.apply$mcZI$sp(ExistsBrokeTest

  .scala:20)

es.ucm.fdi.sscheck.ExistsBrokeTest$$anonfun$2.apply(ExistsBrokeTest.scala:1

  9)

+ Breaking Prop.exists with ScalaTest matchers.exists works ok without Scal

  aTest matchers: OK, proved property.

> x: 1

! Breaking Prop.exists with ScalaTest matchers.exists fails with ScalaTest 

  matchers, for otherwise successful tests: Exception raised on property ev

  aluation.

> x: 3

> Exception: org.scalatest.exceptions.TestFailedException: 3 was not equal 

  to 1

org.scalatest.MatchersHelper$.newTestFailedException(MatchersHelper.scala:1

  60)

org.scalatest.Matchers$ShouldMethodHelper$.shouldMatcher(Matchers.scala:623

  1)

org.scalatest.Matchers$AnyShouldWrapper.should(Matchers.scala:6265)

es.ucm.fdi.sscheck.ExistsBrokeTest$$anonfun$3.apply$mcZI$sp(ExistsBrokeTest

  .scala:31)

es.ucm.fdi.sscheck.ExistsBrokeTest$$anonfun$3.apply(ExistsBrokeTest.scala:3

  0)

+ Breaking Prop.exists with ScalaTest matchers.exists works ok with ScalaTe

  st matchers wrapped by Try: OK, proved property.

> x: 1

+ Breaking Prop.exists with ScalaTest matchers.safeExists works ok with Sca

  laTest matchers, for successful tests: OK, proved property.

> x: 1

! Breaking Prop.exists with ScalaTest matchers.safeExists works ok with Sca

  laTest matchers, for failing tests: Gave up after only 0 passed tests. 10

  1 tests were discarded.

Also, I have noticed that org.scalatest.prop.GeneratorDrivenProperyChecks doesn't define exists, is there any technical reason for that?, I would be glad to work on it if you think it is possible, 

Thanks a lot in advance.

Greetings, 

Juan Rodriguez Hortala

[Juan Rodríguez Hortalá]

Hi,

Any thoughts on this? I have opened an issue at https://github.com/rickynils/scalacheck/issues/174 but I haven't received any answer yet

Thanks in advance

Greetings, 

Juan

[Bill Venners]

Hi Juan,

Sorry I didn't notice your previous email of 8 days ago. I would
suggest you either go more consistently in the ScalaTest or the
ScalaCheck direction. Either use the forAll from ScalaTest's
GeneratorDrivenPropertyChecks, which would allow you to use ScalaTest
matchers (and you might use PropSpec as well), or just use Boolean
expressions with ScalaCheck's forAll. Otherwise you'll be fighting the
frameworks!

The reason there's no exists in GeneratorDrivenPropertyChecks is that
I haven't every had a user request it (until you asked about it now).
I actually asked Rickard about exists years ago when I was first doing
the ScalaTest integration with ScalaCheck. I wondered what the use
case might exist for exists for testing (aside from that question). As
I recall he said he didn't think there was that much need for it for
testing, so it was more just because you have "there exists" and "for
all" in predicate logic, so why not have both in a property-based
testing library for consistency with logic. I did actually finally
decided to add it in for completeness, but only have apparently put it
into TableDrivenPropertyChecks, not GeneratorDrivenPropertyChecks. So
if you want to do that, give it a shot. But please work in the 3.0.x
branch. These are code-generated, so the file to change is this one:

https://github.com/scalatest/scalatest/blob/3.0.x/project/GenGen.scala

I won't include such a PR in 3.0.0 because it is too late, but that's
the correct branch to submit a PR against.

Thanks.

Bill

> --
> You received this message because you are subscribed to the Google Groups
> "scalacheck" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to scalacheck+...@googlegroups.com.
> To post to this group, send email to scala...@googlegroups.com.
> Visit this group at http://groups.google.com/group/scalacheck.
> For more options, visit https://groups.google.com/d/optout.



--
Bill Venners
Artima, Inc.
http://www.artima.com

[Juan Rodríguez Hortalá]

HI Bill, 

Thanks a lot for your answer. I will try to avoid matchers in ScalaChek properties then. 

Greetings, 

Juan