Corner cases are always generated?
Raphael
I started using scalacheck today since I got exposed to Quickcheck and
like the philosophy.
I wanted to use it to test some random number generator I have
written. Some distributions need parameters that are positive, so I
created a generator
`val posInt = Gen.choose(1, 100)`
A call site looks like
`property("Chi²") = forAll(posInt){ v => ... nextChiSq(v) ... }`
I now get, thanks to `require` calls in my generator:
> [info] ! RichRandom.Chi²: Exception raised on property evaluation.
> [info] > ARG_0: 0
> [info] > Exception: java.lang.IllegalArgumentException: requirement failed: Parameter has to be positive
Not even
`val posInt = Gen.choose(1, 100) suchThat (_>0)`
prevents zero from being chosen. The same happens with `listOfN`
(empty list is generated!).
What is going on, am I doing something wrong?
Cheers
Raphael
Rickard Nilsson
You are not doing something wrong, and ScalaCheck is actually not really
_generating_ zeros or empty lists in these cases. This is in fact a
feature of the simplification system in ScalaCheck that tries to simplify
test data as soon as it has found a failing test case. When doing this, it
has no knowledge of the generator that produced the test case originally.
So, in your case, ScalaCheck has found a number in the range 1-100 that
makes your property false. After this, it tries to find smaller numbers
that also will fail the property. 0 is such a number. There are some
different things you can do to get around this:
1) Run the test with increased verbosity, so ScalaCheck also will present
the original argument, not only the simplified one:
scala> val p = Prop.forAll(posInt) { n => 3/n == 3*(1/n) }
scala> Test.check(Test.Params(testCallback = ConsoleReporter(2)), p)
! Exception raised on property evaluation.
> ARG_0: 0 (orig arg: 3)
> Exception: java.lang.ArithmeticException: / by zero
line3$object$$iw$$iw$$iw$$iw$$anonfun$1.apply$mcZI$sp(<console>:9)
line3$object$$iw$$iw$$iw$$iw$$anonfun$1.apply(<console>:9)
line3$object$$iw$$iw$$iw$$iw$$anonfun$1.apply(<console>:9)
org.scalacheck.Prop$$anonfun$forAllShrink$1$$anonfun$3$$anonfun$4.apply(Pro
p.scala:531)
2) Add a precondition to the property:
scala> import Prop._
import Prop._
scala> val p = Prop.forAll(posInt) { n => n > 0 ==> (3/n == 3*(1/n)) }
p: org.scalacheck.Prop = Prop
scala> p.check
! Falsified after 19 passed tests.
> ARG_0: 2
3) Use forAllNoShrink, which won't try to simplify your failing test cases:
scala> val p = Prop.forAllNoShrink(posInt) { n => 3/n == 3*(1/n) }
p: org.scalacheck.Prop = Prop
scala> p.check
! Falsified after 7 passed tests.
> ARG_0: 2
One could also think of ways of "fixing" this in ScalaCheck. Maybe the
precondition could be integrated into the generator in some way, but that
would pollute the clear separation between generators and properties.
Maybe the simplifier should not count exceptions as "real" failures, but
that would just help in this particular type of case. As always, I'm open
for suggestions.
Anyway, when you do run in to this situation, you can be sure there is an
actual failure _within_ the generator's range, but the simplification
algorithm obfuscates this sometimes.
Best regards,
Rickard Nilsson
Raphael
Wow, very helpful answer, thanks a lot!
> This is in fact a
> feature of the simplification system in ScalaCheck that tries to simplify
> test data as soon as it has found a failing test case.
> 1) Run the test with increased verbosity, so ScalaCheck also will present
> the original argument, not only the simplified one:
hints as to how to set this up though. I guess I need to overwrite the
check method somehow?
> 2) Add a precondition to the property:
the test would not be considered failed? That means essentially
writing preconditions down two times, not nice.
> 3) Use forAllNoShrink, which won't try to simplify your failing test cases:
really of interest, anyway; it is just confusing that forbidden values
pop up. In general, not shrinking is not an option, I guess.
> Maybe the simplifier should not count exceptions as "real" failures, but
> that would just help in this particular type of case.
exceptions signify failure by definition.
> As always, I'm open for suggestions.
me), I have two places to specify preconditions. One is when I create
generators, which will effectively result in rejection sampling. The
second place is as precondition of properties, where they will result
in successful tests when not fulfilled. You have to be careful not to
use either wrongly; a very broad generator used on a property with a
very restrictive might result in passed tests without executing the
function under test even once.
Why not factor out preconditions and pass them to generator, property
and shrinker independently? Ideally (for testing intended behaviour on
the definition space), no parameters not fulfilling them should be
generated in the first place. For simple cases, suchThat should be
able to restrict sampling in a clever way. Maybe some predefined
conditions can be added (such as interval boundaries) and
automatically used "correctly". A shrinker should of course never
reduce to a parameter that does not fulfill the preconditions. In
ideal settings, the property need not do anything with the
precondition. Am I missing a use case where property conditions serve
a purpose other than filtering?
In any case, I have to put more thought in my properties. Turns out
testing random number generators is a messy business.
By the way, in which way are you in contact with the creators of
QuickCheck? I ask because I happen to occupy the same office building
currently and think they would be interested in new features you say
you implemented. Also, have you heard of their latest child, QuickSpec
( e.g. http://portal.acm.org/citation.cfm?id=1894408 )?
Best
Raphael
Rickard Nilsson
>> 1) Run the test with increased verbosity, so ScalaCheck also will
>> present the original argument, not only the simplified one:
> That would certainly help coping with this kind of issue. I can't find
> hints as to how to set this up though. I guess I need to overwrite the
> check method somehow?
I'm not sure what you mean. You can give the Text.check method
parameters, one of which controls who should receive callbacks
when properties are evaluated. To get the results printed on the
console, you can use the built-in ConsoleReporter, which you can
control the verbosity of. The default verbosity is 1, but to get
the non-simplified arguments in the reports you need verbosity >= 2:
Test.check(Test.Params(testCallback = ConsoleReporter(2)), p)
You can of course also set your own callback. Or you could investigate
the result object you get back from Test.check programmatically, it
contains the original arguments along with other info.
>> 2) Add a precondition to the property:
> So the shrinker knows about those? Or is it just that at that level,
> the test would not be considered failed? That means essentially
> writing preconditions down two times, not nice.
Yes, the preconditions are baked into the properties, and if the
precondition isn't fulfilled, the property can't fail.
The shrinker only cares for failing cases, so if you put a
precondition on your properties, those forbidden values wouldn't
cause the property to fail.
You are entirely correct that you really need to write the
precondition down twice, both on your generator and on your
property. However, you could get away with a less-than-correct
precondition on your generator. It just needs to be able
to generate "enough" values that pass the precondition. The
precondition on the property, however, should be exact.
To be honest, using suchThat on your generator can almost
always be skipped. suchThat on a generator is technically just the
same as a precondition on a property, just a "stupid" filter
that discards values.
>> 3) Use forAllNoShrink, which won't try to simplify your failing test
>> cases:
> That would work well in my case since the generated parameters are not
> really of interest, anyway; it is just confusing that forbidden values
> pop up. In general, not shrinking is not an option, I guess.
>
>> Maybe the simplifier should not count exceptions as "real" failures,
>> but that would just help in this particular type of case.
> Definitely not! As long as not expected behaviour in the given case,
> exceptions signify failure by definition.
>
>> As always, I'm open for suggestions.
> The way I see it (as a one day scalacheck user, so please bear with
> me), I have two places to specify preconditions. One is when I create
> generators, which will effectively result in rejection sampling. The
> second place is as precondition of properties, where they will result
> in successful tests when not fulfilled. You have to be careful not to
> use either wrongly; a very broad generator used on a property with a
> very restrictive might result in passed tests without executing the
> function under test even once.
This is correct, but as I said there is no reason to put the precondition
as a suchThat-expression on the generator. If you can't be more clever
about the precondition when you define the generator, you can leave it
out completely, and just handle it in the property.
Also, ScalaCheck will protect you from combining a broad generator with
a restrictive property, by telling you that it can't find enough
non-trivially passing test cases. So 100 checks were the precondition
wasn't fulfilled is not considered a passed property. Therefore, you
should never be afraid of defining your properties as completely as
possible, with preconditions and everything.
> Why not factor out preconditions and pass them to generator, property
> and shrinker independently? Ideally (for testing intended behaviour on
> the definition space), no parameters not fulfilling them should be
> generated in the first place. For simple cases, suchThat should be
> able to restrict sampling in a clever way. Maybe some predefined
> conditions can be added (such as interval boundaries) and
> automatically used "correctly". A shrinker should of course never
> reduce to a parameter that does not fulfill the preconditions. In
> ideal settings, the property need not do anything with the
> precondition. Am I missing a use case where property conditions serve
> a purpose other than filtering?
No, preconditions are just filters. But clever generators are more than
that, of course. Maybe some of the cleverness could be done by ScalaCheck,
with some kind of intelligent suchThat.
Factoring out the preconditions could maybe work, it is certainly
something worth thinking about.
> In any case, I have to put more thought in my properties. Turns out
> testing random number generators is a messy business.
>
> By the way, in which way are you in contact with the creators of
> QuickCheck? I ask because I happen to occupy the same office building
> currently and think they would be interested in new features you say
> you implemented. Also, have you heard of their latest child, QuickSpec
> ( e.g. http://portal.acm.org/citation.cfm?id=1894408 )?
I have been in contact with John and Koen some times, but not on
any regular basis. The new features in ScalaCheck is essentially
the state machine-based testing that I have implemented on top of
the core ScalaCheck API in the Commands module. It is documented
in the User Guide. This feature is however inspired by John Hughes'
implementation of Erlang QuickCheck, so it's nothing new for them :)
I've heard about QuickSpec, but never really looked it up. Thanks
for bringing it to my attention again :)
Best regards,
Rickard
Akerbos
Thanks again for your elaborate answer, it helped my understanding a lot.
> You can give the Text.check method parameters, one of which controls
> who should receive callbacks when properties are evaluated.
> ...
> Test.check(Test.Params(testCallback = ConsoleReporter(2)), p)
I see; I never wrote such code, apparently sbt takes care of that. I
will have to check where to adjust verbosity in sbt, then.
> So 100 checks were the precondition wasn't fulfilled is not
> considered a passed property.
That is comforting to know, thanks.
Still, I continue to get peculiar behaviour. Let be give you the big picture:
This is the property I am trying to check:
> property("Sampler") = forAll(posInt.combine(data)(pairComb)){ case (n, d) =>
> (!d.isEmpty && n <= d.size && n >= 0) ==> {
> ... random.sample(n, d)) ...
> }
> }
where sample refers to
> def sample[...](n : Int, data : S[T])(...) : S[T] = {
> require(n >= 0, "Sample size has to be non-negative")
> require(data.size > 0, "Data size has to be positive")
> require(data.size >= n, "Not enough data for sample size")
> ...
> }
Using these generators:
> val posInt = Gen.choose(1, 100)
> val data = Gen.listOfN(100, Arbitrary.arbString.arbitrary)
and the trivial combinator (that should probably be used by default)
> def pairComb[T,U](a : Option[T], b : Option[U]) : Option[(T,U)] =
> (a,b) match {
> case (Some(l), Some(r)) => Some(l, r)
> case _ => None
> }
And this is the test output:
> ! RichRandom.Sampler: Falsified after 0 passed tests.
> > ARG_0: (1,List())
How can that happen, given the precondition? I thought the shrinker
only considers inputs that pass the precondition? Note that the
property is not even evaluated, otherwise require would throw an
exception. Is the problem that ==> is not the outermost method?
By the way, for the purpose of testing random behaviour, it is
necessary to allow for some fuzzyness (e.g. "sample mean is within
confidence interveal in more than 95% of all cases"). Can I do this is
scalacheck or is this too far away from the paradigm and I should do
regular unit testing?
Another thing that keeps popping up is this:
> Exception raised on argument generation.
> Exception: java.lang.StackOverflowError: null
I am not sure how to interpret this. I use posInt as above together
with precondition (v > 0), so it should not be a problem of not
finding enough parameters. Or does this exception come from tested code?
Best Regards
Raphael
----------------------------------------------------------------
This message was sent using IMP, the Internet Messaging Program.
Rickard Nilsson
Sorry for the late reply.
This seems strange, actually. I have been able to reproduce the behavior,
but not found the cause for it. I will look further into it, it looks like
a bug. It's not a general thing with shrinking and preconditions, maybe it
because the tuple is simplified in several steps or something like that.
As I said, I'll look into it.
By the way, you can define your generator in a more straightforward way:
val data = for {
n <- Gen.choose(1,100)
l <- Gen.listOfN(100, Arbitrary.arbitrary[String])
} yield (n,l)
> By the way, for the purpose of testing random behaviour, it is necessary
> to allow for some fuzzyness (e.g. "sample mean is within confidence
> interveal in more than 95% of all cases"). Can I do this is scalacheck
> or is this too far away from the paradigm and I should do regular unit
> testing?
There is no direct support for this, no. But you could maybe let
ScalaCheck generate a list of values, and check your property against each
of the values, and return true if "enough" instances are true. Something
like:
def myFunction(n: Int): Boolean = ...
dev myProp = forAll { l: List[Int] =>
l.filter(myFunction).length >= 0.95*l.length
}
I don't know if this will work at all in your case, though.
> Another thing that keeps popping up is this:
>> Exception raised on argument generation.
>> Exception: java.lang.StackOverflowError: null
> I am not sure how to interpret this. I use posInt as above together with
> precondition (v > 0), so it should not be a problem of not finding
> enough parameters. Or does this exception come from tested code?
According to the message "Exception raised on argument generation.", this
is something that is caused by your generator. Maybe you have defined a
recursive generator? What is the exact definition of your generator?
Best regards,
Rickard Nilsson