How to debug generators?

[sebastien]

Hi,

I made a mistake with a custom generators and it invoked
Gen.choose(min, max) with min > max sometimes. This caused ScalaCheck
(1.9) to discard many tests. It was not obvious to find the mistake
because 'fail' is defined as:

def fail[T]: Gen[T] = Gen(p => None)

which silently discards the error.

Since fail is defined in Gen's companion object it is not possible to
adapt the default behavior. Maybe I missed something but a method like
Gen#debug("label") to let a generator print generated values and
report failures through the test callback reporter would be welcome.

Otherwise I'm very happy with the framework, it already saved me
countless of bugs.

Thanks,
Sebastien

[Rickard Nilsson]

Hi Sebastien,

Den 2011-09-06 19:03:35 skrev sebastien <sebasti...@gmail.com>:

> I made a mistake with a custom generators and it invoked
> Gen.choose(min, max) with min > max sometimes. This caused ScalaCheck
> (1.9) to discard many tests. It was not obvious to find the mistake
> because 'fail' is defined as:
>
> def fail[T]: Gen[T] = Gen(p => None)
>
> which silently discards the error.
>
> Since fail is defined in Gen's companion object it is not possible to
> adapt the default behavior. Maybe I missed something but a method like
> Gen#debug("label") to let a generator print generated values and
> report failures through the test callback reporter would be welcome.

I see your point. The trouble is that Gen.fail really isn't an indication
of an exceptional situation. It is perfectly fine for generators not to
generate a value during some circumstances. We wouldn't want ScalaCheck to
abort the tests and fail the property in such cases. However, in the case
of choose(min,max), you will get a completely useless generator if you
provide min > max. Therefore I think it would make more sense not to
return Gen.fail, but rather throw an assertion exception in that case.
That way, ScalaCheck would abort testing, and would report a sensible
message, too (the exception's message).

There are other generators that also use fail when they maybe rather
should throw an exception. For example Gen.oneOf when it is given an empty
list, or Gen.pick when the arguments doesn't make sense together.

Do you think this is a good solution? I think a Gen.debug method would be
very similar to just throwing an exception. And ScalaCheck already handles
exceptions fine, so why not just use them.

You might still run into situations where a lot of tests are discarded
without any obvious reason, if you have preconditions on your properties
for example, but that is hard to get around. There are some issues on
GitHub regarding generators and preconditions, though. I think there
definitely is room for improvements in that area.

> Otherwise I'm very happy with the framework, it already saved me
> countless of bugs.

Great! I am very grateful for your feedback.

> Thanks,
> Sebastien

Regards,
Rickard

[Sébastien Bocq]

2011/9/6 Rickard Nilsson <rick...@gmail.com>

Hi Sebastien,

Den 2011-09-06 19:03:35 skrev sebastien <sebasti...@gmail.com>:

I made a mistake with a custom generators and it invoked
Gen.choose(min, max) with min > max sometimes. This caused ScalaCheck
(1.9) to discard many tests. It was not obvious to find the mistake
because 'fail' is defined as:

def fail[T]: Gen[T] = Gen(p => None)

which silently discards the error.

Since fail is defined in Gen's companion object it is not possible to
adapt the default behavior. Maybe I missed something but a method like
Gen#debug("label") to let a generator print generated values and
report failures through the test callback reporter would be welcome.

I see your point. The trouble is that Gen.fail really isn't an indication of an exceptional situation. It is perfectly fine for generators not to generate a value during some circumstances. We wouldn't want ScalaCheck to abort the tests and fail the property in such cases. However, in the case of choose(min,max), you will get a completely useless generator if you provide min > max. Therefore I think it would make more sense not to return Gen.fail, but rather throw an assertion exception in that case. That way, ScalaCheck would abort testing, and would report a sensible message, too (the exception's message).

There are other generators that also use fail when they maybe rather should throw an exception. For example Gen.oneOf when it is given an empty list, or Gen.pick when the arguments doesn't make sense together.

Do you think this is a good solution? I think a Gen.debug method would be very similar to just throwing an exception. And ScalaCheck already handles exceptions fine, so why not just use them.

Since you are threading params through the generators anyway, it seems more flexible to call an error method inside the call to 'parameterized' in choose. The default might be to throw an exception. The idea behind Gen.debug is different. The goal is not to see these errors but to print values generated by an intermediate generator. I'm thinking to something like this for example:

for {
      n <- Gen.choose(sz/3, sz/2)

      c <- Gen.vectorOf(n, sizedTree(sz/2)).debug("label")

    } yield Internal(c)

The call to debug could simply return a new generator that prints the value returned by the target generator prefixed with the label before returning itself the value.

Note that I gave only a cursory glance at the sources of the framework, so my advice should be taken as mere suggestions.
 

You might still run into situations where a lot of tests are discarded without any obvious reason, if you have preconditions on your properties for example, but that is hard to get around. There are some issues on GitHub regarding generators and preconditions, though. I think there definitely is room for improvements in that area.

Yes, I knew I had a problem with a generator once I replaced my property check with true and saw that many tests were still being discarded.

Regards,
Sebastien

[Rickard Nilsson]

Hi Sebastien,

Sorry for my very late reply.

So you would like to have the label printed _each_ time the generator is
evaluated, even if the property doesn't fail? Then you are correct that
exceptions wouldn't work out. But you would get lots of output printed to
the console. You can do this yourself if you want, just put a "println"
statement inside your generator. But if it was built into ScalaCheck, the
output could maybe be formatted a bit more nicely.

If you want to investigate properties with lots of discarded tests, you
can also use Prop.collect. It collects the generated values that you tell
it to collect, and in the end of the test round, ScalaCheck will present
statistics on the collected values. Check out the section "Collecting
generated test data" in the User Guide:
http://code.google.com/p/scalacheck/wiki/UserGuide

Best regards,
Rickard Nilsson

[Sébastien Bocq]

2011/10/3 Rickard Nilsson <rickard...@telia.com>

Hi Sebastien,

Sorry for my very late reply.

 

No problem.
 

Yes, that would be only for debugging. I agree that it is possible to use println (this is what I did) but it is quite intrusive. See how it affects the simple example already:

for {
     n <- Gen.choose(sz/3, sz/2)

     c <- {
              println("n:" + n)
              Gen.vectorOf(n, sizedTree(sz/2))
             }
     } yield {
       println("label:" + c)
       Internal(c)
     }

   

If you want to investigate properties with lots of discarded tests, you can also use Prop.collect. It collects the generated values that you tell it to collect, and in the end of the test round, ScalaCheck will present statistics on the collected values. Check out the section "Collecting generated test data" in the User Guide: http://code.google.com/p/scalacheck/wiki/UserGuide

 
Collect gathers data on the resulting generator used in a property but not the intermediate ones. Now 'debug' could have a similar output than collect to summarise debug information.

Kind regards,
Sebastien

[Rickard Nilsson]

Den 2011-10-05 21:00:31 skrev Sébastien Bocq <sebasti...@gmail.com>:

> 2011/10/3 Rickard Nilsson <rickard...@telia.com>

>
>>
>> So you would like to have the label printed _each_ time the generator is
>> evaluated, even if the property doesn't fail? Then you are correct that
>> exceptions wouldn't work out. But you would get lots of output printed
>> to
>> the console. You can do this yourself if you want, just put a "println"
>> statement inside your generator. But if it was built into ScalaCheck,
>> the
>> output could maybe be formatted a bit more nicely.
>>
>
> Yes, that would be only for debugging. I agree that it is possible to use
> println (this is what I did) but it is quite intrusive. See how it
> affects
> the simple example already:
>
> for {
> n <- Gen.choose(sz/3, sz/2)
> c <- {
> println("n:" + n)
> Gen.vectorOf(n, sizedTree(sz/2))
> }
> } yield {
> println("label:" + c)
> Internal(c)
> }

Yes, I agree that a debug method on the generator, even if only a
println, would be nicer.

>> If you want to investigate properties with lots of discarded tests, you
>> can
>> also use Prop.collect. It collects the generated values that you tell
>> it to
>> collect, and in the end of the test round, ScalaCheck will present
>> statistics on the collected values. Check out the section "Collecting

>> generated test data" in the User Guide: http://code.google.com/p/**
>> scalacheck/wiki/UserGuide<http://code.google.com/p/scalacheck/wiki/UserGuide>

>>
>>
> Collect gathers data on the resulting generator used in a property but
> not
> the intermediate ones. Now 'debug' could have a similar output than
> collect
> to summarise debug information.

That is a very good idea. It would be nice to throw in 'collect' on
generators to. Unfortunately, there are no values "threaded"
through the generators like there are for properties, so we can't
collect the values. This really calls for rewriting both Gen and
Prop into proper state monads. Doing that would remove the ugly bits
of mutable state that does exist in Gen today. Hmm, I see a goal
for ScalaCheck 2.0 :)

> Kind regards,
> Sebastien

/ Rickard

[Sébastien Bocq]

2011/10/5 Rickard Nilsson <rick...@gmail.com>

That is even more than what I asked for. I'll be looking forward to it :)
--
Sébastien