Yet Another TODO Model for Test.pm

1 view
Skip to first unread message

Stevan Little

unread,
May 3, 2005, 10:36:40 AM5/3/05
to perl6-c...@perl.org
The ongoing search to find a good model for TODO tests in Pugs
continues ...

To start with, the idea of some kind of "switch" to turn all failures
into TODOs has been removed. It was a bad idea from the start and since
it never actually was implemented, I have ditched it entirely.

Second, I would like to move away from the force_todo() usage, as it
has a number of obvious problems. However, I cannot dismiss it's
usefulness as a quick-fix during release preparation, so i will not yet
remove it.

It was determined (on #perl6 and various emails) that what we really
need in Pugs is a multi-layered TODO model. In Pugs we have several
different kinds of TODO tests, along with the plain old TODO. Here is a
list of some of the more common ones and examples of how the attributes
might look;

- TODO

:todo, :todo(1)

Sometimes, it just doesn't fit anywhere else, so in that case it is
just TODO.

- TODO bug

:todo<bug>, todo => 'bug', :todo('bug')

This is either an outright bug in Pugs, or it is a feature which
/should/ be implemented, but for some reason (usually some kind of
seemingly unrelated bug) it is not. This is a good "layer" for edge
cases tests and other such ugly corners.

- TODO feature

:todo<feature>, todo => 'feature', :todo('feature')

These are tests written for perl6 features which are not yet
implemented in Pugs. Objects is the perfect example of this.

- TODO parsefail

:todo<parsefail>, todo => 'parsefail', :todo('parsefail')

These are tests which fail to parse correctly. These are usually tested
with eval_ok() or eval_is().

- TODO hardfail

:todo<hardfail>, todo => 'hardfail', :todo('hardfail')

These are tests which parse okay, but die with an uncatchable error. I
will be experimenting with using the "is lazy" trait to catch and
not-evaluate these (and therefore not kill pugs).

My goal was to support these different kinds of TODOs with Test.pm, and
also make them visible through Test::Harness (and eventually in our
smoke test). In exploring Test::Harness I noticed that TODO tests can
also have a "reason", just like SKIP tests do (although Test::Harness
seems to ignore them) and I have exploited this feature to accomplish
my goal.

So currently (in the latest revision), Test.pm will support the
following code:

ok(bad_function(), "... testing bad_function", :todo<bug>);

this will produce the following TAP output:

not ok 1 - ... testing bad_function # TODO bug
# Failed (TODO bug) test (-e line 3, column 1-46)
# Got: undef

I have also altered our local copy of Test::Harness (in inc/) so that
if you were to run this with 'prove' or 'make test' you would see
something like this:

t/my_test_file....ok
1/5 TODO bug test
2/5 TODO tests
2/5 TODO feature tests
All tests successful.

This allows a simple top-level overview of not only the number of TODO
tests, but of each specific TODO test "layer". Ideally we can support
this level of TODO granularity in the test smoke interface as well.

Now all that is left is for people to start changing all the :todo
tests to :todo<some_attribute>.

As always comments, question and suggestions are always welcome.

Thanks,

Stevan


Nathan Gray

unread,
May 3, 2005, 12:32:53 PM5/3/05
to Stevan Little, perl6-c...@perl.org
On Tue, May 03, 2005 at 10:36:40AM -0400, Stevan Little wrote:
> The ongoing search to find a good model for TODO tests in Pugs
> continues ...
>
> To start with, the idea of some kind of "switch" to turn all failures
> into TODOs has been removed. It was a bad idea from the start and since
> it never actually was implemented, I have ditched it entirely.

Good.

> Second, I would like to move away from the force_todo() usage, as it
> has a number of obvious problems. However, I cannot dismiss it's
> usefulness as a quick-fix during release preparation, so i will not yet
> remove it.

That also sounds good. If not sure, wait a bit.

> It was determined (on #perl6 and various emails) that what we really
> need in Pugs is a multi-layered TODO model. In Pugs we have several
> different kinds of TODO tests, along with the plain old TODO. Here is a
> list of some of the more common ones and examples of how the attributes
> might look;

This sounds like a really good plan: specify why a test is TODO.

> - TODO
>
> :todo, :todo(1)
>
> Sometimes, it just doesn't fit anywhere else, so in that case it is
> just TODO.

So, vanilla.

> - TODO bug
>
> :todo<bug>, todo => 'bug', :todo('bug')
>
> This is either an outright bug in Pugs, or it is a feature which
> /should/ be implemented, but for some reason (usually some kind of
> seemingly unrelated bug) it is not. This is a good "layer" for edge
> cases tests and other such ugly corners.

So this is a test which should be working, but isn't. Still, the
definition for this one is a little hazy in my mind.

> - TODO feature
>
> :todo<feature>, todo => 'feature', :todo('feature')
>
> These are tests written for perl6 features which are not yet
> implemented in Pugs. Objects is the perfect example of this.

This is what TODO is all about: this is going to be implemented, but
isn't yet, so I'm marking it TODO until it is implemented.

So maybe the 'bug' one means: this was implemented, but now seems to be
unimplemented (because internals changed somehow, etc.).

> - TODO parsefail
>
> :todo<parsefail>, todo => 'parsefail', :todo('parsefail')
>
> These are tests which fail to parse correctly. These are usually tested
> with eval_ok() or eval_is().

So this is like 'feature', except that the parser does not understand
the syntax yet. Maybe it could be called 'syntax', which might make
more sense.

> - TODO hardfail
>
> :todo<hardfail>, todo => 'hardfail', :todo('hardfail')
>
> These are tests which parse okay, but die with an uncatchable error. I
> will be experimenting with using the "is lazy" trait to catch and
> not-evaluate these (and therefore not kill pugs).

This one is hazy in my mind, too. I'm guessing it means the syntax
parses, but is still not understood, so the interpretter dies when
trying to execute. It's the difference between understanding (the
syntax) and comprehending (the meaning). English doesn't really have
good words to describe it. Dutch uses 'verstaan' (for words/syntax) and
'begrijpen' (for comprehension/meaning).

> My goal was to support these different kinds of TODOs with Test.pm, and
> also make them visible through Test::Harness (and eventually in our
> smoke test). In exploring Test::Harness I noticed that TODO tests can
> also have a "reason", just like SKIP tests do (although Test::Harness
> seems to ignore them) and I have exploited this feature to accomplish
> my goal.

Very nice.

> So currently (in the latest revision), Test.pm will support the
> following code:
>
> ok(bad_function(), "... testing bad_function", :todo<bug>);
>
> this will produce the following TAP output:
>
> not ok 1 - ... testing bad_function # TODO bug
> # Failed (TODO bug) test (-e line 3, column 1-46)
> # Got: undef
>
> I have also altered our local copy of Test::Harness (in inc/) so that
> if you were to run this with 'prove' or 'make test' you would see
> something like this:
>
> t/my_test_file....ok
> 1/5 TODO bug test
> 2/5 TODO tests
> 2/5 TODO feature tests
> All tests successful.
>
> This allows a simple top-level overview of not only the number of TODO
> tests, but of each specific TODO test "layer". Ideally we can support
> this level of TODO granularity in the test smoke interface as well.

That looks really useful. We might just be able to leave the TODO tests
as TODO, once we have all the reasons in place.

> Now all that is left is for people to start changing all the :todo
> tests to :todo<some_attribute>.

Can you put the definitions of each attribute in one of the README
files?

> As always comments, question and suggestions are always welcome.

And gladly given. Good work, Stevan.

-kolibrie

Stevan Little

unread,
May 3, 2005, 1:37:43 PM5/3/05
to Nathan Gray, perl6-c...@perl.org
Nathan,

On May 3, 2005, at 12:32 PM, Nathan Gray wrote:
> On Tue, May 03, 2005 at 10:36:40AM -0400, Stevan Little wrote:
>> The ongoing search to find a good model for TODO tests in Pugs
>> continues ...
>>
>> To start with, the idea of some kind of "switch" to turn all failures
>> into TODOs has been removed. It was a bad idea from the start and
>> since
>> it never actually was implemented, I have ditched it entirely.
>
> Good.

Yeah it was a cheap hack, so best left undone :)

>> Second, I would like to move away from the force_todo() usage, as it
>> has a number of obvious problems. However, I cannot dismiss it's
>> usefulness as a quick-fix during release preparation, so i will not
>> yet
>> remove it.
>
> That also sounds good. If not sure, wait a bit.

Agreed. Even cheap hacks can be useful :)

>
>> It was determined (on #perl6 and various emails) that what we really
>> need in Pugs is a multi-layered TODO model. In Pugs we have several
>> different kinds of TODO tests, along with the plain old TODO. Here is
>> a
>> list of some of the more common ones and examples of how the
>> attributes
>> might look;
>
> This sounds like a really good plan: specify why a test is TODO.
>
>> - TODO
>>
>> :todo, :todo(1)
>>
>> Sometimes, it just doesn't fit anywhere else, so in that case it is
>> just TODO.
>
> So, vanilla.
>
>> - TODO bug
>>
>> :todo<bug>, todo => 'bug', :todo('bug')
>>
>> This is either an outright bug in Pugs, or it is a feature which
>> /should/ be implemented, but for some reason (usually some kind of
>> seemingly unrelated bug) it is not. This is a good "layer" for edge
>> cases tests and other such ugly corners.
>
> So this is a test which should be working, but isn't. Still, the
> definition for this one is a little hazy in my mind.

This one is a little hazy for me too. Obviously if it is a bug, then we
label it as such (most of these are in t/pugsbugs), but sometimes there
are edge cases which just have not been fleshed out enough, and this is
where this attr comes in.

And to be honest, I think it is okay that it is a little hazy, as the
reason for the failure may not always be immediately obvious to the
test developer, but a lambda-folk later on may know exactly what the
issue is. So in a way this category almost implies a certain amount of
ambiguity (especially when found outside t/pugsbugs).

>
>> - TODO feature
>>
>> :todo<feature>, todo => 'feature', :todo('feature')
>>
>> These are tests written for perl6 features which are not yet
>> implemented in Pugs. Objects is the perfect example of this.
>
> This is what TODO is all about: this is going to be implemented, but
> isn't yet, so I'm marking it TODO until it is implemented.
>
> So maybe the 'bug' one means: this was implemented, but now seems to be
> unimplemented (because internals changed somehow, etc.).

Yes, internals change so often lately (with Autrijus's speed-ups and
such) and they are likely to change more as the OO model gets put in
too.

>
>> - TODO parsefail
>>
>> :todo<parsefail>, todo => 'parsefail', :todo('parsefail')
>>
>> These are tests which fail to parse correctly. These are usually
>> tested
>> with eval_ok() or eval_is().
>
> So this is like 'feature', except that the parser does not understand
> the syntax yet. Maybe it could be called 'syntax', which might make
> more sense.

Sometimes it is a 'feature', other times it is just some odd
combination/ordering that the parser cannot comprehend. These types of
TODOs tend to not stay around for very long either.

>
>> - TODO hardfail
>>
>> :todo<hardfail>, todo => 'hardfail', :todo('hardfail')
>>
>> These are tests which parse okay, but die with an uncatchable error. I
>> will be experimenting with using the "is lazy" trait to catch and
>> not-evaluate these (and therefore not kill pugs).
>
> This one is hazy in my mind, too. I'm guessing it means the syntax
> parses, but is still not understood, so the interpretter dies when
> trying to execute. It's the difference between understanding (the
> syntax) and comprehending (the meaning). English doesn't really have
> good words to describe it. Dutch uses 'verstaan' (for words/syntax)
> and
> 'begrijpen' (for comprehension/meaning).

This is one Autrijus suggested. It is basically this:

Pugs can parse the code.
Pugs can execute the code.
An error is thrown which Pugs cannot catch.

There is still a difference between Pugs-catchable errors and
non-Pugs-catchable errors. As I said too, I am going to experiment with
the "is lazy" trait and this attr to see if we cannot avoid Pugs dying.

Well once they succeed, they will show up as unexpected successes.

>
>> Now all that is left is for people to start changing all the :todo
>> tests to :todo<some_attribute>.
>
> Can you put the definitions of each attribute in one of the README
> files?

I will put these common ones in there yes. However, the nice aspect of
using the TODO "reason" attribute is that I am not limiting us to a
strict sub-set of TODO attributes. So if you so desire, you can make
them up on your own (or write them in Dutch :)).

>
>> As always comments, question and suggestions are always welcome.
>
> And gladly given. Good work, Stevan.

Danke :)

And thanks for the comments as well.

Stevan

Nathan Gray

unread,
May 4, 2005, 12:00:38 PM5/4/05
to Stevan Little, perl6-c...@perl.org

Ah, flexibility! Very nice.

-kolibrie

Reply all
Reply to author
Forward
0 new messages