Reopening the Test::More API?

[Michael G Schwern]

As vehemently as I might enforce it, I don't like the Test::More API being
closed. I originally put that policy in place because I was getting so many
requests for random test functions to be put into Test::More that it would
have turned into a massive hodge-podge. Instead I encourage people to write
their own modules.

Test::More has become the standard issue, general-purpose Perl testing module.
The Perl testing community has become more sophisticated, Test::More hasn't.

So, let's pretend that the API is now open. What now?

There are three questions to ask:

* What needs fixing?

For example, I don't particularly like how skip and todo work.

* What needs adding?

Some way to print diagnostics to STDOUT, aka note().

* What existing modules might Test::More assimilate?

CPAN modules act list Test::More's laboratory. Off the top of my head, parts
of Test::Plan and Test::Most could be sucked in.

--
We do what we must because we can.
For the good of all of us,
Except the ones who are dead.
-- Jonathan Coulton, "Still Alive"

[Ovid]

On Fri, Aug 22, 2008 at 6:26 PM, Michael G Schwern <sch...@pobox.com> wrote:

 

So, let's pretend that the API is now open.  What now?

There are three questions to ask:

* What needs fixing?

For example, I don't particularly like how skip and todo work.

Clearly we need backwards compatibility, but extending these would be great.  The following should be trivial.

Fix TODO.  The following will work regardless of namespace.  This has great benefits for OO testing libraries where a subclass package doesn't match the $TODO package.

  todo_start('Just testing TODO');
  ok 0, 'this is a todo test';
  todo_end;

 For SKIP (Schwern suggested this to me):

  if ( some_condition ) {
    skip($message, $number_of_tests);
  }
  else {
    # run some tests
  }
 

CPAN modules act list Test::More's laboratory.  Off the top of my head, parts
of Test::Plan and Test::Most could be sucked in.

I'd be happy to add failure callbacks, but I think this should be in Test::Builder, not Test::More.  For Test::More, maybe &die_on_fail and &bail_on_fail conveniences could be added, or the Test::Kit &on_fail function.

Of course, I still like the &explain function :)

Not having used Test::Plan, I can't comment on that.

Cheers,
Ovid

[Michael G Schwern]

Ovid wrote:
> Clearly we need backwards compatibility, but extending these would be
> great. The following should be trivial.
>
> Fix TODO. The following will work regardless of namespace. This has
> great benefits for OO testing libraries where a subclass package doesn't
> match the $TODO package.
>
> todo_start('Just testing TODO');
> ok 0, 'this is a todo test';
> todo_end;

I really dislike start/end functions, but I suppose it couldn't hurt to add it
in and have the option.

I like that the current implementation stacks.

> For SKIP (Schwern suggested this to me):
>
> if ( some_condition ) {
> skip($message, $number_of_tests);
> }
> else {
> # run some tests
> }

Test::More's skip() already does something else, and there's no way I know for
it to detect that it's not in a SKIP block. It would need a different name.
Suggestions?

> CPAN modules act list Test::More's laboratory. Off the top of my
> head, parts
> of Test::Plan and Test::Most could be sucked in.
>
> I'd be happy to add failure callbacks, but I think this should be in
> Test::Builder, not Test::More. For Test::More, maybe &die_on_fail and
> &bail_on_fail conveniences could be added, or the Test::Kit &on_fail
> function.

The current Test::Builder implementation can not support that reliably, but
it'll definitely be considered after Test::More is reimplemented to use TB2.

I'm dubious about making them a Test::More function, easily accessible to the
test author. The person running the test, not writing the test, knows best
what on-fail the behavior should be. Thus it should probably be controllable
via Test::Builder and environment variables.

For example:

TEST_BUILDER_ON_FAIL=die prove

Speaking of Test::Most, could you list out what you had to hack around in
Test::Builder or flat out couldn't do? Then we can try to make them easier.

> Of course, I still like the &explain function :)

Something like explain() is high on the list.

--
There will be snacks.

[Ovid]

On Fri, Aug 22, 2008 at 8:02 PM, Michael G Schwern <sch...@pobox.com> wrote:

> > todo_start('Just testing TODO');
> > ok 0, 'this is a todo test';
> > todo_end;
>
> I really dislike start/end functions, but I suppose it couldn't hurt to add it
> in and have the option.
>
> I like that the current implementation stacks.

I'm not sure about start/end functions either. The problem I see is this:

use Test::More 'no_plan';
use My::Tests;
todo_start;
My::Tests->run;

Note the lack of the 'todo_end' function. This means that someone can
call this function in &My::Tests::run and break things. Should
corresponding start/end functions be required to live in the same
package? That might mitigate the damage to the point of people being
only being able to break it by trying to break it.

> Test::More's skip() already does something else, and there's no way I know for
> it to detect that it's not in a SKIP block. It would need a different name.
> Suggestions?

&ignore? I don't like it, but my thesaurus gives me no love here.
&omit? (yuck)

Here are other ideas:

avoid, bypass, circumvent, connive, discount, disregard, elide,
forget, overlook, pretermit, slight, snub

> > I'd be happy to add failure callbacks, but I think this should be in
> > Test::Builder, not Test::More. For Test::More, maybe &die_on_fail and
> > &bail_on_fail conveniences could be added, or the Test::Kit &on_fail
> > function.
>
> The current Test::Builder implementation can not support that reliably, but
> it'll definitely be considered after Test::More is reimplemented to use TB2.

Agreed that it's not reliable, but it's *somewhat* stable and in
practice seems work rather well.

> I'm dubious about making them a Test::More function, easily accessible to the
> test author. The person running the test, not writing the test, knows best
> what on-fail the behavior should be. Thus it should probably be controllable
> via Test::Builder and environment variables.
>
> For example:
>
> TEST_BUILDER_ON_FAIL=die prove

I don't think it's that clear cut. If I'm trying to &use_ok an
optional module, it's not appropriate to force this on the consumer,
but if the core module fails &use_ok, a BAIL_OUT is not only fine, but
desired.

> Speaking of Test::Most, could you list out what you had to hack around in
> Test::Builder or flat out couldn't do? Then we can try to make them easier.

I needed easy ways of wrapping several Test::Builder methods to allow
ignoring plans (Test::Aggregate). Test::Most sets internal values for
delayed plans and 'all_done' (when you don't care about a plan but you
want to know if your program didn't exit prematurely).

You can also read The Test::Aggregate::Builder code which is where I
encapsulated (hah!) a lot of Test::Builder encapsulation violations :)

http://search.cpan.org/src/OVID/Test-Aggregate-0.32_05/lib/Test/Aggregate/Builder.pm

Cheers,
Ovid

[Michael G Schwern]

Ovid wrote:
> I'm not sure about start/end functions either. The problem I see is this:
>
> use Test::More 'no_plan';
> use My::Tests;
> todo_start;
> My::Tests->run;
>
> Note the lack of the 'todo_end' function. This means that someone can
> call this function in &My::Tests::run and break things. Should
> corresponding start/end functions be required to live in the same
> package? That might mitigate the damage to the point of people being
> only being able to break it by trying to break it.

If it was fatal to fail to end a todo_start() that would be enough to protect
against a dangling TODO. So at the end of the test run if there's anything
left on the TODO stack that would be fatal.

Would you take care of that, please?
http://code.google.com/p/test-more/issues/detail?id=10

>> Test::More's skip() already does something else, and there's no way I know for
>> it to detect that it's not in a SKIP block. It would need a different name.
>> Suggestions?
>
> &ignore? I don't like it, but my thesaurus gives me no love here.
> &omit? (yuck)
>
> Here are other ideas:
>
> avoid, bypass, circumvent, connive, discount, disregard, elide,
> forget, overlook, pretermit, slight, snub

Picking a synonym for skip() doesn't convey that it's to be used in a skip
condition rather than a skip block. Something like skip_if() comes to mind,
but doesn't quite read right.

if( $condition ) {
skip_if( 2, $reason );
}
else {
...
}

>> I'm dubious about making them a Test::More function, easily accessible to the
>> test author. The person running the test, not writing the test, knows best
>> what on-fail the behavior should be. Thus it should probably be controllable
>> via Test::Builder and environment variables.
>>
>> For example:
>>
>> TEST_BUILDER_ON_FAIL=die prove
>
> I don't think it's that clear cut. If I'm trying to &use_ok an
> optional module, it's not appropriate to force this on the consumer,
> but if the core module fails &use_ok, a BAIL_OUT is not only fine, but
> desired.

We have per-test on-fail actions already.

use_ok() || BAIL_OUT;
use_ok() || die;

It'll be available in Test::Builder2, so you can get at it if you want to do
something clever.

>> Speaking of Test::Most, could you list out what you had to hack around in
>> Test::Builder or flat out couldn't do? Then we can try to make them easier.
>
> I needed easy ways of wrapping several Test::Builder methods to allow
> ignoring plans (Test::Aggregate). Test::Most sets internal values for
> delayed plans and 'all_done' (when you don't care about a plan but you
> want to know if your program didn't exit prematurely).
>
> You can also read The Test::Aggregate::Builder code which is where I
> encapsulated (hah!) a lot of Test::Builder encapsulation violations :)
>
> http://search.cpan.org/src/OVID/Test-Aggregate-0.32_05/lib/Test/Aggregate/Builder.pm

Looking through Test::Most, there needs to be a safer way for individual
modules to set their own private attributes.

Could you write out the problems in the issue tracker, please?

--
If at first you don't succeed--you fail.
-- "Portal" demo

[David E. Wheeler]

On Aug 22, 2008, at 12:02, Michael G Schwern wrote:

>> For SKIP (Schwern suggested this to me):
>>
>> if ( some_condition ) {
>> skip($message, $number_of_tests);
>> }
>> else {
>> # run some tests
>> }
>
> Test::More's skip() already does something else, and there's no way
> I know for
> it to detect that it's not in a SKIP block. It would need a
> different name.
> Suggestions?

How about skip(&$)?

skip {
# run some tests that are skipped.
} 3;

Best,

David

[Michael G Schwern]

David E. Wheeler wrote:
> How about skip(&$)?
>
> skip {
> # run some tests that are skipped.
> } 3;

That sort of thing would solve a lot of problems, but anything of the form
"let's pretend this subroutine is a block" falls afoul of adding muck to the
stack.

Let's pretend you want to test anything that looks at caller(), here
represented by Carp::cluck().

$ perl -wle 'use Carp; sub run(&) { my $sub = shift; $sub->() } {
Carp::cluck("Here we are in a real block"); } run { Carp::cluck(q[And now
inside the 'block']) }'
Here we are in a real block at -e line 1
And now inside the block at -e line 1
main::__ANON__() called at -e line 1
main::run('CODE(0x1815a30)') called at -e line 1

cluck() knows that the run() block isn't really a block but an anonymous
subroutine called inside another subroutine. This violates the
non-interference clause of testing. [1]

Sub::Uplevel exists to get around this, but it's complicated and twitchy.
Overriding core functions is too fraught with peril for such a basic testing
construct.

[1] Your code should not behave any different in a test script then in a
normal script. Just like the (en|de)bugger should, but doesn't and how
annoying is that?

--
Whip me, beat me, make my code compatible with VMS!