Test::Builder calling convention

5 views
Skip to first unread message

Michael G Schwern

unread,
Mar 13, 2009, 2:51:04 PM3/13/09
to test-mo...@googlegroups.com, PDX PM
At the last PDX.pm meeting we had a breakthrough on how a testing library says
that they've run a test.

Right now, with Test::Builder, you do this:

my $ok = $builder->ok( $test, $name );
unless( $ok ) {
my $diag = ...;
...build up the diagnostic message...
$builder->diag( $diag );
}

return $ok;

Or you can sometimes compress it to this:

return $builder->ok( $test, $name ) ||
$builder->diag( $diag );

With structured diagnostics, we need to A) strongly associated the diagnostics
with the test and B) allow an arbitrary hash. This lead to the unsatisfactory:

$builder->ok( $test, $name, {
have => $have,
want => $want
});

I never liked this because it seems inflexible. You have to have everything
ready in one shot. And if we have is() or like() or more assert methods they
have to accept all the arguments, too. After some discussion we came up with
this:

return $builder->ok( $test )
->name( $name )
->diag({
have => $have,
want => $want
});

ok() returns a result object upon which you can call more methods to add more
information about the result. The test name, test diagnostics or whatever
else we want to tack on. Each of those return the same result object to allow
chaining. In boolean/string/numeric context it will return true or false to
reflect the test passing or failing.

You can do it all in one shot, chained together, as above. Or you can do it
in pieces like this:

my $ok = $builder->ok($test)->name($name);

if( !$ok ) {
my $diag = ...;
...build up $diag...
$ok->diag($diag);
}

return $ok;

The result object will have a flush() method (better name needed) to output
the results. Normally this will be called by the wrapper around the user's
test function, but I don't want to rely on that 100% of the time, not everyone
is going to use install_test().

The original thought was to have the object flush on destruction, but we're
going to want to store the result object in the TB2 history (ok() will do
that) so it won't actually get destroyed. We can't store a copy because we
want to see any changes after it's stored. Can't store a weak ref because we
do want to keep the object around.

My thought is for ok() to store the real result object in history and return a
thin wrapper object that does nothing but delegates everything to the result.
Then when it gets destroyed it can tell the real result object to flush.

In the end, I really like the flexibility the object chaining gives. Its
going to make it very easy to add new methods. I don't like the complexity
and magic of determining when the result should output and would like to see
and good ideas on that.

As always, +1 and -1s appreciated.


PS At PDX.pm we talked about an or() method which would do this:

$builder->ok( open my $fh, $file )
->diag( "Errno: $!" )
->or
->is( $!, ENOBACON )
->name( "Insufficient bacon" );

Its a short circuit method. If the result is true, it returns a null object
that does nothing. If it's false, it chains the result through and the
following is() starts a new result object. Trouble is, I can't remember what
it bought us over a regular or operator.

$builder->ok( open my $fh, $file )
->diag( "Errno: $!" )
or
$builder->is( $!, ENOBACON )
->name( "Insufficient bacon" );

That would appear to be perfectly sufficient, understandable to all and you
get real short circuiting, not calling empty methods on an empty object.


--
If you want the truth to stand clear before you, never be for or against.
The struggle between "for" and "against" is the mind's worst disease.
-- Sent-ts'an

David E. Wheeler

unread,
Mar 13, 2009, 5:00:57 PM3/13/09
to Michael G Schwern, test-mo...@googlegroups.com, PDX PM

On Mar 13, 2009, at 11:51 AM, Michael G Schwern wrote:

> My thought is for ok() to store the real result object in history
> and return a
> thin wrapper object that does nothing but delegates everything to
> the result.
> Then when it gets destroyed it can tell the real result object to
> flush.

+1, sounds smart.

> In the end, I really like the flexibility the object chaining
> gives. Its
> going to make it very easy to add new methods. I don't like the
> complexity
> and magic of determining when the result should output and would
> like to see
> and good ideas on that.

Can you cite an example of what you dislike here, so we can sink our
teeth into something?

> As always, +1 and -1s appreciated.

+1

Best,

David

Aristotle Pagaltzis

unread,
Mar 13, 2009, 7:29:06 PM3/13/09
to test-mo...@googlegroups.com
* Michael G Schwern <sch...@pobox.com> [2009-03-13 19:55]:

> I don't like the complexity and magic of determining when the
> result should output and would like to see and good ideas on
> that.

Hmm, `undef $ok` sounds like a perfectly good explicit interface
way to flush the test to me. And variable scope is a nice
implicit interface. If you want a less manual interface, you can
always add a method or two to do this:

$builder
->ok($test)
->name($name)
->if_not_ok( sub {
my $diag = ...;
# ...build up $diag...
$_->diag($diag);
} );

Ruby makes this syntactically much nicer, but there’s no reason
we can’t build APIs with at least the nice semantics.

--
*AUTOLOAD=*_;sub _{s/(.*)::(.*)/print$2,(",$\/"," ")[defined wantarray]/e;$1}
&Just->another->Perl->hack;
#Aristotle Pagaltzis // <http://plasmasturm.org/>

Michael G Schwern

unread,
Mar 13, 2009, 10:09:08 PM3/13/09
to test-mo...@googlegroups.com
Aristotle Pagaltzis wrote:
> * Michael G Schwern <sch...@pobox.com> [2009-03-13 19:55]:
>> I don't like the complexity and magic of determining when the
>> result should output and would like to see and good ideas on
>> that.
>
> Hmm, `undef $ok` sounds like a perfectly good explicit interface
> way to flush the test to me.

That's even worse than $ok->flush. It's like having a toilet that you have to
take the lid off the tank and pull the stopper manually [1] and the plumber's
fix is to put up a sign that says "to flush open tank, pull stopper".

The whole problem is that they have to think about this at all. That its an
extra, unexpected step. Doesn't matter so much what that extra step is. That
calling $builder->ok() is not enough and you might run into odd cases where
tests will come out of order.

{
my $ok = $builder->ok($test);


unless( $ok ) {
my $diag = ...;

$ok->diag($diag);
}

$builder->ok( $another_test )
->name( $name );
}

If object destruction causes output, the second test will be output before the
first. To avoid this, the test author has to add an $ok->flush (or undef $ok
or whatever) once they're done with the object. No matter how much we
document it, this is a bug generator.

It also means if a user innocently stores $ok they might not get output until
the process exits.


> And variable scope is a nice
> implicit interface. If you want a less manual interface, you can
> always add a method or two to do this:
>
> $builder
> ->ok($test)
> ->name($name)
> ->if_not_ok( sub {
> my $diag = ...;
> # ...build up $diag...
> $_->diag($diag);
> } );

Reimplementing Perl's control structures is a road I do not want to walk down.


[1] I realize I may have lost the Europeans with this analogy.


--
...they shared one last kiss that left a bitter yet sweet taste in her
mouth--kind of like throwing up after eating a junior mint.
-- Dishonorable Mention, 2005 Bulwer-Lytton Fiction Contest
by Tami Farmer

Michael G Schwern

unread,
Mar 14, 2009, 1:25:29 AM3/14/09
to David E. Wheeler, test-mo...@googlegroups.com, PDX PM
David E. Wheeler wrote:
>> In the end, I really like the flexibility the object chaining gives. Its
>> going to make it very easy to add new methods. I don't like the
>> complexity
>> and magic of determining when the result should output and would like
>> to see
>> and good ideas on that.
>
> Can you cite an example of what you dislike here, so we can sink our
> teeth into something?

Does this help?
http://groups.google.com/group/test-more-users/msg/f55b165f4b0a3ac8


--
If at first you don't succeed--you fail.
-- "Portal" demo

Aristotle Pagaltzis

unread,
Mar 14, 2009, 8:12:42 AM3/14/09
to test-mo...@googlegroups.com
* Michael G Schwern <sch...@pobox.com> [2009-03-14 03:15]:

> The whole problem is that they have to think about this at all.
> That its an extra, unexpected step. Doesn't matter so much
> what that extra step is. That calling $builder->ok() is not
> enough and you might run into odd cases where tests will come
> out of order.
>
> {
> my $ok = $builder->ok($test);
> unless( $ok ) {
> my $diag = ...;
> $ok->diag($diag);
> }
>
> $builder->ok( $another_test )
> ->name( $name );
> }
>
> If object destruction causes output, the second test will be
> output before the first.

Make `$builder` call `flush` on the previously constructed result
object when it’s called upon to generate one (and flush the last
result object if it hasn’t been when the builder gets destroyed).
Also add a method so the user can ask for a result object without
invoking this automatism which means they assume responsibility
for flushing it at the correct time.

> Reimplementing Perl's control structures is a road I do not
> want to walk down.

Then don’t?

$builder
->ok($test)
->name($name)

->do( sub {
my $ok = shift;
if ( $ok ) {
my $diag = ...;
# ...build up $diag...
$ok->diag($diag);
}
} );

I find that uglier, but if that’s your speed, fine. I can always
subclass it if I care.

The point is not whether you do condition checks, the point is to
invert control so the object knows at which point the user is
done with it.

In Ruby you can do the equivalent of

File->new( @blah )->lines( sub {
# process lines in $_ here
} );

and have the file handle come into existence, run the code IFF
there are no errors, and disappear again. No need to write any
error checking and handling in user code and yet errors are
caught rather than silently ignored because the file handle
object knows the scope for which it is relevant.

Regards,
--
Aristotle Pagaltzis // <http://plasmasturm.org/>

Michael G Schwern

unread,
Mar 14, 2009, 4:33:49 PM3/14/09
to test-mo...@googlegroups.com
Aristotle Pagaltzis wrote:
> * Michael G Schwern <sch...@pobox.com> [2009-03-14 03:15]:
>> The whole problem is that they have to think about this at all.
>> That its an extra, unexpected step. Doesn't matter so much
>> what that extra step is. That calling $builder->ok() is not
>> enough and you might run into odd cases where tests will come
>> out of order.
>>
>> {
>> my $ok = $builder->ok($test);
>> unless( $ok ) {
>> my $diag = ...;
>> $ok->diag($diag);
>> }
>>
>> $builder->ok( $another_test )
>> ->name( $name );
>> }
>>
>> If object destruction causes output, the second test will be
>> output before the first.
>
> Make `$builder` call `flush` on the previously constructed result
> object when it’s called upon to generate one (and flush the last
> result object if it hasn’t been when the builder gets destroyed).

This smells suspiciously of "die on the next call to ok()" that Test::Most
does. Has the same problems.

my $ok = $builder->ok( 1 + 1 )->name("Simple test");

...do a long and complicated test...

$builder->ok( $result_of_long_and_complicated_test );

The output of the simple test will be delayed until the second one has completed.


> Also add a method so the user can ask for a result object without
> invoking this automatism which means they assume responsibility
> for flushing it at the correct time.

Oh, sure.

Oh god I want to make use of block syntax so hard, but I can't. This is the
universal problem in Perl: code refs are not blocks, as much as we'd like them
to be. Tests cannot make use of them because they alter the call stack. This
violates the principle that the test code should not alter the outcome of the
code. It should not introduce Heisenbugs.

Sub::Uplevel is my elaborate solution to this problem, but it's too shaky and
magical to put into the baseline testing system that absolutely has to be able
to test everything.

And having the test author captive inside our own little world of do() blocks
and method calls... doesn't seem Perlish. Seems very frameworky in the bad
sense of having to learn all the special ways to do things the way the
framework wants you to do them. I want people to be able to write Perl, not
Test::Builder.

I have a feeling I'm worrying about this corner case too much. That it will
hardly come up and that the small caveat of making the user call ->flush when
it does isn't a big deal. So I'm going to implement some of it and see what
happens.


--
s7ank: i want to be one of those guys that types "s/j&jd//.^$ueu*///djsls/sm."
and it's a perl script that turns dog crap into gold.

David E. Wheeler

unread,
Mar 15, 2009, 12:54:06 AM3/15/09
to test-mo...@googlegroups.com, PDX PM
On Mar 13, 2009, at 10:25 PM, Michael G Schwern wrote:

>> Can you cite an example of what you dislike here, so we can sink our
>> teeth into something?
>
> Does this help?
> http://groups.google.com/group/test-more-users/msg/f55b165f4b0a3ac8

Yes. Why the flush? Sorry if I just missed that.

Best,

David

David E. Wheeler

unread,
Mar 15, 2009, 1:09:03 AM3/15/09
to test-mo...@googlegroups.com
On Mar 14, 2009, at 1:33 PM, Michael G Schwern wrote:

>> Make `$builder` call `flush` on the previously constructed result
>> object when it’s called upon to generate one (and flush the last
>> result object if it hasn’t been when the builder gets destroyed).
>
> This smells suspiciously of "die on the next call to ok()" that
> Test::Most
> does. Has the same problems.
>
> my $ok = $builder->ok( 1 + 1 )->name("Simple test");
>
> ...do a long and complicated test...
>
> $builder->ok( $result_of_long_and_complicated_test );
>
> The output of the simple test will be delayed until the second one
> has completed.

For that case, the author should call $flush. But most often she won't
have to.

> Oh god I want to make use of block syntax so hard, but I can't.
> This is the
> universal problem in Perl: code refs are not blocks, as much as we'd
> like them
> to be. Tests cannot make use of them because they alter the call
> stack. This
> violates the principle that the test code should not alter the
> outcome of the
> code. It should not introduce Heisenbugs.

Couldn't the method to which the code ref is passed (lines() in
Aristotle's example) do something to keep it off the stack?

And isn't this already a problem with, e.g., throws_ok and stdout_is
and other test modules that rely on code refs?

> I have a feeling I'm worrying about this corner case too much. That
> it will
> hardly come up and that the small caveat of making the user call -
> >flush when
> it does isn't a big deal. So I'm going to implement some of it and
> see what
> happens.

That's just what I was starting to think. I also think that
Aristotle's suggestion about autoflushing on the next test is a good
idea, as it'll cover 99% of the cases, and you can provide flush() for
the other 1% (guessing at stats here, obviously).

As for a better term than flush(), how about, I'm thinking of sending
the TAP output to its destination, yes? How about:

* commit()
* deliver()
* post()
* send()
* ship()
* issue()

I think I like ship() best. But post() is good, too, and flush() is
probably fine.

Best,

David

Aristotle Pagaltzis

unread,
Mar 15, 2009, 4:02:04 AM3/15/09
to test-mo...@googlegroups.com
* David E. Wheeler <da...@kineticode.com> [2009-03-15 06:25]:

> Couldn't the method to which the code ref is passed (lines() in
> Aristotle's example) do something to keep it off the stack?

AFAIK it could only keep itself out of the call stack, but not
the sub that’s being invoked via coderef.

> And isn't this already a problem with, e.g., throws_ok and
> stdout_is and other test modules that rely on code refs?

I was wondering the same.

> As for a better term than flush(), how about, I'm thinking of
> sending the TAP output to its destination, yes? How about:
>
> * commit()
> * deliver()
> * post()
> * send()
> * ship()
> * issue()
>
> I think I like ship() best. But post() is good, too, and
> flush() is probably fine.

What happened to a simple `print`… or even `say`? :-)

Michael G Schwern

unread,
Mar 16, 2009, 5:02:06 PM3/16/09
to test-mo...@googlegroups.com
Aristotle Pagaltzis wrote:
> * David E. Wheeler <da...@kineticode.com> [2009-03-15 06:25]:
>> Couldn't the method to which the code ref is passed (lines() in
>> Aristotle's example) do something to keep it off the stack?
>
> AFAIK it could only keep itself out of the call stack, but not
> the sub that’s being invoked via coderef.
>
>> And isn't this already a problem with, e.g., throws_ok and
>> stdout_is and other test modules that rely on code refs?
>
> I was wondering the same.

Test::Exception uses Sub::Uplevel to avoid the problem.

Test::Output does not and thus you can't rely on the call stack.

#!/usr/bin/perl

use Test::More tests => 1;
use Test::Output;

sub call_stack {
my @stack;

my $height = 0;
while( my @caller = (caller($height))[0..3] ) {
push @stack, \@caller;
$height++;
}

return @stack;
}

sub print_stack {
my @stack = call_stack();
for my $caller (@stack) {
print "@$caller\n";
}
}

stdout_unlike \&print_stack, qr/Test::Output/;
note("Stack...");
print_stack();
__END__
1..1
not ok 1
# Failed test at /Users/schwern/tmp/test.plx line 25.
# STDOUT:
# main /Users/schwern/tmp/test.plx 19 main::call_stack
# Test::Output /usr/local/perl/5.10.0/lib/site_perl/5.10.0/Test/Output.pm 824
main::print_stack
# Test::Output /usr/local/perl/5.10.0/lib/site_perl/5.10.0/Test/Output.pm 221
Test::Output::stdout_from
# main /Users/schwern/tmp/test.plx 25 Test::Output::stdout_unlike
#
# matches:
# (?-xism:Test::Output)
# not expected
# Stack...
main /Users/schwern/tmp/test.plx 19 main::call_stack
main /Users/schwern/tmp/test.plx 27 main::print_stack
# Looks like you failed 1 test of 1.

Note that Test::Output introduces a difference in the call stack. The more
you try and pretend a code ref is a block the more likely users will forget
this. If the code runs differently when being tested as in production you
have a heisenbug. This makes hard bugs even harder to test, sometimes impossible.

Sub::Uplevel rewrites caller(). This is useful but evil and it will not be
going into Test::Builder. The reason it can go into Test::Exception and
friends is because if it doesn't work or interferes you can just not use
Test::Exception. But there's no choice with Test::Builder, you're stuck with
it or you hand roll your own TAP writer. So Test::Builder must strive to
alter the environment as little as possible.

You can get around this with a goto &SUB but that only allows you one level
deep of testing functions and severely limits what your testing library can do.

This may seem like a very thin corner case, but making sure that TB can be
used to test everything is part of why it's not just *A* Perl testing system
but why its *THE* Perl testing system.


>> As for a better term than flush(), how about, I'm thinking of
>> sending the TAP output to its destination, yes? How about:
>>
>> * commit()
>> * deliver()
>> * post()
>> * send()
>> * ship()
>> * issue()
>>
>> I think I like ship() best. But post() is good, too, and
>> flush() is probably fine.
>
> What happened to a simple `print`… or even `say`? :-)

Write functions with the same name as core functions at your peril.


--
Reality is that which, when you stop believing in it, doesn't go away.
-- Phillip K. Dick

Aristotle Pagaltzis

unread,
Mar 16, 2009, 8:02:12 PM3/16/09
to test-mo...@googlegroups.com
* Michael G Schwern <sch...@pobox.com> [2009-03-16 22:10]:

> Aristotle Pagaltzis wrote:
> > What happened to a simple `print`… or even `say`? :-)
>
> Write functions with the same name as core functions at your peril.

We’re talking about methods not functions.

Michael G Schwern

unread,
Mar 17, 2009, 6:33:05 PM3/17/09
to test-mo...@googlegroups.com
Aristotle Pagaltzis wrote:
> * Michael G Schwern <sch...@pobox.com> [2009-03-16 22:10]:
>> Aristotle Pagaltzis wrote:
>>> What happened to a simple `print`… or even `say`? :-)
>> Write functions with the same name as core functions at your peril.
>
> We’re talking about methods not functions.

We're talking about Perl, there's no difference.


--
Look at me talking when there's science to do.
When I look out there it makes me glad I'm not you.
I've experiments to be run.
There is research to be done
On the people who are still alive.
-- Jonathan Coulton, "Still Alive"

Reply all
Reply to author
Forward
0 new messages