Right now, with Test::Builder, you do this:

    my $ok = $builder->ok( $test, $name );
    unless( $ok ) {
        my $diag = ...;
        ...build up the diagnostic message...
        $builder->diag( $diag );
    }

    return $ok;

Or you can sometimes compress it to this:

    return $builder->ok( $test, $name ) ||
      $builder->diag( $diag );

With structured diagnostics, we need to A) strongly associated the diagnostics
with the test and B) allow an arbitrary hash.  This lead to the unsatisfactory:

    $builder->ok( $test, $name, {
        have => $have,
        want => $want
    });

I never liked this because it seems inflexible.  You have to have everything
ready in one shot.  And if we have is() or like() or more assert methods they
have to accept all the arguments, too.  After some discussion we came up with
this:

    return $builder->ok( $test )
            ->name( $name )
            ->diag({
                have => $have,
                want => $want
            });

ok() returns a result object upon which you can call more methods to add more
information about the result.  The test name, test diagnostics or whatever
else we want to tack on.  Each of those return the same result object to allow
chaining.  In boolean/string/numeric context it will return true or false to
reflect the test passing or failing.

You can do it all in one shot, chained together, as above.  Or you can do it
in pieces like this:

    my $ok = $builder->ok($test)->name($name);

    if( !$ok ) {
        my $diag = ...;
        ...build up $diag...
        $ok->diag($diag);
    }

    return $ok;

The result object will have a flush() method (better name needed) to output
the results.  Normally this will be called by the wrapper around the user's
test function, but I don't want to rely on that 100% of the time, not everyone
is going to use install_test().

The original thought was to have the object flush on destruction, but we're
going to want to store the result object in the TB2 history (ok() will do
that) so it won't actually get destroyed.  We can't store a copy because we
want to see any changes after it's stored.  Can't store a weak ref because we
do want to keep the object around.

My thought is for ok() to store the real result object in history and return a
thin wrapper object that does nothing but delegates everything to the result.
 Then when it gets destroyed it can tell the real result object to flush.

In the end, I really like the flexibility the object chaining gives.  Its
going to make it very easy to add new methods.  I don't like the complexity
and magic of determining when the result should output and would like to see
and good ideas on that.

As always, +1 and -1s appreciated.

PS  At PDX.pm we talked about an or() method which would do this:

    $builder->ok( open my $fh, $file )
            ->diag( "Errno: $!" )
            ->or
            ->is( $!, ENOBACON )
            ->name( "Insufficient bacon" );

Its a short circuit method.  If the result is true, it returns a null object
that does nothing.  If it's false, it chains the result through and the
following is() starts a new result object.  Trouble is, I can't remember what
it bought us over a regular or operator.

    $builder->ok( open my $fh, $file )
            ->diag( "Errno: $!" )
      or
    $builder->is( $!, ENOBACON )
            ->name( "Insufficient bacon" );

That would appear to be perfectly sufficient, understandable to all and you
get real short circuiting, not calling empty methods on an empty object.

On Mar 13, 2009, at 11:51 AM, Michael G Schwern wrote:

Best,

      $builder
        ->ok($test)
        ->name($name)
        ->if_not_ok( sub {
          my $diag = ...;
          # ...build up $diag...
          $_->diag($diag);
        } );

Ruby makes this syntactically much nicer, but there’s no reason
we can’t build APIs with at least the nice semantics.

The whole problem is that they have to think about this at all.  That its an
extra, unexpected step.  Doesn't matter so much what that extra step is.  That
calling $builder->ok() is not enough and you might run into odd cases where
tests will come out of order.

{
    my $ok = $builder->ok($test);
    unless( $ok ) {
        my $diag = ...;
        $ok->diag($diag);
    }

    $builder->ok( $another_test )
            ->name( $name );

It also means if a user innocently stores $ok they might not get output until
the process exits.

[1] I realize I may have lost the Europeans with this analogy.

    $builder
      ->ok($test)
      ->name($name)
      ->do( sub {
        my $ok = shift;
        if ( $ok ) {
            my $diag = ...;
            # ...build up $diag...
            $ok->diag($diag);
        }
      } );

I find that uglier, but if that’s your speed, fine. I can always
subclass it if I care.

The point is not whether you do condition checks, the point is to
invert control so the object knows at which point the user is
done with it.

In Ruby you can do the equivalent of

    File->new( @blah )->lines( sub {
        # process lines in $_ here
    } );

and have the file handle come into existence, run the code IFF
there are no errors, and disappear again. No need to write any
error checking and handling in user code and yet errors are
caught rather than silently ignored because the file handle
object knows the scope for which it is relevant.

  my $ok = $builder->ok( 1 + 1 )->name("Simple test");

  ...do a long and complicated test...

  $builder->ok( $result_of_long_and_complicated_test );

The output of the simple test will be delayed until the second one has completed.

Sub::Uplevel is my elaborate solution to this problem, but it's too shaky and
magical to put into the baseline testing system that absolutely has to be able
to test everything.

And having the test author captive inside our own little world of do() blocks
and method calls... doesn't seem Perlish.  Seems very frameworky in the bad
sense of having to learn all the special ways to do things the way the
framework wants you to do them.  I want people to be able to write Perl, not
Test::Builder.

I have a feeling I'm worrying about this corner case too much.  That it will
hardly come up and that the small caveat of making the user call ->flush when
it does isn't a big deal.  So I'm going to implement some of it and see what
happens.

Best,

And isn't this already a problem with, e.g., throws_ok and stdout_is  
and other test modules that rely on code refs?

As for a better term than flush(), how about, I'm thinking of sending  
the TAP output to its destination, yes? How about:

* commit()
* deliver()
* post()
* send()
* ship()
* issue()

I think I like ship() best. But post() is good, too, and flush() is  
probably fine.

Best,

Test::Output does not and thus you can't rely on the call stack.

#!/usr/bin/perl

use Test::More tests => 1;
use Test::Output;

sub call_stack {
    my @stack;

    my $height = 0;
    while( my @caller = (caller($height))[0..3] ) {
        push @stack, \@caller;
        $height++;
    }

    return @stack;

Note that Test::Output introduces a difference in the call stack.  The more
you try and pretend a code ref is a block the more likely users will forget
this.  If the code runs differently when being tested as in production you
have a heisenbug.  This makes hard bugs even harder to test, sometimes impossible.

Sub::Uplevel rewrites caller().  This is useful but evil and it will not be
going into Test::Builder.  The reason it can go into Test::Exception and
friends is because if it doesn't work or interferes you can just not use
Test::Exception.  But there's no choice with Test::Builder, you're stuck with
it or you hand roll your own TAP writer.  So Test::Builder must strive to
alter the environment as little as possible.

You can get around this with a goto &SUB but that only allows you one level
deep of testing functions and severely limits what your testing library can do.

This may seem like a very thin corner case, but making sure that TB can be
used to test everything is part of why it's not just *A* Perl testing system
but why its *THE* Perl testing system.