die-on-fail and Test::NoWarnings in Test::Builder2

[Michael G Schwern]

At the QA hackathon rafl and I did a code sprint to make the use case of
Test::NoWarnings work with Test::Builder2. This involved a change in the
structure of Test::Builder2 to allow extensions to hook into test events.

We came up with this list of basic testing events:
assert_start a test function starts (example: is() is entered)
assert_end a test function ends (example: is() returns)
stream_start testing starts (example: plan is called)
stream_end testing ends (example: done_testing() is called)
set_plan the plan is set (example: plan( tests => 42 ) or
done_testing(42))
set_history the history object is changed
set_formatter the formatter object is changed

Rather than implement them as formal hooks, which requires designing a whole
event system, extension authors attach functionality to TB2 methods with
method modifiers. All of the above were implemented as TB2 methods.

Here's an example of how you'd implement die-on-fail.

{
# Declare yourself as a role
package TB2::Assert;
use Mouse::Role;

# Your role has a modified assert_end() method
after assert_end => sub {
my $self = shift;
my $result = shift;

die "Test said to die" if $result->name =~ /\b die \b/x;
};

# Apply this role to the Test::Builder2 singleton
require Test::Simple;
TB2::Assert->meta->apply(Test::Simple->builder);
}

By implementing event hooks as method modifiers in roles you can apply an
extension to a whole class or just a single instance of an object, in this
case the TB2 singleton.

Here's a basic Test::NoWarnings.

{
package Test::NoWarnings;
use Mouse::Role;

my @Warnings;

# When the tests start, install a warn handler to trap
# warnings.
before stream_start => sub {
$SIG{__WARN__} = sub {
push @Warnings, @_;
warn @_;
};
};

# Trap an attempt to change the number of tests and add one
around "set_plan" => sub {
my $orig = shift;
my $self = shift;
my %args = @_;

$args{tests}++ if defined $args{tests};

$self->$orig(%args);
};

# When all the tests are done, do one more test to check
# there were no warnings.
before stream_end => sub {
my $self = shift;
$self->ok( !@Warnings, "no warnings" );
};

# Apply our role to the TB2 singleton
require Test::Simple;
Test::NoWarnings->meta->apply(Test::Simple->builder);
}

Because they are roles, and because they are using method modifiers,
TB2::Assert and Test::NoWarnings can be used together in the same process.

I'm very happy with this system, and it was very easy due to basing TB2 on Mouse.

--
52. Not allowed to yell "Take that Cobra" at the rifle range.
-- The 213 Things Skippy Is No Longer Allowed To Do In The U.S. Army
http://skippyslist.com/list/

[Nick Cleaton]

On Apr 12, 5:43 pm, Michael G Schwern <schw...@pobox.com> wrote:
> At the QA hackathon rafl and I did a code sprint to make the use case of
> Test::NoWarnings work with Test::Builder2.  This involved a change in the
> structure of Test::Builder2 to allow extensions to hook into test events.
>
> We came up with this list of basic testing events:
> assert_start    a test function starts (example: is() is entered)
> assert_end      a test function ends   (example: is() returns)
> stream_start    testing starts (example: plan is called)
> stream_end      testing ends (example: done_testing() is called)
> set_plan        the plan is set (example: plan( tests => 42 ) or
>                  done_testing(42))
> set_history     the history object is changed
> set_formatter   the formatter object is changed

Nice.

One detail I'd change though - assert_start returns an Assert object,
which is a helper class to accumulate the ingredients for the result
object that will be need at the end of the assert.

my $assert = $class->builder->assert_start;

$assert->add_diag(...);

$assert->end(...);

This allows start-without-end to be cleanly trapped and made into a
failed test. Also, You can have more than one assert object active
at a time, think running several IO-bound tests in parallel under
POE.

[Nick Cleaton]

On Mon, 2010-04-12 at 17:43 +0200, Michael G Schwern wrote:
> At the QA hackathon rafl and I did a code sprint to make the use case of
> Test::NoWarnings work with Test::Builder2. This involved a change in the
> structure of Test::Builder2 to allow extensions to hook into test events.
>
> We came up with this list of basic testing events:
> assert_start a test function starts (example: is() is entered)
> assert_end a test function ends (example: is() returns)
> stream_start testing starts (example: plan is called)
> stream_end testing ends (example: done_testing() is called)
> set_plan the plan is set (example: plan( tests => 42 ) or
> done_testing(42))
> set_history the history object is changed
> set_formatter the formatter object is changed
>

> Rather than implement them as formal hooks, which requires designing a whole
> event system, extension authors attach functionality to TB2 methods with
> method modifiers. All of the above were implemented as TB2 methods.

Nice.

One thing I'd change: have assert_start() return an Assert object, a
helper class for accumulating the results of the assert into a Result
object:

my $assert = $class->builder->assert_start;

$assert->diag(...);

$assert->end(...);

This allows several asserts to be in progress at the same time in a
single thread, think running asserts in parallel under POE.

> Because they are roles, and because they are using method modifiers,
> TB2::Assert and Test::NoWarnings can be used together in the same process.
>
> I'm very happy with this system, and it was very easy due to basing TB2 on Mouse.

So, extensions will have to be written in Test::Builder::Mouse ?

[Shawn M Moore]

Not necessarily, but it will make extensions cleaner and more robust.

If you eschew Mouse you will have to reimplement the subclass creation,
population, and rebless logic. Also by doing this, you may break other
extensions that come to depend on Mouse controlling the instance/role
application process.

Shawn

[Michael G Schwern]

Nick Cleaton wrote:
> One detail I'd change though - assert_start returns an Assert object,
> which is a helper class to accumulate the ingredients for the result
> object that will be need at the end of the assert.
>
> my $assert = $class->builder->assert_start;
>
> $assert->add_diag(...);
>
> $assert->end(...);
>
> This allows start-without-end to be cleanly trapped and made into a
> failed test. Also, You can have more than one assert object active
> at a time, think running several IO-bound tests in parallel under
> POE.

I don't really understand. I sort of get how assert_start/end might become
trouble with in-process multi-tasking like POE. If nothing else the TB2
singleton can only track a single stack of test function calls at a time, and
POE could generate several in parallel.

It might not be clear, but the only people who would be actually calling
$builder->assert_start() directly would be doing something like writing their
own way of installing test functions. Like if you wanted to implement your
own version of install_test(). Its unlikely to be called by a test module or
extension author. You certainly wouldn't be using it in the same use case
where you'd want to alter the result.

For everyone else, assert_start/end are wrapped around your test function by
install_test() and called for you.

--
Stabbing you in the face for your own good.

[Nick Cleaton]

On Tue, 2010-04-13 at 21:06 +0200, Michael G Schwern wrote:
> > my $assert = $class->builder->assert_start;
> >
> > $assert->add_diag(...);
> >
> > $assert->end(...);
> >
> > This allows start-without-end to be cleanly trapped and made into a
> > failed test. Also, You can have more than one assert object active
> > at a time, think running several IO-bound tests in parallel under
> > POE.
>
> I don't really understand. I sort of get how assert_start/end might become
> trouble with in-process multi-tasking like POE. If nothing else the TB2
> singleton can only track a single stack of test function calls at a time, and
> POE could generate several in parallel.

Yup, that's what I was getting at.

> It might not be clear, but the only people who would be actually calling
> $builder->assert_start() directly would be doing something like writing their
> own way of installing test functions. Like if you wanted to implement your
> own version of install_test(). Its unlikely to be called by a test module or
> extension author. You certainly wouldn't be using it in the same use case
> where you'd want to alter the result.
>
> For everyone else, assert_start/end are wrapped around your test function by
> install_test() and called for you.

So install_test would wrap the call to the coderef in an eval, and if it
dies rethrow after calling assert_end, to avoid the possibility of an
unmatched assert_start ?

It's not a big deal, but for me the caller information for the test
function is an attribute of the in-progess assert; storing each one in
an Assert object seems a bit more natural than having a stack of them in
the builder.

[Michael G Schwern]

Nick Cleaton wrote:
>> For everyone else, assert_start/end are wrapped around your test function by
>> install_test() and called for you.
>
> So install_test would wrap the call to the coderef in an eval, and if it
> dies rethrow after calling assert_end, to avoid the possibility of an
> unmatched assert_start ?

I hadn't thought of that. Yes, it would be better to have the Assert object
serve as a guard in case of death. Much less hassle in the long run than
messing with eval.

The explicit call to assert_end() is still necessary as it has to be handed
the result and decide if its time to format (print) it. This enables test
authors to write wrappers around other test functions, augment the result, and
not fool around with $Level.

sub is {
my($have, $want, $name) = @_;

my $result = ok( $have eq $want, $name );
$result->diagnostics([
have => $have,
want => $want
]);

return $result;
}

You don't want the result printed by ok(), this allows is() the opportunity to
add information to the result. (I'm working on implementing that at the moment).

To sketch out the install_test() wrapper, something like this?

*{$test_function} = sub {
my $assert = $Builder->assert_start();
my $result = $test_code->(@_);
$assert->end($result);

return $result;
};

> It's not a big deal, but for me the caller information for the test
> function is an attribute of the in-progess assert; storing each one in
> an Assert object seems a bit more natural than having a stack of them in
> the builder.

Some sort of stack still needs to be maintained, as an assert has to know if
its the last one in the stack and should format the result. Its also
necessary to know where the top of the stack is in order to output proper
failure file/line numbers (see TB2->from_top).

Thoughts on how to accomplish that?

--
Insulting our readers is part of our business model.
http://somethingpositive.net/sp07122005.shtml

[Nick Cleaton]

On Wed, 2010-04-14 at 23:56 +0200, Michael G Schwern wrote:
> Nick Cleaton wrote:
> >> For everyone else, assert_start/end are wrapped around your test function by
> >> install_test() and called for you.
> >
> > So install_test would wrap the call to the coderef in an eval, and if it
> > dies rethrow after calling assert_end, to avoid the possibility of an
> > unmatched assert_start ?
>
> I hadn't thought of that. Yes, it would be better to have the Assert object
> serve as a guard in case of death. Much less hassle in the long run than
> messing with eval.
>
> The explicit call to assert_end() is still necessary as it has to be handed
> the result and decide if its time to format (print) it. This enables test
> authors to write wrappers around other test functions, augment the result, and
> not fool around with $Level.
>
> sub is {
> my($have, $want, $name) = @_;
>
> my $result = ok( $have eq $want, $name );
> $result->diagnostics([
> have => $have,
> want => $want
> ]);
>
> return $result;
> }
>
> You don't want the result printed by ok(), this allows is() the opportunity to
> add information to the result. (I'm working on implementing that at the moment).

But sometimes you do want asserts within asserts to be independent with
their own output and history entries:

lives_ok { ok foo(), "foo() is true" } "foo lives";

There needs to be a way for assert code to tell TB2 which behaviour is
required when they cause other asserts to be executed.

> To sketch out the install_test() wrapper, something like this?
>
> *{$test_function} = sub {
> my $assert = $Builder->assert_start();
> my $result = $test_code->(@_);
> $assert->end($result);
>
> return $result;
> };

Maybe the $assert should be passed to the inner function as a parameter;
if it holds a reference to the builder then the inner function can use
it for all its interactions with Test::Builder2

In object terms, an assert is a test that is ongoing and a result is a
test that has finished. Maybe the assert should use the Assert object
to build up the results and diagnostics of the test, and the Assert
should get coerced into a Result at the end ? Hmm, not sure.

> Some sort of stack still needs to be maintained, as an assert has to know if
> its the last one in the stack and should format the result. Its also
> necessary to know where the top of the stack is in order to output proper
> failure file/line numbers (see TB2->from_top).
>
> Thoughts on how to accomplish that?

OK, I think this works:

(1) modules that define asserts via install_test() record their package
names in a central registry.

(2) the TB2 singleton holds a stack of weak references to Assert
objects.

(3) on entering one of the subs installed by install_test(), TB2
generates the Assert object as follows: If there is an Assert in the
stack and the caller information for this sub points to a registered
test module's package then this is an assert called directly from within
another assert, and the Assert at the top of the stack is used for it.
Otherwise it's an independent test in a user code callback, and a new
Assert is created and pushed onto the stack.

The Assert object stores caller details, which are sampled when it is
created. Wrapped asserts don't cause a new Assert object to be
generated, so the caller details will still refer to the caller of the
outer predicate when the Assert is processed into a Result.

A variation of your is() around ok() example:

package Test::Bar;
use Test::Foo qw(foo_ok);

...->install_test(bar_ok => sub {
my ($assert, ...) = @_;

foo_ok( ... );
$assert->diagnostic(...);
});

This would work even if Test::Foo is written to use Test::Builder
rather than TB2, which I like.

The logic in (3) ensures it does the right thing when wrapping a TB1
thing such as lives_ok() which calls back to end user code that might
run tests. By contrast, wrapping lives_ok() correctly under
Test::Builder involves some hoops to jump through:

sub my_lives_ok {
my ($code, $name) = @_;

local $Test::Builder::Level = $Test::Builder::Level + 1;
return lives_ok {
local $Test::Builder::Level = 1;
$code->();
} $name;
}

[Michael G Schwern]

I'm writing up complete notes on the TB2 design in anticipation of taking a
final stab at a release later this month before TPF breaks my legs. I'm
trying to get it all out of my head and into public to let other people help
out before its complete.
http://piratepad.net/MvXoZgnrNB

Sorry I let the discussion fall on the floor. WRT the die-on-fail assert
stack issue I've basically gone with Nick's stack of stacks approach to asserts.

I'd like your input and edits.