Test::Builder vs Test::Builder2 flow diagram

[Michael G Schwern]

I made some ASCII diagrams of how test results flow from your test program to
your screen the TB1 way and the TB2 way.

First, the Test::Builder way

.-------.
| foo.t |
'-------'
|
.-------------. | .----------------.
| Test::More |<--------->| Test::Whatever |
'-------------' '----------------'
| |
| .---------------. |
'---->| Test::Builder |<----'
'---------------'
|
v
.-----.
| TAP |
'-----'
|
v
.---------------.
| Test::Harness |
'---------------'

Test modules call Test::Builder asserts, TAP comes out. Test::Builder is
monolithic. It is wielded to TAP.

Now the TB2 way.

.-------.
.----------------| foo.t |-------------------.
| '-------' |
| | |
| | |
v v v
.------------. .----------------. .------------------.
| Test::More | | Test::Whatever | | Test::NotUpdated |
'------------' '----------------' '------------------'
| | |
| v v
| .----------------. .---------------.
'---------->| Test::Builder2 |<------| Test::Builder |
'----------------' '---------------'
|
v
.--------------. .-------------.
| TB2::History |<---| TB2::Result |
'--------------' '-------------'
|
|
.--------------------------. | .---------------------.
| TB2::Formatter::TAP::v13 |<-----'------>| TB2::Formatter::GUI |
'--------------------------' '---------------------'
| |
v |
.-------------------------------. |
| TB2::Formatter::Streamer::TAP | |
'-------------------------------' |
| |
v |
.-----. |
| TAP | |
'-----' |
| |
v v
.---------------. .-----------------.
| Test::Harness | | Pretty Pictures |
'---------------' '-----------------'

TB2 produces Result objects which are stored in an attached History object and
handed off to a Formatter. The Formatter takes care of turning a result into
something you see.

The example above illustrates two possible formatters. One is the traditional
TAP formatter showing that we can swap out formatters to support different
versions of TAP and change the streamer to change where the TAP actually goes
(STDOUT/STDERR or maybe a file or hell, a Speak and Spell) Another is a (very
much vapor) GUI formatter which takes Result objects and renders them as
graphics. It could be just a moving red/green line or it could hook into a
full blown testing GUI. TB2 doesn't care, it just hands it result objects and
informs it of a few other events like the test starting and ending.

It also shows a Test module that is still using TB1 and that it all fits together.

The relationship between TB1 and TB2 is not as simple as shown. TB1 is not a
simple wrapper around TB2. It doesn't use TB2 directly, most of the TB1 code
is unchanged, but rather they share History and Formatter objects.

.-----. .-----.
| TB2 | | TB1 |
'-----' '-----'
| |
| |
v v
.-------------. .--------------. .-------------.
| TB2::Result |------->| TB2::History |<--------| TB2::Result |
'-------------' '--------------' '-------------'
| |
| |
| .----------------. |
'------------->| TB2::Formatter |<--------------'
'----------------'
|
v
.--------.
| Output |
'--------'

If a test module is written with TB1, TB1 produces a Result, hands it to
History and then the Formatter. If a test module is written with TB2, TB2
produces a Result and hands it to the same History and Formatter objects.

This setup allows the output and history to remain coordinated without the two
builders having to be coupled. This allows radically different builders to be
developed that continue to work together.

What it doesn't allow is event coordination. If you want to write an
end-of-assert action, like die-on-fail, you can do it to TB2 but any modules
still using TB1 won't be informed. I can mostly fix that for TB1, by more
closely coupling it with TB2 [1], but not for unrelated builders.

[1] TB1 is currently serving as a use case for "other builders"

--
"I went to college, which is a lot like being in the Army, except when
stupid people yell at me for stupid things, I can hit them."
-- Jonathan Schwarz

[David E. Wheeler]

This is awesome. A few thoughts:

1. I see now why the formatting classes should decide how to display the different result states. I think you're right.

2. Maybe it shouldn't be called Test::Builder 2. It's clearly a completely different beast.

3. Blog this. Now.

Best,

David

[Michael G Schwern]

On 2010.9.8 8:40 AM, David E. Wheeler wrote:
> This is awesome. A few thoughts:
>
> 1. I see now why the formatting classes should decide how to display the different
> result states. I think you're right.

I'm glad it was illustrative. And I don't like that I'm right. Its easier on
the end user if TB2 is authoritative about the matter. Or at least for the
user to have a simple way of knowing if all tests are passing (or the not
quite equivalent "did anything fail").

> 2. Maybe it shouldn't be called Test::Builder 2. It's clearly a completely different beast.

It serves the same function, works very much the same for shallow uses, but
the guts are radically altered. It is the clear successor. The package name
is literally Test::Builder2 so there's no namespace collision. There's not
much point in calling it Test::Assembler or something obfuscatory if the idea
is for TB1 users to find and use TB2.

> 3. Blog this. Now.

Ok mom.

--
151. The proper way to report to my Commander is "Specialist Schwarz,
reporting as ordered, Sir" not "You can't prove a thing!"
-- The 213 Things Skippy Is No Longer Allowed To Do In The U.S. Army
http://skippyslist.com/list/

[David E. Wheeler]

On Sep 8, 2010, at 2:03 PM, Michael G Schwern wrote:

> I'm glad it was illustrative. And I don't like that I'm right. Its easier on
> the end user if TB2 is authoritative about the matter. Or at least for the
> user to have a simple way of knowing if all tests are passing (or the not
> quite equivalent "did anything fail").

I don't like it, either, but since it seems like different formats can interpret things in different ways, it makes sense. Perhaps there should be a canonical representation in the results that's recommended and tries to cover all the bases, but the formatters are free to make a different choice if they think their users expect it?

> It serves the same function, works very much the same for shallow uses, but
> the guts are radically altered. It is the clear successor. The package name
> is literally Test::Builder2 so there's no namespace collision. There's not
> much point in calling it Test::Assembler or something obfuscatory if the idea
> is for TB1 users to find and use TB2.

It seems pretty clear from your graphs that it is not a replacement, that T::B will in fact run on top of T::B2 result objects, yes? It's not T::B, then, it's something lower. At any rate, if it's not "Test::Builder version 2," then it really ought to have a different name IMHO.

>> 3. Blog this. Now.
>
> Ok mom.

Good job, son.

Best,

Davi