Re: Doubts about TB2 and the TAP stack
Pedro Melo
(moved over from the perl-qa ML to the test-more-users ML)
On Mon, Jan 17, 2011 at 11:38 AM, Michael G Schwern <sch...@pobox.com> wrote:
> On 2010.10.29 8:36 AM, Pedro Melo wrote:
>> I'm sure I'm missing something and there is a plan to glue the Harness
>> with TB2 formatters. If some kind soul could take a little bit of time
>> to point me in the right direction (maybe there is a TAP2 tree
>> somewhere?), I could go and look and try to make use of my hacked
>> TB2::Formatter::JUnit.
>
> Sorry, this message got lost along the way. I'm glad you're interested in TB2
> and want to put together a JUnit formatter for it. Lots has changed since
> October. Fortunately it should be a fairly rote change from the old system of
> to accepting events. I'll be glad to help you fix it up for the new event
> system and bring your formatter into the fold, but probably as an example
> because of the XML::Generator dep. Or you can release it as its own CPAN
> module. We definitely need a maintained JUnit formatter.
I'm sure that I'll release whatever I end up with to CPAN as a
separate dist. Right now, I'm working with Hudson as a CI system, and
the best only takes JUnit as test report.
So I do want to have a clean way to generate JUnit from .t files.
> Also you've noticed that Test::Builder really only works with the TAP
> formatter due to all the extra methods. I'm working on tearing them apart.
> Test::Builder does use Test::Builder2::Formatter almost completely as of alpha
> 2. I still have to make diagnostics work. And then Test::More will come along.
Cool.
> Now, to whether you're missing something, you're only missing it because it
> isn't there.
>
> TAP::Harness explicitly does not know anything about what it's executing
> (beyond enough information to execute it). It just expects TAP. If you want
> to use TB2 tests with a different format you don't use TAP::Harness. [1] Ta da!
>
> What do you use? Presumably you use whatever test runner wants the
> differently formatted output. Let that run your test programs directly.
>
> This might seem like it's creating a redundant system of formatters... and in
> some way it is. But forcing (for example) a JUnit runner to instead run prove
> which then takes test results, translates them into TAP and then translates
> them into JUnit is awkward. There's a lot of information JUnit can't handle
> that TAP can, so information is lost in the translation.
hmms... But wouldn't it make sense for TB2 to emit a stream of events
with all the information it can gather, in a easy-to-parse format, and
then have the harness collect these TAP-agnostic events, and format
them as required?
It would even be backwards compatible AFAICS. For example:
1. Our harness forks each .t file with an extra file-descriptor open,
and a env var TB2_REPORT_FD with the number of the FD;
2. when the TB2 startups up, it notices the TB2_REPORTS_FD var and
enables the generic event formatter, that just writes said events to
the reports FD.
3. the harness collects such events and broadcasts them internally (so
that you can even have several formatters in parallel if you need
them);
If the .t script is invoked directly, TB2 notices that the
TB2_REPORTS_FD is not available and falls back to plain TAP.
What I'm suggesting is Tests => Events => Harness => Formatter in the
common (using prove) case. The Tests => Events is done in the .t
process space, and the Harness => Formatter is done in the harness
(prove) process space.
It seems cleaner to me, more flexible.
> Furthermore, TB1 was having difficulty adapting to TAP extensions. Extensions
> to TAP turn into pretty much different formatters, so internally it needed to
> exist anyway.
I don't get this part, not sure what you mean here... Maybe its
because TB1 did to much?
> Right now, making a TB2 test program use a different format is awkward. You
> have to do it per script. There needs to be a way to control this outside the
> script, perhaps an environment variable one can set to swap out formatters. I
> haven't been doing much convenience features, more focusing on just getting
> the design solid and things working. It is a SMOP to make your own test
> programs respond to an environment variable if you need something now.
See above.
> In the future, discussions about Test::Builder2 are more likely to be noticed
> on test-mo...@groups.google.com.
I was already subscribed to it, forgot about it :)
> [1] Not entirely true. You could, for example, use
> Test::Builder2::Formatter::Multi and attach two formatters, one which outputs
> TAP to stdout and another which writes JUnit to a file. This is handy for
> something like Hudson.
I'll check it out.
Bye,
--
Pedro Melo
http://www.simplicidade.org/
xmpp:me...@simplicidade.org
mailto:me...@simplicidade.org
--
Pedro Melo
http://www.simplicidade.org/
xmpp:me...@simplicidade.org
mailto:me...@simplicidade.org
Michael G Schwern
> I'm sure that I'll release whatever I end up with to CPAN as a
> separate dist. Right now, I'm working with Hudson as a CI system, and
> the best only takes JUnit as test report.
>
> So I do want to have a clean way to generate JUnit from .t files.
Hudson is a use case for the TB2 formatters. I'm using it, but right now I'm
using the TAP::Harness JUnit formatter.
>> This might seem like it's creating a redundant system of formatters... and in
>> some way it is. But forcing (for example) a JUnit runner to instead run prove
>> which then takes test results, translates them into TAP and then translates
>> them into JUnit is awkward. There's a lot of information JUnit can't handle
>> that TAP can, so information is lost in the translation.
>
> hmms... But wouldn't it make sense for TB2 to emit a stream of events
> with all the information it can gather, in a easy-to-parse format, and
> then have the harness collect these TAP-agnostic events, and format
> them as required?
The problem there is "the harness" meaning "the only harness". I do not want
to tie TB2 to any one system. As Fennec illustrates, TAP::Harness is not the
only way to run tests. But it has been for so long that we can't even think
of another way to write tests. TB1 had a similar effect, only outputting TAP
and only letting you write tests in a certain way, and I'm writing TB2 in a
manner that avoids that warping.
BUT this doesn't preclude your plan. You can write a formatter that just
serializes events as JSON or YAML or something and spits them out. That's the
beauty of TB2. :) Giving the harness complete information is desirable. It
breaks the dichotomy between TAP being a human readable format and it
containing complete information.
> It would even be backwards compatible AFAICS. For example:
>
> 1. Our harness forks each .t file with an extra file-descriptor open,
> and a env var TB2_REPORT_FD with the number of the FD;
> 2. when the TB2 startups up, it notices the TB2_REPORTS_FD var and
> enables the generic event formatter, that just writes said events to
> the reports FD.
> 3. the harness collects such events and broadcasts them internally (so
> that you can even have several formatters in parallel if you need
> them);
>
> If the .t script is invoked directly, TB2 notices that the
> TB2_REPORTS_FD is not available and falls back to plain TAP.
I see where you're going, my first concern is it might be difficult to
accomplish in a portable way using file descriptors.
There is an issue of which stream is canonical and what happens if they
differ. For example, what if the harness gets "not ok 2" in the TAP but the
event stream says test #2 passed. Now the harness has a double load. It has
to parse TAP and the event stream and make sure they're in sync.
> What I'm suggesting is Tests => Events => Harness => Formatter in the
> common (using prove) case. The Tests => Events is done in the .t
> process space, and the Harness => Formatter is done in the harness
> (prove) process space.
>
> It seems cleaner to me, more flexible.
If we went with this plan, TB2 would still have to output two formats. TAP
and Yet To Be Named Event Stream. Since there's only three numbers in
computer science, once you need two formatters you might as well do infinite
formatters. :)
The beauty of the TB2 system is this: this is the first I've heard about your
idea and it already supports it. Run with it if you like, because now you can.
>> Furthermore, TB1 was having difficulty adapting to TAP extensions. Extensions
>> to TAP turn into pretty much different formatters, so internally it needed to
>> exist anyway.
>
> I don't get this part, not sure what you mean here... Maybe its
> because TB1 did to much?
Think about how a web program treats HTML. These days you'd use a template to
handle your output. The web program would just hand data to the template.
There's be no display code in the web program. Ten years ago, you'd just have
your code print bits of HTML all over the place. This made it really
difficult to change the HTML in even simple ways.
TB1 treated TAP like an old web program. It had bits of TAP hard coded all
over the code and printed out whatever little bit it needed at the time.
Because of this, any change or addition to the TAP format necessitated a lot
of pain. The TAP version header and structured diagnostics have been on hold
because of this.
Now that TB1 is using TB2::Formatter internally, the code makes a lot more
sense. The TAP formatter can be modified, enhanced and tested independently.
The event system even provides a solution to the Test::Builder::Tester
problem. Right now, if you write a Test module and want to test it, you
either have to capture the TAP output and do string comparisons on it.
Test::Builder::Tester only mitigates the problem, it does not make it go away.
As a result, even the simplest changes to the TAP format cause tests to fail.
Test::Class is failing with TB2 largely because I changed "# skip" to "#
SKIP" to make it consistent and more obvious that it's a directive.
Test::Builder2::Tester would take a completely different approach. Rather
than capturing and comparing the TAP, it captures [1] and compares the test
events directly.
It might look like this example testing Test::More's is() (once it's been
transferred to TB2).
my $result = is( "this", "that", "a name" );
result_is( $result, {
file => __FILE__,
line => __LINE__ - 3,
pass => 0,
name => "a name",
diagnostics => {
have => "this",
want => "that",
},
});
Test::More's responsibility is to produce a valid result, so that's all it has
have to test. It trusts that the formatter will do the right thing with that
result. If there is reason not to trust the formatter, it can unit test it
directly because it's a separate object. See the tests in t/Formatter for
examples.
(This approach also requires establishing two separate event coordinators, one
for Test::More with no formatter so it's detached from the outside world, and
the "real" one for Test::Builder2::Tester which outputs TAP. I'm hand waving
over that.)
Again, this is something I didn't have planned at the start.
Point is, the event and formatter system provides a huge spread of benefits.
[1] The capturing is already done by a Test::Builder2::History object.
>> Right now, making a TB2 test program use a different format is awkward. You
>> have to do it per script. There needs to be a way to control this outside the
>> script, perhaps an environment variable one can set to swap out formatters. I
>> haven't been doing much convenience features, more focusing on just getting
>> the design solid and things working. It is a SMOP to make your own test
>> programs respond to an environment variable if you need something now.
>
> See above.
If you want a test to output something other than TAP, right now you have to
write code in the test like this:
use Test::Builder2::Formatter::Other;
use Test::Simple;
my $tb2 = Test::Simple->Builder;
my $ec = $tb2->event_coordinator;
$ec->clear_formatters;
$ec->add_formatters(
Test::Builder2::Formatter::Other->new
);
This is inconvenient. More importantly it doesn't give you control outside
the code to change the formatter which is what you need to make it practical.
Sometimes you want TAP. Sometimes you want JUnit. You don't want this hard
coded in your test.
Ideally, you'd be able to do something like this:
TB2_FORMATTER=Test::Builder2::Formatter::Other perl t/test.t
Test::Builder2 would see the TB2_FORMATTER environment variable and switch
everything over to using that formatter.
--
44. I am not the atheist chaplain.
-- The 213 Things Skippy Is No Longer Allowed To Do In The U.S. Army
http://skippyslist.com/list/