Events
Michael G Schwern
events have to be fine grained enough that they can completely describe what
goes on in a test and generic enough to fit any conceivable testing situation.
The most obvious event is the result of an assert, and that's pretty well
understood.
The most important event watcher is the Formatter. Events need to tell the
Formatter everything it needs to know to display the test results. This
includes things like the test plan (if any), diagnostics, metadata about the
whole test, sub tests, and so on.
The second most important watcher is user written watchers. These are things
that take side actions when events happen. For example, Test::NoWarnings
would be written as an early watcher (meaning it gets to take action before
the Formatter does) looking for a "start stream" event to set up the warning
handler, a "set plan" event to add one test to (its own warnings check at the
end), and an "end stream" event to remove the warning handler and do it's own
test for warnings. "Die on fail" would be a late watcher (after the
Formatter) looking at results. If there are any failures, it dies.
Events identified so far are:
start stream
end stream
set plan
stream metadata
result
start assert
end assert
A test like this:
ok 1
not ok 2 - something something
1..2
Would be represented by events like so:
start stream
start assert
result
literal_pass: 1
end assert
start assert
result
literal_pass: 0
name: something something
end assert
set plan
asserts_expected: 2
end stream
"start assert" and "end assert" events may be superfluous. They exist because
of the way Test::Builder2 stacks assert functions on top of each other to
build up functionality and so there's an "end of assert" action to hook on to.
Most other systems don't stack asserts, and seeing a result is enough to
signal that an assert has completed. So far there has been no reason to know
that an assert has begun, only that it has ended.
By leaving them in the concern is they won't make sense for other builders and
they won't send those events. If they don't send them, then watchers
expecting them to signal the end of an assert won't see them and thus won't
work as expected. I might take them out and see if anyone needs them back.
So... yeah, that's how things are looking. Thoughts?
--
Being faith-based doesn't trump reality.
-- Bruce Sterling
Pedro Melo
On Fri, Dec 17, 2010 at 3:07 AM, Michael G Schwern <sch...@pobox.com> wrote:
> "start assert" and "end assert" events may be superfluous. They exist because
> of the way Test::Builder2 stacks assert functions on top of each other to
> build up functionality and so there's an "end of assert" action to hook on to.
> Most other systems don't stack asserts, and seeing a result is enough to
> signal that an assert has completed. So far there has been no reason to know
> that an assert has begun, only that it has ended.
I think start/end assert are important to synchronize STDERR in some
situations so I would keep them.
Bye,
--
Pedro Melo
http://www.simplicidade.org/
xmpp:me...@simplicidade.org
mailto:me...@simplicidade.org
Michael G Schwern
> On Fri, Dec 17, 2010 at 3:07 AM, Michael G Schwern <sch...@pobox.com> wrote:
>> "start assert" and "end assert" events may be superfluous. They exist because
>> of the way Test::Builder2 stacks assert functions on top of each other to
>> build up functionality and so there's an "end of assert" action to hook on to.
>> Most other systems don't stack asserts, and seeing a result is enough to
>> signal that an assert has completed. So far there has been no reason to know
>> that an assert has begun, only that it has ended.
>
> I think start/end assert are important to synchronize STDERR in some
> situations so I would keep them.
Could you expand on that please?
Let me clarify that TB2 will differ from TB1 in how it handles STDOUT and
STDERR. Keep in mind that Test::Harness is also its own thing.
For starters, assert code should *never* be sending anything *directly* to
STDOUT or STDERR. It should always be passing events to the Formatter which
then decides how to display. The only thing which may write directly to
STDOUT or STDERR is dodgy output from the code being tested (warnings, errors,
stray print statements) which we really don't have a reasonable expectation of
control over.
Currently, test diagnostics are expressed as free-form comments going to
STDERR such as "# got: foo, expected: bar". These are unparsed by
Test::Harness. In TB2, these will be replaced by structured data going to
STDOUT and thus parsed. This eliminates most of the need for synchronizing
STDOUT and STDERR within Test::Harness... though it is still nice to know
which code warnings and error messages are associated with which test. Here's
details on that:
http://testanything.org/wiki/index.php/TAP_diagnostic_syntax
Some of the details are out of date, but the idea stands.
Most uses of diag() to print out diagnostics associated with an assert (like
the got/expected stuff from is()) will instead be attached to the Result
produced by the assert, usually as structured data rather than free form
comments. The Formatter will then decide what to do with them. So in that
respect, STDOUT and STDERR synchronization will happen in the Formatter. It
will be easy because the Formatter will get everything it needs about the
result of an assert in one tidy package.
Free-form comments, currently produced with diag() and note() (STDERR and
STDOUT respectively) have not yet been designed into TB2. They'll probably be
transmitted to the Formatter as some form of logging event with a message and
severity level and structured data. The Formatter can then decide what to do
with them.
--
60. "The Giant Space Ants" are not at the top of my chain of command.
-- The 213 Things Skippy Is No Longer Allowed To Do In The U.S. Army
http://skippyslist.com/list/