Test.Harness and overrun/underrun/die
Marvin Humphrey
Test.Harness doesn't seem to notice when the number of tests run
doesn't match the number of tests expected. It looks like we need
this in Test.Harness.prototype.outputResults:
if (test.TestResults.length < expected) {
// List unrun tests as failures.
pass = false;
for(var i = test.TestResults.length; i < expected; i++) {
track.failList.push(i + 1);
}
}
else if (test.TestResults.length > expected) {
// List overruns as failures.
pass = false;
this.ran += test.TestResults.length - expected;
for(var i = expected; i < test.TestResults.length; i++) {
track.failList.push(i + 1);
}
}
That gets us halfway there -- at least Test.Harness now has the
information it needs to report overruns/underruns as numbered failures.
What's left is to pass along the diagnostic messages, and that's
where I'm confused. It seems like Test.Harness doesn't pay any
attention to the end output generated by
Test.Builder.prototype._ending() -- so it doesn't see any "# Looks
like you planned " messages, and it doesn't know if the test file died.
What's the intended mechanism for conveying that information from
Test.Builder to Test.Harness?
Marvin Humphrey
Rectangular Research
http://www.rectangular.com/
David E. Wheeler
> Test.Harness doesn't seem to notice when the number of tests run
> doesn't match the number of tests expected. It looks like we need
> this in Test.Harness.prototype.outputResults:
Hrm. As I said, I've had my head out of Test.Simple and friends for
quite some time, but I do remember that I copied that code pretty
much verbatim from Test::Harness. How does *it* do it?
> That gets us halfway there -- at least Test.Harness now has the
> information it needs to report overruns/underruns as numbered
> failures.
>
> What's left is to pass along the diagnostic messages, and that's
> where I'm confused. It seems like Test.Harness doesn't pay any
> attention to the end output generated by
> Test.Builder.prototype._ending() -- so it doesn't see any "# Looks
> like you planned " messages, and it doesn't know if the test file
> died.
Yeah, it doesn't read the output, but rather checks the data
structure inside Test.Builder directly. That's what's in
test.TestResults, IIRC.
HTH, and maybe I'll get back into this stuff one of these days.
Thanks for taking it on a bit.
David
Marvin Humphrey
Back to working on this problem.
On Jul 30, 2007, at 10:29 PM, David E. Wheeler wrote:
> On Jul 30, 2007, at 08:33, Marvin Humphrey wrote:
>
>> Test.Harness doesn't seem to notice when the number of tests run
>> doesn't match the number of tests expected. It looks like we need
>> this in Test.Harness.prototype.outputResults:
>
> Hrm. As I said, I've had my head out of Test.Simple and friends for
> quite some time, but I do remember that I copied that code pretty
> much verbatim from Test::Harness. How does *it* do it?
It does pretty much what I described. However, the code isn't easy
to follow, for two reasons.
First, there are all the IPC complications when you're running in
Perl-ville.
Second, the encapsulation isn't very strong, and many interfaces are
tightly bound to each other -- a lot of complex data structures get
passed around, and it's hard to see exactly what's going on.
If I had my druthers, those data structures would get replaced with
small, purpose-built classes, e.g. Test.Harness.TestResult. Would
you be open to that?
>> That gets us halfway there -- at least Test.Harness now has the
>> information it needs to report overruns/underruns as numbered
>> failures.
>>
>> What's left is to pass along the diagnostic messages, and that's
>> where I'm confused. It seems like Test.Harness doesn't pay any
>> attention to the end output generated by
>> Test.Builder.prototype._ending() -- so it doesn't see any "# Looks
>> like you planned " messages, and it doesn't know if the test file
>> died.
>
> Yeah, it doesn't read the output, but rather checks the data
> structure inside Test.Builder directly. That's what's in
> test.TestResults, IIRC.
>
> HTH, and maybe I'll get back into this stuff one of these days.
It helps. I am kind of frustrated by how hard it's been to produce
this patch, though. I wish I could just give up, but this bug is
unacceptable: any time a test script dies prior to actual test
failures, Test.Harness reports success regardless of how many or how
few tests have been run. :(
David E. Wheeler
> If I had my druthers, those data structures would get replaced with
> small, purpose-built classes, e.g. Test.Harness.TestResult. Would
> you be open to that?
Sure, it makes sense. I often get to that after I've passed around
data structures that keep getting bigger. It's a good refactoring
pattern.
>> Yeah, it doesn't read the output, but rather checks the data
>> structure inside Test.Builder directly. That's what's in
>> test.TestResults, IIRC.
>>
>> HTH, and maybe I'll get back into this stuff one of these days.
>
> It helps. I am kind of frustrated by how hard it's been to produce
> this patch, though. I wish I could just give up, but this bug is
> unacceptable: any time a test script dies prior to actual test
> failures, Test.Harness reports success regardless of how many or how
> few tests have been run. :(
Yeah, that's teh suck alright. I might have some time to look at it
in a few months, but right now, unfortunately, I'm swamped. :-)
Thanks for hacking at it.
Best,
David