Strange interaction between new Test::More and Test::Builder::Tester

[Buddy Burden]

Guys,

I'm getting CPAN Tester failures that look like this:

# STDOUT is:
# ok 1 # SKIP symlink_target_is doesn't work on systems without symlinks!
#
# not:
# ok 1 # skip symlink_target_is doesn't work on systems without symlinks!
#
# as expected

and, in case it doesn't jump out at you what the problem is (I had to
stare at it 4 or 5 times before I caught it), it's the capitalization
of "skip" (or "SKIP", as the case may be). Now, these failures are a
minority, and the trend I'm seeing is that the failures all seem to
come from Test::More 1.005000_002, whereas the passes all seem to come
from versions pre 1.x. Now, I've never really used
Test::Builder::Tester before (I've previously used Test::Tester), but
I got co-maint on this module (Test::File), and that's what it uses,
and it seems to work pretty well ... except for this one thing. So I
don't really want to change it, and anyway I'm sure I'm not the only
person who is/will be seeing this sort of problem.

Any thoughts on what the proper fix for this is/should be? TIA guys.


            -- Buddy

[Ovid]

----- Original Message -----

> From: Buddy Burden <barefo...@gmail.com>
>
>G uys,

Hi Buddy,
I'm not sure what's going on here. You've mentioned the Test::More, Test::Builder::Tester, Test::Tester and Test::File. I don't know exactly what is causing the problem you have with your tests, but if you check the TAP::Parser::Grammar (https://metacpan.org/module/TAP::Parser::Grammar) you'll see that the SKIP (and TODO) directives are case-insensitive. Thus, both SKIP and skip should be fine. If something is marking that as a failure, it's probably ignoring case-sensitivity for directives or it's expecting an exact text match.

In any event, I can't tell how to reproduce the issue from the plethora of modules you've listed. Can you send a small code example of a test failure?

Cheers,
Ovid
--
Live and work overseas - http://overseas-exile.blogspot.com/
Buy the book - http://www.oreilly.com/catalog/perlhks/
Tech blog - http://blogs.perl.org/users/ovid/
Twitter - http://twitter.com/OvidPerl/

[Buddy Burden]

Ovid,

> I'm not sure what's going on here. You've mentioned the Test::More, Test::Builder::Tester, Test::Tester and Test::File.

Sorry; perhaps I overexplained. This is a problem between Test::More
and Test::Builder, like the subject says. The other two are
irrelevant.

> I don't know exactly what is causing the problem you have with your tests, but if you check the TAP::Parser::Grammar (https://metacpan.org/module/TAP::Parser::Grammar) you'll see that the SKIP (and TODO) directives are case-insensitive. Thus, both SKIP and skip should be fine. If something is marking that as a failure, it's probably ignoring case-sensitivity for directives or it's expecting an exact text match.

Okay, that puts the blame squarely on Test::Builder::Tester then. It
definitely is expecting an exact text match. Going through the code a
bit, it looks like it would accept a regex (although this is not
documented in the POD) so I suppose I could code defensively enough to
work around it, but it seems like if the standard says case
insensitive, it might be better for Test::Builder::Tester to ignore
case altogether. Perhaps there are subtleties that would break other
stuff though ...

> In any event, I can't tell how to reproduce the issue from the plethora of modules you've listed. Can you send a small code example of a test failure?

This code:

{
local $^O = 'Win32';

my @subs = qw(
file_mode_is file_mode_isnt
file_executable_ok file_not_executable_ok
);

foreach my $sub ( @subs )
{
no strict 'refs';

test_out("ok 1 # skip $sub doesn't work on Windows!");
&{$sub}();
test_test();
}

}

works on Test::More (say) 0.98, but not on Test::More 1.005000_002.

I think I'll raise an issue on Test::Builder::Tester's github and see
what schwern has to say.


            -- Buddy