Just how far should cmp_ok() go?
Michael G Schwern
cmp_ok $foo, '==', $bar;
should do exactly the same as this:
$foo == $bar;
It goes through some effort to make sure any error messages are done at the
caller's context. A recent change to Test::Builder revealed a problem.
local $^W;
cmp_ok "12 and stuff", "==", 12;
0.80 and back this produced no warning, which is correct. After 0.82 it does
because Test::Builder turns on warnings and that overrides $^W. This breaks a
test in constant.pm.
Now, I could fix that by detecting $^W and turning warnings on/off based on
that. That works for $^W but not for this...
no warnings;
cmp_ok "12 and stuff", "==", 12;
Well, ok, maybe just turn off warnings.pm inside Test::Builder. But then what
about this?
use warnings;
cmp_ok "12 and stuff", "==", 12;
I don't know of any sane way to detect if warnings are on in the caller's context.
It's at this point in contemplating a never ending stack of insane hacks that
I question the base assumption. Just how far should cmp_ok() go to try to
pretend it's being called in the caller's context before A) the user gets
surprised, because there's clearly a function call there, and B) the hacks get
so thick things break?
--
Robrt: People can't win
Schwern: No, but they can riot after the game.
Sébastien Aperghis-Tramoni
Michael G Schwern wrote:
> Ideally, this...
>
> cmp_ok $foo, '==', $bar;
>
> should do exactly the same as this:
>
> $foo == $bar;
>
> It goes through some effort to make sure any error messages are
> done at the
> caller's context. A recent change to Test::Builder revealed a
> problem.
>
> local $^W;
> cmp_ok "12 and stuff", "==", 12;
>
> 0.80 and back this produced no warning, which is correct. After
> 0.82 it does
> because Test::Builder turns on warnings and that overrides $^W.
> This breaks a
> test in constant.pm.
I saw that bug but didn't give it a serious look because it felt more
like a test bug than a constant.pm bug. I now see someone has opened
a ticket on CPAN RT, but I didn't receive any mail.
> Now, I could fix that by detecting $^W and turning warnings on/off
> based on
> that. That works for $^W but not for this...
>
> no warnings;
> cmp_ok "12 and stuff", "==", 12;
>
> Well, ok, maybe just turn off warnings.pm inside Test::Builder.
> But then what
> about this?
>
> use warnings;
> cmp_ok "12 and stuff", "==", 12;
>
> I don't know of any sane way to detect if warnings are on in the
> caller's context.
Why should you? this case is precisely the situation of the
constant.pm tests, and it works with Test::More 0.80:
$ perl -MV=Test::More
Test::More
/System/Library/Perl/5.8.6/Test/More.pm: 0.80
$ perl -MTest::More=tests,1 -Mwarnings -le 'cmp_ok "12 cats", "==", 12'
1..1
ok 1
So I'd say, simply remove warnings from Test::Builder, because as you
said, a hackish solution will become a new hell, and I think you
already have enough hells to deal with :-)
--
Sébastien Aperghis-Tramoni
Close the world, txEn eht nepO.
Michael G Schwern
> It's at this point in contemplating a never ending stack of insane hacks that
> I question the base assumption. Just how far should cmp_ok() go to try to
> pretend it's being called in the caller's context before A) the user gets
> surprised, because there's clearly a function call there, and B) the hacks get
> so thick things break?
I was thinking of some way I could maybe redesign cmp_ok() to avoid the
problem entirely. To have something which would really run the comparison in
the caller's context and yet still print meaningful diagnostics.
This comes to mind...
my $foo = 23;
my $bar = 42;
test { $foo == $bar } $name;
And then use Data::Dump::Streamer to dump out the subroutine and lexical context.
# $bar = 42;
# $foo = 23;
# sub {
# $foo == $bar;
# };
But Test::Builder can't depend on DDS. Interesting idea though. *wink wink,
nudge nudge* at Test::Most.
I think I'm going to have to give up on the illusion that cmp_ok() is running
the comparison in the caller's context. Lifting that curtain requires
removing the code that makes warnings and errors happen at the caller's file
and line number. But having warnings come out at "Test/Builder.pm line 23989"
is no better. Thus...
Argument "12 cats" isn't numeric in numeric eq (==) at cmp_ok [from
t/constant.t line 86] line 1.
It tells the user where the warning came from while reminding them that it's
still from inside a function.
Thoughts?
--
If at first you don't succeed--you fail.
-- "Portal" demo
Michael G Schwern
>> Well, ok, maybe just turn off warnings.pm inside Test::Builder. But
>> then what
>> about this?
>>
>> use warnings;
>> cmp_ok "12 and stuff", "==", 12;
>>
>> I don't know of any sane way to detect if warnings are on in the
>> caller's context.
>
> Why should you? this case is precisely the situation of the constant.pm
> tests, and it works with Test::More 0.80:
>
> $ perl -MV=Test::More
> Test::More
> /System/Library/Perl/5.8.6/Test/More.pm: 0.80
>
> $ perl -MTest::More=tests,1 -Mwarnings -le 'cmp_ok "12 cats", "==", 12'
> 1..1
> ok 1
>
> So I'd say, simply remove warnings from Test::Builder, because as you
> said, a hackish solution will become a new hell, and I think you already
> have enough hells to deal with :-)
That's the same problem in the opposite direction. It should have warned, but
didn't.
--
40. I do not have super-powers.
-- The 213 Things Skippy Is No Longer Allowed To Do In The U.S. Army
http://skippyslist.com/list/
Todd
Michael G Schwern
>> Thoughts?
>
> What about..... SOURCE FILTERS!!!!!
Now you have two problems.
--
52. Not allowed to yell "Take that Cobra" at the rifle range.
Todd Rinaldo
>
> Todd wrote:
>>> Thoughts?
>>
>> What about..... SOURCE FILTERS!!!!!
>
> Now you have two problems.
>
Seriously thought, out of sick curiousity, lemme ask: Would source
filters fix your problem in that you could determine whether warnings
were on? I realize it's a no go for a testing system, but my knowledge
is limited because every time I hear about them, it's followed by
"don't do it".
Michael G Schwern
Source filters would "fix" the problem in that I could try to detect if
there's a warnings pragma in the caller's context.
The reason you don't want to use a source filter is A) it has to touch *all*
the code and B) it has to parse Perl. There's nothing magical about a source
filter. You get the text of the program as a string and you do stuff to it.
So most likely I'd write a filter that mostly gets it right finding the use of
warnings. Oh, and no warnings. Oh, and tracking what particular categories
are on or off. And don't forget figuring out when lexical scope changes,
always fun. And then eval STRING brings it all crashing down.
And then there's the problem of passing this information gleaned at compile
time down into Test::Builder when cmp_ok() is called. I could rewrite the
cmp_ok() call to add extra arguments, but that's going to break. I could
insert a "local %warnings = ..." just before a cmp_ok() call. That's sort of
safe, but there's tons of edge cases in order to find the preceding statement
break.
Oh, don't forget not to mess up the line numbers.
Any filter which thinks it gets this right, I will find code on CPAN that will
break it. Either it will fail to scan the warnings pragmas correctly, or the
code insertion will change the way the code behaves.
And that is why you would then have two problems. Your original problem will
still not be solved and now you have a source filter silently rewriting code
and introducing mysterious bugs.
--
54. "Napalm sticks to kids" is *not* a motivational phrase.