testing module loading output and testing under the debugger
Nadim Khemir
I'm writing a module to help debug application. I have used it in PBS for a
while and I wanted to rewrite it "right" with tests and all.
===== 1 =====
The first problem I have is capturing the module output when it is loaded.
I tried:
my $capture ;
BEGIN
{
$capture = IO::Capture::Stdout->new();
$capture->start();
}
{
local $Plan = {'load without debugger, Debug::Mixin banner NOT shown' => 1} ;
$capture->stop();
my @all_lines = $capture->read;
is("@all_lines", "expected", "message") ;
}
without any success.
===== 2 =====
I will also have a bunch of tests that need to be run under the debugger. I
will use Devel::ebug as it is the only way to control the debugger. There are
tests in the ebug distribution that I might be able to copy but I would also
like to try this in the "standard" perl debugger.
has anyone experimented with something similar?
===== 3 =====
To get a 100% coverage I had to run this test:
dies_ok
{
Bum::Config::new () ;
} "invalid constructor" ;
How do you guys test this?
===== 4 =====
A propos coverage, I have a module (Text:::Vip) that fiddles quite a lot with
member subs, adding them dynamicaly. The more tests I added, the least
coverage I got!
Has anyone seen this before?
Cheers, Nadim.
Adrian Howard
On 17 Dec 2006, at 20:46, Nadim Khemir wrote:
[snip]
> The first problem I have is capturing the module output when it is
> loaded.
> I tried:
You might want to look at Test::Output for this.
[snip]
> I will also have a bunch of tests that need to be run under the
> debugger.
[snip]
Why? (he asks curiously :-)
[snip]
> To get a 100% coverage I had to run this test:
>
> dies_ok
> {
> Bum::Config::new () ;
> } "invalid constructor" ;
>
> How do you guys test this?
[snip]
That's a reasonably thing to do if that's the behaviour you're
interested in. If I was throwing some kind of structured exception I
might use throws_ok and test for it explicitly - but dies_ok is fine
if any-old-exception is what you want to happen.
Personally I wouldn't get /too/ hung up about 100% test coverage - it
can be taken too seriously. See Brian Marick's "How to Misuse Code
Coverage" <http://www.testing.com/writings/coverage.pdf> for example.
[snip]
> A propos coverage, I have a module (Text:::Vip) that fiddles quite
> a lot with
> member subs, adding them dynamicaly. The more tests I added, the least
> coverage I got!
[snip]
Not personally :-) Never having delved into the guts to D::C my guess
would be that you're generating new subs and only every testing part
of the new sub - so you're creating more uncovered code than not...
Without seeing the code it's hard to tell :)
Cheers,
Adrian
Joshua ben Jore
> I will also have a bunch of tests that need to be run under the debugger. I
> will use Devel::ebug as it is the only way to control the debugger. There are
> tests in the ebug distribution that I might be able to copy but I would also
> like to try this in the "standard" perl debugger.
>
> has anyone experimented with something similar?
Not true. If you real perldebguts you'll see that the debugger can be
scripted or you could write your own debugger. They can even be
ultra-simple things that are just a few lines long. I wrote a debugger
the other day and it was somewhere around five lines long. All it did
was report the depth of the call stack but that's all I wanted.
Josh
Nadim Khemir
> Not true. If you real perldebguts you'll see that the debugger can be
> scripted or you could write your own debugger.
right and right but Devel::Ebug is simply much cleaner and I'd rather not mock
a debugger when I can use a real one.
Hi Adrian,
>> The first problem I have is capturing the module output
>>...
>You might want to look at Test::Output for this.
Doesn't work either.
>> I will also have a bunch of tests that need to be run under the
>> debugger.
>Why? (he asks curiously :-)
The module is called Debug::Mixin (though it might be called something else
when released). It's a module to help applications and modules be more
debuggable. it allows you to write a set of files with breakpoints. when
debugging, you choose which set you want. This could certainly be achieved
through the debugger directly but this also works without the debugger to
achieve logging of debug information etc...
Think of this module as a smart TRACE where TRACE is a bunch of subs. one of
those subs may decide it's time for you to have a look at the current code in
a debugger (if any running)
>Personally I wouldn't get /too/ hung up about 100% test coverage - it
>can be taken too seriously. See Brian Marick's "How to Misuse Code
>Coverage" <http://www.testing.com/writings/coverage.pdf> for example.
Thanks for the article link. I've seen bad test code with 100% coverage but
I've never seen good test code with bad coverage. Also, I'd rather not have
98.7% coverage. It's nagging me and I'd rather spend five extra minutes to
get 100%.
>> A propos coverage, I have a module (Text:::Vip) that fiddles quite
>> a lot with
>> member subs, adding them dynamicaly. The more tests I added, the least
>> coverage I got!
>Not personally :-) Never having delved into the guts to D::C my guess
>would be that you're generating new subs and only every testing part
>of the new sub - so you're creating more uncovered code than not...
>Without seeing the code it's hard to tell :)
Want to see the code? Text::Vip on CPAN.
Cheers, Nadim.
Demerphq
> >Personally I wouldn't get /too/ hung up about 100% test coverage - it
> >can be taken too seriously. See Brian Marick's "How to Misuse Code
> >Coverage" <http://www.testing.com/writings/coverage.pdf> for example.
>
> Thanks for the article link. I've seen bad test code with 100% coverage but
> I've never seen good test code with bad coverage. Also, I'd rather not have
> 98.7% coverage. It's nagging me and I'd rather spend five extra minutes to
> get 100%.
Hmm, well, if you are like me then ocassionally you will have branches
to handle "can't happen" cases in your code. Eliminating them makes
your code less robust as at some future time the can't happen just
might, but at the same time since they are can't happen cases you
can't really test them or get coverage for them. Some people go to
inordinate lengths to trigger these, and I have to say im not
convinced that its time well spent.
Yves
--
perl -Mre=debug -e "/just|another|perl|hacker/"
Joshua ben Jore
It'd be nice if there were a pragma or function for use by
Devel::Cover which said just that:
cond ? ... :
cond ? ... :
cond ? ... :
can't::happen;
sub can't::happen { Carp::confess q[This can't happen] }
Josh
Chromatic
> It'd be nice if there were a pragma or function for use by
> Devel::Cover which said just that:
Nicer yet would be if Devel::Cover (and I haven't tried in a few months; too
busy with other things, so if there's a fix now, not only will I learn about
it by shooting off my mouth, but you'll all learn about it too) would
consider constants on the right side of || as, well, constant.
I'm not sure what kind of hack it would take to make the empty string
unconstant, but I promise I'm not using it in my code right now.
-- c
Paul Johnson
> On Tuesday 19 December 2006 16:04, Joshua ben Jore wrote:
>
> > It'd be nice if there were a pragma or function for use by
> > Devel::Cover which said just that:
cond ? ... :
cond ? ... :
cond ? ... :
can't::happen;
sub can't::happen { Carp::confess q[This can't happen] }
That would be the "uncoverable" feature, which I haven't quite finished
yet. It does just what is asked for, that is you can mark constructs as
uncoverable which means that the sense of the error is reversed - the
construct will be in error iff it is executed.
There are three main sections of work still left to be completed:
1. Find some nice way expressing what is uncoverable. For subroutines
this is easy. For statements it is not hard. For branches it is
tricky and for conditions I'm somewhat stumped. The current
method I use is based on implementation details.
2. Make all the reports honour this feature. The text and html_basic
reports do so, but the (default) html report doesn't.
3. Provide a way to specify uncoverable code within the source file.
I'm surprised that many people want this, but they do.
The basics of this functionality have been there for a while, but I've
not found any time to work on things recently. Until the API is
complete it is deliberately not documented. Anyone wanting to play for
it can grep the source for "uncoverable" and use the feature. It
actually works quite well, apart from the limitations above.
> Nicer yet would be if Devel::Cover (and I haven't tried in a few months; too
> busy with other things, so if there's a fix now, not only will I learn about
> it by shooting off my mouth, but you'll all learn about it too) would
> consider constants on the right side of || as, well, constant.
This should already be the case, and I don't think there have been many
changes there in the last few months. I'm fairly sure of this since
there haven't been any releases in the last few months ;-)
So if you (or anyone else) has any examples where this is not doing the
right thing please send them to me, either as code snippets or as
patches to tests/cond_or.
> I'm not sure what kind of hack it would take to make the empty string
> unconstant, but I promise I'm not using it in my code right now.
Such ideas are the bane of my life as Devel::Cover's author. But I
think that anyone who makes undef return 42 randomly probably deserves
whatever they get. Yes, Juerd, I'm scowling at you.
--
Paul Johnson - pa...@pjcj.net
http://www.pjcj.net
Michael G Schwern
> That would be the "uncoverable" feature, which I haven't quite finished
> yet. It does just what is asked for, that is you can mark constructs as
> uncoverable which means that the sense of the error is reversed - the
> construct will be in error iff it is executed.
>
> There are three main sections of work still left to be completed:
>
> 1. Find some nice way expressing what is uncoverable. For subroutines
> this is easy. For statements it is not hard. For branches it is
> tricky and for conditions I'm somewhat stumped. The current
> method I use is based on implementation details.
What's the current syntaxes you're toying with?
> 3. Provide a way to specify uncoverable code within the source file.
> I'm surprised that many people want this, but they do.
Do you mean so we can mark what lines and things are uncoverable inline? Right next to the code itself? Of course! What would the alternative be?
Paul Johnson
> Paul Johnson wrote:
> > That would be the "uncoverable" feature, which I haven't quite finished
> > yet. It does just what is asked for, that is you can mark constructs as
> > uncoverable which means that the sense of the error is reversed - the
> > construct will be in error iff it is executed.
> >
> > There are three main sections of work still left to be completed:
> >
> > 1. Find some nice way expressing what is uncoverable. For subroutines
> > this is easy. For statements it is not hard. For branches it is
> > tricky and for conditions I'm somewhat stumped. The current
> > method I use is based on implementation details.
>
> What's the current syntaxes you're toying with?
from tests/.uncoverable:
tests/uncoverable statement 20f752895295d69b6b73da4d9921a9ed 0 0 Can't get to this statement
tests/uncoverable statement 89c43a57239a432b0517549f6b8f9feb 0 0 Can't get to this statement
tests/uncoverable statement ba33ccd5188f01ebbd59be35038fe0cb 0 0 Can't get to this statement
tests/uncoverable branch 9389290602dbf70585e0467e6cb28669 0 0 Branch can't be true
tests/uncoverable condition 9389290602dbf70585e0467e6cb28669 0 0 $x can't be false
tests/uncoverable condition 9389290602dbf70585e0467e6cb28669 0 2 $y can't be false
tests/uncoverable subroutine ba33ccd5188f01ebbd59be35038fe0cb 0 0 Can't run this subroutine
that is:
file,
criterion,
md5 of line,
count (nth such construct on the line),
type (identification of the part of the branch or condition)
reason
The bit that really needs work is the type field. How can you easily
specify which part of a branch or which part of a condition is
uncoverable?
> > 3. Provide a way to specify uncoverable code within the source file.
> > I'm surprised that many people want this, but they do.
>
> Do you mean so we can mark what lines and things are uncoverable inline?
> Right next to the code itself? Of course! What would the alternative be?
At the moment,
$ cover -add_uncoverable_point "tests/uncoverable statement 73 0 0 Can't get to this statement"
I sometimes dream of some AJAX driven, fully buzzword compliant gui which
would let you navigate your results without having to create acres of static
HTML, and which would allow you to specify this sort of thing in the gui.
But I'm learning that I really shouldn't eat cheese before going to bed.
I suspect that the sets of those who would annotate uncoverable code inline,
those who would use a separate file and those who wouldn't bother at all
probably correlates fairly well with those who add pod inline, those who add
it at the end or separately and those who don't document at all.
Well, maybe not exactly - I can probably make good arguments for inline
annotation.
But I'm probably just scarred by using too many tools that insist I change my
source code to make up for their inadequacies.
Michael G Schwern
>>> 1. Find some nice way expressing what is uncoverable. For subroutines
>>> this is easy. For statements it is not hard. For branches it is
>>> tricky and for conditions I'm somewhat stumped. The current
>>> method I use is based on implementation details.
>> What's the current syntaxes you're toying with?
>
> from tests/.uncoverable:
>
> tests/uncoverable statement 20f752895295d69b6b73da4d9921a9ed 0 0 Can't get to this statement
> tests/uncoverable statement 89c43a57239a432b0517549f6b8f9feb 0 0 Can't get to this statement
> tests/uncoverable statement ba33ccd5188f01ebbd59be35038fe0cb 0 0 Can't get to this statement
> tests/uncoverable branch 9389290602dbf70585e0467e6cb28669 0 0 Branch can't be true
> tests/uncoverable condition 9389290602dbf70585e0467e6cb28669 0 0 $x can't be false
> tests/uncoverable condition 9389290602dbf70585e0467e6cb28669 0 2 $y can't be false
> tests/uncoverable subroutine ba33ccd5188f01ebbd59be35038fe0cb 0 0 Can't run this subroutine
>
> that is:
>
> file,
> criterion,
> md5 of line,
> count (nth such construct on the line),
> type (identification of the part of the branch or condition)
> reason
*snip*
> I suspect that the sets of those who would annotate uncoverable code inline,
> those who would use a separate file and those who wouldn't bother at all
> probably correlates fairly well with those who add pod inline, those who add
> it at the end or separately and those who don't document at all.
I don't think this is a good measure. The reason usually given for not using inline POD is that since there's nearly as much POD as code it clutters up the code and makes it difficult to read. This is not the case with an uncoverable marker. For one, uncover markers should be rare. Another, it should be no more obtrusive than an end-of-line comment.
else {
die "Can't get here"; # uncoverable
}
> Well, maybe not exactly - I can probably make good arguments for inline
> annotation.
Allow me. :)
The current scheme won't track changes to the source file well. Using file + md5 rather than file + line helps some, but problems remain. Not dealing well with code change is a much worse problem than having to markup your source. You markup once. You change code a lot. O(1) vs O(n).
* What happens when I trivially alter an uncoverable line? Add a comment, change the spacing,
move it to a different indentation level?
* What happens when I have two duplicate lines in the same file, one I want to cover one I don't?
* What happens when I rename a file with uncoverable elements?
* What happens when I move code from one file to another?
And the usual drawbacks of not being inline.
* Having to drag around a separate file just for coverage.
* Having to shell out to a separate utility and describe to it which bit not to cover.
Additionally, the use of an md5 sum to identify the line means I cannot easily know which lines are marked as uncoverable even with the uncover file open in front of me.
I predict a lot of churn of the form:
1) Edit the code
2) Run the coverage
3) Read warnings about uncover lines no longer existing in the source file
(maybe, they've just been altered, maybe they've been deleted. Who knows?)
4) Carefully scanning my coverage results to see what was marked as uncovered has lost
that marker because of a trivial change to the line.
5) Updating the uncover file.
This is no fun.
If I might make a suggestion for an inline syntax. This is statement, subroutine, branch and condition respectively.
statement; # uncoverable "Optional Reason"
sub foo { # uncoverable "Optional Reason"
...
}
if( condition ) { # uncoverable "Optional Reason"
if( this || that ||
unreachable # uncoverable "Optional Reason"
)
{
...
}
The condition coverage marker does require some reformatting of the code. That probably won't be a big deal given how infrequently it should be used. However, maybe something a bit more explicit could be used.
if( this || that || unreachable ) { # uncoverable condition/3 "Optional Reason"
...
}
That states that the 3rd part of the condition on that line is uncoverable.
Chris Dolan
IF you want a powerful, inline markup syntax for Devel::Cover (and
that's a BIG if), you might want to look at what we did for
Perl::Critic. We have scoped, blocked, and line-oriented pragma-like
comments that selectively disable Perl::Critic. An optional
parenthesized substring turns off just matching policies.
Single line style:
eval $str; ## no critic(StringyEval)
On/off-style:
## no critic(StringyEval)
eval $str;
## use critic
Scoped style:
sub {
## no critic(StringyEval)
eval $str;
}
eval $str; # triggers a violation...
The scoped style is possible because we are using PPI to parse the
code into a DOM.
Chris
--
Chris Dolan, Software Developer, http://www.chrisdolan.net/
Public key: http://www.chrisdolan.net/public.key
vCard: http://www.chrisdolan.net/ChrisDolan.vcf
Richard Foley
useful for scripting purposes. There's also the recent perl5db.t to take a
look at for an example of usage. Just a thought.
>
> > Not true. If you real perldebguts you'll see that the debugger can be
> > scripted or you could write your own debugger.
>
> right and right but Devel::Ebug is simply much cleaner and I'd rather not
> mock a debugger when I can use a real one.
>
--
Richard Foley
Ciao - shorter than aufwiedersehen