Devel::Cover better practice to identify uncovered files

[Jason Pyeron]

Currently we are using this script to find files (line 48) under the scripts directory and add them to the coverage report.

After running the programs under test, the script is run:

$ ./tests/script/cover-missing.pl
load
ingest
17 known covered file(s) found
preprocess
132 uncovered file(s) found and hashed
process
run: 1492558405.0000.10744
saving

Here it found over 100 files without executions, so our tests are not very comprehensive...

First question, is the code below the "best" way to do this?

Second question, is this something that could be provided by the Module via API or command line? Something like: cover -know fileName . Maybe even support if recursively find files if fileName is a directory.

$ cat -n tests/script/cover-missing.pl [source at https://sourceforge.net/u/jpyeron/logwatch/ci/master/tree/tests/script/cover-missing.pl]
1 #!/usr/bin/perl -w
2
3 use Data::Dumper;
4 use File::Find;
5 use Cwd;
6
7 print "load\n";
8
9 use Devel::Cover::DB;
10
11 my $dbpath="cover_db";
12 my $db = Devel::Cover::DB->new(db => $dbpath);
13 my $timeStart=time;
14 my $runKey="$timeStart.0000.$$";
15 my %known;
16
17 print "ingest\n";
18
19 find({ wanted => \&process_coverfile, no_chdir => 1 }, "$dbpath/runs/");
20 find({ wanted => \&process_coverfile, no_chdir => 1 }, "$dbpath/digests");
21 find({ wanted => \&process_coverfile, no_chdir => 1 }, "$dbpath/structure/");
22
23 sub process_coverfile
24 {
25 if (-f $_)
26 {
27 my $x=$db->read($_);
28 foreach my $run ($x->runs)
29 {
30 my $h=$run->{digests};
31 foreach my $file (keys %$h)
32 {
33 if ( ! exists $known{$file} )
34 {
35 $known{$file}=$run->{digests}{$file};
36 }
37 }
38 }
39 }
40 }
41
42 print scalar keys %known, " known covered file(s) found\n";
43
44 print "preprocess\n";
45
46 my %toadd;
47
48 find({ wanted => \&process_file, no_chdir => 1 }, "scripts");
49
50 sub process_file
51 {
52 if (-f $_)
53 {
54 if ( ! exists $known{$_} )
55 {
56 $toadd{$_}=Devel::Cover::DB::Structure->digest($_);
57 }
58 }
59 }
60
61 print scalar keys %toadd, " uncovered file(s) found and hashed\n";
62
63
64 print "process\n";
65
66 if (scalar keys %toadd == 0)
67 {
68 print "no files to process\n";
69 exit;
70 }
71
72 print "run: $runKey\n";
73
74 $db->{runs}{$runKey}{"OS"}=$^O;
75 $db->{runs}{$runKey}{"collected"}=["branch","condition","pod","statement","subroutine","time"];
76 $db->{runs}{$runKey}{"dir"}=Cwd::abs_path();
77 $db->{runs}{$runKey}{"vec"}={};
78 $db->{runs}{$runKey}{"start"}=$timeStart;
79 $db->{runs}{$runKey}{"run"}=$0;
80 $_=$^V;
81 s/v//;
82 $db->{runs}{$runKey}{"perl"}=$_;
83
84 my $s=$db->{structure}=Devel::Cover::DB::Structure->new;
85
86 foreach my $file (keys %toadd)
87 {
88 $db->{structure}->{f}{$file}{start}{-1}{"__COVER__"}[0]{"branch"}=undef;
89 $db->{structure}->{f}{$file}{start}{-1}{"__COVER__"}[0]{"condition"}=undef;
90 $db->{structure}->{f}{$file}{start}{-1}{"__COVER__"}[0]{"pod"}=undef;
91 $db->{structure}->{f}{$file}{start}{-1}{"__COVER__"}[0]{"subroutine"}=undef;
92 $db->{structure}->{f}{$file}{start}{-1}{"__COVER__"}[0]{"time"}=undef;
93 $db->{structure}->{f}{$file}{start}{-1}{"__COVER__"}[0]{"statement"}=0;
94 $db->{structure}->{f}{$file}{file}=$file;
95 $db->{structure}->{f}{$file}{digest}=$toadd{$file};
96 $db->{structure}->{f}{$file}{statement}=[1];
97 $db->{runs}{$runKey}{"count"}{$file}{'statement'}=[0];
98 $db->{runs}{$runKey}{"digests"}{$file}=$toadd{$file};
99 }
100
101 $db->{runs}{$runKey}{"finish"}=time;
102
103 print "saving\n";
104
105 $db->write("$dbpath/runs/$runKey");


-Jason

[James E Keenan]

On 04/18/2017 07:43 PM, Jason Pyeron wrote:
> Currently we are using this script to find files (line 48) under the scripts directory and add them to the coverage report.
>
> After running the programs under test, the script is run:
>
> $ ./tests/script/cover-missing.pl
> load
> ingest
> 17 known covered file(s) found
> preprocess
> 132 uncovered file(s) found and hashed
> process
> run: 1492558405.0000.10744
> saving
>
> Here it found over 100 files without executions, so our tests are not very comprehensive...
>
> First question, is the code below the "best" way to do this?
>
> Second question, is this something that could be provided by the Module via API or command line? Something like: cover -know fileName . Maybe even support if recursively find files if fileName is a directory.
>

Can I ask you to back up a minute and tell us how you "normally" test
your code -- i.e., test it in the absence of coverage analysis?

The reason I ask that is that, for better or worse, most of the Perl
testing infrastructure is geared toward the testing of *functions*,
whether those be Perl built-in functions or functions or methods
exported by libraries such as CPAN distributions. This typically looks
like this:

#####
use Test::More;
use Some::Module qw( myfunc );

my ($arg, $expect);
$arg = 42;
$expect = 84;
is(myfunc($arg), $expect, "myfunc(): Got expected return value";
#####

Testing of *programs* -- what I presume you have under your scripts/
directory -- is somewhat different. Testing a script from inside a Perl
test script is just a special case of running any program from inside a
Perl program. Typically, you have to use Perl's built-in system()
function to call the program being tested along with any arguments thereto.

#####
my ($rv, @program_args);
$rv = system(qq|/path/to/tests/scripts/myprogram @program_args|);
is($rv, 0, "system call of myprogram exited successfully");
#####

You can get slightly more fancy by asking whether the program threw an
exception or not.

#####
{
local $@;
$expect = "Division-by-zero forbidden";
eval {
$rv = system(qq|/path/to/tests/scripts/myprogram @program_args|);
}
like($@, qr/$exp/, "Got expected fatal error");
}
#####

And you can capture warnings, STDOUT and STDERR with CPAN libraries like
Capture::Tiny, IO::CaptureOutput, etc.

But to get fancier, you have to establish some expectation as to how the
world outside the program being tested will change after you've run
'myprogram' once with a certain set of arguments. For example, you have
to assert that a new value will be stored in a database, and you then
have to make a database call and run the return value of that call
through Test::More::ok(), is, like(), etc. (You can do at least 82% of
anything you reasonably need with just those three functions.)

What you're really doing in this case is *testing the program's
documented interface* -- its API. "'myprogram' is documented to store
this value in the database. Irrespective of how 'myprogram' is written
-- or even of what language it's written in -- does it actually store
that value in the database?"

That's what has sometimes been characterized as "black-box testing." We
can't see into the program; we limit our concerns to what we expect the
program to do based on its published interface.

Coverage analysis, in contrast, is the archetypical case of "white-box
testing." You, the developer, can see the code, and you want to make
sure that your tests exercise every nook-and-cranny of it.

Based on this approach, in cases where I am writing a library and also
writing an executable as a short-hand way of calling functions from that
library, my approach has generally been to use coverage analysis (via
Devel::Cover) focused on the library (the .pm files) and to use system
calls of the executable more sparingly except insofar as the executable
is asserted to make a change in the outside world.

Now, clearly, your mileage may vary. That's why I'm interested in
knowing your concerns about doing coverage analysis on the programs
under scripts/.

Thank you very much.
Jim Keenan

[Jason Pyeron]

Sure. It is not tested.

Your intentions are right, but not aligned to the practical issues. I am working on a massive update to a long running open source project. The code base does not have ANY tests when I started. I am changing much of the rendering engine, but preserving legacy functionality. To assure everyone, we are creating tests in parallel, not before.

Code coverage will tell us how comprehensive our test are. Right now they suck. Nothing helps more than having a finidh line. This does that.

<snip/>

>
> Testing of *programs* -- what I presume you have under your scripts/
> directory -- is somewhat different. Testing a script from
> inside a Perl
> test script is just a special case of running any program
> from inside a

What one has to understand is that the code was written to load perl files dynamically, not all branches load all files. Here in lies the rub.

<snip/>

>
> Now, clearly, your mileage may vary. That's why I'm interested in
> knowing your concerns about doing coverage analysis on the programs
> under scripts/.

I sort of mentioned this above. I am adding XML output to logwatch. Logwatch uses an execution model that loads perl files dynamically. Further we do not have anywhere sufficient test coverage at this point, but that can no longer be a blocker. The coverage reports should indicate zero coverage on files which have not been loaded / run. Having 78% of 2% of the files is not 78% it is closer to 1%.

The more important issue is the future, when new log processors are added the code coverage in Jenkins will catch it by seeing a new file that is never called, reporting the code coverage falling below the threshold.

-Jason