Surviving smoke testing

[Ilya Zakharevich]

Let me start with obvious: smoke testing is a very useful tool.

But HOW ON THE EARTH people manage to live with the results it provides?!

Let's leave aside the topic that the web interface is majorly buggy
(the counts go up and down in a complicated dance, counts in different
"views" do not match etc) - even the puzzling buggy interface provides
heaps of useful information. But I'd like to know tricks people
employ to fight the major limitation of smoke testings: that "original
reports from testers" have the most crucial information cut out.

[The problem is that the "try" is broken into 3 steps
(configuration, build, and test), and only the log of the last of
executed steps is reported. It would not be important for
distributions which do essentially no configuration, and
essentially no "building".

But the distributions with non-trivial configuration and/or
non-trivial build are exactly ones which could benefit most from
widest possible testing - and to debug problems found during the
testing stage, one would need to inspect logs of configuration step
and the build step...]

As an appetizer, here is my trick: it tries to load the module, and if
loading fails, it reports the parameters passed to MakeMaker during
the configuration stage.

The code needs 3 parameters: which module to load, the name of one of
the test files (to find in which directory one is executed), and which
directories contain active Makefile.PL's. You modify these
parameters, and put the code in one of the test files.

One example of the output in

http://www.nntp.perl.org/group/perl.cpan.testers/2009/10/msg5821317.html

(Unfortunately, this particular report does not help - I still do not
know why PIC is not enabled... Looks like assembler flags should
include -KPIC...)

Enjoy,
Ilya

=======================================================
#!/usr/bin/perl -w
use strict;

$| = 1;
print "1..1\nok 1\n"; # report success

exit if eval 'use Math::Pari; 1';

# If failed, report build parameters

sub report_build_parameters ($) {
my ($makefile, $in) = (shift, '');
warn "# reporting $makefile header:\n# ==========================\n";
open M, "< $makefile" or die "Can't open $makefile";
$in = <M> while defined $in and $in !~ /MakeMaker \s+ Parameters/xi;
$in = <M>; # Reload till first non-empty line after "Param" one:
$in = <M> while defined $in and $in !~ /\S/;
warn $in and $in = <M> while defined $in and $in =~ /^#/;
close M;
warn "# ==========================\n";
}

my ($base_d, $in) = (-f "t/000_load-problem.t" ? '.' : '..', '');
report_build_parameters("$base_d/Makefile");
report_build_parameters("$base_d/libPARI/Makefile");