PASS with no test output

[Nathan (Nat) Goodman]

Greetings testers

I'm a module developer, not a tester. For one of my modules, I'm seeing a lot of PASS reports that have no test output; see example below. I'm guessing this means the test didn't run. Any suggestions on how I can get better diagnostics to help me debug the problem?

Many thanks,
Nat Goodman

Begin forwarded message:

> From: ch...@bingosnet.co.uk
> Subject: PASS Data-Babel-1.10_05 v5.14.1 netbsd
> Date: November 28, 2012 7:12:05 PM PST
> To: "Nathan Goodman" <NA...@cpan.org>
> Reply-To: ch...@bingosnet.co.uk
>
> This distribution has been tested as part of the CPAN Testers
> project, supporting the Perl programming language. See
> http://wiki.cpantesters.org/ for more information or email
> questions to cpan-teste...@perl.org
>
>
> --
>
> Dear NATG,
>
> This is a computer-generated error report created automatically by
> CPANPLUS, version 0.9133. Testers personal comments may appear
> at the end of this report.
>
>
> This report was machine-generated by CPANPLUS::Dist::YACSmoke 0.84.
> Powered by minismokebox version 0.54
>
> ------------------------------
> ENVIRONMENT AND OTHER CONTEXT
> ------------------------------
>
> Environment variables:
>
> AUTOMATED_TESTING = 1
> PATH = /home/cpan/pit/thr/conf/perl-5.14.1/.cpanplus/5.14.1/build/DBI-1.622/blib/script:/home/cpan/bin:/bin:/sbin:/usr/bin:/usr/sbin:/usr/X11R7/bin:/usr/X11R6/bin:/usr/pkg/bin:/usr/pkg/sbin:/usr/games:/usr/local/bin:/usr/local/sbin
> PERL5LIB = :/home/cpan/pit/thr/conf/perl-5.14.1/.cpanplus/5.14.1/build/DBI-1.622/blib/lib:/home/cpan/pit/thr/conf/perl-5.14.1/.cpanplus/5.14.1/build/DBI-1.622/blib/arch:/home/cpan/pit/thr/conf/perl-5.14.1/.cpanplus/5.14.1/build/Data-Babel-1.10_05/blib/lib:/home/cpan/pit/thr/conf/perl-5.14.1/.cpanplus/5.14.1/build/Data-Babel-1.10_05/blib/arch
> PERL5_CPANPLUS_IS_RUNNING = 17246
> PERL5_CPANPLUS_IS_VERSION = 0.9133
> PERL5_MINISMOKEBOX = 0.54
> PERL5_YACSMOKE_BASE = /home/cpan/pit/thr/conf/perl-5.14.1
> PERL_EXTUTILS_AUTOINSTALL = --defaultdeps
> PERL_MM_USE_DEFAULT = 1
> SHELL = /usr/pkg/bin/bash
> TERM = screen
>
> Perl special variables (and OS-specific diagnostics, for MSWin32):
>
> Perl: $^X = /home/cpan/pit/thr/perl-5.14.1/bin/perl
> UID: $< = 1001
> EUID: $> = 1001
> GID: $( = 100 100
> EGID: $) = 100 100
>
>
> -------------------------------
>
>
> --
>
> Summary of my perl5 (revision 5 version 14 subversion 1) configuration:
>
> Platform:
> osname=netbsd, osvers=6.0, archname=amd64-netbsd-thread-multi
> uname='netbsd newspeak.bingosnet.co.uk 6.0 netbsd 6.0 (generic) amd64 '
> config_args='-des -Dprefix=/home/cpan/pit/thr/perl-5.14.1 -Dusethreads'
> hint=recommended, useposix=true, d_sigaction=define
> useithreads=define, usemultiplicity=define
> useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef
> use64bitint=define, use64bitall=define, uselongdouble=undef
> usemymalloc=n, bincompat5005=undef
> Compiler:
> cc='cc', ccflags ='-fno-strict-aliasing -pipe -fstack-protector -I/usr/pkg/include',
> optimize='-O',
> cppflags='-fno-strict-aliasing -pipe -fstack-protector -I/usr/pkg/include'
> ccversion='', gccversion='4.5.3', gccosandvers=''
> intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678
> d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16
> ivtype='long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8
> alignbytes=8, prototype=define
> Linker and Libraries:
> ld='cc', ldflags =' -Wl,-rpath,/usr/pkg/lib -Wl,-rpath,/usr/local/lib -fstack-protector -L/usr/pkg/lib'
> libpth=/usr/pkg/lib /lib /usr/lib
> libs=-lgdbm -lm -lcrypt -lutil -lc -lposix -lpthread
> perllibs=-lm -lcrypt -lutil -lc -lposix -lpthread
> libc=/lib/libc.so, so=so, useshrplib=false, libperl=libperl.a
> gnulibc_version=''
> Dynamic Linking:
> dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E '
> cccdlflags='-DPIC -fPIC ', lddlflags='--whole-archive -shared -L/usr/pkg/lib -fstack-protector'
>
>
> Characteristics of this binary (from libperl):
> Compile-time options: MULTIPLICITY PERL_DONT_CREATE_GVSV
> PERL_IMPLICIT_CONTEXT PERL_MALLOC_WRAP
> PERL_PRESERVE_IVUV USE_64_BIT_ALL USE_64_BIT_INT
> USE_ITHREADS USE_LARGE_FILES USE_PERLIO USE_PERL_ATOF
> USE_REENTRANT_API
> Built under netbsd
> Compiled at Oct 21 2012 13:51:03
> %ENV:
> PERL5LIB=":/home/cpan/pit/thr/conf/perl-5.14.1/.cpanplus/5.14.1/build/DBI-1.622/blib/lib:/home/cpan/pit/thr/conf/perl-5.14.1/.cpanplus/5.14.1/build/DBI-1.622/blib/arch:/home/cpan/pit/thr/conf/perl-5.14.1/.cpanplus/5.14.1/build/Data-Babel-1.10_05/blib/lib:/home/cpan/pit/thr/conf/perl-5.14.1/.cpanplus/5.14.1/build/Data-Babel-1.10_05/blib/arch"
> PERL5_CPANPLUS_IS_RUNNING="17246"
> PERL5_CPANPLUS_IS_VERSION="0.9133"
> PERL5_MINISMOKEBOX="0.54"
> PERL5_YACSMOKE_BASE="/home/cpan/pit/thr/conf/perl-5.14.1"
> PERL_EXTUTILS_AUTOINSTALL="--defaultdeps"
> PERL_MM_USE_DEFAULT="1"
> @INC:
> /home/cpan/pit/thr/conf/perl-5.14.1/.cpanplus/5.14.1/build/DBI-1.622/blib/lib
> /home/cpan/pit/thr/conf/perl-5.14.1/.cpanplus/5.14.1/build/DBI-1.622/blib/arch
> /home/cpan/pit/thr/conf/perl-5.14.1/.cpanplus/5.14.1/build/Data-Babel-1.10_05/blib/lib
> /home/cpan/pit/thr/conf/perl-5.14.1/.cpanplus/5.14.1/build/Data-Babel-1.10_05/blib/arch
> /home/cpan/pit/thr/perl-5.14.1/lib/site_perl/5.14.1/amd64-netbsd-thread-multi
> /home/cpan/pit/thr/perl-5.14.1/lib/site_perl/5.14.1
> /home/cpan/pit/thr/perl-5.14.1/lib/5.14.1/amd64-netbsd-thread-multi
> /home/cpan/pit/thr/perl-5.14.1/lib/5.14.1
> .

[Chris 'BinGOs' Williams]

On Thu, Nov 29, 2012 at 03:22:18PM -0800, Nathan (Nat) Goodman wrote:
> Greetings testers
>
> I'm a module developer, not a tester. For one of my modules, I'm seeing a lot of PASS reports that have no test output; see example below. I'm guessing this means the test didn't run. Any suggestions on how I can get better diagnostics to help me debug the problem?
>
> Many thanks,
> Nat Goodman
>

The test likely did run. And it was a PASS.

What is extremely likely is that some edge condition in CPANPLUS meant that the test output
was not captured to pass to the test reporting.

Cheers,

--
Chris Williams
aka BinGOs
PGP ID 0x4658671F
http://www.gumbynet.org.uk
==========================

[Nathan (Nat) Goodman]

Thanks, Chris! I appreciate the response.

Best,
Nat

[Chris 'BinGOs' Williams]

On Tue, Dec 04, 2012 at 11:59:47AM -0800, Nathan (Nat) Goodman wrote:
> Thanks, Chris! I appreciate the response.
>
> Best,
> Nat
>

Further to my previous message, after building a v5.16.2 perl, setting up
with Test::Reporter::Transport::File and doing numerous tests I have tracked
down a bug in CPANPLUS::Dist::Build, the installer type for Module::Build
based distributions in CPANPLUS.

It seems if a Build.PL exits with 0 (like yours does) without generating a
Build script, that CPANPLUS::Dist::Build then proceeds to try and run this
mythical Build script and eventually generates a PASS report without any
content. This is obviously wrong.

I'm working on a solution.

You can pretty much disregard any PASS test reports from me that have
no test output.

[Nathan (Nat) Goodman]

That's what I feared...

I'm getting some strange FAILs from other testers who try to run the tests even though they're missing key prereqs. Perhaps this is a related bug. I can imagine that the tester code might try to run the Build script that's present in the distribution tarfile even though Build.PL exits with 0.

Thanks for persisting!

Best,
Nat

[David Cantrell]

On Wed, Dec 05, 2012 at 12:00:36AM -0500, bulk 88 wrote:

>   "dcantrell: some really old distributions have no Makefile.PL, so CPAN.pm
> makes one up. "), at one point in history, Perl 4 or early 5 days, XS
> modules didn't come with any makefile.pl or build.pl. MakeMaker or CPAN
> shell created a default makefile to compile that module, kindda like make can
> compile things without a makefile present using builtin rules.

These days it will also check for a Build.PL before just making stuff up.

--
David Cantrell | Cake Smuggler Extraordinaire

Never attribute to malice that which can be explained by stupidity
-- Hanlon's Razor

Stupidity maintained long enough is a form of malice
-- Richard Bos's corollary

[Nathan (Nat) Goodman]

Hi Chris

I'm still seeing test reports with no output. Any chance this bug will be fixed soon? Just wondering.

Thanks,
Nat

On Dec 4, 2012, at 2:02 PM, Chris 'BinGOs' Williams wrote:

[Chris 'BinGOs' Williams]

On Sat, Jul 20, 2013 at 12:07:43PM -0700, Nathan (Nat) Goodman wrote:
> Hi Chris
>
> I'm still seeing test reports with no output. Any chance this bug will be fixed soon? Just wondering.
>
> Thanks,
> Nat
>
> On Dec 4, 2012, at 2:02 PM, Chris 'BinGOs' Williams wrote:
>
> > On Tue, Dec 04, 2012 at 11:59:47AM -0800, Nathan (Nat) Goodman wrote:
> >> Thanks, Chris! I appreciate the response.
> >>
> >> Best,
> >> Nat
> >>

I have just had a chance to look at this (after remembering what the hell it was about).

It appears that what I thought had fixed it, actually hadn't (because I wasn't checking
for the existence of the Build script, though I thought I was). Another change I made at
the same time was reverted because it caused too much pain and exposed this flaw.

I've just tried it and it does indeed not generate a 'PASS'.

[Nathan (Nat) Goodman]

Hi Chris

I'm guessing you made some change because I saw some test reports from you reporting FAIL and UNKNOWN. I've decided to take a different tack.

To reiterate what I'm trying to do: The module in question, Data::Babel, relies heavily on MySQL, and the tests require the ability to create and drop MySQL databases, tables, etc. The distribution includes a check for these capabilities. I'm trying to find a clean way to communicate to the installation driver that the required capabilities are not available. This is very common. If I simply generate a FAIL under these conditions, the module will accumulate a large pile of FAIL reports which I deem to be undesirable.

I read on some Perl website (but have no record where and can't find it now) that the correct idiom for communicating such information to the installation driver is for Build.PL to exit(0) and not create the Build script. I've come to realize this is not the correct idiom based on the fact that all testers but one (or maybe two) who've tested my module treat this condition as a PASS and generate a test report with no output.

Here's what I'm doing now. (1) I removed the check for MySQL capabilities from Build.PL and always create the Build script. (2) I moved the check into a test script; this test asserts TAP version 13, and if the check fails, emits a pragma saying 'stop testing'. (3) I check for the pragma in a customized Build::run_tap_harness method, and break out of the main test loop before running any tests that need MYSQL. (BTW, I've been using a customized run_tap_harness method all along (and do so in many distributions) to support compound tests).

The net effect is that if the required MySQL capabilities are not available, 'Build test' runs a truncated test suite and reports PASS. This is better than what's happening now but not ideal for the obvious reason that it overstates the situations in which my module has been successfully tested. It would be better to report UNKOWN or NA or something in these cases, but I don't know how to make this happen robustly.

Thanks for your help with this issue and all the work you do to keep Perl the best programming community in existence!

Best,
Nat

[Tom Molesworth]

Hi Nat,

On 27/07/13 16:40, Nathan (Nat) Goodman wrote:
> Hi Chris
>

> I'm guessing you made some change because I saw some test reports from you reporting FAIL and UNKNOWN. I've decided to take a different tack.
>
> To reiterate what I'm trying to do: The module in question, Data::Babel, relies heavily on MySQL, and the tests require the ability to create and drop MySQL databases, tables, etc. The distribution includes a check for these capabilities. I'm trying to find a clean way to communicate to the installation driver that the required capabilities are not available. This is very common. If I simply generate a FAIL under these conditions, the module will accumulate a large pile of FAIL reports which I deem to be undesirable.

It seems like this would be an ideal candidate for test_requires => {
'Test::mysqld' => 0 }, since if that installs successfully then the
mysql basics have already been tested for you. It would also avoid
polluting any existing mysql databases (I'm not entirely clear on how
Data::Babel/Class::AutoDB select and use a database but a quick glance
at tests seems to indicate it's a db called 'test' on the localhost
mysql server?).

cheers,

Tom

[Nathan (Nat) Goodman]

Hi Tom

Thanks for pointing me to Test::mysqld. You're right - it's exactly what I need! I'll give it a try here and then move it into AutoDB.

You also correctly surmised that both Babel and AutoDB assume it's okay for their tests to scribble on a db called 'test'. Test::mysqld is a much better solution.

Thanks again,
Nat

[David Cantrell]

On Sat, Jul 27, 2013 at 08:40:53AM -0700, Nathan (Nat) Goodman wrote:

> I read on some Perl website (but have no record where and can't find it now) that the correct idiom for communicating such information to the installation driver is for Build.PL to exit(0) and not create the Build script...

You're probably thinking of the "Notes for CPAN authors" page on
http://wiki.cpantesters.org/ but that site is currently down.

--
David Cantrell | Nth greatest programmer in the world

If you can't imagine how I do something, it's
because I have a better imagination than you

[Nathan (Nat) Goodman]

Hi Tom, Chris

I want to update you on my progress with Test::mysqld and the modules we've been discussing, Data::Babel and Class::AutoDB.

1) I started to integrate Test::mysqld into the Class::AutoDB test suite. It's the right idea; thanks again for the suggestion! The module itself has some small problems (not exactly bugs, more like mis-features). I will communicate the issues to the developer in a bug report. To really use the module, I will need to build a simple wrapper that works around the issues, or get the developer to fix the problems (with my help if s/he desires), or just reimplement the bits of the module I actually need.

2) Looking at the test results and test suite for Test::mysqld, I see that it faced the same problem that got this conversation going, namely, how to tell the test/installation driver that MySQL is not available and thus it is impossible to run the tests. The developer adopted a different (better) solution than I did, but the net effect is similar: when MySQL is not available, the test suite skips all the important tests and reports PASS even though the module wasn't actually tested and won't run on the platform in question. So, even if Test::mysqld installs cleanly, this does not mean that MySQL is there, and does not obviate the need for my module to check for MySQL.

3) What I ended up doing with Data::Babel (and will do with Class:AutoDB soon) is to move the check for MySQL into a test script that runs first. If the script detects that MySQL is unavailable, it reports this to the test driver using the pragma capability of TAP version 13, and a slightly customized version of Build::run_tap_harness (included in my distribution) breaks out of the test suite when it sees this pragma, reporting PASS as the outcome. It would be far better to report NA or UNKNOWN, but there doesn't seem to be a documented way to do this. Please correct me if I'm wrong...

The net effect, as with Test::mysqld, is that Data::Babel will install happily on platforms where it cannot run. This is a complete crock but more satisfying than the alternative of FAILing everywhere.

I appreciate your help and any suggestions as to better ways of solving the problem.

All the best,
Nat

[Steffen Schwigon]

"Nathan (Nat) Goodman" <ngoo...@systemsbiology.org> writes:
> 3) What I ended up doing with Data::Babel (and will do with
> Class:AutoDB soon) is to move the check for MySQL into a test script
> that runs first. If the script detects that MySQL is unavailable, it
> reports this to the test driver using the pragma capability of TAP
> version 13, and a slightly customized version of
> Build::run_tap_harness (included in my distribution) breaks out of the
> test suite when it sees this pragma, reporting PASS as the outcome.

Can you please elaborate more on the details of your pragma solution.

I never saw a TAP pragma in real world[1] and would love to understand
it - without reading into your code, sorry for that laziness... :-)

Kind regards,
Steffen

Footnotes:
[1] except that I support highlighting it in Emacs TAP mode
--
Steffen Schwigon <s...@renormalist.net>

[Nathan (Nat) Goodman]

Hi Steffen

TAP as you probably know is a line-oriented protocol whose main purpose it to let test scripts communicate pass/fail results to a test harness. Functions like 'ok' or 'pass' or 'fail' in Test::More print lines that tell the harness whether a test passed or failed.

The main TAP site is http://testanything.org/. The TAP specification is at http://podwiki.hexten.net/TAP/TAP.html?page=TAP

I learned about TAP pragmas by reading the TAP CPAN documentation and code, esp. TAP::Parser , http://search.cpan.org/~ovid/Test-Harness-3.28/lib/TAP/Parser.pm#pragmas

To make it work you have to tell the TAP harness that you're using TAP version 13, whose specification is at http://podwiki.hexten.net/TAP/TAP13.html?page=TAP13

You do this from a test script by printing this line
TAP version 13

The format of a pragma is pragma +/-<whatever>, eg,
pragma +strict

The + or - is mandatory!

To process a non-standard pragma, you have to customize some aspect of the test harness. I did this by creating a subclass of Module::Build which I needed for other purposes anyway. The subclass is in the Data::Babel 1.12 distribution, file t/Build.pm. You also have to tell the main Build driver to use the subclass. This is in Build.PL; the relevant few lines of code are near the top

use lib qw(t);
use t::Build; # my Module::Build subclass
my $class='t::Build';

followed a few lines later by

my $builder = $class->new
....

I hope this helps. Feel free to ask for further clarification as necessary. I'm glad to help.

Best,
Nat

[Steffen Schwigon]

Hi!

And what particular pragma did you invent, when is it generated, how
does it influence the TAP parsing with the particular "MySQL
unavailable" isse?

Thanks.

Kind regards,
Steffen

--
Steffen Schwigon <s...@renormalist.net>

[Nathan (Nat) Goodman]

The pragma is 'stop_testing'.

The test script that deals with the issue is t/babel.000.reqs.t. Near the top, the scripts says
print "TAP version 13\n";

The script then checks that certain essential modules can be loaded - Class::AutoDB, DBI, DBD::mysql. For reasons I don't understand, despite being listed in the 'build_requires' section of Build.PL, Build tries to do the build even when these prerequisites are not met.

Finally the script tries to connect to MySQL and do the necessary manipulations.

If any of the checks fail, the script constructs a detailed diagnostic message and prints it by saying
diag($diag);
Then it prints the pragma:
print "pragma +stop_testing\n";

The TAP parser processes the pragma and makes this information available to the harness. The code that handles the pragma is in my customized version of run_tap_harness in t/Build.pm. The main part of run_tap_harness is a loop that runs each test script. At the end of each iteration, ie, after running each test script, the code says
my @parsers=$agg->parsers;
my $parser=$parsers[$#parsers];
my @pragmas=$parser->pragmas;
last if grep /stop_testing/i,@pragmas;

This reaches into the TAP aggregator (the aggregator collects results from compound tests - not relevant here) to get the list of parsers that were created to parse the test results - here there will only be one, but I wrote the code for the general case. Then it grabs the last parser and gets the list of pragmas processed by the parser - again there will only be one, but I wrote the code for the general case. If one of the pragmas was stop_testing, it breaks out of the loop and the test suite comes to a halt.

This is not the only way one could solve the problem. Instead of using the pragma mechanism (some might say 'mis-using' since stop_testing is really a directive not a pragma in the proper sense), one could invent a new kind of TAP message and create a TAP parser subclass to process the message. The TAP design envisions this sort of customization and it wouldn't be too hard. I chose to go the pragma route because it was a bit simpler for my limited purpose.

What are you trying to do? Are you looking for a way to check that MySQL is available, or are you thinking about using the pragma mechanism for something else?

Best,
Nat

[Steffen Schwigon]

Thanks for that explanation.

I don't need it now, I was only curious about a particular use-case,
just in case I have a similar one. I know TAP and I also know pragmas
syntactically, but never had a real world use-case for it.

Thanks and regards,

--
Steffen Schwigon <s...@renormalist.net>