Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss
Test::More, what we hope to achieve doc.
4 views
Skip to first unread message
Chad Granum
Apr 8, 2015, 1:15:03 PM4/8/15
to Perl QA
Several people have asked for this, so I wrote it up.
-Chad
Buddy Burden
Apr 12, 2015, 5:15:02 PM4/12/15
to Chad Granum, Perl QA
Chad,
> https://docs.google.com/document/d/1RCqf5uOQx0-8kE_pGHqKSQr7zsJDXkblyNJoVR2mF1A/edit?usp=sharing
>
> Several people have asked for this, so I wrote it up.
Excellent document. And, since I haven't said it yet, I applaud your
bravery for taking on this task.
One thing I would like to see which I don't believe I've seen yet (or
perhaps I just missed it) is a some advice for the DarkPAN-ers out
there. If there were some sort of checklist that we could use to go
through our test suites ahead of time, I for one would love to go ahead
and jump on that. Something like: If your test suite infrastructure
does X, then you will need to change it to Y for the new system. Etc.
Also, the solutions would ideally be things that would work with both
old and new, otherwise getting the timing right may be tricky for many
folks ...
-- Buddy
> https://docs.google.com/document/d/1RCqf5uOQx0-8kE_pGHqKSQr7zsJDXkblyNJoVR2mF1A/edit?usp=sharing
>
> Several people have asked for this, so I wrote it up.
bravery for taking on this task.
One thing I would like to see which I don't believe I've seen yet (or
perhaps I just missed it) is a some advice for the DarkPAN-ers out
there. If there were some sort of checklist that we could use to go
through our test suites ahead of time, I for one would love to go ahead
and jump on that. Something like: If your test suite infrastructure
does X, then you will need to change it to Y for the new system. Etc.
Also, the solutions would ideally be things that would work with both
old and new, otherwise getting the timing right may be tricky for many
folks ...
-- Buddy
Chad Granum
Apr 12, 2015, 5:30:02 PM4/12/15
to Buddy Burden, Perl QA
Thanks for the feedback!
A few people have asked for a transition document like what you are asking for, I have yet to write it, but it is on my list.
The main thing for a darkpan maintainer to do is this:
Run all your tests against the latest dev release on cpan (https://metacpan.org/release/EXODIST/Test-Simple-1.301001_100). This can easily be done with a local::lob if you do not want to actually install it. If all your tests pass then there is nothing you need to do, everything just works. If this is the case please mention it somewhere, such as on this list. So far I have gotten only a handful of darkpan results, none of them reporting anything broken. It would be nice if there was a public list of darkpan results I can keep, and I may start one on github.
If something is broken it would be best to send me a bug report on github. My goal is to break as little as possible, so if I do break something for you I would prefer fixing it over making you change you things. The only exceptions are when things are very crazy: Replacing the Test::Builder singleton, or breaking encapsulation by directly accessing hash keys of the Test::Builder object. The new stuff even goes so far as to support legacy monkeypatching of key methods in the Test::Builder namespace.
-Chad
Chad Granum
Apr 12, 2015, 6:00:04 PM4/12/15
to Buddy Burden, Perl QA
https://github.com/Test-More/test-more/blob/stream/replace/lib/Test/Stream/Transition.pod
Initial, and very sparse transition document. If there are specific things you want added please feel free to comment on it, or make requests :-)
Martin J. Evans
Apr 13, 2015, 4:30:03 AM4/13/15
to exod...@gmail.com, per...@perl.org
On 12/04/15 22:14, Chad Granum wrote:
> Thanks for the feedback!
>
> A few people have asked for a transition document like what you are asking for, I have yet to write it, but it is on my list.
>
> The main thing for a darkpan maintainer to do is this:
I have a large darkpan test suite but when attempting to install this:
> Thanks for the feedback!
>
> A few people have asked for a transition document like what you are asking for, I have yet to write it, but it is on my list.
>
> The main thing for a darkpan maintainer to do is this:
https://metacpan.org/release/EXODIST/Test-Simple-1.301001_100)
release I got:
t/xxx-changes_updated.t ............ skipped: Only tested when releasing
t/zzz-check-breaks.t ............... Can't locate object method "requirements_for_module" via package "CPAN::Meta::Requirements" at t/zzz-check-breaks.t line 43.
t/zzz-check-breaks.t ............... Dubious, test returned 25 (wstat 6400, 0x1900)
No subtests run
Test Summary Report
-------------------
t/zzz-check-breaks.t (Wstat: 6400 Tests: 0 Failed: 0)
Non-zero exit status: 25
Parse errors: No plan found in TAP output
Files=30, Tests=214, 5 wallclock secs ( 0.35 usr 0.08 sys + 4.11 cusr 0.52 csys = 5.06 CPU)
Result: FAIL
Failed 1/30 test programs. 0/214 subtests failed.
Run individually:
prove -vb t/zzz-check-breaks.t
t/zzz-check-breaks.t .. Can't locate object method "requirements_for_module" via package "CPAN::Meta::Requirements" at t/zzz-check-breaks.t line 43.
Dubious, test returned 25 (wstat 6400, 0x1900)
No subtests run
Test Summary Report
-------------------
t/zzz-check-breaks.t (Wstat: 6400 Tests: 0 Failed: 0)
Non-zero exit status: 25
Parse errors: No plan found in TAP output
Files=1, Tests=0, 0 wallclock secs ( 0.06 usr 0.01 sys + 0.24 cusr 0.04 csys = 0.35 CPU)
Result: FAIL
This is perl 5, version 16, subversion 0 (v5.16.0) built for i686-linux
I will happily run our darkpan test suite if I get past this point.
I'll post this on github if you prefer.
Martin
Kent Fredric
Apr 13, 2015, 5:00:02 AM4/13/15
to Martin J. Evans, Chad Granum, per...@perl.org
On 13 April 2015 at 20:21, Martin J. Evans <martin...@easysoft.com> wrote:
release I got:
t/xxx-changes_updated.t ............ skipped: Only tested when releasing
t/zzz-check-breaks.t ............... Can't locate object method "requirements_for_module" via package "CPAN::Meta::Requirements" at t/zzz-check-breaks.t line 43.
t/zzz-check-breaks.t ............... Dubious, test returned 25 (wstat 6400, 0x1900)
No subtests run
Test Summary Report
-------------------
t/zzz-check-breaks.t (Wstat: 6400 Tests: 0 Failed: 0)
Non-zero exit status: 25
Parse errors: No plan found in TAP output
Files=30, Tests=214, 5 wallclock secs ( 0.35 usr 0.08 sys + 4.11 cusr 0.52 csys = 5.06 CPU)
Result: FAIL
Failed 1/30 test programs. 0/214 subtests failed.
A quick analysis says that that function call didn't exist before CPAN::Meta::Requirements 2.120920:
https://metacpan.org/changes/release/DAGOLDEN/CPAN-Meta-2.120920#L7
https://metacpan.org/changes/release/DAGOLDEN/CPAN-Meta-2.120920#L7
So that test should call CPAN::Meta::Requirements->VERSION(2.120920) in the eval to skip it at install time if CPAN::Meta::Requirements is not new enough.
Kent Fredric
Apr 13, 2015, 6:00:07 AM4/13/15
to Martin J. Evans, Chad Granum, per...@perl.org
I'll post this on github if you prefer.
Martin
And done on your behalf: https://github.com/Test-More/test-more/issues/563
Chad Granum
Apr 13, 2015, 12:15:02 PM4/13/15
to Kent Fredric, Martin J. Evans, Perl QA
Just cut a new release that fixes that unit test.
Martin J. Evans
Apr 14, 2015, 4:00:03 AM4/14/15
to Chad Granum, Buddy Burden, Perl QA
On 12/04/15 22:14, Chad Granum wrote:
> Thanks for the feedback!
>
> A few people have asked for a transition document like what you are
> asking for, I have yet to write it, but it is on my list.
>
> The main thing for a darkpan maintainer to do is this:
>
> Run all your tests against the latest dev release on cpan
> (https://metacpan.org/release/EXODIST/Test-Simple-1.301001_100). This
> can easily be done with a local::lob if you do not want to actually
> install it. If all your tests pass then there is nothing you need to
> do, everything just works. If this is the case please mention it
> somewhere, such as on this list. So far I have gotten only a handful
> of darkpan results, none of them reporting anything broken. It would
> be nice if there was a public list of darkpan results I can keep, and
> I may start one on github.
>
> If something is broken it would be best to send me a bug report on
> github. My goal is to break as little as possible, so if I do break
> something for you I would prefer fixing it over making you change you
> things. The only exceptions are when things are very crazy: Replacing
> the Test::Builder singleton, or breaking encapsulation by directly
> accessing hash keys of the Test::Builder object. The new stuff even
> goes so far as to support legacy monkeypatching of key methods in the
> Test::Builder namespace.
>
> -Chad
Hi Chad,
>
> A few people have asked for a transition document like what you are
> asking for, I have yet to write it, but it is on my list.
>
> The main thing for a darkpan maintainer to do is this:
>
> Run all your tests against the latest dev release on cpan
> (https://metacpan.org/release/EXODIST/Test-Simple-1.301001_100). This
> can easily be done with a local::lob if you do not want to actually
> install it. If all your tests pass then there is nothing you need to
> do, everything just works. If this is the case please mention it
> somewhere, such as on this list. So far I have gotten only a handful
> of darkpan results, none of them reporting anything broken. It would
> be nice if there was a public list of darkpan results I can keep, and
> I may start one on github.
>
> If something is broken it would be best to send me a bug report on
> github. My goal is to break as little as possible, so if I do break
> something for you I would prefer fixing it over making you change you
> things. The only exceptions are when things are very crazy: Replacing
> the Test::Builder singleton, or breaking encapsulation by directly
> accessing hash keys of the Test::Builder object. The new stuff even
> goes so far as to support legacy monkeypatching of key methods in the
> Test::Builder namespace.
>
> -Chad
I have now run our entire darkpan test suite with the version of Test::Simple above. It takes around 3 hours to run :-( and tests 3 APIs (one of which is very large) using JSON sent over various transports (including HTTP). I found no differences from the the previous (and much older) version.
Our test suite uses, Test::More, Test::Deep, Test::Group and Test::Builder. The only thing we did I was unsure would work was overriding Test::Builder::ok and Test::Builder::BAIL_OUT as follows:
if ($self->{verbose}) {
Test::Group->verbose($args{verbose}); # so we can see the subtests run:
# Part copied from
# http://www.nntp.perl.org/group/perl.qa/2009/05/msg12229.html
# add call frame to ok's if verbose running
my $ok = \&Test::Builder::ok;
#no warnings 'redefined';
*Test::Builder::ok = sub {
my ($package, $filename, $line);
my $frame = 0;
my $frames = '';
for (;;) {
( $package, $filename, $line ) = caller($frame);
last if !defined $package;
$frame++;
next if $filename =~ /^\/usr/;
$frames .= "$package:$line, ";
}
$_[0]->diag("ok, $frames") if $frames;
goto $ok;
};
# redefine the BAIL_OUT method to show call stack:
{
my $bail_out = \&Test::Builder::BAIL_OUT;
*Test::Builder::BAIL_OUT = sub {
my ($package, $filename, $line);
my $frame = 0;
my $frames = '';
for (;;) {
( $package, $filename, $line ) = caller($frame);
last if !defined $package;
$frame++;
#next if $filename =~ /^\/usr/;
$frames .= "$package:$filename:$line, ";
}
$_[1] = "$_[1] - $frames";
goto $bail_out;
};
}
but that did not seem to cause a problem.
I hope this gives you some further confidence.
Martin
0 new messages