making v6 test suite its own distro
Darren Duncan
Now that we are fulfilling an intention for Perl 6 to have multiple
fully decoupled implementations that are distinct from its spec, Pugs
and v6.pm being the largest short term examples, I propose that we
make like Sun and its Java VM validation suite, and start
distributing the Perl 6 test suite as its own distribution on CPAN,
which Perl6::Pugs and v6 then have as optional (for testing only)
external dependencies.
This way, for example, people who just want to use the pure Perl 5
v6.pm implementation of Perl 6 don't have to download Pugs in order
to test it.
Audrey suggested that it could be named v6::tests ... I previously to
that suggested Perl6::Certification ... and other suggestions are
welcome.
From that point onward, neither the Perl6::Pugs nor v6 distros would
include their own test suites, or such would be extremely basic.
Their 'make test' and 'make smoke' would instead call out to
v6::tests.
(A substantial part of the testing and/or smoking harness should
probably also be included, so people that use it get more consistent
formatted results that can be compared, though some details will need
to be in the individual implementation distros, as only they know how
they are invoked. Whether that is written in Perl 6 or Perl 5 or
something else is less of an issue short term.)
On a separate but related matter, I suggest also separating anything
else that isn't Pugs specific, that is common to both it and v6, from
the Pugs distro and distributing those separately. This includes the
contents of: examples/ , ext/, some of docs/, some of util/, and so
on.
These various distros can be distributed on their own schedule, as
frequent or infrequent as makes sense, just as v6 and Moose et al are.
As to whether any of the above are still maintained within Pugs SVN
or moved to separate SVN, that is a possibly orthogonal matter.
Seeing as the next Pugs release, 6.28.0, is psychologically a major
release, I suggest that the various decoupling I am talking about
above be accomplished prior to that, so that Pugs 6.28.0 and later
are slimmed down and don't contain the shared components. The Pugs
only distro will probably feature what is in src/ now and what is
needed to support that. Audrey suggested separation before OSCON,
even.
Of course, feedback always welcome.
-- Darren Duncan
Gaal Yahas
> As briefly discussed on #perl6 ...
As briefly replied there before being jethandled...
> I propose that we make like Sun and its Java VM validation suite,
> and start distributing the Perl 6 test suite as its own distribution
> on CPAN
Here are a number of issues.
1. Naming
> Audrey suggested that it could be named v6::tests ... I previously to
> that suggested Perl6::Certification ... and other suggestions are
> welcome.
Please let's not call it certification. That gives foreground to a
question of authority which I think is (certainly at this stage,
possibly later too) a needless distraction.
2. Modularity
> From that point onward, neither the Perl6::Pugs nor v6 distros would
> include their own test suites, or such would be extremely basic.
> Their 'make test' and 'make smoke' would instead call out to
> v6::tests.
>
> (A substantial part of the testing and/or smoking harness should
> probably also be included, so people that use it get more consistent
> formatted results that can be compared, though some details will need
> to be in the individual implementation distros, as only they know how
> they are invoked. Whether that is written in Perl 6 or Perl 5 or
> something else is less of an issue short term.)
The Pugs test system has grown organically and has functionality at the
following levels, most of which currently reside in the Pugs repo:
A) A Perl 6 Test.pm that borrows heavily from Perl 5 TAP goodies: ok, is,
like, cmp_ok, isa_ok, is_deeply, dies_ok, skip, and related functions;
as well as conventions for extra TAP info supplying message coordinates
and skip/todo reasons, used in reports [ext/Test/ in the Pugs repo].
B) A Perl 5 Straps harness that keeps more metadata around for machine
processing than Test::Harness' default "make smoke", and which
has the extra bonus of working faster on SMP machines [originally
util/yaml_harness.pl in the Pugs repo, which has since been refactored
to mostly be a pugs-specific front-end to Test::TAP::Model on CPAN].
C) Local report generators: a colorful TAP results generator
[Tetst::TAP::HTMLMatrix on CPAN and a frontend in util/testgraph.pl]
and a FIT-and-Project-Xanadu inspired cross-referencer into arbitrary
locations in the spec and the tests [util/catalog-tests.pl].
D) A smoke server system for transmitting and viewing other smoker's
reports [code in util/smokeserv/ ; service maintained by Ingo
Blechschmidt].
E) Scripts and cron jobs on feather to autosmoke pugs every revision and
submit results [currently maintained by Juerd, I believe].
(The slides at http://perlcabal.org/~gaal/pugstest/start.html demonstrate
most of these features.)
The first obvious question that this enumeration presents is: where to
cut? Does every implementation use its own Test.pm? Pugs was able to
start using one after 23 days, but that was a simpler module then than
it is today, and a fresh implementation may find it hard to get started
if its first objective is to implement so much Perl 6 simultaneously.
Perhaps we need a baby-Perl Test::Simple for new implementations, and
by convention have 01-sanity use only those functions present there. The
drawback being somewhat duplicated effort and another distribution worry.
The other question that comes to mind is how to manage SKIP and TODO
tests for release mangement. Currently Pugs, assuming globality, simply
plunges into every t/ file before a release and marks forsaken (for this
release) tests TODO, and commits. But surely different implementations
have their different bugs. In the case of TODO we can fix that easily:
we already have a mechanism for force_todoing tests "at a distance",
though currently that distance is at the head of the test file (but it's
a simple matter to make that a distro-level file). The deeper technical
challenge is how to maintain SKIPs, which after all, are often used
before hanging tests. skip_all we can manage; skip we cannot, until at
least when we have a powerful debugger.
3. Generality
> On a separate but related matter, I suggest also separating anything
> else that isn't Pugs specific, that is common to both it and v6, from
> the Pugs distro and distributing those separately. This includes the
> contents of: examples/ , ext/, some of docs/, some of util/, and so
> on.
>
> These various distros can be distributed on their own schedule, as
> frequent or infrequent as makes sense, just as v6 and Moose et al are.
>
> As to whether any of the above are still maintained within Pugs SVN
> or moved to separate SVN, that is a possibly orthogonal matter.
As a Pugs hacker I can say that the immediacy of access to t/ is a
great benefit, and I would hate to lose that. But what's the best way
of granting that benefit to other implementations without crippling it
for anyone?
And a different tack on all of this: at YAPC I heard from Jesse Vincent
that work has been done in Best Practical to refactor the test system
for a non-Perl 6 project, or rather, collection of products. Their
effort is called Chimps and its repo lives at:
http://svn.bestpractical.com/cgi-bin/index.cgi/bps/browse/Test-Chimps/trunk/
It is still very much work in progress! But I did see a demo and it's
looking good. They are focusing on a multi-product smoke server built
on our stuff (thanks for the compliment, guys!). (Yuval Kogman had also
been putting some work on that before hearing of Chimps, but I think he
put that on low flame due to unvocal demand.) The Best Practical stuff
will be open source and they sound happy to both receive and give back
code contributions.
So, I think there's a very big overlap between several different
users. I'd love for more cooperation and a single code base (though of
course sometimes different smokeserver services) for
- Pugs hackers during development
- Other Perl 6 implementors
- @Larry as people starting from a spec doc and exploring how well a
particular feature is tested and supported by the implementations
(the game is "follow the superscript Ts":
http://perlcabal.org/~gaal/pugstest/t_index/Synopsis/S06.html#175425272
).
- A "product" company with more than one report subject, possibly not
100%-public
- CPAN hackers?
- CPAN users?
As you can see, there are possibly wider and wider scopes of problems to
solve here and we can certainly choose not to attack every problem now.
But yes, let's get the discussion going (thanks, Darren, for starting
this thread) and come up with a plan, so that we can be ambitious and
realistic together.
--
Gaal Yahas <ga...@forum2.org>
http://gaal.livejournal.com/
Darren Duncan
>On Sat, Jul 08, 2006 at 12:47:17AM -0700, Darren Duncan wrote:
>> As briefly discussed on #perl6 ...
>
>As briefly replied there before being jethandled...
As further discussed there ...
>Perhaps we need a baby-Perl Test::Simple for new implementations, and
>by convention have 01-sanity use only those functions present there. The
>drawback being somewhat duplicated effort and another distribution worry.
I believe that the current Test.pm already qualifies as a baby-Perl
implementation, as overall its functionality is quite simple, and it
uses very little of the language (especially since its use of
junctions was removed a few weeks ago).
Have a look at 01-sanity again. Since I moved 08-test out of that
folder (and into 02-test-pm) none of those 01-sanity tests use
Test.pm at all.
My impression of 01-sanity is that it checks that the Perl 6
implementation has the basic language functions that Test.pm has as a
prerequisite, such as being able to print, knowing basic
conditionals, supporting basic subroutines and modules.
If 01-sanity runs, then the normal Test.pm should be useable, and
02-test-pm checks that Test.pm itself actually works. When that
runs, then the other tests can be performed as usual.
>The first obvious question that this enumeration presents is: where to
>cut? Does every implementation use its own Test.pm? Pugs was able to
>start using one after 23 days, but that was a simpler module then than
>it is today, and a fresh implementation may find it hard to get started
>if its first objective is to implement so much Perl 6 simultaneously.
I would say for starters that anything which is written in Perl 6
will be shared by all implementations, and that includes Test.pm.
Considering its few needs, any Perl that can't run the current
Test.pm is hardly useful, so they would be getting up to the level of
supporting Test.pm very quickly. And in the process of getting to
that level, the 01-sanity tests can be tried without Test.pm.
>The other question that comes to mind is how to manage SKIP and TODO
>tests for release mangement. Currently Pugs, assuming globality, simply
>plunges into every t/ file before a release and marks forsaken (for this
>release) tests TODO, and commits. But surely different implementations
>have their different bugs. In the case of TODO we can fix that easily:
>we already have a mechanism for force_todoing tests "at a distance",
>though currently that distance is at the head of the test file (but it's
>a simple matter to make that a distro-level file). The deeper technical
>challenge is how to maintain SKIPs, which after all, are often used
>before hanging tests. skip_all we can manage; skip we cannot, until at
>least when we have a powerful debugger.
All unconditional skips/todos in the test suite should be eliminated
entirely, and be allowed to fail. If hangs are a concern, then the
test harness (presumably not written in Perl 6) can just have a
timeout function and kill any tests that run for too long (the time
threshold can be changed per user in config.yml).
The proper way to have skips/todos is for them to be conditional.
Ideally, one of the first things that any Perl 6 implementation would
provide is something akin to a %FEATURES (or some other named) global
or environmental or config variable, which a Perl 6 script can query
at runtime to see whether that implementation CLAIMS to support
certain features. Only features that are claimed to be supported are
tested, to yield a ok/not ok, and only those features claimed to be
unsupported, either explicitly, or implicity by not being mentioned
at all, will be skipped.
A brand new Perl 6 implementation will declare an empty %FEATURES,
meaning that everything is unsupported, and so everything conditional
skips (which would be most tests, ideally). As the implementation
matures, it starts adding items to %FEATURES, and so skips are
replaced by passes or fails.
Of course, certain sanity items will have to work before %FEATURES
does, such as the ability to print, basic conditionals and boolean
expressions, the ability to read from a variable, etc. But no
support for subroutines or external modules will be needed in order
to do this, so the threshold is lower than for any type of Test.pm.
The %FEATURES variable would be populated in the logical fashion,
derived partly or entirely from explicit declarations in the
implementation code itself, and partly or not at all from user
configuration or environment as is applicable. Entirely from code
may be better so that this doesn't make the CONFIG or ENV or whatever
vars redundant, or they could possibly overlap as makes sense, but my
emphasis is on the code.
If we are going to go with this idea, and I give it my highest level
of recommendation, it would stand to reason that the possible
features to be tested for will be standardized and documented such as
in a Synopsis.
I could start enumerating such a list if no one else wants to, but it
would be long and would stand for a thought out design, like the
tests themselves.
More broadly speaking, I think that a revisiting of the test
hierarchy with a goal to it being able to scale better (see TASKS of
Pugs), and a capabilities/features list could be done somewhat hand
in hand. Features that tend to be implemented together can also be
grouped together for testing, and tests could be grouped generally in
dependency order, such that if feature A needs to be implemented
before feature B can be implemented, feature A will be tested first.
A trivial case of such organizing has started with 01-sanity and
02-test-pm being checked first, but that's as far as it goes so far.
>As a Pugs hacker I can say that the immediacy of access to t/ is a
>great benefit, and I would hate to lose that. But what's the best way
>of granting that benefit to other implementations without crippling it
>for anyone?
From the point of view of a *user* of Perl 6, one that gets it from
CPAN, getting this to work is just a matter of packing the tests in
such a way that they can be "installed" some how and be visible to
the packages for Pugs or v6 or whatever.
This may require something unusual, because afaik, only "modules"
typically get installed and become visible to other packages, while
test scripts don't.
(I get around this perceived limitation with Rosetta by implementing
all the backend-shared tests as Rosetta::Validator::* modules, which
install with the framework and can be invoked like a module. But I
don't know if something like that is feasible for a general Perl 6
test, so something else may be required.)
As for hackers working from version control, where immediacy of
access to and seamless integration of updates of otherwise separately
distributed components is important, I don't know exactly what to
suggest as a solution but perhaps something that lets you checkout
multiple SVN into a shared space where the pieces can see each other
as if they were one big distro, and you can just test how the parts
work together naturally. I don't know how to implement this, though.
>As you can see, there are possibly wider and wider scopes of problems to
>solve here and we can certainly choose not to attack every problem now.
>But yes, let's get the discussion going (thanks, Darren, for starting
>this thread) and come up with a plan, so that we can be ambitious and
>realistic together.
I think it is very important to share whatever resources can be
shared, for easier development and better quality results. If this
causes a few hiccups short term, like test fails rather than skips,
or all tests temporarily skipping, I think it is still worth it.
-- Darren Duncan
Gaal Yahas
> I believe that the current Test.pm already qualifies as a baby-Perl
> implementation, as overall its functionality is quite simple, and it
> uses very little of the language (especially since its use of
> junctions was removed a few weeks ago).
Test.pm currently uses the following non-trivial language features:
- implicitly, the ability to load a module
- function export
- function named args
- parameter default values
- &caller
- $?CALLER::CALLER::POSITION
- &sub.goto
- %*ENV
- END {...}
Some of these can be given up on in a prospective Test::Simple, if only
because they in themselves shouldn't be the first things an implementation
should strive to provide for their own sake.
> Have a look at 01-sanity again. Since I moved 08-test out of that
> folder (and into 02-test-pm) none of those 01-sanity tests use
> Test.pm at all.
>
> My impression of 01-sanity is that it checks that the Perl 6
> implementation has the basic language functions that Test.pm has as a
> prerequisite, such as being able to print, knowing basic
> conditionals, supporting basic subroutines and modules.
>
> If 01-sanity runs, then the normal Test.pm should be useable, and
> 02-test-pm checks that Test.pm itself actually works. When that
> runs, then the other tests can be performed as usual.
Some things are missing in 01-sanity/ if the particular implementation
of our current Test.pm is the par for tests there.
Clearly more tests should be added, but we can permit fledgling
implementations to use their own worse-is-better Test modules. So these
tests should perhaps go into 02-test-pm/, not 01-sanity/.
> I would say for starters that anything which is written in Perl 6
> will be shared by all implementations, and that includes Test.pm.
>
> Considering its few needs, any Perl that can't run the current
> Test.pm is hardly useful, so they would be getting up to the level of
> supporting Test.pm very quickly. And in the process of getting to
> that level, the 01-sanity tests can be tried without Test.pm.
(Responded to above.)
> >The other question that comes to mind is how to manage SKIP and TODO
> >tests for release
>
> All unconditional skips/todos in the test suite should be eliminated
> entirely, and be allowed to fail. If hangs are a concern, then the
> test harness (presumably not written in Perl 6) can just have a
> timeout function and kill any tests that run for too long (the time
> threshold can be changed per user in config.yml).
Can you suggest a portable implementation for killing hanged tests?
What sort of granularity are you thinking of for this variable? Some
test files have one subtest fail for a reason somewhere deep in the
internals. These reasons tend to be implementation specific and lack
good names: "pending AST refactoring" is both spatially and temporally
imprecise, for it affects many surface features and yet there may well
be further refactorings in the future. And yet this, for example, is
one of the things blocking many tests from being fixed for the next
version of Pugs. Nearly all these features sorta work today, but not
100% correctly. What does and does not work from the surface looks quite
arbitrary. How would you set %FEATURES for them?
> More broadly speaking, I think that a revisiting of the test
> hierarchy with a goal to it being able to scale better (see TASKS of
> Pugs), and a capabilities/features list could be done somewhat hand
> in hand. Features that tend to be implemented together can also be
> grouped together for testing, and tests could be grouped generally in
> dependency order, such that if feature A needs to be implemented
> before feature B can be implemented, feature A will be tested first.
> A trivial case of such organizing has started with 01-sanity and
> 02-test-pm being checked first, but that's as far as it goes so far.
I agree that's a good goal, and that sanity tests are the first place to
do it. What's next up?
> >As a Pugs hacker I can say that the immediacy of access to t/ is a
> >great benefit, and I would hate to lose that. But what's the best way
> >of granting that benefit to other implementations without crippling it
> >for anyone?
>
> From the point of view of a *user* of Perl 6, one that gets it from
> CPAN, getting this to work is just a matter of packing the tests in
> such a way that they can be "installed" some how and be visible to
> the packages for Pugs or v6 or whatever.
>
> This may require something unusual, because afaik, only "modules"
> typically get installed and become visible to other packages, while
> test scripts don't.
As a first approximation, this doesn't have to be installable in a
system location.
Jerry Gay
> On Mon, Jul 10, 2006 at 03:55:25PM -0700, Darren Duncan wrote:
> > I believe that the current Test.pm already qualifies as a baby-Perl
> > implementation, as overall its functionality is quite simple, and it
> > uses very little of the language (especially since its use of
> > junctions was removed a few weeks ago).
>
> Test.pm currently uses the following non-trivial language features:
>
> - implicitly, the ability to load a module
> - function export
> - function named args
> - parameter default values
> - &caller
> - $?CALLER::CALLER::POSITION
> - &sub.goto
> - %*ENV
> - END {...}
>
parrot/perl6 implementation.
> Some of these can be given up on in a prospective Test::Simple, if only
> because they in themselves shouldn't be the first things an implementation
> should strive to provide for their own sake.
>
i hesitate to call a test module that uses a limited language
'::Simple,' as this word has been traditionally used for modules which
ease usability, not those that make the language implementor's job
easier. perhaps 'Test::Perl6::Sanity' might better describe testing
the sanity of a perl6 implementation... *ahem* really hard not to make
a snide remark here *cough*.
> > Have a look at 01-sanity again. Since I moved 08-test out of that
> > folder (and into 02-test-pm) none of those 01-sanity tests use
> > Test.pm at all.
> >
> > My impression of 01-sanity is that it checks that the Perl 6
> > implementation has the basic language functions that Test.pm has as a
> > prerequisite, such as being able to print, knowing basic
> > conditionals, supporting basic subroutines and modules.
> >
> > If 01-sanity runs, then the normal Test.pm should be useable, and
> > 02-test-pm checks that Test.pm itself actually works. When that
> > runs, then the other tests can be performed as usual.
>
> Some things are missing in 01-sanity/ if the particular implementation
> of our current Test.pm is the par for tests there.
>
i'd love to see tests refactored to make 01-sanity/ the first wicket
(to use a croquet analogy) for testing a perl6 implementation.
> Clearly more tests should be added, but we can permit fledgling
> implementations to use their own worse-is-better Test modules. So these
> tests should perhaps go into 02-test-pm/, not 01-sanity/.
>
agreed. then higher-level constructs can be used in the *real*
Test.pm. in my opinion, Test.pm should be idiomatic perl6, as its
audience is well beyond we the foolish few perl6 implementors.
> > I would say for starters that anything which is written in Perl 6
> > will be shared by all implementations, and that includes Test.pm.
> >
> > Considering its few needs, any Perl that can't run the current
> > Test.pm is hardly useful, so they would be getting up to the level of
> > supporting Test.pm very quickly. And in the process of getting to
> > that level, the 01-sanity tests can be tried without Test.pm.
>
> (Responded to above.)
>
i'd like to note that in order to get to the 01-sanity/ tests, we
(parrot/perl6 implementors) needed some tests of more basic building
blocks. for these tests, we created 00-parrot/. here we test literals,
math/logic/comparison ops, variable assignment/interpolation, inplace
math/string ops, and regex smartmatching. i suppose now i should call
this the 0th wicket. (that's what i hate about analogies--now i've
limited the audience who understands this one from *all croquet
players* to *all cs-geek croquet players*.)
i'm staked out. tea anyone?
~jerry