Proposal: refactor the test suite according to synopsis

[Patrick R. Michaud]

Since the perl6 compiler on Parrot is moving ahead nicely in
its new incarnation, it's time for us to start looking at the
official test suite again. As a reminder, Pugs is considered
the repository for the "official test suite" at the moment
( http://svn.pugscode.org/pugs/t/ ).

Having looked at the test suite at various times in the past,
I know that I've often found it difficult to determine where
a given test would/should be located. So, I'd like to start
reorganizing the suite to hopefully make it easier to locate
existing tests and to know where to add new ones.

There are certainly plenty of organization schemes we could use
for this, but one that leaps to my mind is to organize tests based
loosely on where the feature is described in the synopses or the
camel book (the p5 camel book for now, the p6 camel book when it
comes out). So, if we named subdirectories of t/ with the
synopsis number and major subsection name, we could end up with
something like (names here are just for illustration at the moment):

t/
01-sanity/ tests for basic sanity items
01-test/ tests for the Test.pm harness
...
02-comments/ whitespace and comments
02-datatypes/ built-in data types
02-variables/ variables
02-names/ names and scopes
02-literals/ literals
...
03-basic/ basic operators and precedence
03-junctive/ junctive operators
03-comparison/ comparison operators
03-range/ range operators
03-smartmatch/ smartmatch tests
03-meta/ meta operators
03-declarators/ my, our, etc.
...
04-conditional/ if, unless, elsif, etc.
04-loop/ while, until, repeat, for
04-switch/ given, when
04-exceptions/ try, CATCH, exception types, die
...etc...

I'm thinking of this organization scheme as primarily a guiding
principle and not something to be strictly applied. For example,
I wouldn't advocate a strict one-to-one correspondence between
subdirectories and major headings, and I don't think we need to
create directories that end up containing only a single test file.
I.e., each chapter could have a XX-other directory to catch
items that don't fit into another top-level directory, such as
02-misc/, 03-misc/, etc.

Also, I'd consider it okay (but hopefully rare) to create
subdirectories within a major section if there are enough
tests that need to be organized that way.

Any thoughts (positive/negative) about this approach? Unless
there are major objections I will probably start looking into
reorganizing some of the basic tests in the suite along these
principles.

Thanks!

Pm

[Larry Wall]

On Thu, Dec 20, 2007 at 11:27:14AM -0600, Patrick R. Michaud wrote:
: I'm thinking of this organization scheme as primarily a guiding

: principle and not something to be strictly applied. For example,
: I wouldn't advocate a strict one-to-one correspondence between
: subdirectories and major headings

Yes, especially considering that most of the Synopses have never
been reorganized from their original braindump. Only S03 has seen
major refactoring from the original, and most of the others are rather
randomly organized, especially the earlier ones. Part of the problem
is that redoing the headings tends to break a lot of smart links.

Larry

[Jerry Gay]

On Dec 20, 2007 9:27 AM, Patrick R. Michaud <pmic...@pobox.com> wrote:
> There are certainly plenty of organization schemes we could use
> for this, but one that leaps to my mind is to organize tests based
> loosely on where the feature is described in the synopses or the
> camel book (the p5 camel book for now, the p6 camel book when it
> comes out). So, if we named subdirectories of t/ with the
> synopsis number and major subsection name, we could end up with
> something like (names here are just for illustration at the moment):
>
> t/
> 01-sanity/ tests for basic sanity items
> 01-test/ tests for the Test.pm harness
> ...
> 02-comments/ whitespace and comments
> 02-datatypes/ built-in data types
> 02-variables/ variables
> 02-names/ names and scopes
> 02-literals/ literals
> ...
>

this is on the right track, but still a bit confusing to me. many
tests for cpan modules use a similar structure, but it often implies
test directory execution order. this is not what you mean by the
directory naming convention you've chosen.

also, as it is, this scheme leaves no room for ordered test
directories at a later date. i can't think of a use for them in this
instance off the top of my head, but people smarter than me might find
one eventually.

here's my suggestions, based on your structure

t/
spec/
S02-comments/
S02-misc/ # or -other

this gives us a nice tidy location for all spec tests, and helps
clarify that the directory naming isn't meant to imply test execution
ordering (although there are few forward references in the synopses by
design, few != none.)

it also gives us room for additional directories, for example
t/compilers/perl6/00-parrot/, t/compilers/kp6/, t/misc/, and as i said
previously, t/01-sanity/ etc.)

> Any thoughts (positive/negative) about this approach? Unless
> there are major objections I will probably start looking into
> reorganizing some of the basic tests in the suite along these
> principles.
>

great idea! i'd love to help. as a start, we can create or use a
utility (like 'ack') to hunt for smartlinks in the existing test files
and move the test files to whatever structure we've agreed upon. that
should address the first 80%. then manual review can be done at the
file level to do the second 80%. the last 80% is the hardest:
reorganizing the subtests within and between the files.

however it gets done, i think it'll help all the perl 6 contributors
immensely. it'll certainly help me find the test files i need to run
in order to implement new bits of perl6.
~jerry