Integrating the Pugs test suite into the Synopses
Agent Zhang
When you see the title, you may wonder what that means. Here is the answer:
http://feather.perl6.nl/~agentzh/syn/S04.html
Search for links named like "Show the snippet from ..." and click on
them to find out what will happen. :)
Well, in short, we have divided the .t files in the Pugs test suite
into pieces and inserted every resulting snippet after the
corresponding paragraph of the Synopses.
The job was done by the Perl 5 script util/smartlinks.pl living in the
Pugs source tree. And all the *.html under
http://feather.perl6.nl/~agentzh/syn/ are updated every hour with the
latest Synopses and Pugs test suite. Thanks to the cron program on
feather. ;-)
At this point, there is a question. How does smartlinks.pl manage to
determine which snippet of the tests corresponds to which paragraph of
a certain synopsis?
Hmm, I'm sure you may have already found the answer from the web
pages. There's hundreds of "smartlinks" spreading over the whole test
suite in the form of comments. Here're two examples:
# L<S04/The do-once loop/"can't" put while or until modifier>
# L<S04/"Switch statements" /"is exactly equivalent to">
You see, the "smartlink" has its deep root in our POD, but contains an
extra keyword list after the second slash. This list of keywords
specifies which paragraph in the given section is relevant to the
tests right below the comment.
Multiple adjacent smartlinks can share the same group of tests:
# L<S02/Context/boolean "?">
# L<S03/Changes to Perl 5 operators/"?" imposes boolean context>
{ ... }
Please see t/README in Pugs repository for more detailed information
on smartlinks:
http://svn.openfoundry.org/pugs/t/README
Embedding test snippets into the Synopses has several benefits:
* Providing more sample code to the Synopses
Synopses are known for their rich samples. Unfortunately, the sample
code our Synopses are not enough at all. Many sections go without
sample code.
On the other hand, the Pugs test suite contains tens of thousands of
tests, which can certainly serve as very good examples as long as they
are correct and up-to-date.
* Intuitive estimation for the quality of the tests
When we browse the web pages with test snippet embedded, it's quite
easy to figure out the validity and also the completeness of the test
suite.
For instance, if a whole section has no snippet links at all, then
that probably means we need to add more tests as well as smartlinks.
These HTML pages are good visual coverage reports per se.
Also, the correctness of a group of tests is easiest to determine if
it appears right in the context of the synopses.
Currently we also have TODO and SKIP marks in the test suite, hence we
can get a good estimation for how much of the Synopses are implemented
by Pugs at a quick glance.
* Easier to detect mismatch between tests and docs
Everyone who has subscribed p6l can see the Synopses are changing
pretty fast. In contrast, many tests in the Pugs test suite are
terribly out of date even though Larry has the habit of updating some
tests after updating the documents.
But if we have enough smartlinks distributed in the test suite, the
mismatch between tests and docs can be detected in the form of
"broken" links. To get a list of broken links, we only need a single
command:
$ util/smartlinks.pl --check t/*/*.t t/*/*/*.t
Unmatched smartlinks (if any) will be reported in great detail:
ERROR: t/operators/quoting.t: line 173: pattern ``/Split result on
words (no quote protection)/'' failed to match any paragraph
in L<S02/Literals>.
ERROR: t/data_types/arglist.t: line 6: section ``Operator renaming''
not found in S03.
As I told Juerd on #perl6, the initial motivation of developing
smartlinks.pl is simply to make myself more motivated to contribute
tests to Pugs.
Thanks to other Pugs members who worked on the smartlinks long before
I joined the Pugs team. Without their work, smartlinks.pl won't
produce anything meaningful today. :)
This mail is also posted to Audrey's blog in the form of HTML:
http://pugs.blogs.com/pugs/2006/08/integrating_the.html
POD++
Cheers,
Agent
Agent Zhang
> On Aug 22, 2006, at 9:55 AM, Agent Zhang wrote:
>
> > Well, in short, we have divided the .t files in the Pugs test suite
> > into pieces and inserted every resulting snippet after the
> > corresponding paragraph of the Synopses.
>
>
Glad you like it. :)
> Often people read the synopses and wonder, "Is feature X from the spec
> working in Pugs today?" I think it would be cool to integrate the smoke
> reports with your inlined tests in the synopses. When a test section is
> displayed, lines from the test that pass could be colored green,
> and those that fail or are todo could be red. The "show snippet"
> link might itself be colored, too.
>
Yes, the good old util/catalog_tests.pl does color the tests with
green and red, and Gaal has also asked me if I'm planning to implement
that feature in util/smartlinks.pl. I'd say "of course", b0ut I intend
to make this feature optional, since just as you have said, smoke
results are implementation specific while tests and docs are not.
> Of course, this idea doesn't quite make sense, because test status
> depends on which runtime is used (and sometimes which O/S is used).
> Maybe we could produce one version of the synopsis for
> each runtime/OS combination listed in smoke.pugscode.org, and update it
> as new smoke reports come in. (Even this approach needs more thought:
> do we mix results from different smoke reports if a new report comes in
> that skips a lot of tests?)
>
Yeah, that will be even cooler. Thank you for this wonderful
suggestion. However, I'm afraid it is not of very high priority for me
given the current status of the smartlinks. Among other things, I'll
focus on polishing the Pugs test suite as well as the smartlinks used
in it so as to get every group fo tests and every Synopsis paragraph
linked. Of course, if you have the tuits to try that, I'll be
delighted.
> Do you think this is worth trying, or would you rather keep the
> smokes separate?
>
This is definitely worth trying. :=)
Regards,
Agent