[P6C] problems with revision 1.5 of lib/Parrot/Test.pm
Allison Randal
The problem is in a few recent changes to lib/Parrot/Test.pm. The Perl
6 test module in languages/perl6/P6C/TestCompiler.pm inherits from
Parrot::Test, but overrides some subs. P6C::TestCompiler's
&generate_pbc_for is writing the .pasm files in subdirectories of
languages/perl6/t, but Parrot::Test's &generate_functions is trying to
read them from a flattened directory structure in /tmp/*.
(P6C::TestCompiler is passing arguments to Parrot::Test's
&generate_functions that it no longer accepts, and Parrot::Test is
passing arguments to P6C::TestCompiler's &generate_pbc_for that it
ignores.)
So, the tests are failing either because the expected file in /tmp
doesn't exist, or more often because there's already a .pasm file in
/tmp with the expected name that was left behind by the main Parrot test
run (it seems that names like "basic.t" are common).
Will or Jerome, could you take a minute to explain the direction you're
taking Parrot::Test? I'll patch P6C::TestCompiler to match the new
setup, but I want to do an "architecture sanity check" first.
Thanks,
Allison
Leopold Toetsch
> Will or Jerome, could you take a minute to explain the direction you're
> taking Parrot::Test? I'll patch P6C::TestCompiler to match the new
> setup, but I want to do an "architecture sanity check" first.
I'd rather have the test files back in our tree. The comment in Test.pm
("put the test in a tempdir, so we're not cluttering") isn't really an
argument.
$ make testclean
*got* rid of all tests. This is of course broken now and /tmp is filled
with tests.
> Thanks,
> Allison
leo
Jerome Quelin
> Okay, languages/perl6 is failing all its tests.
> Will or Jerome, could you take a minute to explain the direction
> you're taking Parrot::Test? I'll patch P6C::TestCompiler to match the
> new setup, but I want to do an "architecture sanity check" first.
I'm sorry, I've been very busy for 2 weeks, and did not have the time to
continue things regarding this topic. :-(
I hope to have some more time this week-end, but I'm not even sure...
So, very quickly:
1. The main goal is to have unified languages tests, that is, make
language-tests would get a list of all the test files for all of the
languages, and then run only _one_ harness with all those files. This
would allow for one nice overview of all the tests, and tinderboxen
would benefit from this too: they were reporting success as long as the
last language was reporting succes.
2. In order for this to work, all the languages should have commit to
having a t/harness file in their directory, responsible to get back a
list of all the test files when called with a -files argument. (Except
if we consider that *.t files are test files, but this may be too
strict) If the language implementer wants to use this harness to also
launch test for the current language only, this is of course welcome
3. Will and I thought that it would be nice for tests to just "use
Parrot::Test" and then have a new function:
language_output_is($language, $code, $expected, $comment)
4. Will worked on a patch for Parrot::Test, with some (little) help from
me to debug it. (Perl 6 tests did not even run because it seems Perl 6
test suite has core knowledge of Parrot::Test)
5. With Dan's agreement, I applied it since Will does not have checkin
rights on the whole tree.
6. The next thing to do was to work on the subject in order to make all
languages aware of this scheme.
7. The patch has some, erm, "action at distance" problems, such as the
pasm files put in /tmp
This is for what happened.
Now, here's my view for the future of languages testing:
1. I really think the scheme one language_output_is function and one
global harness is the way to go. This does not forbid having one
harness per language: indeed, each language still _needs_ one harness
script to fetch the list of test files.
2. Will's patch needs to be updated in (at least) two ways: a) put back
pasm files in test directories and b) remove the need for a
Parrot::Test::<Language> module. It should _allow_ it, but should also
provide a convenient default for languages that do not need fancy
testing things. (This should be the case of all the "little" languages
where running a $parrot languages/$lang/$lang.pasm $code_file should be
enough)
3. All the languages needs to be updated to take this scheme into
account once Parrot::Test is modified accordingly.
That's it, I don't know if I made myself clear? (tricky subject to work
on - and thus to explain :o) )
As we saw, it seems that Perl 6's test suite does weird things. Can you
explain what those weird things are, so we can take them into account
in the Plan?
Also, what do you think of the scheme? (and when I say you, I also mean
the p6i list) Knowing that the scheme will allow (currently it
_forces_) you to extend Parrot::Test with Parrot::Test::Perl6 (or
whatever language)
I'll try to work on it saturday, but maybe we should currently revert
the patch on Parrot::Test so I can have some more time re-working the
languages testing stuff, without annoying Leo and you. (at least)
Once again, sorry for lack of time that led to those problems,
Jerome
--
jqu...@mongueurs.net
Leopold Toetsch
> 7. The patch has some, erm, "action at distance" problems, such as the
> pasm files put in /tmp
Is that a side effect or needed?
leo
Dan Sugalski
It's just a side-effect, and we can put the old behavior back, I expect.
--
Dan
--------------------------------------"it's like this"-------------------
Dan Sugalski even samurai
d...@sidhe.org have teddy bears and even
teddy bears get drunk
Allison Randal
>
> I'm sorry, I've been very busy for 2 weeks, and did not have the time to
> continue things regarding this topic. :-(
> I hope to have some more time this week-end, but I'm not even sure...
It happens. :)
> So, very quickly:
> 1. The main goal is to have unified languages tests, that is, make
> language-tests would get a list of all the test files for all of the
> languages, and then run only _one_ harness with all those files. This
> would allow for one nice overview of all the tests, and tinderboxen
> would benefit from this too: they were reporting success as long as the
> last language was reporting succes.
>
> 2. In order for this to work, all the languages should have commit to
> having a t/harness file in their directory, responsible to get back a
> list of all the test files when called with a -files argument. (Except
> if we consider that *.t files are test files, but this may be too
> strict) If the language implementer wants to use this harness to also
> launch test for the current language only, this is of course welcome
I agree with the idea of allowing for a 'make languages-test', but I'm
not sure I agree that means Parrot::Test has to be smart enough to run
every language's tests directly (more below).
[...]
> Now, here's my view for the future of languages testing:
> 1. I really think the scheme one language_output_is function and one
> global harness is the way to go. This does not forbid having one
> harness per language: indeed, each language still _needs_ one harness
> script to fetch the list of test files.
I'm not clear on why we need a separate language_output_is function.
P6C::TestCompiler is using the standard output_is, it only overrides
&generate_pbc_for so it can compile the p6 source down to pasm before it
passes it off to Parrot::Test::generate_pbc_for to compile it down to
pbc.
I do agree that keeping the language-specific harnesses is very
important. (I'll tear my hair out if I have to run the test suite for
all the languages every time I want to test Perl 6 :).
> 2. Will's patch needs to be updated in (at least) two ways: a) put back
> pasm files in test directories
I agree.
> and b) remove the need for a
> Parrot::Test::<Language> module. It should _allow_ it, but should also
> provide a convenient default for languages that do not need fancy
> testing things. (This should be the case of all the "little" languages
> where running a $parrot languages/$lang/$lang.pasm $code_file should be
> enough)
Hmm... I'd lean toward keeping all the language-specific code in the
languages/* directories. Core Parrot shouldn't have to carry extra
weight for every language it ever ships with, or might ever need to
test. Dropping a language from a specialized distribution should be as
simple as dropping the irrelevant languages/* directory.
> 3. All the languages needs to be updated to take this scheme into
> account once Parrot::Test is modified accordingly.
>
> That's it, I don't know if I made myself clear? (tricky subject to work
> on - and thus to explain :o) )
It seems clear, but if my responses seem like I don't understand, please
clarify.
> As we saw, it seems that Perl 6's test suite does weird things. Can you
> explain what those weird things are, so we can take them into account
> in the Plan?
The only major thing it does is override &generate_pbc_for and pass it
to &generate_functions.
> Also, what do you think of the scheme? (and when I say you, I also mean
> the p6i list) Knowing that the scheme will allow (currently it
> _forces_) you to extend Parrot::Test with Parrot::Test::Perl6 (or
> whatever language)
Would it be possible to define a set of conventions instead of adding to
the weight of Parrot::Test? Say, if a language has a t/harness, it
should return X results in Y format. Cardinal may write tests in Ruby,
and Pirate may write tests in Python, but it should still be possible to
drop them into languages/ and test them with the rest of the languages.
> I'll try to work on it saturday, but maybe we should currently revert
> the patch on Parrot::Test so I can have some more time re-working the
> languages testing stuff, without annoying Leo and you. (at least)
It's Leo's call, but sounds reasonable to me. I don't know that we'll
reach a final solution this weekend, and I'd like to work on other P6C
stuff.
Thanks!
Allison
Dan Sugalski
> So, very quickly:
>> 1. The main goal is to have unified languages tests, that is, make
>> language-tests would get a list of all the test files for all of the
>> languages, and then run only _one_ harness with all those files. This
>> would allow for one nice overview of all the tests, and tinderboxen
>> would benefit from this too: they were reporting success as long as the
>> last language was reporting succes.
>>
>> 2. In order for this to work, all the languages should have commit to
>> having a t/harness file in their directory, responsible to get back a
>> list of all the test files when called with a -files argument. (Except
>> if we consider that *.t files are test files, but this may be too
>> strict) If the language implementer wants to use this harness to also
>> launch test for the current language only, this is of course welcome
>
>I agree with the idea of allowing for a 'make languages-test', but I'm
>not sure I agree that means Parrot::Test has to be smart enough to run
>every language's tests directly (more below).
Well... the big problems we were make-related, with some tinder
issues. Basically if any test suite died really hard the testing
would stop, and only the last test suite run counted for tinder
good/bad reporting.
At this point, though, I'm comfortable making it so that any language
in the distribution with tests should run under the base test
harness. If you need to mess around with it there's a good chance
that something's gone horribly wrong, and twiddling with it on a
per-language basis isn't likely the right way to go.
Allison Randal
>
> Well... the big problems we were make-related, with some tinder
> issues. Basically if any test suite died really hard the testing
> would stop, and only the last test suite run counted for tinder
> good/bad reporting.
>
> At this point, though, I'm comfortable making it so that any language
> in the distribution with tests should run under the base test
> harness. If you need to mess around with it there's a good chance
> that something's gone horribly wrong, and twiddling with it on a
> per-language basis isn't likely the right way to go.
I'm mainly thinking about the fact that Perl 6 tests will eventually be
written in Perl 6. But we can cross that bridge when we get to it. :)
All I really care about at the moment is making the Perl 6 test suite
work so I can work on P6C. The attached patch takes care of it (and
stops putting the tests in /tmp).
Allison
Dan Sugalski
Applied. (Was there anything besides the /tmp change? That's all that
was in the attachment)
Allison Randal
>
> Applied. (Was there anything besides the /tmp change? That's all that
> was in the attachment)
Calling 'rel2abs' was necessary so Parrot::Test would have the right
path to the test file later when it cd's to the main parrot directory
and calls './parrot'. That's all it needed once we got rid of the /tmp
change.
I'll need to make more extensive changes to the Perl 6 test suite when
Jerome is done, but this is good enough for now.
Allison