More testing of common platforms
David Golden
is worth it to have more testers submitting reports even for common
platforms. We're nowhere near having statistically valid samples on
most distributions on common platforms. If we get to the point where
we have 10 or 15 reports from platform X with identical config
settings, then I think we can start encouraging those testers to
diversify more.
David
Andreas J. Koenig
I've also come to Tim's conclusion that we already now need more
diversification than we have. But it's less a matter of encouragement
but a question how to support it.
One problem I see at the moment is the format of the reports-sent.db file:
test FAIL Data-FormValidator-4.63 (perl-5.11.2) x86_64-linux 2.6.30-1-amd64
Right now a reporter will send a maximum of one pass and one fail for
this distro/perl combination.
Imagine a bughunter is trying out variations in the C compiler or in the
toolchain or in installed versions of modules required or is bisecting
perl. Then it would be wonderful to see the resulting variations in
cpantesters. Currently the bughunter has to wipe out the one line in the
reports-sent database to deliver more than one PASS or FAIL.
What would you think about a report_again command?
--
andreas
Serguei Trouchelle
[...]
> One problem I see at the moment is the format of the reports-sent.db file:
>
> test FAIL Data-FormValidator-4.63 (perl-5.11.2) x86_64-linux 2.6.30-1-amd64
>
> Right now a reporter will send a maximum of one pass and one fail for
> this distro/perl combination.
There's already "send_duplicates" configuration variable for CPAN::Reporter, it can be set to "yes", then duplicate
reports would be sent regardless of reports-sent.db.
Though, it may produce a great amount of FAIL reports of common prerequisites when they get tested for every package
that requires them.
I'd add another user (if it's possible) and run smokers for different configurations from different accounts.
--
Serguei Trouchelle
David Golden
Tim's conclusion that we already now need more
> diversification than we have. But it's less a matter of encouragement
> but a question how to support it.
>
> One problem I see at the moment is the format of the reports-sent.db file:
At the risk of sounding like an auto-responder, I'll put that on the
list of design goals for the CT2.0 client.
-- David
Sergey Aleynikov
I'm just entering automated perl testing, so have a question related
to this topic. I want to test as much possible combinations as
possible on a single machine,
but there's no glue what really matters and what - not. So, currently
i've chosen pairs mymalloc/nothreads and nomymalloc/threaded for perls
5.8.8, 5.8.9, 5.10.1, 5.11.2, on
i386 and amd64 Freebsd systems. But, is this a correct choice?
Shouldn't i, for example, add 64-bit-int options to any of theese
configs, or make new triplets (so, varying 3 params)?
Probably, this should go to a new section of testing faq?
Best regards,
Sergey Aleynikov
Andreas J. Koenig
My take on this is that we are in a position where all computers in the
world will not be able to run all possible combinations of our input
parameters. [...insert endless blurb as to why...]
Like always when there is a large array to explore randomness helps to
spot problem areas. [...insert endless blurb about evolution...]
Whenever a problem area is discovered it pays to put the spot on this
area until it is cleaned up and then move elsewhere. So as soon as you
find a hobby horse that discovered a single bug, concentrate your search
on it to find similar cases.
Whenever you discover that you and another tester are doing the same
thing, go somewhere else. It's really boring to see two identical
results. It's like searching mushrooms on the same grid square that
somebody else has just visited before you.
Bottom line is we cannot tell you what to explore. You must experiment
and watch your results and watch the results of the others in the whole
endavour. Reading recent RT tickets is always interesting. Writing new
RT tickets is helpful for the community.
If there is one thing you should avoid then it is to test outdated
distros that are superceded by a higher version. While it is good to
have old versions installed so you can study the impact on other modules
it is nothing but noise if you're testing Archive-Tar-0.23 when the
current version is 1.54[0]
--
andreas
[0] http://www.nntp.perl.org/group/perl.cpan.testers/2009/11/msg6143241.html
Oliver Paukstadt
> I've also come to Tim's conclusion that we already now need more
> diversification than we have. But it's less a matter of encouragement
> but a question how to support it.
>
> One problem I see at the moment is the format of the reports-sent.db file:
>
> test FAIL Data-FormValidator-4.63 (perl-5.11.2) x86_64-linux 2.6.30-1-amd64
>
> Right now a reporter will send a maximum of one pass and one fail for
> this distro/perl combination.
>
> Imagine a bughunter is trying out variations in the C compiler or in the
> toolchain or in installed versions of modules required or is bisecting
> perl. Then it would be wonderful to see the resulting variations in
> cpantesters. Currently the bughunter has to wipe out the one line in the
> reports-sent database to deliver more than one PASS or FAIL.
>
sh Configure -Darchname='bisect-b3dc64032ba370e494dd752a46f2a53d'
for a different perl?
This should generate a different archname which is registered in
reports-sent.db. This would help at least for all new compiled perls,
not for the mentioned module version variations.
Quick hack solution for this might be: Change printed archname in
reports-sent.db to
$Config{archname}."-".$ENV{'CPAN_Reporter_Tag'} if defined.
Oliver
--
Oliver Paukstadt <pst...@sourcentral.org>
Andreas J. Koenig
>> Imagine a bughunter is trying out variations in the C compiler or in the
>> toolchain or in installed versions of modules required or is bisecting
>> perl. Then it would be wonderful to see the resulting variations in
>> cpantesters. Currently the bughunter has to wipe out the one line in the
>> reports-sent database to deliver more than one PASS or FAIL.
>>
> What about
> sh Configure -Darchname='bisect-b3dc64032ba370e494dd752a46f2a53d'
> for a different perl?
Acme::BadExample:) How many parts of the cpantesters ecology would break
this? Imagine a cpan testers matrix with these crap labels:)
> This should generate a different archname which is registered in
> reports-sent.db. This would help at least for all new compiled
> perls, not for the mentioned module version variations.
> Quick hack solution for this might be: Change printed archname in
> reports-sent.db to
> $Config{archname}."-".$ENV{'CPAN_Reporter_Tag'} if defined.
OK, something like this might work, it has no impact on the rest of the
world.
--
andreas
Tim Bunce
I'm not going to address the question directly as Andreas gave a good answer.
This post, and Andreas's answer, did remind me to post my
build-perl-variants script that I've used to generate automate the
creation of multiple perl variants for testing (NYTProf etc).
Currently it build variants of these config values: usemymalloc
DEBUGGING usemultiplicity uselongdouble useshrplib usethreads
use64bitall. With just that set it creates 128 perl builds!
Add one more, like -Dmad, and you'll get 256 perl builds.
Andreas's point should be clear!
On the other hand, I have found it useful.
I've meant to add command line options to enable building with a subset
of config values (i.e. "exclude uselongdouble & use64bitall" and "only
use usethreads & useshrplib & debugging") but haven't got round to it
yet -- I just hack the source instead :)
If anyone hacks on this please send the patches to me.
Perhaps it'll get added to the perl distribution one day.
Tim.
Tim Bunce
>
> Whenever you discover that you and another tester are doing the same
> thing, go somewhere else.
Sadly the definition of "same thing" (ie just archname) is currently too
broad. Hopefully that planned changes to keep more information about
build options will help here.
> Bottom line is we cannot tell you what to explore.
Yet.
I can imagine a future where cpan testers can at least see which
combinations of perl versions and config options are getting little or
no testing on a given platform. Then they'd be able to pick one or more
to add to their testing.
Tim.
David Golden
<andreas.koe...@franz.ak.mind.de> wrote:
> Whenever you discover that you and another tester are doing the same
> thing, go somewhere else. It's really boring to see two identical
> results. It's like searching mushrooms on the same grid square that
> somebody else has just visited before you.
For what it's worth, this is where I sort of disagree. If someone
says "I'm testing X, Y, Z, is that useful?" my answer is "Yes!" We
have nothing like a statistically valid sample even for common
distributions and there are any number of odd ways for Perl 5.X.Y to
be installed that are worth exploring. By not having a cookbook for
how to set up a multi-perl smoker, everyone invents a slightly
different way to do it and here is where we find useful diversity.
On the other hand, if someone say "I want to test a lot of perls, what
are some useful areas to explore", then I'm perfectly happy for us to
point out new grid squares.
And it's not just perl configurations, either. Here are some fun/evil ideas:
* NOEXEC temp partition (or NOEXEC cpan build partition) at least
* Readonly filesystem *except* for temp
* statically compiled perl
* *nix distro of your choice installed on a FAT32 filesystem
* Offline (no internet)
> If there is one thing you should avoid then it is to test outdated
> distros that are superceded by a higher version. While it is good to
> have old versions installed so you can study the impact on other modules
> it is nothing but noise if you're testing Archive-Tar-0.23 when the
> current version is 1.54[0]
I *mostly* agree, with the caveat that I think anything listed in
02packages is fair game for testing. Anything there *can* be
installed on a users system given the right prereq listing and modules
removed from distros but not de-indexed are an authors problem.
Smoking those is one (more) way to harass authors to clean things up.
-- David
Serguei Trouchelle
> it is nothing but noise if you're testing Archive-Tar-0.23 when the
> current version is 1.54[0]
Archive-Tar-0.23 is indexed (see 02packages.details.txt.gz), because it contains Archive::Tar::Std and
Archive::Tar::Win32, missing in Archive-Tar-1.54.
Then it gets attention from CPAN-Reporter-Smoker and got smoked eventually.
So, it may be a noise, but it's a noise in CPAN, not in smoke testing.
--
Serguei Trouchelle
Burak Gürsoy
> From: Serguei Trouchelle [mailto:st...@railways.dp.ua]
> Sent: Friday, November 27, 2009 11:54 AM
> To: cpan-teste...@perl.org
> Subject: Re: More testing of common platforms
>
> Andreas J. Koenig wrote:
>
> > it is nothing but noise if you're testing Archive-Tar-0.23 when the
> > current version is 1.54[0]
>
Perhaps BackPAN & deleting old versions must be couraged more? Maybe a search.cpan.org clone for BackPAN as well, as some authors seem to like the CPAN Search tools (like colored diff) very much and you need to keep old versions in CPAN for these kind of functionality :)
> --
> Serguei Trouchelle
Chris 'BinGOs' Williams
Currently, I'm testing on each of the plethora of OSs I have at my disposal
a number of perls with the following build options:
-Dusethreads -Duse64bitint
-Dusethreads
(no build options)
Not a problem to add more combinations to the mix (besides available diskspace) as
I keep separate CPANPLUS::YACSmoke histories for each perl
Cheers,
--
Chris Williams
aka BinGOs
PGP ID 0x4658671F
http://www.gumbynet.org.uk
==========================
Slaven Rezic
No need for another user. Just define either the environment variable
PERL_CPAN_REPORTER_DIR or PERL_CPAN_REPORTER_CONFIG (documented in
CPAN::Reporter::Config) to use a different reports-sent.db file.
Regards,
Slaven
--
Slaven Rezic - slaven <at> rezic <dot> de
tkruler - Perl/Tk program for measuring screen distances
http://ptktools.sourceforge.net/#tkruler
Slaven Rezic
> I *mostly* agree, with the caveat that I think anything listed in
> 02packages is fair game for testing.
Except for old packages which would actually overwrite a newer
distribution version. And there are a lot of them in
02packages.details.txt.
Regards,
Slaven
--
Slaven Rezic - slaven <at> rezic <dot> de
BBBike - route planner for cyclists in Berlin
WWW version: http://www.bbbike.de
Perl/Tk version for Unix and Windows: http://bbbike.sourceforge.net
Slaven Rezic
Related to this issue, I would like to encourage testers on Unix
platforms to test with an X display available. Currently meaningful test
results for e.g. Tk-related modules are practically missing.
--
Slaven Rezic - slaven <at> rezic <dot> de
tktimex - time recording tool
http://sourceforge.net/projects/ptktools/
David Cantrell
> Related to this issue, I would like to encourage testers on Unix
> platforms to test with an X display available. Currently meaningful test
> results for e.g. Tk-related modules are practically missing.
If you can point me at the software, I'll install a fake X server on at
least some of my machines. However, I really don't want random windows
popping up all over the place on the only one of my testing machines to
actually have a display, because it's also the same machine I use for
watching films on.
--
David Cantrell | Godless Liberal Elitist
Godliness is next to Englishness
pe...@0ne.us
> Slaven Rezic wrote:
>
>
>> Related to this issue, I would like to encourage testers on Unix
>> platforms to test with an X display available. Currently meaningful test
>> results for e.g. Tk-related modules are practically missing.
>>
>
> If you can point me at the software, I'll install a fake X server on at
> least some of my machines. However, I really don't want random windows
> popping up all over the place on the only one of my testing machines to
> actually have a display, because it's also the same machine I use for
> watching films on.
>
>
using it!
David Cantrell
> David Cantrell wrote:
> >If you can point me at the software, I'll install a fake X server on at
> >least some of my machines. However, I really don't want random windows
> >popping up all over the place on the only one of my testing machines to
> >actually have a display, because it's also the same machine I use for
> >watching films on.
> Ah, I've used Xvfb in the past to do this but not recently... Good luck
> using it!
Tried it, but it just spits a load of errors about fonts, and I have no
interest whatsoever in actually putting time into configuring an X
server - that's why I stopped using Linux on the desktop years ago!
--
David Cantrell | http://www.cantrell.org.uk/david
Arbeit macht Alkoholiker
Slaven Rezic
> On Sun, Nov 29, 2009 at 02:53:07PM -0700, pe...@0ne.us wrote:
>> David Cantrell wrote:
>> >If you can point me at the software, I'll install a fake X server on at
>> >least some of my machines. However, I really don't want random windows
>> >popping up all over the place on the only one of my testing machines to
>> >actually have a display, because it's also the same machine I use for
>> >watching films on.
>> Ah, I've used Xvfb in the past to do this but not recently... Good luck
>> using it!
>
> Tried it, but it just spits a load of errors about fonts, and I have no
> interest whatsoever in actually putting time into configuring an X
> server - that's why I stopped using Linux on the desktop years ago!
I also see some "FreeFontPath" errors when using Xvfb, but these seem to
be harmless. At least the Perl/Tk test suite runs without problems in a
Xvfb server.
But Xvfb is not the only solution:
- You can use an Xnest server, a full-featured X11 server running in an
X11 window. This means you can iconify this server, and the test
windows are now popping up within the (possibly iconified) window. To
create one use
Xnest :1 &
env DISPLAY=:1 xterm &
If you like a window manager you can add one with
env DISPLAY=:1 fvwm2 &
Or whatever window manager you prefer. Some systems, notably Debian
systems seem to require calling Xnest with the "-ac" option. Or do
experiment using the startx command instead:
startx -- /usr/local/bin/Xnest :1 &
- You can run another X11 server in another virtual console. At least
this should work on Linux and FreeBSD systems:
Change to a free virtual console by hitting Ctrl-Alt-F1 (or F2 ...).
Login.
Start the X11 server with
startx -- :1
The first X11 server is accessible using Ctrl-Alt-F9 on FreeBSD and
Ctrl-Alt-F7 on Linux systems. The second one may be switched to with
Ctrl-Alt-F10 or Ctrl-Alt-F8 resp.
Regards,
Slaven
--
Slaven Rezic - slaven <at> rezic <dot> de
tknotes - A knotes clone, written in Perl/Tk.
http://ptktools.sourceforge.net/#tknotes