rfc: "safer" smoking

[David Westbrook]

Here's a snippet that gets all the installed distributions (based on
what the CPAN 'autobundle' command does), and then runs
CPAN::Reporter::Smoker against that list of distributions.

use CPAN;
use CPAN::Reporter::Smoker;
my %dists;
foreach my $mod ( CPAN::Shell->expand('Module','/./') ){
my $d = $mod->distribution or next;
$d = $d->pretty_id;
next if exists $dists{$d};
next if ! $mod->inst_file;
$dists{$d} = undef;
};
my @dists = sort grep { ! m#/perl-5\.# } keys %dists;
my $n = scalar @dists;
warn "Got $n distros -- starting smoker";
CPAN::Reporter::Smoker::start( list => \@dists );

Whereas you wouldn't want to run a smoker on a "regular" machine (e.g.
a $work box, or your day-to-day windows desktop, or laptop, etc),
this is reasonably safe since it's only testing distros that you
already installed, which implies that they (and their dependencies)
are trusted.
Also lets you easily submit reports for all the modules you're actual
using, w/o a separate smoke environment.

so, looking for comments/feedback ...

is this a good/bad (any additional risks??) idea?

Any better way to get the distro list? (note this tests the newest
version, even if it's an older one actually installed)

Is this useful for others?
If so, as a snippet (e.g. posted on wiki)?
as a subclass (name??) of CPAN::Reporter::Smoker?
as an additional exported sub (name??) in CPAN::Reporter::Smoker?

thanks,
--david

[David Golden]

On Sun, Jan 18, 2009 at 7:59 PM, David Westbrook <dwest...@gmail.com>wrote:

> Whereas you wouldn't want to run a smoker on a "regular" machine (e.g.
> a $work box, or your day-to-day windows desktop, or laptop, etc),
> this is reasonably safe since it's only testing distros that you
> already installed, which implies that they (and their dependencies)
> are trusted.
> Also lets you easily submit reports for all the modules you're actual
> using, w/o a separate smoke environment.
>

I like it. Among other things, it could be used with the File transport to
save reports to a directory for examination, which could be useful to check
things before upgrading.

> Is this useful for others?
> If so, as a snippet (e.g. posted on wiki)?
> as a subclass (name??) of CPAN::Reporter::Smoker?
> as an additional exported sub (name??) in CPAN::Reporter::Smoker?
>

At the very least, I'd put it on the CPAN Testers wiki. I have no problem
if you want to publish a CPAN::Reporter::Smoker::* module, but I don't think
I want to include it in C::R::Smoker itself, mostly because it's a slipperly
slope of extra maintenance work for me if I start accepting different ways
of generating the list.

-- David

[David Westbrook]

thanks for the comments!

Any suggestions for a good CPAN::Reporter::Smoker::* subclass name?

--david

[David Golden]

CPAN::Reporter::Smoker::Installed?

But I'm terrible at names. Other people always suggest better ones after I
release something to CPAN. ;-)

-- David

[Imacat]

David Westbrook said, on 2009-01-19 10:09:

1. Please do not top-quote.
2. Sorry I posted your mail on the list. I do not see any reason why
this mail is off-list.

> What part of the "reasonably safe" logic do you see as problematic?

You focused on the "safe" issue, and only see the "problematic".
The safer way is to stop the CPAN testers system. The already-installed
system may not meet the new requirement upon new releases. The
even-safer way is not to release anything. No software, no bugs, problems.

What is the right way? The right way is to solve the problem, not
to close your eyes to the problems.

> In regards to the small projects, I think this method would actually
> help them -- people could use this to easily submit a bunch of test
> reports without the (relatively steep) requirement that they have a
> dedicated/separate smoke environment.

I do not see your point. Is this because my English is too poor?
An easier step for my module to have "bunch of test reports" from less
than 1000 real users where none of them have CPAN::Reporter at all?

> I wasn't suggestion that this replace a full smoke run (where "the
> whole ... system will break down") -- but rather to just complement it
> for systems that otherwise wouldn't be producing any tests.

I do not see this also. If you remove all the reports from testers
that are not really installing the module, the CPAN testers report
database will be less than 1% left. Your point does sound like
non-sense from governments officials. Could you please speak in a
simple English tone?

--
Best regards,
imacat ^_*' <ima...@mail.imacat.idv.tw>
PGP Key: http://www.imacat.idv.tw/me/pgpkey.asc

<<Woman's Voice>> News: http://www.wov.idv.tw/
Tavern IMACAT's: http://www.imacat.idv.tw/
TLUG List Manager: http://lists.linux.org.tw/cgi-bin/mailman/listinfo/tlug

[David Westbrook]

On Sun, Jan 18, 2009 at 9:45 PM, imacat <ima...@mail.imacat.idv.tw> wrote:
>
> 2. Sorry I posted your mail on the list. I do not see any reason why
> this mail is off-list.

Your first reply was directly to me, and not to the list (so the list
is probably missing some context here), which is why i continued it
off-list, assuming it was intentional.

> The safer way is to stop the CPAN testers system.

Can you qualify that? Isn't the CPAN testers system the whole reason
we're here (this list, the cpantesters site, test reports, etc)??

> The already-installed system may not meet the new requirement upon new releases.

But that's fine .. and David Golden made a good point that this

"could be useful to check things before upgrading."

> The even-safer way is not to release anything. No software, no bugs, problems.

ummm .. and no fun, and 0 chance of doing anything good. Isn't
writing/releasing the whole point in any of us being programmers??
Yes, of course there's risk in any change/release, but the goal is
that the benefits outweigh the risks.
That aside, i'm not sure what "not to release anything" is in
reference to here ...

> What is the right way? The right way is to solve the problem, not
> to close your eyes to the problems.

I'm not sure i follow what "problem" is referring to in this context?

> I do not see your point. Is this because my English is too poor?

English looks just fine to me (based on this and a couple previous
posts on this list); but i'm a lowly monolingual and far from fluent
in a 2nd language, so definitely not going to pass judgement.

BUT, clearly we're on different wavelengths ... let's start back at
the beginning:

Proposition:
+ Provide an easy way to execute CPAN::Reporter::Smoker, limited
just to distributions that have any version installed on the system
(instead of all of CPAN).

Pros:
+ Smoke testing benefits, in terms of generating reports (albeit
not for all of CPAN).
+ Tests are submitted from actual use systems, as opposed to just
an isolated builddir & LIB path.
+ In theory is safer than full smoke, since only testing "trusted"
distros (and their deps) -- the user trusted the distro enough to
install it in the first place.
+ Potential use is to check things out before upgrading.
+ doesn't require (making the trust assumptions) a
dedicated/isolated environment

Cons:
+ In theory, still same potential risk as full smoke run
+ The "trusted" distros are based on a chain of assumptions.

And then we can tackle each line item individually where we disagree.

> I do not see this also. If you remove all the reports from testers
> that are not really installing the module, the CPAN testers report
> database will be less than 1% left.

I in no way was suggesting that ... just the opposite, i think this
could be helpful to _increase_ the test count. e.g. i wouldn't have
submitted a bunch of reports from my windows boxes w/o something like
this.

--david

[Gabor Szabo]

On Mon, Jan 19, 2009 at 6:07 PM, David Westbrook <dwest...@gmail.com> wrote:

>
> Proposition:
> + Provide an easy way to execute CPAN::Reporter::Smoker, limited
> just to distributions that have any version installed on the system
> (instead of all of CPAN).

In general I think this is a good idea and I probably would run it on my
development machine.

>
> Pros:

> + In theory is safer than full smoke, since only testing "trusted"
> distros (and their deps) -- the user trusted the distro enough to
> install it in the first place.

For additional safety you might want to limit the installations to

1) Modules that already have N reports in the database
meaning that some other testers have already tested it.

2) Modules that have been on CPAN for at least M days
hoping that if the module and its tests do something bad
someone has already caught this, reported to the
"rulers of CPAN" and the module was taken off CPAN.

regards
Gabor

[Alceu R. de Freitas Jr.]

--- Em seg, 19/1/09, Gabor Szabo <sza...@gmail.com> escreveu:

> For additional safety you might want to limit the
> installations to
>
> 1) Modules that already have N reports in the database
> meaning that some other testers have already tested it.
>
> 2) Modules that have been on CPAN for at least M days
> hoping that if the module and its tests do something
> bad
> someone has already caught this, reported to the
> "rulers of CPAN" and the module was taken off
> CPAN.
>
>
> regards
> Gabor

What about those distributions that have only that dummy test file with a single "BEGIN { use Foo::Bar; }" that h2xs generates?

This actually does not means to much for me, but the smoke test (and mannually too) will consider this distribution OK.

Shouldn't the smoke test include something like Devel::Cover?

Regards,
Alceu

Veja quais são os assuntos do momento no Yahoo! +Buscados
http://br.maisbuscados.yahoo.com

[David Westbrook]

On Mon, Jan 19, 2009 at 1:31 PM, Gabor Szabo <sza...@gmail.com> wrote:
>>
>> Pros:
>> + In theory is safer than full smoke, since only testing "trusted"
>> distros (and their deps) -- the user trusted the distro enough to
>> install it in the first place.
>
> For additional safety you might want to limit the installations to
>
> 1) Modules that already have N reports in the database
> meaning that some other testers have already tested it.
>
> 2) Modules that have been on CPAN for at least M days
> hoping that if the module and its tests do something bad
> someone has already caught this, reported to the
> "rulers of CPAN" and the module was taken off CPAN.

ah -- i like those! Suggestions on defaults for N & M? I guess i'll
look at my installed systems as examples and see what gets excluded
for different values ...

Any thoughts on CPAN::Reporter::Smoker::Safer for the namespace? i
also like ::Installed, but taking into account the above, which shifts
the focus a little, I'm leaning toward ::Safer.
(does ::Safer imply a negative connotation towards CPAN::Reporter::Smoker?)

thanks,
--david

[David Golden]

On Mon, Jan 19, 2009 at 3:04 PM, David Westbrook <dwest...@gmail.com>wrote:

> (does ::Safer imply a negative connotation towards CPAN::Reporter::Smoker?)
>

Not one that bothers me. After all, I put this in the C::R::S Pod:
WARNING -- smoke testing is
risk<http://search.cpan.org/%7Edagolden/CPAN-Reporter-Smoker-0.17/lib/CPAN/Reporter/Smoker.pod#___top>

Smoke testing will download and run programs that other people have uploaded
to CPAN. These programs could do *anything* to your system, including
deleting everything on it. Do not run CPAN::Reporter::Smoker unless you are
prepared to take these risks.

[Imacat]

On 01/20/2009 12:07 AM, David Westbrook wrote:
> On Sun, Jan 18, 2009 at 9:45 PM, imacat <ima...@mail.imacat.idv.tw> wrote:
>> 2. Sorry I posted your mail on the list. I do not see any reason why
>> this mail is off-list.
>
> Your first reply was directly to me, and not to the list (so the list
> is probably missing some context here), which is why i continued it
> off-list, assuming it was intentional.

Oh... sorry. I do not know the cpan-testers-discuss list does not
set the Reply-to: header, and I did not check it. Maybe I shall write
to the list manager some other time.

> Proposition:
> + Provide an easy way to execute CPAN::Reporter::Smoker, limited
> just to distributions that have any version installed on the system
> (instead of all of CPAN).

I still do not see at all.

1. How many smokers have installed my module? Only one, that is,
I. How many smokers have installed Mac::iTune? No one.

2. With this new system you suggested, I will only get my own
reports, which is nearly useless. And other authors will get no reports
at all.

3. I will stop running smoke tests, since when I upgrade the
modules weekly, the reports will be sent anyway. If I will not be
sending reports other than the above, running smoke tests is a
duplicated work and CPU waste which could be saved. In fact, currently
the only modules I do not need to run smoke tests are those that I did
not installed.

4. Nobody gets reports except for those popular modules like LWP,
etc.. Also, nobody is running smoke tests anymore.

5. If nobody is running smoke tests, and nobody is receiving smoke
test reports except LWP, how does this CPAN testers system work?

So, why are you thinking that this will increase the amount of
reports, but not demolish the whole CPAN testers system?

If that is not what you mean, then I have a serious English problem.
Either I misunderstand your proposition, or I misunderstand your
expected result. In either way, I may have to restart my English class.

[David Westbrook]

On Tue, Jan 20, 2009 at 9:13 PM, imacat <ima...@mail.imacat.idv.tw> wrote:

>> Proposition:
>> + Provide an easy way to execute CPAN::Reporter::Smoker, limited
>> just to distributions that have any version installed on the system
>> (instead of all of CPAN).
>
> I still do not see at all.
>
> 1. How many smokers have installed my module? Only one, that is,
> I. How many smokers have installed Mac::iTune? No one.

For your module, and Mac::iTune, there won't be an added benefit. But
there will be for more commonly installed modules. Which, of course,
are already tested, but this could reach additional platforms/perls --
that potentially could be a bigger benefit than additional modules.

> 2. With this new system you suggested, I will only get my own
> reports, which is nearly useless. And other authors will get no reports
> at all.

True, it's not reaching all of CPAN, but OTOH it's not zero modules
either -- it's whatever's on your system .. which i'm guessing is
200-500 distros.

> 3. I will stop running smoke tests, since when I upgrade the
> modules weekly, the reports will be sent anyway. If I will not be
> sending reports other than the above, running smoke tests is a
> duplicated work and CPU waste which could be saved.

Here's where are lines of thought split i think ..
You wouldn't need to (and shouldn't) stop running smoke tests if your
running this new system (C::R::S::Safer) .. There's two cases here:
a) The system (read: platform/perl) you're smoking on is the same as
where you're running C::R::S::Safer.
+ In this case, i'd say either don't run C::R::S::Safer, or make
sure the reports-send.db file is shared between the C::R::S::Safer
runs and the C::R::S smoke runs. That way there's no wasted
resources.
b) The platform/perl is different. This situation is what i'm
targeting .. where you're smoking on system A, but also have system
B, which full smoking is unsafe and/or inappropriate on. Running
C::R::S::Safer on system B at least gets some partial smoke
coverage/reports, whereas without it there wouldn't be any.

Basically, it's not proposed to replace smoke testing -- just to be
used where smoke testing can't (or shouldn't) be done.

Your points 4. & 5. and "demolish the whole CPAN testers system"
followed from the assumption that running this would mean ceasing
smoke testing, so i'll skip those.

Take my actual setups for an example ... I have:
* a $work laptop (winXP, 5.10)
* a home desktop (win2K, 5.8.8)
* a spare laptop (no hard drive)
* hosting account (linux, i686, 5.8.8)
* $work dev env (linux, 5.6.1)

On the spare laptop, i'm booting a knoppix livecd and smoking from
there (linux, i486, 5.8.8).
BUT, the other 4 systems w/unique platforms/perls I can't run smoke
testing on (for both safety and resources reasons).
BUT, i am willing to run C::R::S::Safer, to ensure that i've
contributed reports for everything that i have installed.

So this is where I see it only creating additional testers reports.

Basically, i envision the following levels of test report contribution:

a) Install CPAN::Reporter, and send reports for any module that gets
installed from that point on.
b) Install CPAN::Reporter::Smoker::Safer, and run it to submit
reports for all the distros you "trust" (read: have installed). And
run it again in the future to test any newer versions of those
distros, w/o actually installing them.
c) Install & run CPAN::Reporter::Smoker (or other smoker), and
submit reports for everything. --> BUT, this takes an isolated &
dedicated environment.

And it's key to note that these 3 are not mutually exclusive .. can do
(c) on one system, and (a) and/or (b) on others (as in my setup
described above).

> In either way, I may have to restart my English class.

don't waste the time or money :) -- w/o your mention of it and the .tw
email, I would have assumed the above was written by a native english
speaker.

This is a good thread -- parts of it will definitely make their way
into the C::R::S::Safer documentation.

--david

[Gabor Szabo]

On Wed, Jan 21, 2009 at 4:13 AM, imacat <ima...@mail.imacat.idv.tw> wrote:
> On 01/20/2009 12:07 AM, David Westbrook wrote:

>> Proposition:
>> + Provide an easy way to execute CPAN::Reporter::Smoker, limited
>> just to distributions that have any version installed on the system
>> (instead of all of CPAN).
>
> I still do not see at all.
>
> 1. How many smokers have installed my module? Only one, that is,
> I. How many smokers have installed Mac::iTune? No one.
>
> 2. With this new system you suggested, I will only get my own
> reports, which is nearly useless. And other authors will get no reports
> at all.
>
> 3. I will stop running smoke tests, since when I upgrade the
> modules weekly, the reports will be sent anyway. If I will not be
> sending reports other than the above, running smoke tests is a
> duplicated work and CPU waste which could be saved. In fact, currently
> the only modules I do not need to run smoke tests are those that I did
> not installed.
>
> 4. Nobody gets reports except for those popular modules like LWP,
> etc.. Also, nobody is running smoke tests anymore.
>
> 5. If nobody is running smoke tests, and nobody is receiving smoke
> test reports except LWP, how does this CPAN testers system work?
>
> So, why are you thinking that this will increase the amount of
> reports, but not demolish the whole CPAN testers system?
>

This Safer CPAN Tester module could be installed by people like
me who do don't do smoke testing at all (or by you but on your
real development environment) thus it has the potential to increase
the number of reports in more natural environments.

One of my main issues with the tons of reports the main smokers send
- despite the fact that I am totally thankful for them -
that those reports are generated in a lab environment and their sheer number
hides the few reports that come from real setups.

Point: while I am happy to see my modules tested on all version of
5.8.x on NetBSD
it would be more important to get the reports from people using the
*default perl*
of Red Hat 17.0 and Enterprise SuSE 38.7 or whatever version numbers they have.

Those environments are usually more difficult and much more broken (!)
than those of the smokers.

This Safer smoker has the potential to help increasing this number.

Gabor

[David Cantrell]

On Wed, Jan 21, 2009 at 10:13:52AM +0800, imacat wrote:

> Oh... sorry. I do not know the cpan-testers-discuss list does not
> set the Reply-to: header, and I did not check it. Maybe I shall write
> to the list manager some other time.

Here, have a procmail recipe to fix that :-)

:0 fHw
* ^List-Id:.*cpan-testers-discuss.perl.org
| formail -i "Reply-To: cpan-teste...@perl.org"

--
David Cantrell | Nth greatest programmer in the world

fdisk format reinstall, doo-dah, doo-dah;
fdisk format reinstall, it's the Windows way