Here's a snippet that gets all the installed distributions (based on what the CPAN 'autobundle' command does), and then runs CPAN::Reporter::Smoker against that list of distributions.
use CPAN; use CPAN::Reporter::Smoker; my %dists; foreach my $mod ( CPAN::Shell->expand('Module','/./') ){ my $d = $mod->distribution or next; $d = $d->pretty_id; next if exists $dists{$d}; next if ! $mod->inst_file; $dists{$d} = undef; }; my @dists = sort grep { ! m#/perl-5\.# } keys %dists; my $n = scalar @dists; warn "Got $n distros -- starting smoker"; CPAN::Reporter::Smoker::start( list => \@dists );
Whereas you wouldn't want to run a smoker on a "regular" machine (e.g. a $work box, or your day-to-day windows desktop, or laptop, etc), this is reasonably safe since it's only testing distros that you already installed, which implies that they (and their dependencies) are trusted. Also lets you easily submit reports for all the modules you're actual using, w/o a separate smoke environment.
so, looking for comments/feedback ...
is this a good/bad (any additional risks??) idea?
Any better way to get the distro list? (note this tests the newest version, even if it's an older one actually installed)
Is this useful for others? If so, as a snippet (e.g. posted on wiki)? as a subclass (name??) of CPAN::Reporter::Smoker? as an additional exported sub (name??) in CPAN::Reporter::Smoker?
On Sun, Jan 18, 2009 at 7:59 PM, David Westbrook <dwestbr...@gmail.com>wrote:
> Whereas you wouldn't want to run a smoker on a "regular" machine (e.g. > a $work box, or your day-to-day windows desktop, or laptop, etc), > this is reasonably safe since it's only testing distros that you > already installed, which implies that they (and their dependencies) > are trusted. > Also lets you easily submit reports for all the modules you're actual > using, w/o a separate smoke environment.
I like it. Among other things, it could be used with the File transport to save reports to a directory for examination, which could be useful to check things before upgrading.
> Is this useful for others? > If so, as a snippet (e.g. posted on wiki)? > as a subclass (name??) of CPAN::Reporter::Smoker? > as an additional exported sub (name??) in CPAN::Reporter::Smoker?
At the very least, I'd put it on the CPAN Testers wiki. I have no problem if you want to publish a CPAN::Reporter::Smoker::* module, but I don't think I want to include it in C::R::Smoker itself, mostly because it's a slipperly slope of extra maintenance work for me if I start accepting different ways of generating the list.
On Sun, Jan 18, 2009 at 8:31 PM, David Golden <xda...@gmail.com> wrote: > On Sun, Jan 18, 2009 at 7:59 PM, David Westbrook <dwestbr...@gmail.com> > wrote:
>> Whereas you wouldn't want to run a smoker on a "regular" machine (e.g. >> a $work box, or your day-to-day windows desktop, or laptop, etc), >> this is reasonably safe since it's only testing distros that you >> already installed, which implies that they (and their dependencies) >> are trusted. >> Also lets you easily submit reports for all the modules you're actual >> using, w/o a separate smoke environment.
> I like it. Among other things, it could be used with the File transport to > save reports to a directory for examination, which could be useful to check > things before upgrading.
>> Is this useful for others? >> If so, as a snippet (e.g. posted on wiki)? >> as a subclass (name??) of CPAN::Reporter::Smoker? >> as an additional exported sub (name??) in CPAN::Reporter::Smoker?
> At the very least, I'd put it on the CPAN Testers wiki. I have no problem > if you want to publish a CPAN::Reporter::Smoker::* module, but I don't think > I want to include it in C::R::Smoker itself, mostly because it's a slipperly > slope of extra maintenance work for me if I start accepting different ways > of generating the list.
> Any suggestions for a good CPAN::Reporter::Smoker::* subclass name?
> --david
> On Sun, Jan 18, 2009 at 8:31 PM, David Golden <xda...@gmail.com> wrote: > > On Sun, Jan 18, 2009 at 7:59 PM, David Westbrook <dwestbr...@gmail.com> > > wrote:
> >> Whereas you wouldn't want to run a smoker on a "regular" machine (e.g. > >> a $work box, or your day-to-day windows desktop, or laptop, etc), > >> this is reasonably safe since it's only testing distros that you > >> already installed, which implies that they (and their dependencies) > >> are trusted. > >> Also lets you easily submit reports for all the modules you're actual > >> using, w/o a separate smoke environment.
> > I like it. Among other things, it could be used with the File transport > to > > save reports to a directory for examination, which could be useful to > check > > things before upgrading.
> >> Is this useful for others? > >> If so, as a snippet (e.g. posted on wiki)? > >> as a subclass (name??) of CPAN::Reporter::Smoker? > >> as an additional exported sub (name??) in CPAN::Reporter::Smoker?
> > At the very least, I'd put it on the CPAN Testers wiki. I have no > problem > > if you want to publish a CPAN::Reporter::Smoker::* module, but I don't > think > > I want to include it in C::R::Smoker itself, mostly because it's a > slipperly > > slope of extra maintenance work for me if I start accepting different > ways > > of generating the list.
1. Please do not top-quote. 2. Sorry I posted your mail on the list. I do not see any reason why this mail is off-list.
> What part of the "reasonably safe" logic do you see as problematic?
You focused on the "safe" issue, and only see the "problematic". The safer way is to stop the CPAN testers system. The already-installed system may not meet the new requirement upon new releases. The even-safer way is not to release anything. No software, no bugs, problems.
What is the right way? The right way is to solve the problem, not to close your eyes to the problems.
> In regards to the small projects, I think this method would actually > help them -- people could use this to easily submit a bunch of test > reports without the (relatively steep) requirement that they have a > dedicated/separate smoke environment.
I do not see your point. Is this because my English is too poor? An easier step for my module to have "bunch of test reports" from less than 1000 real users where none of them have CPAN::Reporter at all?
> I wasn't suggestion that this replace a full smoke run (where "the > whole ... system will break down") -- but rather to just complement it > for systems that otherwise wouldn't be producing any tests.
I do not see this also. If you remove all the reports from testers that are not really installing the module, the CPAN testers report database will be less than 1% left. Your point does sound like non-sense from governments officials. Could you please speak in a simple English tone?
On Sun, Jan 18, 2009 at 9:45 PM, imacat <ima...@mail.imacat.idv.tw> wrote:
> 2. Sorry I posted your mail on the list. I do not see any reason why > this mail is off-list.
Your first reply was directly to me, and not to the list (so the list is probably missing some context here), which is why i continued it off-list, assuming it was intentional.
> The safer way is to stop the CPAN testers system.
Can you qualify that? Isn't the CPAN testers system the whole reason we're here (this list, the cpantesters site, test reports, etc)??
> The already-installed system may not meet the new requirement upon new releases.
But that's fine .. and David Golden made a good point that this "could be useful to check things before upgrading."
> The even-safer way is not to release anything. No software, no bugs, problems.
ummm .. and no fun, and 0 chance of doing anything good. Isn't writing/releasing the whole point in any of us being programmers?? Yes, of course there's risk in any change/release, but the goal is that the benefits outweigh the risks. That aside, i'm not sure what "not to release anything" is in reference to here ...
> What is the right way? The right way is to solve the problem, not > to close your eyes to the problems.
I'm not sure i follow what "problem" is referring to in this context?
> I do not see your point. Is this because my English is too poor?
English looks just fine to me (based on this and a couple previous posts on this list); but i'm a lowly monolingual and far from fluent in a 2nd language, so definitely not going to pass judgement.
BUT, clearly we're on different wavelengths ... let's start back at the beginning:
Proposition: + Provide an easy way to execute CPAN::Reporter::Smoker, limited just to distributions that have any version installed on the system (instead of all of CPAN).
Pros: + Smoke testing benefits, in terms of generating reports (albeit not for all of CPAN). + Tests are submitted from actual use systems, as opposed to just an isolated builddir & LIB path. + In theory is safer than full smoke, since only testing "trusted" distros (and their deps) -- the user trusted the distro enough to install it in the first place. + Potential use is to check things out before upgrading. + doesn't require (making the trust assumptions) a dedicated/isolated environment
Cons: + In theory, still same potential risk as full smoke run + The "trusted" distros are based on a chain of assumptions.
And then we can tackle each line item individually where we disagree.
> I do not see this also. If you remove all the reports from testers > that are not really installing the module, the CPAN testers report > database will be less than 1% left.
I in no way was suggesting that ... just the opposite, i think this could be helpful to _increase_ the test count. e.g. i wouldn't have submitted a bunch of reports from my windows boxes w/o something like this.
On Mon, Jan 19, 2009 at 6:07 PM, David Westbrook <dwestbr...@gmail.com> wrote:
> Proposition: > + Provide an easy way to execute CPAN::Reporter::Smoker, limited > just to distributions that have any version installed on the system > (instead of all of CPAN).
In general I think this is a good idea and I probably would run it on my development machine.
> Pros: > + In theory is safer than full smoke, since only testing "trusted" > distros (and their deps) -- the user trusted the distro enough to > install it in the first place.
For additional safety you might want to limit the installations to
1) Modules that already have N reports in the database meaning that some other testers have already tested it.
2) Modules that have been on CPAN for at least M days hoping that if the module and its tests do something bad someone has already caught this, reported to the "rulers of CPAN" and the module was taken off CPAN.
> For additional safety you might want to limit the > installations to
> 1) Modules that already have N reports in the database > meaning that some other testers have already tested it.
> 2) Modules that have been on CPAN for at least M days > hoping that if the module and its tests do something > bad > someone has already caught this, reported to the > "rulers of CPAN" and the module was taken off > CPAN.
> regards > Gabor
What about those distributions that have only that dummy test file with a single "BEGIN { use Foo::Bar; }" that h2xs generates?
This actually does not means to much for me, but the smoke test (and mannually too) will consider this distribution OK.
Shouldn't the smoke test include something like Devel::Cover?
On Mon, Jan 19, 2009 at 1:31 PM, Gabor Szabo <szab...@gmail.com> wrote:
>> Pros: >> + In theory is safer than full smoke, since only testing "trusted" >> distros (and their deps) -- the user trusted the distro enough to >> install it in the first place.
> For additional safety you might want to limit the installations to
> 1) Modules that already have N reports in the database > meaning that some other testers have already tested it.
> 2) Modules that have been on CPAN for at least M days > hoping that if the module and its tests do something bad > someone has already caught this, reported to the > "rulers of CPAN" and the module was taken off CPAN.
ah -- i like those! Suggestions on defaults for N & M? I guess i'll look at my installed systems as examples and see what gets excluded for different values ...
Any thoughts on CPAN::Reporter::Smoker::Safer for the namespace? i also like ::Installed, but taking into account the above, which shifts the focus a little, I'm leaning toward ::Safer. (does ::Safer imply a negative connotation towards CPAN::Reporter::Smoker?)
Smoke testing will download and run programs that other people have uploaded to CPAN. These programs could do *anything* to your system, including deleting everything on it. Do not run CPAN::Reporter::Smoker unless you are prepared to take these risks.
> On Sun, Jan 18, 2009 at 9:45 PM, imacat <ima...@mail.imacat.idv.tw> wrote: >> 2. Sorry I posted your mail on the list. I do not see any reason why >> this mail is off-list.
> Your first reply was directly to me, and not to the list (so the list > is probably missing some context here), which is why i continued it > off-list, assuming it was intentional.
Oh... sorry. I do not know the cpan-testers-discuss list does not set the Reply-to: header, and I did not check it. Maybe I shall write to the list manager some other time.
> Proposition: > + Provide an easy way to execute CPAN::Reporter::Smoker, limited > just to distributions that have any version installed on the system > (instead of all of CPAN).
I still do not see at all.
1. How many smokers have installed my module? Only one, that is, I. How many smokers have installed Mac::iTune? No one.
2. With this new system you suggested, I will only get my own reports, which is nearly useless. And other authors will get no reports at all.
3. I will stop running smoke tests, since when I upgrade the modules weekly, the reports will be sent anyway. If I will not be sending reports other than the above, running smoke tests is a duplicated work and CPU waste which could be saved. In fact, currently the only modules I do not need to run smoke tests are those that I did not installed.
4. Nobody gets reports except for those popular modules like LWP, etc.. Also, nobody is running smoke tests anymore.
5. If nobody is running smoke tests, and nobody is receiving smoke test reports except LWP, how does this CPAN testers system work?
So, why are you thinking that this will increase the amount of reports, but not demolish the whole CPAN testers system?
If that is not what you mean, then I have a serious English problem. Either I misunderstand your proposition, or I misunderstand your expected result. In either way, I may have to restart my English class.
On Tue, Jan 20, 2009 at 9:13 PM, imacat <ima...@mail.imacat.idv.tw> wrote: >> Proposition: >> + Provide an easy way to execute CPAN::Reporter::Smoker, limited >> just to distributions that have any version installed on the system >> (instead of all of CPAN).
> I still do not see at all.
> 1. How many smokers have installed my module? Only one, that is, > I. How many smokers have installed Mac::iTune? No one.
For your module, and Mac::iTune, there won't be an added benefit. But there will be for more commonly installed modules. Which, of course, are already tested, but this could reach additional platforms/perls -- that potentially could be a bigger benefit than additional modules.
> 2. With this new system you suggested, I will only get my own > reports, which is nearly useless. And other authors will get no reports > at all.
True, it's not reaching all of CPAN, but OTOH it's not zero modules either -- it's whatever's on your system .. which i'm guessing is 200-500 distros.
> 3. I will stop running smoke tests, since when I upgrade the > modules weekly, the reports will be sent anyway. If I will not be > sending reports other than the above, running smoke tests is a > duplicated work and CPU waste which could be saved.
Here's where are lines of thought split i think .. You wouldn't need to (and shouldn't) stop running smoke tests if your running this new system (C::R::S::Safer) .. There's two cases here: a) The system (read: platform/perl) you're smoking on is the same as where you're running C::R::S::Safer. + In this case, i'd say either don't run C::R::S::Safer, or make sure the reports-send.db file is shared between the C::R::S::Safer runs and the C::R::S smoke runs. That way there's no wasted resources. b) The platform/perl is different. This situation is what i'm targeting .. where you're smoking on system A, but also have system B, which full smoking is unsafe and/or inappropriate on. Running C::R::S::Safer on system B at least gets some partial smoke coverage/reports, whereas without it there wouldn't be any.
Basically, it's not proposed to replace smoke testing -- just to be used where smoke testing can't (or shouldn't) be done.
Your points 4. & 5. and "demolish the whole CPAN testers system" followed from the assumption that running this would mean ceasing smoke testing, so i'll skip those.
Take my actual setups for an example ... I have: * a $work laptop (winXP, 5.10) * a home desktop (win2K, 5.8.8) * a spare laptop (no hard drive) * hosting account (linux, i686, 5.8.8) * $work dev env (linux, 5.6.1)
On the spare laptop, i'm booting a knoppix livecd and smoking from there (linux, i486, 5.8.8). BUT, the other 4 systems w/unique platforms/perls I can't run smoke testing on (for both safety and resources reasons). BUT, i am willing to run C::R::S::Safer, to ensure that i've contributed reports for everything that i have installed.
So this is where I see it only creating additional testers reports.
Basically, i envision the following levels of test report contribution:
a) Install CPAN::Reporter, and send reports for any module that gets installed from that point on. b) Install CPAN::Reporter::Smoker::Safer, and run it to submit reports for all the distros you "trust" (read: have installed). And run it again in the future to test any newer versions of those distros, w/o actually installing them. c) Install & run CPAN::Reporter::Smoker (or other smoker), and submit reports for everything. --> BUT, this takes an isolated & dedicated environment.
And it's key to note that these 3 are not mutually exclusive .. can do (c) on one system, and (a) and/or (b) on others (as in my setup described above).
> In either way, I may have to restart my English class.
don't waste the time or money :) -- w/o your mention of it and the .tw email, I would have assumed the above was written by a native english speaker.
This is a good thread -- parts of it will definitely make their way into the C::R::S::Safer documentation.
On Wed, Jan 21, 2009 at 4:13 AM, imacat <ima...@mail.imacat.idv.tw> wrote: > On 01/20/2009 12:07 AM, David Westbrook wrote: >> Proposition: >> + Provide an easy way to execute CPAN::Reporter::Smoker, limited >> just to distributions that have any version installed on the system >> (instead of all of CPAN).
> I still do not see at all.
> 1. How many smokers have installed my module? Only one, that is, > I. How many smokers have installed Mac::iTune? No one.
> 2. With this new system you suggested, I will only get my own > reports, which is nearly useless. And other authors will get no reports > at all.
> 3. I will stop running smoke tests, since when I upgrade the > modules weekly, the reports will be sent anyway. If I will not be > sending reports other than the above, running smoke tests is a > duplicated work and CPU waste which could be saved. In fact, currently > the only modules I do not need to run smoke tests are those that I did > not installed.
> 4. Nobody gets reports except for those popular modules like LWP, > etc.. Also, nobody is running smoke tests anymore.
> 5. If nobody is running smoke tests, and nobody is receiving smoke > test reports except LWP, how does this CPAN testers system work?
> So, why are you thinking that this will increase the amount of > reports, but not demolish the whole CPAN testers system?
This Safer CPAN Tester module could be installed by people like me who do don't do smoke testing at all (or by you but on your real development environment) thus it has the potential to increase the number of reports in more natural environments.
One of my main issues with the tons of reports the main smokers send - despite the fact that I am totally thankful for them - that those reports are generated in a lab environment and their sheer number hides the few reports that come from real setups.
Point: while I am happy to see my modules tested on all version of 5.8.x on NetBSD it would be more important to get the reports from people using the *default perl* of Red Hat 17.0 and Enterprise SuSE 38.7 or whatever version numbers they have.
Those environments are usually more difficult and much more broken (!) than those of the smokers.
This Safer smoker has the potential to help increasing this number.
On Wed, Jan 21, 2009 at 10:13:52AM +0800, imacat wrote: > Oh... sorry. I do not know the cpan-testers-discuss list does not > set the Reply-to: header, and I did not check it. Maybe I shall write > to the list manager some other time.
