Re: smoking with 5.15, 5.13

[Andreas Koenig]

Ricardo Signes <perl.t...@rjbs.manxome.org> writes:

> Can we all agree that there's no reason at this date to be smoking with 5.13 or
> 5.15?
>
> I love reports from older perls, and from recent 5.17s. Reports from 5.13 and
> 5.15 strike me as utterly pointless. I'm still getting them, though, and being
> told by a smoker that it's my problem to deal with.
>
> I would appreciate some input from the rest of the list on whether I am wrong
> to think that these reports are a waste of the time of everyone involved and
> affected.

+1

One might argue the ultimate goal of cpantesters is to find bugs. I'd
say yes, but the bugs should be relevant. Bugs in very old and
irrelevant combinations are *usually* noise that makes useful work
harder for everybody involved.

Occasionally it happens on my smokers too, that I send a report for
something that is a bit outdated. This usually happens while I'm
bisecting. But to prevent it from happening too often I have a
distroprefs file (recommended for all CPAN.pm based smokers):

match:
perlconfig:
version: '^5\.(7|9|11|13|15)\.'
cpanconfig:
test_report: 0


Frankly I confess I had only the regexp for up to 11 and have added the
13 and 15 right now:)

--
andreas

[MPR]

On Wed, Jan 2, 2013 at 8:06 AM, Andreas Koenig
<andreas.koe...@franz.ak.mind.de> wrote:
> Ricardo Signes <perl.t...@rjbs.manxome.org> writes:
>
>> Can we all agree that there's no reason at this date to be smoking with 5.13 or
>> 5.15?
>

> One might argue the ultimate goal of cpantesters is to find bugs. I'd
> say yes, but the bugs should be relevant. Bugs in very old and
> irrelevant combinations are *usually* noise that makes useful work
> harder for everybody involved.

You say "usually". Is there a case where it might be useful to test
and send reports against an old version?

As an alternative, would it be better to have cpantesters.org not
email the authors when a test is run against old versions of Perl?
That way each tester won't have to maintain an entry in their
distroprefs. We can exclude in a central location.

[David Oswald]

I want the failure reports for older versions of Perl, back through
5.6.x. A distribution can specify a minimum Perl version in its META
information. To me it makes sense that reports be issued for any Perl
from current, back through whatever minimum version is specified with
the exception of releases from retired development tracks.

Andreas mentioned that the goal of the smoke testing is to find bugs.
Certainly the failure to support an older Perl that is within the
range bounded by the minimum version specified in the distribution
*is* a bug.

Additionally, if I've got email notification turned on, and found that
FAILs were being issued on an older Perl without my getting an email,
I'd say that the notification tool's effectiveness is broken.

Of course all this is with respect to older production-track Perls,
not older development-tracks. I agree that it's not useful for
retired development track Perl's to be used in smoke testing modules.

--

David Oswald
daos...@gmail.com

[Barbie]

On Wed, Jan 02, 2013 at 12:27:54PM -0500, Ricardo Signes wrote:
> * David Oswald <daos...@gmail.com> [2013-01-02T12:25:29]

> > Of course all this is with respect to older production-track Perls,
> > not older development-tracks. I agree that it's not useful for
> > retired development track Perl's to be used in smoke testing modules.
>

> I am definitely interested in the results of older stable perls. Reports from
> 5.8 are welcome indeed.
>
> I only want to see an end to old development releases:

>
> >>> Can we all agree that there's no reason at this date to be smoking with
> >>> 5.13 or 5.15?
>

> So far, it seems we can.

I'll mention this in my next summary. Blocking the reports at
www.cpantesters.org is a little too late, so adding appropriate
distroprefs as Andreas suggested is the right way to do this.

I'll also try and remember to ask peeps to add the old dev version to
their distroprefs when the next stable release is out too.

Cheers,
Barbie.
--
Birmingham Perl Mongers <http://birmingham.pm.org>
Memoirs Of A Roadie <http://barbie.missbarbell.co.uk>
CPAN Testers Blog <http://blog.cpantesters.org>
YAPC Conference Surveys <http://yapc-surveys.org>
Ark Appreciation Pages <http://ark.eology.org>

[Andreas Koenig]

MPR <mplist...@gmail.com> writes:

> On Wed, Jan 2, 2013 at 8:06 AM, Andreas Koenig
> <andreas.koe...@franz.ak.mind.de> wrote:
>> Ricardo Signes <perl.t...@rjbs.manxome.org> writes:
>>
>>> Can we all agree that there's no reason at this date to be smoking with 5.13 or
>>> 5.15?
>>
>> One might argue the ultimate goal of cpantesters is to find bugs. I'd
>> say yes, but the bugs should be relevant. Bugs in very old and
>> irrelevant combinations are *usually* noise that makes useful work
>> harder for everybody involved.
>
> You say "usually". Is there a case where it might be useful to test
> and send reports against an old version?

When a test passes with perl 5.14 and fails with 5.16, then we will
probably be interested in the results with 5.15 (e.g.
http://matrix.cpantesters.org/?dist=Attribute-Util%201.06 )

Counter example where two lonely fails on some 5.13.x perls are quite
disturbing: http://matrix.cpantesters.org/?dist=Path-Class-Rule%200.013


--
andreas

[David Golden]

Forgot to copy the list in my reply to Rik.

On Wed, Jan 2, 2013 at 9:46 AM, David Golden <x...@xdg.me> wrote:

> On Wed, Jan 2, 2013 at 8:42 AM, Ricardo Signes
> <perl.t...@rjbs.manxome.org> wrote:
>> Can we all agree that there's no reason at this date to be smoking with 5.13 or
>> 5.15?
>

> +1
>
> Among other things, report storage is not free.
>
> David
>
>
> --
> David Golden <x...@xdg.me>
> Take back your inbox! → http://www.bunchmail.com/
> Twitter/IRC: @xdg



--
David Golden <x...@xdg.me>
Take back your inbox! → http://www.bunchmail.com/
Twitter/IRC: @xdg

[David Golden]

On Wed, Jan 2, 2013 at 2:45 PM, Barbie <bar...@missbarbell.co.uk> wrote:
> I'll mention this in my next summary. Blocking the reports at
> www.cpantesters.org is a little too late, so adding appropriate
> distroprefs as Andreas suggested is the right way to do this.
>
> I'll also try and remember to ask peeps to add the old dev version to
> their distroprefs when the next stable release is out too.

When I eventually have some test boxes, I can add Metabase logic to
reject reports from old dev versions.

[Eirik Berg Hanssen]

On Wed, Jan 2, 2013 at 6:07 PM, MPR <mplist...@gmail.com> wrote:

> As an alternative, would it be better to have cpantesters.org not
> email the authors when a test is run against old versions of Perl?
>

If you go that way, make sure that's old _development_ versions of Perl.
As rjbs wrote: "I love reports from older perls, and from recent 5.17s."

(I'm still prepared to keep my old stuff working on 5.6.2, as long as I
don't need to bend over backwards, but 5.15 is assumed unused and useless.)


Eirik

[David Cantrell]

On Wed, Jan 02, 2013 at 09:07:32AM -0800, MPR wrote:
> On Wed, Jan 2, 2013 at 8:06 AM, Andreas Koenig
> <andreas.koe...@franz.ak.mind.de> wrote:
> > One might argue the ultimate goal of cpantesters is to find bugs. I'd
> > say yes, but the bugs should be relevant. Bugs in very old and
> > irrelevant combinations are *usually* noise that makes useful work
> > harder for everybody involved.
> You say "usually". Is there a case where it might be useful to test
> and send reports against an old version?

Sure. Even if a module's author doesn't care, the module's users do.
And cpXXXan cares deeply.

--
David Cantrell | A machine for turning tea into grumpiness

Aluminum makes a nice hat.
All paranoids will tell you that.
But what most do not know
Is reflections will show
On the CIA's evil landsat.

[Nigel Horne]

On Fri, Jan 04, 2013 at 02:19:43PM +0000, David Cantrell wrote:
> On Wed, Jan 02, 2013 at 09:07:32AM -0800, MPR wrote:
> > On Wed, Jan 2, 2013 at 8:06 AM, Andreas Koenig
> > <andreas.koe...@franz.ak.mind.de> wrote:
> > > One might argue the ultimate goal of cpantesters is to find bugs. I'd
> > > say yes, but the bugs should be relevant. Bugs in very old and
> > > irrelevant combinations are *usually* noise that makes useful work
> > > harder for everybody involved.
> > You say "usually". Is there a case where it might be useful to test
> > and send reports against an old version?
>
> Sure. Even if a module's author doesn't care, the module's users do.

+1

-Nigel

[David Cantrell]

On Fri, Jan 04, 2013 at 10:32:14AM -0500, Ricardo Signes wrote:
> * Nigel Horne <n...@bandsman.co.uk> [2013-01-04T09:43:13]

> ...but who is a "user" on a year old development version?

No-one. And that's not what I was talking about.

--
David Cantrell | Bourgeois reactionary pig

It wouldn't hurt to think like a serial killer every so often.
Purely for purposes of prevention, of course.

[Andreas Koenig]

Ricardo Signes <perl.t...@rjbs.manxome.org> writes:

>> When a test passes with perl 5.14 and fails with 5.16, then we will
>> probably be interested in the results with 5.15 (e.g.
>> http://matrix.cpantesters.org/?dist=Attribute-Util%201.06 )
>

> I'm not sure that this is really a good example. The question is whether
> anyone should be *still* testing and sending reports against old dev versions.
>
> So, the way this would be useful is if:
>
> 1. we have known passes for 5.14
> 2. we have known failures for 5.16
> 3. we have /no existing reports/ for 5.15
> 4. it's useful to know what commit changed things from pass to fail
>
> The first three conditions need to be true, *and* somebody needs to be smoking
> *both* the "still works" earlier 5.15.x *and* the "now fails" 5.15.(x+1), and
> that still only gets you to a 30 day window of commits to test.
>
> It seems like a lot of noise with only a very, very small chance of producing
> signal. Meanwhile, we have git-bisect to quickly get to the actual commit!
>
> I'm delighted to have those smoke reports rolling in during 5.15, though, so we
> CPAN authors can see what 5.16 might break. Once 5.16 is out the door, though,
> it hardly seems to have any value.

Needless to say, I fully agree with you. Thank you for spelling the full
argument out so clearly.

--
andreas