dump.c cleanup

4 views
Skip to first unread message

Jim Cromie

unread,
May 7, 2009, 4:12:38 PM5/7/09
to Perl 5 Porters
following the excellent howto in pod/perlrepository,
Ive pushed the following to github

http://github.com/jimc/perl/commits/dump-pub

[jimc@harpo perl]$ ls 000*
0001-chop-Perl_do_op_dump-into-reusable-chunks.patch
0002-reuse-S_dump_op_flags-in-Perl_do_op_xmldump.patch
0003-keep-a-single-decl-of-S_dump_op_mad.patch
0004-reindent-code-previously-carved-into-separate-funcs.patch
0005-drop-blank-line-in-debug-output-it-doesnt-help-read.patch
0006-update-embed.fnc-with-new-STATICs-in-dump.c.patch
0007-mortalize-new-internals.patch
0008-add-t-run-mad.t-for-Dmad-builds.patch

do we also have a shiny new review process ? <ducks, rolls, then hides>

Nicholas Clark

unread,
May 25, 2009, 11:31:56 AM5/25/09
to Jim Cromie, Perl 5 Porters
On Thu, May 07, 2009 at 02:12:38PM -0600, Jim Cromie wrote:
> following the excellent howto in pod/perlrepository,
> Ive pushed the following to github

> do we also have a shiny new review process ? <ducks, rolls, then hides>

No, we don't. It's still (mostly) Rafael reviewing and merging ever
third-party patch proposed for the core.

For all the comments encouraging switching to git beforehand, five months
on I don't think it's made a measurable difference in either:

1: number of people committing their own changes direct to core
2: number of people reviewing and applying other people's changes
3: amount of activity

I've been trying to generate graphs to understand what's going on, to assess
how confident I am in the above assertions, but the data is very noisy.

For example, for blead, commits per week for the top 4 individual contributors
compared to total commits, unsmoothed: http://xrl.us/betqzi

(From which I observe that my spikes aren't as big as Jarkko's were)

code change and total commits, smoothed: http://xrl.us/bet385

(that one is very noisy just recently because of all the lines removed due to
the deletion of the Changes files, but generally the orange and red lines
wiggle up and down in much the same fashion this year as they did last year
or the year before)


I infer that everyone currently involved is very busy with other things.
(Certainly I am, and I know what some others are busy with)


What's particularly frustrating is that booking.com donated $50,000 to TPF 5
months ago today, "to aid in the further development and maintenance of the
Perl programming language in general, and Perl 5.10 in particular":
http://news.perlfoundation.org/2008/12/bookingcom_makes_a_major_contr.html

That money remains unspent, and there is no plan published by TPF on how it
proposes to put that money to good use, to meet the aims of the donor. I had
hoped that the TPF board would approve Richard Dice's proposal to be employed
full time by TPF, and that this would give Richard some time to work on it.
However, the proposal has been revised, and now gives Richard no time to work
on things outside a specific list of deliverables, so nothing here is going
to change fast.

( http://news.perlfoundation.org/2009/05/continued_review_of_tpf_grant.html
http://use.perl.org/comments.pl?sid=43032&cid=68736
from http://use.perl.org/~domm/journal/39013 )


I don't have the spare time to organise spending the money wisely.
Neither does offering me money create more time for me to do things.
So I'm not able to be part of the solution on this one.

Money isn't the problem. Time and enthusiasm is. If people really wanted
things done, they would do them, or organise spending this money to get them
done. But for all the talk, no-one follows up with action. So I infer that
people don't actually want anything that badly.

Nicholas Clark

David Golden

unread,
May 25, 2009, 3:05:12 PM5/25/09
to Perl 5 Porters
On Mon, May 25, 2009 at 11:31 AM, Nicholas Clark <ni...@ccl4.org> wrote:
> I've been trying to generate graphs to understand what's going on, to assess
> how confident I am in the above assertions, but the data is very noisy.

I'm not sure you'll ever be able to isolate the effect. Certainly not
over such a short timeframe. And with dual-life modules, a lot of
changes may be happening in upstream repos. Is that a git effect? I
don't know. But certainly, I've found it a lot easier to track
activity in blead/maint/ancient now that it's in git and even
contribute the odd patch or two. Or, in the case of Module::Build, try
to make sure that blead and upstream were synchronized.

I also wouldn't have worked on my own "maint-5.X.Y" branches if it
hadn't been for git. (http://github.com/dagolden/perl) Maybe that
hasn't made it upstream into the main repo -- partly because some of
my initial work on it didn't get such a great reception on the p5p
mailing list -- but I never would have even attempted it prior to git.

> Money isn't the problem. Time and enthusiasm is. If people really wanted
> things done, they would do them, or organise spending this money to get them
> done. But for all the talk, no-one follows up with action. So I infer that
> people don't actually want anything that badly.

I infer that people don't want anything that badly only relative to
all the other things in life they want badly right now.

-- David

Nicholas Clark

unread,
May 25, 2009, 5:22:44 PM5/25/09
to David Golden, Perl 5 Porters
On Mon, May 25, 2009 at 03:05:12PM -0400, David Golden wrote:
> On Mon, May 25, 2009 at 11:31 AM, Nicholas Clark <ni...@ccl4.org> wrote:
> > I've been trying to generate graphs to understand what's going on, to assess
> > how confident I am in the above assertions, but the data is very noisy.
>
> I'm not sure you'll ever be able to isolate the effect. Certainly not
> over such a short timeframe. And with dual-life modules, a lot of

There was a lot of talk from people who said that the fact that the core was
using Perforce was a barrier that prevented them from hacking on the core.

Well, we removed that barrier, so if that demand was pent up, and it really
was a barrier, rather than the lowest hanging excuse (yes, I am being
marginally provocative in that choice of word), then we should have seen
some activity due to these people now being able to hack on the core.

But we haven't.

It is possible to see effects at the 2-3 month scale in http://xrl.us/betqzi
For example, the point where the green line for my commit rate disappears in
mid 2006 correlates with starting a job at Shopzilla UK, that I found very
demoralising, and obliterated any energy or enthusiasm I had to hack on the
core. Again, the gap in 2007 correlates with a troublesome contract that I
did, which again robbed my free-time motivation. My spike in early 2009
correlates with the work I did on rearranging ext, and how extensions are
built.


> changes may be happening in upstream repos. Is that a git effect? I
> don't know. But certainly, I've found it a lot easier to track
> activity in blead/maint/ancient now that it's in git and even
> contribute the odd patch or two. Or, in the case of Module::Build, try
> to make sure that blead and upstream were synchronized.

This is useful, and I'm not suggesting that it's unimportant. There's only a
finite amount of volunteer time in the Perl world, so anything that makes
more efficient use of that time is a good thing.

Curiously, it's also consistent with what someone on IRC said (Joshua ben Jore,
I think) - git made it a lot easier for him to trace upstream changes, and
produce an "official release + patches" build for work.

But it's not leading to, or at least not yet directly leading to, increased
participation in core development.

> > Money isn't the problem. Time and enthusiasm is. If people really wanted
> > things done, they would do them, or organise spending this money to get them
> > done. But for all the talk, no-one follows up with action. So I infer that
> > people don't actually want anything that badly.
>
> I infer that people don't want anything that badly only relative to
> all the other things in life they want badly right now.

Agree.

Maybe I'm over-sensitive, but I seem to notice a lot of people (not
yourself, I should note) griping about lack of progress on releases, but
neither proposing anything to help solve the problem, nor asking how they
could help. So I'm starting to conclude that it's all idle chat, or at least
only regarded as a "nice to have" rather than a "must have", however much
the tone might suggest otherwise.

Nicholas Clark

Chromatic

unread,
May 25, 2009, 6:06:06 PM5/25/09
to perl5-...@perl.org, Nicholas Clark
On Monday 25 May 2009 14:22:44 Nicholas Clark wrote:

> Maybe I'm over-sensitive, but I seem to notice a lot of people (not
> yourself, I should note) griping about lack of progress on releases, but
> neither proposing anything to help solve the problem, nor asking how they
> could help. So I'm starting to conclude that it's all idle chat, or at
> least only regarded as a "nice to have" rather than a "must have", however
> much the tone might suggest otherwise.

Several people who volunteered last year to manage a releases from a checklist
of tasks they could work through to produce a release. What happened? What
happened to that checklist? Was it a bad idea? Did no one have time to write
it? Did the volunteers all say "No, thank you!" privately?

Do I recall correctly that this process stalled when someone suggested "It's
just too hard to generate a perldelta, so thanks, but no thanks!" (Once upon
a time a pumpking decided that patches without test cases and documentation
changes were unacceptable, so it seems like a small leap to require delta
patches too.)

As long as producing a release relies on someone spending a day a week
cherrypicking patches from an unstable branch into a stable branch, I believe
that no one who knows how to make a release will ever have time to write up
what it takes to make a release (let alone what it will take to make a release
parallelizable and trivial), and no one who doesn't already know how to make a
release will learn how to make a release. Cart, egg, horse, chicken.

My concrete suggestions are:

0) Release 5.10.1 and 5.11.0 concurrently

1) Require all feature work to occur in branches

2) Declare and enforce trunk stability (no more "bleadperl" from which someone
cherrypicks stable commits)

3) Enable smoking of branches ready to merge to trunk

4) Require all merge-ready branches to include perldelta changes, when
appropriate

5) Write a release manager guide

6) Recruit likely release managers for the next several releases

7) Set a release schedule tied to the calendar, not the state of any feature

8) Follow the release schedule

I believe #2 is the most important change to unblock this process and am happy
to write about it in more detail if anyone cares.

Git enables 1, 2, and 3. 0, 1, 2, 4, 5, and 8 require active pumpking
support. Is there interest?

I'm still working on a support guide, but I'm sure there'll be plenty of
debate about it.

If I'm full of hot air because I spend most of my development time on Parrot
these days (though we have no trouble there with monthly stable releases), or
if none of my proposals are interesting or useful, I can stop talking about
this, though if I'm not completely crazy I'd certainly like feedback on why
these suggestions are impractical or infeasable.

-- c

Craig A. Berry

unread,
May 25, 2009, 6:59:50 PM5/25/09
to Jim Cromie, Perl 5 Porters
On Mon, May 25, 2009 at 10:31 AM, Nicholas Clark <ni...@ccl4.org> wrote:

> For all the comments encouraging switching to git beforehand, five months
> on I don't think it's made a measurable difference in either:
>
> 1: number of people committing their own changes direct to core
> 2: number of people reviewing and applying other people's changes
> 3: amount of activity

I can't say that I've noticed an overall difference in activity
either, but see below.

> I've been trying to generate graphs to understand what's going on, to assess
> how confident I am in the above assertions, but the data is very noisy.
>
> For example, for blead, commits per week for the top 4 individual contributors
> compared to total commits, unsmoothed: http://xrl.us/betqzi

I think a better measure than trends in the top 4 committers would be
whether everybody else has done more relative to the top 4, or whether
the overall amount of work getting done has changed.

> (From which I observe that my spikes aren't as big as Jarkko's were)

Yeah, but the area under the curve is important too :-).

Unfortunately Porting/checkAUTHORS.pl depends on the changelogs that
aren't there any more, so the following is a poor man's version that
does what it can with the git-generated logs. This depends very
heavily on the quality of the conversion from Perforce (which seems
very high as far as I can tell), and specifically that the way author
and committer is recorded in converted data is comparable to the same
in data generated since the conversion.

I created two logs, one covering the roughly one-year period from the
release of 5.10.0 to the switch to git, and one covering the roughly
six months from the switch to git until now:

% git log --pretty=fuller --name-status perl-5.10.0..GitLive-blead >
../fiveten2git.txt
% git log --pretty=fuller --name-status GitLive-blead.. > ../git2now.txt

At first blush it looks like the number of authors of all stripes has
fallen by about 25% since the switch to git:

% grep "Author:" fiveten2git.txt | sort -u | wc -l
125
% grep "Author:" git2now.txt | sort -u | wc -l
96

There are some duplicates in there because I don't have the
intelligent grouping of checkAUTHORS.pl, but I don't think those are
enough to affect the trend. Possibly we are not doing a good enough
job with git am or git commit --author to properly give credit where
credit is due. Some more digging here would be appropriate and could
be done by anyone willing to eyeball a number of changes and see if
the committers are forgetting to record the author with some
frequency.

The number of committers, on the other hand, has roughly doubled:

% grep "Commit:" fiveten2git.txt | sort -u | wc -l
12
% grep "Commit:" git2now.txt | sort -u | wc -l
27

There are some duplicates and noise in here, as well, but the
following people have pushed commits since the switch to git but had
not done so during the final year of Perforce:

Abigail
Ben Morrow
Florian Ragwitz
Jan Dubois
Leon Brocard
Steffen Mueller
Vincent Pit
Yitzchak Scott-Thoennes
Yuval Kogman

That looks like pure good news to me.

Unless I've shot myself in the foot by trying to do too much in
one-liners (and I could probably avoid some of the post-processing
with smarter invocations of git log), the following gives us the
number of total commits per month in the year or so leading up to the
switch to git:

% grep CommitDate: fiveten2git.txt | perl -ane '$m=$F[2].$F[5]; $c++;
if ($pm && (eof || $pm ne $m)) {print qq/$pm $c\n/; $c=0;} $pm = $m;'
Dec2008 75
Nov2008 157
Oct2008 140
Sep2008 99
Aug2008 65
Jul2008 66
Jun2008 97
May2008 103
Apr2008 115
Mar2008 120
Feb2008 176
Jan2008 300
Dec2007 135

and in the six months since the switch:

% grep CommitDate: git2now.txt | perl -ane '$m=$F[2].$F[5]; $c++; if
($pm && (eof || $pm ne $m)) {print qq/$pm $c\n/; $c=0;} $pm = $m;'
May2009 154
Apr2009 131
Mar2009 199
Feb2009 242
Jan2009 310
Dec2008 162

where December 2008 is clearly split between pre-git and post-git, and
December 2007 shows only the post-5.10.0 commits. January, February,
March, and April of 2009 are the months of pure git data available and
each has a higher number of commits than the same month the previous
year on Perforce. In fact it looks like this 4-month period has about
25% more commits this year than the same period last year.

I always thought that the potential benefits of switching to git were
overstated and the costs of making the switch were understated, but it
does look as though the period of its use corresponds to a period of
more committers and more commits. It might also correspond to a
decreasing number of patch authors, though hopefully that's just bad
data.

But more commits doesn't automatically lead to more releases, and I
don't think git ever had the potential to do more than open the doors
to the workshop a bit wider. That's a good thing, but obviously not
the only thing.

Mark Mielke

unread,
May 25, 2009, 7:58:02 PM5/25/09
to Craig A. Berry, Jim Cromie, Perl 5 Porters
I suspect if you trended commits to Perl 5 since 2000, it is probably
pretty consistent and dropping throughout. The transition to git
probably hasn't really + or - anything. Serious commiters could always
get commits in using p4 or by submitting a patch, whatever was worth
their time to learn to figure out. Git opens the door, but without
people behind the door, this means nothing. Really, the demand for
changes to Perl 5 is down and it continues to drop over time. I don't
want Perl 5 to change myself. I'd rather all the "we need change" people
switched over to Perl 6 and made that project successful than muck
around with Perl 5.

So, any comparison of 2007 to 2008 and assuming anything about git needs
to take into account the generally downward trend.

Note that this doesn't mean work has stopped. We have CPAN to accomplish
almost anything we need, with far fewer restrictions on what the
contributions need to look like or what standards they must meet. The
core is stable - which is great. The modules are unrestricted - which is
also great.

I think everything is fine.

Cheers,
mark

--
Mark Mielke <ma...@mielke.cc>

Mark Mielke

unread,
May 25, 2009, 9:00:15 PM5/25/09
to Craig A. Berry, Jim Cromie, Perl 5 Porters
Can you run the same report counting "churn" instead of "commits"? :-)

The 1990 - 2000 years look highly suspect to me, since those are the
days when lots of churn really occurred. Perl 5 was introduced. Two
different threading models. Unicode support. Compiled Perl Support (even
if it never worked perfectly). All sorts of portability work. The
numbers pre 1997 don't jive with my recollection of history. Either the
changes were huge - or there is history missing from your report. (Or,
the changes were combined into larger changes before they made it into
your history)

Cheers,
mark


Craig A. Berry wrote:


> On Mon, May 25, 2009 at 6:58 PM, Mark Mielke <ma...@mark.mielke.cc> wrote:
>
>> I suspect if you trended commits to Perl 5 since 2000, it is probably pretty
>> consistent and dropping throughout.
>>
>

> Don't suspect. Count. 2001 was certainly a bumper crop (what is *in*
> that Finnish coffee?), but there is no general downward trend in
> recent years (2007 has almost twice as many commits as 2003, for
> example). Following shows commits per year, with 2009 looking like it
> will be a lot bigger than 2008 by the time the year is done.
>
> % git log --pretty=fuller | grep CommitDate: | perl -ane '$m=$F[5];


> $c++; if ($pm && (eof || $pm ne $m)) {print qq/$pm $c\n/; $c=0;} $pm =
> $m;'

> 2009 1036
> 2008 1675
> 2007 2377
> 2006 2700
> 2005 2393
> 2004 1175
> 2003 2555
> 2002 3950
> 2001 5040
> 2000 3345
> 1999 1904
> 1998 1755
> 1997 1290
> 1996 790
> 1995 1
> 1996 129
> 1995 1
> 1996 11
> 1995 2
> 1996 16
> 1995 1
> 1996 3
> 1995 1
> 1996 12
> 1995 43
> 1994 6
> 1993 7
> 1992 16
> 1991 23
> 1990 33
> 1989 9
> 1988 16
>
>


--
Mark Mielke <ma...@mielke.cc>

Craig A. Berry

unread,
May 25, 2009, 8:52:26 PM5/25/09
to Mark Mielke, Jim Cromie, Perl 5 Porters
On Mon, May 25, 2009 at 6:58 PM, Mark Mielke <ma...@mark.mielke.cc> wrote:
> I suspect if you trended commits to Perl 5 since 2000, it is probably pretty
> consistent and dropping throughout.

Don't suspect. Count. 2001 was certainly a bumper crop (what is *in*


that Finnish coffee?), but there is no general downward trend in
recent years (2007 has almost twice as many commits as 2003, for
example). Following shows commits per year, with 2009 looking like it
will be a lot bigger than 2008 by the time the year is done.

% git log --pretty=fuller | grep CommitDate: | perl -ane '$m=$F[5];


$c++; if ($pm && (eof || $pm ne $m)) {print qq/$pm $c\n/; $c=0;} $pm =
$m;'

Jan Dubois

unread,
May 25, 2009, 9:06:40 PM5/25/09
to Mark Mielke, Craig A. Berry, Jim Cromie, Perl 5 Porters
On Mon, 25 May 2009, Mark Mielke wrote:
>
> The 1990 - 2000 years look highly suspect to me, since those are the
> days when lots of churn really occurred. Perl 5 was introduced. Two
> different threading models. Unicode support. Compiled Perl Support (even
> if it never worked perfectly). All sorts of portability work. The
> numbers pre 1997 don't jive with my recollection of history. Either the
> changes were huge - or there is history missing from your report. (Or,
> the changes were combined into larger changes before they made it into
> your history)

Perl moved to Perforce in March 1997, so any older patches probably have
been collected from mailing list archives, or tarballs or something. So
I'm not surprised that the number of "commits" are much smaller before
1997.

Cheers,
-Jan

Sam Vilain

unread,
May 25, 2009, 9:14:44 PM5/25/09
to chromatic, perl5-...@perl.org, Nicholas Clark
chromatic wrote:
> My concrete suggestions are:
>
> 0) Release 5.10.1 and 5.11.0 concurrently
>
> 1) Require all feature work to occur in branches
>
> 2) Declare and enforce trunk stability (no more "bleadperl" from which someone
> cherrypicks stable commits)
>
> 3) Enable smoking of branches ready to merge to trunk
>
> 4) Require all merge-ready branches to include perldelta changes, when
> appropriate
>
> 5) Write a release manager guide
>
> 6) Recruit likely release managers for the next several releases
>
> 7) Set a release schedule tied to the calendar, not the state of any feature
>
> 8) Follow the release schedule
>
> I believe #2 is the most important change to unblock this process and am happy
> to write about it in more detail if anyone cares.
>
> Git enables 1, 2, and 3. 0, 1, 2, 4, 5, and 8 require active pumpking
> support. Is there interest?
>
> I'm still working on a support guide, but I'm sure there'll be plenty of
> debate about it.
>
> If I'm full of hot air because I spend most of my development time on Parrot
> these days (though we have no trouble there with monthly stable releases), or
> if none of my proposals are interesting or useful, I can stop talking about
> this, though if I'm not completely crazy I'd certainly like feedback on why
> these suggestions are impractical or infeasable.
>

Yeah. I certainly have to laud most of the suggestions. Numbers 7 and
8 are of course entirely arbitrary and pointless; it's the earlier
suggestions that add the value which is flexibility about when you
release. You still release when you're ready, it just happens more
often. People don't care when releases happen, as long as it's soon and
they're good quality. Employing a 'master', 'stable', etc system is
what I have been suggesting all along.

Chromatic parrot might have no problems making releases, but it's just
not as widely deployed as Perl 5. The userbase is much, much smaller.
So frankly your point that "this is easy for parrot" couldn't be more
irrelevant. With time-based releases you either compromise on features
or release quality. When I tried to contribute to get good metamodel
support I found that 'trunk' was anything but stable and there were huge
architectural hurdles. Not exactly a shining parable of good
development practices. Maybe things have changed in parrot, but I've
lost interest and I think many have lost patience for your arguments
about regular releases.

Sam.

Sam Vilain

unread,
May 25, 2009, 9:29:36 PM5/25/09
to Mark Mielke, Craig A. Berry, Jim Cromie, Perl 5 Porters
Mark Mielke wrote:
> Can you run the same report counting "churn" instead of "commits"? :-)
>
> The 1990 - 2000 years look highly suspect to me, since those are the
> days when lots of churn really occurred. Perl 5 was introduced. Two
> different threading models. Unicode support. Compiled Perl Support (even
> if it never worked perfectly). All sorts of portability work. The
> numbers pre 1997 don't jive with my recollection of history. Either the
> changes were huge - or there is history missing from your report. (Or,
> the changes were combined into larger changes before they made it into
> your history)

Yeah, well, I did the best I could to split the changes out into
individual patches... but you know, that history is pre-Perforce :-).
So it was really just munging whatever breadcrumbs were left behind in
the patchfiles.

Take the threading models. Well, that was I think 5.001m or something
right? Well that probably hit as a single patch which was integrated in
the early Perforce changes. Unicode support all went in much later.
Compiled Perl support wasn't many commits really.

Sure there are certainly gaps in the history - the first of the Perl 5.0
alphas in the history is a big changeset, for instance.

>> 1997 1290
>> 1996 790
>> 1995 1
>> 1996 129
>> 1995 1
>> 1996 11
>> 1995 2
>> 1996 16
>> 1995 1
>> 1996 3
>> 1995 1
>> 1996 12
>> 1995 43
>>

This output doesn't fit with my recollection of history, either - I only
remember 1995 happening once... ;-)

>> 1994 6
>> 1993 7
>> 1992 16
>> 1991 23
>> 1990 33
>> 1989 9
>> 1988 16

Sam.

David Golden

unread,
May 25, 2009, 9:38:13 PM5/25/09
to p5p
On Mon, May 25, 2009 at 9:14 PM, Sam Vilain <s...@vilain.net> wrote:
>> 7) Set a release schedule tied to the calendar, not the state of any feature
>>
>> 8) Follow the release schedule
>
> Yeah.  I certainly have to laud most of the suggestions.  Numbers 7 and
> 8 are of course entirely arbitrary and pointless; it's the earlier

I disagree, for a mixture of practical and psychological reasons

Establishing a habit around regular releases also means that the
amount of grotty housekeeping that needs to be done will be smaller
for each release. I've got to think it's not only easier, but less
intimidating to think of writing a perldelta for 3 months than a
perldelta for nearly 18 months (5.10.0 to 5.10.1) or almost 3 years
(5.8.8 to 5.8.9). There are plenty of real world examples for the
psychology of this (cleaning the bathroom/kitchen/garage, weeding the
garden,

I think it would also be easier to attract volunteers if they knew
their commitment was bounded. Release manager for 3 months instead of
pumpking for a "long time". It's easier to project forward available
time and energy on a shorter horizon.

Another benefit would be less pressure to sync dual-life modules
development with core development right before a release. It's not
like there would be only one shot every couple years to get a module
into the core, so there would be less of a last-minute churn to shove
in as many bug fixes at the last minute and fewer calls to hold up a
perl release for "just a couple more" dual-life module fixes.

More generally, for potential contributors, it would be more
psychologically rewarding to contribute if one knew that work would
manifest within one or two short release cycles.

I think this argument can be taken further, but Sam and I went a
couple rounds on IRC already a while back so I'll leave it at that.
Whether you agree or disagree with me how significant these things are
is up to you, but I wouldn't say that timeboxing is just "arbitrary
and pointless".

-- David

Sam Vilain

unread,
May 25, 2009, 9:41:12 PM5/25/09
to David Golden, p5p

Well, no, you wouldn't, but that's because you missed the point rather
than because you had a good reason.

Yes releases should be regular, but the timeboxing isn't what's
important - what's important is that the housekeeping is performed along
the way as Chromatic was suggesting. Using your analogy think of it as
"clean as you go".

Sam.

Chromatic

unread,
May 25, 2009, 10:06:30 PM5/25/09
to perl5-...@perl.org, Sam Vilain
On Monday 25 May 2009 18:14:44 Sam Vilain wrote:

> parrot might have no problems making releases, but it's just
> not as widely deployed as Perl 5. The userbase is much, much smaller.

That may affect how many users *upgrade* or how often they do but it has
nothing to do with making a release on any schedule. If anything, Perl 5's
greater userbase *should* mean that more people test Perl 5 release candidates
than Parrot releases.

Evidence suggests to discard that line of thinking.

> So frankly your point that "this is easy for parrot" couldn't be more
> irrelevant. With time-based releases you either compromise on features
> or release quality.

That is patently untrue. Do you believe that you can eat an elephant in one
bite? (Even if so, I suspect the quality of your digestion will suffer.)

> When I tried to contribute to get good metamodel
> support I found that 'trunk' was anything but stable and there were huge
> architectural hurdles. Not exactly a shining parable of good
> development practices.

The object system underwent a (scheduled) revision in September 2007. The
most recent related message I can find from you on the Parrot mailing list is
from July 2007.

I can (meander back to the topic at hand to) point you to grotty piece of
Parrot in May 2009, but you will have trouble making the argument that Parrot
has not improved dramatically between November 2006 and May 2009 (and I
believe many other committers will agree that our release policy deserves much
of the credit).

> Maybe things have changed in parrot, but I've
> lost interest and I think many have lost patience for your arguments
> about regular releases.

Then ignore Parrot and consider instead the Linux kernel, or Ubuntu GNU/Linux,
or KDE, or GNOME, or Wine, OpenBSD, or any of many other projects with regular
releases. All of these projects have roadmaps. You can't predict with 100%
certainty which features will be present in which release, but if you keep
trunk passing (which we do), refactor regularly (which we do), and release
regularly, users can create their own roadmaps regarding upgrades.

I'd like Perl 5 to have a roadmap.

-- c

David Golden

unread,
May 25, 2009, 10:20:49 PM5/25/09
to Sam Vilain, p5p
On Mon, May 25, 2009 at 9:41 PM, Sam Vilain <s...@vilain.net> wrote:
>> I think this argument can be taken further, but Sam and I went a
>> couple rounds on IRC already a while back so I'll leave it at that.
>> Whether you agree or disagree with me how significant these things are
>> is up to you, but I wouldn't say that timeboxing is just "arbitrary
>> and pointless".
>>
>
> Well, no, you wouldn't, but that's because you missed the point rather
> than because you had a good reason.

Your actual comment was "Numbers 7 and 8 are of course entirely
arbitrary and pointless". If you'd like to make an actual argument
instead of a dismissing them with "of couse", I'd take your point more
seriously. I get it. You disagree. I respect the fact that you have
a different opinion, but please do me the same courtesy.

> Yes releases should be regular, but the timeboxing isn't what's
> important - what's important is that the housekeeping is performed along
> the way as Chromatic was suggesting.  Using your analogy think of it as
> "clean as you go".

Since you made a point to make a point... :-) ... my response is that
I think timeboxing is what puts discipline around the regular
housekeeping. It's like having the parents or in-laws over for a
visit. It's harder to get lax around the housekeeping if there's a
regular expectation for when things become public.

Note -- I'm not saying Perl should have a "production-ready" release
on a timeboxed basis -- but that it might benefit from a "public
release" meeting certain standards on a time-boxed basis. (5.10.2.1
.. 5.10.2.N .. and so on). Call them "public snapshots" or
"development snapshots" or whatever makes sense.

-- David

Sam Vilain

unread,
May 25, 2009, 11:11:58 PM5/25/09
to chromatic, perl5-...@perl.org
chromatic wrote:
>> With time-based releases you either compromise on features
>> or release quality.
>>
>
> That is patently untrue. Do you believe that you can eat an elephant in one
> bite? (Even if so, I suspect the quality of your digestion will suffer.)
>

I ... just don't understand what your point is here at all. All I was
trying to say is that when the deadline looms you either drop unfinished
features or make a poor quality release. Why is that "patently untrue"?

>> Maybe things have changed in parrot, but I've
>> lost interest and I think many have lost patience for your arguments
>> about regular releases.
>>
>
> Then ignore Parrot and consider instead the Linux kernel, or Ubuntu GNU/Linux,
> or KDE, or GNOME, or Wine, OpenBSD, or any of many other projects with regular
> releases. All of these projects have roadmaps.

Ok ... but Ubuntu is the only one AFAIK which has a regular, timeboxed
release policy out of that bunch.

> You can't predict with 100%
> certainty which features will be present in which release, but if you keep
> trunk passing (which we do), refactor regularly (which we do), and release
> regularly, users can create their own roadmaps regarding upgrades.
>
> I'd like Perl 5 to have a roadmap.
>

I agree on that, I just think that having a vague release schedule, eg
aiming for a maint release about every three months and a major one
every year or two, is just as good as a fixed one where you release on
the first of every month or whatever. I prefer releases that are
slightly irregular, it makes it feel like the release is less of a
"Zombie release" which is just bundling whatever is happening now,
regardless of whether that was the next logical step forward or whatever...

Sam

Craig A. Berry

unread,
May 25, 2009, 11:30:16 PM5/25/09
to Sam Vilain, Mark Mielke, Jim Cromie, Perl 5 Porters
On Mon, May 25, 2009 at 8:29 PM, Sam Vilain <s...@vilain.net> wrote:
> Mark Mielke wrote:

>> The 1990 - 2000 years look highly suspect to me, since those are the
>> days when lots of churn really occurred.

I confess I didn't even look at the numbers from years prior to 2000
before posting and really shouldn't have included them. The data are
clearly wrong and just distract people from what's been happening in
our current decade.

> Yeah, well, I did the best I could to split the changes out into
> individual patches... but you know, that history is pre-Perforce :-).
> So it was really just munging whatever breadcrumbs were left behind in
> the patchfiles.

> This output doesn't fit with my recollection of history, either - I only


> remember 1995 happening once... ;-)

You would know better than I why git log thinks some parts of 1995
happened after some parts of 1996, but undoubtedly the breadcrumbs
were just too scarce and we shouldn't be expecting useful detail of
the sort I posted from the pre-Perforce years.

Chromatic

unread,
May 25, 2009, 11:52:28 PM5/25/09
to Sam Vilain, perl5-...@perl.org
On Monday 25 May 2009 20:11:58 Sam Vilain wrote:

> I ... just don't understand what your point is here at all. All I was
> trying to say is that when the deadline looms you either drop unfinished
> features or make a poor quality release. Why is that "patently untrue"?

Because timeboxed releases make one claim about features (they are done or
they are not) and one claim about quality (trunk quality is always sufficient
for release).

I suppose it's possible to make a poor quality release if you don't have an
automated testing suite (or you do and you ignore it), but I don't believe
projects can make regular timeboxed releases without taking testing seriously.

> > Then ignore Parrot and consider instead the Linux kernel, or Ubuntu
> > GNU/Linux, or KDE, or GNOME, or Wine, OpenBSD, or any of many other
> > projects with regular releases. All of these projects have roadmaps.
> Ok ... but Ubuntu is the only one AFAIK which has a regular, timeboxed
> release policy out of that bunch.

They all have a regular, timeboxed release policy. They may not all predict
it to the day, but they all have a regular timeboxed release policy. (Wine in
particular has *monthly* releases.)

> I agree on that, I just think that having a vague release schedule, eg
> aiming for a maint release about every three months and a major one
> every year or two, is just as good as a fixed one where you release on
> the first of every month or whatever.

The nice part of the calendar-based releases is that it divorces features
landing on trunk from release dates. You remove debates over whether the
release is ready. There are no discussions or arguments over when the release
is. You do not run the risk of slipping the release. You get into a rhythm
of risk versus stability on trunk. (There's a reason we merge branches in the
first couple of weeks after a release and spend three or four days before the
release concentrating on bugfixes.)

The nice part of short interval calendar-based releases is that it doesn't
matter if a feature isn't quite done, because the next release is only a month
away.

> I prefer releases that are
> slightly irregular, it makes it feel like the release is less of a
> "Zombie release" which is just bundling whatever is happening now,
> regardless of whether that was the next logical step forward or whatever...

Why would you do that? Features are done or they aren't. If you want to keep
your trunk stable, you merge only done features:

http://jamesshore.com/Agile-Book/done_done.html

-- c

Sam Vilain

unread,
May 26, 2009, 12:39:36 AM5/26/09
to chromatic, perl5-...@perl.org
On Mon, 2009-05-25 at 20:52 -0700, chromatic wrote:
> > > Then ignore Parrot and consider instead the Linux kernel, or Ubuntu
> > > GNU/Linux, or KDE, or GNOME, or Wine, OpenBSD, or any of many other
> > > projects with regular releases. All of these projects have roadmaps.
> > Ok ... but Ubuntu is the only one AFAIK which has a regular, timeboxed
> > release policy out of that bunch.
>
> They all have a regular, timeboxed release policy. They may not all predict
> it to the day, but they all have a regular timeboxed release policy. (Wine in
> particular has *monthly* releases.)

Well, not to say you're wrong but that did surprise me so I thought I'd
check your claim out.

First, Linux I don't think has such a schedule. Linus tries to make a
-rc release every two weeks and lets each point release cook for about 8
-rc releases. The best I could find as to a roadmap was the "Linux
Weather Forecast". http://www.linuxfoundation.org/collaborate/lwf

| Short-term forecast: the 2.6.30 release will probably happen sometime
| in June, 2009.

KDE had one specifying dates very far into the future, so your point is
true for KDE. But Gnome has no similar road map.

But look this is a tired thread and I think we actually agree on the
important points here about having a roadmap, and using feature branches
which are only merged to master/trunk/whatever when they are *ready*,
including everything needed to make them releasable.

Sam.

Mark Mielke

unread,
May 26, 2009, 2:04:06 AM5/26/09
to Sam Vilain, chromatic, perl5-...@perl.org
Sam Vilain wrote:
> chromatic wrote:
>
>>> With time-based releases you either compromise on features
>>> or release quality.
>>>
>> That is patently untrue. Do you believe that you can eat an elephant in one
>> bite? (Even if so, I suspect the quality of your digestion will suffer.)
>>
>
> I ... just don't understand what your point is here at all. All I was
> trying to say is that when the deadline looms you either drop unfinished
> features or make a poor quality release. Why is that "patently untrue"?
>

That's not the only possible outcomes, though. Another possible outcome
is that the true high priority items bubble to the surface, and get
addressed for end users, without holding up releases for features that
are really a sort of vaporware anyways. Committers who want to see their
changes incorporated need to either put up or shut up, and stop holding
everybody else up. :-)

We've been jumping on the "Agile" band wagon at work. It focuses on
small *complete* changes that can be delivered to "trunk" leaving the
product in a reliable state, but delivering increments in features over
time. "Releases" could be as a frequent as every two weeks since the
product is kept in a "ready to deliver to the customer" state at all
times. There are demonstrations of this out there - Green Hopper
releases a new release every 3 or 4 weeks, and each increment delivers
new functionality that we (the customers in this case) have requested.
The feedback to deployment of change cycle is much shorter, which is
where the "Agile" comes from. It is a more customer responsive model.
Not that it always works - but just to point out that people are working
this way. Content is committed to at the beginning of the "sprint", and
it is the responsibility of the team to complete the full commitment
before the sprint ends. Time boxing in this sense is about focus, not
about poor quality or functionality dropped at the end. Decide what is
really important to the customer. Spend a focused amount of time on
that. Deliver results.

Rafael Garcia-Suarez

unread,
May 26, 2009, 2:14:25 AM5/26/09
to chromatic, perl5-...@perl.org, Nicholas Clark
2009/5/26 chromatic <chro...@wgz.org>:

> Do I recall correctly that this process stalled when someone suggested "It's
> just too hard to generate a perldelta, so thanks, but no thanks!"  (Once upon
> a time a pumpking decided that patches without test cases and documentation
> changes were unacceptable, so it seems like a small leap to require delta
> patches too.)

With git, we now more or less require commit messages too.
(git format-patch is preferred over git diff.) A good commit message is
often the basis for a perldelta entry.

> As long as producing a release relies on someone spending a day a week
> cherrypicking patches from an unstable branch into a stable branch, I believe
> that no one who knows how to make a release will ever have time to write up
> what it takes to make a release (let alone what it will take to make a release
> parallelizable and trivial), and no one who doesn't already know how to make a
> release will learn how to make a release.  Cart, egg, horse, chicken.
>
> My concrete suggestions are:
>
> 0) Release 5.10.1 and 5.11.0 concurrently

Very good idea.

> 1) Require all feature work to occur in branches

Good idea -- but at some point you'll have to merge in trunk, and that's
usually where new features get more broadly tested, and where bugs are
discovered. (Just look at my recent smartmatch branch -- the few
recent changes were made in trunk, after the merge.)

> 2) Declare and enforce trunk stability (no more "bleadperl" from which someone
> cherrypicks stable commits)

I don't think that's a good idea. Nicholas already explained that at
length.

However the situation can be improved -- for example bug fixes could go
in a branch off the 5.10 release point, and that branch be merged in
maint-5.10 and trunk afterwards.

Also, I think there is a possibility to write a tool to help in the
cherry-picking -- at least to distribute the knowledge of what has been
already reviewed for inclusion in maint. There are many solutions.

> 3) Enable smoking of branches ready to merge to trunk

That would go with 1.

> 4) Require all merge-ready branches to include perldelta changes, when
> appropriate
>
> 5) Write a release manager guide

Good idea, embryos exist in Porting/, they need to be updated a bit I
think.

> 6) Recruit likely release managers for the next several releases

Volunteers anyone ?

> 7) Set a release schedule tied to the calendar, not the state of any feature
>
> 8) Follow the release schedule

I shall be clear here: those two points are completely impossible and
unrealistic -- at least as long as no-one's time is bought to make the
pumpking job. You just can't make previsions based on the availability
of a resource you don't control -- here, the time.

--
The truth is that we live out our lives putting off all that can be put off;
perhaps we all know deep down that we are immortal and that sooner or later
all men will do and know all things. -- Borges

Rafael Garcia-Suarez

unread,
May 26, 2009, 2:23:07 AM5/26/09
to Mark Mielke, Sam Vilain, chromatic, perl5-...@perl.org
2009/5/26 Mark Mielke <ma...@mark.mielke.cc>:

There is a very important point here that you are missing.

A programming langage is not a mere collection of features. At any stable point
in time, it should be internally consistent. For example, in 5.10, having "my
$_" without the (_) prototype would have been inconsistent -- it would have
created situations where the language would not have been satisfactory.
However, those are two different features. The same remark can be done
about Unicode improvements, or regex work, etc.

--
A system is nothing more than the subordination of all aspects of
the universe to any one such aspect.
-- Borges

Nicholas Clark

unread,
May 26, 2009, 11:31:35 AM5/26/09
to Craig A. Berry, Mark Mielke, Jim Cromie, Perl 5 Porters
On Mon, May 25, 2009 at 07:52:26PM -0500, Craig A. Berry wrote:
> On Mon, May 25, 2009 at 6:58 PM, Mark Mielke <ma...@mark.mielke.cc> wrote:
> > I suspect if you trended commits to Perl 5 since 2000, it is probably pretty
> > consistent and dropping throughout.
>
> Don't suspect. Count. 2001 was certainly a bumper crop (what is *in*
> that Finnish coffee?), but there is no general downward trend in

I do /not/ know.

> recent years (2007 has almost twice as many commits as 2003, for
> example). Following shows commits per year, with 2009 looking like it

I think you mean than 2004.

> will be a lot bigger than 2008 by the time the year is done.

I don't disagree with that statement

> % git log --pretty=fuller | grep CommitDate: | perl -ane '$m=$F[5];
> $c++; if ($pm && (eof || $pm ne $m)) {print qq/$pm $c\n/; $c=0;} $pm =
> $m;'
> 2009 1036
> 2008 1675
> 2007 2377
> 2006 2700
> 2005 2393
> 2004 1175
> 2003 2555
> 2002 3950
> 2001 5040
> 2000 3345
> 1999 1904
> 1998 1755
> 1997 1290

Extrapolate 2009 up, and I think it will close to 2007 and 2005, more than
2008 and 2004, and less than every other year post 1999. There are high
years and low years. I'm not sure whether it's fair to group the time
pre-5.8.1 (when Jarkko was insanely active) with post that time.

However, "number of commits" isn't that useful if the average commit size
changes because of the different version control system. For example, we've
had someone (I forget the name, sorry) recently send 1 e-mail containing 13
or so documentation patches to one file, rather than 1 large patch.

This is good - it's 13 times as much granularity for future "blame" analysis.
But if everyone does this, than numbers of commits will not be the whole
story, and the absolute number of commits will need to increase for the same
level of change.

(Then again, level of change is actually irrelevant. Level of progress is
what matters. And right now, another measure of progress, the size of the
bug database, is in reverse)

Nicholas Clark

Nicholas Clark

unread,
May 26, 2009, 11:47:37 AM5/26/09
to Craig A. Berry, Jim Cromie, Perl 5 Porters
On Mon, May 25, 2009 at 05:59:50PM -0500, Craig A. Berry wrote:
> On Mon, May 25, 2009 at 10:31 AM, Nicholas Clark <ni...@ccl4.org> wrote:

> I think a better measure than trends in the top 4 committers would be
> whether everybody else has done more relative to the top 4, or whether
> the overall amount of work getting done has changed.

True, but

a: The graph I generated was easy (well, easier.)
b: 4 was arbitrary. (I hit the URL size limit of the Google chart API)

> > (From which I observe that my spikes aren't as big as Jarkko's were)
>
> Yeah, but the area under the curve is important too :-).

His area is larger too.

http://www.ohloh.net/p/perl/contributors


> The number of committers, on the other hand, has roughly doubled:
>
> % grep "Commit:" fiveten2git.txt | sort -u | wc -l
> 12
> % grep "Commit:" git2now.txt | sort -u | wc -l
> 27
>
> There are some duplicates and noise in here, as well, but the
> following people have pushed commits since the switch to git but had
> not done so during the final year of Perforce:
>
> Abigail
> Ben Morrow
> Florian Ragwitz
> Jan Dubois
> Leon Brocard
> Steffen Mueller
> Vincent Pit
> Yitzchak Scott-Thoennes
> Yuval Kogman
>
> That looks like pure good news to me.

That's actually an artefact of how git works.

I believe that of that list, Ben Morrow, Florian Ragwitz, and Yuval Kogman
have not pushed commits direct, but have committed to their own repository,
that someone (who does have direct commit access) has then has merged.

I'm not convinced that, efficiency-wise, that's any more (or less) work than
applying a well-formed git-formatted patch delivered via e-mail. (In that
the real work is in the level of code review it gets)

I'm not sure about Yitzchak Scott-Thoennes - he has an account on dromedary.


> But more commits doesn't automatically lead to more releases, and I
> don't think git ever had the potential to do more than open the doors
> to the workshop a bit wider. That's a good thing, but obviously not
> the only thing.

Agree.

Nicholas Clark

Nicholas Clark

unread,
May 26, 2009, 12:10:32 PM5/26/09
to David Golden, p5p
On Mon, May 25, 2009 at 09:38:13PM -0400, David Golden wrote:

> Another benefit would be less pressure to sync dual-life modules
> development with core development right before a release. It's not
> like there would be only one shot every couple years to get a module
> into the core, so there would be less of a last-minute churn to shove
> in as many bug fixes at the last minute and fewer calls to hold up a
> perl release for "just a couple more" dual-life module fixes.

To the best of my memory I never held up a release for anything other than
critical bugs (specifically CVEs and behaviour regressions). I didn't
always ship the latest version, and my view (which may not have been
explicitly stated) was that it *wasn't my problem* to get anything into
blead. I would only cherry pick what was already there - if it wasn't there -
too bad.

As to syncing dual-life modules - I want the pesky things to be in sync
always. It's annoying that the only time there ends up with a real drive
to sort the mess out is before a release, but I'd prefer to solve the
underlying problem generally, rather than decide that the only way to
effect a solution to it is to couple it with having more releases. Hence
why I've been working on re-arranging ext/, and intend to move as much as
possible from lib/ to ext/, so that the diff process is trivial (and can
be automated - we could establish a daily report on the state of sin)

We didn't actually get this "right" for (my) view of right in 5.10.0 - for
example, it shipped with a completely unreleased Module::Build, and I
believe that it also had a lot of other divergence from CPAN. I believe
that the "pressure" here is because Dave is trying to maintain the same
standards that I aimed at.


Before I forget: CVEs are not fun.

Nicholas Clark

Nicholas Clark

unread,
May 26, 2009, 3:34:50 PM5/26/09
to Rafael Garcia-Suarez, chromatic, perl5-...@perl.org
On Tue, May 26, 2009 at 08:14:25AM +0200, Rafael Garcia-Suarez wrote:
> 2009/5/26 chromatic <chro...@wgz.org>:
> > Do I recall correctly that this process stalled when someone suggested "It's
> > just too hard to generate a perldelta, so thanks, but no thanks!" �(Once upon
> > a time a pumpking decided that patches without test cases and documentation
> > changes were unacceptable, so it seems like a small leap to require delta
> > patches too.)
>
> With git, we now more or less require commit messages too.
> (git format-patch is preferred over git diff.) A good commit message is
> often the basis for a perldelta entry.

True

But not all commits are perldelta entries. (Many need to be edited out)
And some perldelta entries are 3 or 4 fold expansions on the commit message,
and there's nothing wrong with the commit message, because it's terse, to the
point, but assumes a lot of context which then needs to be written out
explicitly in perldelta.

> > As long as producing a release relies on someone spending a day a week
> > cherrypicking patches from an unstable branch into a stable branch, I believe
> > that no one who knows how to make a release will ever have time to write up
> > what it takes to make a release (let alone what it will take to make a release
> > parallelizable and trivial), and no one who doesn't already know how to make a
> > release will learn how to make a release. �Cart, egg, horse, chicken.

Having a binary compatible branch, and a "free-for-all" branch makes
maintaining binary compatibility much easier. For some value of "easier"
probably actually spelled "possible at all", given that it becomes possible
to review patches after the event to work out whether they are compatible
with the stability goals of the maintenance branch.

> > My concrete suggestions are:
> >
> > 0) Release 5.10.1 and 5.11.0 concurrently
>
> Very good idea.

Heck, what's wrong with releasing 5.11.0 *before* then?

Several people have said "what's stopping us?" and really, for blead, all it
is is "write the perldelta". At which point they evaporate.

> > 3) Enable smoking of branches ready to merge to trunk
>
> That would go with 1.

re-ordered, because I think that that's a prerequisite of 1

> > 1) Require all feature work to occur in branches
>
> Good idea -- but at some point you'll have to merge in trunk, and that's
> usually where new features get more broadly tested, and where bugs are
> discovered. (Just look at my recent smartmatch branch -- the few
> recent changes were made in trunk, after the merge.)
>
> > 2) Declare and enforce trunk stability (no more "bleadperl" from which someone
> > cherrypicks stable commits)
>
> I don't think that's a good idea. Nicholas already explained that at
> length.
>
> However the situation can be improved -- for example bug fixes could go
> in a branch off the 5.10 release point, and that branch be merged in
> maint-5.10 and trunk afterwards.

I'm not sure on that one. We have had things that are bug fixes where we're
not sure of the stability issues for a while. Or where they've caused subtle
bugs. (There's one I'm thinking of involving resizing the stack before doing
a require, which seemed like the "right" thing, but caused C undefined
behaviour elsewhere when an exception was thrown:

http://www.xray.mpe.mpg.de/mailing-lists/perl5-porters/2005-11/msg00511.html

)

> Also, I think there is a possibility to write a tool to help in the
> cherry-picking -- at least to distribute the knowledge of what has been
> already reviewed for inclusion in maint. There are many solutions.

Seems useful.


> > 5) Write a release manager guide
>
> Good idea, embryos exist in Porting/, they need to be updated a bit I
> think.
>
> > 6) Recruit likely release managers for the next several releases
>
> Volunteers anyone ?

This works for blead. I'm not convinced that it solves the real bottlenecks
for any "stable" release, be it 5.10.x or 5.12.0. Specifically, the analogous
Parrot document says

A couple of weeks in advance: Ask people to run C<make fulltest> and
report (and hopefully fix!) any problems they find. Check in with
language project leads for release blockers, to allow time to fix them.

( https://svn.parrot.org/parrot/trunk/docs/project/release_manager_guide.pod )


I think that for a stable release the question "is it ready" is more like:

Does CPAN pass?

which comes down to:

for each module that fails its regression tests on $current
did it fail identically on $previous?
if yes, "SEP"
else work out why it failed (a bisect is useful for this)

attempt to group failure causes

for each failure cause
is that a regression?
if yes, figure out how to fix it
(more code? revert the code that broke it)
else
(presumably) it's relying on something un-or-under-documented
should the existing behaviour stay?
yes - goto "regression"
no - note it in perldelta as a significant bugfix
(also, try to inform the module's author)


and the decisions involved in that are never going to be scriptable.

You wrote:

: Several people who volunteered last year to manage a releases from a checklist

: of tasks they could work through to produce a release. What happened? What
: happened to that checklist? Was it a bad idea? Did no one have time to write
: it? Did the volunteers all say "No, thank you!" privately?

and my view is that the above set of steps are

1: What sets Perl apart from certain less well managed projects

(eg PHP, where IIRC 5.0.8 broke something to do with arrays, and the final
conclusion of the PHP maintainers was "oh, well, it stays. fix your
scripts")
(eg Ruby, where 1.9.0 broke Rails. I infer that the Ruby maintainers don't
test prominent Ruby code to be aware of the problem, but worse, the Rails
maintainers don't bother to test upcoming Ruby releases)

2: Not a set of tasks that can be "worked through"

3: Not the work that really needs spreading out

[people volunteered in error after Gabor misread the message I wrote about what
*did* need doing, and wrongly assumed that it meant a scriptable task list]


The set of tasks that can be "scripted" for Perl 5, analogous to steps 2-11 in
https://svn.parrot.org/parrot/trunk/docs/project/release_manager_guide.pod
for 5.8.9 at least were roughly:

0: so you think you have a source control in a state that won't break CPAN,
at least not in "surprising" ways.

1: As there are no regular smokes (yet - please fix?) find out about the state
of VMS. If it's bad, think again.

2: Re-read the perldelta to try to find any embarrassing typos

3: Run Porting/makemeta

4: [used to be run autodoc.pl, but I eliminated that]

5: [used to be run pod/buildtoc, but I eliminated that]

6: update module corelist, but we need to fix that
[it has been holding perforce revisions for releases, but we can't know
hashes in advance for git. We need to agree a plan to move to git tags]

7: [update changes, but Dave has eliminated that]

8: update patchlevel.h to remove all local patches

9: make tarball with Porting/makerel

10: copy tarball to some other machine x 2 [or more - IRC is good for this]

11: check that ./Configure -des && make all test works in one place

12: check that ./Configure ... && make all test_harness install works
[that's likely something that needs fixing in the Parrot checklist -
there's no step to check that it installs, that the installed parrot runs,
or that the installed parrot can be used to build a third-party language
such as Rakudo or Pynie]


for 5.8.x I did this:

13: bootstrap the CPAN client on the clean install

14: install CPANPLUS

15: bootstrap the CPANPLUS client

16: install an XS module

17: if this is good, commit this.
sit, and wait.

18: do the smoke tests pass (particularly Win32)

19: if yes, upload it to PAUSE. This is the point of no return

20: mail p5p to announce it, with a quote I prepared earlier

21: wait 24 hours or so

22: post the announcement to use.perl

And, I repeat, doing the above is actually *quite fun*, after all the stress
and hard work of dealing with the issues of "is it releasable?" The above
steps appear to be analogous to the steps in the Parrot release guide.

So, I was never looking for anyone to do that for me.

> > 7) Set a release schedule tied to the calendar, not the state of any feature
> >
> > 8) Follow the release schedule
>
> I shall be clear here: those two points are completely impossible and
> unrealistic -- at least as long as no-one's time is bought to make the
> pumpking job. You just can't make previsions based on the availability
> of a resource you don't control -- here, the time.

In particular, I can't see how 7 and 8 work given either

1: it is discovered that trunk contains a regression from the previous stable
release
2: it doesn't look like a bug fix will arrive before the release date

or worse

1: it is discovered that a bug fix added in the previous release *also*
introduces a regression
2: it doesn't look like a bug fix will arrive before the release date


The choices are
a: reverting the change that fixes the bug? [introduces a second regression]
b: do nothing [keeps a regression]
c: defer the release until quality is restored.


My view was always that it was more important to avoid creating regressions,
than to fix (non security) bugs. Better the bugs you know. Specifically,
because modules on CPAN *have* to work round existing bugs, whatever you do
in a new release. So keeping the same bugs doesn't increase the workload on
CPAN authors.

Whereas fixing one bug but introducing another

1: increases the complexity for CPAN authors
2: provides a disinsentive for end users to upgrade from their current version.
If they know that the next version, meant to be compatible, isn't quite,
why should they risk upgrading?

It's not good having a reputation for shipping slightly buggy releases.
Because if you do you end up with more spread of installed versions than you
would otherwise, and you have a harder time getting people to do the right
thing ["upgrade to fix this bug", versus "suffer it and hack round it" or
"we want the old version, with just the bug fixes".

This happens. An ActiveState employee has told me that firms want
"DBI version $this, and then just the bug fixes from the later versions".
I'm aware that firms wanting to install Linux want (say) 2.4.18 plus all the
bug fixes. But it has to be 2.4.18. (Consultant's solution was to take the
current version sources, and change the reported version number, because it
was actually the best solution for the customer's real need.)]


I think that 5.8.9 was generally successful in the "don't introduce regressions"
stakes. I believe that there are only two known so far:

1: suidperl doesn't work. no-one who needed it tested it. They get what they
deserve
2: there's a regression with assigning format references

Nicholas Clark

H.Merijn Brand

unread,
May 26, 2009, 3:51:23 PM5/26/09
to perl5-...@perl.org
On Tue, 26 May 2009 20:34:50 +0100, Nicholas Clark <ni...@ccl4.org>
wrote:

> I think that 5.8.9 was generally successful in the "don't introduce regressions"
> stakes. I believe that there are only two known so far:
>
> 1: suidperl doesn't work. no-one who needed it tested it. They get what they
> deserve
> 2: there's a regression with assigning format references

I'm the only one that uses that in production code (with a lot of
customers), and I decided to stick to 5.8.8 hoping 5.10.1 would be the
next best thing.

5.10.x is now on two boxes, where I try to run all non-public customer
code, which is why I found that regression. I know it will be in
5.10.1, so I am a happy camper again.

--
H.Merijn Brand http://tux.nl Perl Monger http://amsterdam.pm.org/
using & porting perl 5.6.2, 5.8.x, 5.10.x, 5.11.x on HP-UX 10.20, 11.00,
11.11, 11.23, and 11.31, OpenSuSE 10.3, 11.0, and 11.1, AIX 5.2 and 5.3.
http://mirrors.develooper.com/hpux/ http://www.test-smoke.org/
http://qa.perl.org http://www.goldmark.org/jeff/stupid-disclaimers/

Nicholas Clark

unread,
May 26, 2009, 3:54:56 PM5/26/09
to H.Merijn Brand, perl5-...@perl.org
On Tue, May 26, 2009 at 09:51:23PM +0200, H.Merijn Brand wrote:
> On Tue, 26 May 2009 20:34:50 +0100, Nicholas Clark <ni...@ccl4.org>
> wrote:
>
> > I think that 5.8.9 was generally successful in the "don't introduce regressions"
> > stakes. I believe that there are only two known so far:
> >
> > 1: suidperl doesn't work. no-one who needed it tested it. They get what they
> > deserve
> > 2: there's a regression with assigning format references
>
> I'm the only one that uses that in production code (with a lot of
> customers), and I decided to stick to 5.8.8 hoping 5.10.1 would be the
> next best thing.

Aha. but that also means that you were too busy to test your production code
with 5.8.9-RCanything. :-(

Whilst both regressions were actually caused by changes I made whilst
refactoring, I'm actually more miffed about the format bug, because the looking
at the change with hindsight, the diff is small, and it's obvious that it's
wrong.

> 5.10.x is now on two boxes, where I try to run all non-public customer
> code, which is why I found that regression. I know it will be in
> 5.10.1, so I am a happy camper again.

Nicholas Clark

Jim Cromie

unread,
May 26, 2009, 3:53:25 PM5/26/09
to Craig A. Berry, Mark Mielke, Jim Cromie, Perl 5 Porters
this is an interesting thread, though I think it has veered
into forensics / epidimiology - picking over old corpses to find
evidence of (dropping the 'guilt' metaphor) errors or
failings, or signs of impending doom (Mayan calendar jokes encouraged)

I think the switch to git was an enormously wise move.
Any delay in uptake you are perceiving is just us chewing..

<parable>
Killing the mammoth(s) (p4=>git transition, several releases)
was a giant task ('we' took several down in this hunt),
done by brave and gifted hunters, and so the tribe will survive.

Maybe this is only for the winter, but for now, theres plenty to eat.
And we have: DaveG is using git, and wouldnt have bothered otherwise,
I myself have started using git (obviously, given the OP github patchset)
</>

This OP was about getting old boring cleanup work done;
I'll note that Ive tried multiple times before on this very patchset,
and Im uninterested in proving it or blaming anyone for not noticing.
But it is a boring patch, and reworking it creates ennui, and I cant
help but want it done ( or 'done and dusted' )

Hedging on that sentiment, git will apparently lower the maintenance of
forward porting (rebasing) it when the mad-galactic-clockwork permits.
Unfortunately I have no astrologer.

Damn, thats got a distinct tone=nasal to it.
Nobody owes me a review.

And I note ruefully that I could be said to have left that OP unfinished.
Thats where the ennui struck - whether to attach multiple patches
(which is ultimately easier for me), or whether that detail
is just enough hassle to keep anyone from looking.

So the github submission was meant as a probe to see what would happen,
vaguely like a child pulling the head off a locust, with some devilish glee,
but with more variance in expected outcome (stops for a chuckle :-)

I didnt really think the process was there, yet, except insofar as
one major branch has been merged ( smartmatch ) and no doubt
RGS has some lessons (scars? does the fix in mainline count ?)
that I'll certainly allow him to assimilate before explaining to the
rest of us.;-)

<hunt metaphor>
Digestion takes time(, and the product isnt always pretty ;-)
</>

my $.02 - we should talk about:

3) Enable smoking of branches ready to merge to trunk

chromatic was right to give this its own number,
ISTM this is how to automate much (AMAP) of the 'overhead'
work needed to consider a patchset.

We're not going to magically create more committers time,
(or committers), except by eliminating skut-work,
like the cat-herding that is CPAN<-->blead<-->releases sync.

And we ought not limit it to 'ready to merge' either.

I should be able to push a topic-branch,
forex: jimc/acme-win32/some-dumb-idea

somewhere where it gets a single test-run thru a windows platform,
and I get to see whats broken,
without asking any favors (or even asking, thats the point)

Now I know next-to-nothing about windows, and that seems
to work out here - I know personally that Steve Hay has
repaired some of my dreck in the past, with nary a whimper (thanks)
And Nick too, who said something intimating he knew who dunnit,
but rather what was wrong - no -U<unremembered> build before submit/commit
(something about unbuilt extensions - B particularly)

Which ties nicely back to the benefits of (3;-)
Both the fixes above are to my optree regression test work.

Tomorrow, loosely, it would be possible that the work could
have percolated in a branch, and those 2 fixes found there.
iirc, it took well over a year for either of those 2 bugs to manifest.

How would we do now ?
How will we do a year from now ? THIS IS THE ONE THAT MATTERS

( he asks, fingering his Laziness, False-Impatience cards throughout
final edit )

How do we get 5-10x perf improvement from Test::Smoke ?

strawman:

use -Dmksymlinks in each of ~27 smoke-config dirs,
named by a decent mangling of the build-options
(default remove ws, fancy chars, prefix with BLD)
this is roughly test-smoke configuring..

AND PRESERVE THEM - keep reusing the build dirs,
even as the source tree is git pulled *
(of course, we'll expose branches here later)

This will give us back some compile-time (and fs overhead too)
for at least some of the changes (those not touching *.h, embed.fnc, etc)

Thats no 5x, but it might be 50% for enough cases to matter.

We will find breakages:

- symlink errors, as files disappear under target links.
these are fixable by hand, probably automatically detectable
after error, and if handled by deleting and replacing directory,
are a functional replacement for current method (always delete target)
Id posit that more subtle fixes are possible.

- incidentally, I think 'make regen test' is now causing a full
recompile before 'make test' part runs, I noticed this a few days ago,
but dwelled on whether/how to find this with git bisect.
make regen is certainly not giving a reason for the remake,
I didnt look for a sneaky touched file..

- false positives/negatives
to know what they might be, we try, and fix by class (like symlink errs).
to HANDLE-IT, delete and remake target should work, provided we
can differentiate the errors from other make failures, and remove them
from reporting.


time to post, see what happens :-D
and Im clearly falling behind..

H.Merijn Brand

unread,
May 26, 2009, 4:02:26 PM5/26/09
to Nicholas Clark, perl5-...@perl.org
On Tue, 26 May 2009 20:54:56 +0100, Nicholas Clark <ni...@ccl4.org>
wrote:

> On Tue, May 26, 2009 at 09:51:23PM +0200, H.Merijn Brand wrote:
> > On Tue, 26 May 2009 20:34:50 +0100, Nicholas Clark <ni...@ccl4.org>
> > wrote:
> >
> > > I think that 5.8.9 was generally successful in the "don't introduce regressions"
> > > stakes. I believe that there are only two known so far:
> > >
> > > 1: suidperl doesn't work. no-one who needed it tested it. They get what they
> > > deserve
> > > 2: there's a regression with assigning format references
> >
> > I'm the only one that uses that in production code (with a lot of
> > customers), and I decided to stick to 5.8.8 hoping 5.10.1 would be the
> > next best thing.
>
> Aha. but that also means that you were too busy to test your production code
> with 5.8.9-RCanything. :-(

Correct, and it was a decision taken

> Whilst both regressions were actually caused by changes I made whilst
> refactoring, I'm actually more miffed about the format bug, because the looking
> at the change with hindsight, the diff is small, and it's obvious that it's
> wrong.

We're all human. I don't blame anyone.
I'm capable enough to rebuild a perl from source with patch to fix that
if fire starts spreading.

FWIW I test my CPAN modules with 10 perls before sending them to CPAN,
including maint and blead

> > 5.10.x is now on two boxes, where I try to run all non-public customer
> > code, which is why I found that regression. I know it will be in
> > 5.10.1, so I am a happy camper again.

--

David Golden

unread,
May 26, 2009, 4:16:29 PM5/26/09
to perl5-...@perl.org
On Tue, May 26, 2009 at 3:34 PM, Nicholas Clark <ni...@ccl4.org> wrote:
> I'm not sure on that one. We have had things that are bug fixes where we're
> not sure of the stability issues for a while. Or where they've caused subtle
> bugs. (There's one I'm thinking of involving resizing the stack before doing
> a require, which seemed like the "right" thing, but caused C undefined
> behaviour elsewhere when an exception was thrown:

In reading this, my mind immediately jumped to thinking in terms of
"unstable, testing, stable".

From what I understand of the development process in Perl, it's not
clear that blead/maint map cleanly to those kinds of categories.
Blead is generally "unstable", and maint is generally "testing", but
maint is not strictly the "testing" equivalent of "blead". "Unstable"
maint is what's in the maint pumpking's local repo/scratch and
"testing" blead is "last blead commit that passed tests".

Here's the conundrum: more explicit unstable/testing/stable branches
would help isolate release-ready changes and bug-fixes that need a
shakedown, but at the cost of more volunteer merge work.

Side note -- on the perldelta vs commit message issue, maybe a simple
convention of putting the text "(perldelta)" in the commit message for
significant commits would make it easier to grep commits for things to
filter/expand to perldelta at release-time.

-- David

Sam Vilain

unread,
May 26, 2009, 7:12:24 PM5/26/09
to Craig A. Berry, Perl 5 Porters
Craig A. Berry wrote:
>> This output doesn't fit with my recollection of history, either - I only
>> remember 1995 happening once... ;-)
>>
> You would know better than I why git log thinks some parts of 1995
> happened after some parts of 1996, but undoubtedly the breadcrumbs
> were just too scarce and we shouldn't be expecting useful detail of
> the sort I posted from the pre-Perforce years.
>

Well, with git log, topology trumps chronology. If a commit is a child
commit of a commit with a later date then 'git log' will show the
'earlier' commit after the 'later' one.

There are also two dates - the 'author' date I will have back-dated to
the date of the p5p e-mail containing the patch, if found in the
archives or otherwise noted in the changelog. The 'commit' date I tried
to fix to the date of the release, so should show a much greater , but
not absolute, respect to chronological consistency. But as you
discovered, it's not absolutely correct. Adding --date-order doesn't
help, either (--date-order is just --topo-order with date-based
tie-breaking).

Oh well, use a hash, then:

wilber:~/src/perl$ git log --pretty=fuller --date-order
origin/{blead,maint-5.{004,005,10,6,8}} origin/perl-1-maint | grep


CommitDate: | perl -ane '$m=$F[5];

$M{$m}++; END { print "$_ $M{$_}\n" for reverse sort keys %M }'
2009 1960
2008 2467
2007 3107
2006 3025
2005 2789
2004 1668
2003 3580
2002 4204
2001 5524
2000 3507
1999 2170
1998 2244
1997 1295
1996 961
1995 49


1994 6
1993 7
1992 16
1991 23
1990 33
1989 9
1988 16

1987 1

Dave Mitchell

unread,
May 26, 2009, 7:40:25 PM5/26/09
to chromatic, perl5-...@perl.org, Nicholas Clark
On Mon, May 25, 2009 at 03:06:06PM -0700, chromatic wrote:
> Several people who volunteered last year to manage a releases from a checklist
> of tasks they could work through to produce a release. What happened? What
> happened to that checklist? Was it a bad idea? Did no one have time to write
> it? Did the volunteers all say "No, thank you!" privately?
>
> Do I recall correctly that this process stalled when someone suggested "It's
> just too hard to generate a perldelta, so thanks, but no thanks!" (Once upon
> a time a pumpking decided that patches without test cases and documentation
> changes were unacceptable, so it seems like a small leap to require delta
> patches too.)
>
> As long as producing a release relies on someone spending a day a week
> cherrypicking patches from an unstable branch into a stable branch, I believe
> that no one who knows how to make a release will ever have time to write up
> what it takes to make a release (let alone what it will take to make a release
> parallelizable and trivial), and no one who doesn't already know how to make a
> release will learn how to make a release. Cart, egg, horse, chicken.

Ok, some comments from a maint-5.10 perspective (some of this has probably
already been covered better by other posters, but I'm too tired to try
summarizing the whole thread).

There are two aspects to producing a release. First is the purely
mechanical, go-through-a-checklist stuff, which results in a tarball. As
Nicholas has pointed out, this is the fun and easy bit, and getting
volunteers to help with that doesn't solve any problems. [ Well, for 5.10.1
it's going to be harder, because no-one has made a release yet since we
moved to git, and a whole bunch of things will need fixing up as I work my
way through it, but by the end of the process I expect to update
Porting/pumpkin.pod so it contains that nice shiny checklist you're so
keen on ]

The second aspect, the really, really, really, *really* hard bit, is
ensuring that maint is in a stable state for release. The thing that makes
this particularly hard, is the fact that perl bundles 114 dual-life CPAN
modules. Most of these modules have external authors not under the control
of p5p, who (quite rightly) have their own agenda and timelines. The trick
is to to issue a release with a set of CPAN modules that somehow all work
together with each other and the perl core, on as many weird and wonderful
platforms as possible (e.g. VMS).

I can announce to the whole world and his second cousin's dog's brother
that 5.10.X will be released on date Y, but if the dual-life modules
haven't stabilised around that time, then I'm buggered.

(Just as an aside, I think this is one area where the maint pumpking role
can be sensibly split into two, with pumpking#2 responsible for all the
dual-life module cherry-picks, and doing all the legwork of tracking
module stability. Volunteers please form an orderly queue...)

I'm not so bothered about the feature stuff. I'm perfectly capable of
temporarily skipping a bunch of commits for some blead feature regardless
of whether they have been moved into a separate feature branch. In fact
if bleed degenerated into zillions of little feature branches, then that
would only make the maint cherry-picking harder.

> My concrete suggestions are:
>
> 0) Release 5.10.1 and 5.11.0 concurrently

What problem does that solve?



> 1) Require all feature work to occur in branches

Please, no.



> 2) Declare and enforce trunk stability (no more "bleadperl" from which someone
> cherrypicks stable commits)

So what *does* maint get cherry-picked from then?



> 3) Enable smoking of branches ready to merge to trunk

As Rafael has pointed out elsewhere, few things in perl equate to a nice
simple isolated "feature" that can go in a branch and be individually
smoked; all things end up interrelated, and most feature branches will end
up spewing black smoke because they rely on changes/fixes/cpan-stuff that
are in other feature branches. Also, we've just increased the requirement
for smoke boxes by a factor of N.

> 4) Require all merge-ready branches to include perldelta changes, when
> appropriate

I can't speak for blead; but for maint, since January I have been
successfully updating perldelta at the same time as I cherry-pick from
blead, and it doesn't seem too onerous. So this isn't something that will
hold up maint releases in future.

> 5) Write a release manager guide

I plan to do this shortly after 5.10.1 release.

> 6) Recruit likely release managers for the next several releases

As has been mentioned before, building the release isn't the hard bit, so
recruiting release managers is a solution without a problem.



> 7) Set a release schedule tied to the calendar, not the state of any feature
>
> 8) Follow the release schedule

let's not, for all the reasons pointed out elsewhere in this thread.

> I believe #2 is the most important change to unblock this process and am happy
> to write about it in more detail if anyone cares.

I don't understand how #2 unblocks anything.



> Git enables 1, 2, and 3. 0, 1, 2, 4, 5, and 8 require active pumpking
> support. Is there interest?

I'm all in favour of things to make my life easier. I don't think what
I've heard so far is the solution though.

> If I'm full of hot air because I spend most of my development time on Parrot
> these days (though we have no trouble there with monthly stable releases), or
> if none of my proposals are interesting or useful, I can stop talking about
> this, though if I'm not completely crazy I'd certainly like feedback on why
> these suggestions are impractical or infeasable.

I think parrot is utterly non-comparable with perl. You haven't got half
of CPAN bundled with parrot, and you haven't got half the world ready to
scream blue murder at you if a release breaks something.

--
Indomitable in retreat, invincible in advance, insufferable in victory
-- Churchill on Montgomery

David Golden

unread,
May 26, 2009, 8:10:27 PM5/26/09
to p5p
On Tue, May 26, 2009 at 7:40 PM, Dave Mitchell <da...@iabyn.com> wrote:
> The second aspect, the really, really, really, *really* hard bit, is
> ensuring that maint is in a stable state for release. The thing that makes
> this particularly hard, is the fact that perl bundles 114 dual-life CPAN
> modules. Most of these modules have external authors not under the control
> of p5p, who (quite rightly) have their own agenda and timelines. The trick
> is to to issue a release with a set of CPAN modules that somehow all work
> together with each other and the perl core, on as many weird and wonderful
> platforms as possible (e.g. VMS).
>
> I can announce to the whole world and his second cousin's dog's brother
> that 5.10.X will be released on date Y, but if the dual-life modules
> haven't stabilised around that time, then I'm buggered.

Interesting that from the maintperl perspective "hard" is about
stability of dual-life modules instead of the stability of features
and bug fixes. Probably not surprising, on reflection. It still
points to the fact that blead/maint branch tips not being
release-ready is the problem.

Some options, not mutually exclusive

(1) stop merging "latest, greatest module release" to blead and *then*
figuring out if its stable. "latest, greatest" can go into a branch
(e.g. unstable, or dual-life-testing or whatever) and only get pulled
to blead once stable. The default can always be the version in the
last release of Perl, with a few exceptions where things are tightly
coupled (EU::MM and testing toolchain probably fall into the category
that can't just)

(2) have maint-perl track dual-life modules at a version in-between
last maint and latest blead. This means not cherry-picking blead for
dual-life modules, but direct patches to maint when a "stable"
dual-life version is deemed fit for maint. (Here's where I think the
move towards ext/ will help make this a lot easier)

(3) remove modules from core and let distribution packagers add them
from CPAN as they see fit; controversial, but would help cut down the
number of things needing to be checked

-- David

Mark Mielke

unread,
May 26, 2009, 11:28:54 PM5/26/09
to Rafael Garcia-Suarez, Sam Vilain, chromatic, perl5-...@perl.org
Rafael Garcia-Suarez wrote:
> There is a very important point here that you are missing.
>
> A programming langage is not a mere collection of features. At any stable point
> in time, it should be internally consistent. For example, in 5.10, having "my
> $_" without the (_) prototype would have been inconsistent -- it would have
> created situations where the language would not have been satisfactory.
> However, those are two different features. The same remark can be done
> about Unicode improvements, or regex work, etc.
>

I think I'm still missing the point - because I don't understand it. All
software is a collection of features. Keeping things internally
consistent and in a "ready to deliver" state at all times does not
prevent a language from changing in smaller increments. Or rather,
"keeping the language consistent" does not justify destabilizing trunk
for extended periods at a time. That's just laziness - and not the good
Perl kind of laziness that we all respect. :-)

Nicholas Clark

unread,
May 27, 2009, 5:26:00 AM5/27/09
to David Golden, p5p

This policy alone doesn't solve the issue that we don't *find out* that
something is duff until it's too late, and it's already been merged
somewhere more central. (Be it blead, because only blead, not branches,
are smoked on some platform, or a "trunk" branch, because a third party
only downloaded a "snapshot" of it to test on their platform or with their
code.)

> (2) have maint-perl track dual-life modules at a version in-between
> last maint and latest blead. This means not cherry-picking blead for
> dual-life modules, but direct patches to maint when a "stable"
> dual-life version is deemed fit for maint. (Here's where I think the
> move towards ext/ will help make this a lot easier)

This isn't consistent with your view that we keep branch tips releasable,
as this would mean that blead is not releasable.

But I also don't agree with patching direct to maint. It's viable for blead
to act as chronological integration for all CPAN releases (and CPAN release
candidates too), and then the choice is made *not* to integrate up to the
most current version.

Heck, I did that for 5.8.9 - Test::More was not on the same version as blead,
and possibly some other modules.

So yes, I agree with the intent, but not the mechanism you propose.

> (3) remove modules from core and let distribution packagers add them
> from CPAN as they see fit; controversial, but would help cut down the
> number of things needing to be checked

I've worked on that, but not dealt with the CPAN/CPANPLUS side of it yet, as
it's for 5.12, and 5.12 doesn't look imminent. So far, only Switch.pm is on
the list of retirees, but my hitlist is longer (and not canonical). For
starters I'm thinking of FileHandle.pm and Shell.pm, which are neither used
by the build process, or for installing modules, or as a key dependency of
CPAN.

(I'm not the person to ask about when 5.12 will arrive, what it will contain,
or how we get from here to there)


However, none of this addresses the fact that the number of people actually
prepared to assess bug reports, let alone investigate them (or other
activities that actually get work done on core), has dropped through the
floor, and doesn't look like it will come back again on its own.

Nicholas Clark

Andy Dougherty

unread,
May 27, 2009, 8:56:18 AM5/27/09
to Sam Vilain, Perl 5 Porters
On Wed, 27 May 2009, Sam Vilain wrote:

> Craig A. Berry wrote:
> >> This output doesn't fit with my recollection of history, either - I only
> >> remember 1995 happening once... ;-)
> >>
> > You would know better than I why git log thinks some parts of 1995
> > happened after some parts of 1996, but undoubtedly the breadcrumbs
> > were just too scarce and we shouldn't be expecting useful detail of
> > the sort I posted from the pre-Perforce years.

Exactly. Sam did a heroic job scavenging useful information wherever
possible, but sometimes the information simply wasn't available. For
example, there are only '6' commits that show up for 1994 (the year
perl5.000 was released), but each of those 6 was actually a huge
agglomeration of many smaller entries. For example, there's this entry:


commit a0d0e21ea6ea90a22318550944fe6cb09ae10cda
Author: Larry Wall <lw...@netlabs.com>
AuthorDate: Mon Oct 17 23:00:00 1994 +0000
Commit: Larry Wall <lw...@netlabs.com>
CommitDate: Mon Oct 17 23:00:00 1994 +0000

perl 5.000

[editor's note: this commit combines approximate 4 months of furious
releases of Andy Dougherty and Larry Wall - see pod/perlhist.pod for
details. Andy notes that;

Alas neither my "Irwin AccuTrack" nor my DC 600A quarter-inch cartridge
backup tapes from that era seem to be readable anymore. I guess 13 years
exceeds the shelf life for that backup technology :-(.
]


Again, I am amazed at how much history Sam was able to find, but I do
caution folks that attempts to gather statistics from the pre-Perforce era
are almost certainly meaningless.

--
Andy Dougherty doug...@lafayette.edu

Rafael Garcia-Suarez

unread,
May 27, 2009, 8:51:48 AM5/27/09
to David Golden, p5p
2009/5/27 David Golden <xda...@gmail.com>:

My preferred solution would be

a. move all dual-lifed modules in ext/
b. force dual-life module maintainers to commit changes themselves
(each module would have a branch or a fork on github). That's your 1st
option.
c. merge those in blead only if tests passes.

I'm not ruling out option 3 completely either.

Andy Dougherty

unread,
May 27, 2009, 9:06:01 AM5/27/09
to Mark Mielke, Perl Porters

I think it's more a reflection of the fact that some things are *hard*,
especially since features often interact in ways that are subtle.

--
Andy Dougherty doug...@lafayette.edu

Mark Mielke

unread,
May 27, 2009, 11:37:55 AM5/27/09
to Andy Dougherty, Perl Porters

By subtle I think you mean in ways not predicted by the committer. :-)

The "make the change, throw it out on blead, wait for it to smoke, and
let other people discover problems in my patch" just seems bad.

Also, there is a lot of software out there significantly more complex
than the Perl internals. Perl isn't really that bad. I'm not convinced
that it being a language makes it more complex or more difficult to keep
stable than any other serious large software project. I would consider
the Linux kernel to be an example of something hairier than Perl.

Nicholas Clark

unread,
May 27, 2009, 11:44:24 AM5/27/09
to Mark Mielke, Andy Dougherty, Perl Porters
On Wed, May 27, 2009 at 11:37:55AM -0400, Mark Mielke wrote:
> Andy Dougherty wrote:
> >On Tue, 26 May 2009, Mark Mielke wrote:
> >
> >>I think I'm still missing the point - because I don't understand it. All
> >>software is a collection of features. Keeping things internally
> >>consistent and
> >>in a "ready to deliver" state at all times does not prevent a language
> >>from
> >>changing in smaller increments. Or rather, "keeping the language
> >>consistent"
> >>does not justify destabilizing trunk for extended periods at a time.
> >>That's
> >>just laziness - and not the good Perl kind of laziness that we all
> >>respect.
> >>:-)
> >>
> >I think it's more a reflection of the fact that some things are *hard*,
> >especially since features often interact in ways that are subtle.
>
> By subtle I think you mean in ways not predicted by the committer. :-)
>
> The "make the change, throw it out on blead, wait for it to smoke, and
> let other people discover problems in my patch" just seems bad.

Counter example: can you see what's wrong in the original bug fix to require
that caused the smoke failure linked in
http://www.xray.mpe.mpg.de/mailing-lists/perl5-porters/2005-11/msg00511.html

A sane, small and (arguably) correct change caused some other unrelated
system to fly apart.

Counter example generally: How good is your understanding of VMS? Of Win32?
Even of AIX.

> Also, there is a lot of software out there significantly more complex
> than the Perl internals. Perl isn't really that bad. I'm not convinced
> that it being a language makes it more complex or more difficult to keep
> stable than any other serious large software project. I would consider
> the Linux kernel to be an example of something hairier than Perl.

I dispute "a lot". Operating system kernels that run on a large variety of
hardware - yes.

Not much else.

It's not the size. It's the age of the code, and the interdependency.

Nicholas Clark

Rafael Garcia-Suarez

unread,
May 27, 2009, 12:00:30 PM5/27/09
to Mark Mielke, Sam Vilain, chromatic, perl5-...@perl.org
2009/5/27 Mark Mielke <ma...@mark.mielke.cc>:

A programming language just cannot change in smaller increments.
That's because it's not a mere collection of features. It's a
_consistent_ collection of features. You can add a feature, test it,
make it pass its tests on all platforms and verify it doesn't break
half of CPAN -- that does not mean that the language is releasable.
Wouldn't it be embarassing if, say, changes were made on pack
templates, but not matching ones on unpack ?

And I don't get your comment about destabilizing trunk. When was the
last time you got a "make test" reporting a failure on a common
platform such as Linux or FreeBSD under a configuration reasonably
close to the default ?

Mark Mielke

unread,
May 27, 2009, 12:14:58 PM5/27/09
to Mark Mielke, Andy Dougherty, Perl Porters
Nicholas Clark wrote:
> On Wed, May 27, 2009 at 11:37:55AM -0400, Mark Mielke wrote:
>
>> Andy Dougherty wrote:
>>
>>> I think it's more a reflection of the fact that some things are *hard*,
>>> especially since features often interact in ways that are subtle.
>>>
>> By subtle I think you mean in ways not predicted by the committer. :-)
>>
>> The "make the change, throw it out on blead, wait for it to smoke, and
>> let other people discover problems in my patch" just seems bad.
>>
>
> Counter example: can you see what's wrong in the original bug fix to require
> that caused the smoke failure linked in
> http://www.xray.mpe.mpg.de/mailing-lists/perl5-porters/2005-11/msg00511.html
>

Other than guessing it has something to do with the tricky fork
emulation on Windows that has always seemed fragile from the start? No
clue. :-)

> A sane, small and (arguably) correct change caused some other unrelated
> system to fly apart.
>

Sure - but, give that Windows is widely supported and used, I have to
wonder if the contributor tried it on Windows or not? This may come down
to resources available to the contributor rather than "what is best".
That is, throwing it on blead and waiting for it to smoke is necessary
because contributors don't have better options available to them.

> Counter example generally: How good is your understanding of VMS? Of Win32?
> Even of AIX.
>

I guess this is relevant from the sense of a "what the average
contributor can be expected to know". I think the conclusion you are
making here is that the average contributor probably has no clue how
their change will affect systems very different from their own. (In
terms of me personally - I know Win32 pretty well, VMS almost nothing,
AIX - I know enough to know it's not UNIX and that it needs some
tweaking to get things to compile back from the days when I regularly
used an AIX box... But, details for me personally? I don't think this
is relevant...)

It seems bad that the average contributor might not understand the
effects of the code they are changing. It seems like this is a problem
in the design - although perhaps that is also one of your points.

If one looks at a solution such as Java, for example, where the majority
of the code that actually defines the language, is written in Java
itself, these problems don't exist any more. Contributors can change
String.java, and know that it will work fine on all platforms out there.
The black voodoo happens in the JIT compiler layer, and anybody changing
that layer *better* understand the consequences. They've made a clear
separation between the 'portable' part of the implementation and the
'non-portable' part of the implementation.

This might be related to your later "age of the code". Design
assumptions, and design evolution over long periods of time, creating
interdependencies that are not always clear and well known and documented.

Ok, I buy this. :-)

>> Also, there is a lot of software out there significantly more complex
>> than the Perl internals. Perl isn't really that bad. I'm not convinced
>> that it being a language makes it more complex or more difficult to keep
>> stable than any other serious large software project. I would consider
>> the Linux kernel to be an example of something hairier than Perl.
>>
>
> I dispute "a lot". Operating system kernels that run on a large variety of
> hardware - yes.
>
> Not much else.
>
> It's not the size. It's the age of the code, and the interdependency.

Well, to throw a few out there that would offer strong competition in
the "age of the code and the interdepedency" - GNOME, PostgreSQL, and
Mozilla come to mind as favourites. These are pretty complex beasts too,
with long enough histories to make things similarly complex.

Andy Dougherty

unread,
May 27, 2009, 12:27:51 PM5/27/09
to Perl Porters
On Wed, 27 May 2009, Mark Mielke wrote:

> Andy Dougherty wrote:
>
>> I think it's more a reflection of the fact that some things are *hard*,
>> especially since features often interact in ways that are subtle.
>

> Also, there is a lot of software out there significantly more complex than


> the Perl internals. Perl isn't really that bad. I'm not convinced that it
> being a language makes it more complex or more difficult to keep stable than
> any other serious large software project. I would consider the Linux kernel
> to be an example of something hairier than Perl.

Perhaps, though the Linux kernel doesn't have the same compatibility
commitments as Perl does. External kernel drivers routinely break under
even "minor" version updates.

Still, I will readily grant it is indeed a big hairy project.

--
Andy Dougherty doug...@lafayette.edu

Mark Mielke

unread,
May 27, 2009, 1:28:09 PM5/27/09
to Rafael Garcia-Suarez, Sam Vilain, chromatic, perl5-...@perl.org
Rafael Garcia-Suarez wrote:
> A programming language just cannot change in smaller increments.
> That's because it's not a mere collection of features. It's a
> _consistent_ collection of features. You can add a feature, test it,
> make it pass its tests on all platforms and verify it doesn't break
> half of CPAN -- that does not mean that the language is releasable.
> Wouldn't it be embarassing if, say, changes were made on pack
> templates, but not matching ones on unpack ?
>

I don't want to really belabour this somewhat off-topic. You guys have
kindly put up with my perhaps naive comments. :-)

There might be a better analogy than pack/unpack (pack/unpack is the
exact sort of example where I would expect an incremental change - to
both simultaneously, to be possible). Let's leave it at that. :-)

> And I don't get your comment about destabilizing trunk. When was the
> last time you got a "make test" reporting a failure on a common
> platform such as Linux or FreeBSD under a configuration reasonably
> close to the default ?
>

Then we agree in principle? Trunk should be kept "ready to deliver",
possibly ignoring the really hard for contributors to test
configurations? :-)

Steve Peters

unread,
May 27, 2009, 3:19:29 PM5/27/09
to Mark Mielke, Andy Dougherty, Perl Porters
On Wed, May 27, 2009 at 11:14 AM, Mark Mielke <ma...@mark.mielke.cc> wrote:

>
> If one looks at a solution such as Java, for example, where the majority of
> the code that actually defines the language, is written in Java itself,
> these problems don't exist any more. Contributors can change String.java,
> and know that it will work fine on all platforms out there. The black voodoo
> happens in the JIT compiler layer, and anybody changing that layer *better*
> understand the consequences. They've made a clear separation between the
> 'portable' part of the implementation and the 'non-portable' part of the
> implementation.


That's not necessarily a correct analogy. Perl makes every attempt to make
sure code written will not only work on bleadperl on my Mac OS X box, but
also a Perl 5.6 on my old Windows 98 box. In fact, several old test files
have been carried largely intact since Perl 4. Java is typically only
forward compatible meaning Java 1.4 code will run on a Java 1.5 runtime, but
Java 1.5 code will at best whine loudly on Java 1.4 runtime but more likely
fail to run at all. Another problem with Java has been the non-standard
JVMs released by several vendors. It is not surprising that sometimes code
that works perfectly with a Sun JDK will fail miserably with an IBM JVM and
vice versa. The people that sell the perfect portability of Java have also
been known to sell previously owned vehicles and bridges and the
metropolitan New York area.


Steve Peters
st...@fisharerojo.org

Mark Mielke

unread,
May 27, 2009, 3:51:48 PM5/27/09
to Steve Peters, Andy Dougherty, Perl Porters

I think you missed my comparison. A LOT of Perl is written in
platform-specific code. Perl has some nice .h pre-processor magic to
hide many things, but platform-specific stuff bubbles to the surface
everywhere. Andy and others were making the case that Perl is hard to
change because it is a language or because it the code has such age, and
it is has evolved so far, that there are plenty of opportunities for
action at a distance. Nicholas pointed to an innocent looking patch that
ended up causing breakage on Windows, and the issue was difficult to
resolve.

I'm not talking about Perl compatibility between versions or Java
compatibility between versions, which is frankly a religious war that I
don't want to get into. I'm not trying to sell you bridges from the
metropolitan New York area or portability kool aid. Far from it - I said
something completely different.

I said that a majority of Java is *written* in Java. The implementation
of the "String" object, is implemented in String.java. Being a nearly
fully self-hosting language, even in terms of bootstrap, it has a unique
benefit that if somebody contributor to OpenJDK changes String.java,
they can have very high certainty that their change will be portable to
ALL platforms that Java is supported under. There are lots of negatives
to this approach - one of them is performance. Perl's strings and hashes
may perform better, since they're written in highly optimized C code.
But, one great positive is that the majority of the self-hosting code
itself - is truly platform independent.

In case, this is truly off topic.

Tom Horsley

unread,
May 27, 2009, 3:52:33 PM5/27/09
to Perl Porters
On Wed, 27 May 2009 15:19:29 -0400
Steve Peters wrote:

> It is not surprising that sometimes code
> that works perfectly with a Sun JDK will fail miserably with an IBM JVM and
> vice versa.

Not to mention one version of Sun not working the same as another :-).
I remember having a discussion on some mailing list about the java
version of the citrix client not working on different systems,
(all of which had some version of Sun's java or openjdk, etc)
and got a response from someone at a university where they run
a telescope driven remotely by a java app, and if I'm recalling
correctly, he said almost 1/3 of the code in the app exists only
to probe java's reactions and work around them so the app will
work on as many versions of java as possible. The whole absolutely
portable thing appears to be more Sun marketing kool-aid than
actual fact.

I've always suspected that the main reason Sun dropped their lawsuit
against Microsoft was that Microsoft was getting ready to demonstrate
an infinite number of cases where Sun's own software was just as
incompatible as the Microsoft version :-).

Mark Mielke

unread,
May 27, 2009, 3:59:49 PM5/27/09
to Tom Horsley, Perl Porters
My opinion: P5P should be above this sort of FUD. "some mailing list"
"response from someone". "1/3 of the code for probing"? Either he was a
horrible code writer, or he was making things up as he went.

I should have known better than dare mention another language as a
comparison. Java is a great language just as Perl is a great language.
They have their pros and cons. I've successfully and happily used both
for serious projects, and been glad with my choice in both cases. The
best tool for the job. FUD has no place in professional discussions over
software.

Cheers,
mark


Tom Horsley wrote:
> Not to mention one version of Sun not working the same as another :-).
> I remember having a discussion on some mailing list about the java
> version of the citrix client not working on different systems,
> (all of which had some version of Sun's java or openjdk, etc)
> and got a response from someone at a university where they run
> a telescope driven remotely by a java app, and if I'm recalling
> correctly, he said almost 1/3 of the code in the app exists only
> to probe java's reactions and work around them so the app will
> work on as many versions of java as possible. The whole absolutely
> portable thing appears to be more Sun marketing kool-aid than
> actual fact.
>
> I've always suspected that the main reason Sun dropped their lawsuit
> against Microsoft was that Microsoft was getting ready to demonstrate
> an infinite number of cases where Sun's own software was just as
> incompatible as the Microsoft version :-).
>


--
Mark Mielke <ma...@mielke.cc>

Ben Evans

unread,
May 27, 2009, 6:31:51 PM5/27/09
to Mark Mielke, Perl Porters
On Wed, May 27, 2009 at 8:51 PM, Mark Mielke <ma...@mark.mielke.cc> wrote:

> Steve Peters wrote:
>
>> Another problem with Java has been the non-standard JVMs released by
>> several vendors. It is not surprising that sometimes code that works
>> perfectly with a Sun JDK will fail miserably with an IBM JVM and vice versa.
>> The people that sell the perfect portability of Java have also been known
>> to sell previously owned vehicles and bridges and the metropolitan New York
>> area.
>
>

Errr, I think this rather overstates the case. Do you have any references
for this claim that there are bytecode sequences that have different
semantic interpretations between Sun and IBM JDKs?

Note that in purely semantic terms, we need to leave to one side the details
of what should be thought of as implementation concerns, such as whether
JITing will optimise away the call frames of tail calls or GC (technically
speaking it is possible to write a conformant Java implementation which
never reclaims any memory until the JVM process exits).


> In case, this is truly off topic.


Seconded.

Ben

Steve Peters

unread,
May 28, 2009, 12:49:15 AM5/28/09
to Ben Evans, Mark Mielke, Perl Porters
On Wed, May 27, 2009 at 5:31 PM, Ben Evans <
benjamin....@googlemail.com> wrote:

> On Wed, May 27, 2009 at 8:51 PM, Mark Mielke <ma...@mark.mielke.cc> wrote:
>
> > Steve Peters wrote:
> >
> >> Another problem with Java has been the non-standard JVMs released by
> >> several vendors. It is not surprising that sometimes code that works
> >> perfectly with a Sun JDK will fail miserably with an IBM JVM and vice
> versa.
> >> The people that sell the perfect portability of Java have also been
> known
> >> to sell previously owned vehicles and bridges and the metropolitan New
> York
> >> area.
> >
> >
> Errr, I think this rather overstates the case. Do you have any references
> for this claim that there are bytecode sequences that have different
> semantic interpretations between Sun and IBM JDKs?


13 years of experience with Java have told me to never mix compilers and
JVMs (including the assorted included class libraries). You can get by with
small, self-contained bits of code, but when dealing with any decent sized
applications, there are invariably incompatibilities.

Steve Peters
st...@fisharerojo.org

Ben Evans

unread,
May 28, 2009, 4:17:34 AM5/28/09
to Steve Peters, Mark Mielke, Perl Porters

If you haven't got evidence (eg incompatibility bugs closed with WONTFIX /
NOTABUG / etc) , then
citing "years of experience" without details is just FUD of the same
standard as "Perl is great for small SA scripts, but maintaining a 50kloc
application in it is just impossible".

Ben

Nicholas Clark

unread,
May 28, 2009, 4:21:27 AM5/28/09
to Mark Mielke, Rafael Garcia-Suarez, Sam Vilain, chromatic, perl5-...@perl.org
On Wed, May 27, 2009 at 01:28:09PM -0400, Mark Mielke wrote:
> Rafael Garcia-Suarez wrote:

> >And I don't get your comment about destabilizing trunk. When was the
> >last time you got a "make test" reporting a failure on a common
> >platform such as Linux or FreeBSD under a configuration reasonably
> >close to the default ?
> >
>
> Then we agree in principle? Trunk should be kept "ready to deliver",
> possibly ignoring the really hard for contributors to test
> configurations? :-)

Oh, I can completely agree to that in principle. :-)

And it's the state I try to maintain. But human error and the world in
general often have short term victories.

Nicholas Clark

Nicholas Clark

unread,
May 28, 2009, 9:00:07 AM5/28/09
to Mark Mielke, Andy Dougherty, Perl Porters
On Wed, May 27, 2009 at 12:14:58PM -0400, Mark Mielke wrote:
> Nicholas Clark wrote:
> >On Wed, May 27, 2009 at 11:37:55AM -0400, Mark Mielke wrote:
> >
> >>Andy Dougherty wrote:
> >>
> >>>I think it's more a reflection of the fact that some things are *hard*,
> >>>especially since features often interact in ways that are subtle.
> >>>
> >>By subtle I think you mean in ways not predicted by the committer. :-)
> >>
> >>The "make the change, throw it out on blead, wait for it to smoke, and
> >>let other people discover problems in my patch" just seems bad.
> >>
> >
> >Counter example: can you see what's wrong in the original bug fix to
> >require
> >that caused the smoke failure linked in
> >http://www.xray.mpe.mpg.de/mailing-lists/perl5-porters/2005-11/msg00511.html
> >
>
> Other than guessing it has something to do with the tricky fork
> emulation on Windows that has always seemed fragile from the start? No
> clue. :-)

Only it's not, as that message shows that I could actually reproduce it on
Linux with a minimal test case. And no, even then, I had no clue.

> >A sane, small and (arguably) correct change caused some other unrelated
> >system to fly apart.
> >
>
> Sure - but, give that Windows is widely supported and used, I have to
> wonder if the contributor tried it on Windows or not? This may come down
> to resources available to the contributor rather than "what is best".
> That is, throwing it on blead and waiting for it to smoke is necessary
> because contributors don't have better options available to them.

The contributor was me, and my way of testing on Win32 is to have Steve smoke
it. But it happened that the problem was cross platform, it's just that only
a test on Win32 crashed because of it, at first.

It's not a Win32 issue, it's a very strange interaction between the code
that sets up the save stack, and the code that unwinds it.

> It seems bad that the average contributor might not understand the
> effects of the code they are changing. It seems like this is a problem
> in the design - although perhaps that is also one of your points.

I don't think it's the design. I think it's an inherent problem with trying
to be portable.

> If one looks at a solution such as Java, for example, where the majority
> of the code that actually defines the language, is written in Java
> itself, these problems don't exist any more. Contributors can change
> String.java, and know that it will work fine on all platforms out there.
> The black voodoo happens in the JIT compiler layer, and anybody changing
> that layer *better* understand the consequences. They've made a clear
> separation between the 'portable' part of the implementation and the
> 'non-portable' part of the implementation.

Yes, that's a fair point. Perl 6 is working t