New contribution workflow for packaging (for GSoC students + current devs)

[Éric Araujo]

Hi,

With packaging due to land in the official CPython repo these days, I’d
like to propose a new development workflow, both for existing developers
and GSoC students.

The workflow I advocate is one I used last summer with great benefits
and which is now even more relevant.

The idea is to start a clone and a named branch for each long-term
feature or difficult bug. The clone can be efficiently done with the
Web UIs of hg.python.org or bitbucket (no details now, just the general
process), then you start a new branch and hack away. Using a named
branch is useful because it gives you an easy name to use with “hg
update”, “hg diff -r default:mywork”, “hg log”, etc. This makes it easy
to work on various things, merge regularly with upstream, generate
diffs, fix mistakes, ports some changes upstream directly, get code
reviews thanks to the review button on bugs.python.org... In short, you
can commit and push to backup and share regularly, but you don’t get a
messy repo in the end.

The only downside is that it’s a bit more difficult to split your work
into more than one patch, but after all Python does not use that: one
bug, one patch. People are free to use MQ for that, but I do not
recommend it for long-term work (neither do the Mercurial developers).
Diffs of diffs are no fun, and MQ does not use the regular merge system.

For small changes or bugfixes, it’s of course not necessary to use a new
clone and branch.

I used named branches last summer with d2 and use them now with CPython.
With the tools support (server-side clone, code reviews) that CPython
has now, I think it makes sense that we adopt this style of development
to transition to working in the stdlib.

Waiting for your feedback

[Kelsey Hightower]

+1 seems more organized. Is this work flow captured in the Python Dev docs?

[Alexis Métaireau]

Agreed, I used this workflow last summer as well for distutils2 and it
worked well. We also can have a repository on hg.python.org to publish
our work so others can see and review it.

Also, as you suggested Éric, we should use more often the python bug
tracker. I don't know what is the python-dev policy about that, for
instance for new features: do we have to open a new ticket and attach
patches / point to named branches also for new features?

--
Alexis

[Éric Araujo]

Le 15/05/2011 14:02, Kelsey Hightower a écrit :
> +1 seems more organized. Is this work flow captured in the Python Dev docs?

I thought so but it’s sadly been removed in favor of MQ. I will bring
this up on the python-committers list. MQ is much less newbie-friendly
and Mercurial devs themselves have told me it’s not well suited to
long-term development. If the core devs disagree, I will find the
previous wording in the devguide repo and put it on the wiki page
describing our worklow.

(Unrelated note: Would you mind not top-posting? For the small part of
the world not using GMail or Google Groups, it’s distracting to see
walls of needless quoted text. Thanks in advance.)

Regards

[Éric Araujo]

Hi Alexis,

Le 15/05/2011 20:32, Alexis Métaireau a écrit :
> Agreed, I used this workflow last summer as well for distutils2 and it
> worked well. We also can have a repository on hg.python.org to publish
> our work so others can see and review it.

And for people without ssh access to hg.python.org, bitbucket has a
CPython mirror.

> Also, as you suggested Éric, we should use more often the python bug
> tracker. I don't know what is the python-dev policy about that, for
> instance for new features: do we have to open a new ticket and attach
> patches / point to named branches also for new features?

It is not required but is useful for coordination, to avoid duplication,
get feedback, report problems, document choices, etc. Code reviews are
also not required (Python core development is very pragmatic, you will
see), but strongly recommended.

If these pieces of info are not in the devguide, please report a bug.

> [snip wall of quotation]

Regards

[Éric Araujo]

Le 15/05/2011 14:02, Kelsey Hightower a écrit :

> +1 seems more organized. Is this work flow captured in the Python Dev docs?

I thought so but it’s sadly been removed in favor of MQ. I will bring

[Tarek Ziadé]

I am in favor of MQ. The learning curve is not that bad, and it's way
better to work on multiple patches.

Cloning cpython for every patch is just a pain, really..

I am curious about the criticism about MQ from the mercurial dev., can
you elaborate ?

> Regards
>

--
Tarek Ziadé | http://ziade.org

[Tarek Ziadé]

On Mon, May 16, 2011 at 6:33 PM, Éric Araujo <win...@netwok.org> wrote:

I am in favor of MQ. The learning curve is not that bad, and it's way

[Daniel Holth]

I've been using MQ for several years. MQ is great for organizing a set of (orthogonal) patches locally, as you can go back and revise each one until they are just right. This makes you look smarter. But it's difficult to collaborate on a patch queue with someone else (patches of patches) and it can be tricky to reconcile patch queues with upstream revisions. I like to use MQ when I'm working on a larger change that should be broken up into several atomic pieces.

I never got the hang of Mercurial's named branches but some people like them. Newer versions of Mercurial also have a 'rebase' command which takes care of some of the use cases of mq (linearizing history instead of merging, cherry-picking, ...)

[Rafael Villar Burke (Pachi)]

On 15/05/2011 12:53, Éric Araujo wrote:
> Hi,
>
> With packaging due to land in the official CPython repo these days, I’d
> like to propose a new development workflow, both for existing developers
> and GSoC students.
>
> The workflow I advocate is one I used last summer with great benefits
> and which is now even more relevant.
>
> The idea is to start a clone and a named branch for each long-term
> feature or difficult bug.

There's a nice tutorial by one of the Mercurial developers that explains this workflow nicely:
http://mercurial.aragost.com/kick-start/en/tasks/

I hope it can be useful for people starting to use mercurial.

Regards,

Rafael Villar Burke

[Éric Araujo]

Hi,

I have finally got around to write in detail the workflow we talked
about and I posted it at http://wiki.python.org/moin/Distutils/Contributing

I’ve also updated the much shorter
http://wiki.python.org/moin/Distutils/FixingBugs

Any feedback is appreciated.

[Tarek]

> I am in favor of MQ. The learning curve is not that bad, and it's way
> better to work on multiple patches.

Sure. For Python patches, multiple patches are not generally used, so
MQ is not needed. I agree that the learning curve is not that bad, but
it‘s less easy than pure Mercurial, and the user experience is worse in
my experience (I want to merge branches, not refresh diffs!)

> Cloning cpython for every patch is just a pain, really..

In what aspect? If it’s time, know that on UNIX you can use cp to copy
a repository with an already built python and thus skip compilation.

> I am curious about the criticism about MQ from the mercurial dev., can
> you elaborate ?

<parren> merwok: mq manages patch files. pbranch manages features
branches. So it's more suited to long-lived and/or collaborative patch
development/maintenance.
<parren> merwok: I use mq for quick edits. I use pbranch to maintain my
customized hg setup, for instance. And to develop features that might
take weeks or months to finish.
[...]
<sjl> merwok: You can use rebase to help make MQ patches apply easier,
but it's not designed to manage many branches of long-running development
<parren> merwok: With pbranch, the rebase becomes a plain hg merge. One
of the things I like about it.
<timeless_mbp> yeah, rebase kinda works

[Pierre-Yves David]

On Thu, Jun 09, 2011 at 03:41:27PM +0200, Éric Araujo wrote:
> Hi,
>
> I have finally got around to write in detail the workflow we talked
> about and I posted it at http://wiki.python.org/moin/Distutils/Contributing

I would discourage the use of named branches for feature related stuff. Named
branch is far too much rooted in history for feature that are typically short
lived. You better use bookmark that are more volatile and fit more your
workflow. Bookmark are available for ages and are push/pullable since
mercurial 1.7.

Most criticism around MQ focus on:

MQ is too fragile: it's far much easier to corrupt your mq repository that
your mercurial one. Working one multiple related patches often lead to
reject.

MQ is too limited: it does handle merge. you can safely only work one your
top most patches.

However mq is still and useful and efficient tool is most common case.
Rewriting mercurial history should be much easier with the 2.0 release due in
november.

--
Pierre-Yves David

http://www.logilab.fr/

[Éric Araujo]

> I would discourage the use of named branches for feature related stuff. Named
> branch is far too much rooted in history for feature that are typically short
> lived.

What do you mean with rooted? Permanent? As explained, the trick is to
use the name only to have an identifier to use with update, log, diff
and co.; the named branch will never be pushed to the main repo, but
only used to produce one patch.

Also, be careful with “short-lived”; some apparently simple changes take
months to complete.

> You better use bookmark that are more volatile and fit more your
> workflow. Bookmark are available for ages and are push/pullable since
> mercurial 1.7.

Good one. I did not think about them since I haven’t had the chance yet
to use them. Would the merge instructions in the workflow guide work
the same way with bookmarks? With a named branch, you don’t have heads
problems (I’m thinking about “hg heads .” and “hg push”); is it the same
with bookmarks?

By all means feel free to update the wiki page to mention bookmarks.

Regards