Walk-through impressions and mercurial workflow

[martin....@gmx.net]

After using sage for quite some time, and contributing some input to the sage-on-gentoo project, I finally decided to get my hands dirty and tackle some annyoing bugs in sage itself, #14403 in particular. In the process, I used the Dev Walk-Through at http://www.sagemath.org/doc/developer/walk_through.html which was a great help. Nevertheless, I had some thoughts which might improve it even further, so I'll share them here.

The text starts out already assuming that you have a sage install. I think there should be more on how to obtain the sources. In particular, it seems as if the right way to start coding on sage is by taking a reasonably recent source release tarball and compile that. Since this is opposed to the “check out the repository head” I'd do for pretty much any other project, this might be worth mentioning.

If I start from a tarball, then I'm likely behind the repository tip. Usually I'd pull in new changes, to ensure that the patches I submit are against the latest state of affairs. But I found no description at all how to do this, or if it is considered good practice in the sage community. Down on that page I find the section on “Upgrading Sage with queues present”, which does mention that I might upgrade at some point, but it only states that a suitable URL might be provided in discussion groups. No single URL to get to repository head, so I assume this is for release upgrades only. Should I try to manually upgrade to what http://hg.sagemath.org/sage-main/src has to offer?

“Creating a sandbox” shows how to create what is later called a “branch”. But I'm a bit surprised that a sage branch apparently is a clone of the repository, with development still going on inside the “default” branch as far as hg is concerned. Is there any reason not to use a mercurial branch for this? The benefit, as far as I see it, would be that you'd only have a single working directory, which means less disk space and perhaps less confusion about which version you currently have opened in your editor. Of course, on the downside you'd have to commit changes before switching branches, and switching branches would mean sources change so targets have to be rebuilt. Therefore I can see some reasons for the current approach, but would consider an approach using branches worth describing as well.

“Updating a change” is a real pain, and this is the main reason I'm writing this post here. The elegant solution to this problem is a simple “hg commit --amend”, which includes all new changes into the previous commit. That way, you don't have to delete, clone and patch, but only to commit and export. I consider this so useful that you might even want to write a method to the hg_sage object for this. A “hg diff -r -2” to see changes relative to the revision before the last commit will come handy as well.

“Mercurial queues” is nice at first glance, since it uses only a single directory, and works more directly with the version control system. One problem I see here is that queues are imho rather ill suited for unrelated modifications. Series are always ordered stacks of patches, not simply collections of them. As far as I know (although I don't know mq too well) there is no way to apply a patch without all the patches that went before it. And there is no way to switch between different series either. So you can't isolate effects, which might be bad under some circumstances. In the simplest case, it means that line numbers for later patches are wrong. In more problematic cases, there might be adverse effects, particularly when doing things like benchmarks to investigate the effect of a single patch on the performance of some operation.

It seems to me that mercurial branches would be better suited for most cases. The one exception are multiple interdependent patches, since the above “commit --amend” will only work for a single commit. So in those cases where you'll submit all your patches to Trac, either to a single ticket or to multiple tickets with explicit interdependencies, queues would be the right tool.

Personally, I'll probably do mercurial branches in the future, and if I need multiple patches, I'll use mq. If I ever want to work at more than one issue with more than one patch, I'll probably clone the repo to get multiple distinct queues.

I hope that this report will help you make sage development easier for newcomers like myself.

[John Cremona]

Were you aware of the fact that Sage is on the brink of leaving
mercurial behind and using git for all this (and more)? I'll leave
others to say more about that, but it means that most of the
deficiencies you mention are well known and on the point of being
solved with a new all-singing-and-dancing git system for Sage
develoment.

John

> --
> You received this message because you are subscribed to the Google Groups "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.
> To post to this group, send email to sage-...@googlegroups.com.
> Visit this group at http://groups.google.com/group/sage-devel.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

[Volker Braun]

We are in the process of updating the developer guide for the new git workflow. The current state is here:

http://sagemath.github.io/git-developer-guide/

Help is of course welcome, the github project page is here: https://github.com/sagemath/git-developer-guide

[martin....@gmx.net]

After writing my post, while skimming some other headers here in this group, I found http://wiki.sagemath.org/WorkflowSEP. So now I'm somewhat aware, but I wasn't before. Thanks for pointing this out.

[martin....@gmx.net]

This is great news, it should simplify things considerably. Will there be a specific switchover day, or are contributors already invited to follow this procedure?

I submitted a pull request with various minor fixes, but what I'm really interested in is the big holes. For example, what will be your relation to github? Will you accept issues filed there, or pull requests submitted there? If so, will you import these into trac? Should contributors make an effort to learn the trac workflow, even if they know github very well?

I also tried this out, cloning the repo from the trac machine. I noticed that I get a number of perl warnings due to unsupported locale settings, particularly for the “info” command, but also one while cloning. This is probably due to the fact that my distro sends locale info over ssh by default, and there appears to be no way to override this. And the locale in question isn't installed on the server. Perhaps you can remove the AcceptEnv setting from the ssh server configuration to avoid these issues. I guess git doesn't really require locale settings in any case, does it?

I know many projects encourage contributors to write bug fixes against the revisions which introduced a bug. That way they can be integrated into multiple lines of development simply by merging that branch. But identifying the relevant base revision may take some work. Do you have any policy on this issue? Do you want to describe it, or encourage users who know how to do this, or would you rather have all commits against a recent code base to obtain a simple and more linear line of development?

[Volker Braun]

The recommended workflow for Sage is the "Simple Workflow". You can use github or any other repo to collaborate, but you'll have to push the final result to the Sage trac server to add it to a ticket. You should base your edits on the most recent Sage version. You can already use git, we'll support both mercurial patches and git branches during a transition period.

I made http://trac.sagemath.org/ticket/14986 to deal with the locale issue (Andrew, please have a look). Though the ssh gitolite commands don't really allow you to do anything besides checking that your public key is set up correctly.

I've added your fixes to the git developer manual (note that it is for now developed on github, so pull requests are fine).