Release management of flask-sagenb
Jeroen Demeyer
Now that there seems to be some effort to finalize the flask-sagenb
spkg, there is a question which is extremely important to me which needs
to be answered:
Will the new flask-sagenb be
a) a separate project
b) part of Sage (as it currently is, despite what William thinks)
If you go for option a), then I expect flask-sagenb to appoint their own
"release manager" and use their own bug-tracker. I will treat sagenb
like any other spkg, like mpir, atlas, python,...
If you go for option b), I will remain release manager also for sagenb,
we keep using the Sage Trac and I will treat sagenb patches like patches
to the Sage library (devel/sage) or patches to sage_scripts (local/bin).
In terms of total development complexity, I think option b) is simpler.
Some Sage tickets require changes to sagenb or vice versa and with
option a), this coordination will certainly be harder.
Please don't hijack this thread, try to keep it on topic.
Jeroen.
Jason Grout
I would vote for (a) for the near future. There will probably be pretty
rapid development in the notebook over the next 1.5, and moving to a
github-type of model (pull requests, branches, etc.) will make it much
easier for us to do this sort of development in the funding time period
that we have. It would be great if we could still install the
development repository into $SAGE_ROOT/devel/sagenb, though, as that
makes the repository accessible to newcomers.
So my preference would be:
(1) Move development to github: This would requires using git, which may
be too much of a change. I personally find git and github better than
google code for rapid development integrating work from a number of
people. Bitbucket also has some of the nice features of github, like
pull requests, comments on lines of code in pull requests, nice forking
that is less burdensome, etc. So if we *must* stay with mercurial, I
would prefer moving development to bitbucket. I've been maintaining a
bitbucket mirror at https://bitbucket.org/jasongrout/sagenb
(2) We'll make sagenb spkgs and submit them via trac
(3) Allow us to keep $SAGE_ROOT/devel/sagenb as a repository, like we
have it now. The Sage release manager wouldn't merge patches into that
repository or make spkgs for that repository, though.
In 2 years, when development on the notebook may switch back to
maintenance mode, then it might make more sense to switch the
administration back to the Sage release manager.
Thanks,
Jason
kcrisman
> > "release manager" and use their own bug-tracker. I will treat sagenb
> > like any other spkg, like mpir, atlas, python,...
>
> > If you go for option b), I will remain release manager also for sagenb,
> > we keep using the Sage Trac and I will treat sagenb patches like patches
> > to the Sage library (devel/sage) or patches to sage_scripts (local/bin).
>
> > In terms of total development complexity, I think option b) is simpler.
> > Some Sage tickets require changes to sagenb or vice versa and with
> > option a), this coordination will certainly be harder.
> development repository into $SAGE_ROOT/devel/sagenb, though, as that
> makes the repository accessible to newcomers.
> bitbucket mirror athttps://bitbucket.org/jasongrout/sagenb
> have it now. The Sage release manager wouldn't merge patches into that
> repository or make spkgs for that repository, though.
As someone who would like to contribute to the NB more and has not
been able to do so much lately because of the separate model, I
strongly request making it possible to review material from sagenb
(and/or make change) in the current way, with hg_sagenb and so
forth. You don't have to use Mercurial internally, but make sure
it's possible for those of us primarily on the Sage end to continue
contributing meaningfully *and* efficiently. It looks like what you
are proposing would do this, but I just want to make sure. This would
particularly useful for getting people (i.e., students) who don't know
as much math but know about web stuff to contribute quickly and
easily, without having to do a separate project and then someone
figure out how to get it into Sage in order to actually test it.
- kcrisman
Jason Grout
> Bitbucket also has some of the nice features of github, like pull
> requests, comments on lines of code in pull requests, nice forking that
> is less burdensome, etc.
Apparently Bitbucket does not let you comment on lines in a commit (only
on the entire commit itself). Bummer. On the other hand, google *does*
let you comment on a specific line, but you have to formally start a
"code review". So that's one nice thing about google and github over
bitbucket. It still seems like a pain to deal with merging different
branches of development from different people using google code, though
(no pull requests, it's hard to set up a fork with just a few changes in
it to be pulled in, etc.)
Also, it looks like hg-review [1] is a very interesting take on
reviewing commits. Basically, it's the github-like commenting on lines,
files, or commits, along with a "sign off" field, but done in a
distributed way. You pull the commits, and then comment and review on
your own computer, and then push your comments and reviews whenever you
like. I like the idea of decentralizing the commenting and review. But
the project only has sporadic activity, so I don't know how supported
the implementation is.
Thanks,
Jason
Dima Pasechnik
Jason Grout
The easy way for you to submit one or two things is to just publish your
repository somewhere and tell us where it is. We can pull your changes.
Or you can email us a patch.
The easy way for us (and better way for you if you are submitting
multiple patches) is to:
1. Make a clone by going to
http://code.google.com/r/jasongrout-flask-sagenb/source/clones and
clicking "Create a clone" at the bottom (after you sign in)
2. Check out your clone: In your clone, go to the Source tab and follow
the instructions at the command line, something like:
hg clone https://<SOME URL>
2. Finding your google authentication key: In your clone, go to the
Source tab and click on "your generated googlecode.com password".
3. In your cloned repository on your own computer, edit the .hg/hgrc
file to look something like:
[paths]
default = https://<YOUR
USERNAME>:<YOUR_GOOGLECODE.COM_PASSWORD_FROM_ABOVE>@code.google.com/r/jasongrout-flask-sagenb/
4. Then you can make changes, commit them to your repository, and then do:
hg push
and it will push changes to your clone of the repository.
5. Then you need to send an email to me or sage-notebook saying that you
have changes that need to be reviewed and merged. We can pull from your
repository and merge the changes.
Step 5 is where things are easier with bitbucket or github. They have
"pull requests", where you just click a button and it sends a
notification to the project lead that changes need to be merged. It
also keeps track of comments on the commits, etc., and that in effect
because the issue ticket. Things are *way* nicer on github, since you
have pull requests *and* inline code comments. Plus, on github, you can
easily work on multiple features at once (each in a separate branch).
Each pull request is tied to a separate branch, so it is easy to isolate
and merge separate features, rather than having all the commits for
different features mixed up. Mercurial does branches too, but I haven't
found a nice way to deal with a pull request for each branch.
Again, I would prefer us moving to github and using git for the
notebook. I've been really impressed with the nice workflow that
matplotlib, scipy, numpy, ipython, and even our own singlecell server
have had using github to manage things.
Thanks,
Jason
kcrisman
> notebook. I've been really impressed with the nice workflow that
> matplotlib, scipy, numpy, ipython, and even our own singlecell server
> have had using github to manage things.
But could "regular" Sage users at least still use hg_sagenb (or just
hg in that repository) to review and create patches, and would there
be a place to post created patches and to review them for such people?
Dima Pasechnik
Fernando Perez
<jason...@creativetrax.com> wrote:
> Again, I would prefer us moving to github and using git for the notebook.
> I've been really impressed with the nice workflow that matplotlib, scipy,
> numpy, ipython, and even our own singlecell server have had using github to
> manage things.
Just as a data point on this front, supporting Jason's comment, that
may help inform your decision: switching to github is one of the best
decision we've made in the ipython project. The 0.12 release we put
out today closed a release cycle that lasted 4 1/2 months, and in
that period we were able to close 515 issues of which 257 were pull
requests (some with over 60 comments and multiple ongoing revisions),
totaling almost 1500 commits. For a very small team like IPython,
achieving this level of code throughput with any other tools would be
unthinkable, but github manages to make it reasonably easy.
Cheers,
f
William Stein
> On Mon, Dec 19, 2011 at 5:57 PM, Jason Grout
> <jason...@creativetrax.com> wrote:
>> Again, I would prefer us moving to github and using git for the notebook.
>> I've been really impressed with the nice workflow that matplotlib, scipy,
>> numpy, ipython, and even our own singlecell server have had using github to
>> manage things.
>
> Just as a data point on this front, supporting Jason's comment, that
> may help inform your decision: switching to github is one of the best
> decision we've made in the ipython project.
What did you switch from?
> The 0.12 release we put
> out today closed a release cycle that lasted 4 1/2 months, and in
> that period we were able to close 515 issues of which 257 were pull
> requests (some with over 60 comments and multiple ongoing revisions),
> totaling almost 1500 commits. For a very small team like IPython,
> achieving this level of code throughput with any other tools would be
> unthinkable, but github manages to make it reasonably easy.
So doing the same using Mercurial or Google Code or Bitbucket is in
your opinion unthinkable?
-- William
>
> Cheers,
>
> f
--
William Stein
Professor of Mathematics
University of Washington
http://wstein.org
Jason Grout
The hg_sage* commands assume a mercurial repository with a mercurial
interface, so I think it would be too much to try to make the underlying
repository a git repository and make everything seamless.
However, here's another idea: the hg-git plugin [1] makes it so that
mercurial can push/pull to/from git repositories. We could host
development on github, but use mercurial repositories in sagenb.
Here is what I really miss from github: having multiple features under
development simultaneously (without having to create a disk clone and
set up a new clone on the server for each different feature), be able to
submit each feature individually for inclusion using something like a
pull request (i.e., comments on the set of commits or lines in the
commit, as well as ability to add more commits if needed to finish the
feature before inclusion). On github, each feature is developed in a
git branch, and it is very easy to push or pull branches to/from github.
When a feature is ready for inclusion (which may include multiple
commits), a pull request is submitted for the branch. Any subsequent
commits are added automatically to the pull request. To merge, you just
pull from that branch and merge.
I know mercurial has tags, which are like the git branches I mention
above. However, does bitbucket or google code support pushing and
pulling to different "branches" using the tags, as well as submitting
just a certain tag (i.e., set of commits) for inclusion in the mainline?
Thanks,
Jason
Jason Grout
There are other people that have github experience around here too. For
example, Cython is on github---what do you guys think about it compared
to google code or bitbucket?
Thanks,
Jason
Tim Joseph Dumol
Tim Joseph Dumol <tim (at) timdumol (dot) com>
http://timdumol.com
Jason Grout
>
>
> As a user of Github, I've found that forking and issuing a pull request
> so easy that I've submitted patches to some projects, whereas if it was
> hosted on, say, Google Code, I probably wouldn't have bothered due to
> the hassle.
I've found the same thing true for myself. In fact, with in-place
editing on the github website, I've contributed grammatical fixes very
easily (to be fair, I think at least one of google code or bitbucket
also has inline editing and commiting of files).
Thanks,
Jason
Fernando Perez
> What did you switch from?
Launchpad and bazaar. While in principle they offer very similar
features, in practice it worked horribly. bzr is sluggish, once
corrupted a repository, and on multiple occasions we were unable to
work because of X11 problems. This is very strange on a version
control tool, eventually we tracked it down to a GTK plugin that was
globally installed on ubuntu, and that could crash bzr completely.
Basically bzr always felt brittle, slow and annoying. The launchpad
website used to be so slow (it may be better now, I don't know) that I
was in the habit of opening 10 or 20 tabs and initiating actions in
all of them, so that I could actually make some progress.
When we switched to github, it was really a breath of fresh air, and
people immediately jumped in who explicitly said they'd wanted to
contribute before but were too turned off by the bzr/lp combination.
>> The 0.12 release we put
>> out today closed a release cycle that lasted 4 1/2 months, and in
>> that period we were able to close 515 issues of which 257 were pull
>> requests (some with over 60 comments and multiple ongoing revisions),
>> totaling almost 1500 commits. For a very small team like IPython,
>> achieving this level of code throughput with any other tools would be
>> unthinkable, but github manages to make it reasonably easy.
>
> So doing the same using Mercurial or Google Code or Bitbucket is in
> your opinion unthinkable?
I can't really say because I don't have hands-on experience with them.
My *superficial* impression is that hg and git are by and large very
similar, although from what I've read it still appears that git's
branching model is in practice nicer (even if in principle they are
equivalent). But I don't want to badmouth hg without really knowing
(I have no problem talking about bzr b/c I suffered it for two years).
One thing I know is that google code's issue tracker is better than
the github one, or at least provides richer semantics:
http://code.google.com/p/support/wiki/IssueTracker
The github issue tracker is a bit bare-bones, but serviceable.
Where github shines is in the amazing fluidity of the fork / pull
request / review / merge workflow. They've really managed to reduce
the friction involved in both forking/submitting requests (for
contributors) and in reviewing/approving/merging (for core members
receiving the requests) to more or less zero. And the effect of that
is really noticeable in the long run.
Here's an example of an open pull request with active discussion:
https://github.com/ipython/ipython/pull/1129
It even shows at the bottom if it can be merged (basically whether
there will be merge conflicts). This one needs rebasing, but once
rebased that gray bar will become green like on this one:
https://github.com/ipython/ipython/pull/1139
At that point, when the committers decide to approve it, they can
simply click on the green 'Merge pull request' button, type a merge
message that summarizes the situation, and bingo. The merge commits
look like this:
https://github.com/ipython/ipython/commit/b24c9d28226120d0c4789f146e635029b6ec6c01
I don't really know whether bitbucket can match this level of
fluidity, though from the comments from others here it seems like
that's not the case.
I occasionally follow the python-dev list, and I get the impression
that they have spent a fair amount of energy in discussing and finding
a fluid worfklow for Python and that it's still somewhat complex and
doesn't look very nice. I think about how much this has become a
complete non-issue for us and I wonder if Python itself wouldn't be
better served on github... I've never said anything because this may
be a completely invalid impression I have as an outsider and that in
fact things work great for them, and I really don't want to trash hg.
So to give a short answer to your question: it *may* be possible to
achieve a similarly fluid and pleasant workflow on hg/google
code/bitbucket, but from the outside I get the impression it's not
quite there. And since github exists now, and is awesome in just
about every respect (I'd *really* like pull requests to accept labels
and other metadata) , I'm happy to stay there...
I think feedback from Robert Bradshaw would be very useful: they had
extensive experience with hg and switched to github (partly based on
my feedback about it at the time). Their data point is probably one
of the most important you could get for Sage.
Cheers,
f
Fernando Perez
>>> unthinkable, but github manages to make it reasonably easy.
>>
>> So doing the same using Mercurial or Google Code or Bitbucket is in
>> your opinion unthinkable?
BTW, I realize now that you may have been asking for a precision in my
strongly worded statement, and I didn't really answer that. What I
mean by that statement is that such throughput would be unthinkable
with any other tools that didn't offer a similar 'zero friction'
worfklow to review/discuss/integrate external contributions. If
hg+{bitbucket, google} can do that, then they would work similarly. I
certainly *know* that bzr/launchpad are a million miles from meeting
this requirement. And it seems to me that the hg combo isn't quite
there either, but I could easily be wrong on that.
I hope that helps clarify my intent, I'd be happy to answer further questions...
Best,
f
Jason Grout
I was just experimenting with bitbucket pull requests to see if I could
duplicate the github branching workflow. I'm using hg bookmarks as the
git branch equivalents (they are pretty much equivalent; hg branches are
totally different than git branches).
Pro: bitbucket recognizes hg bookmarks as "branches" in the interface
Con: every time I create a new branch and try pushing it to bitbucket,
hg complains about pushing multiple heads to bitbucket, which I have to
override by doing hg push -f. Then I have to explicitly push the branch
using hg push -B branchname.
I then tried to fork the repository and submit a pull reequest from a
bookmark.
Pro: I can own multiple forks of a repository (github only allows one
fork per account, which is silly)
Con: While I can submit a pull request from any 'head', it doesn't show
me the bookmarks associated with the heads, so I have to guess which
commit is the right bookmark.
Con: Even when I try to submit a pull request from the fork to the
original repository, it says: Unfortunately this pull request cannot be
automatically pulled and merged until all the heads on the destination
branch have been merged. (see
https://bitbucket.org/jasongrout/test-bitbucket-tags/pull-request/1/new-p).
I don't see any obvious way to merge these changes into the mainline
as-is. On the other hand, github provides a nice address to explicitly
pull in the pull request changes.
Pro: I can comment on the entire pull request (nested too!)
Con: I can't comment on individual lines of code
So, my conclusion is that there is enough overhead to make the process
frustrating. Of course, I hope that I've just missed some things and
the process is really easier than what I've discovered.
Another big +1 for git over hg: it is very easy to track remote
repositories explicitly, get all the changes from a remote repository
downloaded, and then work with remote repositories offline---merging, etc.
Thanks,
Jason
Fernando Perez
<jason...@creativetrax.com> wrote:
> Pro: I can own multiple forks of a repository (github only allows one fork
> per account, which is silly)
Mhh, it had never occurred to me that this would be useful, given how
easily I can keep multiple branches in one repo. What is your use
case for this?
> So, my conclusion is that there is enough overhead to make the process
> frustrating. Of course, I hope that I've just missed some things and the
> process is really easier than what I've discovered.
Well, this is actually very informative, thanks for going through the
exercise. Perhaps someone with good bitbucket-fu can point to
solutions/improvements on these points.
> Another big +1 for git over hg: it is very easy to track remote repositories
> explicitly, get all the changes from a remote repository downloaded, and
> then work with remote repositories offline---merging,
I really like this ability of git. I regularly track the remotes of
all the core ipython devs, and it's great to have this single point of
control over the entire project. I can merge anyone's branches,
compare them, make PRs against one of them if I want, etc. I don't
know if hg supports something like git's model for remotes, but if it
doesn't, that would definitely be a major point against it in my book.
Cheers,
f
William Stein
<jason...@creativetrax.com> wrote:
> I was just experimenting with bitbucket pull requests to see if I could
> duplicate the github branching workflow.
For what it is worth, I was just discussing this situation with Mike
Hansen on gmail chat. Some concerns:
1. It would be weird for SAGE_ROOT/devel/sage-main to be under hg
control, but SAGE_ROOT/devel/sagenb to be under git control. This
will definitely confuse some potential developers (kcrisman?).
2. With sage, we ship hg, but we don't ship git. I suspect git is
not trivial to install on some generic machine "out there" (just as hg
isn't). This is perhaps less of an issue for the sage notebook, since
one is much more likely to be working on the core sage library on a
remote machine.
3. It is important to keep in mind the difficult people have in
learning multiple tools just to contribute to a project. For Jason
"use git" is trivial, since he already knows it, but for other people
there will be a substantial learning curve. (That said, in my
experience contributors to the Sage notebook are often more
technically savvy than typical contributors to Sage.)
One way to mitigate some of the above concerns, which Mike Hansen
suggested, is to use http://hg-git.github.com/. We could make it so
SAGE_ROOT/devel/sagenb would still be an hg repo, and for
casual/occasional contributors (the sort I'm worried about in 3
above), they can just use hg and post a patch somewhere. However, the
core sagenb repo would use git + github. People who work a lot on
sagenb would be assumed to know git well and use github. One worry
is that the above webpage says "This plugin is known to work on
Mercurial 1.7, and is basically functional (with a couple of bugs)
against hg 1.8 and 1.9." We have hg 1.8.4 in sage.
-- William
Jason Grout
> On Mon, Dec 19, 2011 at 9:51 PM, Jason Grout
> <jason...@creativetrax.com> wrote:
>> Pro: I can own multiple forks of a repository (github only allows one fork
>> per account, which is silly)
>
> Mhh, it had never occurred to me that this would be useful, given how
> easily I can keep multiple branches in one repo. What is your use
> case for this?
Testing how forking and pull requests work :).
Yes, that's how I answer that Con when I put on my github hat too. So
maybe it's not much of a con, but it still seems pretty silly, given
that it is trivial for me to create an on-disk clone.)
>
>> So, my conclusion is that there is enough overhead to make the process
>> frustrating. Of course, I hope that I've just missed some things and the
>> process is really easier than what I've discovered.
>
> Well, this is actually very informative, thanks for going through the
> exercise. Perhaps someone with good bitbucket-fu can point to
> solutions/improvements on these points.
>
>> Another big +1 for git over hg: it is very easy to track remote repositories
>> explicitly, get all the changes from a remote repository downloaded, and
>> then work with remote repositories offline---merging,
>
> I really like this ability of git. I regularly track the remotes of
> all the core ipython devs, and it's great to have this single point of
> control over the entire project. I can merge anyone's branches,
> compare them, make PRs against one of them if I want, etc. I don't
> know if hg supports something like git's model for remotes, but if it
> doesn't, that would definitely be a major point against it in my book.
I don't think it does. Here is a very interesting read from the guy who
developed flask, werkzeug, jinja, and a bunch of other very popular
python tools:
http://lucumr.pocoo.org/2010/8/17/git-and-mercurial-branching/. He
claims that hg does not (really) support remote tracking.
Thanks,
Jason
Jason Grout
> On Mon, Dec 19, 2011 at 9:51 PM, Jason Grout
> <jason...@creativetrax.com> wrote:
>> I was just experimenting with bitbucket pull requests to see if I could
>> duplicate the github branching workflow.
>
>
> For what it is worth, I was just discussing this situation with Mike
> Hansen on gmail chat. Some concerns:
>
> 1. It would be weird for SAGE_ROOT/devel/sage-main to be under hg
> control, but SAGE_ROOT/devel/sagenb to be under git control. This
> will definitely confuse some potential developers (kcrisman?).
>
> 2. With sage, we ship hg, but we don't ship git. I suspect git is
> not trivial to install on some generic machine "out there" (just as hg
> isn't). This is perhaps less of an issue for the sage notebook, since
> one is much more likely to be working on the core sage library on a
> remote machine.
>
> 3. It is important to keep in mind the difficult people have in
> learning multiple tools just to contribute to a project. For Jason
> "use git" is trivial, since he already knows it, but for other people
> there will be a substantial learning curve. (That said, in my
> experience contributors to the Sage notebook are often more
> technically savvy than typical contributors to Sage.)
Those three reasons are exactly the reasons why I almost didn't have
hope that a switch might happen. I like your points that these may not
be as much of a problem for heavy sagenb contributors.
> One way to mitigate some of the above concerns, which Mike Hansen
> suggested, is to use http://hg-git.github.com/. We could make it so
> SAGE_ROOT/devel/sagenb would still be an hg repo, and for
> casual/occasional contributors (the sort I'm worried about in 3
> above), they can just use hg and post a patch somewhere. However, the
> core sagenb repo would use git + github. People who work a lot on
> sagenb would be assumed to know git well and use github. One worry
> is that the above webpage says "This plugin is known to work on
> Mercurial 1.7, and is basically functional (with a couple of bugs)
> against hg 1.8 and 1.9." We have hg 1.8.4 in sage.
I mentioned this as well in my other post in response to Karl-Dieter.
I've tried to use hg-git on several projects for the other direction (I
wanted to do my development in git, but the main repositories for the
singlecell or another project were in hg, so I would pull to a hg
repository, then push to github using hg-git, then pull from github
using git). Things kept getting too confusing in trying to maintain the
sync between the git and hg sides (which was probably just me not
understanding things as well as I should have, more than the tools being
faulty...). However, I was trying to sync up lots of development with
multiple branches, etc. Maybe for just a casual contributor who wants
to edit one file and post a patch somewhere, it would be simple and easy.
Thanks,
Jason
William Stein
Well I do think we should seriously consider the possibility of a
switch for sagenb.
William
Fernando Perez
<jason...@creativetrax.com> wrote:
> Yes, that's how I answer that Con when I put on my github hat too. So maybe
> it's not much of a con, but it still seems pretty silly, given that it is
> trivial for me to create an on-disk clone.)
I think that's actually a restriction I *like*: I like knowing that
the naming model github.com/user/repo and knowing automatically how
the fork queue maps, without having to keep track of possibly multiple
forks per user. Since functionally git branches can solve the problem
of keeping many development ideas in flight at the same time, I think
the simpler model in this case is a good thing.
> I don't think it does. Here is a very interesting read from the guy who
> developed flask, werkzeug, jinja, and a bunch of other very popular python
> tools: http://lucumr.pocoo.org/2010/8/17/git-and-mercurial-branching/. He
> claims that hg does not (really) support remote tracking.
Excellent post, thanks for pointing it out! He makes a number of
excellent points: I've also missed the simplicity of 'hg serve' as
years ago it proved to be super useful to me in the same scenario he's
describing (a sprint with poor external connectivity but good local
one). And the syntax for deleting a remote branch in git can make
sense once you understand what it's doing, yet it always feels a bit
absurd.
But the descriptions in that post reinforce my perception that in the
long run git is really the better tool for collaborative development.
One thing that I really like about git, despite the often impenetrable
error messages and UI conventions, is that the underlying model is of
a brutal simplicity. Git has a handful of very simple concepts all of
which can be explained as operations on a DAG, so for anyone who has
the most basic mental model of elementary graph theory, git is really
easy to understand. Once I started thinking about git that way (this
article helped me a lot
http://www.eecs.harvard.edu/~cduan/technical/git), pretty much
everything in git made sense.
Obviously Sage has a very solid workflow already that seems to work
pretty well for everyone, but for us the git/gh combo has really been
very useful.
Cheers,
f
Ralf Hemmecke
> years ago it proved to be super useful to me in the same scenario he's
> describing (a sprint with poor external connectivity but good local
> one).
Just for the record...
Let's say your normal apache installation serves files from your
$HOME/www directory. Then doing
cd $HOME/www
umask 022
git clone --bare SomeRepo
cd SomeRepo.git
git update-server-info
let's everyone clone via http from a URL where $HOME/www is served.
For example http://example.com/myname/SomeRepo.git
Or simply have your repo and checkout under $HOME/www and run 'git
update-server-info' in the post-commit hook.
URL would then be something like
http://example.com/myname/SomeRepo/.git
(Note the slash before .git.)
git doesn't need a separate http server.
Anyway, you can easily make your repo available via ssh. And if you want
to give others write access to a repo on your computer, use gitolite.
That all are options that do not require an extra daemon to be running
(except for the standard httpd and sshd).
> One thing that I really like about git, despite the often impenetrable
> error messages and UI conventions, is that the underlying model is of
> a brutal simplicity.
I've found this article quite helpful.
http://newartisans.com/2008/04/git-from-the-bottom-up/
Ralf
kcrisman
> control, but SAGE_ROOT/devel/sagenb to be under git control. This
> will definitely confuse some potential developers (kcrisman?).
True.
> 3. It is important to keep in mind the difficult people have in
> learning multiple tools just to contribute to a project. For Jason
> "use git" is trivial, since he already knows it, but for other people
> there will be a substantial learning curve. (That said, in my
> experience contributors to the Sage notebook are often more
> technically savvy than typical contributors to Sage.)
Yes, definitely true.
> I mentioned this as well in my other post in response to Karl-Dieter.
> I've tried to use hg-git on several projects for the other direction (I
> wanted to do my development in git, but the main repositories for the
> singlecell or another project were in hg, so I would pull to a hg
> repository, then push to github using hg-git, then pull from github
> using git). Things kept getting too confusing in trying to maintain the
> sync between the git and hg sides (which was probably just me not
> understanding things as well as I should have, more than the tools being
> faulty...). However, I was trying to sync up lots of development with
> multiple branches, etc. Maybe for just a casual contributor who wants
> to edit one file and post a patch somewhere, it would be simple and easy.
I think that is the type of user I'm referring to. Especially when
there are things that require patches to both Sage and sagenb (as has
been the case in the past), you would want someone to be able to
contribute, just in the sense of posting a patch. No commit access
needed. Same thing for *reviewing* a patch - it should be made easy
for "casual contributor A" to do something basically the same as we do
now for sage-main.
There is a meta-question underlying all this. The closer you get to
doing releases with pull and push stuff, as I perceive it (which could
be totally wrong, but so it seems), the closer we get to sagenb having
a "commit privilege" pool, as projects like Maxima, or probably most
SF projects (?) have. The quasi-democratic nature of Sage is a very
good thing, particular with such a huge number of developers doing
things from sage-combinat to number theory to plotting to build stuff,
often quite orthogonally.
Now, sagenb is (as you point out) definitely going to attract the more
savvy developer who knows a lot more about these tools, but once the
"new sagenb" is merged into Sage, there should still be some obvious
way that things are reviewed and merged that makes the "paper" (!)
trail clear. For now I think it's good that Jason et al. can make
quick changes, but the fact that each commit in Sage has at least two/
three people look at it before it goes in is very wise.
But for the hard-core developer of sagenb, as well as for attracting
*non*-mathematicians to work on it, this whole thing probably makes a
lot of sense.
- kcrisman
Jason Grout
> There is a meta-question underlying all this. The closer you get to
> doing releases with pull and push stuff, as I perceive it (which could
> be totally wrong, but so it seems), the closer we get to sagenb having
> a "commit privilege" pool, as projects like Maxima, or probably most
> SF projects (?) have. The quasi-democratic nature of Sage is a very
> good thing, particular with such a huge number of developers doing
> things from sage-combinat to number theory to plotting to build stuff,
> often quite orthogonally.
>
The release push/pull thing you mention above is just an extension of
the current release manager model. By convention, you could have a
single person responsible for merging in the master branch (the release
manager), or you could have multiple people that coordinate together
(which would be much easier than in our current single release manager
model, since there would be a public current master tree)
> Now, sagenb is (as you point out) definitely going to attract the more
> savvy developer who knows a lot more about these tools, but once the
> "new sagenb" is merged into Sage, there should still be some obvious
> way that things are reviewed and merged that makes the "paper" (!)
> trail clear. For now I think it's good that Jason et al. can make
> quick changes, but the fact that each commit in Sage has at least two/
> three people look at it before it goes in is very wise.
The github pull requests, while not as flexible as trac tickets (e.g.,
they don't have custom database queries, trac plugins like dependencies,
custom fields, etc.), are definitely an easier workflow, and do provide
the required paper trail. In fact, in some ways, they provide better
workflow than trac (e.g., per-line comments, complete history of commits
in the ticket).
Also, I see moving to git a very useful experiment. I consider moving
the singlecell to github to be a success. Sagenb would be a bigger
project that would provide useful experience in evaluating how github or
github-type tools might improve our workflow.
Thanks,
Jason
Ralf Hemmecke
> *non*-mathematicians to work on it, this whole thing probably makes a
> lot of sense.
Good to hear.
1) When deciding to switch from one SCM to another most people fear that
the project could lose some people. Maybe there's also a chance to win
some new developers by the transition.
2) A personal question: is the sage notebook also working without sage?
Some time ago, I experimented with it for its use as a frontend to
FriCAS, but having to carry around all of Sage, basically made the
sagenb project uninteresting. I haven't followed the development closely
... is it still a goal to decouple sagenb from sage?
Ralf
Jason Grout
Not very much depends on Sage, I don't think. A quick grep through the
source code for when we import sage shows that we pretty much just use
some small miscellaneous functions from sage, which could easily be
packaged as a module for sagenb.
Thanks,
Jason
Ralf Hemmecke
> the source code for when we import sage shows that we pretty much
> just use some small miscellaneous functions from sage, which could
> easily be packaged as a module for sagenb.
Ah, that very much makes it again interesting.
There were other projects some time ago like CodeNode
(http://codenode.org/) and the notebook from FemHub
(https://github.com/certik/femhub).
Codenode seems to be dead, but what's the relation to the FemHub
notebook. Femhub seemed once pretty standalone
http://axiom-portal.newsynthesis.org/Members/hemmecke/fricas-notebook
but I had problems in actually running FriCAS in that way.
I think femhub was forked from sagenb, but is there some code exchange?
Anyway, any hint that helps me to use sagenb as a frontend to fricas is
welcome.
How, for example, can I install sagenb in a standalone fashion (i.e.
without the need of sage)? Is there some INSTALL file for that use case?
Of course, I would then also need a description of how to connect a
different CAS to the notebook cells.
Ralf
PS: In particular, making a separate github project out of it seems very
attractive (at least to me).
Jason Grout
>> Not very much depends on Sage, I don't think. A quick grep through
>> the source code for when we import sage shows that we pretty much
>> just use some small miscellaneous functions from sage, which could
>> easily be packaged as a module for sagenb.
>
> Ah, that very much makes it again interesting.
>
> There were other projects some time ago like CodeNode
> (http://codenode.org/) and the notebook from FemHub
> (https://github.com/certik/femhub).
>
> Codenode seems to be dead, but what's the relation to the FemHub
> notebook. Femhub seemed once pretty standalone
> http://axiom-portal.newsynthesis.org/Members/hemmecke/fricas-notebook
> but I had problems in actually running FriCAS in that way.
> I think femhub was forked from sagenb, but is there some code exchange?
>
>
> Anyway, any hint that helps me to use sagenb as a frontend to fricas is
> welcome.
>
> How, for example, can I install sagenb in a standalone fashion (i.e.
> without the need of sage)? Is there some INSTALL file for that use case?
sagenb is just a normal python package. You can use
python setup.py install
to install it. For example:
hg clone https://code.google.com/r/jasongrout-flask-sagenb/ sagenb-flask
cd sagenb-flask
python setup.py install
python setup.py develop
that *should* install all the necessary dependencies (though I haven't
tested it on a non-sage install).
There will be errors, of course, in the places where we import
miscellaneous sage functions. But the above is how to start.
>
> Of course, I would then also need a description of how to connect a
> different CAS to the notebook cells.
Look at the sagenb/interfaces directory for how Sage is hooked up:
http://code.google.com/r/jasongrout-flask-sagenb/source/browse/#hg%2Fsagenb%2Finterfaces
>
> Ralf
>
> PS: In particular, making a separate github project out of it seems very
> attractive (at least to me).
Great; I'll count that as a +1.
Jason
Jason Grout
> Well I do think we should seriously consider the possibility of a
> switch for sagenb.
By the way, I was just searching for a way to download just a raw patch
from google code (in github or bitbucket, you just append .patch or
something like that to the URL and you get a patch file). Well,
apparently William has run into this before:
http://groups.google.com/group/psage/browse_thread/thread/2eb280c938d6844f?pli=1
So that's another Pro for trying bitbucket or github.
Jason
Keshav Kini
>doing releases with pull and push stuff, as I perceive it (which could
>be totally wrong, but so it seems), the closer we get to sagenb having
>a "commit privilege" pool, as projects like Maxima, or probably most
>SF projects (?) have. The quasi-democratic nature of Sage is a very
>good thing, particular with such a huge number of developers doing
>things from sage-combinat to number theory to plotting to build stuff,
>often quite orthogonally.
But, importantly, even while the release management might remain basically the same, interactions between individual developers can become a lot richer and easier through pull/push and pull requests, since everything is a lot more seamless thanks to github's architecture having all this stuff built into the interface. So on the whole I think moving to a pull request system would make Sage development more open (other than adding the small participation barrier of "please sign up for an account at github.com"). As Jason alluded to somewhere else in this thread, on github you can even edit the source code of a project from the github webpage itself, which automatically creates a commit and a pull request! So for example we could get a lot more typo-fix / documentation / one-liner contributions from random passersby.
Anyway, I'd just like to jump in and give a big +1 to the idea of moving to GitHub. I have barely started using it but I already see what Fernando is talking about. And git itself is a great tool, I'm beginning to discover. In fact I'm starting to prefer it to Mercurial, for whatever my opinion's worth. (See #sagemath logs for a discussion I had with Dan Drake about it a little earlier).
-Keshav
----
Join us in #sagemath on irc.freenode.net !
kcrisman
On Dec 20, 10:50 pm, Keshav Kini <keshav.k...@gmail.com> wrote:
> >There is a meta-question underlying all this. The closer you get to
> >doing releases with pull and push stuff, as I perceive it (which could
> >be totally wrong, but so it seems), the closer we get to sagenb having
> >a "commit privilege" pool, as projects like Maxima, or probably most
> >SF projects (?) have. The quasi-democratic nature of Sage is a very
> >good thing, particular with such a huge number of developers doing
> >things from sage-combinat to number theory to plotting to build stuff,
> >often quite orthogonally.
>
> I'd like to elaborate a bit on what Jason said. There is nothing inherent
> in the push/pull system which creates a "committer privilege" pool. There
> is a trust model, basically. Each person pulls from people he trusts, and
> the same applies for the release manager. But looking at it more closely,
> I trust". So basically it can really be the same thing as we are doing on
> trac right now.
process remains in place for what ends up in Sage in terms of having
reviewers and such; I've seen so many problems on other lists that
shall remain nameless with "commit privilege" situations with bugs
that didn't need to happen.
> But, importantly, even while the release management might remain basically
> the same, interactions between individual developers can become a lot
> richer and easier through pull/push and pull requests, since everything is
> a lot more seamless thanks to github's architecture having all this stuff
> built into the interface. So on the whole I think moving to a pull request
> github.com"). As Jason alluded to somewhere else in this thread, on github
account while free? At any rate bitbucket has something like that.
I'm probably misinterpreting something.
> you can even edit the source code of a project from the github webpage
> itself, which automatically creates a commit and a pull request! So for
> example we could get a lot more typo-fix / documentation / one-liner
> contributions from random passersby.
mirror of the Sage library there for that potential alone... right now
that kind of thing tends to be more tedious than its worth.
- kcrisman
Keshav Kini
> Not that I understand the details, having not used it. As long as the
> process remains in place for what ends up in Sage in terms of having
> reviewers and such; I've seen so many problems on other lists that
> shall remain nameless with "commit privilege" situations with bugs
> that didn't need to happen.
Yup, shouldn't be a problem.
> Hmm, isn't there some kind of limit on number of developers on an
> account while free? At any rate bitbucket has something like that.
> I'm probably misinterpreting something.
Nope. On github, every user has their own separate account, and every
user can participate in any public repository on Github without
restrictions. It's only when private repos come into the picture that
you start getting these limits (which is how github makes money).
Github is really nice in that respect - the only limitations for free
users are basically 1) no private repos; 2) if you use up too much
disk space, the admins may come knocking asking what you're doing,
though if it's nothing malicious / frivolous they generally let you
use as much disk space as you need, I hear. And the default disk space
limit is 300 MB so we're well within that limit anyway when it comes
to sagenb, I'm sure. Even all the main Sage repos combined, including
all the abortive development version branches that Jeroen's script
produces (not including SPKGs though), have a footprint of <150MB in
Mercurial at the moment.
So, in a nutshell, it's pretty much totally free to use for
open-source free software.
>> you can even edit the source code of a project from the github webpage
>> itself, which automatically creates a commit and a pull request! So for
>> example we could get a lot more typo-fix / documentation / one-liner
>> contributions from random passersby.
>
> Wow, that would be stupendous. Almost would make it worth hosting a
> mirror of the Sage library there for that potential alone... right now
> that kind of thing tends to be more tedious than its worth.
Yes, I agree! Though ideally it wouldn't even be a mirror, because
then we'd need someone to translate pull requests on github into trac
tickets and then get them reviewed, which goes back to the current
"more tedious than it's worth" process. One day, perhaps. For now,
it's baby steps with sagenb :)
Jeroen Demeyer
Suppose I make a change to Sage which also requires a change to sagenb
(say, a doctest). What would be the procedure for that?
Jason Grout
It's clear that we still need close communication between the projects.
We'd probably handle it similar to pynac right now. Say trac #30000
and feature X in sagenb depend on each other. Then we'd open an issue
on github for adding feature X, add it, and produce a new spkg. That
spkg would then be attached to #30000, or maybe the spkg update would be
#30001, and #30000 would depend on #30001.
Does that sound reasonable?
Jason
Jeroen Demeyer
different from sage, you should add instructions on how to contribute to
sagenb:
Robert Bradshaw
<jason...@creativetrax.com> wrote:
> On 12/20/11 12:10 AM, William Stein wrote:
>>
>> On Mon, Dec 19, 2011 at 9:51 PM, Jason Grout
>> <jason...@creativetrax.com> wrote:
>>>
>>> I was just experimenting with bitbucket pull requests to see if I could
>>> duplicate the github branching workflow.
>>
>>
>>
>> For what it is worth, I was just discussing this situation with Mike
>> Hansen on gmail chat. Some concerns:
>>
>> 1. It would be weird for SAGE_ROOT/devel/sage-main to be under hg
>> control, but SAGE_ROOT/devel/sagenb to be under git control. This
>> will definitely confuse some potential developers (kcrisman?).
I agree, though it's already confusing to have two separate, parallel
repositories.
>> 2. With sage, we ship hg, but we don't ship git. I suspect git is
>> not trivial to install on some generic machine "out there" (just as hg
>> isn't). This is perhaps less of an issue for the sage notebook, since
>> one is much more likely to be working on the core sage library on a
>> remote machine.
>>
>> 3. It is important to keep in mind the difficult people have in
>> learning multiple tools just to contribute to a project. For Jason
>> "use git" is trivial, since he already knows it, but for other people
>> there will be a substantial learning curve. (That said, in my
>> experience contributors to the Sage notebook are often more
>> technically savvy than typical contributors to Sage.)
I found git trivial to learn given knowledge of mercurial. I'd expect
the demographic of those working on the sage notebook will be similar
(or at least this will be less of an issue).
> Those three reasons are exactly the reasons why I almost didn't have hope
> that a switch might happen. I like your points that these may not be as
> much of a problem for heavy sagenb contributors.
+1
>> One way to mitigate some of the above concerns, which Mike Hansen
>> suggested, is to use http://hg-git.github.com/. We could make it so
>> SAGE_ROOT/devel/sagenb would still be an hg repo, and for
>> casual/occasional contributors (the sort I'm worried about in 3
>> above), they can just use hg and post a patch somewhere. However, the
>> core sagenb repo would use git + github. People who work a lot on
>> sagenb would be assumed to know git well and use github. One worry
>> is that the above webpage says "This plugin is known to work on
>> Mercurial 1.7, and is basically functional (with a couple of bugs)
>> against hg 1.8 and 1.9." We have hg 1.8.4 in sage.
>
>
> I mentioned this as well in my other post in response to Karl-Dieter. I've
> tried to use hg-git on several projects for the other direction (I wanted to
> do my development in git, but the main repositories for the singlecell or
> another project were in hg, so I would pull to a hg repository, then push to
> github using hg-git, then pull from github using git). Things kept getting
> too confusing in trying to maintain the sync between the git and hg sides
> (which was probably just me not understanding things as well as I should
> have, more than the tools being faulty...). However, I was trying to sync
> up lots of development with multiple branches, etc. Maybe for just a casual
> contributor who wants to edit one file and post a patch somewhere, it would
> be simple and easy.
Yeah, that sounds dangerous.
In terms of development flow, the process of manually
importing/exporting patches and attaching them to webforms is fairly
archaic and cumbersome. The value in the github pull request model is
that it gets (almost) all the technical stuff out of the way and lets
you focus on coding and reviewing, i.e. the interesting and rewarding
part of development. Rather than hg_sage... I'd like to see something
like.
1) [hack, hack, hack]
2) sage --push_ticket [id]
3) sage --pull_ticket id
These may be interactive, verifying the local state is in sync with
"trac," asking for descriptions or dependencies, running tests, etc.
The fact that there's a revision controls system, git or hg, would
only be visible to advanced users who would be free to use whatever
tools they wanted, with the above mapping to a refresh/commit/push or
a pull to an online repo in a very transparent manner. Imagine a git
repo with an up-to-date branch for every open ticket... A pull request
would be a formal request for review, allowing for line-level
commenting or someone to build on that patch. If we went with an
external service like github where the above "simple" commands could
even push raw patches to a proxy to avoid requiring an account (though
that's been a non-issue for us, and I like having someone else do the
account management/spam control.) For something like sagenb,
essentially all users would be "advanced" and so you wouldn't really
need this, but it'd be good for the Sage core. Of course being able to
try things out/edit/add doctests (e.g. directly from your test
session) online would be icing on the cake, but that's a bigger leap
(and less co-opting an existing system).
In terms of Cython, I think our contributions would have been
drastically reduced if it weren't for the ease of submitting a pull
request and reviewing it/merging it in. IIRC, bitbucket had missing
features, and one developer had strong opinions about Google code not
being available in Cuba due to US trade restrictions, but in
retrospect we have been very happy with the move to github. I have to
say I've grown to like git, but it is a steeper learning curve than
hg, and "hg serve" is really nice.
Sage and its notebook may still be tightly coupled, but I think
they're distinct enough components that there's more to gain by
letting the core developers work in a more streamlined fashion (if
they find this to be the case) than there is to loose by going with a
different tool.
- Robert
Keshav Kini
This is beautiful, big +1 from me (though it seems like a far-off
goal). It seems to me that hg_sage sits in an awkward position where
it is neither as powerful as command line hg nor much easier to use
than command line hg. (Maybe someone who uses it can give their
opinion? I don't use it.) A `sage --push` and `sage --pull` would be a
much better system.
If we did use github in particular, and let each developer have their
own fork on their own github account, I guess we would need to
integrate into trac some sort of system for looking for branches on
people's forks and maybe displaying links to them on the trac ticket
page, since the trac ticket is (and should be) a central place for
multiple people working on the ticket to discuss what they're doing.
That way, people could be aware of and pull from each other before
submitting a pull request. `sage --pull` and `sage --push` could be
aware of this too and display a message if branches have diverged.
Once a pull request is submitted, only the person who submitted the
pull request can any longer make any changes to the code, which is a
bit different from how our trac workflow is now. But if that person
drops off the map, one can always open a new pull request, I guess.
The release manager's workflow would change drastically if we moved to
github, of course. I think we should still ask the developers on a
ticket to rebase (or merge) their changes with the latest dev build
themselves, and I think this would in fact be a bit easier with git
(or standard mercurial), rather than mercurial queues. Many people
still seem to be rebasing patches by hand, which is much more tedious
than it need be.
> Of course being able to
> try things out/edit/add doctests (e.g. directly from your test
> session) online would be icing on the cake, but that's a bigger leap
> (and less co-opting an existing system).
Could you explain what you mean by this?
> Sage and its notebook may still be tightly coupled, but I think
> they're distinct enough components that there's more to gain by
> letting the core developers work in a more streamlined fashion (if
> they find this to be the case) than there is to loose by going with a
> different tool.
Right, and I believe the goal is to modularize more in the future anyway.
Jeroen Demeyer
> The release manager's workflow would change drastically if we moved to
> github, of course.
certainly not a drastic change.
Keshav Kini
Well, I meant that for example your current merger scripts would need
to be rehauled unless you want to be stuck with something like the
current hg.sagemath.org which just shows a list of trac tickets that
were fixed in each release instead of a real organic development
history. Maybe it's not as drastic as I'm imagining, though,
especially for someone with your expertise in cooking up scripts :)
Jason Grout
Not drastic, but probably not smooth either. Github is missing:
* workflow states
* attachments to tickets
* attach code to tickets (often, an issue is opened, then an identical
pull request with the code, and then both have to be closed when the
pull request is merged)
* easily merge code from multiple people in a pull request (unlike
different people submitting patches on a ticket)
* custom fields (e.g., dependencies)
Jason
William Stein
It's easier to use hg_sage than the command line hg for people who don't know how to use the command line. That's probably 99% of potential Sage users; certainly, at least 90% of students who took my Sage class (at the beginning). Almost nobody uses/knows the unix command prompt, except computer geeks. I didn't even know how to use it until I took a class on UNIX when I was an undergrad (though back then, a reason I didn't know the command line was that it was very difficult to get access to -- Linux didn't exist, and even when it did, installing it meant downloading dozen of floppies). Today access is also not totally automatic, since most people use Microsoft Windows.
In the notebook or sage prompt (say), one can do hg_sage.commit('commit message') then hg_sage.export('foo.patch'), and get a link to a patch file. You can also do "hg_sage.[tab]" and get just the minimal commands you need to know about to make a patch, with simple documentation.
I never use hg_sage myself, except when helping "newbies"... who really appreciate that I understand well what it feels like to be very ignorant (hence why I wrote hg_sage).
> A `sage --push` and `sage --pull` would be a
> much better system.
I do not see that as a "better system" -- it's just something else that would also be useful to have. It would also mean a command such as "hg_sage.push(...)" and "hg_sage.pull(...)" would be easy to implement.
William
Robert Bradshaw
My feelings exactly, though it does offer tab completion and
sage-centric documentation and the ability to operate without leaving
Sage which is an advantage for non-experts and the simple case. (But
it should be even easier!)
> A `sage --push` and `sage --pull` would be a much better system.
> If we did use github in particular, and let each developer have their
> own fork on their own github account, I guess we would need to
> integrate into trac some sort of system for looking for branches on
> people's forks and maybe displaying links to them on the trac ticket
> page, since the trac ticket is (and should be) a central place for
> multiple people working on the ticket to discuss what they're doing.
It's easy to post links to branches on github, and to discover them
(especially if there's a convention of naming them after the ticket
number). Easier in fact than attaching (and re-attaching, and
re-attaching) a patch. Pointers to code in a live repository are a
better solution than keeping the code on trac, and then there's no
ambiguity about what patches to attach in what order, what
dependencies are needed, what version of Sage it's based on, etc.
(This an example of context that we lose/manually track by leaving the
version control system in our current setup.)
> That way, people could be aware of and pull from each other before
> submitting a pull request. `sage --pull` and `sage --push` could be
> aware of this too and display a message if branches have diverged.
>
> Once a pull request is submitted, only the person who submitted the
> pull request can any longer make any changes to the code, which is a
> bit different from how our trac workflow is now. But if that person
> drops off the map, one can always open a new pull request, I guess.
>
> The release manager's workflow would change drastically if we moved to
> github, of course.
Hopefully it shouldn't. If the release manager is spending a lot of
time with the tools rather than the higher level tasks of coordinating
what tickets and spkgs should go in and verifying sufficient testing
on a wide variety of platforms then I'd say we need better tools.
> I think we should still ask the developers on a
> ticket to rebase (or merge) their changes with the latest dev build
> themselves, and I think this would in fact be a bit easier with git
> (or standard mercurial), rather than mercurial queues. Many people
> still seem to be rebasing patches by hand, which is much more tedious
> than it need be.
Yes. One could easily see what branches are up to date.
>> Of course being able to
>> try things out/edit/add doctests (e.g. directly from your test
>> session) online would be icing on the cake, but that's a bigger leap
>> (and less co-opting an existing system).
>
> Could you explain what you mean by this?
Imagine you go to a ticket, and there's a link to a sage notebook
worksheet with this patch already applied and built. So as well as
inspecting the code, you can try it out right there. You do some
examples, and as you're stress-testing the code you think "this might
make a good doctest" so you clean it up and click on a link to the
side that lets you add that cell to the doctest of a function of your
choice (perhaps bringing up the docstring with your additions appended
in an editable window.) When you're all done, you can click "Save" and
it will create a referee commit on top of the existing ticket adding
your new examples. (As well as doctests, being able to edit the code
as well as make line-by-line changes is an absolutely huge step
forward, both for large things and for general code cleanliness
because it can take less time to ix something than try to describe the
fix.)
Keshav Kini
> On Sat, Dec 24, 2011 at 6:59 PM, Keshav Kini <kesha...@gmail.com> wrote:
>> This is beautiful, big +1 from me (though it seems like a far-off
>> goal). It seems to me that hg_sage sits in an awkward position where
>> it is neither as powerful as command line hg nor much easier to use
>> than command line hg. (Maybe someone who uses it can give their
>> opinion? I don't use it.)
>
> It's easier to use hg_sage than the command line hg for people who don't
> know how to use the command line. That's probably 99% of potential Sage
> users; certainly, at least 90% of students who took my Sage class (at the
> beginning). Almost nobody uses/knows the unix command prompt, except
> computer geeks. I didn't even know how to use it until I took a class on
> UNIX when I was an undergrad (though back then, a reason I didn't know the
> command line was that it was very difficult to get access to -- Linux didn't
> exist, and even when it did, installing it meant downloading dozen of
> floppies). Today access is also not totally automatic, since most people use
> Microsoft Windows.
Certainly, I did not mean that having an interface from the Sage
console is not an important simplification in itself. And of course
`sage --pull` or `--push` could just as easily be implemented as
functions in the Sage library. Let me elaborate on what I meant by the
statement you quoted, and why I think `sage --push` would be a better
tool to use.
hg_sage does not hide the Mercurial mechanisms, but rather just moves
them wholesale into the Sage prompt, which seems to me like it could
be improved upon by providing a frontend that performs common tasks
like "create a patch for the changes I've made [against a vanilla
version of my sage install]" in one step. Unless, of course, hg_sage
is supposed to be exactly that - a direct one-to-one interface to a
subset of Mercurial, to be used for whatever you want to and can do
with Mercurial on the command line, in which case it doesn't compete
for `sage --push`'s use case at all. But this aside, the whole
workflow for Sage development and code submission that hg_sage
encourages doesn't make sense to me. Unlike using command line hg to
manage patch queues, hg_sage's behavior of committing permanently to
the on-disk repository seems prone to cause problems during sage
upgrades when the spkg-install scripts for the major mercurial-tracked
development directories fail to merge your committed changes, pop up
inscrutable merge windows in the middle of the build, etc. etc. These
are some problems that a `sage --push` / `sage --pull` would help to
address simply by abstracting the details away so we are free to do
whatever cleanup we want on the low level without breaking what users
expect to be literally happening / to be literally doing step by step.
Basically, the way that hg_sage works right now does not correspond in
any meaningful way to our real development model. When the user
"commits their changes", it is only as a means to create a patch file,
which then undergoes a review process on trac -- which means that
newbie developers never needed to know anything about Mercurial in the
first place! Any local commits they make have no meaning and are never
propagated anywhere. The real committer of the changes is always the
release manager, assuming the changes are positively reviewed. So this
makes no sense. In the current situation, post-upgrade, not only will
your local commits made by hg_sage sit there side by side with
almost-but-not-quite-identical commits from the release manager based
on the patch file you put on trac, but in fact the spkg-install script
for devel/sage will try to merge your commits into "your commits"! And
that's not even getting into the mess that happens when you try to
upgrade from a dev release to another dev release.
Instead, imagine a git repository with a master branch that just
tracks the version of sage that you have downloaded (and possibly
built), and topic branches named after trac tickets on which you
commit your changes relevant to each particular ticket. Other users
working on the same ticket will have their own such branches, and you
can cross-pollinate your branches until a ready-to-review piece of
code is produced on someone's branch for that ticket, at which point
you get a reviewer to mark the ticket as "positive review", and the
release manager then just merges the relevant reviewed branch into his
own master branch (or more properly the "release-prep" branch or
something) by *pulling*, not by messing around with patch files. When
you upgrade Sage under this system, supposing that you authored code
that was just released in the latest version of Sage, even if you have
that code on your own ticket branch from back when you were writing
it, the actual commit hashes from the released repository you just
downloaded and the commit hashes from your ticket branch will
correspond so your ticket branch will automatically meld into the
master branch - problem solved! This is one way to use a dvcs
effectively.
As a side note, in git, you can just delete topic head tags and the
branch will just go away, e.g. if your changes are rejected and the
ticket is marked invalid/wontfix for whatever reason. Rebasing is
similarly painless and doesn't mar the history of the release
manager's repo since he will only pull the topic branch for a ticket
when the ticket has positive review.
So to return to the topic of `sage --push` and `sage --pull`,
basically newbies would have an even better workflow which doesn't
need them to think about the details at all. Some first-glance idea of
what this workflow could be: they make a trac ticket, hack on sage,
and do `sage --push <ticket-id>`. To talk to other developers about
the ticket they go to the trac ticket webpage and post comments like
usual. On the trac ticket page can see links to all the people on,
say, github, who are working on this ticket. They do `sage --pull
<ticket-id> [username]` to check if someone else has been working on
it too, and can supply a username to pull that person's changes (maybe
this could be made even simpler). A combination of git and our custom
code will give them helpful messages like "You've been editing the
same file as someone else, I'm opening a 3-way diff viewer for you to
fix it" if necessary. (Ideally all the developers will be more or less
in sync with each other's changes.) Or, in a more likely scenario,
they do `sage --pull <another-ticket-id> [username]` in order to do a
review of a ticket. When someone feels their code branch on a
particular ticket is ready for review, they open a pull request on
github and wait for comments, which will be emailed to them.
That seems to me to be simpler than the current process, especially
since we now have a pretty developed codebase so tickets don't clash
with each other all that often. But above and beyond ease of use for
newbie developers, the abstraction also allows us to develop systems
to gain the benefit of using a revision control system properly rather
than as just a glorified diff utility and changelog generator :)
> In the notebook or sage prompt (say), one can do hg_sage.commit('commit
> message') then hg_sage.export('foo.patch'), and get a link to a patch file.
> You can also do "hg_sage.[tab]" and get just the minimal commands you need
> to know about to make a patch, with simple documentation.
Yes, but it would be nice if we could ditch patches altogether.
hg_sage is just a direct interface to Mercurial. If we can build
something more abstract, we can rework and streamline our development
workflow.
-Keshav
Keshav Kini
<robe...@math.washington.edu> wrote:
>> This is beautiful, big +1 from me (though it seems like a far-off
>> goal). It seems to me that hg_sage sits in an awkward position where
>> it is neither as powerful as command line hg nor much easier to use
>> than command line hg. (Maybe someone who uses it can give their
>> opinion? I don't use it.)
>
> My feelings exactly, though it does offer tab completion and
> sage-centric documentation and the ability to operate without leaving
> Sage which is an advantage for non-experts and the simple case. (But
> it should be even easier!)
Yup, which is about what I just replied to William a few minutes ago.
>> A `sage --push` and `sage --pull` would be a much better system.
>> If we did use github in particular, and let each developer have their
>> own fork on their own github account, I guess we would need to
>> integrate into trac some sort of system for looking for branches on
>> people's forks and maybe displaying links to them on the trac ticket
>> page, since the trac ticket is (and should be) a central place for
>> multiple people working on the ticket to discuss what they're doing.
>
> It's easy to post links to branches on github, and to discover them
> (especially if there's a convention of naming them after the ticket
> number). Easier in fact than attaching (and re-attaching, and
> re-attaching) a patch. Pointers to code in a live repository are a
> better solution than keeping the code on trac, and then there's no
> ambiguity about what patches to attach in what order, what
> dependencies are needed, what version of Sage it's based on, etc.
> (This an example of context that we lose/manually track by leaving the
> version control system in our current setup.)
Yes, exactly right! I think we should still keep to trac as a base of
operations, though, since github's issue tracker is pretty bare-bones
compared to what we have on trac (such as patchbot integration for
example, attachments as Jason mentioned, etc.).
>> The release manager's workflow would change drastically if we moved to
>> github, of course.
>
> Hopefully it shouldn't. If the release manager is spending a lot of
> time with the tools rather than the higher level tasks of coordinating
> what tickets and spkgs should go in and verifying sufficient testing
> on a wide variety of platforms then I'd say we need better tools.
Right, good point. It just seems that the release manager tends to
write his own tools, but I guess that's not technically part of the
"release management workflow", but something else.
>>> Of course being able to
>>> try things out/edit/add doctests (e.g. directly from your test
>>> session) online would be icing on the cake, but that's a bigger leap
>>> (and less co-opting an existing system).
>>
>> Could you explain what you mean by this?
>
> Imagine you go to a ticket, and there's a link to a sage notebook
> worksheet with this patch already applied and built. So as well as
> inspecting the code, you can try it out right there. You do some
> examples, and as you're stress-testing the code you think "this might
> make a good doctest" so you clean it up and click on a link to the
> side that lets you add that cell to the doctest of a function of your
> choice (perhaps bringing up the docstring with your additions appended
> in an editable window.) When you're all done, you can click "Save" and
> it will create a referee commit on top of the existing ticket adding
> your new examples. (As well as doctests, being able to edit the code
> as well as make line-by-line changes is an absolutely huge step
> forward, both for large things and for general code cleanliness
> because it can take less time to ix something than try to describe the
> fix.)
I see. Yes, that's certainly a bigger leap, haha. This would of course
only apply to the Sage library, I expect, since you would need to
restart sagenb.org's webserver constantly otherwise. An interesting
idea.
Keshav Kini
> Not drastic, but probably not smooth either. Github is missing:
>
> * workflow states
> * attachments to tickets
> * attach code to tickets (often, an issue is opened, then an identical pull
> request with the code, and then both have to be closed when the pull request
> is merged)
> * easily merge code from multiple people in a pull request (unlike different
> people submitting patches on a ticket)
> * custom fields (e.g., dependencies)
Right, I certainly hope that moving to github wouldn't require us to
abandon trac. We have a lot of good features on trac (not to mention a
long history of tickets that would be difficult or more likely
impossible to migrate). Github would just be used for managing the
code on developers' forks and handling pull requests, I imagine, at
least for the foreseeable future.