git
William Stein
I want to switch to using git to develop the core Sage library (though
I'll still submit patches using hg for now).
How can I do this?
William
--
William Stein
Professor of Mathematics
University of Washington
http://wstein.org
Minh Nguyen
On Mon, Feb 20, 2012 at 6:21 AM, William Stein <wst...@gmail.com> wrote:
> I want to switch to using git to develop the core Sage library (though
> I'll still submit patches using hg for now).
> How can I do this?
To preserve the commit history under Mercurial, we need to convert
from hg to git format. This can be done using various tools. I've
personally used fast-export [1] before to convert a hg repository to a
git repo. From there, we should be able to start using git for
development. Git should be able to import patches produced by hg.
For those transitioning from hg to git, here's a brief dictionary [2]
between hg/git commands.
[2] https://github.com/sympy/sympy/wiki/Git-hg-rosetta-stone
--
Regards,
Minh Van Nguyen
http://sage.math.washington.edu/home/mvngu/
Mike Hansen
> To preserve the commit history under Mercurial, we need to convert
> from hg to git format. This can be done using various tools. I've
> personally used fast-export [1] before to convert a hg repository to a
> git repo. From there, we should be able to start using git for
> development. Git should be able to import patches produced by hg.
> For those transitioning from hg to git, here's a brief dictionary [2]
> between hg/git commands.
This doesn't seem to be what William wants -- he wants to make the
changes in git and be able to contribute those to Sage (as if he
hadn't used git). Other than producing patches with git and then say
qimporting them into hg, the best bet looks to be
https://github.com/offbytwo/git-hg where you can "experimentally" push
to an hg repo.
The other direction (hg client talking to get repo) seems to be much
more fully supported.
--Mike
William Stein
> On Sun, Feb 19, 2012 at 12:39 PM, Minh Nguyen <mvngu...@gmail.com> wrote:
>> To preserve the commit history under Mercurial, we need to convert
>> from hg to git format. This can be done using various tools. I've
>> personally used fast-export [1] before to convert a hg repository to a
>> git repo. From there, we should be able to start using git for
>> development. Git should be able to import patches produced by hg.
>> For those transitioning from hg to git, here's a brief dictionary [2]
>> between hg/git commands.
>
> This doesn't seem to be what William wants -- he wants to make the
> changes in git and be able to contribute those to Sage (as if he
> hadn't used git). Other than producing patches with git and then say
> qimporting them into hg,
That's actually not a bad idea for now. I think I'll do that for a few days.
A little more longterm, I'll do what you suggest below.
> the best bet looks to be
> https://github.com/offbytwo/git-hg where you can "experimentally" push
> to an hg repo.
>
> The other direction (hg client talking to get repo) seems to be much
> more fully supported.
>
> --Mike
--
Keshav Kini
> Hi Keshav (/Sage-devel),
>
> I want to switch to using git to develop the core Sage library (though
> I'll still submit patches using hg for now).
> How can I do this?
Hi William,
Just fork sagemath/sagelib on github, then do
$ cd $SAGE_ROOT/devel/
$ git clone g...@github.com:williamstein/sagelib sage-git
$ rm sage
$ ln -S sage-git sage
$ cd sage
$ git remote add upstream g...@github.com:sagemath/sagelib
$ sage -b
Everything should just work from this point. To generate a patch from
one of your branches will be a little bit more work, but first do this:
$ cd $SAGE_ROOT/devel/sage
$ git remote update upstream
$ git diff upstream/master <your branch> > foo.patch
I am making sure that upstream/master always points to the latest
development release, or stable release if there have been no development
releases since the last stable release.
Finally you'll need to add in some lines at the top of foo.patch, viz.
# HG changeset patch
# User William Stein <wst...@gmail.com>
# Date $(date +%s) 0000
commit message for entire patch
Here $(date +%s) of course should be expanded by the shell. You can
probably write a quick shell script to insert these lines automatically,
maybe accepting the commit message as an argument. (I assume of course
that you will be consolidating multiple small commits into one patch,
which hopefully we won't have to do anymore once we actually switch to
git.)
Hope that helps. Evidently other people in this thread do not know that
I have been maintaining a git version of the Sage library, so hopefully
it will help them too, if they're interested :)
-Keshav
----
Join us in #sagemath on irc.freenode.net !
Keshav Kini
> This doesn't seem to be what William wants -- he wants to make the
> changes in git and be able to contribute those to Sage (as if he
> hadn't used git). Other than producing patches with git and then say
> qimporting them into hg, the best bet looks to be
> https://github.com/offbytwo/git-hg where you can "experimentally" push
> to an hg repo.
Or use hg-git - http://bitbucket.com/durin42/hg-git/ - to pull your
patch into the hg Sage library. hg-git is much less experimental than
git-hg, I believe.
Or, you could just not use Mercurial at all and create fake Mercurial
patches as I described in my response to William in this thread. This is
probably the nicest option, since you don't have to mess around with
multiple version control systems and conversion layers.
-Keshav
William Stein
> William Stein <wst...@gmail.com> writes:
>> Hi Keshav (/Sage-devel),
>>
>> I want to switch to using git to develop the core Sage library (though
>> I'll still submit patches using hg for now).
>> How can I do this?
>
> Hi William,
>
> Just fork sagemath/sagelib on github, then do
>
> $ cd $SAGE_ROOT/devel/
> $ git clone g...@github.com:williamstein/sagelib sage-git
> $ rm sage
> $ ln -S sage-git sage
> $ cd sage
Cool -- that's actually exactly what I had done before sending the
email, thinking it was the natural thing to do.
https://github.com/williamstein/sagelib
In general, what version of the Sage library is at sagemath/sagelib on github?
> $ git remote add upstream g...@github.com:sagemath/sagelib
> $ sage -b
>
> Everything should just work from this point. To generate a patch from
> one of your branches will be a little bit more work, but first do this:
>
> $ cd $SAGE_ROOT/devel/sage
> $ git remote update upstream
> $ git diff upstream/master <your branch> > foo.patch
>
> I am making sure that upstream/master always points to the latest
> development release, or stable release if there have been no development
> releases since the last stable release.
Ah, yes, that answers my main question.
>
> Finally you'll need to add in some lines at the top of foo.patch, viz.
>
> # HG changeset patch
> # User William Stein <wst...@gmail.com>
> # Date $(date +%s) 0000
> commit message for entire patch
>
> Here $(date +%s) of course should be expanded by the shell. You can
> probably write a quick shell script to insert these lines automatically,
> maybe accepting the commit message as an argument. (I assume of course
> that you will be consolidating multiple small commits into one patch,
> which hopefully we won't have to do anymore once we actually switch to
> git.)
>
> Hope that helps. Evidently other people in this thread do not know that
> I have been maintaining a git version of the Sage library, so hopefully
> it will help them too, if they're interested :)
Thanks!!
In particular, could you give me a step-by-step handholding
explanation of how would I replicate (and improve on!) the following
HG workflow:
1. Decide I want to do something awesome.
2. Create a trac ticket describing it.
3. Write code.
4. Post a single patch
5. Rewrite code.
6. Post a single updated patch.
With queues I do "hg qinit trac12345.patch", etc, and "hg qrefresh"
and "hg export".
I would like to improve on this of course by being able to record (at
least for my personal edification) lots of commits, even if they get
folded later for the public.
-- William
>
> -Keshav
>
> ----
> Join us in #sagemath on irc.freenode.net !
>
> --
> To post to this group, send an email to sage-...@googlegroups.com
> To unsubscribe from this group, send an email to sage-devel+...@googlegroups.com
> For more options, visit this group at http://groups.google.com/group/sage-devel
> URL: http://www.sagemath.org
William Stein
> I have been maintaining a git version of the Sage library, so hopefully
> it will help them too, if they're interested :)
Can we add a better .gitignore?
git status
...
# Untracked files:
# (use "git add <file>..." to include in what will be committed)
#
# ../../../../build/
# ../../../../c_lib/.sconsign.dblite
# ../../../../c_lib/libcsage.dylib
# ../../../../c_lib/src/ZZ_pylong.os
# ../../../../c_lib/src/convert.os
# ../../../../c_lib/src/gmp_globals.os
# ../../../../c_lib/src/interrupt.os
# ../../../../c_lib/src/memory.os
# ../../../../c_lib/src/mpn_pylong.os
# ../../../../c_lib/src/mpz_longlong.os
and a *LOT* more...
William Stein
I think the reason I implemented the kludgy "sage -clone" Sage branch
thing (way back in maybe March 2006) was because we had no dependency
checking for Pyrex files. Now that we have dependency checking,
lightweight branching should be able to accomplish the same thing much
more efficiently, right?
-- William
--
Keshav Kini
> Hi Keshav (et al.),
>
> I think the reason I implemented the kludgy "sage -clone" Sage branch
> thing (way back in maybe March 2006) was because we had no dependency
> checking for Pyrex files. Now that we have dependency checking,
> lightweight branching should be able to accomplish the same thing much
> more efficiently, right?
Well, I normally wouldn't write this contentless response, but since
you've addressed me specifically:
I have no idea how Cython works, it all seems like magic to me :) So I
have no idea whether there's some problem with getting rid of `sage
-clone`. But even if not, then I think it's Cython which needs to be
fixed. `sage -clone` should not exist, IMO.
We should be able to put whatever we want at $SAGE_ROOT/devel/sage,
whether it be a symlink to some other directory, or just a directory
itself, and expect Sage to read its library from there. Sage should not
know or care about where a symlink at $SAGE_ROOT/devel/sage points to.
But since you're calling it a kludge, I guess you agree with me on this. :)
Keshav Kini
Yes, we definitely should make a .gitignore once we actually start
moving to git, or even earlier. But I think that this time we should
make sure lines in the .gitignore file are commented to mention why
we're ignoring certain things, unlike how the .hgignore file is now.
"Boring file regexps" and "sage files to ignore" is not good enough,
especially for newbie developers :)
Anyway, for now, you can do
$ cd $SAGE_ROOT/devel/sage
$ git config status.showUntrackedFiles no
This will, as the name of the config variable suggests, stop unknown
files from appearing in the `git status` report. Of course, this also
means that you won't notice if you write a new file and forget to add
it, but if you always use the -v option with `git commit` to check what
you're committing, this shouldn't be a problem.
Keshav Kini
> Cool -- that's actually exactly what I had done before sending the
> email, thinking it was the natural thing to do.
> https://github.com/williamstein/sagelib
>
> In general, what version of the Sage library is at sagemath/sagelib on github?
>
>> development release, or stable release if there have been no development
>> releases since the last stable release.
>
> Ah, yes, that answers my main question.
Exactly what the various branches mean is pointed out on the README.rst
you see at the github page. This README.rst does not actually appear in
the file structure if you checkout the master branch, though, because
it's on an extra branch I made just in order to display that file on the
github page.
Well, let me just give you an overview of the workflow I'm envisioning:
1. Decide I want to do something awesome.
2. $ git remote update upstream
3. $ git checkout -b something-awesome upstream/master
4. Write code.
5. Commit changes with explanatory message in commit message; no need
to refer to "something awesome" in every message. Suggestion: use
`git commit -av`, or `git commit -pv` if you don't want to commit
everything you've changed.
6. $ git push origin something-awesome # optional
7. Repeat steps 4-6 until you have something you think is ready to put
into Sage.
8. Make a trac ticket describing something awesome.
Here's where it diverges. What you would do right now:
9. Create a patch from `git diff something-awesome upstream/master`, as
I described in my previous mail, and upload it to the trac ticket.
10. Wait for patchbot to notice your patch, download it, apply it, test
it, and put feedback on the trac ticket. Hopefully this feedback is
not "patch did not apply".
11. Wait for reviewers to give you comments.
12. If more changes are necessary, repeat steps 4-6 as required, then
repeat steps 9-11, overwriting your old patch.
13. If no more changes are necessary, wait for the release manager to
merge your patch.
14. Forcibly delete your branch something-awesome because your code is
already in Sage, albeit not in the form of the commits you made.
15. Wait for your deleted branch to be garbage collected. Feel a great
disturbance in the Force.
That should more or less be all you need to know for now. But here's how
I hope we will be doing it in the near future:
9. Create a pull request on github from williamstein:something-awesome
to sagemath:master , mentioning "trac 12345" in the pull request
title. (Make sure you've done step 6 sufficiently recently.)
10. Manually tell the patchbot to test your branch by clicking a "test
me" button on the trac ticket. Wait for patchbot to arrive at your
ticket in its request queue, search pull requests on github for a
single open pull request with "trac 12345" in its title, locally
"accept" the pull request and run doctests, and finally report to
the trac ticket on the success of "accepting" the pull request and
on the success of the doctests.
11. Wait for reviewers to give you comments on the trac ticket.
12. If more changes are necessary, repeat steps 4-6 and 9-11 as
required.
13. If no more changes are necessary, wait for the release manager to
merge the pull request on github.
You'll notice that I recommend currently just creating a new patch
against upstream/master each time you update the trac ticket, rather
than uploading multiple tickets. This is just because otherwise you
might feel like making fewer commits because you don't want to have too
many patches on trac, which I think is probably a bad habit.
I also envision the patchbot being "on request" because otherwise it
will probably be quickly overloaded by the higher commit frequency we
would have in the new system.
I put making a trac ticket as late as step #8 so that if something
awesome turned out not to be so awesome after all, you wouldn't need to
bother the release manager about closing the ticket. As long as
everyone's doing step #6 frequently, people can always see what other
people are working on by looking at the `github network page`_, so that
removes the incentive to create a trac ticket early so people know what
you're doing. (Nobody reads the list of new trac tickets these days
anyway, do they? There are too many of them!)
.. _github network page: http://github.com/sagemath/sagelib/network
Anyway, this "near future" stuff is all speculation so far. It will also
be facilitated for newbies by some kind of script, and for the release
manager by another script. I still have to write up something coherent
in the SEP, but the above is a preview :)
Jason Grout
> Or use hg-git -http://bitbucket.com/durin42/hg-git/ - to pull your
> patch into the hg Sage library. hg-git is much less experimental than
> git-hg, I believe.
That's what I've done. I develop in git, then when I want to make a
patch in mercurial, I use hg-git to pull down my changes into an hg
repository, then use hg to export a patch.
1. Install hg-git
2. hg clone the git repository
3. hack in git
4. hg pull from the git repository
5. hg export patch
It's a little confusing sometimes to keep track of the bookmarks in hg,
which correspond to the branches in git, and it seems that sometimes I
get my bookmarks messed up in hg. So I don't know that I would
recommend this for new people to git or hg.
Keshav: is there an easy way to make a new git command which generates a
mercurial patch by adding those lines to the top of a generated patch?
Thanks,
Jason
Keshav Kini
> Keshav: is there an easy way to make a new git command which generates
> a mercurial patch by adding those lines to the top of a generated
> patch?
Sure, just drop a script called `git-sage-patch` somewhere in your $PATH.
Here's such a script::
#!/bin/bash
# Script to produce a patch to the Sage library.
# Usage:
# git sage-patch filename.patch "commit message"
echo "# HG changeset patch
# User $(git config user.name) <$(git config user.email)>
# Date $(date +%s) 0000
$2
" > $1
git diff upstream/master HEAD >> $1
Disclaimer: this script is not very robust and fails with a cryptic
error message if you use it somewhere other than inside the Sage library
repo, or don't provide it the correct command line arguments. It also
doesn't check that the filename exists before exporting the patch. Use
with caution! :)
Robert Bradshaw
> William Stein <wst...@gmail.com> writes:
>> Hi Keshav (et al.),
>>
>> I think the reason I implemented the kludgy "sage -clone" Sage branch
>> thing (way back in maybe March 2006) was because we had no dependency
>> checking for Pyrex files. Now that we have dependency checking,
>> lightweight branching should be able to accomplish the same thing much
>> more efficiently, right?
>
> Well, I normally wouldn't write this contentless response, but since
> you've addressed me specifically:
>
> I have no idea how Cython works, it all seems like magic to me :) So I
> have no idea whether there's some problem with getting rid of `sage
> -clone`. But even if not, then I think it's Cython which needs to be
> fixed. `sage -clone` should not exist, IMO.
Cython is just a py(x) to C compiler. The problem is not Cython per
se, but re-building when changing branches. Each "clone" keeps it's
own independent set of build artifacts, so switching between them does
not require a re-build.
> We should be able to put whatever we want at $SAGE_ROOT/devel/sage,
> whether it be a symlink to some other directory, or just a directory
> itself, and expect Sage to read its library from there. Sage should not
> know or care about where a symlink at $SAGE_ROOT/devel/sage points to.
>
> But since you're calling it a kludge, I guess you agree with me on this. :)
>
> -Keshav
>
> ----
> Join us in #sagemath on irc.freenode.net !
>
Keshav Kini
> On Mon, Feb 20, 2012 at 1:00 AM, Keshav Kini <kesha...@gmail.com> wrote:
>> William Stein <wst...@gmail.com> writes:
>>> I think the reason I implemented the kludgy "sage -clone" Sage branch
>>> thing (way back in maybe March 2006) was because we had no dependency
>>> checking for Pyrex files. Now that we have dependency checking,
>>> lightweight branching should be able to accomplish the same thing much
>>> more efficiently, right?
>>
>> have no idea whether there's some problem with getting rid of `sage
>> -clone`. But even if not, then I think it's Cython which needs to be
>> fixed. `sage -clone` should not exist, IMO.
>
> Cython is just a py(x) to C compiler. The problem is not Cython per
> se, but re-building when changing branches. Each "clone" keeps it's
> own independent set of build artifacts, so switching between them does
> not require a re-build.
Sure, that much I certainly understand. I don't know what William is
saying about dependency checking, though.
Well, on second thought maybe I can guess. Is it that Cython now knows
how to check which files have been touched, and then do a cascading
rebuild of all outdated .so files? If previously you required a total
rebuild every time you ran `sage -b`, I can see why you would want to
keep a separate copy of the Sage library which you never ran `sage -b`
on.
William Stein
> Robert Bradshaw <robe...@math.washington.edu> writes:
>> On Mon, Feb 20, 2012 at 1:00 AM, Keshav Kini <kesha...@gmail.com> wrote:
>>> William Stein <wst...@gmail.com> writes:
>>>> I think the reason I implemented the kludgy "sage -clone" Sage branch
>>>> thing (way back in maybe March 2006) was because we had no dependency
>>>> checking for Pyrex files. Now that we have dependency checking,
>>>> lightweight branching should be able to accomplish the same thing much
>>>> more efficiently, right?
>>>
>>> I have no idea how Cython works, it all seems like magic to me :) So I
>>> have no idea whether there's some problem with getting rid of `sage
>>> -clone`. But even if not, then I think it's Cython which needs to be
>>> fixed. `sage -clone` should not exist, IMO.
>>
>> Cython is just a py(x) to C compiler. The problem is not Cython per
>> se, but re-building when changing branches. Each "clone" keeps it's
>> own independent set of build artifacts, so switching between them does
>> not require a re-build.
>
> Sure, that much I certainly understand. I don't know what William is
> saying about dependency checking, though.
If you switch between git branches all the build artifacts are for the
branch right
*before* you did the switch. It would seem likely this could confuse
the hell out
of Pyrex, but maybe not Cython (?). Even if it doesn't confuse
them, it can cause
huge build times -- e.g., each time you switch branches, you might have to wait
30 minutes for "sage -br", in case the branches touch something that is depended
on by a lot of other Cython code. The Sage version of branches
completely solves
this problem.
> Well, on second thought maybe I can guess. Is it that Cython now knows
> how to check which files have been touched, and then do a cascading
> rebuild of all outdated .so files? If previously you required a total
> rebuild every time you ran `sage -b`, I can see why you would want to
> keep a separate copy of the Sage library which you never ran `sage -b`
> on.
It's worse in some cases, as mentioned above. However, for a lot of
everyday work, there won't be many files that are forced to be rebuilt
when switching branches.
-- William
>
> -Keshav
>
> ----
> Join us in #sagemath on irc.freenode.net !
>
> --
> To post to this group, send an email to sage-...@googlegroups.com
> To unsubscribe from this group, send an email to sage-devel+...@googlegroups.com
> For more options, visit this group at http://groups.google.com/group/sage-devel
> URL: http://www.sagemath.org
--
Robert Bradshaw
Yep, that is exactly the issue. Distutils doesn't understand Cython
dependencies, and we have our own custom code to handle that (though
we could move over to Cython's implementation).
And before this we used to have to manually touch dependent .pyx files
or do a "-ba" just to be sure, in the snow, up the hill both ways...
>> Well, on second thought maybe I can guess. Is it that Cython now knows
>> how to check which files have been touched, and then do a cascading
>> rebuild of all outdated .so files? If previously you required a total
>> rebuild every time you ran `sage -b`, I can see why you would want to
>> keep a separate copy of the Sage library which you never ran `sage -b`
>> on.
>
> It's worse in some cases, as mentioned above. However, for a lot of
> everyday work, there won't be many files that are forced to be rebuilt
> when switching branches.
In particular, when working on coercion (parent.pxd) and more basic
structures (matrix0.pxd) a huge number of files had to be rebuilt. I
think these days a higher percentage of development happens in the
"leaves" so it's not near as painful.
- Robert
Keshav Kini
> On Mon, Feb 20, 2012 at 10:06 AM, Keshav Kini <kesha...@gmail.com> wrote:
>> Robert Bradshaw <robe...@math.washington.edu> writes:
>>> Cython is just a py(x) to C compiler. The problem is not Cython per
>>> se, but re-building when changing branches. Each "clone" keeps it's
>>> own independent set of build artifacts, so switching between them does
>>> not require a re-build.
>>
>> Sure, that much I certainly understand. I don't know what William is
>> saying about dependency checking, though.
>
> If you switch between git branches all the build artifacts are for the
> branch right
> *before* you did the switch. It would seem likely this could confuse
> the hell out
> of Pyrex, but maybe not Cython (?). Even if it doesn't confuse
> them, it can cause
> huge build times -- e.g., each time you switch branches, you might have to wait
> 30 minutes for "sage -br", in case the branches touch something that is depended
> on by a lot of other Cython code. The Sage version of branches
> completely solves
> this problem.
>
OK, why not cache object files by hashes of input or something? Of
course I'm probably vastly underestimating the complexity of this
problem...
Robert Bradshaw
IMHO, this is the correct solution. Of course the "input" to a binary
typically consists of a file, all its dependencies, and possibly some
of the environment. Possible (and I really hope we get there), but a
bit more complicated. See ccache or NixOS for prior art.
- Robert