SVN to Git

[Daniel Lee]

Any objections to a move to Git? I can take care of it sometime within
the next week, as long as no one objects.

Ben, if you have any experience migrating repositories, could you help
me out? If not, directions online look straightforward.

[Ben Goodrich]

I've never done a conversion, so I don't have anything to add to the
instructions Google Code has for conversions.

I still think everyone should use git-svn on the client side for a
week to make sure everyone is on board before we convert anything on
the server side.

Ben

[Daniel Lee]

We're moving to Git tomorrow, Friday. I'll send an email when it's
done, but for the most part, feel free to operate normally.

Ben, is there a decent git tutorial around? Especially those of us
migrating from SVN?

Daniel

[Ben Goodrich]

The git documentation at

http://git-scm.com/documentation

is pretty good, whatever your taste in documentation may be.
Basically, for everything worthwhile in svn, there is a corresponding
git command. However, there are worthwhile features of git that are
not in svn, so you have to remind yourself to make use of them. You
can see most of the commands by doing

master!stan-git> git --help
usage: git [--version] [--exec-path[=<path>]] [--html-path] [--man-
path] [--info-path]
[-p|--paginate|--no-pager] [--no-replace-objects] [--bare]
[--git-dir=<path>] [--work-tree=<path>] [--
namespace=<name>]
[-c name=value] [--help]
<command> [<args>]

The most commonly used git commands are:
add Add file contents to the index
bisect Find by binary search the change that introduced a bug
branch List, create, or delete branches
checkout Checkout a branch or paths to the working tree
clone Clone a repository into a new directory
commit Record changes to the repository
diff Show changes between commits, commit and working tree,
etc
fetch Download objects and refs from another repository
grep Print lines matching a pattern
init Create an empty git repository or reinitialize an
existing one
log Show commit logs
merge Join two or more development histories together
mv Move or rename a file, a directory, or a symlink
pull Fetch from and merge with another repository or a local
branch
push Update remote refs along with associated objects
rebase Forward-port local commits to the updated upstream head
reset Reset current HEAD to the specified state
rm Remove files from the working tree and from the index
show Show various types of objects
status Show the working tree status
tag Create, list, delete or verify a tag object signed with
GPG

See 'git help <command>' for more information on a specific command.

Many of these subcommands have a --dry-run that you can use if you are
unsure of what will happen. It is almost always possible to roll
mistakes back but that requires careful understanding of the relevant
documentation.

If you use Emacs, there is also some helpful Emacs modes and stuff at

http://www.emacswiki.org/emacs/Git

The main difference is that whereas development under svn primarily
takes place in "the" trunk, development under git takes place
primarily in local branches (plural). When the conversion goes down
Friday, there will at first be just a "master" branch. But I would
strongly favor a rule that no one should edit source code directly in
the master branch. Maybe we can update TODO, add code comments, etc.
directly in the master branch, but the main thing is that the master
branch should never, ever be broken because everyone (possibly
including random people on the internet that we don't even know about)
will be branching off our master branch.

To create a local branch from master and switch to it, do

git checkout master
git branch foo
git checkout foo

At this moment in time, foo is identical to master. Once you start
making changes to foo, then it diverges from master but can be merged
back into master (possibly much) later on. Pretend you have made some
changes to the files in stan and saved them to the hard disk. Git
actually doesn't care at this point. If you do

git status

it will tell you those files have been modified and if you do

git diff

you can see the diff relative to the last *committed* change. To be
relevant, you have to "add" the modified files to the staging area so
that they can subsequently be committed locally. The use of the word
"add" is a bit confusing because it is the same word regardless of
whether you are adding a brand new file from scratch or adding a
modified file that already exists in the tree to the staging area. In
other words, you are "adding a file to the staging area", not
necessarily "adding a file to the tree". If you just do

git add .

(note there is a "dot" in the last line) it will add all modified
files to the staging area. This is often not what you want either
because you only want to add a particular modified file to the staging
area or because make test-all has created a lot of temporary binary
junk that does not (and should not) be tracked by git. So you can
instead do

git add path/to/modified/file.hpp

or

git add path/to/directory/where/modified/files/are

to do a more tailored adding of files to the staging area.

At that point, you can do

git commit -m "commit message" -m "optional longer commit explanation"

or if you omit the last half, git will open $EDITOR so that you can
write a commit message. Tradition dictates that there should be one
line explaining the commit, optionally followed by two line breaks and
one or more paragraphs explaining the commit in more detail. At this
point, the modifications are committed to the local branch but your
local branch is not (yet) relevant to anyone else. You can iterate the
process of adding to the staging area and locally committing in order
to make smaller and more self-contained steps. But if you know that
you have just made a few changes that you want to all be staged in a
single commit, you could instead streamline the above by doing

git commit -a -m "commit message" -m "optional longer commit
explanation"

which would add with the -a flag ALL modified files to the staging
area and commit them as a group to the local branch.

You can and should have multiple local branches. For example, I have
the following branches now

master!stan-git> git branch -a
BB
ice
lkj
logit
* master
multinormal
policy
remotes/git-svn

So, I have a branch where I was working on the BetaBinomial several
weeks ago, a branch where I was trying to figure out WTF was up with
the ice example in BUGS. A branch with ongoing work on the LKJ
transformation / distribution. The master branch which currently
tracks to the svn trunk. A branch with stuff on multivariate normal. A
branch with my hapless attempt to figure out that bug with policy
under g++-4.4. Since Bob / Daniel fixed that bug, now would be a good
time to delete that branch with

master!stan-git> git branch -d policy
Deleted branch policy (was 3f2631e).

So, basically my hacking around with that policy thing never happened
as far as you all are concerned. There is also the non-local branch
remotes/git-svn which tracks the svn trunk. This will be called
remotes/master when the conversion goes down; i.e. there is a master
branch in a certain state on the Google Code server and the "same"
master branch --- possibly in a different state very temporarily ---
on each of our machines.

Basically, the moral of the story is that you should create a local
branch every time you want to accomplish some halfway well-defined
goal with the code. So, if we were using git yesterday, Bob would
likely have made a better_debugging local branch, an inv_logit local
branch, a more_nuts_options local branch, and possibly some other
branches that the rest of us don't even need to know about. Another
good principle is not to let a branch drift from its original purpose.
Say Bob is implementing the features in the more_nuts_options branch
and discovers some unrelated bug. It would be better to

git checkout master
git branch unrelated_bug
git checkout unrelated_bug

and fix, test, etc. the unrelated bug in its branch, rather than
lumping it in with the rest of the stuff in the more_nuts_options
branch. One reason for this is that it is clearer, but the main reason
is that it may take a LONG time before Bob is happy enough with the
(more invasive) more_nuts_options branch to merge it back into the
master branch, whereas the fix to the unrelated bug should get pushed
to the master branch as soon as it is thoroughly fixed. There is an
option called "cherry picking" where Bob could extract and push just
the commits that fix the unrelated bug, but that is a bit messy.

I would say that the leading cause of problems with git is forgetting
which branch you are currently working on and / or remembering that
you have made some changes to the code but forgetting that those
changes were actually made to a different branch than the one you are
currently working on. You can configure your shell prompt to remind
you the branch name, but it is still easy to overlook. One place where
this manifests itself is accidentally branching from another local
branch instead of from the (local) master branch. Say Bob is working
on the more_nuts_options branch when he discovers the unrelated bug.
If he just does

git branch unrelated_bug
git checkout unrelated_bug

then the unrelated_bug branch will branch off of more_nuts_options
instead of master. Occasionally, you want to branch from a local
branch (... from a local branch)^n but most of the time, inclusive of
this case, you want to branch from the (local) master branch. If
unrelated_bug branches from more_nuts_options, then the commits that
Bob makes to fix the unrelated bug will be on top of Bob's half-baked
commits that implement more NUTS options. In that case, merging the
fix to the unrelated bug to master would also merge in the half-baked
commits implementing more NUTS options and thus the master branch
would be in a less-than-perfect state.

I've mentioned "merging" a few times already, which can get a bit
frustrating. When a branch is "done", meaning at a minimum you are
happy with it, all the tests pass, the stuff is documented if
necessary, etc. then you have to merge it back so that the rest of the
world can use it. There may be better ways but these are the steps I
do with the mi package.

# STEP 1: Sync your local master branch to the state of the master
branch on Google Code (origin)

git checkout master
git pull origin master

This is because during the time that Bob was working on
more_nuts_options, Matt may have fully implemented nutsN and pushed it
to Google Code, so Bob needs to catch up with that. Actually, people
should execute those two lines often, just to see what other people
have done, which is easy with some of the graphical tools or

git log

# STEP 2: Merge the local master branch into the local feature branch

git checkout more_nuts_options
git merge . master

(note that there is a "dot" in the last line). This ostensibly lays
Matt's work on top of Bob's work. In practice, there often is a "merge
conflict", which is annoying but usually is easily fixed. A merge
conflict happens when Matt and Bob change the same line(s) of code in
different ways. In this case, git will say something like "there is a
merge conflict in file.hpp, commit a fix and do git merge --continue
or you can do git merge --abort". At that point, if you do

git diff path/to/conflicted/file.hpp

you will see that git has inserted some markers indicating where the
conflict is and stacked the two sets of changes on top of each other.
The diff will look something like

>>>>>BOB>>>>>
- x = 1;
+ x = 2;
>>>>>MATT>>>>
- x = 1;
+ x = 3;
>>>>>>>>>>>>>

except it doesn't actually say BOB and MATT but rather HEAD (this
refers to the state of Bob's more_nuts_options branch) and a SHA
string (this refers to the SHA of the last commit that Matt pushed to
the master branch on Google Code). The >>> business actually is there
and facilitates finding the conflicts in large files. At this point,
Bob has to decide whether x should be 2 or 3 (or anything else), erase
all the lines that git inserted, commit to the more_nuts_options
branch and do

git merge --continue

to continue the mergification of other files that may have diverged.

Sometimes, there will be a huge merge conflict where multiple people
do fundamentally different things to the same code. In this situation,
it may be best to

git merge --abort

which will leave more_nuts_options in the state it was before the
aborted merge. Then, Bob and Matt can talk / email about which
direction to go. So, merge conflicts inevitably happen with multiple
developers, branches, etc. going on but because they are anticipated,
it is usually not too hard to resolve them.

Assuming the resolution of all merge conflicts, Bob can go onto

# STEP 3: Make an "empty" commit explaining what the more_nuts_options
branch accomplished

git commit --allow-empty

this will launch $EDITOR to allow Bob to write a longer commit message
even though no code has changed since the commit(s) that merged master
into more_nuts_options. So Bob could write something like:

"The more_nuts_options branch implements more NUTS options as follows

--foo does foo
--bar does bar
--baz does baz

The reason for these changes is because it is a good idea in one or
more of the following situations ..."

Not everybody does step 3 but I think it is a good practice because
when somebody is scrolling through the log it will give them a place
to get an overview of the branch, as opposed to wading through the
tens or hundreds of small commits in the more_nuts_options branch that
actually implement more NUTS options. Probably it wouldn't be
necessary if the purpose of the branch was just to fix some bug.

# STEP 4: Merge the more_nuts_options branch into the local master
branch

git checkout master
git merge . more_nuts_options

(note there is a "dot" in the last line). This step should never
generate a merge conflict because more_nuts_options already has
everything from master (with any merge conflicts resolved at this
point) and more_nuts_options just has the extra "empty" commit giving
an overview of more_nuts_options.

# STEP 5: Sync the master branch on Google Code with your local master
branch

git push origin master

Make double, super, extra sure that you are in your local master
branch before pushing to the master branch on Google Code. In other
words, you really don't want to accidentally be in some half-baked
branch and push that to the master branch on Google Code because then
everyone else would be branching from a polluted master branch.

# STEP 6: Optionally delete the local feature branch

git branch -d more_nuts_options

Since everything from the more_nuts_options branch is now not only in
the local master branch but also the master branch on Google Code, the
more_nuts_options branch isn't needed, although it doesn't hurt
anything to keep it around for a little while longer in case a problem
is discovered.

Some people have different ideas about the best workflow and
procedures. One possible rule is that "Only Bob gets to merge into the
master branch"; this is basically the rule they use for the Linux
kernel where Bob = Linus. Under that rule, everybody else would push
their local branches to Google Code when they are ostensibly ready.
For example, Matt would do steps 1, 2, and 3 and then

git push origin nutsN

at which point the Google Code server would (or at least should, I
have never actually done this with Google Code) have both "our" master
branch and Matt's nutsN branch. At which point, Matt would communicate
to Bob that he should look at and merge the nutsN branch to the master
branch, which Bob would do with

git pull origin nutsN
git checkout nutsN

After Bob verifies that nutsN is in good shape (and / or possibly
makes some more commits to it), Bob could merge the nutsN branch into
the master branch by following steps 4 and 5 above.

I don't know if we need to have a rule that says "Only Bob gets to
merge into the master branch". It would help Bob keep up to speed with
what other people are doing, but it also makes more work for him. If
the test suite is good and people are very careful to not break the
master branch, then we could do without it.

In either case, pushing a local branch to Google Code and sending an
email to the list saying "Can someone pull, help with, test my
potentially invasive changes" can be useful, in which case the second
person could edit / stage / commit to the local copy of that branch
and push to the first person's remote branch on Google Code (instead
of master) whereupon the first person can merge that remote branch
back into his local branch. Taking this to its logical extreme, some
projects do "peer review" where the first person announces that a
branch is available for review, a second person (not necessarily Bob)
approves of it or not, and if so, either the first or second person
merges into the master branch. This can work well if there are
multiple people who understand the same parts of the code and thus can
do good reviewing. I don't know if we could pull that off.

Winding down, one very useful command is

git grep REGEX DIR

which quickly finds all occurrences of the regular expression in or
under DIR. Often DIR would be src/ because you usually don't want
matches from Boost or Eigen in lib/ .

Also, if you do need to (re)move a file, make sure to use one of

git rm path/to/first_file.hpp
git mv path/to/second_file.hpp
git commit -a -m "(re)moving some files"

so that git can reconcile everything when those commits get pushed
into the master branch and subsequently pulled by other people with
potentially conflicting branches, etc.

Finally, because development under git is not chronological, it is not
possible to refer to "revision 793" because there is no such thing as
of Friday. But many of the commands have the option of operating on
specific commits. So git hashes every single commit, and you can refer
to it by the SHA. For example, in my git log, I see that r793 is
officially known as dee...

commit dee7347823ab7ae9a9e90a36991796fcafb5c68f
Author: bea...@alum.mit.edu <bea...@alum.mit.edu@9a304bd9-dce1-
f7c0-8d5c-bb1642157d4e>
Date: Thu Feb 16 23:30:01 2012 +0000

removed warning in printf.

THE END

Ben

[Ben Goodrich]

Let me know when you want to do this and I'll come by your office. --
Ben

[Daniel Lee]

We're using git now.

Before you start committing, I suggest changing your author
information. I managed to commit as Daniel Lee
<dan...@Daniel-Lees-MacBook-Air.local>

To see your information:
$ git config -l

To set your information:
$ git config user.name "Daniel Lee"
$ git config user.email bea...@alum.mit.edu

[Ben Goodrich]

On Feb 17, 10:55 am, Daniel Lee <bear...@alum.mit.edu> wrote:
> We're using git now.
>
> Before you start committing, I suggest changing your author
> information. I managed to commit as Daniel Lee
> <dan...@Daniel-Lees-MacBook-Air.local>
>
> To see your information:
> $ git config -l
>
> To set your information:
>  $ git config user.name "Daniel Lee"

>  $ git config user.email bear...@alum.mit.edu

I would also suggest

git config color.ui true

but I don't know if colorizing the shell propagates to Emacs.

I also use

https://github.com/rtomayko/git-sh

as my Bash-like shell when working with git repositories.

Ben

[Bob Carpenter]

On 2/17/12 10:55 AM, Daniel Lee wrote:
> We're using git now.

Great.

I managed to get everything checked out from Git.
It's nice how Google code generates the command
you need. I'll config before committing.

I installed on my Mac using this:

http://code.google.com/p/git-osx-installer/

which seemed to work.

Thanks for the doc pointers, Ben.

- Bob

[Bob Carpenter]

This is a nice tutorial for Git for people
familiar with SVN:

http://git.or.cz/course/svn.html

- Bob

[Ben Goodrich]

The top of that page says that it is (slightly) out of date and points
people to

https://git.wiki.kernel.org/articles/g/i/t/GitSvnCrashCourse_512d.html

Ben

[Bob Carpenter]

I thought I'd committed, but I hadn't. Now I
get what Daniel was saying about commits requiring
two commands. But when I try, it fails:

~/stan>git push
fatal: remote error: Invalid username/password.
You may need to use your generated googlecode.com password; see
https://code.google.com/hosting/settings

Where do I put my password? 20 minutes of web searching
and reading the tutorials again and studying the man page
has me scratching my head about things I've never heard
of like SSH keyrings, git: vs. ssh: vs. http: and so on.

I have my google code code password. And my git e-mail
matches my project e-mail:

~/stan>git config -l
user.name=Bob Carpenter
user.email=ca...@alias-i.com
color.diff=auto
color.status=auto
color.branch=auto
core.repositoryformatversion=0
core.filemode=true
core.bare=false
core.logallrefupdates=true
core.ignorecase=true
remote.origin.fetch=+refs/heads/*:refs/remotes/origin/*
remote.origin.url=https://carp%40alia...@code.google.com/p/stan/
branch.master.remote=origin
branch.master.merge=refs/heads/master

- Bob

[Ben Goodrich]

The simplest thing to do is to create a .netrc file in your $HOME
directory, which should show up as Option 2 under

http://code.google.com/p/stan/source/checkout

if you are logged into your Gmail account. You have to click through
to see the password that Google generated for you.

Ben

On Feb 18, 6:34 pm, Bob Carpenter <c...@alias-i.com> wrote:
> I thought I'd committed, but I hadn't.  Now I
> get what Daniel was saying about commits requiring
> two commands.  But when I try, it fails:
>
>    ~/stan>git push
>    fatal: remote error: Invalid username/password.

>    You may need to use your generated googlecode.com password; seehttps://code.google.com/hosting/settings

>
> Where do I put my password?  20 minutes of web searching
> and reading the tutorials again and studying the man page
> has me scratching my head about things I've never heard
> of like SSH keyrings, git: vs. ssh: vs. http: and so on.
>
> I have my google code code password.  And my git e-mail
> matches my project e-mail:
>
> ~/stan>git config -l
> user.name=Bob Carpenter

> user.email=c...@alias-i.com

> color.diff=auto
> color.status=auto
> color.branch=auto
> core.repositoryformatversion=0
> core.filemode=true
> core.bare=false
> core.logallrefupdates=true
> core.ignorecase=true
> remote.origin.fetch=+refs/heads/*:refs/remotes/origin/*

> remote.origin.url=https://carp%40alias-i....@code.google.com/p/stan/
> branch.master.remote=origin
> branch.master.merge=refs/heads/master
>
> - Bob

[Daniel Lee]

Git Pro (online and free, also available for purchase) has proven useful:
http://progit.org/book/

[Ben Goodrich]

At a variety of places, git can call a script before taking action.
One such hook is called "pre-commit". I have done

cp .git/hooks/pre-commit.sample .git/hooks/pre-commit

then I commented out the last line (which prohibits trailing
whitespace) and added this line

exec make CC=clang++ 0=3 -j8 test-all > /dev/null

This way, if one of the tests fails, I can't commit my saved changes
to my local repository.

Under a more Subversion-based paradigm, it would seem better for the
central server to reject commits that cause tests to fail, but as far
as I know, Google Code does not allow projects to run unit tests on
Google's servers. So, it might seem better to have a pre-push hook on
the client side, but the Git developers have never agreed on whether
pre-push hooks are appropriate given the alternatives.

Thus, even though rejecting a commit because it fails a test seems a
bit extreme, that seems like the best we can do for now. Also, you can
skip the pre-commit hook by passing -n to git commit, so you can thus
commit broken stuff if you really want to. Are there other things that
we should do in the pre-commit hook or other hooks?

Ben

[Jiqiang Guo]

Do not be surprised to see I pushed as maverick. For some reason,
my git global configuration was used. I will figure it out how to set
up a local configuration of user.name and user.email.

On Feb 17, 10:55 am, Daniel Lee <bear...@alum.mit.edu> wrote:
> We're using git now.
>
> Before you start committing, I suggest changing your author
> information. I managed to commit as Daniel Lee
> <dan...@Daniel-Lees-MacBook-Air.local>
>
> To see your information:
> $ git config -l
>
> To set your information:
>  $ git config user.name "Daniel Lee"

>  $ git config user.email bear...@alum.mit.edu