Stan repo is now ready to use. Instructions inside email.

[Daniel Lee]

First off, thanks everyone for your patience while we go through this change. If you haven't read this blog post (http://nvie.com/posts/a-successful-git-branching-model/), please do if you want to contribute. I'm going to walk through our process, but it pretty much mimics the one outlined in the post.

If you don't read anything else, just remember these two things:

1. Do not push directly into the "master" or "develop" branch.
2. For working on code: branch from "develop" into a branch called "feature/<some-descriptive-name>", then create a pull request back into "develop".

Permanent Branches

We have two permanent branches:

1. master
2. develop

The master branch will always be at our last release. It is currently set to v1.3.0. @nvie calls this production-ready.

The develop branch will be the current working branch. This will be like how we treated master in the past, but with one additional requirement. We will expect this branch to be in a fairly usable state. For the time being, that means it passes all tests: test-unit, test-distributions, test-models.

In order to enforce the good state of the develop branch, all things that get into the develop branch will be done by pull requests.

Note: we are working on an honor system. The active developers will still have the ability to push to the develop and master branches.

Non-Permanent Branches

(quick description. Instructions further below.)

We will have 3 types of non-permanent branches:

1. feature
2. hotfix
3. release
   

Feature branches are the most common. Any type of new feature, fixing of bugs in a feature that hasn't been released, or pretty much any new development will go in a feature branch. These branch from "develop" and are named "feature/<some-descriptive-name>". After development is complete on the feature, submit a pull request back to "develop".

Hotfix branches are fixes to bugs in the current release. There should be an issue created for the bug in our issue tracker prior to branching. These branch from "master". These are named "hotfix/<next patch version number>". ONce the patch is complete, this will be merged into both "master" and "develop".

Release branches are branches for new releases. These branch from "develop" and are eventually merged into "master". Most developers won't need to worry about this step.

How to Contribute with a Clone of the Repository

(Most of us have been using the repository this way. Using this method, we can share feature branches with ease for collaboration.)

1. Clone the GitHub stan-dev/stan repository. From the command line:
   > git clone https://github.com/stan-dev/stan.git
   > cd stan
   
   The default repository is develop and you can verify you are there by typing
   > git branch
   
2. Create a branch. For a feature branch (most common):
   > git checkout develop
   > git checkout -b feature/foo
   > git push
   
   For a hotfix branch (less common):
   If we're currently at v3.2.1
   > git checkout master
   > git checkout -b hotfix/v3.2.2
   > git push
   
   The last command, git push, is to make the branch public. This is necessary for when a pull request is created.
   
3. Do work.
   Use the regular git commands to work. Collaboration is easy since all the developers have push access to the branch created.
4. Create a pull request.
   A pull request indicates that the work is done and should be merged back to the appropriate place. When the pull request is made, it is expected that the current code passes:
   > make manual
   > make doxygen
   > make test-unit
   > make test-distributions
   > make test-models
   
   If it does not pass on your local machine, it won't be accepted. Passing these tests do not guarantee acceptance.
   
   For a feature branch:
   
   1. Merge the current state of "develop" back into the current feature branch:
      Update "develop", merge "develop" into current branch (deal with any conflicts using git), push so the changes are available to github.
      > git pull origin develop
      > git merge --no-ff develop
      > git push
   2. Create a pull request.
   • Go to https://github.com/stan-dev/stan (refresh page if necessary)
     
   • Select the feature branch you are working on in the drop-down list. (Below the "Clone in ..." row)
     
   • A "Pull Request" button should pop up underneath the feature branch drop-down list. Click it.
   • Fill in form and press "Send pull request".
     
     Alternate directions:
     
   • Go to https://github.com/stan-dev/stan (refresh page if necessary)
   • Click "Pull Request" button. It's towards the top right, on the same row as the repo name "stan-dev/stan"
   • Select the correct base repo, "stan-dev/stan", and base branch, "develop"
   • Select the correct head repo, "stan-dev/stan", and head branch, "feature/foo"
   • Fill in form and press "Send pull request".
     
     

   For a hotfix branch:
   

   1. Create a pull request.
   • Go to https://github.com/stan-dev/stan (refresh page if necessary)
     
   • Select the hotfix branch you are working on in the drop-down list. (Below the "Clone in ..." row)
     
   • A "Pull Request" button should pop up underneath the hotfix branch drop-down list. Click it.
   • Fill in form and press "Send pull request".
     
     Alternate directions:
     
   • Go to https://github.com/stan-dev/stan (refresh page if necessary)
   • Click "Pull Request" button. It's towards the top right, on the same row as the repo name "stan-dev/stan"
   • Select the correct base repo, "stan-dev/stan", and base branch, "develop"
   • Select the correct head repo, "stan-dev/stan", and head branch, "feature/foo"
   • Fill in form and press "Send pull request".
5. Relax.
   At this point, one or more of the developers will check the pull request and do an informal code review. At the same time, our Jenkins box will verify that all tests pass on Windows. If both are good, then the pull request will be accepted, the code will get into the correct branch, and the branch will be deleted.
6. If the pull request is not accepted immediately, fix the issues.
   
   1. Check out the branch
      > git checkout feature/foo
      > git pull origin feature/foo
      (Or you can set upstream tracking so you can do "git pull feature/foo")
      
   2. Do work.
      Others can help at this point since the branch is available to everyone else.
      The work will update the pull request.
      
   3. Go back to "5. Relax"

That's it. That's how it works.

How to Contribute with a Fork of the Repository (we should be using a clone)

(This is how some of us have been working. The downside: if you want others to help, you need to give people permissions to your personal fork of the stan repository.)

1. Fork the GitHub stan-dev/stan repository. Done through either the web or the GitHub application.
   
2. Create a branch. For a feature branch (most common):
   > git checkout develop
   > git checkout -b feature/foo
   > git push
   
   For a hotfix branch (less common):
   If we're currently at v3.2.1
   > git checkout master
   > git checkout -b hotfix/v3.2.2
   > git push
   
   The last command, git push, is to make the branch public. This is necessary for when a pull request is created.
   
3. Do work.
   Use the regular git commands to work. If you want collaborators, let them have access to the branch.
   
4. Create a pull request.
   A pull request indicates that the work is done and should be merged back to the appropriate place. When the pull request is made, it is expected that the current code passes:
   > make manual
   > make doxygen
   > make test-unit
   > make test-distributions
   > make test-models
   
   If it does not pass on your local machine, it won't be accepted. Passing these tests do not guarantee acceptance.
   
   For a feature branch:
   
   1. Merge the current state of "develop" back into the current feature branch:
      Update "develop", merge "develop" into current branch (deal with any conflicts using git), push so the changes are available to github.
      > git pull origin develop
      > git merge --no-ff develop
      > git push
   2. Create a pull request.
   • Go to https://github.com/stan-dev/stan (refresh page if necessary)
     
   • Select the feature branch you are working on in the drop-down list. (Below the "Clone in ..." row)
     
   • A "Pull Request" button should pop up underneath the feature branch drop-down list. Click it.
   • Fill in form and press "Send pull request".
     
     Alternate directions:
     
   • Go to https://github.com/stan-dev/stan (refresh page if necessary)
   • Click "Pull Request" button. It's towards the top right, on the same row as the repo name "stan-dev/stan"
   • Select the correct base repo, "stan-dev/stan", and base branch, "develop"
   • Select the correct head repo, "personal/stan", and head branch, "feature/foo"
   • Fill in form and press "Send pull request".

   For a hotfix branch:
   

   1. Create a pull request.
   • Go to https://github.com/stan-dev/stan (refresh page if necessary)
     
   • Select the hotfix branch you are working on in the drop-down list. (Below the "Clone in ..." row)
     
   • A "Pull Request" button should pop up underneath the hotfix branch drop-down list. Click it.
   • Fill in form and press "Send pull request".
     
     Alternate directions:
     
   • Go to https://github.com/stan-dev/stan (refresh page if necessary)
   • Click "Pull Request" button. It's towards the top right, on the same row as the repo name "stan-dev/stan"
   • Select the correct base repo, "stan-dev/stan", and base branch, "master"
   • Select the correct head repo, "personal/stan", and head branch, "feature/foo"
   • Fill in form and press "Send pull request".
   • 
     
5. Relax.
   At this point, one or more of the developers will check the pull request and do an informal code review. At the same time, our Jenkins box will verify that all tests pass on Windows. If both are good, then the pull request will be accepted, the code will get into the correct branch, and the branch will be deleted.
6. If the pull request is not accepted immediately, fix the issues.
   
   1. Check out the branch
      > git checkout feature/foo
      > git pull origin feature/foo
      (Or you can set upstream tracking so you can do "git pull feature/foo")
      
   2. Do work.
      Others can help if you've given them access. If this workflow is a problem, follow directions for cloning.
      The work will update the pull request.
      
   3. Go back to "5. Relax"

How to Update Your Current Stan Directory

If you have no changes, just follow the above directions on cloning. If that doesn't work, do:

> git pull

That will pull down all the current branches. If you were on working on branch "foo", the new branch will be at "feature/foo".

Commit all changes to foo:

> git checkout foo

> git commit -m "mesage" -a

Check out the new feature branch named "feature/foo"

> git checkout feature/foo

Merge the changes in your current branch:

> git merge foo

Delete old branch:

> git branch -d foo

(shouldn't get warnings -- you've already merged into feature/foo)

Start working like normal. You're at step "3. Do Work" from the sections above.

(Informal) Code Review

We need to define what we're looking for, but to start:

- unit tests

- style

- correctness (by eye)

Here, it helps to be on the shared repo because this will allow other developers to help create tests if you're bogged down to finish out the process.

That should be most of the things we need for developers. If anything is unclear, please let me know. The Stan repo is back to being operational.

Daniel

[Marco Inacio]

This is probably useless to say since is already warned in git 1.8, but I would recommend everybody that haven't already done it, to run the following:

git config --global push.default simple

This is to avoid the command git push origin to inadvertently push to all branches which could cause, for example, one to "update" master in case one has the old version of it.

After doing this, git push origin will only push to the current branch (this will be default for git 2+).


Also, I have corrected my local and fork branches without so much trouble using the simple commands below.

Note that I'm not suggesting anybody to run them though. I'm just writing this to say that it worked well following this path.

git fetch --all
git checkout -b develop origin/develop
git branch -d master #error would prompt if I had worked directly in master branch
git checkout -b master origin/master
git push -f fork master
git push -f fork develop

where fork is the name I gave to my fork remote. After this I changed develop to be the base branch at Github's config and also set-uptream to a local metropolis branch I have.

[Ben Goodrich]

On Saturday, May 18, 2013 12:53:50 AM UTC-4, Marco Inacio wrote:

This is probably useless to say since is already warned in git 1.8, but I would recommend everybody that haven't already done it, to run the following:

git config --global push.default simple

This is to avoid the command git push origin to inadvertently push to all branches which could cause, for example, one to "update" master in case one has the old version of it.

After doing this, git push origin will only push to the current branch (this will be default for git 2+).

I would strongly favor a rule that everyone with access to the Stan organization on GitHub install git 1.8 or greater so that you can do as Marco suggests. In addition, you want to do (inside STAN_HOME locally)

git config merge.ff false

This prevents fast-forward merges which are evil under the gitflow paradigm. I didn't add the --global flag in case you are using git in another project that tolerates fast-forward merges.

Also, the attached file is a pre-push hook that should be saved at STAN_HOME/.git/hooks/pre-push (without any extension) and verify that its executable bit is set. This will prevent you from accidentally pushing to origin/master or origin/develop but requires git >= 1.8. Daniel and Bob might want to modify this hook so it merely asks "Are you sure you want to push to origin?" [y/n] and requires a confirmation.

Ben

[Marcus Brubaker]

(I'll probably respond more later but, a thought just occurred to me.)

If we're using pull requests to merge to develop, do we know what kind
of merge github does? I.E., does it do a ff or not? Is this
configurable?

Cheers,
Marcus

> --
> You received this message because you are subscribed to the Google Groups
> "stan development mailing list" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to stan-dev+u...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

[Marco Inacio]

the same occurred to me yesterday, this answered:

http://stackoverflow.com/questions/9994093/automatic-merge-of-pull-requests-on-github-without-the-merge-bubble

[Marcus Brubaker]

Ah, excellent, thanks Marco!

Cheers,
Marcus

[Bob Carpenter]

The writeup looks fantastic.

I'm going to drop Daniel's writeup into our Wiki:

https://github.com/stan-dev/stan/wiki/Developer-Process

Feel free to edit, especially if something's unclear and you
can clarify. It seemed like a steep learning curve to me,
but then I didn't have Daniel's writeup for our concrete case
to work from.

In addition to seconding the thanks to everyone for their patience,
I'd like to thank Daniel and Ben, who hashed the current process out,
as well as everyone who chipped in with critical bits of advice and config.
It's one thing to read about it on the web in pictures and quite another
to move an existing project into it using GitHub without losing our
history.

In retrospect, we should have listened to Ben the first time around when
he suggested the git flow process, but we didn't understand
the benefits of doing it that way or the dangers of doing
it the SVN way. This all became much more apparent as (a) the
project grew in terms of number of contributors, and (b) we started
to undertake bigger reorganizations of the code.

Next up, I think we're going to split the RStan repository off from
Stan Core. One reason is that we want to set up permissions so Jiqiang
and Ben control RStan instead of me and Daniel. Another is to make the
releases independent.

We should also be getting more integrations soon (Python, MATLAB, Julia,
Stata are on our list), and it should be easier all around if
these are separate from Stan Core. Then the RStan getting
started guide can move to GitHub's wiki.

I'm really psyched to start the new dev process. It'll probably
take us a while to clear out the clutter of old branches and
catch up with all the pull requests for non-critical features.
So thanks in advance for a bit more patience.

- Bob

[Ben Goodrich]

On Saturday, May 18, 2013 1:11:43 AM UTC-4, Ben Goodrich wrote:

Also, the attached file is a pre-push hook that should be saved at STAN_HOME/.git/hooks/pre-push (without any extension) and verify that its executable bit is set. This will prevent you from accidentally pushing to origin/master or origin/develop but requires git >= 1.8. Daniel and Bob might want to modify this hook so it merely asks "Are you sure you want to push to origin?" [y/n] and requires a confirmation.

I forgot to also check whether the push is directed toward origin/hotfix-* or origin/release-*, which should be precluded for everyone except Daniel and Bob. The attached pre-push hook fixes that.

Ben

[Marco Inacio]

On 13-05-18 02:11 AM, Ben Goodrich wrote:

I would strongly favor a rule that everyone with access to the Stan organization on GitHub install git 1.8 or greater so that you can do as Marco suggests. In addition, you want to do (inside STAN_HOME locally)

git config merge.ff false

This prevents fast-forward merges which are evil under the gitflow paradigm. I didn't add the --global flag in case you are using git in another project that tolerates fast-forward merges.

This seems to cause a side-effect: whenever you want to update (git pull) your local develop branch you are not able to fast forward, and thus have to create a commit "Merge branch 'develop' of git://github.com/stan-dev/stan into develop" and a merge bubble.

[Bob Carpenter]

Thanks for the feedback.

Could you explain what a "merge bubble" is and why you
think it's bad?

I thought we wanted to keep track of the merges rather
than ever using fast forward. I hadn't thought about it
from the perspective of merging develop back into feature
branches.

- Bob

[Ben Goodrich]

This is a side effect but is it a problem? I could see how it would make the history less pretty for local master and local development branches, but does it adversely affect anything else? I had been presuming that we would rebase feature branches anyway.

Ben

[Marco Inacio]

On 13-05-20 03:45 AM, Ben Goodrich wrote:
>
> This is a side effect but is it a problem? I could see how it would
> make the history less pretty for local master and local development
> branches, but does it adversely affect anything else?

You got a point, I really don't see a problem there other than making
the history less pretty, I'm just so used to ideia of local branch being
identical to remotes that I found this a little scary, but I guess
there's no harm.

Also, it seems that git pull --ff is able to overwrite this config.

> I had been presuming that we would rebase feature branches anyway.
>

I didn't get very well this point in the other discussions, is it better
to rebase all your commits to a single one before sending a pull
request? I think this is related to git bisect, right? But I never used
it so I don't know what is a desirable history.

[Marco Inacio]

On 13-05-20 03:45 AM, Bob Carpenter wrote:

Thanks for the feedback.

Could you explain what a "merge bubble" is and why you
think it's bad?

Merge bubbles are those cool diagrams of commits with branches that are generated in history every time a non fast forward merge is done.

I thought we wanted to keep track of the merges rather
than ever using fast forward.  I hadn't thought about it
from the perspective of merging develop back into feature
branches.

In fact, the "problem" is about merging the local develop branch with the remote develop branch (either via git pull or git fetch && git merge). Agreed that non-ff is good everywhere else.

[Ben Goodrich]

On Monday, May 20, 2013 3:15:30 AM UTC-4, Marco Inacio wrote:

Also, it seems that git pull --ff is able to overwrite this config.

Yeah, not a big deal. We're mostly trying to configure things to prevent ourselves from accidentally doing the wrong thing. But, if you want to explicitly git pull --ff to improve the looks of your history, that is fine.
 

> I had been presuming that we would rebase feature branches anyway.
>
I didn't get very well this point in the other discussions, is it better
to rebase all your commits to a single one before sending a pull
request? I think this is related to git bisect, right? But I never used
it so I don't know what is a desirable history.

We'll try to establish a wiki page about commits soon. It is generally good to git rebase a feature branch but often not to a single commit. Basically, you can git rebase to organize your history retroactively. So, if you had a feature branch like

commit A: add feature
commit B: oops, fix something with commit A
commit C: really fix the problem
commit D: add tests
commit E: add documentation
commit F: add a subfeature, with tests, and documentation

you could rebase this so that commits B through E became part of commit A, so it could publicly look like

commit ABCDE: add feature, plus tests and documentation
commit F: add a subfeature with tests and documentation

This is not only easier to read and understand, if it later becomes necessary to do a git bisect or a git revert, then things go smoother. For example, if git bisect landed on commit B or commit C, it would mark the commit as bad even though it was looking for a different problem. Or if we had to revert the feature committed in A, commit E would still be there documenting a feature that was reverted.

So, the basic principle is to make commits that consist of logical parts, add value, are easy to understand, are easy to bisect, and are easy to revert. And all that is hard to do in one take, so you can use git rebase to accomplish it after the fact. See especially the -i option for an interactive rebase and the ability to execute, for example, make test-unit on every commit.

Ben
 

[Marco Inacio]

Thanks, I'll try to follow this principle from now on.