Repo and switching manifest branches / Production branches management

[Florian Vallee]

Hi All,

I'm new to repo and I'm currently having a hard time understanding how the manifest branches are supposed to be handled.

Here is my situation : I work in a software company where we have a master branch, then a limited number of supported release branches forked off the master branch. I'm trying to see which architecture/workflow could suit our need using repo+git. (We're transitioning from CVS)

So far I've kind of hit a wall trying to see how developers could switch back and forth between the development branch (master) and the various maintenance branches for already released products.

My current setup is the following. In the manifest repository master I've set the default tracking branch to master :
   <default revision="master"
            remote="myremote"
            sync-j="4" />
Then I've created a manifest-branch for a maintenance branch, let's call in maintenance-1.0. The manifest stays the same only for the default revision which is in this case : default revision="prod_branch_v1.0"

No let's try this out, I'm a developer I start by checking out the repository :
* repo init -u ~/public/manifest.git
* repo sync
* repo start --all master

[Do some work ...]

Until this point everything works fine.

Now, at some point let's say that the local repository is in sync with the central repository and the topic branch master is checked out.

The developer needs to switch to a production branch, for instance in order to port some hotfixes there.

I would expect the following sequence to allow this seamlessly
* repo init -b maintenance-1.0
* repo sync
* repo start --all prod_branch_v1.0

However trying to use this workflow I'm running into some troubles :
* First of all, it looks like repo will try rebasing the contents of the master-manifest onto the production-manifest. For instance, if the two have diverged repo will drop me at the beginning of the procedure complaining that the rebase failed.

Looking into this I stumbled upon the following post :
https://groups.google.com/forum/?fromgroups#!searchin/repo-discuss/manifest$20branch/repo-discuss/VIJEv1kSuNU/L_s7iXKZJoMJ

Which seem to suggest that this a bug in repo, however this post has not been updated since 2009, is this still valid ?
* If the two manifests did not diverge, the init -b works just fine.
However the sync is giving the same troubles : instead of seeing that the manifest branch has changed and switching to the new default revision, repo is trying to rebase again :

Deleting obsolete path ~/repotest/docs
Deleting obsolete path ~/repotest/thirdparty
Deleting obsolete path ~/repotest/wrappers
code/: manifest switched master...prod_branch_v1.0
code/: discarding 314 commits removed from upstream
project code_2/
First, rewinding head to replay your work on top of it...
[..]

Rebase fails ..

If instead of this I start from a "detached" repo (ie I don't create the initial master topic using repo start --all master) then this sync step looks better. However I still get
code/: discarding 314 commits
messages, which I don't really know what to think about.
Anyway from that point the repositories seem to be in a good state with the correct revision being checked out and branch tracking correctly set up :
* (no branch)
  remotes/m/master -> myremote/master
  remotes/m/maintenance-1.0 -> myremote/prod_branch_v1.0
  remotes/myremote/master
  remotes/myremote/prod_branch_v1.0

But starting there however, if I try to go back to master :
repo init -b master
repo sync

Then "nothing happens" and the tree stays checked out on maintenance-1.0.
Again this is because repo rebased the maintenance-1.0 manifest branch onto master, which result into the final manifest being the production one instead of the expected one.

Also note that
* repo init -u ~/public/manifest.git
* repo sync
* repo start --all master
* repo sync -d
* repo init -b maintenance-1.0
* repo sync

Does not work either.  I would expect repo sync -d to detach all repositories from master, but in practice it doesn't, issuing a repo status after repo sync -d shows that all repositories are still attached to master.

Now I think I must be getting it wrong. Seeing that any action I try out ends up with an unwanted rebase It looks like I'm misunderstanding the role of manifest branches : I don''t see how they could be used in practice if switching from one to the other always triggers rebases.
I've looked at how this is used in android, and it seems that the use case is pretty much the same and should suffer the same shortcomings (ie: first init on a specific manifest branch will work, but switching branches is not something you should do ...)

So to sum it up, I've got a few questions here :
* Is switching *manifest* branches in an already checked out repo supported at all ? Does this mean the only way to achieve what I want is for every developer to keep as many separate "checkouts" as we have production branches (+ master) ?
* Do you have any suggestion regarding how we could get this ability to switch between branches that are not in sync / not meant to be re-based on each others ? Because it looks there is currently no way to prevent a "sync" from re-basing everything
* More generally what would currently the right workflow to handle production branches using repo ?
* Which of the test case described above are "expected" and which are bugs ? For instance shouldn't repo sync -d return to a detached head state even if the tree is already on the right revision ?

Thank you in advance for your time,

Regards,

Florian

[Magnus Bäck]

On Monday, April 30, 2012 at 12:38 EDT,
Florian Vallee <florian...@gmail.com> wrote:

[...]

> However trying to use this workflow I'm running into some troubles :
> * First of all, it looks like repo will try rebasing the contents of
> the master-manifest onto the production-manifest. For instance, if the
> two have diverged repo will drop me at the beginning of the procedure
> complaining that the rebase failed.
>
> Looking into this I stumbled upon the following post :
> https://groups.google.com/forum/?fromgroups#!searchin/repo-discuss/manifest$20branch/repo-discuss/VIJEv1kSuNU/L_s7iXKZJoMJ
>
> Which seem to suggest that this a bug in repo, however this post has
> not been updated since 2009, is this still valid ?

I don't know. I do know that I've used "repo init -b branchname" a
number of times without problems. Have you made any changes to the
default branch of the manifest, i.e. does the local default branch
point to the commit from the upstream branch?

> * If the two manifests did not diverge, the init -b works just fine.
> However the sync is giving the same troubles : instead of seeing that
> the manifest branch has changed and switching to the new default
> revision, repo is trying to rebase again :
>
> Deleting obsolete path ~/repotest/docs
> Deleting obsolete path ~/repotest/thirdparty
> Deleting obsolete path ~/repotest/wrappers
> code/: manifest switched master...prod_branch_v1.0
> code/: discarding 314 commits removed from upstream
> project code_2/
> First, rewinding head to replay your work on top of it...
> [..]

If you've made commits on a topic branch (without uploading those
changes) and switch the manifest branch, the commits on the topic
branch will be rebased to whatever branch the new manifest branch
points to for this git. This is by design.

> Rebase fails ..
>
> If instead of this I start from a "detached" repo (ie I don't create
> the initial master topic using repo start --all master) then this sync
> step looks better. However I still get
> code/: discarding 314 commits
> messages, which I don't really know what to think about.

That message means that the branch you're switching *from* (in the git
in question) reaches 314 commits that aren't reachable from the branch
you're switching *to*.

[...]

> So to sum it up, I've got a few questions here :
> * Is switching *manifest* branches in an already checked out repo
> supported at all ? Does this mean the only way to achieve what I want
> is for every developer to keep as many separate "checkouts" as we have
> production branches (+ master) ?

This should work and I've done it many times myself. I think we need
more details to diagnose this.

Now, depending on what type of software you build, which tools you use,
how much source code you have etc, it might make sense to have separate
trees for each manifest branch. Recall that Git will update the mtime as
files are checked out, and with a build system that uses the mtime for
build avoidance you'll end up rebuilding a lot of stuff if you switch
back and forth between branches.

[...]

> * Which of the test case described above are "expected" and which are
> bugs ? For instance shouldn't repo sync -d return to a detached head
> state even if the tree is already on the right revision ?

That's what I'd expect.

--
Magnus Bäck
ba...@google.com

[Michael Livshin]

On Monday, April 30, 2012 9:23:12 PM UTC+3, Magnus Bäck wrote:

 

I don't know. I do know that I've used "repo init -b branchname" a
number of times without problems. Have you made any changes to the
default branch of the manifest, i.e. does the local default branch
point to the commit from the upstream branch? 

[ I'm not the topic originator, I just happen to be puzzled by similar repo behavior ]

"repo init -b branchname" will "work without problems" whenever git can cleanly merge

the "branchname" contents into the "default" manifest branch, and will encounter

problems otherwise.  It doesn't matter whether there are actual local changes.

I've just tried to manually "detach the head" of the manifest repository (.repo/manifests) 

prior to switching manifest branch, and no manifest merge occurred at all (as expected).

"rm -rf .repo/manifest*" obviously works too.

So basically the questions are:

- Why does this local "default" branch in the manifest repository get created at all?  This

repository seems generally to be treated inconsistently by repo: "repo init" does not leave 

it in the friendly "detached head" state like "repo sync" does with the normal ones, but all other

logic treats it just like any other managed repository.  Come to think about it, why does it

even have to be a proper repository (as opposed to just dropping the desired manifest file

under .repo)?  What use case is this, er, git meta-circling supposed to enable (this is probably 

documented somewhere but I wasn't able to find anything...)?

- Seeing as switching manifest branches can only work without manual intervention if

you are lucky, is such switching really supported or is it better avoided?

Thanks,

--m

[Florian Vallee]

Hi Magnus,

Thank you for your comments, please find my answers in-line below

On Monday, April 30, 2012 8:23:12 PM UTC+2, Magnus Bäck wrote:

> Looking into this I stumbled upon the following post :
> https://groups.google.com/forum/?fromgroups#!searchin/repo-discuss/manifest$20branch/repo-discuss/VIJEv1kSuNU/L_s7iXKZJoMJ
>
> Which seem to suggest that this a bug in repo, however this post has
> not been updated since 2009, is this still valid ?

I don't know. I do know that I've used "repo init -b branchname" a
number of times without problems. Have you made any changes to the
default branch of the manifest, i.e. does the local default branch
point to the commit from the upstream branch?

 

No I did not make any local changes, the manifest points by default to the HEAD of
the master branch and my current checkout is unmodified (both in the detached case
and the "master topic" case).

> * If the two manifests did not diverge, the init -b works just fine.
> However the sync is giving the same troubles : instead of seeing that
> the manifest branch has changed and switching to the new default
> revision, repo is trying to rebase again :
>
> Deleting obsolete path ~/repotest/docs
> Deleting obsolete path ~/repotest/thirdparty
> Deleting obsolete path ~/repotest/wrappers
> code/: manifest switched master...prod_branch_v1.0
> code/: discarding 314 commits removed from upstream
> project code_2/
> First, rewinding head to replay your work on top of it...
> [..]

If you've made commits on a topic branch (without uploading those
changes) and switch the manifest branch, the commits on the topic
branch will be rebased to whatever branch the new manifest branch
points to for this git. This is by design.

That is understandable, the thing here is that I'm creating a topic branch
from a fresh checkout and leaving the topic branch unmodified. So the
topic branch is synced with master.

When I switch manifest branches it looks like repo will find the common
ancestor between head and the production branch I'm switching to, then
try to apply all the changes that occurred since this forking point on top of
the new default revision.

I would have expected it to just detach from the current topic branch in that
case since it was in sync before the manifest switch

> So to sum it up, I've got a few questions here :
> * Is switching *manifest* branches in an already checked out repo
> supported at all ? Does this mean the only way to achieve what I want
> is for every developer to keep as many separate "checkouts" as we have
> production branches (+ master) ?

This should work and I've done it many times myself. I think we need
more details to diagnose this.

I've created some test repositories that you can use to see these behaviours
for yourself.

I've packed these in a tar file that is available here :
http://www.ey3ball.net/~florian/repo_test.tar

It's quite simple to use :
cd ~/test
wget http://www.ey3ball.net/~florian/repo_test.tar
tar xvf repo_test.tar
mkdir client
cd client
repo init -u ../repo_test/manifest.git/
repo sync

The only requirement to use this package is that the repo_test directory is
located in ../repo_test/ starting from the location where you're running repo init.
 
These test repositories are quite simple, the tarball contains :
* manifest.git
* code.git
* doc.git

These 3 repositories have two branches each : master and prod_branch_v1.0
in all repositories (including the manifest one).
Moreover I've made sure that master and prod_branch_v1.0 diverge enough
so that any rebase in one direction or the other will trigger conflicts.

Now a few test cases (all of these start with a fresh repo checkout):

(1) repo sync -d behaviour :
$ repo init -u ../repo_test/manifest.git/
$ repo sync
$ repo start --all master
$ repo status
project code/                                   branch master
project doc/                                    branch master
$ repo sync -d
$ repo status
project code/                                   branch master
project doc/                                    branch master

In this case the expected behaviour (IMHO) would be to end up detached
(repo status should show :
$ repo status
nothing to commit (working directory clean))

(1b) repo sync -d behaviour :
$ repo init -u ../repo_test/manifest.git/
$ repo sync
$ repo start --all master
$ repo status
project code/                                   branch master
project doc/                                    branch master
$ cd doc
$ echo "test" > README
$ git commit README -m "detach test"
$ cd ..
$ repo sync -d
$ repo status
project code/                                   branch master

Result:
doc has been detached (expected)
code has not (unexpected ?)

(2) Switching manifest branches
$ repo init -u ../repo_test/manifest.git/
$ repo sync
$ repo init -b prod_branch_v1.0
.repo/manifests/: manifest switched refs/heads/master...prod_branch_v1.0
project .repo/manifests/

First, rewinding head to replay your work on top of it...

Applying: master: diverge from prod_branch_v1.0
Using index info to reconstruct a base tree...
Falling back to patching base and 3-way merge...
Auto-merging default.xml
CONFLICT (content): Merge conflict in default.xml
Recorded preimage for 'default.xml'
Failed to merge in the changes.
Patch failed at 0001 master: diverge from prod_branch_v1.0
[..]

In that case I would have expected the branch switch to just checkout manifest branch prod_branch_v1.0 without trying to rebase

(3) Switching manifest branches when there is no conflict :
$ pushd ../repo_test/manifest.git/
$ git reset --soft aa625abba2562671b52fc967d0de9646c9153714
$ # Comes back to initial manifest on HEAD. This one does not conflicts with prod_branch_v1.0
$ popd
$ repo init -u ../repo_test/manifest.git/
$ repo sync
$ repo init -b prod_branch_v1.0
.repo/manifests/: manifest switched refs/heads/master...prod_branch_v1.0
project .repo/manifests/
Updating aa625ab..0126108
Fast-forward
 default.xml |    4 ++--
 1 files changed, 2 insertions(+), 2 deletions(-)

Your identity is: Florian <florian.va...@gmail.com>
If you want to change this, please re-run 'repo init' with --config-name

repo initialized in /users/vallee/test/client
$ # Rebase happens because of (2), but we'll ignore that here since it just works
$ repo sync
code/: discarding 2 commits
doc/: discarding 2 commits
$ # Look around, everything is fine
$ repo init -b master
$ # Go back to master

Your identity is: Florian <florian.va...@gmail.com>
If you want to change this, please re-run 'repo init' with --config-name

repo initialized in /users/vallee/test/client
$ repo sync

$ # Nothing happens, tree is stuck on production_v1.0
 

Now, depending on what type of software you build, which tools you use,
how much source code you have etc, it might make sense to have separate
trees for each manifest branch. Recall that Git will update the mtime as
files are checked out, and with a build system that uses the mtime for
build avoidance you'll end up rebuilding a lot of stuff if you switch
back and forth between branches.

I agree having different workstate would be the best way to got most of the time,
but I still think switching branches using -b is a valid use case.

For instance in my case we release a bundle of several components. It would
make sense to have two release branches that are almost identical expect for
a few components. In that case you'll end up rebuilding these few components
only, instead of having to do everything from scratch.
Also one developer could have a local manifest branch with custom versions
of certain components he wants to stick with, along with code from master.

Finally since I can't prevent devs from trying to do this anyway, so it would be
better to get it to work as expected :)

--
Florian Vallee

[Florian Vallee]

Also, here is the output of repo --version on my computer :

$ repo --version
repo version v1.8.2
       (from https://code.google.com/p/git-repo/)
repo launcher version 1.15
       (from /users/vallee/bin/repo)
git version 1.7.2.5
Python 2.6.6 (r266:84292, Dec 26 2010, 22:31:48)
[GCC 4.4.5]

--
Florian Vallee

[Florian Vallee]

Hi All,

If you want to change this, please re-run 'repo init' with --config-name

repo initialized in /users/vallee/test/client
$ # Rebase happens because of (2), but we'll ignore that here since it just works
$ repo sync
code/: discarding 2 commits
doc/: discarding 2 commits
$ # Look around, everything is fine
$ repo init -b master
$ # Go back to master

Your identity is: Florian <>

If you want to change this, please re-run 'repo init' with --config-name

repo initialized in /users/vallee/test/client
$ repo sync

$ # Nothing happens, tree is stuck on production_v1.0

 
Following this discussion I've proposed two patches, One to fix the sync -d behaviour, the other to fix repo init -b on an already checked out repo tree.

Here are the links to gerrit-review for reference :
https://gerrit-review.googlesource.com/#/c/35900/
https://gerrit-review.googlesource.com/#/c/35890/

Also the bugtracker issue for sync -d :
https://code.google.com/p/git-repo/issues/detail?id=46

I've been personally running repo with these patches for a few weeks now, they seem to work just fine.

Hope it helps,

Regards

--
Florian Vallee

[Brian Harring]

On Mon, Apr 30, 2012 at 02:23:12PM -0400, Magnus B??ck wrote:
> On Monday, April 30, 2012 at 12:38 EDT,
> Florian Vallee <florian...@gmail.com> wrote:
>
> [...]
>
> > However trying to use this workflow I'm running into some troubles :
> > * First of all, it looks like repo will try rebasing the contents of
> > the master-manifest onto the production-manifest. For instance, if the
> > two have diverged repo will drop me at the beginning of the procedure
> > complaining that the rebase failed.
> >
> > Looking into this I stumbled upon the following post :
> > https://groups.google.com/forum/?fromgroups#!searchin/repo-discuss/manifest$20branch/repo-discuss/VIJEv1kSuNU/L_s7iXKZJoMJ
> >
> > Which seem to suggest that this a bug in repo, however this post has
> > not been updated since 2009, is this still valid ?
>
> I don't know. I do know that I've used "repo init -b branchname" a
> number of times without problems. Have you made any changes to the
> default branch of the manifest, i.e. does the local default branch
> point to the commit from the upstream branch?

We've ran into this a fair bit in chrome-os actually, although I've
not yet nailed down a fix (and we have a simple workaround via
checking out detached head, then wiping the default branch).


It's replicated via the following:

mkdir tmp;
cd tmp;

repo init -u http://git.chromium.org/chromiumos/manifest.git -b master
repo init -u http://git.chromium.org/chromiumos/manifest.git \
-b release-R18-1660.B

Pretty simple to replicate; in terms of history, basically R18 has
diverged by one rev from an earlier point in masters history;
specifically the commit rewriting revision fields to the new branch
namespace.

Last time I looked at this, the separation of Sync_NetworkHalf and
Sync_LocalHalf looked to be a core blocker; specifically, local, to
decide if anything needs to be rebased across (versus just resetting),
has to know what tracking was *prior* to the network half syncing.

Since that isn't preserved, it can't do the quick "should I even try
to rebase" check, resulting in it going down the failing pathway
mentioned above.

Re: simple workaround; running the following prior to a reinit
addresses it:

git checkout -f --detach HEAD
git branch -D default

~brian