The state of the branches.

[Jean-Baptiste Queru]

I've just spent a day and a half in the quietness of the week-end to
merge donut into master, which should bring the codelines in a "clean"
state as far as the process is concerned (working on a week-end
allowed me to work with less risk that the codelines would change
under my feet).

Here are the various relevant "moving" development branches and their
respective descriptions:

-release-1.0 was loosely based on the code that went into Android 1.0
and 1.1. It's only here for historical purposes at this point, hasn't
received any change in a long time and probably never will receive
additional changes.
-cupcake is a close copy of Google's main cupcake branch. 1.5 SDKs
(and the matching online documentation) are generated from Google's
variant. it's not getting much activity any more.
-cupcake-release is a variant of cupcake that matches more closely the
builds that end up on Google-Experience 1.5 devices. It receives
changes cherry-picked from cupcake.
-donut is the branch in which the next significant release is being
developed. Its current state matches the state of Google's donut
branch (minus non-Open-Source parts) as of July 8th. This isn't a
final version by any stretch of the imagination. Google's internal
version of the donut branch is already several weeks ahead and is
still being being actively worked on.
-master is the bleeding edge on the Open-Source side of things, it
contains all the current contributions from Google, all the community
contributions, and changes made there so far have almost all been
merged into Google's master tree (which is being used for post-donut
research and development). It sits "in between" two of Google's trees,
being a superset of the donut tree but a subset of Google's master
tree. This is the only Open-Source branch currently open for
contributions, as Google's source management process can't currently
deal with changes done in other Open-Source branches.

There are also a few "fixed" branches/tags, which are close copies of
previous releases:

-android-1.0 matches the very first code drop of 1.0; it's not very
meaningful as it was only a very loose match for the real 1.0.
-android-sdk-1.5-pre, android-sdk-1.5_r1 and android-sdk-1.5_r3 match
the different releases of the 1.5 SDK (and _r2 is missing for lack of
time). Those are along the cupcake branch.
-android-1.5 and android-1.5r2 respectively match the CRB17 and CRB43
device builds of the 1.5 release (CRB21 is identical to CRB17 as far
as the Open-Source side is concerned). Those are along the
cupcake-release branch. I'm hoping to put CRC01 there when I get the
opportunity.

As a general rule, the so-called Android platform only contains the
projects that are referenced by the official manifests, i.e. code that
is used to build Google-Experience devices. Projects that are hosted
on android.git.kernel.org but aren't referenced by manifests are
provided as extras, independently from and/or unmaintained by Google
and OHA. I'm hoping to make the distinction clearer in the future.

Looking closely at the details of the current state, there are
actually a few notable exceptions to the rules I mentioned above:
-the hardware/msm7k project is somewhat out of date across the board
when compared to the other projects.
-the vendor/htc/dream-open project is currently entirely untested and
pretty much unmaintained.
-the external/opencore project isn't merged from donut to master, and
gets imported into Google's server on its own schedule.
-the frameworks/base and packages/apps/Phone projects don't currently
have their true history exposed. Google's copies contain information
that it currently confidential and therefore can't be disclosed yet,
so the currently visible recent history for those projects is a mere
snapshot.
-the history of the manifests is still quite dirty, and as a result
it's hard to switch between branches.

It's also worth mentioning that Google's primary copy of cupcake
doesn't live in git, and that the script that replicates from our
legacy system into git has been known to have bugs. The git copy
probably deviates a bit from Google's primary version, and isn't
tested at all. Also, for all branches, the Android Open-Source Project
code only represents a part of the code used to build
Google-Experience devices, and Google's current testing process
doesn't test the code from the Android Open-Source Process separately
from the non-Open-Source code; as a result, it's known that some parts
of the Android Open-Source Project depend on non-Open-Source projects,
sometimes accidentally.

Finally, there's been an overall process issue with the way merges are
managed around the open-source master branch, which has been creating
some drift with the other codelines, such that there could be tidbits
code from donut that don't appear in master, or tidbits of code from
master that don't appear in Google's internal tree. Those were all
accidental results of merge conflicts (other than for opencore), but
they are an annoyance. I'll try to limit such drift going forward. I
think that I have an idea for a process change inside Google that
should eliminate that drift, but (assuming that it gets accepted and
implemented) its effects won't be visible until the first code drop
from eclair, so we'll have to live with the current situation for a
while longer.

JBQ

--
Jean-Baptiste M. "JBQ" Queru
Software Engineer, Android Open-Source Project, Google.

Questions sent directly to me that have no reason for being private
will likely get ignored or forwarded to a public forum with no further
warning.

[Johan de Koning]

JBQ, I saw today that there is an android-1.5r3 tag. Can you give some
details about what this tag contains?

[Jean-Baptiste Queru]

In trying to avoid cross-posting (gotta practice what I preach), I
only sent the information to another list where I thought that
information was more relevant.

This morning, I uploaded one new change to the cupcake-release branch,
and tagged the resulting state as android-1.5r3. It matches the CRC01
build that was deployed to user devices recently. It is the latest
"official" release that device manufacturers should be using.

As a general rule, I expect that platform contributors (i.e. the
audience of this list) will be mostly using the heads of the primary
branches, i.e. cupcake, donut, master (especially master), and usually
won't care much for tags or release branches.

JBQ

[Brad Larson]

Hi JBQ,

Thanks for the update! I'm curious about your code-merging process...
any hints or tips you could share? Any strict rules you have the
developers follow? I'm getting pretty good at merging our internal
changes with changes upstream, but I always wonder if there is a
better way. If you have a solid process down and get time to document
any of it ( ;) ) I'm sure it would come in handy for several of us.

Thanks!
Brad

[Samuel]

Hi JBQ,

First off, thanks for your efforts!!

After reading through all posts in the current thread (mainly from you-
JBQ), I got the big picture of how to pick out/deal with android
version amongst different branches/heads, or tags.

Meanwhile, I also extract some highlights from what is described
above, as shown below.
a. Advanced Hacker ==> Master (due to the so bleeding-edge)
==> Newest Project(branch) (currently,
that is "donut")

b. Basic Hacker ==> xxx (here's "xxx" specifying project name,
e.g. cupcake, donot..., constantly growing)

c. OEM ==> xxx-release (here's "xxx" indicating
project name, available and stable)

d. Normal player ==> xxx (above)
==> xxx-release (above)

Additionally, I still wonder that what separately the following
special terms (or jargon) stand for.
CRB17 CRB21 CRB43
CRC01

Thanks a lot!!


B. R.
Samuel

On Aug 1, 6:56 am, Jean-Baptiste Queru <j...@android.com> wrote:
> In trying to avoid cross-posting (gotta practice what I preach), I
> only sent the information to another list where I thought that
> information was more relevant.
>
> This morning, I uploaded one new change to the cupcake-release branch,

> and tagged the resulting state asandroid-1.5r3. It matches the CRC01

> build that was deployed to user devices recently. It is the latest
> "official" release that device manufacturers should be using.
>
> As a general rule, I expect that platform contributors (i.e. the
> audience of this list) will be mostly using the heads of the primary
> branches, i.e. cupcake, donut, master (especially master), and usually
> won't care much for tags or release branches.
>
> JBQ
>
>
>

> On Fri, Jul 31, 2009 at 3:03 PM, Johan de Koning<ikbennu...@gmail.com> wrote:
>
> > JBQ, I saw today that there is anandroid-1.5r3 tag. Can you give some

> > details about what this tag contains?
>
> > On 26 jul, 23:20, Jean-Baptiste Queru <j...@android.com> wrote:
> >> I've just spent a day and a half in the quietness of the week-end to
> >> merge donut into master, which should bring the codelines in a "clean"
> >> state as far as the process is concerned (working on a week-end
> >> allowed me to work with less risk that the codelines would change
> >> under my feet).
>
> >> Here are the various relevant "moving" development branches and their
> >> respective descriptions:
>
> >> -release-1.0 was loosely based on the code that went intoAndroid1.0
> >> and 1.1. It's only here for historical purposes at this point, hasn't
> >> received any change in a long time and probably never will receive
> >> additional changes.
> >> -cupcake is a close copy of Google's main cupcake branch. 1.5 SDKs
> >> (and the matching online documentation) are generated from Google's
> >> variant. it's not getting much activity any more.
> >> -cupcake-release is a variant of cupcake that matches more closely the
> >> builds that end up on Google-Experience 1.5 devices. It receives
> >> changes cherry-picked from cupcake.
> >> -donut is the branch in which the next significant release is being
> >> developed. Its current state matches the state of Google's donut
> >> branch (minus non-Open-Source parts) as of July 8th. This isn't a

> >> finalversionby any stretch of the imagination. Google's internal
> >>versionof the donut branch is already several weeks ahead and is

> >> still being being actively worked on.
> >> -master is the bleeding edge on the Open-Source side of things, it
> >> contains all the current contributions from Google, all the community
> >> contributions, and changes made there so far have almost all been
> >> merged into Google's master tree (which is being used for post-donut
> >> research and development). It sits "in between" two of Google's trees,
> >> being a superset of the donut tree but a subset of Google's master
> >> tree. This is the only Open-Source branch currently open for
> >> contributions, as Google's source management process can't currently
> >> deal with changes done in other Open-Source branches.
>
> >> There are also a few "fixed" branches/tags, which are close copies of
> >> previous releases:
>
> >> -android-1.0 matches the very first code drop of 1.0; it's not very
> >> meaningful as it was only a very loose match for the real 1.0.

> >> -android-sdk-1.5-pre,android-sdk-1.5_r1 andandroid-sdk-1.5_r3 match

> >> the different releases of the 1.5 SDK (and _r2 is missing for lack of
> >> time). Those are along the cupcake branch.

> >> -android-1.5 andandroid-1.5r2 respectively match the CRB17 and CRB43

> >> device builds of the 1.5 release (CRB21 is identical to CRB17 as far
> >> as the Open-Source side is concerned). Those are along the
> >> cupcake-release branch. I'm hoping to put CRC01 there when I get the
> >> opportunity.
>

> >> As a general rule, the so-calledAndroidplatform only contains the

> >> projects that are referenced by the official manifests, i.e. code that
> >> is used to build Google-Experience devices. Projects that are hosted

> >> onandroid.git.kernel.org but aren't referenced by manifests are

> >> provided as extras, independently from and/or unmaintained by Google
> >> and OHA. I'm hoping to make the distinction clearer in the future.
>
> >> Looking closely at the details of the current state, there are
> >> actually a few notable exceptions to the rules I mentioned above:
> >> -the hardware/msm7k project is somewhat out of date across the board
> >> when compared to the other projects.
> >> -the vendor/htc/dream-open project is currently entirely untested and
> >> pretty much unmaintained.
> >> -the external/opencore project isn't merged from donut to master, and
> >> gets imported into Google's server on its own schedule.
> >> -the frameworks/base and packages/apps/Phone projects don't currently
> >> have their true history exposed. Google's copies contain information
> >> that it currently confidential and therefore can't be disclosed yet,
> >> so the currently visible recent history for those projects is a mere
> >> snapshot.
> >> -the history of the manifests is still quite dirty, and as a result
> >> it's hard to switch between branches.
>
> >> It's also worth mentioning that Google's primary copy of cupcake
> >> doesn't live in git, and that the script that replicates from our
> >> legacy system into git has been known to have bugs. The git copy

> >> probably deviates a bit from Google's primaryversion, and isn't
> >> tested at all. Also, for all branches, theAndroidOpen-Source Project

> >> code only represents a part of the code used to build
> >> Google-Experience devices, and Google's current testing process

> >> doesn't test the code from theAndroidOpen-Source Process separately

> >> from the non-Open-Source code; as a result, it's known that some parts

> >> of theAndroidOpen-Source Project depend on non-Open-Source projects,

> >> sometimes accidentally.
>
> >> Finally, there's been an overall process issue with the way merges are
> >> managed around the open-source master branch, which has been creating
> >> some drift with the other codelines, such that there could be tidbits
> >> code from donut that don't appear in master, or tidbits of code from
> >> master that don't appear in Google's internal tree. Those were all
> >> accidental results of merge conflicts (other than for opencore), but
> >> they are an annoyance. I'll try to limit such drift going forward. I
> >> think that I have an idea for a process change inside Google that
> >> should eliminate that drift, but (assuming that it gets accepted and
> >> implemented) its effects won't be visible until the first code drop
> >> from eclair, so we'll have to live with the current situation for a
> >> while longer.
>
> >> JBQ
>
> >> --
> >> Jean-Baptiste M. "JBQ" Queru

> >> Software Engineer,AndroidOpen-Source Project, Google.

>
> >> Questions sent directly to me that have no reason for being private
> >> will likely get ignored or forwarded to a public forum with no further
> >> warning.
>
> --
> Jean-Baptiste M. "JBQ" Queru

> Software Engineer,AndroidOpen-Source Project, Google.

[Samuel]

Hi JBQ,

First off, thanks for your efforts!!

After reading through all posts in the current thread (mainly from you-
JBQ), I got the big picture of how to pick out/deal with android
version amongst different branches/heads, or tags.

Meanwhile, I also extract some highlights from what is described
above, as shown below.
a. Advanced Hacker ==> Master (due to the so bleeding-edge)
==> Newest Project(branch) (currently,
that is "donut")

b. Basic Hacker ==> xxx (here's "xxx" specifying project name,
e.g. cupcake, donot..., constantly growing)

c. OEM ==> xxx-release (here's "xxx" indicating
project name, available and stable)

d. Normal player ==> xxx (above)
==> xxx-release (above)

Additionally, I still wonder that what separately the following
special terms (or jargon) stand for.
CRB17 CRB21 CRB43
CRC01

Thanks a lot!!


B. R.
Samuel

On Aug 1, 6:56 am, Jean-Baptiste Queru <j...@android.com> wrote:

> In trying to avoid cross-posting (gotta practice what I preach), I
> only sent the information to another list where I thought that
> information was more relevant.
>
> This morning, I uploaded one new change to the cupcake-release branch,

> and tagged the resulting state asandroid-1.5r3. It matches the CRC01

> build that was deployed to user devices recently. It is the latest
> "official" release that device manufacturers should be using.
>
> As a general rule, I expect that platform contributors (i.e. the
> audience of this list) will be mostly using the heads of the primary
> branches, i.e. cupcake, donut, master (especially master), and usually
> won't care much for tags or release branches.
>
> JBQ
>
>
>

> On Fri, Jul 31, 2009 at 3:03 PM, Johan de Koning<ikbennu...@gmail.com> wrote:
>
> > JBQ, I saw today that there is anandroid-1.5r3 tag. Can you give some

> > details about what this tag contains?
>
> > On 26 jul, 23:20, Jean-Baptiste Queru <j...@android.com> wrote:
> >> I've just spent a day and a half in the quietness of the week-end to
> >> merge donut into master, which should bring the codelines in a "clean"
> >> state as far as the process is concerned (working on a week-end
> >> allowed me to work with less risk that the codelines would change
> >> under my feet).
>
> >> Here are the various relevant "moving" development branches and their
> >> respective descriptions:
>
> >> -release-1.0 was loosely based on the code that went intoAndroid1.0
> >> and 1.1. It's only here for historical purposes at this point, hasn't
> >> received any change in a long time and probably never will receive
> >> additional changes.
> >> -cupcake is a close copy of Google's main cupcake branch. 1.5 SDKs
> >> (and the matching online documentation) are generated from Google's
> >> variant. it's not getting much activity any more.
> >> -cupcake-release is a variant of cupcake that matches more closely the
> >> builds that end up on Google-Experience 1.5 devices. It receives
> >> changes cherry-picked from cupcake.
> >> -donut is the branch in which the next significant release is being
> >> developed. Its current state matches the state of Google's donut
> >> branch (minus non-Open-Source parts) as of July 8th. This isn't a

> >> finalversionby any stretch of the imagination. Google's internal
> >>versionof the donut branch is already several weeks ahead and is

> >> still being being actively worked on.
> >> -master is the bleeding edge on the Open-Source side of things, it
> >> contains all the current contributions from Google, all the community
> >> contributions, and changes made there so far have almost all been
> >> merged into Google's master tree (which is being used for post-donut
> >> research and development). It sits "in between" two of Google's trees,
> >> being a superset of the donut tree but a subset of Google's master
> >> tree. This is the only Open-Source branch currently open for
> >> contributions, as Google's source management process can't currently
> >> deal with changes done in other Open-Source branches.
>
> >> There are also a few "fixed" branches/tags, which are close copies of
> >> previous releases:
>
> >> -android-1.0 matches the very first code drop of 1.0; it's not very
> >> meaningful as it was only a very loose match for the real 1.0.

> >> -android-sdk-1.5-pre,android-sdk-1.5_r1 andandroid-sdk-1.5_r3 match

> >> the different releases of the 1.5 SDK (and _r2 is missing for lack of
> >> time). Those are along the cupcake branch.

> >> -android-1.5 andandroid-1.5r2 respectively match the CRB17 and CRB43

> >> device builds of the 1.5 release (CRB21 is identical to CRB17 as far
> >> as the Open-Source side is concerned). Those are along the
> >> cupcake-release branch. I'm hoping to put CRC01 there when I get the
> >> opportunity.
>

> >> As a general rule, the so-calledAndroidplatform only contains the

> >> projects that are referenced by the official manifests, i.e. code that
> >> is used to build Google-Experience devices. Projects that are hosted

> >> onandroid.git.kernel.org but aren't referenced by manifests are

> >> provided as extras, independently from and/or unmaintained by Google
> >> and OHA. I'm hoping to make the distinction clearer in the future.
>
> >> Looking closely at the details of the current state, there are
> >> actually a few notable exceptions to the rules I mentioned above:
> >> -the hardware/msm7k project is somewhat out of date across the board
> >> when compared to the other projects.
> >> -the vendor/htc/dream-open project is currently entirely untested and
> >> pretty much unmaintained.
> >> -the external/opencore project isn't merged from donut to master, and
> >> gets imported into Google's server on its own schedule.
> >> -the frameworks/base and packages/apps/Phone projects don't currently
> >> have their true history exposed. Google's copies contain information
> >> that it currently confidential and therefore can't be disclosed yet,
> >> so the currently visible recent history for those projects is a mere
> >> snapshot.
> >> -the history of the manifests is still quite dirty, and as a result
> >> it's hard to switch between branches.
>
> >> It's also worth mentioning that Google's primary copy of cupcake
> >> doesn't live in git, and that the script that replicates from our
> >> legacy system into git has been known to have bugs. The git copy

> >> probably deviates a bit from Google's primaryversion, and isn't
> >> tested at all. Also, for all branches, theAndroidOpen-Source Project

> >> code only represents a part of the code used to build
> >> Google-Experience devices, and Google's current testing process

> >> doesn't test the code from theAndroidOpen-Source Process separately

> >> from the non-Open-Source code; as a result, it's known that some parts

> >> of theAndroidOpen-Source Project depend on non-Open-Source projects,

> >> sometimes accidentally.
>
> >> Finally, there's been an overall process issue with the way merges are
> >> managed around the open-source master branch, which has been creating
> >> some drift with the other codelines, such that there could be tidbits
> >> code from donut that don't appear in master, or tidbits of code from
> >> master that don't appear in Google's internal tree. Those were all
> >> accidental results of merge conflicts (other than for opencore), but
> >> they are an annoyance. I'll try to limit such drift going forward. I
> >> think that I have an idea for a process change inside Google that
> >> should eliminate that drift, but (assuming that it gets accepted and
> >> implemented) its effects won't be visible until the first code drop
> >> from eclair, so we'll have to live with the current situation for a
> >> while longer.
>
> >> JBQ
>
> >> --
> >> Jean-Baptiste M. "JBQ" Queru

> >> Software Engineer,AndroidOpen-Source Project, Google.

>
> >> Questions sent directly to me that have no reason for being private
> >> will likely get ignored or forwarded to a public forum with no further
> >> warning.
>
> --
> Jean-Baptiste M. "JBQ" Queru

> Software Engineer,AndroidOpen-Source Project, Google.

[Jean-Baptiste Queru]

Let me give you my gut feeling about the use cases:

-platform contributor: master (we're not equipped to accept
contributions into any other branch at the moment). On this list, this
should be the case for most people.

-observer trying to figure out what Google is doing: donut (or the
latest named branch).

-developer using the SDK and trying to match stack traces: the tagged
version matching the SDK release. Same thing for on-device
development, though we haven't done a good job at matching the build
numbers with tags.

-OEM: ultimately, the latest numbered tag for the target release (e.g.
android-1.5r3). If working from a named project that doesn't have a
numbered tag yet, head of that project (e.g. donut).

Discussions about the last three cases don't normally belong on
android-platform (typically, android-discuss, android-developers and
android-porting respectively).


As for the build names, here's the "decoder ring":

-the first letter is the codename. C is cupcake, D is donut, E is eclair, etc...

-the second letter is the sub-branch used for the build. R is the main
release branch, but other branches sometimes exist (for cupcake there
are CD and CO in addition to CR).

-the next letter + 2 digits are a date code: the letter is the
quarter, with A being Q1'09, B being Q2'09, etc... and the next 2
digits are the day within the quarter.

-finally, if there are multiple named builds on the same day,
subsequent builds get an extra letter, starting with B (the first
build is an official A).

So, CRB43 is the first build of the day on May 13 2009 from the
cupcake-release branch. DRC31 is what I just pushed a few hours ago,
i.e. last Friday's first build from donut-release.

We moved to that scheme because people kept chopping off half of the
previous scheme and it was hard to know what people were talking
about.

CRB17 and CRB21 match android-1.5 (the difference between CRB17 and
CRB21 was in non-open-source parts), CRB43 matches android-1.5r2, and
CRC01 matches android-1.5r3.

JBQ

Software Engineer, Android Open-Source Project, Google.

[Jean-Baptiste Queru]

There are really 3 merges going on in the process:

-(1) inside Google, the donut branch gets merged onto the master
branch in real time.
-(2) when I do a code drop from Google's donut onto the AOSP one, I
merge it into master as well.
-(3) changes done in the AOSP master are merged into Google's master
when I get a chance, a few times a week (we're looking at automating
that to make it happen in real time).

When it comes to managing merges, there are two basic recommendations
for the team:
-avoid making the same change independently in two codelines (or at
least do a git cherry-pick).
-resolve the conflicts found by the auto-merger as quickly as possible.
As for the rest, it boils down to experience, experience, experience.
I've been involved somewhat closely with merges for a good part of my
career so far, so I got used to reading the clues and resolving the
conflicts.

As for the merges that I do myself (i.e. donut into master on the open
side), I try to handle them on our internal server as much as
possible. I maintain mirrors of the open-source trees on our internal
server (which costs nothing thanks to git), and I stage the pushes
there as well, so once I have a staged push for donut and a mirror of
the open-source master I can just merge them internally and I push the
result of the merge. There are situations where I "cheat", e.g. with
frameworks/base being only published as a snapshot: I have access to
the "non-snapshot" version with the detailed history, so I can
actually do the merge against that and copy the result as if I had
done it directly with the snapshot. Git actually makes all those
operations surprisingly easy.

The one interesting aspect is to manage the changes in the manifests.
For that, I maintain clean clients with all of the primary branches of
both AOSP and Google on my local machine (it weighs in at quite a few
GB), and I run local diffs between them: making sure that the mirrors
are proper manifests, finding which projects exist e.g. in master but
not in donut, or in the internal tree but not the public one, making
sure that after a push the manifest in AOSP matches the manifest that
was staged, etc... I try to manage that part as independently from the
rest as I can, such that on one side I take care of the manifests
pretty much on their own, and on the other side I just use repo forall
commands from those manifests to do all the other operations.

Really, the one part that I need to ask the Android team to be careful
about is to not leak confidential information in commits. This is
something that every engineer needs to be mindful about on a routine
basis, and mistakes either require to re-write a history or to embargo
the detailed history of some projects until the confidentiality issues
clear. Of course, if you're not open-sourcing your private tree,
that's not a concern.

JBQ