Re: [android-building] Mirroring multiple branches of AOSP

[Jean-Baptiste Queru]

See "Using a local mirror" at https://source.android.com/source/downloading.html

TL;DR: repo init -u https://android.googlesource.com/a/mirror/manifest
--mirror -b master ; repo sync

JBQ

On Thu, Nov 22, 2012 at 12:12 AM, Loren M. Lang <pengu...@gmail.com> wrote:
> I am trying to create a local mirror of AOSP that I can use to check out
> multiple branches from. I first tried mirroring the master branch with
> this:
>>
>> repo init -u https://android.googlesource.com/a/platform/manifest --mirror
>> repo sync
>
>
> I assumed that that would mirror everything needed to checkout older
> branches as well as master itself, but, while all mirrored repositories have
> tags/branches for older releases, the manifest does not contain older git
> repositories no longer used in master. When I tried to checkout
> android-4.2_r1 from my mirror with these commands:
>
>> repo init -u ../mirror/platform/manifest.git -b android-4.2_r1
>>
>> repo sync
>
>
> I found that platform/dalvik was missing and platform/dalvik2 was in it's
> place. After taking a closer look at manifest.git, I discovered the
> master-dalvik branch which seems to be maintained as a master branch that
> still uses dalvik. I then started over and tried initializing my mirror
> with that branch instead:
>
>> repo init -u https://android.googlesource.com/a/platform/manifest --mirror
>> -b master-dalvik
>> repo sync
>
>
> But when I tried to checkout android-4.2_r1, I found different repos that
> were missing like device/lge/mako-kernel. What's the proper way of
> mirroring AOSP so that I can checkout multiple branches? Do I need to use
> repo init for each branch I want and just use the --references option to
> save space? Or can I just use repo sync with a copy of a manifest from each
> branch I want to sync?
>
> --
> You received this message because you are subscribed to the "Android
> Building" mailing list.
> To post to this group, send email to android-...@googlegroups.com
> To unsubscribe from this group, send email to
> android-buildi...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/android-building?hl=en



--
Jean-Baptiste M. "JBQ" Queru
Technical Lead, Android Open Source Project, Google.

Questions sent directly to me that have no reason for being private
will likely get ignored or forwarded to a public forum with no further
warning.

[Jean-Baptiste Queru]

Indeed, I believe that it's actually implied, even with --mirror

JBQ

On Thu, Jan 17, 2013 at 9:15 AM, Justin Buser <jus...@justinbuser.com> wrote:
> Isn't -b master implied? I'm having a similar problem switching branches
> using a local mirror, every branch I try results in revision blah in
> manifests not found...

[Antonio Marin Cerezuela]

Hi Loren,

From my understanding, when you sync, you get a working copy of the tag you select, and it may, and may not contain some projects.

The repo manifest itself is a git project, so if you log it, you can find that in commit I90c596116c5ae781d21d0d45b4245c50df8f055a they changed dalvik from platform/dalvik to platform/dalvik2.

-  <default revision="master"
+  <default revision="jb-mr1-dev"

-  <project path="dalvik" name="platform/dalvik2" />
+  <project path="dalvik" name="platform/dalvik" />

That means that every tag you sync to after that commit would have dalvik2 instead of dalvik.

If your intention is to have a mirror to checkout multiple branches, you are on the good way, but keep in mind that some branches differ on the projects involved.

BR,
Tony

El 22/11/12 09:12, Loren M. Lang escribió:

I am trying to create a local mirror of AOSP that I can use to check out multiple branches from.  I first tried mirroring the master branch with this:

repo init -u https://android.googlesource.com/a/platform/manifest --mirror
repo sync

I assumed that that would mirror everything needed to checkout older branches as well as master itself, but, while all mirrored repositories have tags/branches for older releases, the manifest does not contain older git repositories no longer used in master.  When I tried to checkout android-4.2_r1 from my mirror with these commands:

repo init -u ../mirror/platform/manifest.git -b android-4.2_r1

repo sync 

I found that platform/dalvik was missing and platform/dalvik2 was in it's place.  After taking a closer look at manifest.git, I discovered the master-dalvik branch which seems to be maintained as a master branch that still uses dalvik.  I then started over and tried initializing my mirror with that branch instead:

repo init -u https://android.googlesource.com/a/platform/manifest --mirror -b master-dalvik
repo sync

But when I tried to checkout android-4.2_r1, I found different repos that were missing like device/lge/mako-kernel.  What's the proper way of mirroring AOSP so that I can checkout multiple branches?  Do I need to use repo init for each branch I want and just use the --references option to save space?  Or can I just use repo sync with a copy of a manifest from each branch I want to sync?

--
You received this message because you are subscribed to the "Android Building" mailing list.
To post to this group, send email to android-...@googlegroups.com
To unsubscribe from this group, send email to
android-buildi...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/android-building?hl=en

-- 
Antonio Marin

Research Dept
Swiss Mobility Solutions
Av. Jaime I El Conquistador 1, Local 2
03560 El Campello
Spain

www.swissmobilitysolutions.com


Este mensaje contiene informacion reservada y confidencial destinada exclusivamente al destinatario. Si usted no es el destinatario no esta autorizado a copiar, reproducir o distribuir este mensaje ni su contenido. Si ha recibido este mensaje por error, le rogamos que nos lo notifique inmediatamente.

The information in this e-mail is confidential and may be legally privileged. It is intended solely for the addressee(s). It may contain privileged and confidential information. If you are not the intended recipient, please destroy this message and notify us immediately. Any disclosure, copying or distribution of this message is prohibited and may be unlawful.

[Jean-Baptiste Queru]

Note that mirror/manifest (as opposed to platform/manifest or
toolchain/manifest) contains all 485 AOSP projects. Its point is
explicitly to be used to mirror everything, it's not useful for
anything else. From there, you can initialize platform client,
toolchain clients, clone kernels, etc...

JBQ

[Jean-Baptiste Queru]

There's no jb-mr1-release manifest. Use android-4.2.1_r1 to get that
exact version.

JBQ

On Thu, Jan 17, 2013 at 12:34 PM, Justin Buser <jus...@justinbuser.com> wrote:
> Ok, so then please let me know if I'm missing something, I do:
>
> mkdir -p ~/android/aosp/mirror
> cd ~/android/aosp/mirror
> repo init -u https://android.googlesource.com/mirror/manifest --mirror
> repo sync
>
> then several hours later…
>
> mkdir -p ~/android/aosp/jb-mr1-release
> cd ~/android/aosp/jb-mr1-release
> repo init -u /home/justin/android/aosp/mirror/platform/manifest.git -b
> jb-mr1-release
>
> That last command then throws: "revision jb-mr1-release in manifests not
> found"
>
>
> Basically I have a Sprint Galaxy Nexus that I was hoping to install my own
> build on, just for the sake of going through the process and learning how it
> works. I'd ultimately like to be an AOSP contributor, but before doing so I
> feel that I need a more in depth understanding of the entire process.
>
>
>
> Incidentally:
>
> % cat < /proc/sys/kernel/ostype < /proc/sys/kernel/hostname <
> /proc/sys/kernel/osrelease < /proc/sys/kernel/version
> Linux
> foundry
> 3.6.11-1-ARCH
> #1 SMP PREEMPT Tue Dec 18 08:57:15 CET 2012
>
>
>
> I'm not sure if it makes a difference but as you can see I'm not running
> Ubuntu or any Debian variant.
>
>
> ( If you need any additional info, u name it. )

[Jean-Baptiste Queru]

I should add: if you're using a Sprint Galaxy Nexus (i.e. toroplus),
you have to use master, it's the only supported branch.

JBQ

[Jean-Baptiste Queru]

There is a jb-mr1-release branch in individual projects, but not for
the manifest. That's to work around a limitation in git/repo. I can go
into deeper detail if you'd like.

"Supported" means "I'll try to help you if something goes wrong". With
toroplus In 4.2, you might find that some things work, but if things
don't I'm not going to spend time investigating, so you're on your
own.

There is no branch today where you'll have a fully functional Galaxy
Nexus (regardless of models). To my knowledge there is no branch today
where a Sprint Galaxy Nexus will be able to access the cell network
(regardless of whether I call it "supported").

JBQ

On Thu, Jan 17, 2013 at 1:42 PM, Justin Buser <jus...@justinbuser.com> wrote:
> Tried that one already actually, built and flashed via fastboot with no
> problem. Is there anyway to get the cellular functionality working in 4.2
> yet? I tried syncing device/samsung/toroplus and followed the instructions
> for extracting the binaries you've mentioned and everything else worked (as
> I was informed would be the case). I'm not quite sure what you mean when you
> say that it's the only supported branch, and I'm not trying to be nit-picky,
> just understand here.
>
> What I would take that to mean is that the device will be fully functional
> with a master build, but non-functional if I built and installed the same
> version that is provided in the factory images here:
> https://developers.google.com/android/nexus/images.
>
> However I don't believe this to be the case, at least not as it pertains to
> using the master branch as I've already done that and it was exactly as the
> disclaimer in the building for devices info states (no cell network).
>
> To make it simple, all I really need to know is whether or not there is a
> branch I can build and flash that won't compromise the functionality of the
> device, and if so which one it is. I appreciate the assistance btw, I hope
> I'm not being a pest.

[Jean-Baptiste Queru]

We've been thinking about separating proprietary hardware-related
components files into their own partition (that's why we've been
trying to isolate them into /system/vendor - that could become the
separate /vendor partition). We actually started to move in that
direction for Nexus S. Nexus 4 and Nexus 10 use the build system to
actually enforce the separation.

Unfortunately, using a separate partition means that we would need to
reserve more space for the system (since we would have to reserve
enough room to make sure that neither partition fills up). And we'd
need to adapt all the tools accordingly: the build system, fastboot,
the boot sequence, all the OTA and recovery mechanisms. It's quite a
massive task.

On top of that, that wouldn't help for anything that uses apks (as
opposed to native code), because the signature of the apks wouldn't
match that of the system. This tends to be an issue with all CDMA and
LTE devices. At a glance from the log, though, that might not be the
issue (yet) with the Sprint Galaxy Nexus, I suspect that the telephony
apks might be using private Samsung APIs that don't exist in Google's
tree (and therefore in AOSP).

JBQ

On Thu, Jan 17, 2013 at 3:09 PM, Justin Buser <jus...@justinbuser.com> wrote:

>
>
> On Thursday, January 17, 2013 5:12:20 PM UTC-5, Jean-Baptiste Queru wrote:
>>
>> There is a jb-mr1-release branch in individual projects, but not for
>> the manifest. That's to work around a limitation in git/repo. I can go
>> into deeper detail if you'd like.
>>
>

> No need, that makes sense, I'm starting to get a better grasp on the way
> things are structured.

>
>> "Supported" means "I'll try to help you if something goes wrong". With
>> toroplus In 4.2, you might find that some things work, but if things
>> don't I'm not going to spend time investigating, so you're on your
>> own.
>>
>> There is no branch today where you'll have a fully functional Galaxy
>> Nexus (regardless of models). To my knowledge there is no branch today
>> where a Sprint Galaxy Nexus will be able to access the cell network
>> (regardless of whether I call it "supported").
>>
>

> Got it, so supported basically refers to what current collaborative efforts
> are focused on and what I should be concentrating on as well if I want to
> make any sort of contribution. That is my ultimate goal, but you'll have to
> bear with me as I get up to speed. Are there any methods I might employ to
> work around the driver issue? For instance being as the factory images are
> separated by partition would it be possible to only build the
> non-proprietary components and combine the two in such a way that I could
> experiment a bit without losing functionality? I know that for instance
> Cyanogen mods employ some sort of work around, but I'm guessing it has
> something to do with the way they are installed. I'd prefer to stick with
> fastboot and adb though, so if you wouldn't mind and have some time in the
> next day or two and could go into a bit more depth re: the driver issue I
> would sincerely appreciate it. Especially if there is something I can do
> within the boundaries of whatever legal constraints are in place to help
> augment the current usability of the Galaxy Nexus as a test device.
>
> BTW, did you catch my uname joke earlier? No on seemed to notice it despite
> my ridiculous concatenation alternative...

[Jean-Baptiste Queru]

The idea is that the partition wouldn't be overwritten during an AOSP
build, so that there wouldn't be any need to distribute it or any of
the files it contains for the purpose of AOSP.

Using a partition image wouldn't work, because it would get deleted by
fastboot and therefore there would need to be a mechanism to re-create
it as part of an AOSP build.

There's no unique lists of files, that varies between devices, between
variants of devices, and between android versions.

JBQ

On Sun, Jan 20, 2013 at 9:06 PM, Justin Buser <jus...@justinbuser.com> wrote:
> Interesting, so if I understand correctly the driver partition would be a
> static image that would remain untouched during the build process to ensure
> that the individual file signatures within would retain their integrity? Or
> would the point be that the partition in question would be installed by the
> manufacturer and already reside on the device, eliminating the need to
> include it in the source?
>
> Couldn't the same result be achieved by mounting a partition image on a loop
> device at whatever location the system would expect to find the binaries?
> Failing that have you considered or looked into using lvm or some other
> virtual partition utility? That's all pretty light weight and seems like it
> would require a lot less work, on top of eliminating the possibility of
> running out of space on a partition. Also, it would seem to me that if the
> vendor apk signatures matched the ones on the vendors drivers then they
> should be considered safe, assuming that security is the issue at hand.
>
> Is there any chance you could let me know specifically what files are
> involved re: the cell problem, and / or any other relevant info you may
> have? I'd love to have a look, for my own edification at the very least,
> even if you just pointed me to specific log entries etc... I just feel
> better when I fully understand the problem I'm facing, even if I'm unable to
> solve it at the moment. That way I can at least can stew on it over time
> while I do other things :)

[Jean-Baptiste Queru]

If we put an image for /vendor into /system, it'll get deleted by
fastboot. The whole point is very explicitly to be in a separate
fastboot partition (the fact that it's a separate filesystem is not a
goal).

JBQ

On Wed, Jan 23, 2013 at 10:36 AM, Justin Buser <jus...@justinbuser.com> wrote:
> Well the idea would be that the image itself would be pre-signed and
> included as an image in lieu of the files in /vendor. Adding it as a single
> image file to the system partition image for example, and then mounting it
> to whichever directory the files should be found in on /dev/loop0. Doing
> this in the boot loader after mounting the system partition, if done
> correctly, should result in the files contained within being compatible with
> whatever process the system uses to verify their integrity. This would of
> course make the varying nature of what files are required less of an issue,
> and separate images could be created based on the target Android version.
>
> I obviously could be missing something however, is there a specific reason
> why this may not be possible?

> --

> --
> You received this message because you are subscribed to the "Android
> Building" mailing list.
> To post to this group, send email to android-...@googlegroups.com
> To unsubscribe from this group, send email to
> android-buildi...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/android-building?hl=en
>

> ---
> You received this message because you are subscribed to the Google Groups
> "Android Building" group.

> To unsubscribe from this group, send email to

> android-buildi...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.

[Jean-Baptiste Queru]

There's currently no device support in the head manifests (master,
4.2.2, jb-mr1.1-dev)

JBQ

On Thu, Feb 21, 2013 at 7:46 AM, Justin Buser <jus...@justinbuser.com> wrote:
> So I was able to figure out how to get the other projects to at least show
> up as being available by specifying a different manifest i.e.:
>
> repo init -u /home/justin/android/aosp/mirror/platform/manifest.git -m
> base-for-4.2-gpl.xml
>
> Is there any reason I shouldn't be doing this or maybe a better/more
> appropriate method?

>
> --
> --
> You received this message because you are subscribed to the "Android
> Building" mailing list.
> To post to this group, send email to android-...@googlegroups.com
> To unsubscribe from this group, send email to
> android-buildi...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/android-building?hl=en
>
> ---
> You received this message because you are subscribed to the Google Groups
> "Android Building" group.

> To unsubscribe from this group and stop receiving emails from it, send an

[Magnus Bäck]

On Saturday, February 23, 2013 at 07:28 EST,
Justin Buser <jus...@justinbuser.com> wrote:

> Gotcha, so I figure just cloning the devices with git is the best way
> to do this.

Why not use a local manifest and let Repo do the cloning and updating?

> I wrote a shell script to automate doing so, has worked well for me so
> I'll share it for anyone else who's interested:
>
> #!/bin/sh
> MIRROR=$1
> MANIFEST=$(echo "$MIRROR/.repo/manifests/default.xml" | sed "s://:/:")
> if [ -f $MANIFEST ] ; then
> PROJECTS=$(grep 'name="device/' $MANIFEST | sed -E "s:^(\ *)<project(.+?name=\"):: ; s:\".*::")

It would be more robust to use XPath for the extraction.

xpath -q -e "//project/@name" $MIRROR/.repo/manifests/default.xml | \
sed -n 's#^ name="\(platform/hardware/.*\)"$#\1#p'

[...]

--
Magnus Bäck
ba...@google.com

[Magnus Bäck]

On Tuesday, February 26, 2013 at 00:57 EST,
Justin Buser <jus...@justinbuser.com> wrote:

> Thanks for the tip, I would much prefer to work within the repo framework
> if possible but thought that changing manifests/locations would cause
> issues... Is there a specific manifest that you would suggest duplicating
> and modifying, or a specific location that would be best suited to place
> it? Bear in mind that I do have a local mirror, if that makes any
> difference.

A local manifest is a separate manifest file that isn't under version
control that can be used to "patch" the original manifest. See details
at the end of "repo help manifest". A better place to discuss Repo is
the repo-discuss mailing list.

> I've never used XPath before, and it doesn't seem to be part of the
> standard Arch distribution, what would the typical premise for using
> it be and what does it do exactly?

XPath is a query language for XML documents, and the xpath tool is a
simple command-line interface to it. In Ubuntu it's included in the
libxml-xpath-perl package. If you want to generate a local manifest
file, XSLT is an excellent vehicle. The stylesheet below copies all the
project elements for all device/* and platform/hardware/* gits to a new
manifest file. Feed it to e.g. xsltproc.

<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="1.0">
<xsl:output method="xml" indent="yes"/>

<xsl:template match="/manifest">
<manifest>
<xsl:apply-templates select="project[starts-with(@name, 'device/') or starts-with(@name, 'platform/hardware/')]"/>
</manifest>
</xsl:template>

<xsl:template match="project">
<project>
<xsl:copy-of select="@*"/>
</project>
</xsl:template>

</xsl:stylesheet>

--
Magnus Bäck
ba...@google.com

[Magnus Bäck]

On Friday, March 22, 2013 at 08:45 EDT,
Justin Buser <jus...@justinbuser.com> wrote:

[...]

> So my question is, (and I apologize if this would be more
> appropriately covered elsewhere but I'm not sure if this is a
> mirroring/building/repo/etc... issue ) has something changed remotely
> that is causing the device projects to be included in the default
> manifest now or am I missing something locally?
>
> I have checked default.xml after pulling it down to a temp dir using git
> clone https://android.googlesource.com/platform/manifest and there are
> definitely device/hardware entries in there. Unfortunately I don't have any
> old copies of the manifest from when I initially attempted all of this so
> I'm unsure as to whether or not there is some component to the process that
> should be excluding the device projects that I have altered along the way
> and need to fix, or if that's just how it is now and I don't need to worry
> about it.

You don't need any old copies, you have the full Git history of every
thing that's been going on. Sometimes a quick glance at "git log" will
spot the change you're looking for, other times "git blame" will help.
Running

git blame default.xml | grep device

will quickly show that commit 70c902cf added a bunch of lines for
device/* repositories, and "git show 70c902cf" will show you exactly
what happened (but not necessarily when -- the date in the commit header
only indicates when the original commit was created, not when it was
pushed for the world to see).

--
Magnus Bäck
ba...@google.com

[Magnus Bäck]

On Friday, April 19, 2013 at 02:16 EDT,
Jean-Marc GRIMAUD <jmgr...@yahoo.com> wrote:

> From your message, I understand that all the branches are contained in
> this mirroring.
> On my side, after having seting up it, I tried:
> repo init -u /work/android/mirror/android/platform/manifest.git
> it work fine
>
> but if I try
> repo init -u /work/android/mirror/android/platform/manifest.git -b
> android-4.0.4_r2.1
> I got errors, like
>
> Fetching projects: 6% (14/230) fatal:
> '/work/android/mirror/android/device/samsung/crespo' does not appear
> to be a git repository
> fatal: Could not read from remote repository.
>
> Please make sure you have the correct access rights
> and the repository exists.
>
> So I have the feeling that only the master branch has been mirrored,
> but not all the branches.
>
> How can I make a mirror and ensure that I can get all the branches
> after that, for 4.0, 4.1 and 4.2 ?

default.xml from mirror/manifest does list device/samsung/crespo,
so it should've been downloaded just fine. Is there, in fact, no
/work/android/mirror/android/device/samsung/crespo on the machine?
What if you explicitly try to sync that git by running

repo sync device/samsung/crespo

in the mirror workspace?

--
Magnus Bäck
ba...@google.com