An experiment: installing AOSP via recovery

[Jean-Baptiste Queru]

I've been investigating the possibility of installing AOSP via recovery instead of fastboot. The major benefit is that recovery can work at the granularity of one file instead of one partition, so that allows leaving certain files in place while replacing others.

The instructions are a fair bit complex. There might be opportunities to simplify and automate most of this in the future.

This only works on GSM Galaxy Nexus ("maguro") originally running a true yakju builds. This is not supported on yakju variants (e.g. yakjuxw or yakjuux), on CDMA Galaxy Nexus ("toro") or on any other devices.

I'd like to hear some feedback from people who attempt to follow those instructions.

JBQ

# Create a directory to store all the temporary files
mkdir -p ~/aosp-ota-exp

# Download all the IML74K maguro binaries from https://code.google.com/android/nexus/drivers.html into ~/aosp-ota-exp
# download the ICL53F yakju factory image from https://code.google.com/android/nexus/images.html into ~/aosp-ota-exp
# download the matching stub target_files.zip directly from https://dl.google.com/dl/android/aosp/stub-yakju-target_files-icl53f.zip into ~/aosp-ota-exp

# Extract the proprietary binaries
for i in ~/aosp-ota-exp/*maguro-iml74k* ; do tar zxvf $i ; done
for i in ./extract-*-maguro.sh ; do $i ; done

# Extract the individual factory images
(cd ~/aosp-ota-exp ; tar zxvf yakju-icl53f-factory-89fccaac.tgz)
(cd ~/aosp-ota-exp/yakju-icl53f ; unzip image-yakju-icl53f.zip)

# Patch the OTA-packaging tool
repo forall build -c 'git pull https://android.googlesource.com/platform/build refs/changes/64/31464/1'

# Set up the build. Insert dummy files where the original files should be preserved
source build/envsetup.sh
lunch full_maguro-userdebug
make installclean
for i in vendor/firmware/bcm4330.hcd vendor/etc/sirfgps.conf vendor/lib/hw/gps.omap4.so vendor/lib/libinvensense_mpl.so vendor/firmware/libpn544_fw.so vendor/firmware/ducati-m3.bin ; do mkdir -p out/target/product/maguro/system/$(dirname $i) ; echo "DUMMY AOSP FILE" > out/target/product/maguro/system/$i ; done

# Do the build (this is a dist build, not a plain build)
time nice -n19 make -j32 dist

# Create the OTA package and the custom cache partition
rm -rf ~/aosp-ota-exp/cache
mkdir -p ~/aosp-ota-exp/cache
build/tools/releasetools/ota_from_target_files -w -i ~/aosp-ota-exp/stub-yakju-target_files-icl53f.zip -k build/target/product/security/testkey out/dist/full_maguro-target_files-eng.*.zip ~/aosp-ota-exp/cache/aosp_update.zip
make_ext4fs -s -l 209715200 -a cache ~/aosp-ota-exp/cache.img ~/aosp-ota-exp/cache

# Flash the device
fastboot flash bootloader ~/aosp-ota-exp/yakju-icl53f/bootloader-maguro-primekk15.img
fastboot reboot-bootloader
sleep 5
fastboot flash radio ~/aosp-ota-exp/yakju-icl53f/radio-maguro-i9250xxkk6.img
fastboot reboot-bootloader
sleep 5
fastboot flash system ~/aosp-ota-exp/yakju-icl53f/system.img
fastboot flash boot
fastboot flash recovery
fastboot flash cache ~/aosp-ota-exp/cache.img

# Boot into recovery (in the bootloader, navigate with volume up/down, and select with the power button)
# Get the recovery menu (hold power, press volume up)
# In recovery, apply /cache/aosp_update.zip, wipe the cache, and reboot.

--
Jean-Baptiste M. "JBQ" Queru
Software Engineer, Android Open-Source Project, Google.

Questions sent directly to me that have no reason for being private will likely get ignored or forwarded to a public forum with no further warning.

[Jean-Baptiste Queru]

I'd like to hear some feedback about this. I know that there've been
quite some questions about getting support for the camera, for GPS,
for Bluetooth, and I believe that this approach allows all of those to
work, but without anyone testing it I can't be quite sure.

JBQ

[Pau Oliva Fora]

Hi,

was about to test this, but got an error thrown right after make:

build/core/base_rules.mk:78: *** Module name: uix
build/core/base_rules.mk:79: *** Makefile location:
sdk/layoutopt/libs/uix/src
build/core/base_rules.mk:80: *
build/core/base_rules.mk:81: * Each module must use a LOCAL_MODULE_TAGS
in its
build/core/base_rules.mk:82: * Android.mk. Possible tags declared by a
module:

[...]

If I add the 'LOCAL_MODULE_TAGS := optional' into the Android.mk for
this module, then it fails on the next:

build/core/base_rules.mk:78: *** Module name: layoutopt
build/core/base_rules.mk:79: *** Makefile location: sdk/layoutopt/app/etc

[...]

build/core/base_rules.mk:78: *** Module name: layoutopt
build/core/base_rules.mk:79: *** Makefile location: sdk/layoutopt/app/src

And so on.... so I guess I am doing something wrong. Any hint would be
appreciated.

Cheers,

Pau Oliva

[Pau Oliva Fora]

never mind, it started compiling after I edited just the 3 makefiles in
sdk/layoutopt. However now it's stopping here:

make: *** No rule to make target
`prebuilts/qemu-kernel/arm/LINUX_KERNEL_COPYING', needed by
`out/target/product/maguro/obj/NOTICE_FILES/src/kernel.txt'. Stop.

[Jean-Baptiste Queru]

I'm trying to reproduce those failures, but I'm not seeing any of them
at the moment. Those do ring a bell, from many weeks ago, but I really
thought we had fixed that issue.

JBQ

> --
> You received this message because you are subscribed to the "Android
> Building" mailing list.
> To post to this group, send email to android-...@googlegroups.com
> To unsubscribe from this group, send email to
> android-buildi...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/android-building?hl=en

[Pau Oliva Fora]

maybe the problem is caused because I did repo init specifying branch -b
android-4.0.3_r1 and it should be master ?

[Jean-Baptiste Queru]

Yes, that'd be likely to cause that kind of problem. The git pull
command would bring in many changes from master, which could cause the
breakages you've been seeing.

JBQ

[JF Dionne]

I get the following error when running

build/tools/releasetools/ota_from_target_files -w -i ~/aosp-ota-exp/stub-yakju-target_files-icl53f.zip -k build/target/product/security/testkey out/dist/full_maguro-target_files-eng.*.zip ~/aosp-ota-exp/cache/aosp_update.zip
make_ext4fs -s -l 209715200 -a cache ~/aosp-ota-exp/cache.img ~/aosp-ota-exp/cache

ERROR: failed to unzip input target-files "/home/jfdionne/aosp-ota-exp/stub-yakju-target_files-icl53f.zip"

[JF Dionne]

Forgot some lines

build/tools/releasetools/ota_from_target_files -w -i ~/aosp-ota-exp/stub-yakju-target_files-icl53f.zip -k build/target/product/security/testkey out/dist/full_maguro-target_files-eng.*.zip ~/aosp-ota-exp/cache/aosp_update.zip

unzipping target target-files...
using device-specific extensions in device/samsung/tuna
unzipping source target-files...
unzip: cannot find or open /home/jfdionne/aosp-ota-exp/stub-yakju-target_files-icl53f.zip, /home/jfdionne/aosp-ota-exp/stub-yakju-target_files-icl53f.zip.zip or /home/jfdionne/aosp-ota-exp/stub-yakju-target_files-icl53f.zip.ZIP.

[Pau Oliva Fora]

I've completed all the steps, however the screen is not working after I
flash the aosp_update.zip, it shows some weird lines as in this picture:

http://pof.eslack.org/tmp/gnex.jpg

just to make sure there is no mistake on my side, this is the content of
the generated aosp_update.zip file:

http://pof.eslack.org/tmp/aosp_update_contents.txt

Let me know if there's anything else I can try.

Cheers,

Pau Oliva

[Jean-Baptiste Queru]

Could it be that you don't have that file in there? You need to
download it directly from
https://dl.google.com/dl/android/aosp/stub-yakju-target_files-icl53f.zip

JBQ

[JF Dionne]

weird lines are appearing when your binaries are not installed properly.

try to rerun these

[Jean-Baptiste Queru]

This is the symptom of missing proprietary graphics libraries: the
boot animation code falls back to the kernel graphics driver, and that
driver is buggy so the boot animation is displayed slanted. If you
look further in the log, the system server will repeatedly fail to
start because it can't find the OpenGL libraries.

My guess is that you forgot to extract the proprietary binaries from
imgtec (and maybe samsung) before doing your build, and the scripts I
posted don't re-use those binaries from the existing system (they
wouldn't work anyway, the 4.0.2 graphics libraries don't work on the
4.0.3 kernel).

JBQ

[JF Dionne]

yeah found out by reading all the steps again...

now I'm stuck at make_ext4fs: command not found

Not sure where to get that command

[Jean-Baptiste Queru]

That is strange.

source build/envsetup.sh
# ...
lunch full_maguro-userdebug
# ...
make make_ext4fs
# ...
which make_ext4fs
# .../out/host/linux-x86/bin/make_ext4fs

JBQ

[Pau Oliva Fora]

you where right, forgot to put the proprietary libraries in place (had
extracted them in the wrong folder!).

Now that those files are in place, I flashed again and it boots nicely
on full AOSP... camera working perfect after following your steps, thank
you :)

http://pof.eslack.org/archives/images/maguro-full-aosp.jpg

[Jean-Baptiste Queru]

Superb, thank you very much for working your way through all the
steps, and thanks to everyone else who also had a look.

This gives me 2 conclusions:

-It's possible to make this work.

-It really needs to be automated a lot more before it can be unleashed
on the general public, since even with a lot of care it's easy to miss
a step or two.

JBQ

[Pau Oliva Fora]

A good point of this method is that using the 'DUMMY AOSP FILE' trick,
one can preserve also Google apps (market, gmail, maps...) from the
original factory image, and if we do not use '-w' parameter when
generating the OTA package in ota_from_target_files, the new OTA can be
applied without wiping the device.

Do you see something that might not work by using this method? If there
are no obvious stoppers I'll give it a try later today.

Cheers,

Pau Oliva

[Jean-Baptiste Queru]

First, the issue of wiping data: you're unlikely to escape it. The
signatures for the system apps won't match between a retail build and
an AOSP build, so there'll be trouble. If you're lucky, you'll have
issues right away. If you're less lucky, you'll have issues later at
the point where wiping is painful and not wiping is painful. Note also
that the package names don't all match (e.g. I know that the Email app
has a different name in retail builds and in AOSP builds).

As for preserving apks, here are some issues that come to mind:

-If any apk needs to be signed with the platform key, you'll be out of
luck, since that's not going to match the key you use for the AOSP
build.

-You'll need to preserve the odex files. Since those aren't fully
portable, that limits the types of changes you can do to dalvik.

-You'll probably need to trim the AOSP build a bit. E.g. Provision
isn't used in retail builds as there's a full-featured equivalent.

-You'll have to make choices for some of the apps: e.g. the AOSP
LatinIME doesn't have all the features of the retail one, so you could
end up deciding to keep the retail one even though there's an
Open-Source variant.

Finally, if any of those apps rely on proprietary overlays that'd be
applied to e.g. the framework, you could have mismatches with unknown
consequences. I don't expect that'd be the case, but you never know.

JBQ

[Magnus Bäck]

On Thursday, February 02, 2012 at 21:42 CET,
Jean-Baptiste Queru <j...@android.com> wrote:

[...]

> As for preserving apks, here are some issues that come to mind:
>
> -If any apk needs to be signed with the platform key, you'll be out
> of luck, since that's not going to match the key you use for the AOSP
> build.
>
> -You'll need to preserve the odex files. Since those aren't fully
> portable, that limits the types of changes you can do to dalvik.

Will this work in practice? Doesn't an odex file require that all
jar files it uses are identical to what they were in the environment in
which the odex file was created? In other words, touch framework.jar and
all odex files will be void.

[...]

--
Magnus B�ck
ba...@swipnet.se

[Jean-Baptiste Queru]

I got confirmation that odex optimizations follow dependencies.

For practical purposes, that means that we can't replace the framework
(or any other libraries) and continue using apks that had been
pre-dex-opt'ed on a different build.

JBQ

> Magnus Bäck
> ba...@swipnet.se

[Leon]

Hi. I followed your instructions and got an error message with the
command

repo forall build -c 'git pull https://android.googlesource.com/platform/build
refs/changes/64/31464/1'

So I can't go to the next step.
Error message is below.

error: project build not found

Is there something wrong with repo command?

[Conley Owens]

On Wed, Mar 7, 2012 at 7:44 PM, Leon <leo...@gmail.com> wrote:
> Hi. I followed your instructions and got an error message with the
> command
>
> repo forall build -c 'git pull https://android.googlesource.com/platform/build
> refs/changes/64/31464/1'
>
> So I can't go to the next step.
> Error message is below.
>
> error: project build not found

Hi Leon,

Are you in the root of the android sources? Have you synced? Does
the build directory exist?

FYI, this command is more or less the same as doing:
cd build

git pull https://android.googlesource.com/platform/build refs/changes/64/31464/1

and it sounds like it's having trouble with the "cd build" part.

~cco3