Building the kernel on the fly in Android build system

[Leo Barnes]

Hello!

Is there any standardized way of getting the Android build system to
compile the kernel on the fly while building the rest of the system? I
don't believe that the phone manufacturers compile the kernel
separately since that is a setup that is bound to create mistakes. I
assume that I would need a Android.mk file in the kernel dir, and that
it should be placed/linked into the android build tree. Any
suggestions?

Best regards,
//Leo

[Jean-Baptiste Queru]

I don't think that we've ever done it. Re-writing the kernel build
system with Android.mk would be a huge task and a significant
maintenance time-sink.

JBQ

> --
> You received this message because you are subscribed to the "Android Building" mailing list.
> To post to this group, send email to android-...@googlegroups.com
> To unsubscribe from this group, send email to
> android-buildi...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/android-building?hl=en
>

--
Jean-Baptiste M. "JBQ" Queru
Software Engineer, Android Open-Source Project, Google.

Questions sent directly to me that have no reason for being private
will likely get ignored or forwarded to a public forum with no further
warning.

[Patrick Georgi]

android-x86.org has a build/core/kernel.mk that takes care of that. It doesn't rewrite everything in Android.mk but uses a sub-make, but that's a trade-off that some take, some not.

The main advantage is that kernel modules can be built right before they end up in the system images, no need to first build a kernel, and pick up binary files explicitely, and so I use it for my AOSP system.

[Jean-Baptiste Queru]

The very large drawback when doing that is that it doesn't manage
parallel builds properly. With modern machines capable of running 24
tasks simultaneously (2x 6-core hyperthreaded), having to fall back to
no-parallel builds is a very damaging tradeoff.

JBQ

[Al Sutton]

During the viewpad 10 work I did on android-x86 I had some exposure to this and the main reason for the in-build kernel compile seems to be because most of the relied upon "drivers" are kernel modules from the kernel build. With most 'phones the main issue I'm aware of is that some component manufacturers don't feed their code into the mainline kernel and have binary only drivers tied to a specific kernel release, and so binary pre-builts are the only way to guarantee a working system.

In all honesty I'd happily switch to a pre-built system from the android-x86 kernel on the fly system if I could guarantee all of the components would work. Don't get me wrong, tweaking the kernel is very useful, but most users would rather see things like sound through the speakers (as opposed to be limited to just the headphones which I'm currently limited to on the vp10 build) than know that I could compile a kernel from scratch.

Al.

[Magnus Bäck]

On Monday, January 24, 2011 at 11:23 CET,
Leo Barnes <barne...@gmail.com> wrote:

> Is there any standardized way of getting the Android build system to
> compile the kernel on the fly while building the rest of the system?
> I don't believe that the phone manufacturers compile the kernel
> separately since that is a setup that is bound to create mistakes.

I can't speak for other OEMs, but we build the kernel together with
the platform in order to ease dependency and compatibility issues and
generally avoid the hassle of storing prebuilt binaries. Those who wish
so can of course build just the kernel to decrease turnaround time for
changes.

> I assume that I would need a Android.mk file in the kernel dir, and
> that it should be placed/linked into the android build tree. Any
> suggestions?

You're on the right track. Assuming what we're doing is right ;-)

You'll indeed need an Android.mk in the kernel directory. Define a
suitable rule to recursively invoke make for the kernel, and tack the
result onto the goal for prebuilt files. Very incomplete example:

TARGET_PREBUILT_KERNEL := <suitable path>/zImage

$(TARGET_PREBUILT_KERNEL):
$(hide) $(MAKE) ARCH=... CROSS_COMPILE=...

ALL_PREBUILT += $(TARGET_PREBUILT_KERNEL)

I'm a complete idiot when it comes to kernel builds, but you'd probably
want to place the kernel objects in a separate tree, build and install
modules, generate the .config file based on the defconfig etc. Also, the
minimal example above doesn't have any proper dependencies set up, so
once $(TARGET_PREBUILT_KERNEL) has been built it won't ever be remade.
Either set up reasonable dependencies or force the rule to execute every
time.

--
Magnus B�ck Opinions are my own and do not necessarily
SW Configuration Manager represent the ones of my employer, etc.
Sony Ericsson

[Leo Barnes]

Exactly what I was looking for, and for exactly the same reason. How
do you get the main Android make to call the build/core/kernel.mk?

[Magnus Bäck]

On Tuesday, January 25, 2011 at 09:30 CET,
Leo Barnes <barne...@gmail.com> wrote:

> On 24 Jan, 22:33, Patrick Georgi <patr...@georgi-clan.de> wrote:
>
> > android-x86.org has a build/core/kernel.mk that takes care of that.
> > It doesn't rewrite everything in Android.mk but uses a sub-make, but
> > that's a trade-off that some take, some not.
> >
> > The main advantage is that kernel modules can be built right before
> > they end up in the system images, no need to first build a kernel,
> > and pick up binary files explicitely, and so I use it for my AOSP
> > system.
>
> Exactly what I was looking for, and for exactly the same reason. How
> do you get the main Android make to call the build/core/kernel.mk?

See the commit below. I'm not sure its change to core/Makefile makes
sense though, unless the system image actually depends on the kernel
image. If you don't want to patch platform/build.git my suggestion from
earlier today works too, or you can add a dependency to the added kernel
build goal to the default goal (i.e. the goal being run with a plain
"make" command):

$(DEFAULT_GOAL): your-kernel-build-goal

http://git.android-x86.org/?p=platform/build.git;a=commit;h=51352905ba7da1b94f1c4a6f87b151fb0ab2b387

[Patrick Georgi]

gnu make shares parallel job counters between invocations (and the Android build system only works with gnu make anyway). It's not perfect, but it works reasonably well, esp. when it's just one sub-make to call.

There are other good reasons to avoid recursive make, but I this is one of the places where bending the rules helps (if you want to build the kernel in place).

[Bruce Beare]

We do this, it's not terribly difficult.
Here's what I do for the IA virtualbox kernel:
project build:

From 84f888373bf9120a85f818b7e9c624b20b9cac16 Mon Sep 17 00:00:00 2001
From: Bruce Beare <bruce....@intel.com>
Date: Sun, 2 Jan 2011 19:22:02 -0800
Subject: [PATCH] generic_x86: enable kernel source builds

Change-Id: Ide85b47552eb7cfe0273ca5a4d1537152eaeb78a
Signed-off-by: $ <bruce....@intel.com>
---
target/board/generic_x86/AndroidBoard.mk | 21 +++++++++++++++++----
target/board/generic_x86/BoardConfig.mk | 1 +
2 files changed, 18 insertions(+), 4 deletions(-)

diff --git a/target/board/generic_x86/AndroidBoard.mk b/target/board/
generic_x86/AndroidBoard.mk
index 01ca1b7..8a8ae83 100644
--- a/target/board/generic_x86/AndroidBoard.mk
+++ b/target/board/generic_x86/AndroidBoard.mk
@@ -1,11 +1,24 @@
LOCAL_PATH := $(call my-dir)

-ifeq ($(TARGET_PREBUILT_KERNEL),)
-LOCAL_KERNEL := prebuilt/android-x86/kernel/kernel
+
+KERNEL_VERSION ?= 2.6
+KERNEL_DIRECTORY ?= hardware/intel/linux-$(KERNEL_VERSION)
+-include $(TOP)/$(KERNEL_DIRECTORY)/AndroidKernel.mk
+
+.PHONY: build_kernel
+ifeq ($(TARGET_KERNEL_SOURCE_IS_PRESENT),true)
+build_kernel: get_kernel_from_source
else
-LOCAL_KERNEL := $(TARGET_PREBUILT_KERNEL)
+build_kernel: get_kernel_from_tarball
endif

PRODUCT_COPY_FILES += \
- $(LOCAL_KERNEL):kernel \
+ $(PRODUCT_OUT)/bzImage:kernel \
$(LOCAL_PATH)/init.rc:root/init.rc
+
+.PHONY: get_kernel_from_tarball
+get_kernel_from_tarball:
+ tar -xv -C $(PRODUCT_OUT) -f $(TARGET_KERNEL_TARBALL)
+
+.PHONY: installer_vdi
+installer_vdi: build_kernel
diff --git a/target/board/generic_x86/BoardConfig.mk b/target/board/
generic_x86/BoardConfig.mk
index 2268d41..f99d35e 100644
--- a/target/board/generic_x86/BoardConfig.mk
+++ b/target/board/generic_x86/BoardConfig.mk
@@ -1,4 +1,5 @@
TARGET_ARCH=x86
+CUSTOM_BOARD := vbox
DISABLE_DEXPREOPT := true
TARGET_COMPRESS_MODULE_SYMBOLS := false
TARGET_PRELINK_MODULE := false


Then, in your kernel directory:
[bjbeare@bjbeare-svr linux-2.6]$ cat AndroidKernel.mk
# This makefile is included from vendor/intel/*/AndroidBoard.mk.
# If the kernel source is present, AndroidBoard.mk will perform a
kernel build
# otherise, AndroidBoard.mk will find the kernel binaries in a
tarball.

# This can be overridden on the make command line.
TARGET_KERNEL_SOURCE_IS_PRESENT ?= true

.PHONY: get_kernel_from_source
get_kernel_from_source:
TARGET_TOOLS_PREFIX="$(ANDROID_BUILD_TOP)/$(TARGET_TOOLS_PREFIX)"
vendor/intel/support/kernel-build-hack.sh -c $(CUSTOM_BOARD)

# This rule is useful for creating a kernel that will be
# shared with a tree that does not have kernel source.
make_kernel_tarball: get_kernel_from_source bootimage
@echo Building kernel tarball: $(TARGET_KERNEL_TARBALL)
@rm -rf $(PRODUCT_OUT)/kerneltarball
@mkdir -p $(PRODUCT_OUT)/kerneltarball/system/lib/modules
@cp $(PRODUCT_OUT)/system/lib/modules/* $(PRODUCT_OUT)/kerneltarball/
system/lib/modules
@cp $(PRODUCT_OUT)/bzImage $(PRODUCT_OUT)/kerneltarball/
tar cvzf $(TARGET_KERNEL_TARBALL) -C $(PRODUCT_OUT)/kerneltarball
bzImage system

$(PRODUCT_OUT)/bzImage: get_kernel_from_source

[Chih-Wei]

Hi,
I'm the author of build/core/kernel.mk of android-x86.
Thanks to Patrick for introducing our efforts
and JBQ's comments. Glad to see such discussions.

For those who is interesting in the
"build kernel image on-fly" feature,
I would like to invite you to review it:

http://git.android-x86.org/?p=platform/build.git;a=blob;f=core/kernel.mk;h=0b34a70c33828754cb8b9a2299da8b8754159147;hb=6efbb187c186024021d4ef701bb9bf31edbd770f

To use it, simply put kernel.mk to build/core/
and add a small patch to build/core/Makefile
(see below)
It works for x86 and arm (and other arch) kernel.

It's not perfect and may violate Android build system
policy. But it works as what we need.
Comments and patches are welcome.

JBQ, I don't see any parallel build issues in my kernel.mk.
The kernel image and modules can be paralleled built,
as well as other targets -- make handles that very well.
Let me know if you think otherwise.

ps. the patch of build/core/Makefile for gingerbread:

diff --git a/core/Makefile b/core/Makefile
index a74f855..fb2ad3a 100644
--- a/core/Makefile
+++ b/core/Makefile
@@ -362,6 +362,8 @@ $(INSTALLED_BOOTIMAGE_TARGET): $(MKBOOTIMG) $
(INTERNAL_BOOTIMAGE_FILES)
$(hide) $(call assert-max-image-size,$@,$
(BOARD_BOOTIMAGE_PARTITION_SIZE),raw)
endif # TARGET_BOOTIMAGE_USE_EXT2

+include $(BUILD_SYSTEM)/kernel.mk
+
else # TARGET_NO_KERNEL
# HACK: The top-level targets depend on the bootimage. Not all
targets
# can produce a bootimage, though, and emulator targets need the
ramdisk
@@ -725,7 +727,7 @@ define build-systemimage-target
endef
endif # INTERNAL_USERIMAGES_USE_EXT

-$(BUILT_SYSTEMIMAGE): $(INTERNAL_SYSTEMIMAGE_FILES) $
(INTERNAL_USERIMAGES_DEPS)
+$(BUILT_SYSTEMIMAGE): $(INSTALLED_KERNEL_TARGET) $
(INTERNAL_SYSTEMIMAGE_FILES) $(INTERNAL_USERIMAGES_DEPS)
$(call build-systemimage-target,$@)

INSTALLED_SYSTEMIMAGE := $(PRODUCT_OUT)/system.img

[Jean-Baptiste Queru]

Yes, it looks like parallelism works now, at least on linux (haven't
tried on MacOS).

There are some other concerns:
-can you build the kernel on MacOS?
-what's the impact on build time?
-what's the best way to deal with multiple kernel trees (android
routinely uses 5 primary kernel trees)?
-what's the impact on the repository size (over-the-wire and on-disk)?

JBQ

[Patrick Georgi]

On 31 Jan., 16:32, Jean-Baptiste Queru <j...@android.com> wrote:
> There are some other concerns:
> -can you build the kernel on MacOS?

No idea. But given that it's cross compiled... No, no idea.

> -what's the impact on build time?

5 minutes of about an hour of total build time here.
When doing incremential builds the ratio is slightly worse as the
linux build system (which needs to stat all files, and track
dependencies and rules) dominates the process of kernel building then.
But this probably depends a whole lot on the memory configuration (ie.
how much of the build system is stored in caches).

> -what's the best way to deal with multiple kernel trees (android
> routinely uses 5 primary kernel trees)?

What's the scenario - a single tree for multiple devices, needing
different kernels?
I assume that you need one kernel per board. In that case, you could
add multiple kernel sources to your tree, add the desired kernel tree
to your board's config (USE_KERNEL_SOURCE := kernel/tegra) and
parameterize kernel.mk on that (make -C kernel, KBUILD_OUTPUT, ...).
That's probably the smallest issue with this proposal and build system
patch.

> -what's the impact on the repository size (over-the-wire and on-disk)?

linux checkout in my tree:
~500mb for the repo (over-the-wire and on-disk)
~500mb for the checkout (on-disk)
~500mb for the object files (on-disk)

Multiply as appropriate for several trees - if the multiple kernels
were branches in a single repo (so they share the 95% common code and
history) and repo were smart enough to fetch it only once for multiple
checkouts (seems to be possible since it symlinks parts of .git to a
central set of repositories stored below .repo), the required
repository size (wire and disk) could be reduced for multi-kernel
environments.


Regards,
Patrick

[Jean-Baptiste Queru]

The scenario for multiple kernel trees is Android development at
Google. It's common to have kernels from at least 5 different trees
floating around (common, msm, omap, samsung, tegra) to deal with our
usual build targets, not counting the unusual ones.

Things could get a bit worse as the same platform build might be using
kernels for different devices built from the same kernel tree but at
different revisions (or even from different branches). E.g. sapphire
and passion were both built from the msm kernel tree. That means that
we might actually need multiple simultaneous independent checkouts of
the same kernel tree.

Theoretically, I'm guessing that repo could be taught to initialize
additional kernel trees by seeding their .git from kernel/common.
That'd take care of the initial sync, which is typically where things
are painful.

JBQ

[Patrick Georgi]

On 2 Feb., 15:50, Jean-Baptiste Queru <j...@android.com> wrote:
> Things could get a bit worse as the same platform build might be using
> kernels for different devices built from the same kernel tree but at
> different revisions (or even from different branches). E.g. sapphire
> and passion were both built from the msm kernel tree. That means that
> we might actually need multiple simultaneous independent checkouts of
> the same kernel tree.

That should be possible with today's repo tool already. The main
drawback is even more disk space use (500mb per checkout, as long as
it isn't built)

> Theoretically, I'm guessing that repo could be taught to initialize
> additional kernel trees by seeding their .git from kernel/common.
> That'd take care of the initial sync, which is typically where things
> are painful.

That would also be great for people like me (rather short on space/
traffic) who want to host kernel changes: tell repo to use the
official kernel repo, then add the local revs (small transfer) on
top :)


Patrick

[Jean-Baptiste Queru]

At some point in the past repo wouldn't allow mapping the same remote
project twice (i.e. same name attribute). This might have changed.

On a pure gerrit instance I know that we've once worked around it by
creating a server-side symlink (ewww). I still feel dirty.

JBQ

[Magnus Bäck]

On Thursday, February 03, 2011 at 17:10 CET,
Jean-Baptiste Queru <j...@android.com> wrote:

> At some point in the past repo wouldn't allow mapping the same remote
> project twice (i.e. same name attribute). This might have changed.

I'm afraid not, the project name is still the (sole) primary key.
Quoting manifest_xml.py:

for node in config.childNodes:
if node.nodeName == 'project':
project = self._ParseProject(node)
if self._projects.get(project.name):
raise ManifestParseError, \
'duplicate project %s in %s' % \
(project.name, self.manifestFile)
self._projects[project.name] = project

But I guess just changing the primary key in Repo's internal data
structures won't solve much; having multiple worktrees from a single git
does work but isn't without caveats. If we'd skip that and just rely on
Git's ability to clone gits locally by using hard links we'd be really
well off right after the initial repo init/sync, but once a shared git
is repacked the used disk space would be back to normal again.

(We're veering off-topic again. Follow-ups to repo-discuss?)

[...]

[Michael Barabanov]

On Wednesday, February 2, 2011 6:50:41 AM UTC-8, JBQ wrote:

Things could get a bit worse as the same platform build might be using
kernels for different devices built from the same kernel tree but at
different revisions (or even from different branches). E.g. sapphire
and passion were both built from the msm kernel tree. That means that
we might actually need multiple simultaneous independent checkouts of
the same kernel tree.

Theoretically, I'm guessing that repo could be taught to initialize
additional kernel trees by seeding their .git from kernel/common.
That'd take care of the initial sync, which is typically where things
are painful.

If the kernel source is the same, but the config files are different, one can use O= option to the Linux kernel make to put the .config and all build output into separate directories. out/target/product/<product>/kernel would be a good place.

Michael.

[Jean-Baptiste Queru]

Different sources, even when coming from the same repository (like I
said, sapphire and passion are both built out of kernel/msm, but
they're not built from the same commit, so it's not possible to
checkout kernel/msm to a commit that allows building both).

JBQ

[Chih-Wei]

On 2月2日, 下午6時34分, Patrick Georgi <patr...@georgi-clan.de> wrote:
> On 31 Jan., 16:32, Jean-Baptiste Queru <j...@android.com> wrote:> There are some other concerns:
> > -can you build the kernel on MacOS?
>
> No idea. But given that it's cross compiled... No, no idea.

I'm not sure.
But I recall some of our developers work on MacOS.
So I believe it should just work without problems.

--
Chih-Wei
Android-x86 project
http://www.android-x86.org

[Chih-Wei]

On 2月4日, 上午10時51分, Michael Barabanov <michaelwr...@gmail.com> wrote:
> If the kernel source is the same, but the config files are different, one
> can use O= option to the Linux kernel make to put the .config and all build
> output into separate directories. out/target/product/<product>/kernel would
> be a good place.

Indeed in my kernel.mk I put
O=out/target/product/<product>/obj/kernel

(cannot use out/target/product/<product>/kernel
since it is actual the kernel image)