Future of the Android.mk WebView build

[Torne (Richard Coles)]

Hi folks,

I maintain the gyp backend "android" which outputs Android.mk files suitable for building inside the Android OS's build system. This is used to build the Android WebView, which consists of the content layer and also the code in src/android_webview, and has some special cases in Chromium gyp files (controlled by the gyp define "android_webview_build=1"); it's built/tested by the "android_aosp" bot on the main waterfall. This is *not* what's used for building the Chrome for Android browser port, just to be clear.

Right now I don't anticipate many problems caused by the transition while it still generates gyp files, but I'm wondering what the long term plan is. In initial meetings about gn it was mentioned that it might be possible to have a backend that continues to generate Android.mk files; is that still considered a viable option? Does anyone have even a rough idea how much work that might be?

Some background on how Android builds and what its requirements are:

1) Android builds using make, with individual modules using an Android.mk file to describe their build targets. These are largely declarative (defining variables that specify sources/flags then including a target-type-specific .mk that sets up the rules based on those variables) but arbitrary make syntax is permitted including custom rules. These are normally written by hand, not autogenerated. The build system defines standard rules for compiling/linking various languages and library types (including ones that gyp does not provide first class support for, like protobuf or AIDL) and these are expected to be used as-is, with custom rules only used for other kinds of processing.

2) Android supports building for multiple target platforms in the same output tree, with non-target-specific parts like host binaries or Java compilation shared in common directories. This has caused problems with gyp since gyp doesn't prevent host binaries from depending on target-platform-specific settings; we've had to special-case the places where this happens.

3) Android supports building on both linux and mac hosts.

4) Android forbids writing any files into the source tree at build time, and the way the build system finds and includes makefiles makes it difficult (as well as undesirable for performance reasons) to generate makefiles at build time. Currently we generate the Chromium makefiles at the time we merge the sources from Chromium and check them into the tree (for all six combinations of {linux,mac} host and {arm,x86,mips} target). The Android team are reluctant to accept any changes that would make build startup time any worse, since it already can take a minute or more just to parse the existing makefiles on a Z620 (yes, really, and as a bonus everything is much slower on mac).

Feel free to ask me any questions you have about this; I realise this is probably going to look like a lot of work for a special case project to many of you, but the Chromium-based WebView is a very important project to the Android team maintained by a pretty small team, and we realllly need your help at least deciding what the path forward should be :)

Thanks in advance ;)

[Brett Wilson]

On Mon, Dec 9, 2013 at 6:43 AM, Torne (Richard Coles)
<to...@chromium.org> wrote:
> Hi folks,
>
> I maintain the gyp backend "android" which outputs Android.mk files suitable
> for building inside the Android OS's build system. This is used to build the
> Android WebView, which consists of the content layer and also the code in
> src/android_webview, and has some special cases in Chromium gyp files
> (controlled by the gyp define "android_webview_build=1"); it's built/tested
> by the "android_aosp" bot on the main waterfall. This is *not* what's used
> for building the Chrome for Android browser port, just to be clear.
>
> Right now I don't anticipate many problems caused by the transition while it
> still generates gyp files, but I'm wondering what the long term plan is. In
> initial meetings about gn it was mentioned that it might be possible to have
> a backend that continues to generate Android.mk files; is that still
> considered a viable option? Does anyone have even a rough idea how much work
> that might be?

I think make is pretty similar to ninja files so I think something
that works 80% could be hacked up in a few days to generate Android
makefiles.

There are some subtle cases that will take longer. The thing that will
also slow it down is that we don't want to just duplicate all of the
Ninja code, probably a bunch of it can be shared, so there would be
some refactoring involved. I've done some for the GYP generator, but
it has simpler requirements (since it doesn't actually link anything
and I shell out to Ninja to run scripts). In all, I'd say this is a
several week project.

What I'm planning on doing for script targets in GYP is to write a GYP
action that invokes ninja. Have you considered writing a makefile that
runs ninja? This would be very easy to do and also the easiest to
maintain. From a build purity standpoint this is bad and easy to
reject. But all that's required is a ninja binary on the host system,
it's super fast to implement, and it will probably compile and start
up faster also.

> Some background on how Android builds and what its requirements are:
>
> 1) Android builds using make, with individual modules using an Android.mk
> file to describe their build targets. These are largely declarative
> (defining variables that specify sources/flags then including a
> target-type-specific .mk that sets up the rules based on those variables)
> but arbitrary make syntax is permitted including custom rules. These are
> normally written by hand, not autogenerated. The build system defines
> standard rules for compiling/linking various languages and library types
> (including ones that gyp does not provide first class support for, like
> protobuf or AIDL) and these are expected to be used as-is, with custom rules
> only used for other kinds of processing.
>
> 2) Android supports building for multiple target platforms in the same
> output tree, with non-target-specific parts like host binaries or Java
> compilation shared in common directories. This has caused problems with gyp
> since gyp doesn't prevent host binaries from depending on
> target-platform-specific settings; we've had to special-case the places
> where this happens.

GN will let you depend on any settings, although if you want host
binaries to depend on target configurations, it's rather difficult and
you have to go out of your way. As a result, I would not expect much,
if any, of theses places to exist in the GN build.

Brett

[Torne (Richard Coles)]

On 9 December 2013 18:17, Brett Wilson <bre...@chromium.org> wrote:

On Mon, Dec 9, 2013 at 6:43 AM, Torne (Richard Coles)
<to...@chromium.org> wrote:
> Hi folks,
>
> I maintain the gyp backend "android" which outputs Android.mk files suitable
> for building inside the Android OS's build system. This is used to build the
> Android WebView, which consists of the content layer and also the code in
> src/android_webview, and has some special cases in Chromium gyp files
> (controlled by the gyp define "android_webview_build=1"); it's built/tested
> by the "android_aosp" bot on the main waterfall. This is *not* what's used
> for building the Chrome for Android browser port, just to be clear.
>
> Right now I don't anticipate many problems caused by the transition while it
> still generates gyp files, but I'm wondering what the long term plan is. In
> initial meetings about gn it was mentioned that it might be possible to have
> a backend that continues to generate Android.mk files; is that still
> considered a viable option? Does anyone have even a rough idea how much work
> that might be?

I think make is pretty similar to ninja files so I think something
that works 80% could be hacked up in a few days to generate Android
makefiles.

Android makefiles are not really that similar to ninja files, because the rules are almost all predefined. They're much more similar to gyp files (they have separate defined modules, each with lists of source files, lists of library dependencies, etc, and most modules have no actual make rules whatsoever, just variable definitions). See something like https://android.googlesource.com/platform/external/chromium_org/+/kitkat-mr1-release/base/base.target.linux-arm.mk for an example of one of the makefiles generated by the gyp android backend.

 

There are some subtle cases that will take longer. The thing that will
also slow it down is that we don't want to just duplicate all of the
Ninja code, probably a bunch of it can be shared, so there would be
some refactoring involved. I've done some for the GYP generator, but
it has simpler requirements (since it doesn't actually link anything
and I shell out to Ninja to run scripts). In all, I'd say this is a
several week project.

What I'm planning on doing for script targets in GYP is to write a GYP
action that invokes ninja. Have you considered writing a makefile that
runs ninja? This would be very easy to do and also the easiest to
maintain. From a build purity standpoint this is bad and easy to
reject. But all that's required is a ninja binary on the host system,
it's super fast to implement, and it will probably compile and start
up faster also.

I don't think running ninja from make is a viable solution, because a large amount of the logic about how to correctly build things for the current target platform is only known by Android's core makefiles. For example, we currently get all CPU-specific flags automatically because they're included for us in every invocation of gcc, we get the toolchain selected for us, we get symbols stashed in the correct location for automatic backtrace handling, and many other things that come because we're using Android's compile and link rules instead of defining them ourselves. This is also the only reason we manage to successfully build on mac hosts (which Chrome for Android can't) - because we don't actually specify the command lines used, the build system does the right thing for mac without us having to worry about it. gyp+ninja is unable to do this correctly at present, since ends up using linux-specific linker commands for host binaries because there's no way to tell it the *host* OS.

In fact, a desirable feature would be to be able to build *more* types of files using the Android build system's default rules: currently we have to define our java targets in hand-written Android.mk files because gyp has no way to represent "compile some java" that can be translated automatically, and we still manually invoke aidl/protoc/various other tools that Android already knows how to run. It would be nice to use Android's existing support for *all* these things, but right now there's no real way for the generator to do this that isn't a giant hack.

Running ninja from make is going to break parallelism, unless ninja knows how to talk to the make jobserver...

 

> Some background on how Android builds and what its requirements are:
>
> 1) Android builds using make, with individual modules using an Android.mk
> file to describe their build targets. These are largely declarative
> (defining variables that specify sources/flags then including a
> target-type-specific .mk that sets up the rules based on those variables)
> but arbitrary make syntax is permitted including custom rules. These are
> normally written by hand, not autogenerated. The build system defines
> standard rules for compiling/linking various languages and library types
> (including ones that gyp does not provide first class support for, like
> protobuf or AIDL) and these are expected to be used as-is, with custom rules
> only used for other kinds of processing.
>
> 2) Android supports building for multiple target platforms in the same
> output tree, with non-target-specific parts like host binaries or Java
> compilation shared in common directories. This has caused problems with gyp
> since gyp doesn't prevent host binaries from depending on
> target-platform-specific settings; we've had to special-case the places
> where this happens.

GN will let you depend on any settings, although if you want host
binaries to depend on target configurations, it's rather difficult and
you have to go out of your way. As a result, I would not expect much,
if any, of theses places to exist in the GN build.

Well, V8 is the place in chromium where this is unavoidably done, and we fixed that by munging target_arch into the names of the host targets. As long as no new places turn up that works :)

 

Brett

To unsubscribe from this group and stop receiving emails from it, send an email to gn-dev+un...@chromium.org.

[Brett Wilson]

On Mon, Dec 9, 2013 at 10:40 AM, Torne (Richard Coles)

Certainly you can do something like we do for GYP that exports lists
of files. My approach to the GYP exporter though is the opposite of
what you describe. I inherit nothing from the surrounding environment
and specify all command lines exactly. Otherwise the conditionals
expressed in the BUILD.gn files will be useless.

Depending on how much fidelity you need, you could certainly export
file lists into whatever format you want, and this should be pretty
easy. Doing something where the flags are combined with Android's
flags will be nontrivial, though. GN files are really scripts that
execute and produce some results, so by the time the generator is run,
the context of how stuff was set is lost. I suspect the ability of GYP
to keep such context around is used in your current output scheme.

Brett

[Torne (Richard Coles)]

I'm not sure I follow what you mean by the conditionals being useless. In the android gyp backend I just process the gyp files exactly as normal and put all the cflags/etc that gyp arrived at into the appropriate variables in the output makefile so that the normal android rules will add those flags to the command lines where they're needed.

>
> Depending on how much fidelity you need, you could certainly export
> file lists into whatever format you want, and this should be pretty
> easy. Doing something where the flags are combined with Android's
> flags will be nontrivial, though. GN files are really scripts that
> execute and produce some results, so by the time the generator is run,
> the context of how stuff was set is lost. I suspect the ability of GYP
> to keep such context around is used in your current output scheme.

I'm not sure what you mean here either. We don't have any context at gyp time; at gyp time we don't even know exactly what product config we are building for (only the architecture, since that affects the source file choices). The flags emitted by gyp into the makefiles are just appended to the command line after the default ones for that product, by the default rules.

[Brett Wilson]

On Mon, Dec 9, 2013 at 10:59 AM, Torne (Richard Coles)

Okay, then maybe it's about the same difficulty as with GYP.

Brett

[David Turner]

Now with correct sender address. Damn.

---------- Forwarded message ----------
From: "David Turner" <di...@google.com>
Date: Dec 9, 2013 10:47 PM
Subject: Re: Future of the Android.mk WebView build
To: "Brett Wilson" <bre...@chromium.org>
Cc: "be...@chromium.org" <be...@chromium.org>, "gn-dev" <gn-...@chromium.org>, "Torne (Richard Coles)" <to...@chromium.org>

One thing I did for the OpenSSL sources: use one config file that is parsed by a custom script that generates Android.mk fragments and .gypi files.

The config file lists source files, compiler flags, include paths, and arch-specific overrides.

Works pretty well, though it doesn't handle Java, and requires running the script on each config change.

Maybe something similar could be used, due to the vast differences in these build systems?. Each one has carefully custom written Android.mk and .gyp files that include the generated ones.

[Torne (Richard Coles)]

On 9 December 2013 21:47, David Turner <di...@google.com> wrote:

One thing I did for the OpenSSL sources: use one config file that is parsed by a custom script that generates Android.mk fragments and .gypi files.

The config file lists source files, compiler flags, include paths, and arch-specific overrides.

Works pretty well, though it doesn't handle Java, and requires running the script on each config change.

Maybe something similar could be used, due to the vast differences in these build systems?. Each one has carefully custom written Android.mk and .gyp files that include the generated ones.

Maintaining this for the entire chromium sources seems unlikely to be a popular option with chromium developers :) 

[David Turner]

I agree :-) However the alternative seems to teach GN about Android build system quirks, or to teach the Android build about GN ones, which sounds equally hard / impossible, from your previous posts.

I suspect some sort of conversion script will be needed, if you don't want to manually do this every time.

[Torne (Richard Coles)]

On 10 December 2013 10:39, David Turner <di...@chromium.org> wrote:

I agree :-) However the alternative seems to teach GN about Android build system quirks, or to teach the Android build about GN ones, which sounds equally hard / impossible, from your previous posts.

If that's equally hard then we have a serious problem. I am hoping that it's much, much easier! :)

The requirements of the Android build system can potentially be changed; I have made changes to the build system in the past to support things required by the gyp android backend. However, the general approach of wanting to build inside the system instead of outside it is not likely to be something we can change at this time, and I'm reluctant to even try and have that conversation unless it really seems like there's no alternative, especially since we currently have a pretty much working solution using gyp.

Is there any particular requirement here that seems like it's going to be a major problem for gn to handle? If so I'd like to discuss that now, so that I can investigate what can potentially be changed on the Android side..

There is also a question about who is going to do this work and when. I hate to do this to people but the WebView team is pretty small and while I'm the de facto "expert" on the current state of the build system integration it's not the only thing on my plate; if we're going to need to devote a lot of development time to working on this then we need to know that as soon as possible too, so that we can make our plans accordingly.

[Milan Hauth]

> Running ninja from make is going to break parallelism, unless ninja knows how to talk to the make jobserver...

there is a ninja fork with jobclient and jobserver

https://github.com/stefanb2/ninja/tree/topic-tokenpool-master

via

https://github.com/ninja-build/ninja/issues/1139#issuecomment-461750356

to start the main build process:

ninja --tokenpool-master