How do folks build for both Android and non-Android?

[Jeffrey Yasskin]

I have a Linux checkout of Chromium, and need to write an Android patch. The Android instructions say to run `fetch android`, which of course creates an entirely new checkout, which wastes space and time compared to just creating an extra output directory.

Is there any documentation on developing for both Android and Linux in the same tree?

Thanks,

Jeffrey

[Nico Weber]

Add

  target_os = ['android']

as the last line to your .gclient file and rerun `gclient sync`. Now your checkout can do both linux and android builds. Run `GYP_DEFINES=OS=android build/gyp_chromium -Goutput_dir=out_android` to have a separate build dir for android. Or, with gn, `gn args out/and` and add a line saying `target_os = "android"` to the file that's opened in your editor.

This is documented in several threads on this list; I don't know if any of the "real" docs say it. Maybe add it to where you expected it to be :-)

Nico

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev

[Brett Wilson]

I am pretty sure this is possible with GYP but I've never done it and is likely annoying (involving switching GYP defines and output directories).

It should be much easier with GN. I do Android builds and Linux builds from the same tree (since GN supports user-defined output directories). Disclaimer: I've never actually tried to run any of the Android binaries I've produced so I don't know if this works 100%.

There is a section at the bottom of https://www.chromium.org/developers/gn-build-configuration that describes how to get the Android necessities from a Linux checkout. Then just follow the Android GN build instructions: https://www.chromium.org/developers/how-tos/android-build-instructions

Brett

On Mon, Nov 2, 2015 at 7:09 PM, Jeffrey Yasskin <jyas...@chromium.org> wrote:

--

[Lei Zhang]

Are there any down sides to adding the target_os line in the .gclient
file? Will gclient forget to run some hooks as a result due to
thinking os=Android instead of os=Linux somewhere?

FWIW, the disk space waste of an extra Android checkout is 25 GB here.

[Nico Weber]

On Mon, Nov 2, 2015 at 7:31 PM, Lei Zhang <the...@chromium.org> wrote:

Are there any down sides to adding the target_os line in the .gclient
file? Will gclient forget to run some hooks as a result due to
thinking os=Android instead of os=Linux somewhere?

I haven't seen any problems caused by it. The only downside is that `gclient sync` downloads way more stuff – the android ndk is a bunch of large-ish binary files which we store in a git repo, and this isn't the file composition that git is particularly good at.

[Jeffrey Yasskin]

Thanks! https://sites.google.com/a/chromium.org/dev/developers/how-tos/android-build-instructions?pli=1 now includes that instruction, and explicitly says to run `gclient sync` at what I'm guessing is the right point in the flow.

I also removed the broken mention of "--filter 57", since filter indices aren't stable.

On Mon, Nov 2, 2015 at 7:19 PM, Nico Weber <tha...@chromium.org> wrote:

[Colin Blundell]

On Tue, Nov 3, 2015 at 4:31 AM Lei Zhang <the...@chromium.org> wrote:

Are there any down sides to adding the target_os line in the .gclient
file? Will gclient forget to run some hooks as a result due to
thinking os=Android instead of os=Linux somewhere?

My understanding of how target_os works is that it's an add-on to the host OS, not a replacement. This is why target_os_only exists, because in an iOS checkout Mac cruft is not desired (I can't recall the specific reasons at the moment).

[Anthony Berent]

Note that, if you haven't, you will need to run src/build/install-build-deps-android.sh once on your machine. This installs all the packages needed to build Chrome for Android.

[Carlos Knippschild]

(sorry for the late reply on this topic)

I've used to use an Android checkout to also develop for Linux and it worked well for most situations. The issue I had was that GYP configurations wouldn't be fully correct when compiling for Linux resulting in a slightly different build. In my case some tests that were expected to be built and run did not and I it took me a while to understand what was going on. This is apparently the case due to limitations on how GYP configuration works; maybe GN could to better in that area.

Now I'm working with two separate checkouts and finding it better overall. It does use more space but the ability to work on two different things simultaneously is useful. I don't recall exactly the numbers but I think the 25 GB previously mentioned might include more than one output folder (each of them amounts to a few GBs).

[Thiago Farina]

On Wed, Nov 11, 2015 at 8:42 AM, Carlos Knippschild <car...@chromium.org> wrote:

(sorry for the late reply on this topic)

I've used to use an Android checkout to also develop for Linux and it worked well for most situations. The issue I had was that GYP configurations wouldn't be fully correct when compiling for Linux resulting in a slightly different build. In my case some tests that were expected to be built and run did not and I it took me a while to understand what was going on. This is apparently the case due to limitations on how GYP configuration works; maybe GN could to better in that area.

Did you use two separare output directories? I have built the two configurations in a single checkout (using separate output directories for them) and never had any problem.

--

Thiago Farina

[Torne (Richard Coles)]

An android checkout is a strict superset of a Linux checkout, so if you have issues something is wrong and you should let us know exactly what so we can debug it. :)

I use btrfs snapshots to be able to work on an arbitrary number of things at once, by snapshotting my checkout to make a copy-on-write copy that can throw away when I'm done, with all unchanged files (including all the git data) shared.

--

[Carlos Knippschild]

On Wed, Nov 11, 2015 at 12:07 PM Torne (Richard Coles) <to...@chromium.org> wrote:

An android checkout is a strict superset of a Linux checkout, so if you have issues something is wrong and you should let us know exactly what so we can debug it. :)

In terms of the artifacts I'm pretty sure it is but not for the build configuration. In my specific case I was seeing differences between the set of *Prerender* and *SafeBrowsing* browser_tests when compiling for Linux on an (internal) Android checkout and a Chromium checkout. And yes, I was using separate output directories.

As android has a much more limited SafeBrowsing implementation less tests are run for it and this characteristic seemed to be "escaping" the Android boundaries and affecting the Linux builds.

But I just tried reproducing this (it originally happened around mid-September) and I couldn't: the lists of tests are now identical. So whatever the problem was it seems to be fixed. Maybe I just got a bad GYP configuration back then?

I use btrfs snapshots to be able to work on an arbitrary number of things at once, by snapshotting my checkout to make a copy-on-write copy that can throw away when I'm done, with all unchanged files (including all the git data) shared.

Looks like a good workflow to save on disk space (just learned what btrfs is). Is it really stable enough for regular usage?

[Torne (Richard Coles)]

On Fri, 13 Nov 2015 at 13:17 Carlos Knippschild <car...@chromium.org> wrote:

On Wed, Nov 11, 2015 at 12:07 PM Torne (Richard Coles) <to...@chromium.org> wrote:

An android checkout is a strict superset of a Linux checkout, so if you have issues something is wrong and you should let us know exactly what so we can debug it. :)

In terms of the artifacts I'm pretty sure it is but not for the build configuration. In my specific case I was seeing differences between the set of *Prerender* and *SafeBrowsing* browser_tests when compiling for Linux on an (internal) Android checkout and a Chromium checkout. And yes, I was using separate output directories.

As android has a much more limited SafeBrowsing implementation less tests are run for it and this characteristic seemed to be "escaping" the Android boundaries and affecting the Linux builds.

But I just tried reproducing this (it originally happened around mid-September) and I couldn't: the lists of tests are now identical. So whatever the problem was it seems to be fixed. Maybe I just got a bad GYP configuration back then?

If you're using separate output directories there aren't many ways for things to leak between the configs, and if we find one we should fix it. It sounds like it's gone, so hey. 

I use btrfs snapshots to be able to work on an arbitrary number of things at once, by snapshotting my checkout to make a copy-on-write copy that can throw away when I'm done, with all unchanged files (including all the git data) shared.

Looks like a good workflow to save on disk space (just learned what btrfs is). Is it really stable enough for regular usage?

That depends what you want out of a filesystem, to be honest ;)

It's not considered suitable for production by most people. You should use at least the most recent kernel and btrfs-tools that your linux distro offers as a stable option (if you're on goobuntu there is a goobuntu-config option for install_backport_kernel which will give you a newer than default kernel from ubuntu's backport repos).

I use it on a giant fileserver at home (in raid1 storage mode across many disks) and on my workstation at work (in raid0 storage mode across two SSDs). I've not experienced any crashes or data loss.

However, it does periodically (maybe every few weeks) get into a state where it can't allocate any blocks, failing with ENOSPC when there is plenty of space free, and I have to mess about manually rebalancing the FS which can occasionally be quite problematic (sometimes it decides it can't do a rebalance because of there not being any space). I've always been able to do this successfully with enough fiddling around, but once or twice it's taken me an hour or so of messing about and asking people on the btrfs irc channel for advice to get the FS back into a state where writes are possible. It's possible with an even newer kernel this wouldn't happen any more, I'm only running 3.16 currently (the Vivid backport) since that's what goobuntu installs for me, I may try upgrading again sometime.

It's not yet been such a drain on my time that it hasn't been worth the enormous convenience of being able to just clone entire source checkouts "for free" :)

[Ian Clelland]

I'm a bit late to this thread, but I've been using the instructions at https://www.chromium.org/developers/how-tos/get-the-code/multiple-working-directories to set up parallel source trees pointing to the same local git repository. It seems to save a significant amount of space (the secondary working dir is only 2.3GB, rather than 21GB, excluding build artifacts)

Sharing the git repo this way also means that I can be working on different branches for my Linux and Android builds, but when I commit in one directory, I can just check that branch out immediately in the other one if I want to compile and test it there as well.

--

[Torne (Richard Coles)]

Yeah, that is also useful (and much safer), but with btrfs snapshots I am also (until modification) sharing the actual checked out files and also all the build artifacts: I can snapshot the whole checkout, change something, and do an incremental build, and it will 1) reuse all the existing build artifacts from the old checkout instead of building from scratch and 2) only use as much disk space as the artifacts it had to actually rebuild ;)

However note that with snapshots the git repo and branches are *also* copy-on-write, and so if you commit to a branch in one snapshot it *doesn't* appear in the other, and if you then delete the snapshot then your commit is gone, woohoo. :)

You could avoid this by doing *both* things together: having a central set of .git repositories and then snapshot the workdir/outdir, but I haven't bothered and if I really need to move a commit from one to another I just "git fetch" it from the other dir.

[Carlos Knippschild]

Being able to share branches among the different working directories and saving on disk space and having to update from the remote branch only once sounds like the better solution so far. Thanks for those instructions, I'll take a look.

[Nico Weber]

On Wed, Nov 11, 2015 at 2:42 AM, Carlos Knippschild <car...@chromium.org> wrote:

(sorry for the late reply on this topic)

I've used to use an Android checkout to also develop for Linux and it worked well for most situations. The issue I had was that GYP configurations wouldn't be fully correct when compiling for Linux resulting in a slightly different build. In my case some tests that were expected to be built and run did not and I it took me a while to understand what was going on. This is apparently the case due to limitations on how GYP configuration works; maybe GN could to better in that area.

I'm almost certain you're holding it wrong somehow – this should work just fine, and it does over here. (Maybe it's harder to hold it wrong in gn though! If this is too hard to use with gyp, that's definitely a shortcoming of gyp.)