Build System high level feature proposal

Showing 1-56 of 56 messages
Build System high level feature proposal Xavier Ducrohet 8/3/12 5:36 PM
As I've mentioned before, we are looking are a new version of the build system provided in the SDK.

Our goal is to make it easy for beginners but flexible and powerful enough for teams working on large complex application(s).

We started looking at the high level features that the build system would offer. This is independent from the implementation of the build system and how those features would be exposed to devs (this will come later). The following is a proposal for these features (not including the basic features currently available, those are a given), and we are looking for feedback.

So what are we trying to do?

  • Better extensibility
    • Allow devs to configure every steps.
      • expose all options for aapt, dex, …
      • allow changing input/output for all steps.
        • For instance, bypass proguard for a given jar file.
        • Use different jar for compile vs. package
    • Better control over debug vs. release builds
      • manifest info
      • BuildConfig
      • jni compilation
      • different res/code depending on build.
    • Let devs add new logic in the build system
      • New tasks
      • Change dependencies
      • Integrate into other build systems.
  • Help devs making different versions of the same app.
    • Library Projects can be cumbersome to make 2 apps that are 99% identical.
    • Different versions for different app store could only differ in package name and signing key.
  • Better dependency management for 3rd party jar files and Library Projects
    • Use Maven/Ivy for dependency management
    • Depend on Library Projects distributed as binary blob
    • Generate Library Projects blob


To implement this, we are introducing new build concepts.

Product Flavors

A product flavor defines a customized version of the application build by the project. A single project can have different flavors which change the generated application.

Although different flavors could be very different applications, using Library Projects is a better use case for this, and the build system still supports this.
This new concept is designed to help when the differences are very, very minimum, for instance when using Google Play’s multi-apk support (e.g. the difference could be only gl textures compressed using a different format.)
If the answer to “Is this the same application?” is yes, then this is probably the way to go over Library Projects.

Product flavors can customize the following properties:
  • minSdkVersion
  • targetSdkVersion
  • versionCode
  • versionName
  • package name (overrides value from manifest)
  • release signing info (keystore, key alias, passwords,...).
  • native abi filter
  • test info
    • package name for test app (optional, default is <base>.test)
    • InstrumentationTestRunner class (optional)

Additionally, Product Flavor can provide their own source code, resources and manifest.

A project starts with a main product configuration. All the above properties can be set and it has its own source code, resources and manifest.
If the project does not create product flavors, then the application is built from this main configuration.

When a project declares Product Flavors, these extends the main configuration.

The source code of the product flavor is used in addition to the source code of the main configuration (1).
The resources of the product flavor overrides the resources of the main configuration.
The Manifest of the product flavor is merged on top of the Manifest of the main configuration.

(1) This allows some flexibility but also adds some restrictions:
  • Classes in the main configuration and in the product flavors can reference each other. All the source sets are used to generate a single output (instead of each generating its own output with a one-way dependency).
  • Classes exposed by product flavors and referenced by classes in the main configuration must be present in all product flavors, and must have the same API for all builds to succeed.

When a Product Flavor is defined, the main configuration is not available as a buildable product. To create more than one product, two or more flavors must be created and customized.

Build Types

A build type allows configuration of how an application is packaged for debugging or release purpose.

This concept is not meant to be used to create different versions of the same application. This is orthogonal to Product Flavor.

A Build Type provides configuration for the following build properties:
  • manifest debuggable flag
  • native compilation debug flag
  • proguard enabled + rules
  • debug signing flag (ie whether to use debug key or release key)
  • package name suffix (2)
  • Buildconfig
    • DEBUG flag
    • other custom properties?

By default the build system provides two build types, debug and release, which can be reconfigured. New types can also be created to provide any combinations of the above properties.
For instance, one could make a Build Type that uses Proguard but creates a debuggable apk.

(2) This is appended to the manifest package name. The goal here is allow having different build types create different apks that can all be installed at the same time. This is optional.

Like Product Flavors, Build Types can provide their own source code, resources and manifest.
The typical use cases are:
  • different MapView key in release and debug mode as this is linked to the signing key (3)
  • require extra permissions for a debug build.

(3) If Product Flavors are used to create different versions built with different keys, the MapView API Key can be provided by the resources of the Product Flavor instead. See Build Variants below.

The source code and resources provided by the Build Types affects the project the same way Product Flavors do.

Build Variants

We’ve seen Product Flavors and Build Types and how both configure the output of a project.

In fact, the output of a project can only be the cross product of the Product Flavor and the Build Type. This is call a build variant.

It is not possible to build a project without selecting both a Build Type and a Product Flavor.

For a project not customizing any Build Type or Product Flavor, the available build variants are debug and release versions, as the main configuration is implied.

However, if a project declares two flavors, for instance free and forpay, the available build variants are:
  • Free-Debug
  • Free-Release
  • Forpay-Debug
  • Forpay-Release

We have seen above how source code and resources provided by Build Types and Product Flavors extend or override the main configuration. When actually building an app, the Build Type and Product Flavor are both applied with the following rules:

All the source folders are used together (main config, build type, product flavor, and generated source code(4)) to create a single output.

This means the same restrictions mentioned above applies, with the addition that Build Types and Product Flavors cannot provide different versions of the same class.

(4) generated source is output from resource compilation (R.java), aidl and renderscript compilation, and things like BuildConfig which can be impacted by the BuildType. These are different per build variants.

Resources combination uses the following priority (lower number is higher priority and override higher number):
  1. generated from Build Typed (5)
  2. build type
  3. generated from Product Flavor (5)
  4. flavor
  5. generated from main configuration (5)
  6. main configuration

(5) These are the outputs of the image crunch step, and of the renderscript compilation.

Packaging of Assets, Java Resources, JNI libraries follow a similar priority when encountering duplicates:
  1. Build Type
  2. Product Flavor
  3. main configuration

Sourcesets

Due to Build Types and Product Flavors all providing their own sources, resources and manifest, the project structure need to change from the root level src/res/manifest used in the current build system.

Instead the build system moves to a top level src folder, containing folders for each types and flavors. In addition, the main folder contains the source for the main configuration.

A typical project will then have the following folder structure:

  • src
    • main
      • java
      • jni
      • resources -- this is java resources
      • aidl
      • res
      • res -- this is Android resources
      • assets
      • AndroidManifest.xml
    • debug
      • … (same as main)
    • release
      • … (same as main)
    • flavor1
      • … (same as main)
    • flavor2
      • … (same as main)

The current gen folder is to move under the project output (build folder), and follow a similar pattern. This is accompanied by the generated resources, and the generated (merged) Manifest.

This is of course be completely customizable, but this shows the default, and the type of inputs that are available to tasks.

BuildConfig

BuildConfig is an automatically generated class with a single boolean DEBUG, set to true in debug builds only.

With custom source sets for both Build Types and Product Flavors, the build system covers adding extra flags and triggers (through a user created class) to control what the code does in each build variants. It is a manual process to maintain each version of the same class but it is the most flexible for devs who can do anything they want.

The new build system still makes use of BuildConfig as it is an automatic class creation, and extends its capability.
It provides developers with the ability to programmatically add new items in BuildConfig based on logic defined in the build script.
Details TBD.

Testing

Test Application

In the current build system, test applications are created from a different project that is flagged as a “test project”. This project references the main project in order to get the main project’s classes on its classpath.

With possibly multiple Product Flavors it is important to test all of them.

Some Java build systems have introduced having the test code directly in the project under src/test/ and the new build system follows this convention.

Since JUnit test cannot be run against Android code, the build system reuses this concept for test applications.
There is a mirror of the <span style="font-size:15px;font-family:Arial;font-style:italic;vertical-align:baseli
...
Re: Build System high level feature proposal Jake Wharton 8/4/12 12:28 AM
Hi Xavier,

I have complained very openly, loudly, and in arguably to excess about the lack of a proper build system for Android. The current system is unacceptable with regard to scalability and the breadth at which it needs to be applied. Being no fault of you or your team, the legacy system is simply under-equipped and lacking any true semblance of dynamism to deal with modern requirements.

While I represent only a single person (and occasionally a part of a small company), I cannot stress enough how useful nearly everything you mentioned above will be to both beginners and advanced users alike. Please make every effort to involve the community in testing early iterations and considerations of this new system upon implementation. Nearly everything that you say seems exceedingly relevant to my and my company's interests but true discoverability comes through usage.

I cannot describe how much I want to test this new system during the development stage so please, please, please allow the community to be involved as early as possible. While we have a ridiculous amount of requisite configurations, the benefit can only make this new system even better.

Thank you for the work you've already done and for the work on this new system that we're about to embark on.



<span style="font-size:15px;fo
...
Re: Build System high level feature proposal Alexei Kuznetsov 8/4/12 12:28 AM
How about this one:
 - drop ant and have scripts in other languages
?
Re: Build System high level feature proposal Andrew Kelly 8/4/12 12:51 AM
Hi Xavier,

As an Android developer the main problem I had with the existing build system was when integrating it with Bamboo. The project.properties file is used by Eclipse to find the required libraries used by my app, but my local filesystem layout is completely different to the layout used by Bamboo. This means that the SVN managed version of the file has to have one set of directory paths and my local version has to have another set...and I must remember not to commit these changes back to the SVN repository.

I tried using a maven plugin to get around this problem I believe the plugin zips up library projects at build time then extracts them as dependancies when they're references in the main pom.xml file...this again seems a little clunky.

I've used maven on other J2EE projects and the dependancy management has been great, it would be nice to see a solution to Android builds which can utilise this system better. And this would make using versioned libraries easier too, as legacy builds would pull the older library from the maven (or Nexus) repository.

Many of the topics in your post seem like great features, but are things that I've never come across or needed, perhaps my projects aren't too complex. But the ability to build locally and using Bamboo easily when referencing library projects would be a great benefit to me (and hopefully others) personally.

Regards,

Andy.

<span style="font-size:16px;font-family:Arial;color:rgb(102,102,102);vertica

...
Re: Build System high level feature proposal Nicolas Klein 8/4/12 12:54 AM
This new system looks really promising. Not really useful for the beginner developer which is only going to use the default option everywhere of course. But for the advanced/expert user, it opens so much more possibility :)
I don't have the same complexity/needs as Jake but really can't wait to be able to beta-test it :)

Thanks for the work of the whole ADT team once again :)

Nicolas Klein aka Foxykeep
<span style="font-
...
Re: Build System high level feature proposal Jake Wharton 8/4/12 1:15 AM
I just want to follow up on a great point that you make here, Nicolas, the fact that "the beginner developer [who] is only going to use the default option everywhere." There have been skeptics who've commented as a response to my heat-of-the-moment blog post who think that somehow this system will be of no benefit (or even worse so that it would be a detractor) to the average developer. They could not be more wrong.

With the prevalence of the v4 support library and the addition of the v7 support library and libraries like AppCompat and ActionBarSherlock it is a farce to think that the majority of developers will not need dependency management and resolution.

I cannot stress the following point enough: if you every have to download a .jar manually, you've lost. If you have to manually resolve a version difference between projects, you've lost. If you have you create an IDE project for a library dependency, you've lost.

Proper dependency management is a magical experience and I cannot wait for the every single Android developer to realize the simplicity and elegance of when it's done properly. Once experienced you will continuously wonder how you ever managed to copy .jars into various libs/ folders...

<span style="font-size:19px;font-family:Arial;vertical

...
Re: Build System high level feature proposal Xavier Ducrohet 8/4/12 1:42 AM
First, I'm glad you find some of those features useful.

Yes we do want to involve people very early on. This was a first step of hopefully many.

It's going to take us a while to get the new build system to the point where we unleash it on every devs out there. So for a while we'll keep Ant/Eclipse around and have this new one available for people to test and give feedback.
As all other tools it'll be developed in the open so you can stay very close to where we are.
...
Re: Build System high level feature proposal Xavier Ducrohet 8/4/12 1:46 AM
I agree.

First we want it to be easy to just create a project and build it with little (if any) configuration.

After that we do want people to progressively extend their build as they need. But realistically, discovery of how to do things is never going to be trivial. Even something like setting a dependency on a lib coming from Maven requires some basic knowledge that devs who never used Maven don't have.
Proper documentation on developer.android.com will be critical.


  • src
    • <li style="list-style-type:circle;font-size:15px;font-family:Arial;vertical-align:basel
...
Re: Build System high level feature proposal Xavier Ducrohet 8/4/12 1:55 AM
I have explicitly said at the beginning: 
"This is independent from the implementation of the build system and how those features would be exposed to devs (this will come later)"

If you are trying to convince me we can't implement our goals using Ant, there's no need, I'm already convinced.

However, you say "languages", plural. I fear some people are going to be disappointed. Even if you remove some of the more advanced features described above, building a basic Android app takes a LOT of steps with a lot of logic between those steps.

To make a build system that is truly flexible and extensible for developers while making it build agnostic so that anyone can use it using whatever tech they want (Maven, ant, gradle, sbt, Make, whatever...) seems totally impossible to me.

We are going to choose one, and stick to it. We will make the IDE use this same build system (this is a critical point).
We won't be able to support other build system (besides making whatever code we have available under an easy license, like we do today).

Some people are not going to like it. The best we can do is do the best build system we can so that people are tempted by something else (I guess this means no one should have to rewrite their own system because they can't do something with ours. If this happens we'll have to fix ours.)

Xav
--
Xavier Ducrohet
Android SDK Tech Lead
Google Inc.
http://developer.android.com | http://tools.android.com

Please do not send me questions directly. Thanks!
Re: Build System high level feature proposal Mark Murphy (a Commons Guy) 8/4/12 3:47 AM
On Sat, Aug 4, 2012 at 4:15 AM, Jake Wharton <jakew...@gmail.com> wrote:
> I just want to follow up on a great point that you make here, Nicolas, the
> fact that "the beginner developer [who] is only going to use the default
> option everywhere." There have been skeptics who've commented as a response
> to my heat-of-the-moment blog post who think that somehow this system will
> be of no benefit (or even worse so that it would be a detractor) to the
> average developer.

I suspect that you are referring to me:

http://commonsware.com/blog/2012/07/26/beware-one-percent-solution.html

> With the prevalence of the v4 support library and the addition of the v7
> support library and libraries like AppCompat and ActionBarSherlock it is a
> farce to think that the majority of developers will not need dependency
> management and resolution.

Replace "will not need" with "could benefit from", and I'd be in
agreement. But the further problem is ensuring that the ramp-up to
full-on dependency management and resolution is *graceful* and takes
into account that, based on what I am seeing out in the support
trenches:

- A majority of people coming into Android now have limited Java
programming experience, and a fair number have no programming
experience whatsoever

- A majority of people coming into Android now are not native English
speakers, yet the documentation and support resources are largely in
English, thereby magnifying the effects of complexity

========

The #1 problem I see with the design as proposed is that it
invalidates every Web page and every printed page ever written about
Android app development that covers how to create projects, where
files reside, etc. We'll be able to get a tiny fraction of those
rewritten. Hence, when newcomers search for Android material, they
will come up with the wrong answer much of the time. We already have
to field too many questions where people can't get their heads wrapped
around the R16 libs/ change, because too many Web pages provide
formerly-correct-but-outdated instructions. Heck, there are a number
of people who throw up their hands in despair from the differences
between R19 and R20, which were minuscule compared to what's proposed
here.

Hence, my #1 recommendation is to find some way to have the new
structure be a strict superset of the old structure, so that "Hello,
world" looks pretty much the same, old and new.

--
Mark Murphy (a Commons Guy)
http://commonsware.com | http://github.com/commonsguy
http://commonsware.com/blog | http://twitter.com/commonsguy

_The Busy Coder's Guide to Android Development_ Version 4.0 Available!
Re: Build System high level feature proposal Richard Critten 8/4/12 7:25 AM
I agree with all of Mark's comments and would like to add a comment about documentation.

There are other build systems out there but integrating with them is difficult because of the lack of documentation of the existing build tools.  The one I keep running up against is AAPT.  I think I can workout how to use by reverse engineering the Ant scripts - but where is the documentation?

Second a question: how do you see your build system proposal integrating with the NDK build system based around make?
Re: Build System high level feature proposal Fred Grott 8/4/12 9:49 AM
Xavier,

I would like to point out a certain strategy. It may not work in all use cases.
Because, I knew that there was work on this I redid my own codeqa build stuff. I simply let the IDE build the project before the codeqa build script run.
For me it works as I do not have very complex, multi test projects.

As far as proposal do not attempt to re-invent the wheel look at the Gradle Android plugin and borrow as many ideas as possible. I think the Gradle, groovy as build script, might fit but we never had enough developers bandwidth on the Gradle Android project to keep up with Android Tools changes. As far as dependency management Gradle has not only maven, ivy but also can do file systems like the installed Sdk for example.

There is also a python ant-tools android project but its not as far as the Gradle Android plugin and from my analysis it woudl requrie double the dev time to do it in python as opposed to doing it ing Groovy via Gradle.

The Gradle Android plugin project is here:


Please Xavier take a look and at least borrow the ideas. so you can see the double amount of work using python as oposed to Groovy/gradle the python android build project:


As the python project does not have dependency integration.

There is also Apache Buildr which is ruby build system with dependency integration but its in the same boat as the python project.

Gradle Android plugin would have the nice thing of the build scripts would be understandable Groovy which is understandable to most java programmers.

<span style="font-size:15px;font-family:Arial;vertical-align:baseline;white-s
...
Re: Build System high level feature proposal Manfred Moser 8/4/12 9:54 AM
Hi Xavier,

Thanks for writing up this proposal. I will have a more detailed look and feedback in a week or so  but here are a few first thoughts

- all artifacts should be available in a repository for the dependency management system (libraries open source like android.jar, compat lib .. as well as closed source that are used for Android development like admob .. as well as sdk internal libraries) so they can easily be used by other IDE's and tools as well as during the build process, the repository should use the standard maven repo format and ideally be the central repository so it can be used by ivy, gradle, maven, sbt, clojure and anything else that can do http gets..(http://search.maven.org)

- all aosp internal android apps should use the new build system (dogfooding!!!) and not be built with make

- all example apps should be set up to use it as well

- google example apps like iosched and others should use the new build system

- there should be no dependencies of the build system to some IDE specific project files like currently lint relies on the eclipse directory structure and project files

- ideally it should leverage an existing build system and the associated knowledge and community, it should NOT be a new build system but rather an extension of an existing one (don't fall into the Not Invented Here trap)

manfred


<span style="font-size:15px;font-family:Arial;vertical-align:baseline;white-sp
...
Re: Build System high level feature proposal Jake Wharton 8/4/12 11:51 AM
The #1 problem I see with the design as proposed is that it 
invalidates every Web page and every printed page ever written about 
Android app development that covers how to create projects, where 
files reside, etc.

This should almost never be a limiting factor in software development and definitely not for this endeavor. If nobody ever wanted to break things we be slumming it up in decades-old tools and using far-too-long-in-the-tooth techniques. First understand what ant's benefits were and where it came short (if you like long lists) and then build something from scratch that scales down to the beginner and up to the advancer.

Bump the version number (build system 2 woo!), deprecate but support the old one for 6 months, and ship it. The bookers will book, the webbers will web, and the community will be dragged, kicking and screaming, to a better system for everyone. Good riddance ant(iquated).

Why does the fact that it's new have to mean that it will be more complicated? The base-case right now for a "Hello, World" app can be greatly simplified. I absolutely should not need a build.xml, project.properties, and local.properties to compile an apk that exists as src/, res/, and AndroidManifest.xml. Simply executing android should build with whatever is the target SDK version is set to by default (or the latest if not set) and use the ANDROID_HOME environment variable to find the SDK (if it's not present, prompt and set it).
Re: Build System high level feature proposal Xavier Ducrohet 8/4/12 11:58 AM
Thanks all for your comments.

I'd like to refocus the discussion on the features themselves, not on how we will implement it.

To do that, some details about the implementation:
- As I mentioned a week or so ago on the list: yes we're not going to reinvent the wheel, we want to build on top of an existing build toolkit. Yes we are looking at gradle. We are actually prototyping the build using gradle.
- Yes having a dependency management will mean the android artifacts will have to live in a maven repository somewhere. As I said though, while this is a starting point for a lot of build type, it's not going to help people wanting to use sbt for instance.

Alright now to some more interesting comments.

Mark, you have a very valid points. There is a ton of doc out there that's already out dated. I'm not sure how we can completely deal with this. The current system is broken, and trying to migrate gracefully from it might prevent us from using a better solution (at least with regard to the multiple sourcesets).

One thing is for sure: we do want to have both build systems at the same time before the new one becomes for the default one. During that time we'll have to fully document the new one as well as migrating from one to the other (our current doc is severely lacking in that area I think, and it is critical it gets better). Also the "android create project" will handle both build type. We can even have "android update project" create a new build script using the old project structure.

I hope people who have no (or limited) programming experience will go read the doc (or hire you!). If they don't there isn't much we can do about it.

I'll add that I do know we've been pretty bad at doing changes in the build scripts that woud break people's build. We certainly don't have a good rep around this right now. One of the reason to have this build tested by a lot of devs before it becomes the standard build. We can't screw it up this time.

Richard, the aapt command line has a fairly extensive command line help. You should look at it.
Regarding NDK build we do intend to simply call out to Make (ndk_build actually) passing some parameters that are currently in the makefile to properly integrate. Build native code is non trivial when dealing with multiple (cross compile) toolchains, and toolchain options, etc...


Fred, I'm aware of the gradle plugin (discovered it a couple of weeks ago) but it seems to rely a lot on the Ant build script. We need to start from a clean slate.

Manfred,
Yes we will migrate all the samples to work with this. I do want to clean up a bit the samples. We currently have samples per API level and this is very restrictive to me because samples in older API level don't get updated. I'd be interested in feedback on this too (new thread please).

The aosp internal apps, well that's going to be more of a challenge :) First because, since they are internal they may use internal APIs, which means the build system would have to handle Android APIs differently. Then we have a very different structure for the output, and we have to deal with different build targets (different devices). I'm not sure how we would do this yet.

Your comment about lint is interesting. Yes right now, the command line lint does expect a certain project structure. It's actually not going to work in our case either as we could have different source sets. We need to extend the command line options to make it more flexible for any project structure. Best will be to extend the Java api so that it can be started from the build natively.

Xav
Re: Build System high level feature proposal Xavier Ducrohet 8/4/12 12:04 PM
On Sat, Aug 4, 2012 at 11:51 AM, Jake Wharton <jakew...@gmail.com> wrote:
>>
>> The #1 problem I see with the design as proposed is that it
>> invalidates every Web page and every printed page ever written about
>> Android app development that covers how to create projects, where
>> files reside, etc.
>
>
> This should almost never be a limiting factor in software development and
> definitely not for this endeavor. If nobody ever wanted to break things we
> be slumming it up in decades-old tools and using far-too-long-in-the-tooth
> techniques. First understand what ant's benefits were and where it came
> short (if you like long lists) and then build something from scratch that
> scales down to the beginner and up to the advancer.
>
> Bump the version number (build system 2 woo!), deprecate but support the
> old one for 6 months, and ship it. The bookers will book, the webbers will
> web, and the community will be dragged, kicking and screaming, to a better
> system for everyone. Good riddance ant(iquated).

I agree. We do need to provide as much side support as possible, but
we can't completely limit ourselves because of the old system.

> Why does the fact that it's new have to mean that it will be more
> complicated? The base-case right now for a "Hello, World" app can be greatly
> simplified. I absolutely should not need a build.xml, project.properties,
> and local.properties to compile an apk that exists as src/, res/, and
> AndroidManifest.xml. Simply executing android should build with whatever is
> the target SDK version is set to by default (or the latest if not set) and
> use the ANDROID_HOME environment variable to find the SDK (if it's not
> present, prompt and set it).

that's interesting.
I was already thinking that we should bundle gradle in our sdk so that
it has our plugin natively (having to find and set the dependency on a
custom plugin in build.gradle is too complex), but this would be even
simpler.

The "android" command could act as both build and the equivalent of
"android update project". Bonus point if it detects old-style projects
and convert them automatically.
Re: Build System high level feature proposal Mark Murphy (a Commons Guy) 8/4/12 12:40 PM
On Sat, Aug 4, 2012 at 2:58 PM, Xavier Ducrohet <x...@android.com> wrote:
> One thing is for sure: we do want to have both build systems at the same
> time before the new one becomes for the default one...
> I'll add that I do know we've been pretty bad at doing changes in the build
> scripts that woud break people's build.

I had been assuming that this proposed build system change would
necessitate a hard-slam cutover to a new system, since that has been
the modus operandi for prior changes. If you are planning on having
the two supported in parallel for a significant period of time (e.g.,
Jake's suggested six months), that solves a fair number of problems,
if during the early weeks of that period, the old system is still the
default, but the new system is available. Basically, the community
needs time with a production-ready version of the new system to
evangelize its use, before newcomers to Android start stubbing their
toes on it.
Re: Build System high level feature proposal Dirk Jäckel 8/4/12 1:03 PM


On Saturday, August 4, 2012 at 8:58 PM, Xavier Ducrohet wrote:

> I'd like to refocus the discussion on the features themselves, not on how we will implement it.

1. I would like to have an efficient build system that caches all the intermediate results and only rebuilds whats really necessary. The build system should know the dependencies for every artifact it creates. Not only for the class files but also for dex-files and other generated files. If I use Scala it should extract and dex the Scala libraries class files only once. The build system should only compile/create/copy/modify/filter whats absolutely necessary (and nothing more or less).  

2. It would be great if it does not have to create an APK and install that on the emulator/device but only send the changed files, if there is already an installation on the emulator/device.

This would really speed up development turn around times. Right now I am mostly waiting for my build if I am trying out something new. This would be especially helpful for beginners.

The time it takes from a single line of code change to running the app on an emulator/device is embarrassingly big compared to other popular mobile platforms.

Regards,
Dirk

 
Re: Build System high level feature proposal Xavier Ducrohet 8/4/12 2:00 PM
Mark,

yes of course. I should have mentioned this too.

Step 0 is spec'ing the features here, and start of implementation.
Step 1 is people testing it as a custom download. People here, and
whoever else want to give feedback. Hopefully we can do quick
iterations.

Once we have something robust with most APIs done, we can move to:
Step 2a: package it in the SDK but this is not the default build
Step 2b: package it and make it the default, but keep Ant

Step 3: remove Ant.

Step 2 will take as long as it has to, and as long as Ant can stay viable.
Re: Build System high level feature proposal Xavier Ducrohet 8/4/12 2:07 PM
On Sat, Aug 4, 2012 at 1:03 PM, Dirk Jäckel <dirk.j...@gmail.com> wrote:
> 1. I would like to have an efficient build system that caches all the intermediate results and only rebuilds whats really necessary. The build system should know the dependencies for every artifact it creates. Not only for the class files but also for dex-files and other generated files. If I use Scala it should extract and dex the Scala libraries class files only once. The build system should only compile/create/copy/modify/filter whats absolutely necessary (and nothing more or less).

We have part of this already in both Ant and Eclipse, but yes dex is
the missing part. We are working on this, and we'll have it in
Ant/Eclipse too.

> 2. It would be great if it does not have to create an APK and install that on the emulator/device but only send the changed files, if there is already an installation on the emulator/device.
>
> This would really speed up development turn around times. Right now I am mostly waiting for my build if I am trying out something new. This would be especially helpful for beginners.
>
> The time it takes from a single line of code change to running the app on an emulator/device is embarrassingly big compared to other popular mobile platforms.

Yes. Now that Google Play does partial update we are going to look at
reusing this mechanism as well. It might be a bit tricky because you
can have any random version of your app installed but that should be
possible anyway.

Xav
Re: Build System high level feature proposal Mark Murphy (a Commons Guy) 8/4/12 2:08 PM
On Sat, Aug 4, 2012 at 5:00 PM, Xavier Ducrohet <x...@android.com> wrote:
> yes of course. I should have mentioned this too.
>
> Step 0 is spec'ing the features here, and start of implementation.
> Step 1 is people testing it as a custom download. People here, and
> whoever else want to give feedback. Hopefully we can do quick
> iterations.
>
> Once we have something robust with most APIs done, we can move to:
> Step 2a: package it in the SDK but this is not the default build
> Step 2b: package it and make it the default, but keep Ant
>
> Step 3: remove Ant.
>
> Step 2 will take as long as it has to, and as long as Ant can stay viable.

That all sounds great!
Re: Build System high level feature proposal Joe Bowbeer 8/4/12 2:33 PM
Xavier,

I generally like the proposed features.

The build variants (combinations of Product Flavor and Build Type) are similar to features that I've had to build on top of the existing build system.  In one product, before library projects were introduced, we had a main project and five differently-branded project flavors.  When library projects were introduced, the main project morphed into a library project -- but it is really a main project by nature, and this should be easy to express with your new feature set.  In another product, there are 100's of flavors that are only known at build time, so I'm hoping that the new system will support dynamic flavors.  More on this below.  I've never needed more than a handful of build types, by the way, so what you propose seems fine.

Questions, comments, suggestions:

1. What about sharing of portable Java code?

In the past, I've gained a lot of leverage by splitting most of the logic into a platform-independent Java project that could be tested with JUnit, and even run from the command line with very rapid turnaround.  In the first product described above, this was implemented as a Java project whose source was included by the main Android project, plus another Java project that added the command-line interface.

2. Where is the libs folder?  Where do the 3rd-party .jars go?

This is related to question #1 about leveraging portable Java code.  The most common way to do this is to add the .jars to the libs folder, but I don't see this folder in your outline.  Where are the libs?  Are there separate libs per flavor, or what?

3. Packing the main project as a library for 3rd-party flavoring.

I need to package the main project in some form (library blob?) so that I can distribute it to 3rd parties for added flavoring and repackaging.  Will this be supported?

4. Dynamic flavors

This was mentioned in the first paragraph.  I need to be able to create flavors dynamically during the build.  I'm imaging a hook or step in the build process that will let me inject a custom flavor into the build.  I don't care if this ability is present in the IDE, because IDE builds are for developer use only and are not distributed to others, but it is essential for the batch build system that creates the "real" builds that are tested and distributed.

5. Native code?

I'm also concerned, as are others, about integration with the jni code and its build system.  Native code has become increasingly important in a larger and larger set of applications.

Currently, in a cobbled-together build script that I am familiar with, the build variant information is passed to the NDK make file using environment variables (which seems to be the only way to do this), and the jni is rebuilt for every different build variant.  That is, it is rebuilt from scratch every time the build variant changes.  The Android.mk file converts the build variant environment variables into a set of #defines that result in conditional compilation of large sections of code throughout the jni layer.  This raises a question about where this native code will live in the new build layout, and how the build variant information is to be conveyed to the jni build.

6. *the* IDE?

I'm concerned about the reference to "the" IDE.  Also the mention of an ide.jar.  (Which ide.jar?)

As my projects have become more complex, as is the nature of real applications, the Eclipse ADT has become more and more a PITA.  I strongly urge some IDE-agnosticism, for everyone's sake.  Currently, all major IDE's (Eclipse, IDEA, NetBeans) have solid support for both Ant and Maven, so it makes sense for the new build system to leverage these tools.

I'm currently a fan of the NetBeans Android plugin because its project model is exactly the Ant model represented by the AndroidManifest and the .properties files.  NetBeans is the only IDE that has been able to quickly and reliably accommodate the many changes in Android's build scripts and tools starting with r14.  Granted, NetBeans, my choice, is not the most popular IDE, but its approach demonstrates the power and economy of leveraging the standards -- and keeping things simple in the face of rapid change.

[ Aside: What is the most popular IDE among serious Android developers?  I would venture that it is now IDEA? ]

I should mention that I'm not a user of the ADT layout editor or other fancier UI tools.  And I don't know a single developer of a commercial Android app who uses these GUI tools.  I think these editors are valuable as an easy-on-ramp for beginners, but then they quickly lose their appeal.  I hope, of course, that this will change, because I believe in the power of a good IDE.  However, I think the development of layout editors and great RAD tools is probably something that 3rd parties can do better than Google -- if they have the hooks/api/stability to enable it.

I should also mention that I'm still waiting for source-level native code debugging that actually works in practice.  (That would be cool...)

7. Maven: +1

I would likely be using a Maven Android plugin if Maven were supported by the Android Tools team.  Unless the Android Tools team supports Maven, however, I will continue to avoid it for commercial apk development, because I can't risk the version lag, etc.

8. Lint not tied to IDE

I second this recommendation from one respondent.  Please make Lint IDE-agnostic.

--Joe


On Friday, August 3, 2012 5:36:15 PM UTC-7, Xavier Ducrohet wrote:
  • src
    • main
      • <li style="list-style-type:square;f
...
Re: Build System high level feature proposal James Moore 8/4/12 4:01 PM
It looks like this provides the right hooks we need to improve Scala
integration in Eclipse.  However, it'd be great to see some sort of
preshrink/predex built into the build system, and not just for Scala.

The Scala problem is the 64k method reference limit issue (lots of
links at http://blog.restphone.com/2012/07/scala-on-android-what-you-need-to-know.html).
 Briefly, the Scala standard libraries are too large to fit on
Android.  It's not at all Scala-specific though; I suspect that more
and more large projects are going to hit that limit.

Traditional Scala development gets around the problem by running
Proguard on every build.  Slow and painful, but it works.  I write an
Eclipse plugin to speed this up (it caches Proguarded jars and uses
the one that it thinks matches the current build, so you end up with
far fewer Proguard runs).  Hooks for this built into the build system
you're talking about would improve the user experience - in
particular, I'd definitely use the hook that says "use $SOME_JAR for
code completion, but replace it with
$SOME_JAR_THATS_BEEN_RUN_THROUGH_SOMETHING_LIKE_PROGUARD for dexing".

In a perfect world I'd love for my plugin to be put out of business by
the new build system.  At a high level, I'd just like the build system
to solve the 64k limit for me without me even knowing about it.

Maybe:

Detect that a jar file contains more than 64k references, then dex it
separately.  Then do something like generate intermediate classes that
use a classloader that knows where these predexed jars live, and
transparently rewrite things like Android.xml to start up
activities/services/etc with the intermediate classes.

That might speed things up considerably for all large projects; if
most of your jars only need to be dexed once, the change/build/deploy
cycle might see a big speed boost.

FYI, The Scala plugin is available at
https://github.com/banshee/AndroidProguardScala.

--
James Moore
ja...@restphone.com
http://blog.restphone.com/
http://www.linkedin.com/in/jamesmmooreiv
Re: Build System high level feature proposal Xavier Ducrohet 8/4/12 5:08 PM
On Sat, Aug 4, 2012 at 2:33 PM, Joe Bowbeer <joe.b...@gmail.com> wrote:
> Xavier,
>
> I generally like the proposed features.
>
> The build variants (combinations of Product Flavor and Build Type) are
> similar to features that I've had to build on top of the existing build
> system.  In one product, before library projects were introduced, we had a
> main project and five differently-branded project flavors.  When library
> projects were introduced, the main project morphed into a library project --
> but it is really a main project by nature, and this should be easy to
> express with your new feature set.  In another product, there are 100's of
> flavors that are only known at build time, so I'm hoping that the new system
> will support dynamic flavors.  More on this below.  I've never needed more
> than a handful of build types, by the way, so what you propose seems fine.

Can you describe how the flavors are known at build time? How are they
described?
If there is some file+metadata that describe them, it should be fairly
easy to put some logic in the build script that process this data and
dynamically create new flavors.

> Questions, comments, suggestions:
>
> 1. What about sharing of portable Java code?
>
> In the past, I've gained a lot of leverage by splitting most of the logic
> into a platform-independent Java project that could be tested with JUnit,
> and even run from the command line with very rapid turnaround.  In the first
> product described above, this was implemented as a Java project whose source
> was included by the main Android project, plus another Java project that
> added the command-line interface.

Yes this would be supported. Multi-projects build will allow this, or
you can create a jar from it and reference it.

> 2. Where is the libs folder?  Where do the 3rd-party .jars go?
>
> This is related to question #1 about leveraging portable Java code.  The
> most common way to do this is to add the .jars to the libs folder, but I
> don't see this folder in your outline.  Where are the libs?  Are there
> separate libs per flavor, or what?

First you should really reference dependency using Maven or Ivy if
this is a 3rd party library.
If this is your own, you could create a dependency on a locale file.
It's up to you to put it where you want.
With gradle we aren't as tied as libs/ are we were with Ant.

> 3. Packing the main project as a library for 3rd-party flavoring.
>
> I need to package the main project in some form (library blob?) so that I
> can distribute it to 3rd parties for added flavoring and repackaging.  Will
> this be supported?

I think this would have to be a library. The added flavoring can come
from a project using this library.

> 4. Dynamic flavors
>
> This was mentioned in the first paragraph.  I need to be able to create
> flavors dynamically during the build.  I'm imaging a hook or step in the
> build process that will let me inject a custom flavor into the build.  I
> don't care if this ability is present in the IDE, because IDE builds are for
> developer use only and are not distributed to others, but it is essential
> for the batch build system that creates the "real" builds that are tested
> and distributed.

Yes. As I mentioned above this should be easy.

> 5. Native code?
>
> I'm also concerned, as are others, about integration with the jni code and
> its build system.  Native code has become increasingly important in a larger
> and larger set of applications.
>
> Currently, in a cobbled-together build script that I am familiar with, the
> build variant information is passed to the NDK make file using environment
> variables (which seems to be the only way to do this), and the jni is
> rebuilt for every different build variant.  That is, it is rebuilt from
> scratch every time the build variant changes.  The Android.mk file converts
> the build variant environment variables into a set of #defines that result
> in conditional compilation of large sections of code throughout the jni
> layer.  This raises a question about where this native code will live in the
> new build layout, and how the build variant information is to be conveyed to
> the jni build.

This is interesting.
Right now the ndk relies on its own build system because of the
complexity of building native code and dealing with several
toolchains.

We intend to keep this and have the build system call out to this.
One of the param is whether you want debug code or not. For instance
you can call "ndk_builld DEBUG=1" for a debug build. This will be done
automatically if the build type indicates that we want the jni code to
be debuggable.

We will have to extends the params we can pass to ndk_build. For
instance output location is one param we'll have to add.
Also ndk_build does parse the manifest to figure out some info
(minSdkVersion at least) but with the build variants we'll have to
pass it instead.

I'm really interested in knowing what kind of data you want to be able
to pass. I'm guessing at least #define values would be valuable.
Anything else?

From the flavor perspective this could be set similarly to the custom
fields set in BuildConfig.

> 6. *the* IDE?
>
> I'm concerned about the reference to "the" IDE.  Also the mention of an
> ide.jar.  (Which ide.jar?)
>
> As my projects have become more complex, as is the nature of real
> applications, the Eclipse ADT has become more and more a PITA.  I strongly
> urge some IDE-agnosticism, for everyone's sake.  Currently, all major IDE's
> (Eclipse, IDEA, NetBeans) have solid support for both Ant and Maven, so it
> makes sense for the new build system to leverage these tools.
>
> I'm currently a fan of the NetBeans Android plugin because its project model
> is exactly the Ant model represented by the AndroidManifest and the
> .properties files.  NetBeans is the only IDE that has been able to quickly
> and reliably accommodate the many changes in Android's build scripts and
> tools starting with r14.  Granted, NetBeans, my choice, is not the most
> popular IDE, but its approach demonstrates the power and economy of
> leveraging the standards -- and keeping things simple in the face of rapid
> change.

This is both tricky and critical. NetBeans has the advantage that it
relies solely on Ant to descrive the project. any change in there is
automatically supported.

Ultimately we want the same. This is the only way to make the new
build system viable. It has to be 100% supported in the ide.

We'll have to see how we can make this work in IntelliJ and Eclipse.
>> AndroidManifest.xml
>>
>> debug
>>
>> … (same as main)
>>
>> release
>>
>> … (same as main)
>>
>> flavor1
>>
>> … (same as main)
>>
>> flavor2
>>
>> … (same as main)
>>
>>
>> The current gen folder is to move under the project output (build folder),
>> and follow a similar pattern. This is accompanied by the generated
>> resources, and the generated (merged) Manifest.
>>
>> This is of course be completely customizable, but this shows the default,
>> and the type of inputs that are available to tasks.
>>
>> BuildConfig
>>
>> BuildConfig is an automatically generated class with a single boolean
>> DEBUG, set to true in debug builds only.
>>
>> With custom source sets for both Build Types and Product Flavors, the
>> build system covers adding extra flags and triggers (through a user created
>> class) to control what the code does in each build variants. It is a manual
>> process to maintain each version of the same class but it is the most
>> flexible for devs who can do anything they want.
>>
>> The new build system still makes use of BuildConfig as it is an automatic
>> class creation, and extends its capability.
>> It provides developers with the ability to programmatically add new items
>> in BuildConfig based on logic defined in the build script.
>> Details TBD.
>>
>> Testing
>>
>> Test Application
>>
>> In the current build system, test applications are created from a
>> different project that is flagged as a “test project”. This project
>> references the main project in order to get the main project’s classes on
>> its classpath.
>>
>> With possibly multiple Product Flavors it is important to test all of
>> them.
>>
>> Some Java build systems have introduced having the test code directly in
>> the project under src/test/ and the new build system follows this
>> convention.
>>
>> Since JUnit test cannot be run against Android code, the build system
>> reuses this concept for test applications.
>> There is a mirror of the main configuration, plus flavors which allows
>> creating flavor specific test applications.
>> The main test configuration uses src/test, and flavors uses
>> src/test<flavor>
>>
>> The manifest of a test application is pretty basic and it is automatically
>> created it. The build system provides the ability to configure the test
>> package name and the InstrumentationTestRunner class.
>>
>> Deployment and running all test flavors TBD.
>>
>> JUnit tests
>>
>> JUnit tests cannot be run against code accessing Android APIs.
>> Instrumentation tests are the sanctioned way to go.
>> To have code that does not access Android APIs tested with JUnit tests, a
>> multi-project build should be setup with the code to be tested put in a
>> Java-only project.
>>
>> Note: The build system shouldn’t prevent usage of some third party test
>> frameworks such as Roboelectric. It should actually facilitate integrating
>> those. Details TBD.
>>
>> aapt / dex / other options.
>>
>> The build needs to expose all available options and to set them globally
>> or per build variant.
>> Details TBD.
>>
>> Building Library Projects.
>>
>> Library Projects are built in a very similar way to regular projects with
>> a few exceptions.
>>
>> There are no Product Flavor. This would not make sense as the output is
>> not a Product. Different versions of the same Library should be different
>> libraries and use different projects (possibly depending on the same root
>> Library).
>>
>> There are debug and release build types but they are used slightly
>> differently.
>> Testing the library is important and this is done through the test
>> component of the project. However, unlike a normal project, this component
>> doesn’t generate a separate application from only its source set. Instead it
>> implicitly references the library itself and packages it in its own apk.
>> This build is always a debug build.
>>
>> When the Library Project is built and packaged into a distribution blob,
>> then the build is always a release build.
>>
>> Note that the Build Type can here only configure the following properties:
>>
>> native compilation debug flag
>> Buildconfig
>>
>> DEBUG flag
>> other custom properties?
>>
>>
>> The other properties either don’t apply, or only one possible value makes
>> sense.
>>
>> Libraries must be packaged in a very specific way, no matter what the
>> library source folder structure is.
>>
>> classes.jar -- this is the main code used for compile and packaging.
>> api.jar -- this is an optional jar to used at compile time instead of
>> classes.jar
>> res/ -- this is the res folder(s) extended with:
>>
>> crunched images
>> renderscript bitcode
>>
>> AndroidManifest.xml (includes the merging from other Library Projects?)
>> lib/ -- this is the jni component(s)
>> aidl -- this is a straight copy of all the src/../aidl files
>>
>>
>> In all the above cases what is packaged can be extended by the release
>> Build Type, similar to a normal apk.
>>
>> Additionally, the blob may contain:
>>
>> proguard.txt -- custom Proguard rules
>> lint.jar -- custom lint rules
>> ide.jar -- custom IDE extensions
>> docs/ -- this is the javadoc
>> source/ -- this is optional source package (for open source libraries).
>>
>>
>> Using Library Projects.
>>
>> Library projects can be used two ways:
>>
>> Dependency through a repository (local/maven/ivy)
>>
>> this downloads the blob and unarchives it somewhere
>>
>> As part of a multi-project build.
>>
>>
>> #1 is fairly standard and can be added as as normal dependency.
>> #2 is a bit more complicated and needs to be figured out. The main issue
>> is that dependency on Library Projects have a clear dependency order and the
>> setup of the dependency must clearly spell this out.
>>
>> Looking at IDE integration, #2 is important for cases where Library
>> Projects are not distributed but are instead just a way to modularize an
>> application. This would allow better experience in the IDE (for instance
>> allowing refactoring across the main app and the library(ies.))
>> Details TBD.
>>
>> Dependencies
>>
>> There are 2 main types of dependency configurations:
>>
>> compile
>> package
>>
>>
>> While package will in most case include the content of compile, it is
>> important to be able to have a dependency that is compile only. This allows
>> having a different version of a dependency for compilation vs. packaging.
>> Details TBD.
>> It should also be possible to have a dependency skip the Proguard step.
>>
>> Note that the Android API jar(s) is automatically added to the compile
>> (only) dependency list, and doesn’t have to be declared manually.
>>
>> Product Flavors (not Build Type) impact dependencies, as well as tests.
>> This means other dependency configurations are automatically created:
>>
>> <flavor>Compile
>> <flavor>Package
>> testCompile
>> testPackage
>> test<flavor>Compile
>> test<flavor>Package
>>
>> Lint Integration
>>
>> The new build system integrates Lint at two levels.
>>
>> Running Lint
>>
>> A Lint tasks is created and allows testing any build variant.
>> Lint errors will break the build.
>> Lint rules can be enabled/disabled and set priority (warning/error).
>> HTML and/or XML reports can be generated.
>> Lint should run on all available Build Variants but remove duplicates
>> (since code common to all variants is compiled for each of them)
>>
>> Extending Lint
>>
>> Creating custom Lint is directly possible using the src/lint/ folder.
>>
>> This creates a lint.jar file that is passed to Lint when running it on the
>> project itself. The dependency on the Lint API is done automatically. All
>> that’s needed is creating the source folder.
>>
>> If the project depends on Library Projects, their own custom rules are
>> used as well.
>>
>>
>> Feedback welcome. thanks.
>> Xav
>> --
>> Xavier Ducrohet
>> Android SDK Tech Lead
>> Google Inc.
>> http://developer.android.com | http://tools.android.com
>>
>> Please do not send me questions directly. Thanks!
>
>
>> AndroidManifest.xml
>>
>> debug
>>
>> … (same as main)
>>
>> release
>>
>> … (same as main)
>>
>> flavor1
>>
>> … (same as main)
>>
>> flavor2
>>
>> … (same as main)
>>
>>
>> The current gen folder is to move under the project output (build folder),
>> and follow a similar pattern. This is accompanied by the generated
>> resources, and the generated (merged) Manifest.
>>
>> This is of course be completely customizable, but this shows the default,
>> and the type of inputs that are available to tasks.
>>
>> BuildConfig
>>
>> BuildConfig is an automatically generated class with a single boolean
>> DEBUG, set to true in debug builds only.
>>
>> With custom source sets for both Build Types and Product Flavors, the
>> build system covers adding extra flags and triggers (through a user created
>> class) to control what the code does in each build variants. It is a manual
>> process to maintain each version of the same class but it is the most
>> flexible for devs who can do anything they want.
>>
>> The new build system still makes use of BuildConfig as it is an automatic
>> class creation, and extends its capability.
>> It provides developers with the ability to programmatically add new items
>> in BuildConfig based on logic defined in the build script.
>> Details TBD.
>>
>> Testing
>>
>> Test Application
>>
>> In the current build system, test applications are created from a
>> different project that is flagged as a “test project”. This project
>> references the main project in order to get the main project’s classes on
>> its classpath.
>>
>> With possibly multiple Product Flavors it is important to test all of
>> them.
>>
>> Some Java build systems have introduced having the test code directly in
>> the project under src/test/ and the new build system follows this
>> convention.
>>
>> Since JUnit test cannot be run against Android code, the build system
>> reuses this concept for test applications.
>> There is a mirror of the main configuration, plus flavors which allows
>> creating flavor specific test applications.
>> The main test configuration uses src/test, and flavors uses
>> src/test<flavor>
>>
>> The manifest of a test application is pretty basic and it is automatically
>> created it. The build system provides the ability to configure the test
>> package name and the InstrumentationTestRunner class.
>>
>> Deployment and running all test flavors TBD.
>>
>> JUnit tests
>>
>> JUnit tests cannot be run against code accessing Android APIs.
>> Instrumentation tests are the sanctioned way to go.
>> To have code that does not access Android APIs tested with JUnit tests, a
>> multi-project build should be setup with the code to be tested put in a
>> Java-only project.
>>
>> Note: The build system shouldn’t prevent usage of some third party test
>> frameworks such as Roboelectric. It should actually facilitate integrating
>> those. Details TBD.
>>
>> aapt / dex / other options.
>>
>> The build needs to expose all available options and to set them globally
>> or per build variant.
>> Details TBD.
>>
>> Building Library Projects.
>>
>> Library Projects are built in a very similar way to regular projects with
>> a few exceptions.
>>
>> There are no Product Flavor. This would not make sense as the output is
>> not a Product. Different versions of the same Library should be different
>> libraries and use different projects (possibly depending on the same root
>> Library).
>>
>> There are debug and release build types but they are used slightly
>> differently.
>> Testing the library is important and this is done through the test
>> component of the project. However, unlike a normal project, this component
>> doesn’t generate a separate application from only its source set. Instead it
>> implicitly references the library itself and packages it in its own apk.
>> This build is always a debug build.
>>
>> When the Library Project is built and packaged into a distribution blob,
>> then the build is always a release build.
>>
>> Note that the Build Type can here only configure the following properties:
>>
>> native compilation debug flag
>> Buildconfig
>>
>> DEBUG flag
>> other custom properties?
>>
>>
>> The other properties either don’t apply, or only one possible value makes
>> sense.
>>
>> Libraries must be packaged in a very specific way, no matter what the
>> library source folder structure is.
>>
>> classes.jar -- this is the main code used for compile and packaging.
>> api.jar -- this is an optional jar to used at compile time instead of
>> classes.jar
>> res/ -- this is the res folder(s) extended with:
>>
>> crunched images
>> renderscript bitcode
>>
>> AndroidManifest.xml (includes the merging from other Library Projects?)
>> lib/ -- this is the jni component(s)
>> aidl -- this is a straight copy of all the src/../aidl files
>>
>>
>> In all the above cases what is packaged can be extended by the release
>> Build Type, similar to a normal apk.
>>
>> Additionally, the blob may contain:
>>
>> proguard.txt -- custom Proguard rules
>> lint.jar -- custom lint rules
>> ide.jar -- custom IDE extensions
>> docs/ -- this is the javadoc
>> source/ -- this is optional source package (for open source libraries).
>>
>>
>> Using Library Projects.
>>
>> Library projects can be used two ways:
>>
>> Dependency through a repository (local/maven/ivy)
>>
>> this downloads the blob and unarchives it somewhere
>>
>> As part of a multi-project build.
>>
>>
>> #1 is fairly standard and can be added as as normal dependency.
>> #2 is a bit more complicated and needs to be figured out. The main issue
>> is that dependency on Library Projects have a clear dependency order and the
>> setup of the depend...
Re: Build System high level feature proposal Xavier Ducrohet 8/4/12 5:14 PM
The first step should be doable easily:

- in the debug build type, enable proguard with some rules specific to
what you need (I guess: remove unused method/fields but don't
obfuscate)
- set a dependency on what you need to be "compile" but not "package"
- set a different dependency for the runtime, but make it skip proguard.

Note that in gradle you can hook into any task to add work before or
after the task happen.
You can also rework the input / output of each task.

For step two (automatically deal with case where the 64K limit is
reached), this could be done as a 3rd party plugin that sits on top of
the main plugin at least at the beginning. We could integrate it in
the main plug-in if it works well

By this I mean this is probably out of the scope of the first version,
but if someone wants to work on it as we build the new build system,
we can make sure the hooks are in it.

Xav
Re: Build System high level feature proposal Xavier Ducrohet 8/4/12 5:42 PM
A bit more info about the jni part. I have to admit we didn't really
think a whole lot about this.

The first draft kind of implied that each flavor/built type would
generate its own lib<blah>.so but I don't think this is very flexible
now that I think about it.

I think it would make more sense to have 2 folders under ../jni/. One
for the source and one for headers.

Therefore, building a given build variant would give you 3 source folders:
src/main/jni/src/
src/myflavor/jni/src/
src/debug/jni/src/

and three headers folders:
src/main/jni/include/
src/myflavor/jni/include/
src/debug/jni/include/

This would generate a single lib<blah>.so for the build variant.

This gives maximum flexibility in configuring the flavor using custom
headers and even custom code.

thoughts?
Xav
--
Xavier Ducrohet
Android SDK Tech Lead
Google Inc.
http://developer.android.com | http://tools.android.com

Please do not send me questions directly. Thanks!
Re: Build System high level feature proposal Joe Bowbeer 8/4/12 6:14 PM


On Saturday, August 4, 2012 5:08:24 PM UTC-7, Xavier Ducrohet wrote:
On Sat, Aug 4, 2012 at 2:33 PM, Joe Bowbeer wrote:
> Xavier,
>
> I generally like the proposed features.
>
> The build variants (combinations of Product Flavor and Build Type) are
> similar to features that I've had to build on top of the existing build
> system.  In one product, before library projects were introduced, we had a
> main project and five differently-branded project flavors.  When library
> projects were introduced, the main project morphed into a library project --
> but it is really a main project by nature, and this should be easy to
> express with your new feature set.  In another product, there are 100's of
> flavors that are only known at build time, so I'm hoping that the new system
> will support dynamic flavors.  More on this below.  I've never needed more
> than a handful of build types, by the way, so what you propose seems fine.

Can you describe how the flavors are known at build time? How are they
described?
If there is some file+metadata that describe them, it should be fairly
easy to put some logic in the build script that process this data and
dynamically create new flavors.

In this case, a custom flavor consists almost entirely of resources (and assets, too, if possible) that are selected according to the requirements of the build, as specified in command-line arguments, or in a properties file or json response or whatever.  The next steps are to create the flavor on disk, by invoking a custom script, and then to build the product, which presumably entails creating a build variant that selects this newly created flavor.  It is the "build the product" step where I would like the ability to register the new flavor, create the new build variant, and do whatever else is needed in order to build with this new flavor.


> 5. Native code?
>
> I'm also concerned, as are others, about integration with the jni code and
> its build system.  Native code has become increasingly important in a larger
> and larger set of applications.
>
> Currently, in a cobbled-together build script that I am familiar with, the
> build variant information is passed to the NDK make file using environment
> variables (which seems to be the only way to do this), and the jni is
> rebuilt for every different build variant.  That is, it is rebuilt from
> scratch every time the build variant changes.  The Android.mk file converts
> the build variant environment variables into a set of #defines that result
> in conditional compilation of large sections of code throughout the jni
> layer.  This raises a question about where this native code will live in the
> new build layout, and how the build variant information is to be conveyed to
> the jni build.

This is interesting.
Right now the ndk relies on its own build system because of the
complexity of building native code and dealing with several
toolchains.

We intend to keep this and have the build system call out to this.
One of the param is whether you want debug code or not. For instance
you can call "ndk_builld DEBUG=1" for a debug build. This will be done
automatically if the build type indicates that we want the jni code to
be debuggable.

We will have to extends the params we can pass to ndk_build. For
instance output location is one param we'll have to add.
Also ndk_build does parse the manifest to figure out some info
(minSdkVersion at least) but with the build variants we'll have to
pass it instead.

I'm really interested in knowing what kind of data you want to be able
to pass. I'm guessing at least #define values would be valuable.
Anything else?

From the flavor perspective this could be set similarly to the custom
fields set in BuildConfig.

At the least, we need the Product Flavor and Build Type.  Most everything else can be derived from those.

A simple mechanism for passing properties (#define values) to the NDK build would be great.

--Joe
Re: Build System high level feature proposal Joe Bowbeer 8/4/12 6:21 PM
I'm liking the jni structure you describe below.
Re: Build System high level feature proposal Xavier Ducrohet 8/4/12 11:27 PM
I'm still a bit blurry on your actual use case but let me give you a
few more details and then you can tell me if that makes sense to you.

Since we are looking at gradle we have the ability in our plug-in to
extend the DSL that gradle provides.

To declare a new flavor you'd do something like:

flavor {
   myflavor {
      prop1 = value;
      ...
   }
   (repeat for other flavor)
}

Now by default this will use src/myflavor/... where you'll have a res
folder, an assets folder, etc...
You could go an customize the location of each of those if needed with
something like

sourcesets {
   myflavor {
      assetsDir = <new path>
   }
}

Now if you have 100s of flavor you probably wouldn't want to do this
manually. We can make sure that you could use an API to dynamically do
the same.

You could for instance look at all the folders in a given root folder
and for each create a flavor, possibly reading some property file to
setup minSdkVersion, package name, etc..

Note that flavors don't exist on disk. It's purely a configuration in
the build script. They do use folders on disk but without the
configuration in the build script the folders go unused. The
configuration merely creates a new build tasks (build target in Ant
lingo) to let you build this flavor using the parameters defined in
the configuration.

However, do realize that this would be execute every time gradle is
running, in the configuration phase, as new flavors add new build
tasks.
So running "gradle build<flavor42>Debug will always parse all the data
needed to generate the 100 flavors.

Depending on what is required to dynamically create those flavors we
might want  to see if we can control the configuration phase to
disable this step when it's not needed. i'm not sure yet but I'll ask
the gradle guys.


Unrelated to this: I just remember I forgot to answer about ide.jar
This is a secondary jar, like lint. We have APIs (which aren't really
final at this stage) to let developers create custom interactivty in
the layout editors for custom views.

The idea here is that if you have src/ide, this would automatically
get compiled against those APIs and create a jar file (ide.jar) that
gets automatically loaded in the IDE to extend its editors. If your
project depend on a Library Project that has a custom view this would
get used automatically as well so that you get a good experience in
the IDE when using the views provided by the library.

And as we add features to the IDE, we'll see if we can extend it in
other places that make sense too.
Re: Build System high level feature proposal Tor Norbye 8/5/12 8:11 AM
"- there should be no dependencies of the build system to some IDE specific project files like currently lint relies on the eclipse directory structure and project files"

The idea was that there would be a lint driver for each build system; an ant task to run it for ant, a maven task to run it for maven etc, and there is a Java API that these tasks would use to properly initialize source paths and class paths.  However, to make lint more convenient to use in the mean time, the lint command line driver tries to just find things on its own. If it finds a .classpath file it will use it, but it doesn't require it; it will look for known folder locations to handle not just the ADT directory structure but the AOSP one as well. Just a couple of days ago I also made it work for the default Maven structure -- and I also added a couple of command line flags which you can use to tell it to look for sources and/or classes in other locations, if your build is doing something unusual, so in ADT 21 this will hopefully be easier to do. https://android-review.googlesource.com/#/c/40234/

-- Tor


On Sat, Aug 4, 2012 at 9:54 AM, Manfred Moser <mos...@gmail.com> wrote:
Hi Xavier,

Thanks for writing up this proposal. I will have a more detailed look and feedback in a week or so  but here are a few first thoughts

- all artifacts should be available in a repository for the dependency management system (libraries open source like android.jar, compat lib .. as well as closed source that are used for Android development like admob .. as well as sdk internal libraries) so they can easily be used by other IDE's and tools as well as during the build process, the repository should use the standard maven repo format and ideally be the central repository so it can be used by ivy, gradle, maven, sbt, clojure and anything else that can do http gets..(http://search.maven.org)

- all aosp internal android apps should use the new build system (dogfooding!!!) and not be built with make

- all example apps should be set up to use it as well

- google example apps like iosched and others should use the new build system

- there should be no dependencies of the build system to some IDE specific project files like currently lint relies on the eclipse directory structure and project files

- ideally it should leverage an existing build system and the associated knowledge and community, it should NOT be a new build system but rather an extension of an existing one (don't fall into the Not Invented Here trap)

manfred


On Aug 3, 2012 5:36 PM, "Xavier Ducrohet" <x...@android.com> wrote:
As I've mentioned before, we are looking are a new version of the build system provided in the SDK.

Our goal is to make it easy for beginners but flexible and powerful enough for teams working on large complex application(s).

We started looking at the high level features that the build system would offer. This is independent from the implementation of the build system and how those features would be exposed to devs (this will come later). The following is a proposal for these features (not including the basic features currently available, those are a given), and we are looking for feedback.

So what are we trying to do?

  • Better extensibility
    • Allow devs to configure every steps.
      • expose all options for aapt, dex, …
      • allow changing input/output for all steps.
        • For instance, bypass proguard for a given jar file.
        • Use different jar for compile vs. package
    • Better control over debug vs. release builds
      • manifest info
      • BuildConfig
      • jni compilation
      • different res/code depending on build.
    • Let devs add new logic in the build system
      • New tasks
      • Change dependencies
      • Integrate into other build systems.
  • Help devs making different versions of the same app.
    • Library Projects can be cumbersome to make 2 apps that are 99% identical.
    • Different versions for different app store could only differ in package name and signing key.
  • Better dependency management for 3rd party jar files and Library Projects
    • Use Maven/Ivy for dependency management
    • Depend on Library Projects distributed as binary blob
    • Generate Library Projects blob


To implement this, we are introducing new build concepts.

Product Flavors

A product flavor defines a customized version of the application build by the project. A single project can have different flavors which change the generated application.

Although different flavors could be very different applications, using Library Projects is a better use case for this, and the build system still supports this.
This new concept is designed to help when the differences are very, very minimum, for instance when using Google Play’s multi-apk support (e.g. the difference could be only gl textures compressed using a different format.)
If the answer to “Is this the same application?” is yes, then this is probably the way to go over Library Projects.

Product flavors can customize the following properties:
  • minSdkVersion
  • targetSdkVersion
  • versionCode
  • versionName
  • package name (overrides value from manifest)
  • release signing info (keystore, key alias, passwords,...).
  • native abi filter
  • test info
    • package name for test app (optional, default is <base>.test)
    • InstrumentationTestRunner class (optional)

Additionally, Product Flavor can provide their own source code, resources and manifest.

A project starts with a main product configuration. All the above properties can be set and it has its own source code, resources and manifest.
If the project does not create product flavors, then the application is built from this main configuration.

When a project declares Product Flavors, these extends the main configuration.

The source code of the product flavor is used in addition to the source code of the main configuration (1).
The resources of the product flavor overrides the resources of the main configuration.
The Manifest of the product flavor is merged on top of the Manifest of the main configuration.

(1) This allows some flexibility but also adds some restrictions:
  • Classes in the main configuration and in the product flavors can reference each other. All the source sets are used to generate a single output (instead of each generating its own output with a one-way dependency).
  • Classes exposed by product flavors and referenced by classes in the main configuration must be present in all product flavors, and must have the same API for all builds to succeed.

When a Product Flavor is defined, the main configuration is not available as a buildable product. To create more than one product, two or more flavors must be created and customized.

Build Types

A build type allows configuration of how an application is packaged for debugging or release purpose.

This concept is not meant to be used to create different versions of the same application. This is orthogonal to Product Flavor.

A Build Type provides configuration for the following build properties:
  • manifest debuggable flag
  • native compilation debug flag
  • proguard enabled + rules
  • debug signing flag (ie whether to use debug key or release key)
  • package name suffix (2)
  • Buildconfig
    • DEBUG flag
    • other custom properties?

By default the build system provides two build types, debug and release, which can be reconfigured. New types can also be created to provide any combinations of the above properties.
For instance, one could make a Build Type that uses Proguard but creates a debuggable apk.

(2) This is appended to the manifest package name. The goal here is allow having different build types create different apks that can all be installed at the same time. This is optional.

Like Product Flavors, Build Types can provide their own source code, resources and manifest.
The typical use cases are:
  • different MapView key in release and debug mode as this is linked to the signing key (3)
  • require extra permissions for a debug build.

(3) If Product Flavors are used to create different versions built with different keys, the MapView API Key can be provided by the resources of the Product Flavor instead. See Build Variants below.

The source code and resources provided by the Build Types affects the project the same way Product Flavors do.

Build Variants

We’ve seen Product Flavors and Build Types and how both configure the output of a project.

In fact, the output of a project can only be the cross product of the Product Flavor and the Build Type. This is call a build variant.

It is not possible to build a project without selecting both a Build Type and a Product Flavor.

For a project not customizing any Build Type or Product Flavor, the available build variants are debug and release versions, as the main configuration is implied.

However, if a project declares two flavors, for instance free and forpay, the available build variants are:
  • Free-Debug
  • Free-Release
  • Forpay-Debug
  • Forpay-Release

We have seen above how source code and resources provided by Build Types and Product Flavors extend or override the main configuration. When actually building an app, the Build Type and Product Flavor are both applied with the following rules:

All the source folders are used together (main config, build type, product flavor, and generated source code(4)) to create a single output.

This means the same restrictions mentioned above applies, with the addition that Build Types and Product Flavors cannot provide different versions of the same class.

(4) generated source is output from resource compilation (R.java), aidl and renderscript compilation, and things like BuildConfig which can be impacted by the BuildType. These are different per build variants.

Resources combination uses the following priority (lower number is higher priority and override higher number):
  1. generated from Build Typed (5)
  2. build type
  3. generated from Product Flavor (5)
  4. flavor
  5. generated from main configuration (5)
  6. main configuration

(5) These are the outputs of the image crunch step, and of the renderscript compilation.

Packaging of Assets, Java Resources, JNI libraries follow a similar priority when encountering duplicates:
  1. Build Type
  2. Product Flavor
  3. main configuration

Sourcesets

Due to Build Types and Product Flavors all providing their own sources, resources and manifest, the project structure need to change from the root level src/res/manifest used in the current build system.

Instead the build system moves to a top level src folder, containing folders for each types and flavors. In addition, the main folder contains the source for the main configuration.

A typical project will then have the following folder structure:

  • src
    • main
      • java
      • jni
      • resources -- this is java resources
      • aidl
      • res
      • res -- this is Android resources
      • assets
      • AndroidManifest.xml
    • debug
      • … (same as main)
    • release
      • <span st
...
Re: Build System high level feature proposal b0b 8/5/12 9:09 AM


I cannot stress the following point enough: if you every have to download a .jar manually, you've lost. If you have to manually resolve a version difference between projects, you've lost. If you have you create an IDE project for a library dependency, you've lost.

Proper dependency management is a magical experience and I cannot wait for the every single Android developer to realize the simplicity and elegance of when it's done properly. Once experienced you will continuously wonder how you ever managed to copy .jars into various libs/ folders...


Yes, this point is absolutely essential. In the regular Java / J2EE development world this is a given, mostly through maven, an area it really excels at.
Imagine a complex J2EE projects having to manually handle dozens of versionned and complicated dependency jars by copying them in a libs/ folder.
Absolutely inconcevable. The ADT build system feels totally antiquated in this department.

Do my app need a specific ActionBarSherlock version or some other third party library, either and Android Library or a pure Java Library ?
I want to be able to add a few lines to a file (or through a GUI for mr newbie coder) and bam....libraries and dependecies downloaded, with source code if available.
Manually handling dependencies and copying jars = fail.

Another area of major interest is to speed up the build process considerably. On large projects, I waste a lot of time
waiting for the damn thing to generate the final APK. Dirk Jakel post on this topic is spot on. I'm sick of the tools dexing the same pure java libs
that never change over and over and over...

Please do not go the NIH syndrome route on this. You will lose considerable time and nobody will be satisfied by the final solution.
Personnaly I would leverage maven to the maximum and make the integration with the tools first class, with an easy mode hiding complexity for simple projects
and newcomers.

Re: Build System high level feature proposal Joe Bowbeer 8/5/12 10:31 AM

On Saturday, August 4, 2012 5:08:24 PM UTC-7, Xavier Ducrohet wrote:
On Sat, Aug 4, 2012 at 2:33 PM, Joe Bowbeer wrote:
> 2. Where is the libs folder?  Where do the 3rd-party .jars go?
>
> This is related to question #1 about leveraging portable Java code.  The
> most common way to do this is to add the .jars to the libs folder, but I
> don't see this folder in your outline.  Where are the libs?  Are there
> separate libs per flavor, or what?

First you should really reference dependency using Maven or Ivy if
this is a 3rd party library.
If this is your own, you could create a dependency on a locale file.
It's up to you to put it where you want.
With gradle we aren't as tied as libs/ are we were with Ant.

I'm on board with Maven.  I've used a Maven plugin on a few small Android projects, and it is likely that I would be using a Maven Android plugin for "real" applications if it were endorsed by your team.  Maven is mature and ubiquitous, which are two qualities I look for in build system tools.  I like that Maven is supported by all IDEs, and many CI frameworks such as Jenkins and Hudson.  (Hudson is now an Eclipse project with contributions from Sonatype as well as Oracle.)  Ease of integration at the CI level is another important consideration when choosing the build system tools.

This is also a question about dependencies: Do flavors have their own set of source dependencies?

Would it make sense to extend the dependency management to handle assets and resources as well?

--Joe
Re: Build System high level feature proposal Manfred Moser 8/5/12 12:04 PM
On Sun, Aug 5, 2012 at 10:31 AM, Joe Bowbeer <joe.b...@gmail.com> wrote:
>
> ...
> I'm on board with Maven.  I've used a Maven plugin on a few small Android
> projects, and it is likely that I would be using a Maven Android plugin for
> "real" applications if it were endorsed by your team.  Maven is mature and
> ubiquitous, which are two qualities I look for in build system tools.  I
> like that Maven is supported by all IDEs, and many CI frameworks such as
> Jenkins and Hudson.  (Hudson is now an Eclipse project with contributions
> from Sonatype as well as Oracle.)  Ease of integration at the CI level is
> another important consideration when choosing the build system tools.

Thanks for making the case for Maven quite nicely there Joe. As core
committer on the Android Maven Plugin I would like add a bit more
information.

The plugin and documentation is here
http://code.google.com/p/maven-android-plugin/

and more documentation is in the book Maven: The Complete Reference

http://www.sonatype.com/Support/Books/Maven-The-Complete-Reference

(Disclaimer ... I wrote that chapter)

Maven is by far the most widely used build system in the Java/JVM
ecosystem with very mature tool integration in all IDE's and beyond.

E.g. check out the existing Eclipse integration http://www.eclipse.org/m2e/

It is already used by many organizations using it for Android
development (github, groupon, square and many more...) and of course
way beyond for Java, Scala, Groovy.. development (e.g. Twitter
http://discursive.com/2012/04/18/twitters-moving-to-maven/, standard
build tool for Eclipse, Apache, Spring...).

Due to its focus on convention over configuration it is easy to get
started with yet can be scripted nicely with custom plugin
invocations. It has a bit of a bad reputation due to it being
"opinionated" steering you towards best practices rather than allowing
you to easily script together something. This is intentional and is
great when it is being used by teams of developers or even just a few
developers that work on the build setup after the core author changed
job or so.

Gradle on the other hand definitely has the coolness factor now and is
a very good and powerful build system with a built in scripting
language (since it is just a groovy dsl). It just recently had a 1.0
release and tool support is slowly emerging in the latest tool
releases. At this stage it is definitely getting to a mature stage but
there is still a way to go. The powerful scripting is however imho a
danger for most users that will just hack together what they think is
best rather than learning from years of best practice in a large user
community.

In terms of the proposed features I think most of the proposed
features are already available in the Android Maven Plugin and I sure
we would be more than happy for Google taking on the codebase as a
fork or inspiration and many of our contributors would be happy to
work with Google going forwards as well. We already have Eclipse
integration as well as good support in Intellij and Netbeans for
Android specifics beyond the general Maven integration.

Whatever you decide I hope that most of the functionality is
implemented in build system agnostic jars that are available easily in
a repository somewhere and the Android build system mostly wraps these
jars just like e.g. the Android Maven Plugin already wraps ddms and so
on.
This way we are going to implement a good build system across all JVM
languages for whatever the communities want (e.g. how about a
Ruby/Rake based one for Ruboto users or a Leiningen one for Clojure on
Android (pipedream), or an SBT one for scala...)

I am looking forward to see whats is coming..

manfred
Re: Build System high level feature proposal James Moore 8/9/12 8:49 AM
On Fri, Aug 3, 2012 at 5:36 PM, Xavier Ducrohet <x...@android.com> wrote:
> Use different jar for compile vs. package

It would be useful if this also worked for Android library projects,
not just jars.  I need a way of getting the list of library projects
(already provided in the classpath container, I think), but I'll be
handing dex a minified version of the library code.

(You may have been thinking this already, just wanted to make sure)

--
James Moore
ja...@restphone.com
http://blog.restphone.com/
http://www.linkedin.com/in/jamesmmooreiv
Re: Build System high level feature proposal Xavier Ducrohet 8/9/12 12:48 PM
Yes. if you look at the section "Building library Projects" there's
classes.jar which gets packaged and an optional api.jar that would be
used for compilation instead of classes.jar.
Re: Build System high level feature proposal Jake Wharton 8/9/12 11:17 PM
  • Help devs making different versions of the same app.
    • Library Projects can be cumbersome to make 2 apps that are 99% identical.
    • Different versions for different app store could only differ in package name and signing key.

This step needs extra care because I feel that it's vital to anyone that needs production and development (and optionally other variants, betas?) installed side-by-side.

You mentioned a folder structure which would align with a potentially unlimited number of configurations through Maven-like directories that exist in parallel with the 'main' and 'test' conventions that already exist. This could be extremely powerful and versatile when coupled with manifest manipulation of package name. Where we must tread lightly is in R.java and BuildConfig.java generation and the effect it has on imports.

For example, one of Square's applications requires both a development and beta version of the application which differ only by a few resources (icons, values/ xml, and manifest package name). At present the difference in package name requires either the sources exist in a library project which generates its own R/BuildConfig in an unchanging package or a script which facilitates changing the imports based on the current manifest configuration. This is unacceptable and doesn't scale at all. Ideally we would be able to produce these variants without having to make variant-specific modifications or relying on a common, shared library with what is essentially a stub app to change three things.
Re: Build System high level feature proposal Ievgenii Nazaruk 8/9/12 11:40 PM
Even current build system can be slightly modified to produce .apk from same code with different package names _without_ updating imports or moving code to library. I believe this feature was added to aapt sometime at the end of 2010 (just before 2.3 release), the option you need is "--rename-manifest-package". Last time I checked (and this was long time ago) this worked without any issues. 

One thing to keep in mind, is that tester .apk should then also be built with using "--rename-instrumentation-target-package" to actually test those different versions of .apk, and not the main one only. I didn't use this one myself though.

So, as a conclusion, this scenario can be supported by using extra options to aapt when producing .apk. 
Re: Build System high level feature proposal Xavier Ducrohet 8/10/12 9:37 AM
That's exactly right.

--rename-manifest-package acts when the manifest is converted to
binary format, injecting the new value in the package attribute (and
updating the activity/service/provider/.. classes to not be relative
to the package anymore).

This means the original package, declared in
src/main/AndroidManifest.xml will be used to generate all
R.java/Manifest.java/BuildConfig.java no matter what the flavor is.
Re: Build System high level feature proposal Tomáš Procházka 8/12/12 1:23 AM
For example in Netbeans IDE it is solved very nice. User can create Maven project add dependencies, set configuration and everything from IDE GUI, no maven knowledge is necessary 
Then, biggest problem for beginners maven users is add dependency something what is not in any repository. There should be something like "Add local dependency" which simply manually install jars/javadoc/source to the local maven repository.

Dne sobota, 4. srpna 2012 10:46:33 UTC+2 Xavier Ducrohet napsal(a):
I agree.

First we want it to be easy to just create a project and build it with little (if any) configuration.

After that we do want people to progressively extend their build as they need. But realistically, discovery of how to do things is never going to be trivial. Even something like setting a dependency on a lib coming from Maven requires some basic knowledge that devs who never used Maven don't have.
Proper documentation on developer.android.com will be critical.


Re: Build System high level feature proposal Manfred Moser 8/20/12 11:47 AM
Hi Xavier and others interested,

It took a bit longer to get to it but below as some comments and ideas of mine regarding the proposal.


On Friday, 3 August 2012 17:36:15 UTC-7, Xavier Ducrohet wrote:
As I've mentioned before, we are looking are a new version of the build system provided in the SDK.

So what are we trying to do?

  • Better extensibility
    • Allow devs to configure every steps.
      • expose all options for aapt, dex, …

Hopefully expose all these in an API in java artifacts so it can be reused by other tools and then implement wrappers for the build system of choice, keeping them as shallow as possible
 
  • Help devs making different versions of the same app.
    • Library Projects can be cumbersome to make 2 apps that are 99% identical.
    • Different versions for different app store could only differ in package name and signing key.

When it comes to interaction with the store application and so on the changes can get more and more complex. Whatever you do here should seemlessly work together with libraries.
 
  • Better dependency management for 3rd party jar files and Library Projects
    • Use Maven/Ivy for dependency management
    • Depend on Library Projects distributed as binary blob
    • Generate Library Projects blob

These libraries or 3rd party jar files should work with only java code, only android code, only android code in .class files, java code in .class files and any combination with resources of varying setups. The resources should also be able to be a located in a resource only jar and used correctly and it should be possible to also use java resources like properties files in case a java library needs them. At least any magic removal or filtering should be possible to be turned off.

Product Flavors


A product flavor defines a customized version of the application build by the project. A single project can have different flavors which change the generated application.

Although different flavors could be very different applications, using Library Projects is a better use case for this, and the build system still supports this.

Library projects is only useful if it is possible to use them as a dependency in a project without having to rebuild it but rather just include it as a jar. Otherwise the overhead of adding it in and building it as part of the applications is just too much waste of time.
 
When a Product Flavor is defined, the main configuration is not available as a buildable product. To create more than one product, two or more flavors must be created and customized.

Why is that? Would it not be easier to just support the main configuration as a buildable product as well? 
 

Build Types


A Build Type provides configuration for the following build properties:
  • manifest debuggable flag
  • native compilation debug flag
  • proguard enabled + rules
  • debug signing flag (ie whether to use debug key or release key)
  • package name suffix (2)
  • Buildconfig
    • DEBUG flag
    • other custom properties?

I believe this should maybe also include zipalign and maybe some sort of check if the pngs are optimized.

In fact a tool to optimize the png files would be nice..  
 
By default the build system provides two build types, debug and release, which can be reconfigured. New types can also be created to provide any combinations of the above properties.
For instance, one could make a Build Type that uses Proguard but creates a debuggable apk.

I think you should by default add another build type, which is QA/ Staging/Testing or whatever you want to call it. Everybody needs that.. e.g. to configured a  QA build that is configured like a production release build but e.g. talks to the testing infrastructure on the server side.. 
 
 

Sourcesets

Due to Build Types and Product Flavors all providing their own sources, resources and manifest, the project structure need to change from the root level src/res/manifest used in the current build system.

Instead the build system moves to a top level src folder, containing folders for each types and flavors. In addition, the main folder contains the source for the main configuration.

A typical project will then have the following folder structure:

  • src
    • main
      • java
      • jni
      • resources -- this is java resources
      • aidl
      • res
      • res -- this is Android resources
      • assets
      • AndroidManifest.xml
    • debug
      • … (same as main)
    • release
      • … (same as main)
    • flavor1
      • … (same as main)
    • flavor2
      • … (same as main)


Great.. this is pretty much the maven standard directory layout. One thing you are completely leaving out is a overriding set up files for test runs. I would suggest to mirror the structure above in src/test just like Maven does it (and Gradle and others implement as well).
 

This is of course be completely customizable, but this shows the default, and the type of inputs that are available to tasks.

BuildConfig

BuildConfig is an automatically generated class with a single boolean DEBUG, set to true in debug builds only.

Please provide an API that produces that file so we dont have to reimplement (as we did not in the android maven plugin).
 

Testing

Test Application

In the current build system, test applications are created from a different project that is flagged as a “test project”. This project references the main project in order to get the main project’s classes on its classpath.

With possibly multiple Product Flavors it is important to test all of them.

Some Java build systems have introduced having the test code directly in the project under src/test/ and the new build system follows this convention.

Great... while this is could I think it would still be good to work in the traditional way as well where a separate project that has code in src/test reference the tested application as a dependency. This allows for e.g. testing different versions with the same test by just specifying a different version of the tested application. It also reduces build time for the application by simply omitting tests .. 

In the maven world we introduce a so called aggregator project that ties them together. I am sure Gradle can do something similar. This would be really good also to tie together e.g. building a pure java library, as server side component, an android app and a test that deploys and runs it all (server and android app). We can do this already in Maven and it should work for the new Android build system as well.. 
 
Since JUnit test cannot be run against Android code, the build system reuses this concept for test applications.
There is a mirror of the main configuration, plus flavors which allows creating flavor specific test applications.
The main test configuration uses src/test, and flavors uses src/test<flavor>

The manifest of a test application is pretty basic and it is automatically created it. The build system provides the ability to configure the test package name and the InstrumentationTestRunner class.

Deployment and running all test flavors TBD.

JUnit tests

JUnit tests cannot be run against code accessing Android APIs. Instrumentation tests are the sanctioned way to go.
To have code that does not access Android APIs tested with JUnit tests, a multi-project build should be setup with the code to be tested put in a Java-only project.

Note: The build system shouldn’t prevent usage of some third party test frameworks such as Roboelectric. It should actually facilitate integrating those. Details TBD.

Great... please work with these projects including e.g. Robotium that sits on top of the instrumentation tests.

Also when running tests on devices look at implementing a feature that produces the test results on the machine running the tests and not the devices. Note that you can look at the Android Maven Plugin codebase where I implemented that for junit output creation using the ddms library from the sdk. Then you just have to come up with a naming convention for the files when you run the tests onto multiple devices.
 

Building Library Projects.


Libraries must be packaged in a very specific way, no matter what the library source folder structure is.
  • classes.jar -- this is the main code used for compile and packaging.
  • api.jar -- this is an optional jar to used at compile time instead of classes.jar
  • res/ -- this is the res folder(s) extended with:
    • crunched images
    • renderscript bitcode
  • AndroidManifest.xml (includes the merging from other Library Projects?)
  • lib/ -- this is the jni component(s)
  • aidl -- this is a straight copy of all the src/../aidl files


And all that goes into a jar right? Why not have the classes right in there instead of a classes jar .. just like normal jar's do it? The current proposal just makes it more complicated for no benefit I can tell of.
 
In all the above cases what is packaged can be extended by the release Build Type, similar to a normal apk.

Additionally, the blob may contain:
  • proguard.txt -- custom Proguard rules
  • lint.jar -- custom lint rules
  • ide.jar -- custom IDE extensions
  • docs/ -- this is the javadoc
  • source/ -- this is optional source package (for open source libraries).


Would it not be better to have a separate source artifact as well as separate lint and ide jar artifacts .. otherwise you are completely mixing up concerns and have to deal with the fact that e.g. lint and ide jar should not go into a release build.. 

Using Library Projects.

Library projects can be used two ways:
  1. Dependency through a repository (local/maven/ivy)
    • this downloads the blob and unarchives it somewhere
  2. As part of a multi-project build.

#1 is fairly standard and can be added as as normal dependency.
#2 is a bit more complicated and needs to be figured out. The main issue is that dependency on Library Projects have a clear dependency order and the setup of the dependency must clearly spell this out.

When deployed to a repository server or in general the library project has to be able to declare its dependencies so that these can be automatically be pulled in as well. E.g. 

- App declares dependency on ActionBarSherlock in project file
- build gets it from the repo server and sees in the meta data file (in the maven and also gradle world the pom.xml that is deployed to the repo server)
that ABS in turn depends on the compatibility library
- so the build gets it as well and hooks it in

Notice that this works via a local cache just the same (for gradle and maven) as for a repository server. In both cases the meta data file needs to be there. In turn the build tool has to support authoring such a library and deploying a library to the repo server so it can be reused by others (or better pushed to the Central Repository) so everybody can use it.

Dependencies


Note that the Android API jar(s) is automatically added to the compile (only) dependency list, and doesn’t have to be declared manually.

But it should be possible to override it imho with e.g. a custom android jar. 


Lint Integration

The new build system integrates Lint at two levels.

Running Lint

A Lint tasks is created and allows testing any build variant.
Lint errors will break the build.

It should be possible to skip the lint run and to configure it to run but NOT break the build.
 
Lint rules can be enabled/disabled and set priority (warning/error).

Disabling all might be the equivalent to disabling lint altogether.
 
HTML and/or XML reports can be generated.
Lint should run on all available Build Variants but remove duplicates (since code common to all variants is compiled for each of them

And I hope an API will be available to invoke it.. 
 
 

Extending Lint

Creating custom Lint is directly possible using the src/lint/ folder.

This creates a lint.jar file that is passed to Lint when running it on the project itself. The dependency on the Lint API is done automatically. All that’s needed is creating the source folder.


It should also be possible to just e.g. create a lint jar for all android project in a company or even one to share via open source , where you push to a repository server as a binary jar and declare it as a dependency on all the project you want to use it on.. 

The same thought applies to ide.jar imho..

I hope this all helps.. 

manfred
Re: Build System high level feature proposal Xavier Ducrohet 8/20/12 1:29 PM
thanks Manfred for the feedback.

A general comment that applies to several of your points: Yes we are
making a java library that will do all of the work. All you need is to
configure it with info coming from the front end you choose (Maven,
gradle, whatever). Yes we'll put this on a repo so that you can use it
easily in the maven plugin.

More answers inline.
We already support java resources, so there should be no problem.
We'll make sure that filtering is override-able, configurable and
replaceable.

>> Product Flavors
>>
>>
>> A product flavor defines a customized version of the application build by
>> the project. A single project can have different flavors which change the
>> generated application.
>>
>> Although different flavors could be very different applications, using
>> Library Projects is a better use case for this, and the build system still
>> supports this.
>
>
> Library projects is only useful if it is possible to use them as a
> dependency in a project without having to rebuild it but rather just include
> it as a jar. Otherwise the overhead of adding it in and building it as part
> of the applications is just too much waste of time.

Yes of course.

>>
>> When a Product Flavor is defined, the main configuration is not available
>> as a buildable product. To create more than one product, two or more flavors
>> must be created and customized.
>
>
> Why is that? Would it not be easier to just support the main configuration
> as a buildable product as well?

It's a good question. There are some case where main will not be a
buildable product.
For instance 2 flavors each providing a different version of a class.
This means the "main" configuration won't have this class and likely
won't compile.

So we can either have a switch that says "main is a valid product" or
just automatically disable it when you add a flavor.

>> Build Types
>>
>>
>> A Build Type provides configuration for the following build properties:
>>
>> manifest debuggable flag
>> native compilation debug flag
>> proguard enabled + rules
>> debug signing flag (ie whether to use debug key or release key)
>> package name suffix (2)
>> Buildconfig
>>
>> DEBUG flag
>> other custom properties?
>
>
> I believe this should maybe also include zipalign and maybe some sort of
> check if the pngs are optimized.

zipalign should be automatic if the build signs the package. There's
no reason not to zipalign the package if you can.
Am I missing something here?

> In fact a tool to optimize the png files would be nice..

Why not. Any reason why this shouldn't be part of the build system
though? It seems much safer to just do it all the time.

>>
>> By default the build system provides two build types, debug and release,
>> which can be reconfigured. New types can also be created to provide any
>> combinations of the above properties.
>> For instance, one could make a Build Type that uses Proguard but creates a
>> debuggable apk.
>
>
> I think you should by default add another build type, which is QA/
> Staging/Testing or whatever you want to call it. Everybody needs that.. e.g.
> to configured a  QA build that is configured like a production release build
> but e.g. talks to the testing infrastructure on the server side..

That's an interesting idea. However there needs to be some flags
that's different from debug/release by default to have it as a
built-in build type.
None of the current build type properties seem like they would have a
special value in "staging".
Is there any other properties you're thinking of?

My point is that while yes, staging could be configured by a dev to
point to a test server, out of the box the "staging" build type must
be different from debug/release without having to do put some custom
file somewhere to make sense.

BTW, I am happy to add new properties to both ProductFlavor and
BuildType if you think of new ones that make sense.
Overriding what kind of files?

>>
>> This is of course be completely customizable, but this shows the default,
>> and the type of inputs that are available to tasks.
>>
>> BuildConfig
>>
>> BuildConfig is an automatically generated class with a single boolean
>> DEBUG, set to true in debug builds only.
>
>
> Please provide an API that produces that file so we dont have to reimplement
> (as we did not in the android maven plugin).

Yes, this will be part of the builder library.

>> Testing
>>
>> Test Application
>>
>> In the current build system, test applications are created from a
>> different project that is flagged as a “test project”. This project
>> references the main project in order to get the main project’s classes on
>> its classpath.
>>
>> With possibly multiple Product Flavors it is important to test all of
>> them.
>>
>> Some Java build systems have introduced having the test code directly in
>> the project under src/test/ and the new build system follows this
>> convention.
>>
> Great... while this is could I think it would still be good to work in the
> traditional way as well where a separate project that has code in src/test
> reference the tested application as a dependency. This allows for e.g.
> testing different versions with the same test by just specifying a different
> version of the tested application. It also reduces build time for the
> application by simply omitting tests ..

What do you mean by testing different versions? different revisions or
different flavors of the app?
The former doesn't seem that useful. The latter will be handled
directly (we'll have separate test folder for each flavor).

Not building the test should be taken care of depending on which tasks you call.

> In the maven world we introduce a so called aggregator project that ties
> them together. I am sure Gradle can do something similar. This would be
> really good also to tie together e.g. building a pure java library, as
> server side component, an android app and a test that deploys and runs it
> all (server and android app). We can do this already in Maven and it should
> work for the new Android build system as well..

Yes, multi-project build in gradle should handle this.
You mean storing the result of the tests on the host machine?

>>
>> Building Library Projects.
>>
>>
>> Libraries must be packaged in a very specific way, no matter what the
>> library source folder structure is.
>>
>> classes.jar -- this is the main code used for compile and packaging.
>> api.jar -- this is an optional jar to used at compile time instead of
>> classes.jar
>> res/ -- this is the res folder(s) extended with:
>>
>> crunched images
>> renderscript bitcode
>>
>> AndroidManifest.xml (includes the merging from other Library Projects?)
>> lib/ -- this is the jni component(s)
>> aidl -- this is a straight copy of all the src/../aidl files
>>
>>
>
> And all that goes into a jar right? Why not have the classes right in there
> instead of a classes jar .. just like normal jar's do it? The current
> proposal just makes it more complicated for no benefit I can tell of.

see below.

>>
>> In all the above cases what is packaged can be extended by the release
>> Build Type, similar to a normal apk.
>>
>> Additionally, the blob may contain:
>>
>> proguard.txt -- custom Proguard rules
>> lint.jar -- custom lint rules
>> ide.jar -- custom IDE extensions
>> docs/ -- this is the javadoc
>> source/ -- this is optional source package (for open source libraries).
>>
>>
>
> Would it not be better to have a separate source artifact as well as
> separate lint and ide jar artifacts .. otherwise you are completely mixing
> up concerns and have to deal with the fact that e.g. lint and ide jar should
> not go into a release build..

I'm not sure I see the point of multiple artifacts (except maybe for sources).

However I'll start with one concern:
A library dependency needs to have more than just classes in it. there are:
- java resources
- android resources
- jni components
- aidl import files.
- proguard rules

#1 and #2-5 collides with each other. ie right now all non .class
files inside a jar dependency are added to the apk as they are java
resources. We cannot just simply put other files that are not java
resources and process them differently. No matter how we put it there
is a risk of having a conflict with actual java resources.

On top of this, lint.jar should be there too. It's not another
artifact that you have to depend on, it's part of the library. It
should be automatic. You depend on the library and you get lint.jar
that gets used during the lint step.
Same thing for the proguard file.

So it makes a lot more sense to me to have a bundle (zip archive) that
contains the jar used for compile (that would contain the java
resources as well) and other items used during build, that may or may
not find their way into the apk (for instance aidl import files,
lint.jar and proguard file are merely used during the build but are
not packaged).
This bundled is downloaded and extracted locally somewhere in the
folder of cache of the dependency management system and its various
bits are used by the build system.


>> Using Library Projects.
>>
>> Library projects can be used two ways:
>>
>> Dependency through a repository (local/maven/ivy)
>>
>> this downloads the blob and unarchives it somewhere
>>
>> As part of a multi-project build.
>>
>>
>> #1 is fairly standard and can be added as as normal dependency.
>> #2 is a bit more complicated and needs to be figured out. The main issue
>> is that dependency on Library Projects have a clear dependency order and the
>> setup of the dependency must clearly spell this out.
>
>
> When deployed to a repository server or in general the library project has
> to be able to declare its dependencies so that these can be automatically be
> pulled in as well. E.g.

Yes of course. The idea is that the bundle defined above are
uploadable to a Maven or Ivy repository.
I'm not worried about that. I was a bit more concerned about
multi-project build that actually bypass
repositories, but that should be easy to do as well.

> - App declares dependency on ActionBarSherlock in project file
> - build gets it from the repo server and sees in the meta data file (in the
> maven and also gradle world the pom.xml that is deployed to the repo server)
> that ABS in turn depends on the compatibility library
> - so the build gets it as well and hooks it in
>
> Notice that this works via a local cache just the same (for gradle and
> maven) as for a repository server. In both cases the meta data file needs to
> be there. In turn the build tool has to support authoring such a library and
> deploying a library to the repo server so it can be reused by others (or
> better pushed to the Central Repository) so everybody can use it.

Yes, the build tool will allow creating artifacts that can be
uploaded. Gradle has built-in support for this.

>> Dependencies
>
>
>> Note that the Android API jar(s) is automatically added to the compile
>> (only) dependency list, and doesn’t have to be declared manually.
>
>
> But it should be possible to override it imho with e.g. a custom android
> jar.

hmm I would rather not allow this, at least not through a simple
property somewhere.
What's the use case here?

>> Lint Integration
>>
>> The new build system integrates Lint at two levels.
>>
>> Running Lint
>>
>> A Lint tasks is created and allows testing any build variant.
>> Lint errors will break the build.
>
>
> It should be possible to skip the lint run and to configure it to run but
> NOT break the build.

Lint as 2 levels of severity: warning and error.
Like configuring any java compiler, if you set a particular check to
be an error, it will break the build.

Maybe we do need several profiles that allow you to set everything to
be warning instead of errors. But if you run with some checks sets to
be errors these will break the build.

>> Lint rules can be enabled/disabled and set priority (warning/error).
>
>
> Disabling all might be the equivalent to disabling lint altogether.

Yes, this would work.

>>
>> HTML and/or XML reports can be generated.
>> Lint should run on all available Build Variants but remove duplicates
>> (since code common to all variants is compiled for each of them
>
>
> And I hope an API will be available to invoke it..

Yep.

>>
>> Extending Lint
>>
>> Creating custom Lint is directly possible using the src/lint/ folder.
>>
>> This creates a lint.jar file that is passed to Lint when running it on the
>> project itself. The dependency on the Lint API is done automatically. All
>> that’s needed is creating the source folder.
>>
>
> It should also be possible to just e.g. create a lint jar for all android
> project in a company or even one to share via open source , where you push
> to a repository server as a binary jar and declare it as a dependency on all
> the project you want to use it on..
>
> The same thought applies to ide.jar imho..

Indeed.

The lint.jar in a library is to provide custom checks for people using
a particular API provided by the library, but lint does support for
both configuration and custom rules that can live anywhere.

We'll have to make sure this can be configured through the build system.
Re: Build System high level feature proposal Tor Norbye 8/20/12 1:43 PM

Actually, it has a third severity, "fatal", which is like error but also abort (when you do export apk, lint runs automatically and checks only the fatal severity issues and aborts the export if any are found. Fatal let's us have non-aborting errors, though I should probably make a pass over the issues and adjust the error vs fatal severities since it's a bit arbitrary at the moment.

-- Tor

Re: Build System high level feature proposal Manfred Moser 8/20/12 10:12 PM
More inline..

On Mon, Aug 20, 2012 at 1:29 PM, Xavier Ducrohet <x...@android.com> wrote:
>>> When a Product Flavor is defined, the main configuration is not available
>>> as a buildable product. To create more than one product, two or more flavors
>>> must be created and customized.
>>
>>
>> Why is that? Would it not be easier to just support the main configuration
>> as a buildable product as well?
>
> It's a good question. There are some case where main will not be a
> buildable product.
> For instance 2 flavors each providing a different version of a class.
> This means the "main" configuration won't have this class and likely
> won't compile.
>
> So we can either have a switch that says "main is a valid product" or
> just automatically disable it when you add a flavor.

I would default to main as a valid product and allow it be switched to
invalid, since that seems more common and straightforward to use.

>>> Build Types
>> I believe this should maybe also include zipalign and maybe some sort of
>> check if the pngs are optimized.
>
> zipalign should be automatic if the build signs the package. There's
> no reason not to zipalign the package if you can.
> Am I missing something here?

Not sure. My thinking would be not to invoke anything that is not
necessary during a dev build including not invoking proguard, signing,
zipalign and whatever takes time. I would completely disable lint by
default too btw.

>> In fact a tool to optimize the png files would be nice..
>
> Why not. Any reason why this shouldn't be part of the build system
> though? It seems much safer to just do it all the time.

Maybe as part of a release build. But for normal dev builds the more
you add in the longer it takes and that kills productivity.

>> I think you should by default add another build type, which is QA/
>> Staging/Testing or whatever you want to call it. Everybody needs that.. e.g.
>> to configured a  QA build that is configured like a production release build
>> but e.g. talks to the testing infrastructure on the server side..
>
> That's an interesting idea. However there needs to be some flags
> that's different from debug/release by default to have it as a
> built-in build type.
> None of the current build type properties seem like they would have a
> special value in "staging".
> Is there any other properties you're thinking of?
>
> My point is that while yes, staging could be configured by a dev to
> point to a test server, out of the box the "staging" build type must
> be different from debug/release without having to do put some custom
> file somewhere to make sense.

A staging build imho is running all test and verifications like the
release build (lint, proguard, .. enabled) but does NOT go to
production but rather to a QA/testing deployment and therefore might
be configured very differently. E.g. it probably has debug enabled so
that QA can send you stack traces of failures with more info and so
on.

> BTW, I am happy to add new properties to both ProductFlavor and
> BuildType if you think of new ones that make sense.

Where is a list of them I can look at?

>>> Sourcesets
>> Great.. this is pretty much the maven standard directory layout. One thing
>> you are completely leaving out is a overriding set up files for test runs. I
>> would suggest to mirror the structure above in src/test just like Maven does
>> it (and Gradle and others implement as well).
>
> Overriding what kind of files?

Anything that can be configured differently for testing as compared to
the actual build. Could be e.g database connection, or a template data
in a db, or different resource files and so on and so on

>>> Testing

>> Great... while this is could I think it would still be good to work in the
>> traditional way as well where a separate project that has code in src/test
>> reference the tested application as a dependency. This allows for e.g.
>> testing different versions with the same test by just specifying a different
>> version of the tested application. It also reduces build time for the
>> application by simply omitting tests ..
>
> What do you mean by testing different versions?

E.g. you could have a test module that has UI tests that are
independent of the implementation and you have a version 1.0 on the
play store and are working on 1.1. While working on 1.1. you are
writing more tests and you need to ensure that the tests keep working
on version 1.0 and also want to test against 1.1. So you could e.g.
create a test group for 1.0 features and run all these test agains the
1.0 version to ensure your refactoring of the tests is still ok and in
a second run e.g. on a CI server run the same test against the 1.1
version to ensure that existing functionality is not broken in the
upcoming 1.1. version

>>
>> Great... please work with these projects including e.g. Robotium that sits
>> on top of the instrumentation tests.
>>
>> Also when running tests on devices look at implementing a feature that
>> produces the test results on the machine running the tests and not the
>> devices. Note that you can look at the Android Maven Plugin codebase where I
>> implemented that for junit output creation using the ddms library from the
>> sdk. Then you just have to come up with a naming convention for the files
>> when you run the tests onto multiple devices.
>
> You mean storing the result of the tests on the host machine?

Yes. That way you can e.g. build and run the test against a few
attached devices and running emulator at the same time and all test
outputs junit xml files will be available on the host machine in the
build output folder. That way e.g. they can be picked up easily by a
CI server without having to pull them off the devices with adb. imho
this is the more common need than having them on the device. What good
are they there anyway? This would be good for emma as well btw..

> So it makes a lot more sense to me to have a bundle (zip archive) that
> contains the jar used for compile (that would contain the java
> resources as well) and other items used during build, that may or may
> not find their way into the apk (for instance aidl import files,
> lint.jar and proguard file are merely used during the build but are
> not packaged).
> This bundled is downloaded and extracted locally somewhere in the
> folder of cache of the dependency management system and its various
> bits are used by the build system.

I understand what you are saying and I kind of agree and kind of
disagree. You are basically requiring all the logic for working with
the different files in the bundle into the build system rather than
having the user understand and configure it. Thats fine by me as long
as we can expose configurations to override these settings.

>>> Dependencies
>>
>>> Note that the Android API jar(s) is automatically added to the compile
>>> (only) dependency list, and doesn’t have to be declared manually.
>>
>> But it should be possible to override it imho with e.g. a custom android
>> jar.
>
> hmm I would rather not allow this, at least not through a simple
> property somewhere.
> What's the use case here?

E.g. you have a custom build of android that you want to create an app
against that has additional proprietary api's in it that are more
openly exposed.. this approach could then even be used to build the
AOSP apps with a more open android.jar.

Btw I still think it will be crucial to get all open source apps from
Google to be built with the new build tool e.g. at least the iosched
and other examples that use only public api's but ideally also all
apps in the aosp tree. That would be the best proofing ground for the
build system!

>>> Lint Integration
>> Disabling all might be the equivalent to disabling lint altogether.
>
> Yes, this would work.

So if I understand this correctly I should be able to configure e.g. a
pure dev build to have a lint config that disables it all to save
time, but also e.g. a verification dev build as a second one that does
all the lint stuff.

Cool.. thats all for now. Thanks for working on all this and taking in
our feedback in terms of features (even if you already decided on
Gradle.. ;-) )

Manfred
Re: Build System high level feature proposal Mark Murphy (a Commons Guy) 8/21/12 4:02 AM
On Tue, Aug 21, 2012 at 1:12 AM, Manfred Moser <mos...@gmail.com> wrote:
> I would completely disable lint by default too btw.

Please don't.

--
Mark Murphy (a Commons Guy)
http://commonsware.com | http://github.com/commonsguy
http://commonsware.com/blog | http://twitter.com/commonsguy

_The Busy Coder's Guide to Android Development_ Version 4.0 Available!
Re: Build System high level feature proposal Xavier Ducrohet 8/21/12 1:27 PM
On Mon, Aug 20, 2012 at 10:12 PM, Manfred Moser <mos...@gmail.com> wrote:
> More inline..
>
> On Mon, Aug 20, 2012 at 1:29 PM, Xavier Ducrohet <x...@android.com> wrote:
>>>> When a Product Flavor is defined, the main configuration is not available
>>>> as a buildable product. To create more than one product, two or more flavors
>>>> must be created and customized.
>>>
>>>
>>> Why is that? Would it not be easier to just support the main configuration
>>> as a buildable product as well?
>>
>> It's a good question. There are some case where main will not be a
>> buildable product.
>> For instance 2 flavors each providing a different version of a class.
>> This means the "main" configuration won't have this class and likely
>> won't compile.
>>
>> So we can either have a switch that says "main is a valid product" or
>> just automatically disable it when you add a flavor.
>
> I would default to main as a valid product and allow it be switched to
> invalid, since that seems more common and straightforward to use.

I'm not sure. I'm going to start a new thread about flavors as there
are some more details I want to discuss) and we'll see.

>>>> Build Types
>>> I believe this should maybe also include zipalign and maybe some sort of
>>> check if the pngs are optimized.
>>
>> zipalign should be automatic if the build signs the package. There's
>> no reason not to zipalign the package if you can.
>> Am I missing something here?
>
> Not sure. My thinking would be not to invoke anything that is not
> necessary during a dev build including not invoking proguard, signing,
> zipalign and whatever takes time. I would completely disable lint by
> default too btw.

proguard: yes, but that should be disabled by default for debug builds
signing is required to install an app, no way around this.
zipalign can have significant performance impact if not done. I would
think this should be done.
However I need to look at the partial update that Play can do now and
see how that would impact the build process.

Lint is another issue. see at the end.

>>> In fact a tool to optimize the png files would be nice..
>>
>> Why not. Any reason why this shouldn't be part of the build system
>> though? It seems much safer to just do it all the time.
>
> Maybe as part of a release build. But for normal dev builds the more
> you add in the longer it takes and that kills productivity.

Note that before the 2nd half of last year we didn't have incremental
process of images and 9 patch, which made everything slow. This is why
we first processed the resources to generate R.java only and later
processed the resources again to package them (including processing
images and 9-patch). This 2nd step could be delayed to install time in
Eclipse.

Ant followed this even though it was useless. So now we'll process the
resources only once and both generate R.java and process the
resources. The image and 9-patch processing will still be done ahead
of that in an incremental way.

This should speed up things compared to now (at least compared to ant
and eclipse, not sure what Maven did).

>>> I think you should by default add another build type, which is QA/
>>> Staging/Testing or whatever you want to call it. Everybody needs that.. e.g.
>>> to configured a  QA build that is configured like a production release build
>>> but e.g. talks to the testing infrastructure on the server side..
>>
>> That's an interesting idea. However there needs to be some flags
>> that's different from debug/release by default to have it as a
>> built-in build type.
>> None of the current build type properties seem like they would have a
>> special value in "staging".
>> Is there any other properties you're thinking of?
>>
>> My point is that while yes, staging could be configured by a dev to
>> point to a test server, out of the box the "staging" build type must
>> be different from debug/release without having to do put some custom
>> file somewhere to make sense.
>
> A staging build imho is running all test and verifications like the
> release build (lint, proguard, .. enabled) but does NOT go to
> production but rather to a QA/testing deployment and therefore might
> be configured very differently. E.g. it probably has debug enabled so
> that QA can send you stack traces of failures with more info and so
> on.

Yes, I understand this. However this is something that is set by the
developer putting some different configuration files in different
folder, not by setting a flag somewhere.

The buildType properties are:

manifest debuggable flag
native compilation debug flag
proguard enabled + rules
debug signing flag (ie whether to use debug key or release key)
package name suffix
Buildconfig DEBUG flag

This is it.

I think the staging idea is great, and should be advertise a good
practice. I'm not sure there's a point in creating a staging type by
default as it would essentially be the same as the debug one when you
look at the configuration properties above.

>> BTW, I am happy to add new properties to both ProductFlavor and
>> BuildType if you think of new ones that make sense.
>
> Where is a list of them I can look at?
>
>>>> Sourcesets
>>> Great.. this is pretty much the maven standard directory layout. One thing
>>> you are completely leaving out is a overriding set up files for test runs. I
>>> would suggest to mirror the structure above in src/test just like Maven does
>>> it (and Gradle and others implement as well).
>>
>> Overriding what kind of files?
>
> Anything that can be configured differently for testing as compared to
> the actual build. Could be e.g database connection, or a template data
> in a db, or different resource files and so on and so on

Wouldn't that be the staging one then?

It's a good point though that building an app that talks to a server
probably shouldn't talk to the production server when running
instrumentation tests.

I think we need to be able to configure the test to declare which
buildtype variant is used for testing (could be debug, could be
staging or any other).
This should be sufficient to let devs create a buildType that
overrides code and/or resources to build the app the way he/she wants
it to be tested.

>>>> Testing
>
>>> Great... while this is could I think it would still be good to work in the
>>> traditional way as well where a separate project that has code in src/test
>>> reference the tested application as a dependency. This allows for e.g.
>>> testing different versions with the same test by just specifying a different
>>> version of the tested application. It also reduces build time for the
>>> application by simply omitting tests ..
>>
>> What do you mean by testing different versions?
>
> E.g. you could have a test module that has UI tests that are
> independent of the implementation and you have a version 1.0 on the
> play store and are working on 1.1. While working on 1.1. you are
> writing more tests and you need to ensure that the tests keep working
> on version 1.0 and also want to test against 1.1. So you could e.g.
> create a test group for 1.0 features and run all these test agains the
> 1.0 version to ensure your refactoring of the tests is still ok and in
> a second run e.g. on a CI server run the same test against the 1.1
> version to ensure that existing functionality is not broken in the
> upcoming 1.1. version

This is interesting. I'm not sure where the 1.0 apk would come from though.

>>>
>>> Great... please work with these projects including e.g. Robotium that sits
>>> on top of the instrumentation tests.
>>>
>>> Also when running tests on devices look at implementing a feature that
>>> produces the test results on the machine running the tests and not the
>>> devices. Note that you can look at the Android Maven Plugin codebase where I
>>> implemented that for junit output creation using the ddms library from the
>>> sdk. Then you just have to come up with a naming convention for the files
>>> when you run the tests onto multiple devices.
>>
>> You mean storing the result of the tests on the host machine?
>
> Yes. That way you can e.g. build and run the test against a few
> attached devices and running emulator at the same time and all test
> outputs junit xml files will be available on the host machine in the
> build output folder. That way e.g. they can be picked up easily by a
> CI server without having to pull them off the devices with adb. imho
> this is the more common need than having them on the device. What good
> are they there anyway? This would be good for emma as well btw..

Right that's what I figured. right now the instrumentation runner
prints them I think.

Right now Eclipse gets this output and then display them in the
standard jUnit window.
Ant just output them to the console.
However Ant is able to get the emma data off of a device and process
it locally to create a report.

We definitively want to do all of that in here too.

>> So it makes a lot more sense to me to have a bundle (zip archive) that
>> contains the jar used for compile (that would contain the java
>> resources as well) and other items used during build, that may or may
>> not find their way into the apk (for instance aidl import files,
>> lint.jar and proguard file are merely used during the build but are
>> not packaged).
>> This bundled is downloaded and extracted locally somewhere in the
>> folder of cache of the dependency management system and its various
>> bits are used by the build system.
>
> I understand what you are saying and I kind of agree and kind of
> disagree. You are basically requiring all the logic for working with
> the different files in the bundle into the build system rather than
> having the user understand and configure it. Thats fine by me as long
> as we can expose configurations to override these settings.

What exactly do you want to override here? I don't think there's any
reason to package a library as simply a standard Java jar file.

It has to be an archive that contains the jar plus at least the
android resources (in most cases). We can't have those flat in the
same archive due to possible conflict with the java resources inside
the jar.

Once we agree on that point, we may as well just it the rest in the
main archive bundle and call it a day ;) I see little point in
generating such a bundle while omitting some parts of it (be it
source, lint.jar or anything else).

I understand this is different from the normal way of doing things in
Maven, but this is very different from just a java build.

>>>> Dependencies
>>>
>>>> Note that the Android API jar(s) is automatically added to the compile
>>>> (only) dependency list, and doesn’t have to be declared manually.
>>>
>>> But it should be possible to override it imho with e.g. a custom android
>>> jar.
>>
>> hmm I would rather not allow this, at least not through a simple
>> property somewhere.
>> What's the use case here?
>
> E.g. you have a custom build of android that you want to create an app
> against that has additional proprietary api's in it that are more
> openly exposed.. this approach could then even be used to build the
> AOSP apps with a more open android.jar.

I already want (and am planning) to allow building unbundled apps with
this scripts.
Those are apps that use public apis only but are built as part of the
Android source tree.

They don't use a normal SDK but instead look at the files in
platform/prebuilts/sdk/

This would be done through a simple property in gradle that lets you
point to a full Android source tree instead of an SDK.
In the builder library there'll be an SDK parser and a parser for the
Android source tree to resolve the compile target and find the
location of android.jar and other tools and files.

So we could technically add a different properties that tells the app
to build using the full api jar instead of the stubbed android.jar
This would only be possible when using a full Android source tree
instead of an SDK, so replacing android.jar itself is not needed.

People who use this will work directly in the source tree and will,
hopefully, know what they are doing (as in, breaking CTS and other fun
stuff)


> Btw I still think it will be crucial to get all open source apps from
> Google to be built with the new build tool e.g. at least the iosched
> and other examples that use only public api's but ideally also all
> apps in the aosp tree. That would be the best proofing ground for the
> build system!

Yes! Internally, people who work on apps are struggling to build with
the source tree build system (oh, makefiles...) while using our tools.
My goal is to move all those apps to the new build system to let them
use the IDE and only have a single way of configuring how their app is
built.

>>>> Lint Integration
>>> Disabling all might be the equivalent to disabling lint altogether.
>>
>> Yes, this would work.
>
> So if I understand this correctly I should be able to configure e.g. a
> pure dev build to have a lint config that disables it all to save
> time, but also e.g. a verification dev build as a second one that does
> all the lint stuff.

Maybe, or maybe not.

In gradle you get different task to do various levels of the build.

In a regular java project, gradle assemble will do build the app, but
nothing else. This is enough to run the app.
gradle build does assemble + builds and run the tests. This is what
should be done for deployment.

We will have the same. "gradle assemble<Variant>" will build the
variant and that's it. From there you can install on a device and use
the app.
"gradle build<variant>" will build the app, the test apps, (maybe) run
the test app, and run lint as part of the "check" task.

(calling just "gradle build" will build all variants of the app)

See the tasks graph here:
http://www.gradle.org/docs/current/userguide/java_plugin.html#N11B1C

So yes, you can build the app with not running lint, but doing "gradle
build<variant>" will run lint by default always. If you want to
disable it completely (why woud you?) you'll have to manually disable
all the checks.

> Cool.. thats all for now. Thanks for working on all this and taking in
> our feedback in terms of features (even if you already decided on
> Gradle.. ;-) )

I think Gradle is more suited to our needs but I definitively want to
make sure devs who want to use Maven can use it too.

I already mentioned it, but I'll say it again: all of the code dealing
with actually building the app will be in a separate library that you
can reuse. All gradle will do is handle the configuration of the build
and pass that on to the library.

In fact, parts of it (and of the gradle build sytem) is already
checked in (we are using a new git project in AOSP: tools/build.git)

You can see the current contributions (only a few of them, we just
started putting them in this week) here:
https://android.googlesource.com/platform/tools/build/+/master

Comments on this is welcome too. Look in particular at this:
https://android-review.googlesource.com/#/c/41861/5 which is the
initial commit of said library.

Ok, off to write a new post on dimensions of product flavors...
Re: Build System high level feature proposal Xavier Ducrohet 8/21/12 3:28 PM
On Tue, Aug 21, 2012 at 1:27 PM, Xavier Ducrohet <x...@android.com> wrote:
> zipalign can have significant performance impact if not done. I would
> think this should be done.

After thinking about it a bit more I guess we should at least have
this configurable. We can figure out the default value later.
Re: Build System high level feature proposal Manfred Moser 8/22/12 11:32 PM
More answers/discussion inline below


On Tuesday, 21 August 2012 13:27:44 UTC-7, Xavier Ducrohet wrote:

Ant followed this even though it was useless. So now we'll process the
resources only once and both generate R.java and process the
resources. The image and 9-patch processing will still be done ahead
of that in an incremental way.


For the image processing I think it would actually be good to be able to optimize them in place as a manual step so you can check the optimized version back into your source control and then have that process skipped or at least run even faster in the future.. 
This is it.

I think the staging idea is great, and should be advertise a good
practice. I'm not sure there's a point in creating a staging type by
default as it would essentially be the same as the debug one when you
look at the configuration properties above.

Fair enough. I guess we can always do that later if we find that there are more flags that are actually different. Need to test this and think about it a bit more.. 
 
I think we need to be able to configure the test to declare which
buildtype variant is used for testing (could be debug, could be
staging or any other).

I like that idea. 
 
>>>> Testing

>> What do you mean by testing different versions?
>
> E.g. you could have a test module that has UI tests that are
> independent of the implementation and you have a version 1.0 on the
> play store and are working on 1.1. While working on 1.1. you are
> writing more tests and you need to ensure that the tests keep working
> on version 1.0 and also want to test against 1.1. So you could e.g.
> create a test group for 1.0 features and run all these test agains the
> 1.0 version to ensure your refactoring of the tests is still ok and in
> a second run e.g. on a CI server run the same test against the 1.1
> version to ensure that existing functionality is not broken in the
> upcoming 1.1. version

This is interesting. I'm not sure where the 1.0 apk would come from though.

Well .. part of a well managed release process of an app imho is that it is build and deployed to a repository server together with e.g. the proguard map file so that stacktraces can be deciphered later and whatever else is important as part of the release. All artifacts that come out of the release process should be stored and managed in a repo server and be available in an archive or archives there.

That in turn then enables the use case I mentioned above because any old apk (release or even development versions since you can deploy them to the repo server too) can be retrieved as a dependency from the repo and then used in e.g. the testing for deployment to the devices the tests run on.. 

>>>
>>> Great... please work with these projects including e.g. Robotium that sits
>>> on top of the instrumentation tests.
>>>
>>> Also when running tests on devices look at implementing a feature that
>>> produces the test results on the machine running the tests and not the
>>> devices. Note that you can look at the Android Maven Plugin codebase where I
>>> implemented that for junit output creation using the ddms library from the
>>> sdk. Then you just have to come up with a naming convention for the files
>>> when you run the tests onto multiple devices.
>>
>> You mean storing the result of the tests on the host machine?
>
> Yes. That way you can e.g. build and run the test against a few
> attached devices and running emulator at the same time and all test
> outputs junit xml files will be available on the host machine in the
> build output folder. That way e.g. they can be picked up easily by a
> CI server without having to pull them off the devices with adb. imho
> this is the more common need than having them on the device. What good
> are they there anyway? This would be good for emma as well btw..

Right that's what I figured. right now the instrumentation runner
prints them I think.

Maybe.. but it does not create the junit xml format which is what build servers and other tools can change into browsable html and so on. The android maven plugin does that.. feel free to lift the code .. it is all apache licensed and I wrote it all so if you need any further copyright transfer or whatever .. I will sign it..  
 
Right now Eclipse gets this output and then display them in the
standard jUnit window.
Ant just output them to the console.
However Ant is able to get the emma data off of a device and process
it locally to create a report.
 
We definitively want to do all of that in here too.


Great.. the android maven plugin can pull down the emma report and so as well but really it should just be created on the host imho.. I see no value of it on the device/emulator.
 
 
>> So it makes a lot more sense to me to have a bundle (zip archive) .... 
 override these settings.

What exactly do you want to override here? I don't think there's any
reason to package a library as simply a standard Java jar file.

 
It has to be an archive that contains the jar plus at least the
android resources (in most cases). We can't have those flat in the
same archive due to possible conflict with the java resources inside
the jar.

The java resources go in classes and the  android ones in res .. there is no real conflict imho. Both of these are needed at runtime.

By lumping all this other stuff into the library zip you are completely overloading what it is used for. E.g. the ide.jar is for eclipse stuff.. so not useful for other ide users. The lint jar is needed at build time but NOT at run time... but it should be e.g. pulled into another library if you have e.g. one library depend on another. And so on.. it is going to get really complex in terms of a build if you have say an application that uses three libraries. Two of which depend on 1 more library each. Now the build system needs to work with say 5 ide.jar files, up to 5 lint.jar files, and so on and it need to know for all those artifacts what they are needed for ... on the other hand if we split it up into the library only containing what is needed at runtime and have additional artifacts that are optional the user can decide to pull them into their build as dependency if they e.g. want those lint check or not. Just like they can do for e.g. custom developed lint checks that are generic and distributed as a jar. imho this would be way cleaner.. 

javadoc and source could be similar artifacts that are shipped inside separately just like is the convention for all systems now (including gradle). E.g. look at http://search.maven.org/#artifactdetails%7Ccom.google.inject%7Cguice%7C3.0%7Cjar

and you can see a bunch of artifacts for google guice 3.0 in the same repository server location. The folder is defined by 

groupId: com.google.inject which translates to com/google/inject
artifactId: guice translates to the next nested folder guice
version: 3.0

So in that folder com/google/inject/guice/3.0

you find

guice-3.0.pom - the file that defines the dependencies of guice
guice-3.0.jar - guice itself
guice-3.0-javadoc.jar - the javadoc for it
guice-3.0-no_aop.jar - a flavour of the jar using no aspected oriented programming (useful e.g. for android)
guice-3.0-sources.jar - the source jar
guice-3.0-tests.jar

Extending this same concept that is proven to work for all build systems using the Central repo including Ant/Ivy, SBT, Gradle, Maven, Leiningen and others for Android libraries we could e.g. add

mylibrary-1.0-lint.jar
mylibrary-1.0-ide.jar

and so on. 

And when you declare a dependency to it it would be done with a classifier just like you do for javadoc and sources now.. 

Once we agree on that point, we may as well just it the rest in the
main archive bundle and call it a day ;) I see little point in
generating such a bundle while omitting some parts of it (be it
source, lint.jar or anything else).

 
;-) it seems like we don't. 
 
I understand this is different from the normal way of doing things in
Maven, but this is very different from just a java build.


Maven does a whole lot more than just Java builds.. but that is a different story.

I already want (and am planning) to allow building unbundled apps with
this scripts.
Those are apps that use public apis only but are built as part of the
Android source tree.


Awesome!!
 
They don't use a normal SDK but instead look at the files in
platform/prebuilts/sdk/

This would be done through a simple property in gradle that lets you
point to a full Android source tree instead of an SDK.
In the builder library there'll be an SDK parser and a parser for the
Android source tree to resolve the compile target and find the
location of android.jar and other tools and files.

So we could technically add a different properties that tells the app
to build using the full api jar instead of the stubbed android.jar
This would only be possible when using a full Android source tree
instead of an SDK, so replacing android.jar itself is not needed.

People who use this will work directly in the source tree and will,
hopefully, know what they are doing (as in, breaking CTS and other fun
stuff)

Thats sound good.. 

> Btw I still think it will be crucial to get all open source apps from
> Google to be built with the new build tool e.g. at least the iosched
> and other examples that use only public api's but ideally also all
> apps in the aosp tree. That would be the best proofing ground for the
> build system!

Yes! Internally, people who work on apps are struggling to build with
the source tree build system (oh, makefiles...) while using our tools.
My goal is to move all those apps to the new build system to let them
use the IDE and only have a single way of configuring how their app is
built.

Great... btw. while you are at it.. do you want to move the sdk build to a new build system too ;-) 

>>>> Lint Integration 
  
So yes, you can build the app with not running lint, but doing "gradle 
build<variant>" will run lint by default always. If you want to
disable it completely (why woud you?) you'll have to manually disable
all the checks.

Ok.. so maybe what we need is a way to configure different sets of lint rules.. like a simple verification one eg. for regular development and stricter one for QA build and so on.. then one of them could if so desired turn everything off. 
 
https://android.googlesource.com/platform/tools/build/+/master


Sounds good.. can you document how to build this branch somewhere? 

Thanks for all the work and I look forward to more coming in ;-) 

manfred
Re: Build System high level feature proposal Xavier Ducrohet 8/23/12 11:23 AM
On Wed, Aug 22, 2012 at 11:32 PM, Manfred Moser <mos...@gmail.com> wrote:
> More answers/discussion inline below
>
>
> On Tuesday, 21 August 2012 13:27:44 UTC-7, Xavier Ducrohet wrote:
>>
> For the image processing I think it would actually be good to be able to
> optimize them in place as a manual step so you can check the optimized
> version back into your source control and then have that process skipped or
> at least run even faster in the future..

We would have to have a global switch where we disable all
optimization. It seems safer to me to
always run this. However I see the point of performance.
How about we don't run it in debug builds but run it in release mode?
(by default, overridable in build types)

>> I think the staging idea is great, and should be advertise a good
>> practice. I'm not sure there's a point in creating a staging type by
>> default as it would essentially be the same as the debug one when you
>> look at the configuration properties above.
>
>
> Fair enough. I guess we can always do that later if we find that there are
> more flags that are actually different. Need to test this and think about it
> a bit more..

Yes, it would be easy to add it later. This is not a structural change
so that's easy.

>> >>>> Testing
>>
>> >> What do you mean by testing different versions?
>> >
>> > E.g. you could have a test module that has UI tests that are
>> > independent of the implementation and you have a version 1.0 on the
>> > play store and are working on 1.1. While working on 1.1. you are
>> > writing more tests and you need to ensure that the tests keep working
>> > on version 1.0 and also want to test against 1.1. So you could e.g.
>> > create a test group for 1.0 features and run all these test agains the
>> > 1.0 version to ensure your refactoring of the tests is still ok and in
>> > a second run e.g. on a CI server run the same test against the 1.1
>> > version to ensure that existing functionality is not broken in the
>> > upcoming 1.1. version
>>
>> This is interesting. I'm not sure where the 1.0 apk would come from
>> though.
>
>
> Well .. part of a well managed release process of an app imho is that it is
> build and deployed to a repository server together with e.g. the proguard
> map file so that stacktraces can be deciphered later and whatever else is
> important as part of the release. All artifacts that come out of the release
> process should be stored and managed in a repo server and be available in an
> archive or archives there.
>
> That in turn then enables the use case I mentioned above because any old apk
> (release or even development versions since you can deploy them to the repo
> server too) can be retrieved as a dependency from the repo and then used in
> e.g. the testing for deployment to the devices the tests run on..

Yeah that's good. We'll have to see how we do that.


> Maybe.. but it does not create the junit xml format which is what build
> servers and other tools can change into browsable html and so on. The
> android maven plugin does that.. feel free to lift the code .. it is all
> apache licensed and I wrote it all so if you need any further copyright
> transfer or whatever .. I will sign it..

Yes, we need to generate those files. As for lifting the code, I can't do that.
You'd have to contribute it to AOSP.

> Great.. the android maven plugin can pull down the emma report and so as
> well but really it should just be created on the host imho.. I see no value
> of it on the device/emulator.

It cannot work any other way.
The test runner is on the device, and it's generating the report.
Nothing running on the device has access to the host computer. So it
has to be generated there and then pulled locally.
I think I see maybe why we are thinking differently.
In Java you could have an app that is made of different jar files all
linked together through clear dependencies (declared in a POM file or
whatever), and those dependencies are resolved at runtime and the jars
are downloaded and the app is run. So yeah having artifacts that are
"optimized" (in content) for runtime is useful.

But this is not how Android works at all. All the jar files are
combined into a single APK during the build and there is no
dependencies resolved nor any artifacts downloaded at runtime.

Android Library projects artifacts are used at build time only.

Even if we ever add the ability to have runtime artifacts, those would
have to be critically different: the Android resources will have to be
compiled and the bytecode will have to be converted to dalvik
bytecode. So this would be a new type artifact different from the one
I described.

So yes we can have separate artifacts for build and runtime, except
right now the runtime one does exist and the one I describe is the
build-only one. In this case we may as well put lint.jar and other
stuff in it :)

As for having to deal with several lint.jar and ide.jar that's really
not a problem. When running lint, we have the list of libraries from
the dependency graph, so gathering those are easy. I really don't see
how that's more complex than already dealing with the same dependency
graph for compilation and packaging. It seems to me that having to
deal with other artifacts is actually more complex.

I really don't see why having to explicitly say that you want to get
the lint checks associated with the library you just added to your
dependencies is a good idea. If you add the library, you should get
the checks automatically.

By the way, ide.jar is not Eclipse specific. We've been very careful
about making it IDE agnostic, like all the other libraries we create.
Right now (it's actually not finished and APIs aren't set yet) it's
mostly about dealing with custom views, which IntelliJ could benefit
from since they have a layout editor based on our rendering library.
Any IDE could benefit, really.


>
> Great... btw. while you are at it.. do you want to move the sdk build to a
> new build system too ;-)

Yes.
There are some changes going in repo right now that will allow us to
download only some git projects instead of the all source tree.
I want to enable this for the sdk projects, and I want to allow
building those separately from the whole tree.

I do want to move all the Java build away from makefiles. However some
other tools (emulator for instance) will still require makefiles. I'd
like to be able to build only the Java libraries without touching
makefiles though.


>>
>> https://android.googlesource.com/platform/tools/build/+/master
>
> Sounds good.. can you document how to build this branch somewhere?

Just use gradle ;)

So right now, this is in its own (new) git project (tools/build) and
only builds with gradle and is independent from the whole build
system.

It generates a local Maven repo under tools/build/repo with a single
fat jar with everything. I'm going to split this into the gradle
plugin and the AndroidBuilder library so that you can look at the
latter.

Xav
Re: Build System high level feature proposal Ricardo Gladwell 8/25/12 5:36 AM
Hi Xavier

Sorry for coming late to this discussion, I've been busy on my own projects but please give me your thoughts on my comments below:

This proposal looks great. Really pleased to see support to include Android Library archives packages. This will make the task of automating builds much easier and get new developers up and running much faster.

I'd like to second Manfred and Joe's support for Maven: I must admit, I find it odd that you're looking at Gradle to replace the build system, given that the ADT is built on Eclipse and the Eclipse project has more mature Maven support.

I'm the maintainer of the Android Connector for M2E[1] which creates a bridge between the ADT and Maven/M2E. So the problems we have are slightly different: the discussion so far has focussed much more on the SDK but I think we should address issues with the ADT as well:

The biggest problem we face is that the ADT isn't built with third-party extensions in mind. We rely on making calls to internal OSGi ADT APIs and these frequently change, breaking out project between releases. We have opened a ticket for this[2] but this is not close to being resolved and I don't think it addresses a deeper point about refactoring the ADT to be more extensible.

Another problem is that there is a large mismatch between the configuration the Android SDK command line tools support and the configuration the ADT supports.

For example, the android-maven-plugin integrates directly with the command line utilities and therefore supports the ability to disable /libs/ folder dependency look-up or ignore duplicate class errors on DEX compilation. Yet the ADT doesn't support this.

Our job would be much easier if the SDK and the ADT matched up more feature-wise as well as  making the ADT more extensible.

One possibility would be to simply contribute the m2e-android project to Android development team as an optional plugin installed as part of the ADT. Supporting this would mean users get more consistent Maven support, as well as making ADT developers more aware of the issues of making the ADT more extensible making it easier, for example, to write Gradle-ADT Eclipse plugins and so forth.

[1] http://rgladwell.github.com/m2e-android
[2] http://code.google.com/p/android/issues/detail?id=27811

--
Ricardo
Twitter: @rgladwell



On Saturday, 4 August 2012 01:36:15 UTC+1, Xavier Ducrohet wrote:
As I've mentioned before, we are looking are a new version of the build system provided in the SDK.

Our goal is to make it easy for beginners but flexible and powerful enough for teams working on large complex application(s).

We started looking at the high level features that the build system would offer. This is independent from the implementation of the build system and how those features would be exposed to devs (this will come later). The following is a proposal for these features (not including the basic features currently available, those are a given), and we are looking for feedback.

So what are we trying to do?

  • Better extensibility
    • Allow devs to configure every steps.
      • expose all options for aapt, dex, …
      • allow changing input/output for all steps.
        • For instance, bypass proguard for a given jar file.
        • Use different jar for compile vs. package
    • Better control over debug vs. release builds
      • manifest info
      • BuildConfig
      • jni compilation
      • different res/code depending on build.
    • Let devs add new logic in the build system
      • New tasks
      • Change dependencies
      • Integrate into other build systems.
  • Help devs making different versions of the same app.
    • Library Projects can be cumbersome to make 2 apps that are 99% identical.
    • Different versions for different app store could only differ in package name and signing key.
  • Better dependency management for 3rd party jar files and Library Projects
    • Use Maven/Ivy for dependency management
    • Depend on Library Projects distributed as binary blob
    • Generate Library Projects blob


To implement this, we are introducing new build concepts.

Product Flavors

A product flavor defines a customized version of the application build by the project. A single project can have different flavors which change the generated application.

Although different flavors could be very different applications, using Library Projects is a better use case for this, and the build system still supports this.
This new concept is designed to help when the differences are very, very minimum, for instance when using Google Play’s multi-apk support (e.g. the difference could be only gl textures compressed using a different format.)
If the answer to “Is this the same application?” is yes, then this is probably the way to go over Library Projects.

Product flavors can customize the following properties:
  • minSdkVersion
  • targetSdkVersion
  • versionCode
  • versionName
  • package name (overrides value from manifest)
  • release signing info (keystore, key alias, passwords,...).
  • native abi filter
  • test info
    • package name for test app (optional, default is <base>.test)
    • InstrumentationTestRunner class (optional)

Additionally, Product Flavor can provide their own source code, resources and manifest.

A project starts with a main product configuration. All the above properties can be set and it has its own source code, resources and manifest.
If the project does not create product flavors, then the application is built from this main configuration.

When a project declares Product Flavors, these extends the main configuration.

The source code of the product flavor is used in addition to the source code of the main configuration (1).
The resources of the product flavor overrides the resources of the main configuration.
The Manifest of the product flavor is merged on top of the Manifest of the main configuration.

(1) This allows some flexibility but also adds some restrictions:
  • Classes in the main configuration and in the product flavors can reference each other. All the source sets are used to generate a single output (instead of each generating its own output with a one-way dependency).
  • Classes exposed by product flavors and referenced by classes in the main configuration must be present in all product flavors, and must have the same API for all builds to succeed.

When a Product Flavor is defined, the main configuration is not available as a buildable product. To create more than one product, two or more flavors must be created and customized.

Build Types

A build type allows configuration of how an application is packaged for debugging or release purpose.

This concept is not meant to be used to create different versions of the same application. This is orthogonal to Product Flavor.

A Build Type provides configuration for the following build properties:
  • manifest debuggable flag
  • native compilation debug flag
  • proguard enabled + rules
  • debug signing flag (ie whether to use debug key or release key)
  • package name suffix (2)
  • Buildconfig
    • DEBUG flag
    • other custom properties?

By default the build system provides two build types, debug and release, which can be reconfigured. New types can also be created to provide any combinations of the above properties.
For instance, one could make a Build Type that uses Proguard but creates a debuggable apk.

Sourcesets

Due to Build Types and Product Flavors all providing their own sources, resources and manifest, the project structure need to change from the root level src/res/manifest used in the current build system.

Instead the build system moves to a top level src folder, containing folders for each types and flavors. In addition, the main folder contains the source for the main configuration.

A typical project will then have the following folder structure:

  • src
    • main
      • java
      • jni
      • resources -- this is java resources
      • aidl
      • res
      • res -- this is Android resources
      • assets
      • AndroidManifest.xml
    • debug
      • … (same as main)
    • release
      • … (same as main)
    • flavor1
      • … (same as main)
    • flavor2
      • … (same as main)

The current gen folder is to move under the project output (build folder), and follow a similar pattern. This is accompanied by the generated resources, and the generated (merged) Manifest.


This is of course be completely customizable, but this shows the default, and the type of inputs that are available to tasks.

BuildConfig

BuildConfig is an automatically generated class with a single boolean DEBUG, set to true in debug builds only.

With custom source sets for both Build Types and Product Flavors, the build system covers adding extra flags and triggers (through a user created class) to control what the code does in each build variants. It is a manual process to maintain each version of the same class but it is the most flexible for devs who can do anything they want.
<span style="font-si
...
Re: Build System high level feature proposal Xavier Ducrohet 8/28/12 11:16 PM
Hi Ricardo,

I completely recognize the lack of extensibility of ADT but to me this is orthogonal to the build system (sort of, see below), and I'd prefer we keep this thread on the topic of build features and implementation.

Yes we would like to make ADT extensible, but it has to be driven by a use case (yours is fine). I'm not sure why you require so many APIs though, and making those public, maintained APIs is a huge undertaking. We should discuss this in a different thread.

The mismatch between the command line and ADT is a big issue indeed. We definitively want to use a single build system for both. We're designing the build system first. We'll integrate it in the IDE after.

As for contributing m2e-android to aosp, why not. It would help make sure it's up to date. I would also like to hear from the maintainers of the maven plugin for Android what they think of moving the plug-in to use the AndroidBuilder library (when it's finisehd. I think it would be nice and would ensure that all new build features are supported automatically (or close to it) and would ensure that there is parity between Gradle and Maven and that both builds do the same thing.

Getting to a point where you can use ADT with either Maven or Gradle sounds like a great goal to have. Definitively something we should look into when we work on the integration with the IDE.


  • src
    • <span style="vertical-align:baseline
...
Re: Build System high level feature proposal Manfred Moser 8/29/12 12:13 AM
On Tue, Aug 28, 2012 at 11:16 PM, Xavier Ducrohet <x...@android.com> wrote:
> I would also like to hear from the maintainers of the maven
> plugin for Android what they think of moving the plug-in to use the
> AndroidBuilder library (when it's finisehd. I think it would be nice and
> would ensure that all new build features are supported automatically (or
> close to it) and would ensure that there is parity between Gradle and Maven
> and that both builds do the same thing.

That is definitely the plan for the android maven plugin from my point
of view. But it will have to rely on contributions from others as well
as continue to support our current users and features so we will have
to see how that works out..

I definitely plan to refactor things to more and more use the builder
library. Doing that will probably also drive changes to the library so
it will take some time.

manfred
http://simpligility.com
Re: Build System high level feature proposal Ricardo Gladwell 8/29/12 3:24 AM
Hi Xavier

On Tue, 2012-08-28 at 23:16 -0700, Xavier Ducrohet wrote:
>
> I completely recognize the lack of extensibility of ADT but to me this
> is orthogonal to the build system (sort of, see below), and I'd prefer
> we keep this thread on the topic of build features and implementation.

Fair enough, would you like me to start a new thread now or leave things
until the build system is sorted out?

Regards...

--
Ricardo
Twitter: @rgladwell

Re: Build System high level feature proposal Xavier Ducrohet 8/29/12 10:46 AM
Just start a new thread.
Re: Build System high level feature proposal Xavier Ducrohet 8/29/12 10:47 AM
That's great.
I'm still changing the library a lot, but I definitively want to make
sure it's usable by Maven.
Re: Build System high level feature proposal Ricardo Gladwell 8/31/12 5:19 AM
Another quick question on this topic: will you also be changing the
tools that build the SDK/ADT to Gradle as well? Or will Gradle just be
for consumers of the SDK/ADT?

--
Ricardo
Twitter: @rgladwell
Re: Build System high level feature proposal Xavier Ducrohet 8/31/12 10:00 AM
I'm looking at building all our jars with Gradle. Eclipse is more complex so I'm not sure what we'll do there.
More topics »