|Build System high level feature proposal||Xavier Ducrohet||8/3/12 5:36 PM|
As I've mentioned before, we are looking are a new version of the build system provided in the SDK.
Our goal is to make it easy for beginners but flexible and powerful enough for teams working on large complex application(s).
We started looking at the high level features that the build system would offer. This is independent from the implementation of the build system and how those features would be exposed to devs (this will come later). The following is a proposal for these features (not including the basic features currently available, those are a given), and we are looking for feedback.
So what are we trying to do?
To implement this, we are introducing new build concepts.
Product FlavorsA product flavor defines a customized version of the application build by the project. A single project can have different flavors which change the generated application.
Although different flavors could be very different applications, using Library Projects is a better use case for this, and the build system still supports this.
This new concept is designed to help when the differences are very, very minimum, for instance when using Google Play’s multi-apk support (e.g. the difference could be only gl textures compressed using a different format.)
If the answer to “Is this the same application?” is yes, then this is probably the way to go over Library Projects.
Product flavors can customize the following properties:
Additionally, Product Flavor can provide their own source code, resources and manifest.
A project starts with a main product configuration. All the above properties can be set and it has its own source code, resources and manifest.
If the project does not create product flavors, then the application is built from this main configuration.
When a project declares Product Flavors, these extends the main configuration.
The source code of the product flavor is used in addition to the source code of the main configuration (1).
The resources of the product flavor overrides the resources of the main configuration.
The Manifest of the product flavor is merged on top of the Manifest of the main configuration.
(1) This allows some flexibility but also adds some restrictions:
When a Product Flavor is defined, the main configuration is not available as a buildable product. To create more than one product, two or more flavors must be created and customized.
Build TypesA build type allows configuration of how an application is packaged for debugging or release purpose.
This concept is not meant to be used to create different versions of the same application. This is orthogonal to Product Flavor.
A Build Type provides configuration for the following build properties:
By default the build system provides two build types, debug and release, which can be reconfigured. New types can also be created to provide any combinations of the above properties.
For instance, one could make a Build Type that uses Proguard but creates a debuggable apk.
(2) This is appended to the manifest package name. The goal here is allow having different build types create different apks that can all be installed at the same time. This is optional.
Like Product Flavors, Build Types can provide their own source code, resources and manifest.
The typical use cases are:
(3) If Product Flavors are used to create different versions built with different keys, the MapView API Key can be provided by the resources of the Product Flavor instead. See Build Variants below.
The source code and resources provided by the Build Types affects the project the same way Product Flavors do.
Build VariantsWe’ve seen Product Flavors and Build Types and how both configure the output of a project.
In fact, the output of a project can only be the cross product of the Product Flavor and the Build Type. This is call a build variant.
It is not possible to build a project without selecting both a Build Type and a Product Flavor.
For a project not customizing any Build Type or Product Flavor, the available build variants are debug and release versions, as the main configuration is implied.
However, if a project declares two flavors, for instance free and forpay, the available build variants are:
We have seen above how source code and resources provided by Build Types and Product Flavors extend or override the main configuration. When actually building an app, the Build Type and Product Flavor are both applied with the following rules:
All the source folders are used together (main config, build type, product flavor, and generated source code(4)) to create a single output.
This means the same restrictions mentioned above applies, with the addition that Build Types and Product Flavors cannot provide different versions of the same class.
(4) generated source is output from resource compilation (R.java), aidl and renderscript compilation, and things like BuildConfig which can be impacted by the BuildType. These are different per build variants.
Resources combination uses the following priority (lower number is higher priority and override higher number):
(5) These are the outputs of the image crunch step, and of the renderscript compilation.
Packaging of Assets, Java Resources, JNI libraries follow a similar priority when encountering duplicates:
SourcesetsDue to Build Types and Product Flavors all providing their own sources, resources and manifest, the project structure need to change from the root level src/res/manifest used in the current build system.
Instead the build system moves to a top level src folder, containing folders for each types and flavors. In addition, the main folder contains the source for the main configuration.
A typical project will then have the following folder structure:
The current gen folder
|Re: Build System high level feature proposal||Jake Wharton||8/4/12 12:28 AM|
I have complained very openly, loudly, and in arguably to excess about the lack of a proper build system for Android. The current system is unacceptable with regard to scalability and the breadth at which it needs to be applied. Being no fault of you or your team, the legacy system is simply under-equipped and lacking any true semblance of dynamism to deal with modern requirements.
While I represent only a single person (and occasionally a part of a small company), I cannot stress enough how useful nearly everything you mentioned above will be to both beginners and advanced users alike. Please make every effort to involve the community in testing early iterations and considerations of this new system upon implementation. Nearly everything that you say seems exceedingly relevant to my and my company's interests but true discoverability comes through usage.
I cannot describe how much I want to test this new system during the development stage so please, please, please allow the community to be involved as early as possible. While we have a ridiculous amount of requisite configurations, the benefit can only make this new system even better.
Thank you for the work you've already done and for the work on this new system that we're about to embark on.
|Re: Build System high level feature proposal||Alexei Kuznetsov||8/4/12 12:28 AM|
How about this one:
- drop ant and have s...
|Re: Build System high level feature proposal||Andrew Kelly||8/4/12 12:51 AM|
As an Android developer the main problem I had with the existing build system was when integrating it with Bamboo. The project.properties file is used by Eclipse to find the required libraries used by my app, but my local filesystem layout is completely different to the layout used by Bamboo. This means that the SVN managed version of the file has to have one set of directory paths and my local version has to have another set...and I must remember not to commit these changes back to the SVN repository.
I tried using a maven plugin to get around this problem I believe the plugin zips up library projects at build time then extracts them as dependancies when they're references in the main pom.xml file...this again seems a little clunky.
I've used maven on other J2EE projects and the dependancy management has been great, it would be nice to see a solution to Android builds which can utilise this system better. And this would make using versioned libraries easier too, as legacy builds would pull the older library from the maven (or Nexus) repository.
Many of the topics in your post seem like great features, but are things that I've never come across or needed, perhaps my projects aren't too complex. But the ability to build locally and using Bamboo easily when referencing library projects would be a great benefit to me (and hopefully others) personally.
|Re: Build System high level feature proposal||Nicolas Klein||8/4/12 12:54 AM|
This new system looks really promising. Not really useful for the beginner developer which is only going to use the default option everywhere of course. But for the advanced/expert user, it opens so much more possibility :)
I don't have the same complexity/needs as Jake but really can't wait to be able to beta-test it :)
Thanks for the work of the whole ADT team once again :)
Nicolas Klein aka Foxykeep...
|Re: Build System high level feature proposal||Jake Wharton||8/4/12 1:15 AM|
I just want to follow up on a great point that you make here, Nicolas, the fact that "the beginner developer [who] is only going to use the default option everywhere." There have been skeptics who've commented as a response to my heat-of-the-moment blog post who think that somehow this system will be of no benefit (or even worse so that it would be a detractor) to the average developer. They could not be more wrong.
With the prevalence of the v4 support library and the addition of the v7 support library and libraries like AppCompat and ActionBarSherlock it is a farce to think that the majority of developers will not need dependency management and resolution.
I cannot stress the following point enough: if you every have to download a .jar manually, you've lost. If you have to manually resolve a version difference between projects, you've lost. If you have you create an IDE project for a library dependency, you've lost.
Proper dependency management is a magical experience and I cannot wait for the every single Android developer to realize the simplicity and elegance of when it's done properly. Once experienced you will continuously wonder how you ever managed to copy .jars into various libs/ folders...
|Re: Build System high level feature proposal||Xavier Ducrohet||8/4/12 1:42 AM|
First, I'm glad you find some of those features useful.
Yes we do want to involve people very early on. This was a first step of hopefully many.
It's going to take us a while to get the new build system to the point where we unleash it on every devs out there. So for a while we'll keep Ant/Eclipse around and have this new one available for people to test and give feedback.
As all other tools it'll be developed in the open so you can stay very close to where we are....
|Re: Build System high level feature proposal||Xavier Ducrohet||8/4/12 1:46 AM|
First we want it to be easy to just create a project and build it with little (if any) configuration.
After that we do want people to progressively extend their build as they need. But realistically, discovery of how to do things is never going to be trivial. Even something like setting a dependency on a lib coming from Maven requires some basic knowledge that devs who never used Maven don't have.
Proper documentation on developer.android.com will be critical....
|Re: Build System high level feature proposal||Xavier Ducrohet||8/4/12 1:55 AM|
I have explicitly said at the beginning:
If you are trying to convince me we can't implement our goals using Ant, there's no need, I'm already convinced.
However, you say "languages", plural. I fear some people are going to be disappointed. Even if you remove some of the more advanced features described above, building a basic Android app takes a LOT of steps with a lot of logic between those steps.
To make a build system that is truly flexible and extensible for developers while making it build agnostic so that anyone can use it using whatever tech they want (Maven, ant, gradle, sbt, Make, whatever...) seems totally impossible to me.
We are going to choose one, and stick to it. We will make the IDE use this same build system (this is a critical point).
We won't be able to support other build system (besides making whatever code we have available under an easy license, like we do today).
Some people are not going to like it. The best we can do is do the bes...
|Re: Build System high level feature proposal||Mark Murphy (a Commons Guy)||8/4/12 3:47 AM|
On Sat, Aug 4, 2012 at 4:15 AM, Jake Wharton <jakew...@gmail.com> wrote:
> I just want to follow up on a great point that you make here, Nicolas, theI suspect that you are referring to me:
Replace "will not need" with "could benefit from", and I'd be in
agreement. But the further problem is ensuring that the ramp-up to
full-on dependency management and resolution is *graceful* and takes
into account that, based on what I am seeing out in the support
- A majority of people coming into Android now have limited Java
programming experience, and a fair number have no programming
- A majority of people coming into Android now are not native English
speakers, yet the documentation and support resources are largely in
English, thereby magnifying the effects of complexity
The #1 problem I see with the design...
|Re: Build System high level feature proposal||Richard Critten||8/4/12 7:25 AM|
I agree with all of Mark's comments and would like to add a comment about documentation.
There are other build systems out there but integrating with them is difficult because of the lack of documentation of the existing build tools. The one I keep running up against is AAPT. I think I can workout how to use by reverse engineering the Ant scripts - but where is the documentation?
Second a question: how do you see your build system proposal integrating with the NDK build system based around make?...
|Re: Build System high level feature proposal||Fred Grott||8/4/12 9:49 AM|
I would like to point out a certain strategy. It may not work in all use cases.
Because, I knew that there was work on this I redid my own codeqa build stuff. I simply let the IDE build the project before the codeqa build script run.
For me it works as I do not have very complex, multi test projects.
As far as proposal do not attempt to re-invent the wheel look at the Gradle Android plugin and borrow as many ideas as possible. I think the Gradle, groovy as build script, might fit but we never had enough developers bandwidth on the Gradle Android project to keep up with Android Tools changes. As far as dependency management Gradle has not only maven, ivy but also can do file systems like the installed Sdk for example.
There is also a python ant-tools android project but its not as far as the Gradle Android plugin and from my analysis it woudl requrie double the dev time to do it in python as opposed to doing it ing Groovy via Gradle.
The Gradle Android plugin project is here:
Please Xavier take a look and at least borrow the ideas. so you can see the double amount of work using python as oposed to Groovy/gradle the python android build project:
As the python project does not have dependency integration.
There is also Apache Buildr which is ruby build system with dependency integration but its in the same boat as the python project.
Gradle Android plugin would have the nice thing of the build scripts would be understandable Groovy which is understandable to most java programmers.
|Re: Build System high level feature proposal||Manfred Moser||8/4/12 9:54 AM|
Thanks for writing up this proposal. I will have a more detailed look and feedback in a week or so but here are a few first thoughts
- all artifacts should be available in a repository for the dependency management system (libraries open source like android.jar, compat lib .. as well as closed source that are used for Android development like admob .. as well as sdk internal libraries) so they can easily be used by other IDE's and tools as well as during the build process, the repository should use the standard maven repo format and ideally be the central repository so it can be used by ivy, gradle, maven, sbt, clojure and anything else that can do http gets..(http://search.maven.org)
- all aosp internal android apps should use the new build system (dogfooding!!!) and not be built with make
- all example apps should be set up to use it as well
- google example apps like iosched and others should use the new build system
- there should be no dependencies of the build system to some IDE specific project files like currently lint relies on the eclipse directory structure and project files
- ideally it should leverage an existing build system and the associated knowledge and community, it should NOT be a new build system but rather an extension of an existing one (don't fall into the Not Invented Here trap)
|Re: Build System high level feature proposal||Jake Wharton||8/4/12 11:51 AM|
The #1 problem I see with the design as proposed is that it
This should almost never be a limiting factor in software development and definitely not for this endeavor. If nobody ever wanted to break things we be slumming it up in decades-old tools and using far-too-long-in-the-tooth techniques. First understand what ant's benefits were and where it came short (if you like long lists) and then build something from scratch that scales down to the beginner and up to the advancer.
Bump the version number (build system 2 woo!), deprecate but support the old one for 6 months, and ship it. The bookers will book, the webbers will web, and the community will be dragged, kicking and screaming, to a better system for everyone. Good riddance ant(iquated).
Why does the fact that it's new have to mean that it will be more complicated? The base-case right now for a "Hello, World" app can be greatly simplified. I absolutely should not need a build.xml, project.properties, and local.properties to compile an apk that exists as src/, res/, and AndroidManifest.xml. Simply executing android should build with whatever is the target SDK version is set to by default (or the latest if not set) and use the ANDROID_HOME environment variable to find the SDK (if it's not present, prompt and set it).
|Re: Build System high level feature proposal||Xavier Ducrohet||8/4/12 11:58 AM|
Thanks all for your comments.
I'd like to refocus the discussion on the features themselves, not on how we will implement it.
To do that, some details about the implementation:
- As I mentioned a week or so ago on the list: yes we're not going to reinvent the wheel, we want to build on top of an existing build toolkit. Yes we are looking at gradle. We are actually prototyping the build using gradle.
- Yes having a dependency management will mean the android artifacts will have to live in a maven repository somewhere. As I said though, while this is a starting point for a lot of build type, it's not going to help people wanting to use sbt for instance.
Alright now to some more interesting comments.
Mark, you have a very valid points. There is a ton of doc out there that's already out dated. I'm not sure how we can completely deal with this. The current system is broken, and trying to migrate gracefully from it might prevent us from using a better solution (at least with regard to the multiple sourcesets).
One thing is for sure: we do want to have both build systems at the same time before the new one becomes for the default one. During that time we'll have to fully document the new one as well as migrating from one to the other (our current doc is severely lacking in that area I think, and it is critical it gets better). Also the "android create project" will handle both build type. We can even have "android update project" create a new build script using the old project structure.
I hope people who have no (or limited) programming experience will go read the doc (or hire you!). If they don't there isn't much we can do about it.
I'll add that I do know we've been pretty bad at doing changes in the build scripts that woud break people's build. We certainly don't have a good rep around this right now. One of the reason to have this build tested by a lot of devs before it becomes the standard build. We can't screw it up this time.
Richard, the aapt command line has a fairly extensive command line help. You should look at it.
Regarding NDK build we do intend to simply call out to Make (ndk_build actually) passing some parameters that are currently in the makefile to properly integrat...
|Re: Build System high level feature proposal||Xavier Ducrohet||8/4/12 12:04 PM|
On Sat, Aug 4, 2012 at 11:51 AM, Jake Wharton <jakew...@gmail.com> wrote:I agree. We do need to provide as much side support as possible, but
we can't completely limit ourselves because of the old system.
> simplified. I abs...
|Re: Build System high level feature proposal||Mark Murphy (a Commons Guy)||8/4/12 12:40 PM|
On Sat, Aug 4, 2012 at 2:58 PM, Xavier Ducrohet <x...@android.com> wrote:> time before the new one becomes for the default one...
> I'll add that I do know we've been pretty bad at doing changes in the buildI had been assuming that this proposed build system change would
necessitate a hard-slam cutover to a new system, since that has been
the modus operandi for prior changes. If you are planning on having
the two supported in parallel for a significant period of time (e.g.,
Jake's suggested six months), that solves a fair number of problems,
|Re: Build System high level feature proposal||Dirk Jäckel||8/4/12 1:03 PM|
1. I would like to have an efficient build system that caches all the intermediate results and only rebuilds whats really necessary. The build system should know the dependencies for every artifact it creates. Not only for the class files but also for dex-files and other generated files. If I use Scala it should extract and dex the Scala libraries class files only once. The build system should only compile/create/copy/modify/filter whats absolutely necessary (and nothing more or less).
2. It would be great if it does not have to ...
|Re: Build System high level feature proposal||Xavier Ducrohet||8/4/12 2:00 PM|
yes of course. I should have mentioned this too.
Step 0 is spec'ing the features here, and start of implementation.
Step 1 is people testing it as a custom download. People here, and
whoever else want to give feedback. Hopefully we can do quick
Once we have something robust with most APIs done, we can move to:
Step 2a: package it in the SDK but this is not the default build
Step 2b: package it and make it the default, but keep Ant
Step 3: remove Ant.
Step 2 will take as long as it has to, and as long as Ant can stay viable.
> the modus operandi for prior changes. If you are planni...
|Re: Build System high level feature proposal||Xavier Ducrohet||8/4/12 2:07 PM|
On Sat, Aug 4, 2012 at 1:03 PM, Dirk Jäckel <dirk.j...@gmail.com> wrote:We have part of this already in both Ant and Eclipse, but yes dex is
the missing part. We are working on this, and we'll have it in
> 2. It would be great if it does not have to create an APK and install that on the emulator/device but only send the changed files, if there is already an installation on the emulator/device.
> This would really speed up development turn a...
|Re: Build System high level feature proposal||Mark Murphy (a Commons Guy)||8/4/12 2:08 PM|
On Sat, Aug 4, 2012 at 5:00 PM, Xavier Ducrohet <x...@android.com> wrote:> Step 2b: package it and make it ...
|Re: Build System high level feature proposal||Joe Bowbeer||8/4/12 2:33 PM|
I generally like the proposed features....
The build variants (combinations of Product Flavor and Build Type) are similar to features that I've had to build on top of the existing build system. In one product, before library projects were introduced, we had a main project and five differently-branded project flavors. When library projects were introduced, the main project morphed into a library project -- but it is really a main project by nature, and this should be easy to express with your new feature set. In another product, there are 100's of flavors that are only known at build time, so I'm hoping that the new system will support dynamic flavors. More on this below. I've never needed more than a handful of build types, by the way, so what you propose seems fine.
Questions, comments, suggestions:
1. What about sharing of portable Java code?
In the past, I've gained a lot of leverage by splitting most of the logic into a platform-independent Java project that could be tested with JUnit, and even run from the command line with very rapid turnaround. In the first product described above, this was implemented as a Java project whose source was included by the main Android project, plus another Java project that added the command-line interface.
2. Where is the libs folder? Where do the 3rd-party .jars go?
This is related to question #1 about leveraging portable Java code. The most common way to do this is to add the .jars to the libs folder, but I don't see this folder in your outline. Where are the libs? Are there separate libs per flavor, or what?
3. Packing the main project as a library for 3rd-party flavoring.
I need to package the main project in some form (library blob?) so that I can distribute it to 3rd parties for added flavoring and repackaging. Will this be supported?
4. Dynamic flavors
This was mentioned in the first paragraph. I need to be able to create flavors dynamically during the build. I'm imaging a hook or step in the build process that will let me inject a custom flavor into the build. I don't care if this ability is present in the IDE, because IDE builds are for developer use only and are not distributed to others, but it is essential for the batch build system that creates the "real" builds that are tested and distributed.
5. Native code?
I'm also concerned, as are others, about integration with the jni code and its build system. Native code has become increasingly important in a larger and larger set of applications.
Currently, in a cobbled-together build script that I am familiar with, the build variant information is passed to the NDK make file using environment variables (which seems to be the only way to do this), and the jni is rebuilt for every different build variant. That is, it is rebuilt from scratch every time the build variant changes. The Android.mk file converts the build variant environment variables into a set of #defines that result in conditional compilation of large sections of code throughout the jni layer. This raises a question about where this native code will live in the new build layout, and how the build variant information is to be conveyed to the jni build.
6. *the* IDE?
I'm concerned about the reference to "the" IDE. Also the mention of an ide.jar. (Which ide.jar?)
As my projects have become more complex, as is the nature of real applications, the Eclipse ADT has become more and more a PITA. I strongly urge some IDE-agnosticism, for everyone's sake. Currently, all major IDE's (Eclipse, IDEA, NetBeans) have solid support for both Ant and Maven, so it makes sense for the new build system to leverage these tools.
I'm currently a fan of the NetBeans Android plugin because its project model is exactly the Ant model represented by the AndroidManifest and the .properties files. NetBeans is the only IDE that has been able to quickly and reliably accommodate the many changes in Android's build scripts and tools starting with r14. Granted, NetBeans, my choice, is not the most popular IDE, but its approach demonstrates the power and economy of leveraging the standards -- and keeping things simple in the face of rapid change.
[ Aside: What is the most popular IDE among serious Android developers? I would venture that it is now IDEA? ]
I should mention that I'm not a user of the ADT layout editor or other fancier UI tools. And I don't know a single developer of a commercial Android app who uses these GUI tools. I think these editors are valuable as an easy-on-ramp for beginners, but then they quickly lose their appeal. I hope, of course, that this will change, because I believe in the power of a good IDE. However, I think the development of layout editors and great RAD tools is probably something that 3rd parties can do better than Google -- if they have the hooks/api/stability to enable it.
I should also mention that I'm still waiting for source-level native code debugging that actually works in practice. (That would be cool...)
7. Maven: +1
I would likely be using a Maven Android plugin if Maven were supported by the Android Tools team. Unless the Android Tools team supports Maven, however, I will continue to avoid it for commercial apk development, because I can't risk the version lag, etc.
8. Lint not tied to IDE
I second this recommendation from one respondent. Please make Lint IDE-agnostic.
|Re: Build System high level feature proposal||James Moore||8/4/12 4:01 PM|
It looks like this provides the right hooks we need to improve Scala
integration in Eclipse. However, it'd be great to see some sort of
preshrink/predex built into the build system, and not just for Scala.
The Scala problem is the 64k method reference limit issue (lots of
links at http://blog.restphone.com/2012/07/scala-on-android-what-you-need-to-know.html).
Briefly, the Scala standard libraries are too large to fit on
Android. It's not at all Scala-specific though; I suspect that more
and more large projects are going to hit that limit.
Traditional Scala development gets around the problem by running
Proguard on every build. Slow and painful, but it works. I write an
Eclipse plugin to speed this up (it caches Proguarded jars and uses
the one that it thinks matches the current build, so you end up with
far fewer Proguard runs). Hooks for this built into the build system
you're talking about would improve the user experience - in
particular, I'd definitely use the hook that says "use $SOME_JAR for
code completion, but replace it with
$SOME_JAR_THATS_BEEN_RUN_THROUGH_SOMETHING_LIKE_PROGUARD for dexing".
In a perfect world I'd love for my plugin to be ...
|Re: Build System high level feature proposal||Xavier Ducrohet||8/4/12 5:08 PM|
On Sat, Aug 4, 2012 at 2:33 PM, Joe Bowbeer <joe.b...@gmail.com> wrote:Can you describe how the flavors are known at build time? How are they
If there is some file+metadata that describe them, it should be fairly
easy to put some logic in the build script that process this data and
dynamically create new flavors.
Yes this would be supported. Multi-projects build will allow this, or
you can create a jar from it and reference it.
First you should really reference dependency using Maven or Ivy if
this is a 3rd party library.
If this is your own, you could create a dependency on a locale file.
It's up to you to put it where you want.
With gradle we aren't as tied as libs/ are we were with Ant.
I think this would have to be a library. The added flavoring can come
from a project using this library.
Yes. As I mentioned above this should be easy.
This is interesting.
Right now the ndk relies on its own build system because of the
complexity of building native code and dealing with several
We intend to keep this and have the build system call out to this.
One of the param is whether you want debug code or not. For instance
you can call "ndk_builld DEBUG=1" for a debug build. This will be done
automatically if the build type indicates that we want the jni code to
We will have to extends the params we can pass to ndk_build. For
instance output location is one param we'll have to add.
Also ndk_build does parse the manifest to figure out some info
(minSdkVersion at least) but with the build variants we'll have to
pass it instead.
I'm really interested in knowing what kind of data you want to be able
to pass. I'm guessing at least #define values would be valuable.
From the flavor perspective this could be set similarly to the custom
fields set in BuildConfig.
This is both tricky and critical. NetBeans has the advantage that it
relies solely on Ant to descrive the project. any change in there is
Ultimately we want the same. This is the only way to make the new
build system viable. It has to be 100% supported in the ide.
We'll have to see how we can make this work in IntelliJ and Eclipse.
>> There are no Product Flavor. This would not make sense as the output is
>> not a Product. Different versions of the same Library should be different
>> libraries and use different projects (possibly depending on the same root
>> There are debug and release build types but they are used slightly
>> Testing the library is important and this is done through the test
>> component of the project. However, unlike a normal project, this component
>> doesn’t generate a separate application from only its source set. Instead it
>> implicitly references the library itself and packages it in its own apk.
>> This build is always a debug build.
>> When the Library Project is built and packaged into a distribution blob,
>> then the build is always a release build.
>> Note that the Build Type can here only configure the following properties:
>> native compilation debug flag
>> Buildconfig>> The other properties either don’t apply, or only one possible value makes
>> Libraries must be packaged in a very specific way, no matter what the
>> library source folder structure is.
>> classes.jar -- this is the main code used for compile and packaging.
>> api.jar -- this is an optional jar to used at compile time instead of
>> res/ -- this is the res folder(s) extended with:
>> crunched images
>> renderscript bitcode
>> AndroidManifest.xml (includes the merging from other Library Projects?)
>> lib/ -- this is the jni component(s)
>> aidl -- this is a straight copy of all the src/../aidl files
>> In all the above cases what is packaged can be extended by the release
>> Build Type, similar to a normal apk.
>> Additionally, the blob may contain:
>> proguard.txt -- custom Proguard rules
>> lint.jar -- custom lint rules
>> ide.jar -- custom IDE extensions
>> docs/ -- this is the javadoc
>> source/ -- this is optional source package (for open source libraries).
>> Using Library Projects.
>> Library projects can be used two ways:
>> Dependency through a repository (local/maven/ivy)
>> this downloads the blob and unarchives it somewhere
>> As part of a multi-project build.
>> #1 is fairly standard and can be added as as normal dependency.
>> #2 is a bit more complicated and needs to be figured out. The main issue
>> is that dependency on Library Projects have a clear dependency order and the
>> setup of the dependency must clearly spell this out.
>> Looking at IDE integration, #2 is important for cases where Library
>> Projects are not distributed but are instead just a way to modularize an
>> application. This would allow better experience in the IDE (for instance
>> allowing refactoring across the main app and the library(ies.))
>> Details TBD.
>> There are 2 main types of dependency configurations:
>> While package will in most case include the content of compile, it is
>> important to be able to have a dependency that is compile only. This allows
>> having a different version of a dependency for compilation vs. packaging.
>> Details TBD.
>> It should also be possible to have a dependency skip the Proguard step.
>> Note that the Android API jar(s) is automatically added to the compile
>> (only) dependency list, and doesn’t have to be declared manually.
>> Product Flavors (not Build Type) impact dependencies, as well as tests.
>> This means other dependency configurations are automatically created:
>> Lint Integration
>> The new build system integrates Lint at two levels.
>> Running Lint
>> A Lint tasks is created and allows testing any build variant.
>> Lint errors will break the build.
>> Lint rules can be enabled/disabled and set priority (warning/error).
>> HTML and/or XML reports can be generated.
>> Lint should run on all available Build Variants but remove duplicates
>> (since code common to all variants is compiled for each of them)
>> Extending Lint
>> Creating custom Lint is directly possible using the src/lint/ folder.
>> This creates a lint.jar file that is passed to Lint when running it on the
>> project itself. The dependency on the Lint API is done automatically. All
>> that’s needed is creating the source folder.
>> If the project depends on Library Projects, their own custom rules are
>> used as well.
>> Feedback welcome. thanks.
>> Xavier Ducrohet
>> Android SDK Tech Lead
>> Google Inc.
>> http://developer.android.com | http://tools.android.com
>> Please do not send me questions directly. Thanks!
>> This is of course be comp...
|Re: Build System high level feature proposal||Xavier Ducrohet||8/4/12 5:14 PM|
The first step should be doable easily:
- in the debug build type, enable proguard with some rules specific to
what you need (I guess: remove unused method/fields but don't
- set a dependency on what you need to be "compile" but not "package"
- set a different dependency for the runtime, but make it skip proguard.
Note that in gradle you can hook into any task to add work before or
after the task happen.
You can also rework the input / output of each task.
For step two (automatically deal with case where the 64K limit is
reached), this could be done as a 3rd party plugin that sits on top of
the main plugin at least at the beginning. We could integrate it in
the main plug-in if it works well
By this I mean this is probably out of the scope of the first version,
but if someone wants to work on it as we build the new build system,
we can make sure the hooks are in it.
> far fewer Proguard runs). Hooks for this built into the bui...
|Re: Build System high level feature proposal||Xavier Ducrohet||8/4/12 5:42 PM|
A bit more info about the jni part. I have to admit we didn't really
think a whole lot about this.
The first draft kind of implied that each flavor/built type would
generate its own lib<blah>.so but I don't think this is very flexible
now that I think about it.
I think it would make more sense to have 2 folders under ../jni/. One
for the source and one for headers.
Therefore, building a given build variant would give you 3 source folders:
and three headers folders:
This would generate a single lib<blah>.so for the build variant.
This gives maximum flexibility in configuring the flavor using custom
headers and even custom code.
Android SDK Tech Lead
http://developer.android.com | http://tools.android.com
Please do not send me questions directly. Thanks!
|Re: Build System high level feature proposal||Joe Bowbeer||8/4/12 6:14 PM|
On Saturday, August 4, 2012 5:08:24 PM UTC-7, Xavier Ducrohet wrote:
On Sat, Aug 4, 2012 at 2:33 PM, Joe Bowbeer wrote:
In this case, a custom flavor consists almost entirely of resources (and assets, too, if possible) that are selected according to the requirements of the build, as specified in command-line arguments, or in a properties file or json response or whatever. The next steps are to create the flavor on disk, by invoking a custom script, and then to build the product, which presumably entails creating a build variant that selects this newly created flavor. It is the "build the product" step where I would like the ability to register the new flavor, create the new build variant, and do whatever else is needed in order to build with this new flavor.
> 5. Native code?
At the least, we need the Product Flavor and Build Type. Most everything else can be derived from those.
A simple mechanism for passing properties (#define values) to the NDK build would be great.
|Re: Build System high level feature proposal||Joe Bowbeer||8/4/12 6:21 PM|
I'm liking the jni structure you describe below.
|Re: Build System high level feature proposal||Xavier Ducrohet||8/4/12 11:27 PM|
I'm still a bit blurry on your actual use case but let me give you a
few more details and then you can tell me if that makes sense to you.
Since we are looking at gradle we have the ability in our plug-in to
extend the DSL that gradle provides.
To declare a new flavor you'd do something like:
prop1 = value;
(repeat for other flavor)
Now by default this will use src/myflavor/... where you'll have a res
folder, an assets folder, etc...
You could go an customize the location of each of those if needed with
assetsDir = <new path>
Now if you have 100s of flavor you probably wouldn't want to do this
manually. We can make sure that you could use an API to dynamically do
You could for instance look at all the folders in a given root folder
and for each create a flavor, possibly reading some property file to
setup minSdkVersion, package name, etc..
Note that flavors don't exist on disk. It's purely a configuration in
the build script. They do use folders on disk but without the
configuration in the build script the folders go unused. The
configuration merely creates a new build tasks (build target in Ant
lingo) to let you build this flavor using the parameters defined in
However, do realize that this would be execute every time gradle is
running, in the configuration phase, as new flavors add new build
So running "gradle build<flavor42>Debug will always parse all the data
needed to generate the 100 flavors.
Depending on what is required to dynamically create those flavors we
might want to see if we can control the configuration phase to
disable this step when it's not needed. i'm not sure yet but I'll ask
the gradle guys.
Unrelated to this: I just remember I forgot to answer about ide.jar
This is a secondary jar, like lint. We have APIs (which aren't really
final at this stage) to let developers create custom interactivty in
the layout editors for custom views.
The idea here is that if you have src/ide, this would automatically
get compiled against those APIs and create a jar file (ide.jar) that
gets automatically loaded in the IDE to extend its editors. If your
project depend on a Library Project that has a custom view this would
get used automatically as well so that you get a good experience in
the IDE when using the views provided by the library.
And as we add features to the IDE, we'll see if we can extend it in
other places that make sense too.
|Re: Build System high level feature proposal||Tor Norbye||8/5/12 8:11 AM|
"- there should be no dependencies of the build system to some IDE specific project files like currently lint relies on the eclipse directory structure and project files"
The idea was that there would be a lint driver for each build system; an ant task to run it for ant, a maven task to run it for maven etc, and there is a Java API that these tasks would use to properly initialize source paths and class paths. However, to make lint more convenient to use in the mean time, the lint command line driver tries to just find things on its own. If it finds a .classpath file it will use it, but it doesn't require it; it will look for known folder locations to handle not just the ADT directory structure but the AOSP one as well. Just a couple of days ago I also made it work for the default Maven structure -- and I also added a couple of command line flags which you can use to tell it to look for sources and/or classes in other locations, if your build is doing something unusual, so in ADT 21 this will hopefully be easier to do. https://android-review.googlesource.com/#/c/40234/
On Sat, Aug 4, 2012 at 9:54 AM, Manfred Moser <mos...@gmail.com> wrote:
|Re: Build System high level feature proposal||b0b||8/5/12 9:09 AM|
Yes, this point is absolutely essential. In the regular Java / J2EE development world this is a given, mostly through maven, an area it really excels at.
Imagine a complex J2EE projects having to manually handle dozens of versionned and complicated dependency jars by copying them in a libs/ folder.
Absolutely inconcevable. The ADT build system feels totally antiquated in this department.
Do my app need a specific ActionBarSherlock version or some other third party library, either and Android Library or a pure Java Library ?
I want to be able to add a few lines to a file (or through a GUI for mr newbie coder) and bam....libraries and dependecies downloaded, with source code if available.
Manually handling dependencies and copying jars = fail.
Another area of major interest is to speed up the build process considerably. On large projects, I waste a lot of time
waiting for the damn thing to generate the final APK. Dirk Jakel post on this topic is spot on. I'm sick of the tools dexing the same pure java libs
that never change over and over and over...
Please do not go the NIH syndrome route on this. You will lose considerable time and nobody will be satisfied by the final solution.
Personnaly I would leverage maven to the maximum and make the integration with the tools first class, with an easy mode hiding complexity for simple projects
|Re: Build System high level feature proposal||Joe Bowbeer||8/5/12 10:31 AM|
On Sat, Aug 4, 2012 at 2:33 PM, Joe Bowbeer wrote:> 2. Where is the libs folder? Where do the 3rd-party .jars go?
I'm on board with Maven. I've used a Maven plugin on a few small Android projects, and it is likely that I would be using a Maven Android plugin for "real" applications if it were endorsed by your team. Maven is mature and ubiquitous, which are two qualities I look for in build system tools. I like that Maven is supported by all IDEs, and many CI frameworks such as Jenkins and Hudson. (Hudson is now an Eclipse project with contributions from Sonatype as well as Oracle.) Ease of integration at the CI level is another important consideration when choosing the build system tools.
This is also a question about dependencies: Do flavors have their own set of source dependencies?
Would it make sense to extend the dependency management to handle assets and resources as well?
|Re: Build System high level feature proposal||Manfred Moser||8/5/12 12:04 PM|
On Sun, Aug 5, 2012 at 10:31 AM, Joe Bowbeer <joe.b...@gmail.com> wrote:
> I'm on board with Maven. I've used a Maven plugin on a few small AndroidThanks for making the case for Maven quite nicely there Joe. As core
committer on the Android Maven Plugin I would like add a bit more
The plugin and documentation is here
and more documentation is in the book Maven: The Complete Reference
(Disclaimer ... I wrote that chapter)
Maven is by far the most widely used build system in the Java/JVM
ecosystem with very mature tool integration in all IDE's and beyond.
E.g. check out the existing Eclipse integration http://www.eclipse.org/m2e/
It is already used by many organizations using it for Android
development (github, groupon, square and many more...) and of course
way beyond for Java, Scala, Groovy.. development (e.g. Twitter
build tool for Eclipse, Apache, Spring...).
Due to its focus on convention over configuration it is easy to get
started with yet can be scripted nicely with custom plugin
invocations. It has a bit of a bad reputation due to it being
"opinionated" steering you towards best practices rather than allowing
you to easily script together something. This is intentional and is
great when it is being used by teams of developers or even just a few
developers that work on the build setup after the core author changed
job or so.
Gradle on the other hand definitely has the coolness factor now and is
a very good and powerful build system with a built in scripting
language (since it is just a groovy dsl). It just recently had a 1.0
release and tool support is slowly emerging in the latest tool
releases. At this stage it is definitely getting to a mature stage but
there is still a way to go. The powerful scripting is however imho a
danger for most users that will just hack together what they think is
best rather than learning from years of best practice in a large user
In terms of the proposed features I think most of the proposed
features are already available in the Android Maven Plugin and I sure
we would be more than happy for Google taking on the codebase as a
fork or inspiration and many of our contributors would be happy to
work with Google going forwards as well. We already have Eclipse
integration as well as good support in Intellij and Netbeans for
Android specifics beyond the general Maven integration.
Whatever you decide I hope that most of the functionality is
implemented in build system agnostic jars that are available easily in
a repository somewhere and the Android build system mostly wraps these
jars just like e.g. the Android Maven Plugin already wraps ddms and so
This way we are going to implement a good build system across all JVM
languages for whatever the communities want (e.g. how about a
Ruby/Rake based one for Ruboto users or a Leiningen one for Clojure on
Android (pipedream), or an SBT one for scala...)
I am looking forward to see whats is coming..
|Re: Build System high level feature proposal||James Moore||8/9/12 8:49 AM|
On Fri, Aug 3, 2012 at 5:36 PM, Xavier Ducrohet <x...@android.com> wrote:It would be useful if this also worked for Android library projects,
not just jars. I need a way of getting the list of library projects
(already provided in the classpath container, I think), but I'll be
handing dex a minified version of the library code.
(You may have been thinking this already, just wanted to make sure)
|Re: Build System high level feature proposal||Xavier Ducrohet||8/9/12 12:48 PM|
Yes. if you look at the section "Building library Projects" there's
classes.jar which gets packaged and an optional api.jar that would be
used for compilation instead of classes.jar.
|Re: Build System high level feature proposal||Jake Wharton||8/9/12 11:17 PM|
This step needs extra care because I feel that it's vital to anyone that needs production and development (and optionally other variants, betas?) installed side-by-side.
You mentioned a folder structure which would align with a potentially unlimited number of configurations through Maven-like directories that exist in parallel with the 'main' and 'test' conventions that already exist. This could be extremely powerful and versatile when coupled with manifest manipulation of package name. Where we must tread lightly is in R.java and BuildConfig.java generation and the effect it has on imports.
For example, one of Square's applications requires both a development and beta version of the application which differ only by a few resources (icons, values/ xml, and manifest package name). At present the difference in package name requires either the sources exist in a library project which generates its own R/BuildConfig in an unchanging package or a script which facilitates changing the imports based on the current manifest configuration. This is unacceptable and doesn't scale at all. Ideally we would be able to produce these variants without having to make variant-specific modifications or relying on a common, shared library with what is essentially a stub app to change three things.
|Re: Build System high level feature proposal||Ievgenii Nazaruk||8/9/12 11:40 PM|
Even current build system can be slightly modified to produce .apk from same code with different package names _without_ updating imports or moving code to library. I believe this feature was added to aapt sometime at the end of 2010 (just before 2.3 release), the option you need is "--rename-manifest-package". Last time I checked (and this was long time ago) this worked without any issues.
One thing to keep in mind, is that tester .apk should then also be built with using "--rename-instrumentation-target-package" to actually test those different versions of .apk, and not the main one only. I didn't use this one myself though.
So, as a conclusion, this scenario can be supported by using extra options to aapt when producing .apk.
|Re: Build System high level feature proposal||Xavier Ducrohet||8/10/12 9:37 AM|
That's exactly right.
--rename-manifest-package acts when the manifest is converted to
binary format, injecting the new value in the package attribute (and
updating the activity/service/provider/.. classes to not be relative
to the package anymore).
This means the original package, declared in
src/main/AndroidManifest.xml will be used to generate all
R.java/Manifest.java/BuildConfig.java no matter what the flavor is.
|Re: Build System high level feature proposal||Tomáš Procházka||8/12/12 1:23 AM|
For example in Netbeans IDE it is solved very nice. User can create Maven project add dependencies, set configuration and everything from IDE GUI, no maven knowledge is necessary
Then, biggest problem for beginners maven users is add dependency something what is not in any repository. There should be something like "Add local dependency" which simply manually install jars/javadoc/source to the local maven repository.
Dne sobota, 4. srpna 2012 10:46:33 UTC+2 Xavier Ducrohet napsal(a):
|Re: Build System high level feature proposal||Manfred Moser||8/20/12 11:47 AM|
Hi Xavier and others interested,
It took a bit longer to get to it but below as some comments and ideas of mine regarding the proposal.
As I've mentioned before, we are looking are a new version of the build system provided in the SDK.So what are we trying to do?
Hopefully expose all these in an API in java artifacts so it can be reused by other tools and then implement wrappers for the build system of choice, keeping them as shallow as possible
When it comes to interaction with the store application and so on the changes can get more and more complex. Whatever you do here should seemlessly work together with libraries.
These libraries or 3rd party jar files should work with only java code, only android code, only android code in .class files, java code in .class files and any combination with resources of varying setups. The resources should also be able to be a located in a resource only jar and used correctly and it should be possible to also use java resources like properties files in case a java library needs them. At least any magic removal or filtering should be possible to be turned off.
Library projects is only useful if it is possible to use them as a dependency in a project without having to rebuild it but rather just include it as a jar. Otherwise the overhead of adding it in and building it as part of the applications is just too much waste of time.
Why is that? Would it not be easier to just support the main configuration as a buildable product as well?
I believe this should maybe also include zipalign and maybe some sort of check if the pngs are optimized.
In fact a tool to optimize the png files would be nice..
I think you should by default add another build type, which is QA/ Staging/Testing or whatever you want to call it. Everybody needs that.. e.g. to configured a QA build that is configured like a production release build but e.g. talks to the testing infrastructure on the server side..
Great.. this is pretty much the maven standard directory layout. One thing you are completely leaving out is a overriding set up files for test runs. I would suggest to mirror the structure above in src/test just like Maven does it (and Gradle and others implement as well).
Please provide an API that produces that file so we dont have to reimplement (as we did not in the android maven plugin).
Great... while this is could I think it would still be good to work in the traditional way as well where a separate project that has code in src/test reference the tested application as a dependency. This allows for e.g. testing different versions with the same test by just specifying a different version of the tested application. It also reduces build time for the application by simply omitting tests ..
In the maven world we introduce a so called aggregator project that ties them together. I am sure Gradle can do something similar. This would be really good also to tie together e.g. building a pure java library, as server side component, an android app and a test that deploys and runs it all (server and android app). We can do this already in Maven and it should work for the new Android build system as well..
Great... please work with these projects including e.g. Robotium that sits on top of the instrumentation tests.
Also when running tests on devices look at implementing a feature that produces the test results on the machine running the tests and not the devices. Note that you can look at the Android Maven Plugin codebase where I implemented that for junit output creation using the ddms library from the sdk. Then you just have to come up with a naming convention for the files when you run the tests onto multiple devices.
And all that goes into a jar right? Why not have the classes right in there instead of a classes jar .. just like normal jar's do it? The current proposal just makes it more complicated for no benefit I can tell of.
Would it not be better to have a separate source artifact as well as separate lint and ide jar artifacts .. otherwise you are completely mixing up concerns and have to deal with the fact that e.g. lint and ide jar should not go into a release build..
When deployed to a repository server or in general the library project has to be able to declare its dependencies so that these can be automatically be pulled in as well. E.g.
- App declares dependency on ActionBarSherlock in project file
- build gets it from the repo server and sees in the meta data file (in the maven and also gradle world the pom.xml that is deployed to the repo server)
that ABS in turn depends on the compatibility library
- so the build gets it as well and hooks it in
Notice that this works via a local cache just the same (for gradle and maven) as for a repository server. In both cases the meta data file needs to be there. In turn the build tool has to support authoring such a library and deploying a library to the repo server so it can be reused by others (or better pushed to the Central Repository) so everybody can use it.
But it should be possible to override it imho with e.g. a custom android jar.
It should be possible to skip the lint run and to configure it to run but NOT break the build.
Disabling all might be the equivalent to disabling lint altogether.
And I hope an API will be available to invoke it..
It should also be possible to just e.g. create a lint jar for all android project in a company or even one to share via open source , where you push to a repository server as a binary jar and declare it as a dependency on all the project you want to use it on..
The same thought applies to ide.jar imho..
I hope this all helps..
|Re: Build System high level feature proposal||Xavier Ducrohet||8/20/12 1:29 PM|
thanks Manfred for the feedback.
A general comment that applies to several of your points: Yes we are
making a java library that will do all of the work. All you need is to
configure it with info coming from the front end you choose (Maven,
gradle, whatever). Yes we'll put this on a repo so that you can use it
easily in the maven plugin.
More answers inline.
We already support java resources, so there should be no problem.
We'll make sure that filtering is override-able, configurable and
Yes of course.
It's a good question. There are some case where main will not be a
For instance 2 flavors each providing a different version of a class.
This means the "main" configuration won't have this class and likely
So we can either have a switch that says "main is a valid product" or
just automatically disable it when you add a flavor.
zipalign should be automatic if the build signs the package. There's
no reason not to zipalign the package if you can.
Am I missing something here?
Why not. Any reason why this shouldn't be part of the build system
though? It seems much safer to just do it all the time.
That's an interesting idea. However there needs to be some flags
that's different from debug/release by default to have it as a
built-in build type.
None of the current build type properties seem like they would have a
special value in "staging".
Is there any other properties you're thinking of?
My point is that while yes, staging could be configured by a dev to
point to a test server, out of the box the "staging" build type must
be different from debug/release without having to do put some custom
file somewhere to make sense.
BTW, I am happy to add new properties to both ProductFlavor and
BuildType if you think of new ones that make sense.
Overriding what kind of files?
Yes, this will be part of the builder library.
What do you mean by testing different versions? different revisions or
different flavors of the app?
The former doesn't seem that useful. The latter will be handled
directly (we'll have separate test folder for each flavor).
Not building the test should be taken care of depending on which tasks you call.
Yes, multi-project build in gradle should handle this.
You mean storing the result of the tests on the host machine?
I'm not sure I see the point of multiple artifacts (except maybe for sources).
However I'll start with one concern:
A library dependency needs to have more than just classes in it. there are:
- java resources
- android resources
- jni components
- aidl import files.
- proguard rules
#1 and #2-5 collides with each other. ie right now all non .class
files inside a jar dependency are added to the apk as they are java
resources. We cannot just simply put other files that are not java
resources and process them differently. No matter how we put it there
is a risk of having a conflict with actual java resources.
On top of this, lint.jar should be there too. It's not another
artifact that you have to depend on, it's part of the library. It
should be automatic. You depend on the library and you get lint.jar
that gets used during the lint step.
Same thing for the proguard file.
So it makes a lot more sense to me to have a bundle (zip archive) that
contains the jar used for compile (that would contain the java
resources as well) and other items used during build, that may or may
not find their way into the apk (for instance aidl import files,
lint.jar and proguard file are merely used during the build but are
This bundled is downloaded and extracted locally somewhere in the
folder of cache of the dependency management system and its various
bits are used by the build system.
Yes of course. The idea is that the bundle defined above are
uploadable to a Maven or Ivy repository.
I'm not worried about that. I was a bit more concerned about
multi-project build that actually bypass
repositories, but that should be easy to do as well.
Yes, the build tool will allow creating artifacts that can be
uploaded. Gradle has built-in support for this.
hmm I would rather not allow this, at least not through a simple
What's the use case here?
Lint as 2 levels of severity: warning and error.
Like configuring any java compiler, if you set a particular check to
be an error, it will break the build.
Maybe we do need several profiles that allow you to set everything to
be warning instead of errors. But if you run with some checks sets to
be errors these will break the build.
Yes, this would work.
The lint.jar in a library is to provide custom checks for people using
a particular API provided by the library, but lint does support for
both configuration and custom rules that can live anywhere.
We'll have to make sure this can be configured through the build system.
|Re: Build System high level feature proposal||Tor Norbye||8/20/12 1:43 PM|
Actually, it has a third severity, "fatal", which is like error but also abort (when you do export apk, lint runs automatically and checks only the fatal severity issues and aborts the export if any are found. Fatal let's us have non-aborting errors, though I should probably make a pass over the issues and adjust the error vs fatal severities since it's a bit arbitrary at the moment.
|Re: Build System high level feature proposal||Manfred Moser||8/20/12 10:12 PM|
I would default to main as a valid product and allow it be switched to
invalid, since that seems more common and straightforward to use.
>>> Build Types
>> I believe this should maybe also include zipalign and maybe some sort ofNot sure. My thinking would be not to invoke anything that is not
necessary during a dev build including not invoking proguard, signing,
zipalign and whatever takes time. I would completely disable lint by
default too btw.
Maybe as part of a release build. But for normal dev builds the more
you add in the longer it takes and that kills productivity.
A staging build imho is running all test and verifications like the
release build (lint, proguard, .. enabled) but does NOT go to
production but rather to a QA/testing deployment and therefore might
be configured very differently. E.g. it probably has debug enabled so
that QA can send you stack traces of failures with more info and so
Where is a list of them I can look at?
>> Great.. this is pretty much the maven standard directory layout. One thingAnything that can be configured differently for testing as compared to
the actual build. Could be e.g database connection, or a template data
in a db, or different resource files and so on and so on
E.g. you could have a test module that has UI tests that are
independent of the implementation and you have a version 1.0 on the
play store and are working on 1.1. While working on 1.1. you are
writing more tests and you need to ensure that the tests keep working
on version 1.0 and also want to test against 1.1. So you could e.g.
create a test group for 1.0 features and run all these test agains the
1.0 version to ensure your refactoring of the tests is still ok and in
a second run e.g. on a CI server run the same test against the 1.1
version to ensure that existing functionality is not broken in the
upcoming 1.1. version
Yes. That way you can e.g. build and run the test against a few
attached devices and running emulator at the same time and all test
outputs junit xml files will be available on the host machine in the
build output folder. That way e.g. they can be picked up easily by a
CI server without having to pull them off the devices with adb. imho
this is the more common need than having them on the device. What good
are they there anyway? This would be good for emma as well btw..
I understand what you are saying and I kind of agree and kind of
disagree. You are basically requiring all the logic for working with
the different files in the bundle into the build system rather than
having the user understand and configure it. Thats fine by me as long
as we can expose configurations to override these settings.
E.g. you have a custom build of android that you want to create an app
against that has additional proprietary api's in it that are more
openly exposed.. this approach could then even be used to build the
AOSP apps with a more open android.jar.
Btw I still think it will be crucial to get all open source apps from
Google to be built with the new build tool e.g. at least the iosched
and other examples that use only public api's but ideally also all
apps in the aosp tree. That would be the best proofing ground for the
>>> Lint Integration
>> Disabling all might be the equivalent to disabling lint altogether.So if I understand this correctly I should be able to configure e.g. a
pure dev build to have a lint config that disables it all to save
time, but also e.g. a verification dev build as a second one that does
all the lint stuff.
Cool.. thats all for now. Thanks for working on all this and taking in
our feedback in terms of features (even if you already decided on
Gradle.. ;-) )
|Re: Build System high level feature proposal||Mark Murphy (a Commons Guy)||8/21/12 4:02 AM|
On Tue, Aug 21, 2012 at 1:12 AM, Manfred Moser <mos...@gmail.com> wrote:Please don't.
Mark Murphy (a Commons Guy)
http://commonsware.com | http://github.com/commonsguy
http://commonsware.com/blog | http://twitter.com/commonsguy
_The Busy Coder's Guide to Android Development_ Version 4.0 Available!
|Re: Build System high level feature proposal||Xavier Ducrohet||8/21/12 1:27 PM|
On Mon, Aug 20, 2012 at 10:12 PM, Manfred Moser <mos...@gmail.com> wrote:I'm not sure. I'm going to start a new thread about flavors as there
are some more details I want to discuss) and we'll see.
proguard: yes, but that should be disabled by default for debug builds
signing is required to install an app, no way around this.
zipalign can have significant performance impact if not done. I would
think this should be done.
However I need to look at the partial update that Play can do now and
see how that would impact the build process.
Lint is another issue. see at the end.
Note that before the 2nd half of last year we didn't have incremental
process of images and 9 patch, which made everything slow. This is why
we first processed the resources to generate R.java only and later
processed the resources again to package them (including processing
images and 9-patch). This 2nd step could be delayed to install time in
Ant followed this even though it was useless. So now we'll process the
resources only once and both generate R.java and process the
resources. The image and 9-patch processing will still be done ahead
of that in an incremental way.
This should speed up things compared to now (at least compared to ant
and eclipse, not sure what Maven did).
Yes, I understand this. However this is something that is set by the
developer putting some different configuration files in different
folder, not by setting a flag somewhere.
The buildType properties are:
Buildconfig DEBUG flag
This is it.
I think the staging idea is great, and should be advertise a good
practice. I'm not sure there's a point in creating a staging type by
default as it would essentially be the same as the debug one when you
look at the configuration properties above.
Wouldn't that be the staging one then?
It's a good point though that building an app that talks to a server
probably shouldn't talk to the production server when running
I think we need to be able to configure the test to declare which
buildtype variant is used for testing (could be debug, could be
staging or any other).
This should be sufficient to let devs create a buildType that
overrides code and/or resources to build the app the way he/she wants
it to be tested.
This is interesting. I'm not sure where the 1.0 apk would come from though.
Right that's what I figured. right now the instrumentation runner
prints them I think.
Right now Eclipse gets this output and then display them in the
standard jUnit window.
Ant just output them to the console.
However Ant is able to get the emma data off of a device and process
it locally to create a report.
We definitively want to do all of that in here too.
What exactly do you want to override here? I don't think there's any
reason to package a library as simply a standard Java jar file.
It has to be an archive that contains the jar plus at least the
android resources (in most cases). We can't have those flat in the
same archive due to possible conflict with the java resources inside
Once we agree on that point, we may as well just it the rest in the
main archive bundle and call it a day ;) I see little point in
generating such a bundle while omitting some parts of it (be it
source, lint.jar or anything else).
I understand this is different from the normal way of doing things in
Maven, but this is very different from just a java build.
I already want (and am planning) to allow building unbundled apps with
Those are apps that use public apis only but are built as part of the
Android source tree.
They don't use a normal SDK but instead look at the files in
This would be done through a simple property in gradle that lets you
point to a full Android source tree instead of an SDK.
In the builder library there'll be an SDK parser and a parser for the
Android source tree to resolve the compile target and find the
location of android.jar and other tools and files.
So we could technically add a different properties that tells the app
to build using the full api jar instead of the stubbed android.jar
This would only be possible when using a full Android source tree
instead of an SDK, so replacing android.jar itself is not needed.
People who use this will work directly in the source tree and will,
hopefully, know what they are doing (as in, breaking CTS and other fun
Yes! Internally, people who work on apps are struggling to build with
the source tree build system (oh, makefiles...) while using our tools.
My goal is to move all those apps to the new build system to let them
use the IDE and only have a single way of configuring how their app is
Maybe, or maybe not.
In gradle you get different task to do various levels of the build.
In a regular java project, gradle assemble will do build the app, but
nothing else. This is enough to run the app.
gradle build does assemble + builds and run the tests. This is what
should be done for deployment.
We will have the same. "gradle assemble<Variant>" will build the
variant and that's it. From there you can install on a device and use
"gradle build<variant>" will build the app, the test apps, (maybe) run
the test app, and run lint as part of the "check" task.
(calling just "gradle build" will build all variants of the app)
See the tasks graph here:
So yes, you can build the app with not running lint, but doing "gradle
build<variant>" will run lint by default always. If you want to
disable it completely (why woud you?) you'll have to manually disable
all the checks.
I think Gradle is more suited to our needs but I definitively want to
make sure devs who want to use Maven can use it too.
I already mentioned it, but I'll say it again: all of the code dealing
with actually building the app will be in a separate library that you
can reuse. All gradle will do is handle the configuration of the build
and pass that on to the library.
In fact, parts of it (and of the gradle build sytem) is already
checked in (we are using a new git project in AOSP: tools/build.git)
You can see the current contributions (only a few of them, we just
started putting them in this week) here:
Comments on this is welcome too. Look in particular at this:
https://android-review.googlesource.com/#/c/41861/5 which is the
initial commit of said library.
Ok, off to write a new post on dimensions of product flavors...
|Re: Build System high level feature proposal||Xavier Ducrohet||8/21/12 3:28 PM|
On Tue, Aug 21, 2012 at 1:27 PM, Xavier Ducrohet <x...@android.com> wrote:After thinking about it a bit more I guess we should at least have
this configurable. We can figure out the default value later.
|Re: Build System high level feature proposal||Manfred Moser||8/22/12 11:32 PM|
More answers/discussion inline below
For the image processing I think it would actually be good to be able to optimize them in place as a manual step so you can check the optimized version back into your source control and then have that process skipped or at least run even faster in the future..
This is it.
Fair enough. I guess we can always do that later if we find that there are more flags that are actually different. Need to test this and think about it a bit more..
I like that idea.
Well .. part of a well managed release process of an app imho is that it is build and deployed to a repository server together with e.g. the proguard map file so that stacktraces can be deciphered later and whatever else is important as part of the release. All artifacts that come out of the release process should be stored and managed in a repo server and be available in an archive or archives there.
That in turn then enables the use case I mentioned above because any old apk (release or even development versions since you can deploy them to the repo server too) can be retrieved as a dependency from the repo and then used in e.g. the testing for deployment to the devices the tests run on..
Maybe.. but it does not create the junit xml format which is what build servers and other tools can change into browsable html and so on. The android maven plugin does that.. feel free to lift the code .. it is all apache licensed and I wrote it all so if you need any further copyright transfer or whatever .. I will sign it..
Great.. the android maven plugin can pull down the emma report and so as well but really it should just be created on the host imho.. I see no value of it on the device/emulator.
>> So it makes a lot more sense to me to have a bundle (zip archive) ....
override these settings.
The java resources go in classes and the android ones in res .. there is no real conflict imho. Both of these are needed at runtime.
By lumping all this other stuff into the library zip you are completely overloading what it is used for. E.g. the ide.jar is for eclipse stuff.. so not useful for other ide users. The lint jar is needed at build time but NOT at run time... but it should be e.g. pulled into another library if you have e.g. one library depend on another. And so on.. it is going to get really complex in terms of a build if you have say an application that uses three libraries. Two of which depend on 1 more library each. Now the build system needs to work with say 5 ide.jar files, up to 5 lint.jar files, and so on and it need to know for all those artifacts what they are needed for ... on the other hand if we split it up into the library only containing what is needed at runtime and have additional artifacts that are optional the user can decide to pull them into their build as dependency if they e.g. want those lint check or not. Just like they can do for e.g. custom developed lint checks that are generic and distributed as a jar. imho this would be way cleaner..
javadoc and source could be similar artifacts that are shipped inside separately just like is the convention for all systems now (including gradle). E.g. look at http://search.maven.org/#artifactdetails%7Ccom.google.inject%7Cguice%7C3.0%7Cjar
and you can see a bunch of artifacts for google guice 3.0 in the same repository server location. The folder is defined by
groupId: com.google.inject which translates to com/google/inject
artifactId: guice translates to the next nested folder guice
So in that folder com/google/inject/guice/3.0
guice-3.0.pom - the file that defines the dependencies of guice
guice-3.0.jar - guice itself
guice-3.0-javadoc.jar - the javadoc for it
guice-3.0-no_aop.jar - a flavour of the jar using no aspected oriented programming (useful e.g. for android)
guice-3.0-sources.jar - the source jar
Extending this same concept that is proven to work for all build systems using the Central repo including Ant/Ivy, SBT, Gradle, Maven, Leiningen and others for Android libraries we could e.g. add
and so on.
And when you declare a dependency to it it would be done with a classifier just like you do for javadoc and sources now..
;-) it seems like we don't.
Maven does a whole lot more than just Java builds.. but that is a different story.
Thats sound good..
Great... btw. while you are at it.. do you want to move the sdk build to a new build system too ;-)
>>>> Lint Integration
Ok.. so maybe what we need is a way to configure different sets of lint rules.. like a simple verification one eg. for regular development and stricter one for QA build and so on.. then one of them could if so desired turn everything off.
Sounds good.. can you document how to build this branch somewhere?
Thanks for all the work and I look forward to more coming in ;-)
|Re: Build System high level feature proposal||Xavier Ducrohet||8/23/12 11:23 AM|
On Wed, Aug 22, 2012 at 11:32 PM, Manfred Moser <mos...@gmail.com> wrote:
> For the image processing I think it would actually be good to be able toWe would have to have a global switch where we disable all
optimization. It seems safer to me to
always run this. However I see the point of performance.
How about we don't run it in debug builds but run it in release mode?
(by default, overridable in build types)
Yes, it would be easy to add it later. This is not a structural change
so that's easy.
Yeah that's good. We'll have to see how we do that.
Yes, we need to generate those files. As for lifting the code, I can't do that.
You'd have to contribute it to AOSP.
It cannot work any other way.
The test runner is on the device, and it's generating the report.
Nothing running on the device has access to the host computer. So it
has to be generated there and then pulled locally.
I think I see maybe why we are thinking differently.
In Java you could have an app that is made of different jar files all
linked together through clear dependencies (declared in a POM file or
whatever), and those dependencies are resolved at runtime and the jars
are downloaded and the app is run. So yeah having artifacts that are
"optimized" (in content) for runtime is useful.
But this is not how Android works at all. All the jar files are
combined into a single APK during the build and there is no
dependencies resolved nor any artifacts downloaded at runtime.
Android Library projects artifacts are used at build time only.
Even if we ever add the ability to have runtime artifacts, those would
have to be critically different: the Android resources will have to be
compiled and the bytecode will have to be converted to dalvik
bytecode. So this would be a new type artifact different from the one
So yes we can have separate artifacts for build and runtime, except
right now the runtime one does exist and the one I describe is the
build-only one. In this case we may as well put lint.jar and other
stuff in it :)
As for having to deal with several lint.jar and ide.jar that's really
not a problem. When running lint, we have the list of libraries from
the dependency graph, so gathering those are easy. I really don't see
how that's more complex than already dealing with the same dependency
graph for compilation and packaging. It seems to me that having to
deal with other artifacts is actually more complex.
I really don't see why having to explicitly say that you want to get
the lint checks associated with the library you just added to your
dependencies is a good idea. If you add the library, you should get
the checks automatically.
By the way, ide.jar is not Eclipse specific. We've been very careful
about making it IDE agnostic, like all the other libraries we create.
Right now (it's actually not finished and APIs aren't set yet) it's
mostly about dealing with custom views, which IntelliJ could benefit
from since they have a layout editor based on our rendering library.
Any IDE could benefit, really.
There are some changes going in repo right now that will allow us to
download only some git projects instead of the all source tree.
I want to enable this for the sdk projects, and I want to allow
building those separately from the whole tree.
I do want to move all the Java build away from makefiles. However some
other tools (emulator for instance) will still require makefiles. I'd
like to be able to build only the Java libraries without touching
Just use gradle ;)
So right now, this is in its own (new) git project (tools/build) and
only builds with gradle and is independent from the whole build
It generates a local Maven repo under tools/build/repo with a single
fat jar with everything. I'm going to split this into the gradle
plugin and the AndroidBuilder library so that you can look at the
|Re: Build System high level feature proposal||Ricardo Gladwell||8/25/12 5:36 AM|
Sorry for coming late to this discussion, I've been busy on my own projects but please give me your thoughts on my comments below:
This proposal looks great. Really pleased to see support to include Android Library archives packages. This will make the task of automating builds much easier and get new developers up and running much faster.
I'd like to second Manfred and Joe's support for Maven: I must admit, I find it odd that you're looking at Gradle to replace the build system, given that the ADT is built on Eclipse and the Eclipse project has more mature Maven support.
I'm the maintainer of the Android Connector for M2E which creates a bridge between the ADT and Maven/M2E. So the problems we have are slightly different: the discussion so far has focussed much more on the SDK but I think we should address issues with the ADT as well:
The biggest problem we face is that the ADT isn't built with third-party extensions in mind. We rely on making calls to internal OSGi ADT APIs and these frequently change, breaking out project between releases. We have opened a ticket for this but this is not close to being resolved and I don't think it addresses a deeper point about refactoring the ADT to be more extensible.
Another problem is that there is a large mismatch between the configuration the Android SDK command line tools support and the configuration the ADT supports.
For example, the android-maven-plugin integrates directly with the command line utilities and therefore supports the ability to disable /libs/ folder dependency look-up or ignore duplicate class errors on DEX compilation. Yet the ADT doesn't support this.
Our job would be much easier if the SDK and the ADT matched up more feature-wise as well as making the ADT more extensible.
One possibility would be to simply contribute the m2e-android project to Android development team as an optional plugin installed as part of the ADT. Supporting this would mean users get more consistent Maven support, as well as making ADT developers more aware of the issues of making the ADT more extensible making it easier, for example, to write Gradle-ADT Eclipse plugins and so forth.
...As I've mentioned before, we are looking are a new version of the build system provided in the SDK.Our goal is to make it easy for beginners but flexible and powerful enough for teams working on large complex application(s).
|Re: Build System high level feature proposal||Xavier Ducrohet||8/28/12 11:16 PM|
I completely recognize the lack of extensibility of ADT but to me this is orthogonal to the build system (sort of, see below), and I'd prefer we keep this thread on the topic of build features and implementation.
Yes we would like to make ADT extensible, but it has to be driven by a use case (yours is fine). I'm not sure why you require so many APIs though, and making those public, maintained APIs is a huge undertaking. We should discuss this in a different thread.
The mismatch between the command line and ADT is a big issue indeed. We definitively want to use a single build system for both. We're designing the build system first. We'll integrate it in the IDE after.
As for contributing m2e-android to aosp, why not. It would help make sure it's up to date. I would also like to hear from the maintainers of the maven plugin for Android what they think of moving the plug-in to use the AndroidBuilder library (when it's finisehd. I think it would be nice and would ensure that all new build features are supported automatically (or close to it) and would ensure that there is parity between Gradle and Maven and that both builds do the same thing.
Getting to a point where you can use ADT with either Maven or Gradle sounds like a great goal to have. Definitively something we should look into when we work on the integration with the IDE.
|Re: Build System high level feature proposal||Manfred Moser||8/29/12 12:13 AM|
On Tue, Aug 28, 2012 at 11:16 PM, Xavier Ducrohet <x...@android.com> wrote:That is definitely the plan for the android maven plugin from my point
of view. But it will have to rely on contributions from others as well
as continue to support our current users and features so we will have
to see how that works out..
I definitely plan to refactor things to more and more use the builder
library. Doing that will probably also drive changes to the library so
it will take some time.
|Re: Build System high level feature proposal||Ricardo Gladwell||8/29/12 3:24 AM|
Fair enough, would you like me to start a new thread now or leave things
until the build system is sorted out?
|Re: Build System high level feature proposal||Xavier Ducrohet||8/29/12 10:46 AM|
Just start a new thread.
|Re: Build System high level feature proposal||Xavier Ducrohet||8/29/12 10:47 AM|
I'm still changing the library a lot, but I definitively want to make
sure it's usable by Maven.
|Re: Build System high level feature proposal||Ricardo Gladwell||8/31/12 5:19 AM|
Another quick question on this topic: will you also be changing the
tools that build the SDK/ADT to Gradle as well? Or will Gradle just be
for consumers of the SDK/ADT?
|Re: Build System high level feature proposal||Xavier Ducrohet||8/31/12 10:00 AM|
I'm looking at building all our jars with Gradle. Eclipse is more complex so I'm not sure what we'll do there.