How about this one:
- drop ant and have scripts in other languages
?
The #1 problem I see with the design as proposed is that it
invalidates every Web page and every printed page ever written about
Android app development that covers how to create projects, where
files reside, etc.
On Sat, Aug 4, 2012 at 2:33 PM, Joe Bowbeer wrote:
> Xavier,
>
> I generally like the proposed features.
>
> The build variants (combinations of Product Flavor and Build Type) are
> similar to features that I've had to build on top of the existing build
> system. In one product, before library projects were introduced, we had a
> main project and five differently-branded project flavors. When library
> projects were introduced, the main project morphed into a library project --
> but it is really a main project by nature, and this should be easy to
> express with your new feature set. In another product, there are 100's of
> flavors that are only known at build time, so I'm hoping that the new system
> will support dynamic flavors. More on this below. I've never needed more
> than a handful of build types, by the way, so what you propose seems fine.
Can you describe how the flavors are known at build time? How are they
described?
If there is some file+metadata that describe them, it should be fairly
easy to put some logic in the build script that process this data and
dynamically create new flavors.
I cannot stress the following point enough: if you every have to download a .jar manually, you've lost. If you have to manually resolve a version difference between projects, you've lost. If you have you create an IDE project for a library dependency, you've lost.Proper dependency management is a magical experience and I cannot wait for the every single Android developer to realize the simplicity and elegance of when it's done properly. Once experienced you will continuously wonder how you ever managed to copy .jars into various libs/ folders...
On Sat, Aug 4, 2012 at 2:33 PM, Joe Bowbeer wrote:
> 2. Where is the libs folder? Where do the 3rd-party .jars go?
>
> This is related to question #1 about leveraging portable Java code. The
> most common way to do this is to add the .jars to the libs folder, but I
> don't see this folder in your outline. Where are the libs? Are there
> separate libs per flavor, or what?
First you should really reference dependency using Maven or Ivy if
this is a 3rd party library.
If this is your own, you could create a dependency on a locale file.
It's up to you to put it where you want.
With gradle we aren't as tied as libs/ are we were with Ant.
- Help devs making different versions of the same app.
- Library Projects can be cumbersome to make 2 apps that are 99% identical.
- Different versions for different app store could only differ in package name and signing key.
I agree.First we want it to be easy to just create a project and build it with little (if any) configuration.After that we do want people to progressively extend their build as they need. But realistically, discovery of how to do things is never going to be trivial. Even something like setting a dependency on a lib coming from Maven requires some basic knowledge that devs who never used Maven don't have.
As I've mentioned before, we are looking are a new version of the build system provided in the SDK.
So what are we trying to do?
- Better extensibility
- Allow devs to configure every steps.
- expose all options for aapt, dex, …
- Help devs making different versions of the same app.
- Library Projects can be cumbersome to make 2 apps that are 99% identical.
- Different versions for different app store could only differ in package name and signing key.
- Better dependency management for 3rd party jar files and Library Projects
- Use Maven/Ivy for dependency management
- Depend on Library Projects distributed as binary blob
- Generate Library Projects blob
Product Flavors
A product flavor defines a customized version of the application build by the project. A single project can have different flavors which change the generated application.
Although different flavors could be very different applications, using Library Projects is a better use case for this, and the build system still supports this.
When a Product Flavor is defined, the main configuration is not available as a buildable product. To create more than one product, two or more flavors must be created and customized.
Build Types
A Build Type provides configuration for the following build properties:
- manifest debuggable flag
- native compilation debug flag
- proguard enabled + rules
- debug signing flag (ie whether to use debug key or release key)
- package name suffix (2)
- Buildconfig
- DEBUG flag
- other custom properties?
By default the build system provides two build types, debug and release, which can be reconfigured. New types can also be created to provide any combinations of the above properties.
For instance, one could make a Build Type that uses Proguard but creates a debuggable apk.
Sourcesets
Due to Build Types and Product Flavors all providing their own sources, resources and manifest, the project structure need to change from the root level src/res/manifest used in the current build system.
Instead the build system moves to a top level src folder, containing folders for each types and flavors. In addition, the main folder contains the source for the main configuration.
A typical project will then have the following folder structure:
- src
- main
- java
- jni
- resources -- this is java resources
- aidl
- res
- res -- this is Android resources
- assets
- AndroidManifest.xml
- debug
- … (same as main)
- release
- … (same as main)
- flavor1
- … (same as main)
- flavor2
- … (same as main)
This is of course be completely customizable, but this shows the default, and the type of inputs that are available to tasks.
BuildConfig
BuildConfig is an automatically generated class with a single boolean DEBUG, set to true in debug builds only.
Testing
Test Application
In the current build system, test applications are created from a different project that is flagged as a “test project”. This project references the main project in order to get the main project’s classes on its classpath.
With possibly multiple Product Flavors it is important to test all of them.
Some Java build systems have introduced having the test code directly in the project under src/test/ and the new build system follows this convention.
Since JUnit test cannot be run against Android code, the build system reuses this concept for test applications.
There is a mirror of the main configuration, plus flavors which allows creating flavor specific test applications.
The main test configuration uses src/test, and flavors uses src/test<flavor>
The manifest of a test application is pretty basic and it is automatically created it. The build system provides the ability to configure the test package name and the InstrumentationTestRunner class.
Deployment and running all test flavors TBD.JUnit tests
JUnit tests cannot be run against code accessing Android APIs. Instrumentation tests are the sanctioned way to go.
To have code that does not access Android APIs tested with JUnit tests, a multi-project build should be setup with the code to be tested put in a Java-only project.
Note: The build system shouldn’t prevent usage of some third party test frameworks such as Roboelectric. It should actually facilitate integrating those. Details TBD.
Building Library Projects.
Libraries must be packaged in a very specific way, no matter what the library source folder structure is.
- classes.jar -- this is the main code used for compile and packaging.
- api.jar -- this is an optional jar to used at compile time instead of classes.jar
- res/ -- this is the res folder(s) extended with:
- crunched images
- renderscript bitcode
- AndroidManifest.xml (includes the merging from other Library Projects?)
- lib/ -- this is the jni component(s)
- aidl -- this is a straight copy of all the src/../aidl files
In all the above cases what is packaged can be extended by the release Build Type, similar to a normal apk.
Additionally, the blob may contain:
- proguard.txt -- custom Proguard rules
- lint.jar -- custom lint rules
- ide.jar -- custom IDE extensions
- docs/ -- this is the javadoc
- source/ -- this is optional source package (for open source libraries).
Using Library Projects.
Library projects can be used two ways:
- Dependency through a repository (local/maven/ivy)
- this downloads the blob and unarchives it somewhere
- As part of a multi-project build.
#1 is fairly standard and can be added as as normal dependency.
#2 is a bit more complicated and needs to be figured out. The main issue is that dependency on Library Projects have a clear dependency order and the setup of the dependency must clearly spell this out.
Dependencies
Note that the Android API jar(s) is automatically added to the compile (only) dependency list, and doesn’t have to be declared manually.
Lint Integration
The new build system integrates Lint at two levels.
Running Lint
A Lint tasks is created and allows testing any build variant.
Lint errors will break the build.
Lint rules can be enabled/disabled and set priority (warning/error).
HTML and/or XML reports can be generated.
Lint should run on all available Build Variants but remove duplicates (since code common to all variants is compiled for each of them
Extending Lint
Creating custom Lint is directly possible using the src/lint/ folder.
This creates a lint.jar file that is passed to Lint when running it on the project itself. The dependency on the Lint API is done automatically. All that’s needed is creating the source folder.
Actually, it has a third severity, "fatal", which is like error but also abort (when you do export apk, lint runs automatically and checks only the fatal severity issues and aborts the export if any are found. Fatal let's us have non-aborting errors, though I should probably make a pass over the issues and adjust the error vs fatal severities since it's a bit arbitrary at the moment.
-- Tor
Ant followed this even though it was useless. So now we'll process the
resources only once and both generate R.java and process the
resources. The image and 9-patch processing will still be done ahead
of that in an incremental way.
This is it.
I think the staging idea is great, and should be advertise a good
practice. I'm not sure there's a point in creating a staging type by
default as it would essentially be the same as the debug one when you
look at the configuration properties above.
I think we need to be able to configure the test to declare which
buildtype variant is used for testing (could be debug, could be
staging or any other).
>>>> Testing
>> What do you mean by testing different versions?
>
> E.g. you could have a test module that has UI tests that are
> independent of the implementation and you have a version 1.0 on the
> play store and are working on 1.1. While working on 1.1. you are
> writing more tests and you need to ensure that the tests keep working
> on version 1.0 and also want to test against 1.1. So you could e.g.
> create a test group for 1.0 features and run all these test agains the
> 1.0 version to ensure your refactoring of the tests is still ok and in
> a second run e.g. on a CI server run the same test against the 1.1
> version to ensure that existing functionality is not broken in the
> upcoming 1.1. version
This is interesting. I'm not sure where the 1.0 apk would come from though.
>>>
>>> Great... please work with these projects including e.g. Robotium that sits
>>> on top of the instrumentation tests.
>>>
>>> Also when running tests on devices look at implementing a feature that
>>> produces the test results on the machine running the tests and not the
>>> devices. Note that you can look at the Android Maven Plugin codebase where I
>>> implemented that for junit output creation using the ddms library from the
>>> sdk. Then you just have to come up with a naming convention for the files
>>> when you run the tests onto multiple devices.
>>
>> You mean storing the result of the tests on the host machine?
>
> Yes. That way you can e.g. build and run the test against a few
> attached devices and running emulator at the same time and all test
> outputs junit xml files will be available on the host machine in the
> build output folder. That way e.g. they can be picked up easily by a
> CI server without having to pull them off the devices with adb. imho
> this is the more common need than having them on the device. What good
> are they there anyway? This would be good for emma as well btw..
Right that's what I figured. right now the instrumentation runner
prints them I think.
Right now Eclipse gets this output and then display them in the
standard jUnit window.
Ant just output them to the console.
However Ant is able to get the emma data off of a device and process
it locally to create a report.
We definitively want to do all of that in here too.
>> So it makes a lot more sense to me to have a bundle (zip archive) ....
override these settings.
What exactly do you want to override here? I don't think there's any
reason to package a library as simply a standard Java jar file.
It has to be an archive that contains the jar plus at least the
android resources (in most cases). We can't have those flat in the
same archive due to possible conflict with the java resources inside
the jar.
Once we agree on that point, we may as well just it the rest in the
main archive bundle and call it a day ;) I see little point in
generating such a bundle while omitting some parts of it (be it
source, lint.jar or anything else).
I understand this is different from the normal way of doing things in
Maven, but this is very different from just a java build.
I already want (and am planning) to allow building unbundled apps with
this scripts.
Those are apps that use public apis only but are built as part of the
Android source tree.
They don't use a normal SDK but instead look at the files in
platform/prebuilts/sdk/
This would be done through a simple property in gradle that lets you
point to a full Android source tree instead of an SDK.
In the builder library there'll be an SDK parser and a parser for the
Android source tree to resolve the compile target and find the
location of android.jar and other tools and files.
So we could technically add a different properties that tells the app
to build using the full api jar instead of the stubbed android.jar
This would only be possible when using a full Android source tree
instead of an SDK, so replacing android.jar itself is not needed.
People who use this will work directly in the source tree and will,
hopefully, know what they are doing (as in, breaking CTS and other fun
stuff)
> Btw I still think it will be crucial to get all open source apps from
> Google to be built with the new build tool e.g. at least the iosched
> and other examples that use only public api's but ideally also all
> apps in the aosp tree. That would be the best proofing ground for the
> build system!
Yes! Internally, people who work on apps are struggling to build with
the source tree build system (oh, makefiles...) while using our tools.
My goal is to move all those apps to the new build system to let them
use the IDE and only have a single way of configuring how their app is
built.
>>>> Lint Integration
So yes, you can build the app with not running lint, but doing "gradle
build<variant>" will run lint by default always. If you want to
disable it completely (why woud you?) you'll have to manually disable
all the checks.
https://android.googlesource.com/platform/tools/build/+/master
As I've mentioned before, we are looking are a new version of the build system provided in the SDK.
Our goal is to make it easy for beginners but flexible and powerful enough for teams working on large complex application(s).
We started looking at the high level features that the build system would offer. This is independent from the implementation of the build system and how those features would be exposed to devs (this will come later). The following is a proposal for these features (not including the basic features currently available, those are a given), and we are looking for feedback.
So what are we trying to do?
- Better extensibility
- Allow devs to configure every steps.
- expose all options for aapt, dex, …
- allow changing input/output for all steps.
- For instance, bypass proguard for a given jar file.
- Use different jar for compile vs. package
- Better control over debug vs. release builds
- manifest info
- BuildConfig
- jni compilation
- different res/code depending on build.
- Let devs add new logic in the build system
- New tasks
- Change dependencies
- Integrate into other build systems.
- Help devs making different versions of the same app.
- Library Projects can be cumbersome to make 2 apps that are 99% identical.
- Different versions for different app store could only differ in package name and signing key.
- Better dependency management for 3rd party jar files and Library Projects
- Use Maven/Ivy for dependency management
- Depend on Library Projects distributed as binary blob
- Generate Library Projects blob
To implement this, we are introducing new build concepts.
Product Flavors
A product flavor defines a customized version of the application build by the project. A single project can have different flavors which change the generated application.
Although different flavors could be very different applications, using Library Projects is a better use case for this, and the build system still supports this.
This new concept is designed to help when the differences are very, very minimum, for instance when using Google Play’s multi-apk support (e.g. the difference could be only gl textures compressed using a different format.)
If the answer to “Is this the same application?” is yes, then this is probably the way to go over Library Projects.
Product flavors can customize the following properties:
- minSdkVersion
- targetSdkVersion
- versionCode
- versionName
- package name (overrides value from manifest)
- release signing info (keystore, key alias, passwords,...).
- native abi filter
- test info
- package name for test app (optional, default is <base>.test)
- InstrumentationTestRunner class (optional)
Additionally, Product Flavor can provide their own source code, resources and manifest.
A project starts with a main product configuration. All the above properties can be set and it has its own source code, resources and manifest.
If the project does not create product flavors, then the application is built from this main configuration.
When a project declares Product Flavors, these extends the main configuration.
The source code of the product flavor is used in addition to the source code of the main configuration (1).
The resources of the product flavor overrides the resources of the main configuration.
The Manifest of the product flavor is merged on top of the Manifest of the main configuration.
(1) This allows some flexibility but also adds some restrictions:
- Classes in the main configuration and in the product flavors can reference each other. All the source sets are used to generate a single output (instead of each generating its own output with a one-way dependency).
- Classes exposed by product flavors and referenced by classes in the main configuration must be present in all product flavors, and must have the same API for all builds to succeed.
When a Product Flavor is defined, the main configuration is not available as a buildable product. To create more than one product, two or more flavors must be created and customized.
Build Types
A build type allows configuration of how an application is packaged for debugging or release purpose.
This concept is not meant to be used to create different versions of the same application. This is orthogonal to Product Flavor.
A Build Type provides configuration for the following build properties:
- manifest debuggable flag
- native compilation debug flag
- proguard enabled + rules
- debug signing flag (ie whether to use debug key or release key)
- package name suffix (2)
- Buildconfig
- DEBUG flag
- other custom properties?
By default the build system provides two build types, debug and release, which can be reconfigured. New types can also be created to provide any combinations of the above properties.
For instance, one could make a Build Type that uses Proguard but creates a debuggable apk.
Sourcesets
Due to Build Types and Product Flavors all providing their own sources, resources and manifest, the project structure need to change from the root level src/res/manifest used in the current build system.
Instead the build system moves to a top level src folder, containing folders for each types and flavors. In addition, the main folder contains the source for the main configuration.
A typical project will then have the following folder structure:
- src
- main
- java
- jni
- resources -- this is java resources
- aidl
- res
- res -- this is Android resources
- assets
- AndroidManifest.xml
- debug
- … (same as main)
- release
- … (same as main)
- flavor1
- … (same as main)
- flavor2
- … (same as main)
The current gen folder is to move under the project output (build folder), and follow a similar pattern. This is accompanied by the generated resources, and the generated (merged) Manifest.
This is of course be completely customizable, but this shows the default, and the type of inputs that are available to tasks.
BuildConfig
BuildConfig is an automatically generated class with a single boolean DEBUG, set to true in debug builds only.
With custom source sets for both Build Types and Product Flavors, the build system covers adding extra flags and triggers (through a user created class) to control what the code does in each build variants. It is a manual process to maintain each version of the same class but it is the most flexible for devs who can do anything they want.
The new build system still makes use of BuildConfig as it is an automatic class creation, and extends its capability.
It provides developers with the ability to programmatically add new items in BuildConfig based on logic defined in the build script.
Details TBD.
Testing
Test Application
In the current build system, test applications are created from a different project that is flagged as a “test project”. This project references the main project in order to get the main project’s classes on its classpath.
With possibly multiple Product Flavors it is important to test all of them.
Some Java build systems have introduced having the test code directly in the project under src/test/ and the new build system follows this convention.
Since JUnit test cannot be run against Android code, the build system reuses this concept for test applications.
There is a mirror of the main configuration, plus flavors which allows creating flavor specific test applications.
The main test configuration uses src/test, and flavors uses src/test<flavor>
The manifest of a test application is pretty basic and it is automatically created it. The build system provides the ability to configure the test package name and the InstrumentationTestRunner class.
Deployment and running all test flavors TBD.JUnit tests
JUnit tests cannot be run against code accessing Android APIs. Instrumentation tests are the sanctioned way to go.
To have code that does not access Android APIs tested with JUnit tests, a multi-project build should be setup with the code to be tested put in a Java-only project.
Note: The build system shouldn’t prevent usage of some third party test frameworks such as Roboelectric. It should actually facilitate integrating those. Details TBD.
aapt / dex / other options.
The build needs to expose all available options and to set them globally or per build variant.
Details TBD.
Building Library Projects.
Library Projects are built in a very similar way to regular projects with a few exceptions.
There are no Product Flavor. This would not make sense as the output is not a Product. Different versions of the same Library should be different libraries and use different projects (possibly depending on the same root Library).
There are debug and release build types but they are used slightly differently.
Testing the library is important and this is done through the test component of the project. However, unlike a normal project, this component doesn’t generate a separate application from only its source set. Instead it implicitly references the library itself and packages it in its own apk.
This build is always a debug build.
When the Library Project is built and packaged into a distribution blob, then the build is always a release build.
Note that the Build Type can here only configure the following properties:
- native compilation debug flag
- Buildconfig
- DEBUG flag
- other custom properties?
The other properties either don’t apply, or only one possible value makes sense.
Libraries must be packaged in a very specific way, no matter what the library source folder structure is.
- classes.jar -- this is the main code used for compile and packaging.
- api.jar -- this is an optional jar to used at compile time instead of classes.jar
- res/ -- this is the res folder(s) extended with:
- crunched images
- renderscript bitcode
- AndroidManifest.xml (includes the merging from other Library Projects?)
- lib/ -- this is the jni component(s)
- aidl -- this is a straight copy of all the src/../aidl files
In all the above cases what is packaged can be extended by the release Build Type, similar to a normal apk.
Additionally, the blob may contain:
- proguard.txt -- custom Proguard rules
- lint.jar -- custom lint rules
- ide.jar -- custom IDE extensions
- docs/ -- this is the javadoc
- source/ -- this is optional source package (for open source libraries).
Using Library Projects.
Library projects can be used two ways:
- Dependency through a repository (local/maven/ivy)
- this downloads the blob and unarchives it somewhere
- As part of a multi-project build.
#1 is fairly standard and can be added as as normal dependency.
#2 is a bit more complicated and needs to be figured out. The main issue is that dependency on Library Projects have a clear dependency order and the setup of the dependency must clearly spell this out.
Looking at IDE integration, #2 is important for cases where Library Projects are not distributed but are instead just a way to modularize an application. This would allow better experience in the IDE (for instance allowing refactoring across the main app and the library(ies.))
Details TBD.Dependencies
There are 2 main types of dependency configurations:
- compile
- package
While package will in most case include the content of compile, it is important to be able to have a dependency that is compile only. This allows having a different version of a dependency for compilation vs. packaging. Details TBD.
It should also be possible to have a dependency skip the Proguard step.
Note that the Android API jar(s) is automatically added to the compile (only) dependency list, and doesn’t have to be declared manually.
Product Flavors (not Build Type) impact dependencies, as well as tests.
This means other dependency configurations are automatically created:
- <flavor>Compile
- <flavor>Package
- testCompile
- testPackage
- test<flavor>Compile
- test<flavor>Package
Lint Integration
The new build system integrates Lint at two levels.
Running Lint
A Lint tasks is created and allows testing any build variant.
Lint errors will break the build.
Lint rules can be enabled/disabled and set priority (warning/error).
HTML and/or XML reports can be generated.
Lint should run on all available Build Variants but remove duplicates (since code common to all variants is compiled for each of them)
Extending Lint
Creating custom Lint is directly possible using the src/lint/ folder.
This creates a lint.jar file that is passed to Lint when running it on the project itself. The dependency on the Lint API is done automatically. All that’s needed is creating the source folder.
If the project depends on Library Projects, their own custom rules are used as well.
Feedback welcome. thanks.Xav