Gradle builds really slow with multi-project setup

[Ashton Cummings]

My Stack post

Using gradle 1.9-rc-3
When building with gradle on a multi-project setup containing roughly 140 projects/libraries, the build time took 1 hour and 22 minutes. And i was using "--parallel". And our ANT build takes less than 20 minutes without parallel building.

Here is what i was initially doing:

./gradlew clean
./gradlew build --parallel

This took 1 hour and 22 minutes, roughly. I did a little testing it seems like the dexing is taking the longest amount of time. Is there a way to get the gradle process to re-use the stuff it has already dexed? If the libraries have already been built, it should re-use the already dexed libraries.

I saw the option --no-rebuild, but when i run with that option it says the following:

File '/path/to/project/build/libs/project.aar' specified for property 'bundle' does not exist.

I replaced the path and the project name with generic stuff.

***

After some assistance i added "preDexLibraries = false" to all of my "build.gradle"  files. However, i still would like to know a centralize place that i can put that entry and it affect all the other "build.gradle" files. Because having to add that to all of the "build.gradle" files is annoying and if i ever have to remove it, it will be more annoying. This did not reduce the time to what we are needing.

After some more assistance, i ran "assembleDebug" instead of "build". This way it will only do debug builds instead of both release and debug. This still did not bring us down to the time we are getting with ANT. With all these changes we are only down to 58 minutes. And our ANT process is less than 20 minutes. Our ANT process is what android provides only modified to support large multi-project setups and to re-use dexed libraries. The dex process in the gradle process is taking 1+ minute for each project. It is by far the largest time consumer.

***

So, how can we speed the process up more? Is gradle re-using the dexed libraries after they have been dexed? What are your suggestions? For us to move over to gradle, it has to be able to compete with the ANT process.

Thanks

[Xavier Ducrohet]

Right now each project will pre-dex its dependencies on its own. This means 2 components depending on the same library will both run pre-dex on that library's classes.jar which is silly. We're looking at fixing this.

Another thing we are looking at is parallelization at the task level. Right now Gradle only supports it at the projects level. It'll build 2 sub projects in parallel (if there are no dependencies between the two), but in a single project it won't parallelize its task. We are working with the Gradle developers to change this.

Of course the tasks themselves can do their work in parallel, and most of our tasks do already. Pre-dexing doesn't though, so if a project has more than one dependencies it'll run pre-dex one by one. This is fixed in the next Gradle version.

We do take performance very seriously, but we also wanted to provide a lot of features and flexibility in our build. This means we do some extra work. For instance our resource merger is extra work that aapt does in a way but differently (more efficiently in some ways, less in others, but with much less flexibility). We are working to do more to optimize each task and, as I mentioned above, provide generic performance improvements as well.

I'm a bit surprised your build went from 1:20 to only 0:58 when running assembleDebug vs build. 'build' does both debug and release but also runs lint. It should be a lot more work than just assembleDebug. Also, do you have flavors?

One other thing to consider is incremental build. Pre-dexing for instance makes clean build clearly slower but incremental builds faster. The rest of the build is also very good at only doing what needs to be done and nothing extra, which Ant wasn't very good at.

Disabling pre-dexing on all projects without touching each project's build.gradle doesn't seem to be easy. The issue is that doing (in the main build.gradle):

subprojects {

  /// some config

}

this gets applied before all the other projects' own build.gradle are applied. So unless you can apply the 'android' or 'android-library' in there, you won't be able to use our DSL. I need to look into this.

--
You received this message because you are subscribed to the Google Groups "adt-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to adt-dev+u...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Xavier Ducrohet
Android SDK Tech Lead
Google Inc.
http://developer.android.com | http://tools.android.com

Please do not send me questions directly. Thanks!

[Ashton Cummings]

Xavier,

Thanks for the information. We are coming to a point now where the decision to switch our build process is coming to an end. We would really like to move to gradle, hopefully you guys are able to get these optimizations in soon. Once we make the decision, we are going to be stuck with it for a while.

Where can i keep up to date with the exact changes that are going into each build of the gradle plugin? I need to know when these changes are in place as soon as possible.

Thanks

[Xavier Ducrohet]

http://tools.android.com/tech-docs/new-build-system contains our changelog.

We generally don't include Gradle specific changes in it even when we start requiring newer versions of Gradle, but performance fixes will probably be indicated in it because we know it's a pain point.

[Xavier Ducrohet]

BTW, we are still a ways off from 1.0. I totally understand not wanting to switch to Gradle right now (especially given your build times), but if you decide to not switch right now, please don't plan to stick with Ant for 2+ years (or some other random long duration).

The Gradle build system will see a lot of changes in the next 6 months. I would recommend revisiting it mid-year at least :)

[Ashton Cummings]

Thanks, that is the one i was using. Just wanted to make sure i was using the right one.

[Ashton Cummings]

I hope that we can. Our alternatives that i am testing are Maven and ANT. Right now our maven approach is showing times that are comparable to our ANT. I personally want to stick with gradle, however, the decision is not up to me. I just give the higher ups my results and conclusions. =)

I will keep an eye out for those performance improvements and will keep our people informed.

Thank you for your time and assistance. If you figure out anything else or have any other suggestions, please continue to contact me. I really hope to convince them to move to gradle.

[Xavier Ducrohet]

Do you remember the time gain you had when disabling pre-dexing?

If you wouldn't mind. I'd love to get the output of running gradle with --profile. It will create a report in the build folder of the main project which will show me build time for all projects and all tasks in all projects.

[Ashton Cummings]

I have been using --profile to figure out the bottlenecks. However, due to the nature of our project, i can not share that information. =/

I can give you times and stuff, just not project names and details. So, unless i modify the data substantially, i can not share the profile info. Sorry.

However, i will give you the build time improvements and where the bottlenecks are. If you need any specifics, let me know. I will try and get you details that i can.

I will get the info to you as soon as i can. I have so much profile data now, i am going to have to run through it and see if i can find the correct data. Might just do the runs again to make sure to get you the correct data.

[Ashton Cummings]

"clean build" with no changes = 3h1m53.88s

Top 15 tasks (roughly the same for each project)

:projectA:preDexRelease	1m26.29s
:projectA:preDexDebug	1m25.14s
:projectA:processDebugManifest	18.452s
:projectA:processReleaseManifest	17.705s
:projectA:lint	10.699s
:projectA:mergeReleaseResources	8.264s
:projectA:dexDebug	7.742s
:projectA:dexRelease	7.449s
:projectA:mergeDebugResources	4.303s
:projectA:compileDebugJava	1.672s
:projectA:processReleaseResources	1.662s
:projectA:processDebugResources	1.325s
:projectA:compileReleaseJava	1.219s
:projectA:packageDebug	1.179s
:projectA:packageRelease	0.923s

clean assembleDebug with pre-dex turned off = 58m58.29s

Top 15 tasks (roughly the same for each project)

:projectA:dexDebug	56.567s
:projectA:mergeDebugResources	5.135s
:projectA:processDebugManifest	3.517s
:projectA:compileDebugJava	1.579s
:projectA:packageDebug	1.174s
:projectA:processDebugResources	0.972s
:projectA:processDebugJavaRes	0.373s	UP-TO-DATE
:projectA:prepareLibrary1UnspecifiedLibrary	0.340s
:projectA:prepareLibrary2UnspecifiedLibrary	0.218s
:projectA:compileDebugAidl	0.057s
:projectA:compileDebugRenderscript	0.046s
:projectA:prepareLibrary3UnspecifiedLibrary	0.045s
:projectA:prepareLibrary4UnspecifiedLibrary	0.043s
:projectA:prepareLibrary5UnspecifiedLibrary	0.029s
:projectA:prepareLibrary6UnspecifiedLibrary	0.028s

I can not find my data for the parallel run. I will do another one of those runs and post the data. Let me know if this is helpful for you.

Thanks

[Xavier Ducrohet]

(pre)dexing is a known issue. I'm not sure how much faster it's going to be in 0.8 with parallel dexing since you have a lot of projects and you can use parallelization there already.

18secs for a manifest merge seems like a crazy amount of time. do you have a lot of manifest data to merge?

I'm going to build a fake multi-project setup with 100s of project to look at scaling issues. Can you tell me a bit more about your inter-project dependencies? Basically do you have 1 apps and 139 flat libs that don't depend on each other, or do you have a fairly deep dependency graph with lots of transitive dependencies?

[Ashton Cummings]

It is a dependency spaghetti. =P

It is a fairly deep dependency graph. With lots of libraries depending on other libraries.

I am going to put a tool together real quick to get you the dependency tree. I will just rename all the projects and libraries. I will try to get it to you today some time.

Thanks for your continued interest and assistance, it is greatly appreciated.

[Siva Velusamy]

You can try "gradle androidDep" to see the dependency tree.

[Xavier Ducrohet]

androidDep is kinda crapy (I did it!), and it's kind of obsolete now that we have composite Configuration object per-variant anyway. It also doesn't show non Android dependencies I think.

What's really missing is a global dependency graph output that shows:

- inter-project dependencies

- dependencies of each project (it could combine the different Configuration, it wouldn't really matter) + local jars (which is missing from the "dependencies" task).

- all of this in a single graph.

[Jake Wharton]

I'd love to see it write a .dot file of this info as well with good labels, colors, and weights to allow visualizing the graph.

---
Jake Wharton
http://about.me/jakewharton

[Xavier Ducrohet]

indeed.

[Ashton Cummings]

Xavier,

I am not sure that you actually got my previous message. Attached are the dependencies.

I "genericized" the names. The dependencies are a nightmare right now, we are planning to go back and clean that up in a later release. Stuff like that happens when you have several teams working on the same project. =/

[Xavier Ducrohet]

Yes, I did, thanks. I'll look into it soon.

[Ashton Cummings]

Thanks

[Ashton Cummings]

Xavier,

Just checking in for an update on this situation.

Thanks

[Xavier Ducrohet]

There are a bunch of things going on, but no silver bullet yet.

I'm working on reenabling the incremental mode of dx but there'll be some limitations until dx is properly fixed. There is some medium term work that will make build faster but nothing to announce yet.

It's going to take me a week or two before I have time to create a similar project to do profiling on my own and see what the actual bottlenecks are (and possibly make this project also compile with Ant to do comparison). The challenge is making a fake project that's big enough in number of libraries, amount of code and resources.

[Ashton Cummings]

Didn't seem like it took us too long to accrue the mess of projects and libraries we have today. lol.

That would take some time, though. I understand, this task for me is just going to be blocked until i get more news from you.

Thanks for your time and efforts, keep up the good work. You guys are doing great. =)

[Ashton Cummings]

Xavier,

Since 0.4.3 update.
I updated my build tools to the most recent version.
I updated my "parent" build.gradle to "0.8.+".
I updated each of my build.gradle files to buildToolsVersion "19.0.1".
And i took out the preDex disabling stuff in each of the build.gradle files.
I also took out the preDex disabling stuff in the gradle-wrapper.properties.
I also updated the properties file to be gradle-1.10-bin.zip.

I ran "gradlew clean assembleDebug --profile" and the build actually took 30 minutes longer than the way i had it before with the older version of everything.

Below is an example build.gradle i have for each of our projects. Excluding the dependencies.

apply plugin: 'android'

dependencies {
    compile fileTree(dir: 'libs', include: '*.jar')
}

android {

    compileSdkVersion 17
    buildToolsVersion "19.0.1"

    sourceSets {
main {
    manifest.srcFile 'AndroidManifest.xml'
    java.srcDirs = ['src']
    resources.srcDirs = ['src']
    aidl.srcDirs = ['src']
    renderscript.srcDirs = ['src']
    res.srcDirs = ['res']
    assets.srcDirs = ['assets']
}

instrumentTest.setRoot('tests')

debug.setRoot('build-types/debug')
release.setRoot('build-types/release')
    }

    lintOptions {
abortOnError false
    }

}

Here is my parent build.gradle.

buildscript{
    repositories {
mavenCentral()
    }

    dependencies {
classpath 'com.android.tools.build:gradle:0.8.+'
    }
}

Here is my gradle-wrapper.properties

distributionBase=GRADLE_USER_HOME
distributionPath=wrapper/dists
zipStoreBase=GRADLE_USER_HOME
zipStorePath=wrapper/dists
distributionUrl=http\://services.gradle.org/distributions/gradle-1.10-bin.zip

I read the patch notes and i was under the impression that you guys added in the ability for it to re-use the already dexed files when dexing projects. However, it still seems to not be doing that, because the dexing is still taking a long time.

1. preDexDebug takes 1+ minutes (sometimes longer)
   
2. dexDebug takes ~10 seconds

If you did add the feature for it to re-use the dexed stuff, how do i enable it or am do i have a setting wrong?

Thanks

[Xavier Ducrohet]

There should be no difference in build time, though if you re-enabled predexing yes it's going to be slower than without it (at least for clean build, since pre-dexing means more build time for clean builds in order to make incremental build faster.)

Note also that the version of Studio should have no impact on build time (there is a little bit of overhead but for large projects this is negligible).

Anyway, performance is important, but there's no easy fix. We are aware of some of the things that are bad, and we have plans to fix them. It'll take some time to get there though, so don't expect major improvements every release.

[Ashton Cummings]

I understand, i was just making sure i did not miss something. Thanks

[Sebastian Schuberth]

On Thursday, January 16, 2014 5:48:52 PM UTC+1, Xavier Ducrohet wrote:

Right now each project will pre-dex its dependencies on its own. This means 2 components depending on the same library will both run pre-dex on that library's classes.jar which is silly. We're looking at fixing this.

Is there a ticket which tracks this? I was unable to find a matching one at http://code.google.com/p/android/issues/list.

Also, can it be that this applies not only to the new Gradle-based build system, but also to the legacy Ant-based build system?

Regards,
Sebastian

[Xavier Ducrohet]

There isn't any issue for it so I created one: https://code.google.com/p/android/issues/detail?id=66807

For Ant, yet technically but it doesn't show up because:

- you don't build variant in Ant

- you can't have more than one "app" project really.

--
You received this message because you are subscribed to the Google Groups "adt-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to adt-dev+u...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Sebastian Schuberth]

On Fri, Mar 7, 2014 at 5:25 PM, Xavier Ducrohet <x...@android.com> wrote:

> There isn't any issue for it so I created one:
> https://code.google.com/p/android/issues/detail?id=66807

Thanks!

> For Ant, yet technically but it doesn't show up because:
> - you don't build variant in Ant
> - you can't have more than one "app" project really.

Hmm, my understanding is that this could also apply to Ant builds if
e.g. the app depends on library projects A and B, and A and B both
depend on e.g. guava.jar (and have it in their "libs" directories),
then guava.jar would be dexed both as part of building A and building
B, or?

--
Sebastian Schuberth

[Xavier Ducrohet]

Libraries don't run dex in Ant. Packaging a library only adds jar files, not dex files.

In Gradle however, if you run tests, a library project will generate a test apk which will indeed (pre)dex its dependencies.

--
You received this message because you are subscribed to the Google Groups "adt-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to adt-dev+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.