Lint Performance Tips

[Tor Norbye]

Lint tends to get slower for every release. There's a reason for that: It keeps checking more and more things (as of 3.3 canary we're up to around 350 separate issues), and existing issues are often made more complex as well, doing deeper flow and data analysis, and occasionally there are new features which also adds cost, such as baselines, Kotlin analysis, etc. And this is all compounded by the codebases lint is getting run on also growing bigger and bigger, and being broken up into more and more modules.

A number of you have complained about the performance and asked how to make it better, so I figured it would be helpful to try to write a "performance tuning guide" with tips. There are several things you can do in your setup to run lint a LOT faster.

This is a draft, and I'd like your feedback and results on this, and then one of our doc writers can take the raw material and turn it into real documentation.

-----------------------------------------

Use lintRelease target, not lint

-----------------------------------------

If you run "./gradlew tasks" you'll see that there's a "lint" task, and you may be tempted to run it. But lint also adds specific tasks for each variant. For example, if you only have "debug" and "release" variants (e.g. you haven't defined any product flavors or additional build types), lint will create 3 targets:

"lint"

"lintDebug"

"lintRelease"

Don't configure your CI job to run the "lint" task; pick a specific variant instead, such as "lintDebug". The reason this is so important is that what the "lint" task really does is run lint over and over again, for *each* variant, and then it collates the results in the end; combines all the errors it found, and shows which specific variants every error applies to, unless it applies to all (which is commonly the case).   The reason for this is that you could have a variant-specific issue; for example, you may have a resource file which is only present in a variant-specific source set (such as src/debug/res/values), so the general lint target checks everything. Of course, you're probably only shipping your release variant, so you could limit yourself to just running "lintRelease" and you're not going to miss much.

If you have lots of variants this makes a huge difference; with 7 variants, "lint" will take ~7x longer than "lintDebug" !!  (modulo some caching)

---------------------------------------

Only analyze app/leaf modules

---------------------------------------

If you have divided your project into many smaller modules - a number of libraries and just a couple of app modules, it's much better to (a) turn on checkDependencies mode and (b) only run lint on the app modules, instead of recursively running lint on each module.

To do this, first add this to your app module's build.gradle file:

```

android {

    ...

    lintOptions {

        ...

        checkDependencies true

        ...

    }

}         

```

Then instead of running ./gradlew lintDebug, run ./gradlew :app:lintDebug.

Since you've turned on check-dependencies-mode, running lint on the app module will also run it on all the dependent modules the app depends on -- e.g. all the libraries. But this should also make things run a lot faster, since it's a single lint invocation, where lint shares a lot more computation (such as symbol resolution in the SDK and various shared libraries, etc).  

This isn't just a good idea from a performance perspective; you'll also get more accurate results. For example, the unused resource analysis will now correctly know whether a resource defined in a library is consumed by the app module; that doesn't happen when you run lint first on the library, and later on just the app, which is what happens when you run ./gradlew lintDebug. As a bonus you'll also get a single HTML report containing all the results from your codebase, instead of having to hunt around the tree for all the various reports and look at each one separately.

---------------------------------------

Don't analyze test sources

---------------------------------------

Lint has two flags controlling what to do about test sources: checkTestSources and ignoreTestSources. These mean different things.

"checkTestSources" controls whether lint should run all the normal checks on the test sources. This is already off by default, so there's really no performance reason to tweak it. The only reason you'd turn this on is if you really want to make sure your test sources are pristine as well.

"ignoreTestSources" on the other hand controls whether lint should really ignore all test sources. Even if lint doesn't run normal checks on the test sources, it still has to include them in analysis for some of the regular lint checks for production code. For example, what if you have a resource which is only referenced from a test; do you then want to flag this as unused when the test is still referencing it?

Add this to your app build.gradle file to tell lint to completely ignore your test sources, which should help lint run faster since (for well tested code) this can help skip a lot of code analysis and type resolution:

```

android {

    ...

    lintOptions {

        ...

        ignoreTestSources true

        ...

    }

}         

```

---------------------------------------

Don't use checkAllWarnings

---------------------------------------

Lint has a "checkAllWarnings" option you can use to turn on checking of all warnings, including those that are off by default. This flag is off by default, but some developers turn it on ("because I want all the tips I can get" is what I heard from one developer). 

Be careful doing that, because there are some lint checks, in particular the one with the id "WrongThreadInterprocedural", which is *incredibly* slow. If you're going to do this, consider at least disabling that one (add android.lintOptions.disable 'WrongThreadInterprocedural') and adding it in explicitly when you really want to run that check.

---------------------------------------

Use latest version

---------------------------------------

Try using the latest versions of lint (the Android Gradle plugin), as well as of Gradle -- even if that means using a canary version. Yes, by all means, for your production builds use a stable version of the Gradle plugin to build your release APK/bundle, but for your CI server, consider running the latest preview version of lint, since generally those versions will have the most up to date fixes. (Yes, regressions happen and canaries go through less testing, but in general lint has really good coverage so regressions do not happen very often.)

---------------------------------------

Give lint a lot of RAM

---------------------------------------

This one is kind of obvious, but lint (especially the code analysis part) is really memory hungry; it does a lot of the same work as a compiler, except that it also hangs on to a lot more data (e.g. the full ASTs with whitespace and comments etc).  Explore bumping up the memory given to the Gradle daemon significantly to see if it helps bring down the overall analysis time.

This is *especially* important if you happen to run many lint jobs in parallel! If you follow the first advice above (where you're running a single lint job with recursive dependencies on the app module) this is less likely to happen, but I've seen various thread dumps from users asking about performance where there was 16-20 concurrent separate lint jobs running in the Gradle daemon, and each one requires a lot of memory; these separate lint job threads are not sharing data. 

---------------------------------------

Did I forget anything? Is the above clear? Did it help, and if so, how much?

-- Tor

[snic...@groupon.com]

Thx for the tips Tor.

Though, instead of using lint on the leaf module and checking dependencies, it looks like we would have better performances if each module could be analyzed and if this analysis could be cached. It should not be very complicated to check that things have not changed and that lint output should be the same. This is really a feature that that would help reducing build times, especially on CI.

I am not asking for incrementality of lint, which would actually rock but is hard to achieve, but caching is definitely needed and **it's not hard to achieve**.

[Tor Norbye]

On Mon, Jul 16, 2018 at 10:22 AM snicolas via lint-dev <lint...@googlegroups.com> wrote:

Thx for the tips Tor.

Though, instead of using lint on the leaf module and checking dependencies, it looks like we would have better performances if each module could be analyzed and if this analysis could be cached. It should not be very complicated to check that things have not changed and that lint output should be the same. This is really a feature that that would help reducing build times, especially on CI.

I am not asking for incrementality of lint, which would actually rock but is hard to achieve, but caching is definitely needed and **it's not hard to achieve**.

I don't think it's quite that easy. Consider unused resource analysis for example. The way that works is that it accumulates all resource declarations, and all resource references, across the codebase. When that's all done, it produces warnings for all resources that were declared, but not referenced.  It's not clear to me how this would be cached. One way would be for the unused resource detector to write the set of resource definitions and resource references to a file that could then be loaded downstream, or for the unused resource detector to report unused resources for the library but then somehow have this filtered out during app analysis if it finds references that references a resource already reported in the library as unused.

In both scenarios, the specific detector would have to be updated; this isn't just a matter of modifying lint's infrastructure to handle caching of errors from each library. And once we're in the world of updating detectors for this, there are hundreds of detectors so not trivial.

There are many reasons it's advantageous for lint to look at the whole project:

(1) Many errors take into consideration factors like the minSdkVersion, the compileSdkVersion, etc. These are typically defined in the app module. A library can have a lower minSdkVersion than the app module, so when performing analysis you really want to apply the effective API level. In global analysis, that's what lint looks for (detectors are passed in not just local information, but a reference to the main project as well (typically the app project), and lint rules which look up the minSdkVersion pretty much always look up the main minSdkVersion, not the local one.

(2) Lint analysis benefits from considering the *source code* of libraries, not just the compiled output. For example, consider this code:

         if (Utils.isLollipop()) {

              new ToolBar().attach();

        }

Here, new ToolBar() requires API level 21. But lint is smart enough to not print a warning here. How does it do that? It looks at all the surrounding calls, notices the "if (Utils.isLollipop()") call, and wonders to itself "is that a version check call?"  To find out, it resolves the call, finds the corresponding source code, and peeks inside the method body to determine if that method applies a version check which implies that the version code is at least 21. Being able to do this requires access to the source code (in theory it could be done using bytecode analysis on the library, but the bytecode based version checks aren't nearly as good as the source code based ones.)

(3) Performance wise, doing the analysis for each module and then caching the output for downstream modules sounds good in theory, but in practice the most costly part of the lint analysis today is the top down analysis phase with type attribution. This is where lint passes in all the source files and jars and it figures out all the type resolution matches -- types, resolve results, etc. This is very expensive; much more expensive than the checks lint runs after the fact on the ASTs. That's why turning off a lint check typically doesn't make lint run any faster; it's the up front computation that's expensive, not the iterative checks. Anyway, a lot of these modules tend to repeat the same large jars -- android.jar, support lib jars etc. When you're analyzing each module in isolation there's a lot more repeated work reading and analyzing these jar files.

-- Tor

[Cristian Garcia]

On a project with 39 modules (Gradle projects)
Parallel enabled, only 2 cores (I could try with 4 to improve the results on the `lint` option), remote BuildCache enabled (doesn't affect Lint but it reduces time for other tasks).

Running `lint` - 6 mins
nearly all the time both cores are used

Running `:app:lintRelease:` with checkDependencies enabled - 3 mins 18 sec
Only one core is used.

Removing checks on tests only gave us 3 extra seconds (it's not a project with a lot of unit tests really, it may be more relevant on other projects).

Data from Gradle Scans.

It's the opposite approach of what I "wanted", running it by modules so I can paralelize and take advantage of BuildCache to run only analysis on what changed...but the results are quite good so I can't really complain.

Is it ignoreTestSources only on AGP 3.2+ ?
(Just saying because I see it complains in AGP 3.1.3)

[Roberto Leinardi]

Hi Tor, thanks for the tips!

I have one question about variants and link: currently I am working on an app that has multiple flavors and the :app:lintDebug does not exist since we have the flavor specific tasks (eg. :app:lintFoo1Debug, :app:lintFoo2Debug, :app:lintFoo3Debug, etc).

I would like to run lint on all possible Debug variants and I would like to collect the report in a single file (so, not split by variant). Unfortunately I have not found a solution yet because if I run :app:lint it will check all the variant but also both Debug and Release, and if run all the wanted specific lint tasks (:app:lintFoo1Debug, :app:lintFoo2Debug, :app:lintFoo3Debug, etc) I will check only the variants I want but I will get multiple report files (one per each variant).

Is there a way to run link on all the Debug variants and get just a single, common report?

[Tor Norbye]

No, that isn't possible right now. Would you mind filing a request for this? (Help > Submit Feedback.)

-- Tor

[andreas.r...@airbnb.com]

On Sunday, July 15, 2018 at 10:12:04 PM UTC-7, Tor Norbye wrote:

---------------------------------------

Give lint a lot of RAM

---------------------------------------

This one is kind of obvious, but lint (especially the code analysis part) is really memory hungry; it does a lot of the same work as a compiler, except that it also hangs on to a lot more data (e.g. the full ASTs with whitespace and comments etc).  Explore bumping up the memory given to the Gradle daemon significantly to see if it helps bring down the overall analysis time.

This is *especially* important if you happen to run many lint jobs in parallel! If you follow the first advice above (where you're running a single lint job with recursive dependencies on the app module) this is less likely to happen, but I've seen various thread dumps from users asking about performance where there was 16-20 concurrent separate lint jobs running in the Gradle daemon, and each one requires a lot of memory; these separate lint job threads are not sharing data. 

We hat exactly the opposite experience (and this is on AGP 3.0 just FYI, I still have to run tests on 3.2 or even 3.3). We were running lint with 29g of RAM (don't know where the number is coming from, but it was there). And without just a few checkins our lint time went from 10m to over 120m.

Through experimentation I found out that given the VM less heap (12g) makes it more deterministically taking about 20m (still twice as slow as before). I'm going to try a few of the tricks from your post Tor, but I would also like to further debug what is going on here. Any tips what I could look at? (The machine itself is not swapping (it has 64g of RAM). 

-- 

Andreas

[Niek Haarman]

About "checkDependencies true", for 'normal' apps often there is a single 'app' leaf module. What if you have multiple (say 5) leaf modules that depend on one single library module? Will "checkDependencies true" cause lint to run five times for the library module?

[Said Tahsin Dane]

I think `checkDependencies true` makes it only run once for the `app` module but also checks the sources from dependant modules. 

Other benefits include smarter checks (especially for UnusedResources) since it knows the whole system. Also it makes use of the final merged manifest file which has the complete information. 

In my personal tests, I didn't see any performance benefit of `checkDependencies` over running lint separately on every single module. In fact running lint many times on every single module was faster. 

On Tue, Aug 21, 2018 at 5:51 PM Niek Haarman <haarma...@gmail.com> wrote:

About "checkDependencies true", for 'normal' apps often there is a single 'app' leaf module. What if you have multiple (say 5) leaf modules that depend on one single library module? Will "checkDependencies true" cause lint to run five times for the library module?

--
You received this message because you are subscribed to the Google Groups "lint-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lint-dev+u...@googlegroups.com.
To post to this group, send email to lint...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/lint-dev/8340c428-2cf4-48a2-8303-31a56ee0fbc9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Tor Norbye]

On Tuesday, August 21, 2018 at 7:51:11 AM UTC-7, Niek Haarman wrote:

About "checkDependencies true", for 'normal' apps often there is a single 'app' leaf module. What if you have multiple (say 5) leaf modules that depend on one single library module? Will "checkDependencies true" cause lint to run five times for the library module?

Yes. That's why (with quite a bit of hesitation) I added the checkDependencies flag and turned it off by default (the recursive analysis used to be how it always behaved). It helps performance in deeply nested projects with lots and lots of multiply reused libraries. But it does have the downside of less accurate analysis in the context of the app (when a library is analyzed on its own, we don't know the true minSdkVersion, targetSdkVersion, compileSdkVersion, permissions available, resources consumed elsewhere, etc etc).

-- Tor

[Niek Haarman]

On Wednesday, August 22, 2018 at 3:22:26 PM UTC+2, Tor Norbye wrote:

Yes. That's why (with quite a bit of hesitation) I added the checkDependencies flag and turned it off by default (the recursive analysis used to be how it always behaved). It helps performance in deeply nested projects with lots and lots of multiply reused libraries. But it does have the downside of less accurate analysis in the context of the app (when a library is analyzed on its own, we don't know the true minSdkVersion, targetSdkVersion, compileSdkVersion, permissions available, resources consumed elsewhere, etc etc).

-- Tor

Good to know! Perhaps a little useful feedback: I turned on checkDependencies, and found a new Lint error in a java module. Lint was never run on the java module before of course, and now the recursive behavior triggered a lint error on an @CheckResult violation. So having this off by default might cause a false sense of security for java modules (I hadn't realised Lint wouldn't run on the java module). 

[Michael Bailey]

Is there a way to get timing info on each lint check for our build, so that we can analyze the tradeoff between value of the check, versus cost in added build time?

-Michael

On Sunday, July 15, 2018 at 10:12:04 PM UTC-7, Tor Norbye wrote:

[Michael Bailey]

Is the expected behavior of  `checkDependencies true` that it would also check dependent modules that don't apply the Android plugin (just plain java/kotlin) ? 

-Michael

On Sunday, July 15, 2018 at 10:12:04 PM UTC-7, Tor Norbye wrote:

[Michael Bailey]

Regrading "checkDependencies true" ... " this should also make things run a lot faster, since it's a single lint invocation" 

Is `checkDependencies` faster for incremental lint checks? Seems like if you have per-module tasks for lint, then you will save time on incremental lint check because some of those tasks will be "up-to-date", no?

On Sunday, July 15, 2018 at 10:12:04 PM UTC-7, Tor Norbye wrote:

[Said Tahsin Dane]

In my experience, lint task is never up to date. Even when you run lint for each task, it runs all of them all the time.

I've done a benchmarking myself with our project. Running "checkDependencies true" with main module vs running lint on all modules. If we compare the speeds of these 2, checkDependencies is slower for me. I know they are not the same but since I've seen the comparison, I wanted to let you know.

--

You received this message because you are subscribed to the Google Groups "lint-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lint-dev+u...@googlegroups.com.
To post to this group, send email to lint...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/lint-dev/06169137-3314-46fa-a441-1beef04d5e43%40googlegroups.com.

[Tor Norbye]

The primary reason to run checkDependencies true is to

(1) let libraries be analyzed with the context from the app module (e.g. having a correct minSdkVersion coming from the app), considering the whole picture (e.g. all resources and resource overrides, calls, included libraries etc). Some checks, such as the unused resource check, is only correct when it gets to look at the whole project in one piece.

(2) get a single, integrated report, instead of having to dig through individual smaller report files, one per module.

It can also have the side benefit of being significantly faster, but this depends on the shape of your project. The savings are probably biggest if you have a single app module referencing lots of libraries. If you have many leaf nodes it's probably not helpful.

-- Tor

To unsubscribe from this group and stop receiving emails from it, send an email to lint-dev+unsubscribe@googlegroups.com.

[Said Tahsin Dane]

I have a mixed feeling about unused resources. UnusedResources is always given as the biggest example for checkDependencies but I have the contrary opinion. I like to keep my modules fully encapsulated. If a resource is not used within the module where it is defined, that's a problem for me. What do you think?

To unsubscribe from this group and stop receiving emails from it, send an email to lint-dev+u...@googlegroups.com.

To post to this group, send email to lint...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/lint-dev/06169137-3314-46fa-a441-1beef04d5e43%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "lint-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to lint-dev+u...@googlegroups.com.

To post to this group, send email to lint...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/lint-dev/463a0798-175f-4f52-b6c1-40bacc4a7848%40googlegroups.com.

[gle...@fitbit.com]

We use common resources (especially strings - and not always used in the common UX module), because having to update 15 modules that use the same localized text for "Network not available", or making sure that they all use the same theme, etc is a huge pain and is definitely error prone.  That said, everything lint is getting out of hand in terms of the performance cost.  Our builds are 2 minutes, tests are 8 minutes, and lint is another 17 minutes - and that's on our fastest CI executor.  This is with checkDependencies false and the UnusedResources check turned off.  One app module, ~80 library modules.  

Last time I tested checkDependencies true and UnusedResources turned on, linting our main app module was north of 40 minutes.

[Kostya Vasilyev]

It seems that Lint also spends a lot of time resolving unnecessary depenendencies...

Seeing this with Gradle 4.10.2 and Android Tools 3.2, but don't think it's new.

We have several build flavors (for Google Play, for direct web site download, for OEMs...) and I see that running

gradle lintMarketRelease

( Market here is build flavor, Release is build type )

spends *a lot* of time "resolving dependencies" on build tasks that that have to do with 1) other build flavors (not Market) and 2) other build types and which therefore aren't needed at all.

Once that is done - the lint check itself is quite fast, because we only enable a few inspections like this:

android {

lintOptions {

checkReleaseBuilds true

disable '...

check ...

fatal ...

warning ...

textReport true

textOutput 'stdout'

xmlReport false

htmlReport false

}

}

Is there some way to troubleshoot why running lint on specific build flavor / build type - resolves dependencies for apparently all flavors / types?

-- K

[Tor Norbye]

Lint's Gradle integration depends on having Gradle construct a full tooling model (with dependencies etc for lint to then consider). It sounds like we might be computing too much there. I'll mention this to the Gradle team.

-- Tor

[Tor Norbye]

If you've been suffering from lint performance problems, please give 3.4.0-alpha07 a try. It has a pretty massive speed improvement for larger projects. Let us know how it goes! 

[gle...@fitbit.com]

With the new performance improvement, does it require specific version of gradle?  Is it now better to use checkDependencies and only run lint on the app module, or is it still more performant to run lint on everything individually with checkDependencies off?

Regardless, I'll give it a try this weekend and report back.

[Michael Bailey]

Our `:app:lintDebug` with `checkDependencies` is TWICE as fast with 3.4a7 ! thanks! 

[Tor Norbye]

On Fri, Dec 7, 2018 at 10:50 AM gleach via lint-dev <lint...@googlegroups.com> wrote:

With the new performance improvement, does it require specific version of gradle? 

Yes, 3.4.0-alpha07. We're also cherrypicking it back to an upcoming 3.3 RC.

 

Is it now better to use checkDependencies and only run lint on the app module, or is it still more performant to run lint on everything individually with checkDependencies off?

There's no change there (there are tradeoffs depending on the shape of your module graph.)

-- Tor

[gle...@fitbit.com]

On Friday, December 7, 2018 at 11:28:32 AM UTC-8, Tor Norbye wrote:

On Fri, Dec 7, 2018 at 10:50 AM gleach via lint-dev <lint...@googlegroups.com> wrote:

With the new performance improvement, does it require specific version of gradle? 

Yes, 3.4.0-alpha07. We're also cherrypicking it back to an upcoming 3.3 RC.

Sorry, I meant Gradle itself - 4.10.2, 5.0, etc.  Great to hear that it's come to 3.3 too though.  

 

 

Is it now better to use checkDependencies and only run lint on the app module, or is it still more performant to run lint on everything individually with checkDependencies off?

There's no change there (there are tradeoffs depending on the shape of your module graph.)

Can you speak to what kind of module graphs will be improved the most?

[gle...@fitbit.com]

Quick update: the performance improvements look very nice!  "./gradlew :app:lintDebug" (with checkDependencies=true) is about 2x faster (51minutes down to 26min on engineer laptop; 28 minutes down to 9 minutes on our CI server), and a little over 2x improvement for the all-module "./gradlew lintDebug" (with checkDependencies=false) - 12 minutes down to 5 minutes on an engineer laptop. [haven't run that test on CI yet, but expect similar]

On Thursday, December 6, 2018 at 12:34:49 PM UTC-8, Tor Norbye wrote:

[Cristan Meijer]

I have a very large project (with over 2000 classes) and giving lint more memory had a huge effect:

org.gradle.jvmargs=-Xmx2g: app:lintDebug takes 25m 18s
org.gradle.jvmargs=-Xmx8g: app:lintDebug takes 8m 57s!

You're testing your entire app at once while performing lint recursively, so it makes sense you need a large amount of memory.

Op maandag 16 juli 2018 07:12:04 UTC+2 schreef Tor Norbye:

[gle...@fitbit.com]

Hey Tor,

Has the idea of forking the lint process out of the gradle daemon been explored?  Especially with the new Gradle 5.0 heap defaults, it seems like Gradle might be wanting to push more processing out of the daemon process itself.  If that happened, it'd be a great way to be able to give the lint process more memory via it's own jvm args.

Thoughts?

[Tor Norbye]

On Monday, December 17, 2018 at 10:17:38 PM UTC-8, gle...@fitbit.com wrote:

Hey Tor,

Has the idea of forking the lint process out of the gradle daemon been explored?  Especially with the new Gradle 5.0 heap defaults, it seems like Gradle might be wanting to push more processing out of the daemon process itself.  If that happened, it'd be a great way to be able to give the lint process more memory via it's own jvm args.

It's not something we've explored. If we did it we'd need to make it optional. This seems like a good architecture for large CI machines, but we want Studio to run well on machines with 8GB, and every time we split out a separate process we run the risk of running out of memory since the separate processes add up: studio, emulator, gradle daemon, kotlin daemon, plus other processes developers want to run simultaneously such as Chrome for stackoverflow etc. (And we'd like to make it possible to run on 4GB too though that's obviously challenging.)

Is there some obvious performance advantage to running lint out of process rather than giving the daemon more ram than the default sizes?

-- Tor

[pcha...@pandora.com]

Thanks for all the suggestions. Our lint performance is pretty poor at the moment, so I'm going to try out several of these ideas and see what works. 

Another idea we've been discussing internally is to just run lint on the files that have changed since the last build on our CI environment. It seems like lint only support exclusions (via the ignore element); no direct support for including files. But I thought I might be able to dynamically generate a lint.xml that essentially excludes everything except the files that have been modified. Before I attempt this, I wanted to get your thoughts on whether this would work. Is it practical, or does the nature of lint's work not really lend itself to running incrementally?

---------------------------------------

Give lint a lot of RAM

---------------------------------------

This one is kind of obvious, but lint (especially the code analysis part) is really memory hungry; it does a lot of the same work as a compiler, except that it also hangs on to a lot more data (e.g. the full ASTs with whitespace and comments etc).  Explore bumping up the memory given to the Gradle daemon significantly to see if it helps bring down the overall analysis time.

This is *especially* important if you happen to run many lint jobs in parallel! If you follow the first advice above (where you're running a single lint job with recursive dependencies on the app module) this is less likely to happen, but I've seen various thread dumps from users asking about performance where there was 16-20 concurrent separate lint jobs running in the Gradle daemon, and each one requires a lot of memory; these separate lint job threads are not sharing data. 

[Tor Norbye]

On Tuesday, February 5, 2019 at 6:36:05 PM UTC, pcha...@pandora.com wrote:

Thanks for all the suggestions. Our lint performance is pretty poor at the moment, so I'm going to try out several of these ideas and see what works. 

First of all just making sure you've tried 3.3.0 since we fixed a big performance problem in the last 3.3 RC that had been there since 3.1.

 

Another idea we've been discussing internally is to just run lint on the files that have changed since the last build on our CI environment. It seems like lint only support exclusions (via the ignore element); no direct support for including files. But I thought I might be able to dynamically generate a lint.xml that essentially excludes everything except the files that have been modified. Before I attempt this, I wanted to get your thoughts on whether this would work. Is it practical, or does the nature of lint's work not really lend itself to running incrementally?

If you have a large source tree with lots of modules and only a few files have changed in one of them, this should help a lot since you won't be running lint at all in all the non-affected modules. 

But generating lint.xml probably won't avoid this work; lint will simply not *report* the bugs but it will still do all the work up front to compute them. What you instead have to do is invoke lint more directly by computing your own project definition file, "project.xml". There's some examples of how to do this in ProjectInitializerTest.kt -- and also that when you do this you miss out on some of the Gradle integration of lint.

-- Tor

[pcha...@pandora.com]

Thanks for the prompt reply Tor. And yes, I'm evaluating AGP 3.3.0 - it is indeed substantially faster than AGP 3.2.1. However, we are running into the "Lint infrastructure errors" described below, about 40% of the time:

https://issuetracker.google.com/issues/119306837

https://issuetracker.google.com/issues/122674823

I see that these issues have been resolved, and are being backported to 3.3.x - can you share any information around when these fixes might land in a potential AGP 3.3.1 release? I hesitate to upgrade to an alpha/beta 3.4 release in our CI environment.

[Tor Norbye]

On Wed, Feb 6, 2019 at 5:31 PM <pcha...@pandora.com> wrote:

Thanks for the prompt reply Tor. And yes, I'm evaluating AGP 3.3.0 - it is indeed substantially faster than AGP 3.2.1. However, we are running into the "Lint infrastructure errors" described below, about 40% of the time:

https://issuetracker.google.com/issues/119306837

https://issuetracker.google.com/issues/122674823

I see that these issues have been resolved, and are being backported to 3.3.x - can you share any information around when these fixes might land in a potential AGP 3.3.1 release? I hesitate to upgrade to an alpha/beta 3.4 release in our CI environment.

It looks like the release is imminent (but as always, the brake can be pulled at the last minute etc etc so this is not a promise or a commitment). I'll ping this thread when it goes out.

-- Tor

[Tor Norbye]

On Wednesday, February 6, 2019 at 5:35:30 PM UTC, Tor Norbye wrote:

I see that these issues have been resolved, and are being backported to 3.3.x - can you share any information around when these fixes might land in a potential AGP 3.3.1 release? I hesitate to upgrade to an alpha/beta 3.4 release in our CI environment.

It looks like the release is imminent (but as always, the brake can be pulled at the last minute etc etc so this is not a promise or a commitment). I'll ping this thread when it goes out.

It should be live now!

-- Tor

[Jeroen Mols]

Thanks for the performance tips!

One thing that is key to our workflow is to have blazing fast pull request CI builds. Here we try to balance quality vs integration speed. (e.g. for our Espresso tests, we run a subset (~4min) on our PRs and the full suite (~25 min) when the PR gets integrated to master.)

We are exploring to do something similar for lint: run a basic set of checks on every PR (super fast) and run the full suite of checks on integration to master (time is less of a concern).

For this it would be interesting to know how much time lint is spending on each check, so we can make a good choice of what checks to run where.

Is it possible to generate such "lint performance" reports and get timing info per check? (seems like Michael Bailey has also expressed interest in this) 

[Tor Norbye]

On Wed, Feb 20, 2019 at 6:43 AM Jeroen Mols <jmols....@gmail.com> wrote:

Thanks for the performance tips!

One thing that is key to our workflow is to have blazing fast pull request CI builds. Here we try to balance quality vs integration speed. (e.g. for our Espresso tests, we run a subset (~4min) on our PRs and the full suite (~25 min) when the PR gets integrated to master.)

We are exploring to do something similar for lint: run a basic set of checks on every PR (super fast) and run the full suite of checks on integration to master (time is less of a concern).

For this it would be interesting to know how much time lint is spending on each check, so we can make a good choice of what checks to run where.

Is it possible to generate such "lint performance" reports and get timing info per check? (seems like Michael Bailey has also expressed interest in this) 

Yes, it's possible, though it's not trivial; Matthew figured out how to do it (using a JVM agent) and we're hoping to write about it in our new Project Marble blog series. This is how we found a couple of unexpectedly slow detectors that we fixed; we just didn't have that insight before.

One caution though: If you enable *any* detector which looks at Java or Kotlin code, then lint needs to look at (=parse and symbol resolve) all the code, and that's really expensive. So it's generally not the case that if you disable half the checks, lint will run twice as fast; there's fixed cost up front which is pretty large.

-- Tor

[ja...@orionlabs.io]

Thanks for the tips. 

One small thing I noticed is that I had to re-establish my baselines when switching from running lint on every module to running on just leaf modules + adding checkDependencies true.