Recent changes to dependency resolution

[Daz DeBoer]

G'day

For Gradle 2.5, we'll be adding support for replacing external dependencies with project dependencies (and vice versa). This work has necessitated some pretty major changes to dependency resolution, which have now been incorporated into master.

Here's a short description of some of the changes, for your information.

1. Project dependencies can now appear at any node in the dependency graph

With the change to allow project and external dependencies to be substituted, it is now possible for a resolved configuration to transitively include a project via an external dependency.

It is now possible to have a graph like this:

:my-project -> "my.org:published-module:1.0" -> :my-other-project

Any code that assumes that an external module can only ever transitively depend on other external modules will break. This includes our IDE plugins, which currently fail to deal correctly with transitive project dependencies like this.

2. Any configuration that is a task input is resolved when adding the task to the execution graph

When "taskA" takes "configuration.compile" as an input, this implies that any build dependencies of "configuration.compile" become task dependencies of "taskA". If a configuration includes a project dependency, then this includes the tasks required to build that project dependency.

Previously, we were only looking at the directly declared dependencies of a configuration for any ProjectDependency instances. However, now that a ProjectDependency can appear anywhere in the dependency graph, it is necessary to resolve the configuration to detect all project dependencies transitively.

This means that more dependency resolution will be happening up-front, rather than just before a task is executed. This will likely delay the execution of the first task, and could result in slower builds in case of failure (because we are now resolving configurations for tasks that may never end up executing).

3. Configurations used as task input will be re-resolved if modified during execution

As per point 2), input configurations are now resolved early, when building the task graph. However, execution of one task can modify the dependencies of a configuration that is input to another task. An example is a task that looks up a tool version and adds that tool to the dependencies.

In the case that an already-resolved configuration is modified during task execution, that configuration will be resolved _again_ when used as a task input. Modifying a configuration in this way is now deprecated and we've added some extra checks to catch this case.

4. Artifact resolution is now (somewhat) separated from graph resolution.

The artifacts of a project can be modified during task execution. Instead of forcing a re-resolve when artifacts change (like we do for configuration _dependencies_ as noted in 3), we no defer the resolution of configuration artifacts until they are required for task execution.

This means that dependency resolution is now effectively split into 2 separate actions:

a) Resolve the configuration to a graph of module nodes. Conflict resolution happens at this stage.

b) Resolve the artifacts of each module node in the graph.

We've made efforts to minimise the additional state that is held in memory between resolving the dependency graph and resolving the artifacts, however it is possible that this will have an impact on heap allocation for very large builds. Optimisation and profiling will follow.

--

Darrell (Daz) DeBoer

http://www.gradleware.com

Join us for Gradle Summit 2015, June 11 - 12 in Santa Clara, CA: http://www.gradlesummit.com

[Szczepan Faber]

Hey,

Thanks for info. This sounds great.

Is re-resolution of the configuration planned to be supported in general?

Are those changes impact configure-on-demand in any way?

I'm curious about the impact of more upfront configuration on our
network build. Are those changes included in Gradle master already so
that I can test it out?

Cheers!

> --
> You received this message because you are subscribed to the Google Groups
> "gradle-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to gradle-dev+...@googlegroups.com.
> To post to this group, send email to gradl...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/gradle-dev/CAPRP4rvb-yVd6QeNGfjNQeUT2p1OvhEceC2Ugwm72RWwuMcXXQ%40mail.gmail.com.
> For more options, visit https://groups.google.com/d/optout.



--
Szczepan Faber
Founder mockito.org; Core dev gradle.org
tweets as @szczepiq; blogs at blog.mockito.org

[Daz DeBoer]

On 15 April 2015 at 07:02, Szczepan Faber <szcz...@gmail.com> wrote:

Hey,

Thanks for info. This sounds great.

Is re-resolution of the configuration planned to be supported in general?

No. We'll only re-resolve if a configuration is changed after it is resolved to calculate the task graph dependencies.

- Changes made after a 'task dependency' resolve are deprecated, but will trigger a re-resolve

- Changes made after a 'true' resolve (which also resolves the artifacts) are deprecated, and ignored

 

Are those changes impact configure-on-demand in any way?

Not that I'm aware of.

 

I'm curious about the impact of more upfront configuration on our
network build. Are those changes included in Gradle master already so
that I can test it out?

Yep, please do. The sooner we get feedback on this, the better.

 

To view this discussion on the web visit https://groups.google.com/d/msgid/gradle-dev/CADb1c%2BPWynVpb-Mv6JK-GRQX112qG4GK%3D-7V7%2Bp3NXB3FBHteQ%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

[Szczepan Faber]

Thanks! I'm very happy to see those features getting into gradle.

Cheers!

To view this discussion on the web visit https://groups.google.com/d/msgid/gradle-dev/CAPRP4rs%2BKZinLtfd-1q_qsPOB_-RYb-nLaoaLg1YBAp1gMog%2Bg%40mail.gmail.com.