Transitive dependencies

[fabian....@gmail.com]

Hi,

I just came across this discussion, and would like to share my experience as a Java developer:

The main problems we have in many projects are version conflicts with transitive dependencies:

Library A depends on Library C version 1.0
Library B depends on Library C version 1.1

You want to use Library A and Library B.

By default, Maven does not care about these conflicts, it packs both versions of library C into the Classpath, and it is kind of random which one will be used at runtime. This often results in hard-to-find run-time errors.

Therefore, many Java developers use the `maven-enforcer-plugin` which can be configured to make the build break if version conflicts are detected.

Maven supports an `exclude` option to selectively disable transitive dependencies. For example, you could configure Maven to use Library A, but `exclude` the dependency to Library C.

With the `maven-enforcer-plugin` enabled, you are forced to use `excludes` to manually resolve dependency conflicts, otherwise you cannot build the project. This is sometimes a painstaking process, but it is the best solution i'm aware of.

My opinion: I find transitive dependencies useful and would recommend to enable them by default. However, I also find it useful to break the build and force the user to fix it as soon as any conflict occurs.


/Fabian

[Henrik Johansson]

I thought the best way to handle this in maven was via dependencyManagement where you can state that "let's always use version 1.1 for C".

Anyway you are right that this will happen in Go as well and afaik there are many thoughts and discussions going on regarding this within the community.

Simply providing the override mechanism to abort and allow the user to choose somehow is simple and something any tool can choose to do.

I seem to remember discussions somewhere about performing some static analysis to provide a best guess of which version to use but that seem much harder and possibly dangerous if applied automatically. Cool though if it could be made to work.

--
You received this message because you are subscribed to the Google Groups "Go Package Management" group.
To unsubscribe from this group and stop receiving emails from it, send an email to go-package-manag...@googlegroups.com.
To post to this group, send email to go-package...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Fabian Stäber]

I thought the best way to handle this in maven was via dependencyManagement where you can state that "let's always use version 1.1 for C".

I think it would be dangerous to have an option like "let's always use version 1.1 for C". When you use this option, and then later introduce Library A as a dependency, there is nothing that makes you aware that you are changing Library A's dependency. I would prefer to explicitly exclude the transitive dependencies for each library. It might be more verbose, because if you have multiple dependencies on Library C you have to exclude it multiple times, but at least you will always explicitly know what you are doing.

BTW: This is off-topic, but to my knowledge dependencyManagement in Java is only for explicit dependencies, not for transitive dependencies that come with a library.

[Henrik Johansson]

Maven 1 used to work that way but everyone got tired of explicitly stating all dependencies so therefore transitive resolution came about in Maven 2.

I think locking down a given version as in the C 1.1 case isn't that bad. Certainly not worse than randomly picking whatever gets resolved last.
How is you suggestion safer? Perhaps I am missing the point? 

[Fabian Stäber]

I totally agree that automatic resolution of transitive dependencies is a good thing. My point is only about what happens if there is a version conflict.

Here's an example:

Library A depends on Library C version 1.0 
Library B depends on Library C version 1.1 

Library X depends on Library C version 1.0

Imagine a project that needs only Library A and Library B.

Given a global option to lock the version of C, a config would be like this:

  - global: C version is 1.1

  - dependency: A

  - dependency: B

Now if you decide at a later point to add a dependency to X, it would be like this:

  - global: C version is 1.1

  - dependency: A

  - dependency: B

  - dependency: X

At this point, nothing will warn you that you changed Library X's dependency from C 1.0 to C 1.1. These things are sometimes very hard to see if there's a large project.

On the other hand, if there is no option to globally lock C's version, the initial config would be like this:

  - dependency: A, excluding C

  - dependency: B

after adding X, the config would be like this:

  - dependency: A, excluding C

  - dependency: B

  - dependency: X, excluding C

You repeat excluding C where ever needed. It is more verbose to specify the excludes for each dependency, but it is explicitly clear what you are doing for each library.

To unsubscribe from this group and stop receiving emails from it, send an email to go-package-management+unsub...@googlegroups.com.

[Henrik Johansson]

And if you forget to exclude C from X what happens then?

To unsubscribe from this group and stop receiving emails from it, send an email to go-package-manag...@googlegroups.com.

To post to this group, send email to go-package...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "Go Package Management" group.

To unsubscribe from this group and stop receiving emails from it, send an email to go-package-manag...@googlegroups.com.

[Fabian Stäber]

On Wednesday, August 3, 2016 at 2:46:10 PM UTC+2, Henrik Johansson wrote:

And if you forget to exclude C from X what happens then?

My opinion is: The build should fail if you forget to exclude C from X. This forces the user to explicitly resolve all conflicts. No conflict should be resolved automatically, and no version should be implicitly set by some global parameter.

That's exactly what happens if you use the maven-enforcer-plugin in Maven, and in my experience this is a good thing.

[Konstantin Shaposhnikov]

> By default, Maven does not care about these conflicts, it packs both versions of library C into the Classpath

I'd like to correct this statement. Maven always uses only one version
of each dependency. The rules to determine the exact version are
somewhat complicated
(https://maven.apache.org/guides/introduction/introduction-to-dependency-mechanism.html#Transitive_Dependencies)
but deterministic.And as Henrik pointed out it is always possible to
lock the exact version of any transitive dependency.

Using excludes and maven-enforcer-plugin to prevent conflicting
versions is possible but it very quickly becomes complicated when
number of dependencies grows. Transient dependencies of transient
dependencies makes things even more complicated. To be honest I have
never seen excludes being used this way.

[Henrik Johansson]

I think this at least highlight that there is some things to consider when talking about versions of dependencies.

There are lot's of cases and problematic things that at least affect usability.

Dropping down to user input is safe but if it is at all possible to make good automatic decisions that would be great if at the same time the dependency graph and how it was constructed can be visualised.

[Fabian Stäber]

On Wednesday, August 3, 2016 at 3:55:15 PM UTC+2, Konstantin Shaposhnikov wrote:

> By default, Maven does not care about these conflicts, it packs both versions of library C into the Classpath

I'd like to correct this statement. Maven always uses only one version
of each dependency. The rules to determine the exact version are
somewhat complicated
(https://maven.apache.org/guides/introduction/introduction-to-dependency-mechanism.html#Transitive_Dependencies)
but deterministic.

Ok, I should have written that 'both versions may potentially end up in the Classpath'. If you aren't aware of a conflict, it doesn't help that the winning version is derived by a deterministic process.

 

And as Henrik pointed out it is always possible to
lock the exact version of any transitive dependency.

Cool, I didn't know that dependencyManagement can be used to overwrite the version of transient dependencies. I only used it for dependencies where no version has been specified.

 

Using excludes and maven-enforcer-plugin to prevent conflicting
versions is possible but it very quickly becomes complicated when
number of dependencies grows. Transient dependencies of transient
dependencies makes things even more complicated. To be honest I have
never seen excludes being used this way.

Using excludes and the enforcer-plugin that way is the only way to be sure that you didn't introduce any implicit dependency conflicts without knowing it. Of course it's a radical approach, because when you add a dependency you must exclude all conflicts, but after all it happens not too often and it forces you to keep the dependency tree clean.

If you don't like this as default behaviour, it would at least be good to have an 'enforce' option for people like me :)

 

On 3 August 2016 at 14:39, Fabian Stäber <fabian....@gmail.com> wrote:
> On Wednesday, August 3, 2016 at 2:46:10 PM UTC+2, Henrik Johansson wrote:
>>
>> And if you forget to exclude C from X what happens then?
>>
>>
>
> My opinion is: The build should fail if you forget to exclude C from X. This
> forces the user to explicitly resolve all conflicts. No conflict should be
> resolved automatically, and no version should be implicitly set by some
> global parameter.
>
> That's exactly what happens if you use the maven-enforcer-plugin in Maven,
> and in my experience this is a good thing.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Go Package Management" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to go-package-management+unsub...@googlegroups.com.

[Sam Boyer]

This is a rough, human language description of how gps' solver approaches the problem: https://github.com/sdboyer/gps/wiki/gps-for-Contributors#the-solving-algorithm

In considering approaches to this, I think it's important to remember that this run's user choice should become next run's automation. That is, we can't expect users to have to re-make the same conflict resolving choices repeatedly, otherwise repeatability of dependency selection is lost. That effectively loses you repeatable builds (as well as your team's sanity).

> email to go-package-management+unsub...@googlegroups.com.

[Konstantin Shaposhnikov]

Dependency resolution in Maven when version ranges are not used is
fairly straightforward:
- if a project explicitly defines a version in dependencyManagement
section use this version
- otherwise choose the first version that is found by BFS when
traversing dependencies graph

This algorithm never fails to resolve versions but of course can
select wrong versions and result in compile or runtime errors. If this
happens it is always possible (and recommended) to specify versions
explicitly though.

It gets much more complicated with version ranges. The exact behaviour
is not fully documented and there are some edge cases that are not
handled very well (like this
https://issues.apache.org/jira/browse/MNG-3092 ticket open for 9
years!). When using version ranges:
- the build can fail because version ranges of a transient dependency
do not overlap
- maven finds the latest version from the overlapping range
- the build can be non-reproducible as it relies on the versions
available in the remote repository.

Fortunately version ranges are used very rarely. In fact I have never
used version ranges with Maven and don't recall any project that used
them.

I doubt the usefulness of version ranges. The way they are resolved
are often hard to explain and often they lie to users. For example:
- specifying >=1.2.7 when 1.3.0 doesn't even exist. What if 1.3.0 (or
1.2.8) is not backward compatible?
- specifying >= 1.2.7 <= 1.2.12. Has the library been tested with all
5 versions (1.2.7, 1.2.8, ....)? Most of the time not. And when 1.2.13
is out it might still be compatible, so the version range
unnecessarily prevents 1.2.13 from being used.

>> > email to go-package-manag...@googlegroups.com.

>> > To post to this group, send email to go-package...@googlegroups.com.
>> > For more options, visit https://groups.google.com/d/optout.
>

> --
> You received this message because you are subscribed to the Google Groups
> "Go Package Management" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to go-package-manag...@googlegroups.com.

[Sam Boyer]

On Thursday, August 4, 2016 at 4:22:21 AM UTC-4, Konstantin Shaposhnikov wrote:

Dependency resolution in Maven when version ranges are not used is
fairly straightforward:
- if a project explicitly defines a version in dependencyManagement
section use this version
- otherwise choose the first version that is found by BFS when
traversing dependencies graph 

This algorithm never fails to resolve versions but of course can
select wrong versions and result in compile or runtime errors. If this
happens it is always possible (and recommended) to specify versions
explicitly though.

gps would also work just fine without ranges - a tool could choose to disallow them (as indicated in the README). In that case, I think it would have similar solving characteristics to what you're describing here with Maven.

An additional toggle I plan to ship in the next couple months will also allow type-level checking to at least ensure that selected versions of dependencies have API-level compatibility (thanks, go/types!). That's not sufficient to guarantee a correct outcome, of course, but it could get you closer.

 

It gets much more complicated with version ranges. The exact behaviour
is not fully documented and there are some edge cases that are not
handled very well (like this
https://issues.apache.org/jira/browse/MNG-3092 ticket open for 9
years!).

It strikes me as odd that that would be an issue that's open for nine years. npm, for example, addresses this by simply excluding any kind of prerelease from ranges. Dart partially addresses it (and gps currently does the same as Dart) by pushing alpha/beta/rc/unstable/etc. releases down to be less prioritized than any other (semantic) version. I imagine something else about the design must make fixing it difficult.

 

When using version ranges:
- the build can fail because version ranges of a transient dependency
do not overlap

This is a strictly *more* likely occurrence if you don't allow version ranges, not less, as version ranges (inherently) allow for more overlap than an explicitly set version constraint. If everyone leaves a given dep unconstrained and everything happens to work out on that version, then great. When it doesn't work out, explicitly setting a version is one approach - but it's less transitively/ecosystem-friendly than setting a range. (At least, that's the case with how bundler/cargo/npm/gps intersect constraints - idk if Maven has the same mechanism)

 

- maven finds the latest version from the overlapping range 

- the build can be non-reproducible as it relies on the versions
available in the remote repository.

Non-reproducibility here is not a function of version ranges themselves, but an implementation detail of where Maven stores information. I don't think it's a valid criticism of ranges in general.

 

Fortunately version ranges are used very rarely. In fact I have never
used version ranges with Maven and don't recall any project that used
them.

I doubt the usefulness of version ranges. The way they are resolved
are often hard to explain

Again, this is not an argument against ranges per se, but against implementations of them with bad help text.

 

and often they lie to users. For example:
- specifying >=1.2.7 when 1.3.0 doesn't even exist. What if 1.3.0 (or
1.2.8) is not backward compatible?

Versions lie, period. But this is no more of a lie than not providing a constraint and hoping things work out with whatever version gets picked. I'd argue it's less of a lie, because it at least identifies the ballpark of versions that should work. Wouldn't it be a bigger lie to provide no constraint, which is generally equivalent to the range of 0 < v < ∞?

 

- specifying >= 1.2.7 <= 1.2.12. Has the library been tested with all
5 versions (1.2.7, 1.2.8, ....)? Most of the time not. 

And when 1.2.13
is out it might still be compatible, so the version range
unnecessarily prevents 1.2.13 from being used.

These two points have some internal conflict. On the one hand, you dislike the range because it allows versions that maybe haven't been tested. On the other hand, you dislike the range because it disallows versions because they maybe have been tested (or would pass if they were to be tested). In other words, "ranges are bad because they can include bad stuff and could exclude good stuff, so instead we should just..."  what? Always allow everything? Allow everything or only one exact thing? (The answer, btw, to a dep overly constraining another dep is something like https://github.com/sdboyer/gps/wiki/gps-for-Implementors#overrides)

I think I understand the point you're trying to drive at. And in general, I think ranges are things we shouldn't be wanting people to express. But they do serve important purposes at times. 

 

>> > email to go-package-management+unsub...@googlegroups.com.

>> > To post to this group, send email to go-package...@googlegroups.com.
>> > For more options, visit https://groups.google.com/d/optout.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Go Package Management" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to go-package-management+unsub...@googlegroups.com.

[Konstantin Shaposhnikov]

Sam, thank you for your comments. I realize that my exaplanation was
probably not very clear, so let me add a few more examples.

Again I am not suggesting that the Go package manager should work the
same way as Maven but I beleive there are some valuable lessons can be
learnt from the one of the most widely used package manager out there.

On 4 August 2016 at 15:20, Sam Boyer <samuel....@gmail.com> wrote:
>
> On Thursday, August 4, 2016 at 4:22:21 AM UTC-4, Konstantin Shaposhnikov
> wrote:
>>
>> Dependency resolution in Maven when version ranges are not used is
>> fairly straightforward:
>> - if a project explicitly defines a version in dependencyManagement
>> section use this version
>> - otherwise choose the first version that is found by BFS when
>> traversing dependencies graph
>>
>>
>> This algorithm never fails to resolve versions but of course can
>> select wrong versions and result in compile or runtime errors. If this
>> happens it is always possible (and recommended) to specify versions
>> explicitly though.
>
>
> gps would also work just fine without ranges - a tool could choose to
> disallow them (as indicated in the README). In that case, I think it would
> have similar solving characteristics to what you're describing here with
> Maven.

That is good news.

> An additional toggle I plan to ship in the next couple months will also
> allow type-level checking to at least ensure that selected versions of
> dependencies have API-level compatibility (thanks, go/types!). That's not
> sufficient to guarantee a correct outcome, of course, but it could get you
> closer.

The more checks the better as long as they produce easy to understand
error messages.

>>
>>
>> It gets much more complicated with version ranges. The exact behaviour
>> is not fully documented and there are some edge cases that are not
>> handled very well (like this
>> https://issues.apache.org/jira/browse/MNG-3092 ticket open for 9
>> years!).
>
>
>
> It strikes me as odd that that would be an issue that's open for nine years.
> npm, for example, addresses this by simply excluding any kind of prerelease
> from ranges. Dart partially addresses it (and gps currently does the same as
> Dart) by pushing alpha/beta/rc/unstable/etc. releases down to be less
> prioritized than any other (semantic) version. I imagine something else
> about the design must make fixing it difficult.

I guess that might be related to a few things:
- backward compatiblity. Generally maven is fairly stable and having
different versions of maven resolve dependencies differently would
only bring more chaos.
- maven doesn't require SemVer. Instead it uses a heuristic to sort
arbitrary version strings (see
https://github.com/apendragon/Java-Maven-Artifact-Version/wiki/Full-Maven-artifact-versions-comparison-algo-description)
- maven has so called SNAPSHOT versions which are generally a version
built from the latest code on a particular branch (e.g. 2.0-SNAPSHOT
could be from master and 1.2-SNAPSHOT could be from branch-1.2). Some
think that they shold be treated differently from the released
versions, some argue otherwise.

>> When using version ranges:
>> - the build can fail because version ranges of a transient dependency
>> do not overlap
>
>
> This is a strictly *more* likely occurrence if you don't allow version
> ranges, not less, as version ranges (inherently) allow for more overlap than
> an explicitly set version constraint.

I beleive there is a misunderstanding here. Maven treats versions (but
not version ranges!) as recommendations, not as constraints So for
example:

A depends on B and C
B depends on D 1.0
C depends on D 1.1

Maven will use D 1.0 as A's dependency.

If you change an order of dependencies for A (A depends on C and B)
Maven will chose D 1.1 as A's dependency.

If you tell Maven (in dependencyManagement section for A) to always
use D 1.3 then it will chose D 1.3

Also If you declare that A depends on D 1.3 then Maven wil use D 1.3
as A's dependency.

This becomes more complicated with version ranges.

E.g. if in the example above B depends on D <= 1.2 then even if A
depends on D 1.3 Maven will still chose D 1.2. However if
dependencyManagement is used to set D's version to 1.3 it will fail
with a conflict.

I hope this makes sense ;)

From my experience most developers hardly know any of these rules
though they use Maven every day ;) And when it comes to version ranges
I am not so sure either as I have never used them.

> If everyone leaves a given dep
> unconstrained and everything happens to work out on that version, then
> great. When it doesn't work out, explicitly setting a version is one
> approach - but it's less transitively/ecosystem-friendly than setting a
> range. (At least, that's the case with how bundler/cargo/npm/gps intersect
> constraints - idk if Maven has the same mechanism)
>
>>
>> - maven finds the latest version from the overlapping range
>>
>> - the build can be non-reproducible as it relies on the versions
>> available in the remote repository.
>
>
> Non-reproducibility here is not a function of version ranges themselves, but
> an implementation detail of where Maven stores information. I don't think
> it's a valid criticism of ranges in general.

I assume that you are talking about lock files here that Maven doesn't
have. I don't have experience using them so cannot comment.

>>
>> Fortunately version ranges are used very rarely. In fact I have never
>> used version ranges with Maven and don't recall any project that used
>> them.
>>
>> I doubt the usefulness of version ranges. The way they are resolved
>> are often hard to explain
>
>
> Again, this is not an argument against ranges per se, but against
> implementations of them with bad help text.

From my experience there are two type of developers. Some people have
awareness of dependencies and versioning and they usually understand
these concepts quite well. Some people do not care about them and just
want to include a library to use some functionality (such as leftpad
;). For them these rules and versions are like black magic ;)

>> and often they lie to users. For example:
>> - specifying >=1.2.7 when 1.3.0 doesn't even exist. What if 1.3.0 (or
>> 1.2.8) is not backward compatible?
>
>
> Versions lie, period. But this is no more of a lie than not providing a
> constraint and hoping things work out with whatever version gets picked. I'd
> argue it's less of a lie, because it at least identifies the ballpark of
> versions that should work. Wouldn't it be a bigger lie to provide no
> constraint, which is generally equivalent to the range of 0 < v < ∞?

>> - specifying >= 1.2.7 <= 1.2.12. Has the library been tested with all
>> 5 versions (1.2.7, 1.2.8, ....)? Most of the time not.
>>
>> And when 1.2.13
>> is out it might still be compatible, so the version range
>> unnecessarily prevents 1.2.13 from being used.
>
>
> These two points have some internal conflict. On the one hand, you dislike
> the range because it allows versions that maybe haven't been tested. On the
> other hand, you dislike the range because it disallows versions because they
> maybe have been tested (or would pass if they were to be tested).

That is exactly my point - there is always internal conflict with
version ranges.

> In other
> words, "ranges are bad because they can include bad stuff and could exclude
> good stuff, so instead we should just..." what? Always allow everything?
> Allow everything or only one exact thing? (The answer, btw, to a dep overly
> constraining another dep is something like
> https://github.com/sdboyer/gps/wiki/gps-for-Implementors#overrides)

I am not pretending that I know the solution but as I mentioned Maven
works just fine without version ranges most of the time.

How about using version lists instead of version ranges (as an idea to explore)?

So basically be able to say that my library works with the following
versions of A: 1.1, 1.2, 2.0, 2.0.1. And build process can even build
and run tests with different versions of A (though I understand that
this doesn't scale ;).

Anyone can try to use other versions of A in their projects (probably
they can guess compatibility based on SemVer rules) but there are no
guarantees that they will work with my library (because I have never
tested other versions).

[Matt Farina]

On Wednesday, August 3, 2016 at 7:17:05 AM UTC-4, Fabian Stäber wrote:

Hi,

I just came across this discussion, and would like to share my experience as a Java developer:

The main problems we have in many projects are version conflicts with transitive dependencies:

Library A depends on Library C version 1.0
Library B depends on Library C version 1.1

You want to use Library A and Library B.

This is the diamond dependency problem. Most package managers help you solve this problem. A few, when the language environments allow it, will allow different dependencies to have different versions.

 

By default, Maven does not care about these conflicts, it packs both versions of library C into the Classpath, and it is kind of random which one will be used at runtime. This often results in hard-to-find run-time errors.

Therefore, many Java developers use the `maven-enforcer-plugin` which can be configured to make the build break if version conflicts are detected.

Maven supports an `exclude` option to selectively disable transitive dependencies. For example, you could configure Maven to use Library A, but `exclude` the dependency to Library C.

With the `maven-enforcer-plugin` enabled, you are forced to use `excludes` to manually resolve dependency conflicts, otherwise you cannot build the project. This is sometimes a painstaking process, but it is the best solution i'm aware of.

My opinion: I find transitive dependencies useful and would recommend to enable them by default. However, I also find it useful to break the build and force the user to fix it as soon as any conflict occurs.

Pretty much all of the other package managers for other languages handle transitive dependencies. It's a common feature in all the modern package managers. Even Rust already has this solved. Solid handling is going to be a must for Go to be competitive.

 

/Fabian

[atd...@gmail.com]

No. There is no solution that really scales and having this problem in the first place is not a positive thing to have. It involves unwarranted complexity.

A good write up on this: https://blogs.msdn.microsoft.com/eric_brechner/2015/06/30/diamond-dependencies/

Using SAT solvers for this is not a scalable nor very good solution. We shouldn't get into this in the first place. It involves more upfront effort of course.

But for a cleaner, more maintainable overall architecture.

[Konstantin Shaposhnikov]

On 8 August 2016 at 17:08, <atd...@gmail.com> wrote:
> No. There is no solution that really scales and having this problem in the
> first place is not a positive thing to have. It involves unwarranted
> complexity.
>
> A good write up on this:
> https://blogs.msdn.microsoft.com/eric_brechner/2015/06/30/diamond-dependencies/
>
> Using SAT solvers for this is not a scalable nor very good solution. We
> shouldn't get into this in the first place. It involves more upfront effort
> of course.
> But for a cleaner, more maintainable overall architecture.

I agree. Package manager and SemVer is not a substitute for maintaing
backward compatibility. Just as an illustration Hadoop has been trying
to update its Guava (a library that is used by many other Java
libraries) version for 3 years:
https://issues.apache.org/jira/browse/HADOOP-10101

> This is the diamond dependency problem. Most package managers help you solve this problem. A few, when the language environments allow it, will allow different dependencies to have different versions.

I also would like to point out that in some cases the current Go build
system supports dependencies with different versions via nested vendor
directories.

[Dave Cheney]

Yes, this is a critical flaw that requires additional tooling to "flatten" various copies of dependencies into their parent.

>
> --
> You received this message because you are subscribed to the Google Groups "Go Package Management" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to go-package-manag...@googlegroups.com.

[Konstantin Shaposhnikov]

>>
>> I also would like to point out that in some cases the current Go build
>> system supports dependencies with different versions via nested vendor
>> directories.
>
> Yes, this is a critical flaw that requires additional tooling to "flatten" various copies of dependencies into their parent.
>

I don't think it is a flaw though. This can be used to include
incompatible dependencies (that use the same import path) without
import path rewriting. I understand that It is not always possible.

[Dave Cheney]

The lesson that GB took from maven is that dependencies should specify only the things they depend on, not which version. It's up to the project that integrates all that code to decide on which single version of a jar will be compiled against and available on the computed class path.

In go, we get the first part of this sorry for free, the import decl already tells us _what_ is the name of the package this code depends on, all that is missing is _which_ copy (or version if you're being imprecise) should be used.

[Konstantin Shaposhnikov]

On 8 August 2016 at 23:07, Dave Cheney <da...@cheney.net> wrote:
> The lesson that GB took from maven is that dependencies should specify only
> the things they depend on, not which version. It's up to the project that
> integrates all that code to decide on which single version of a jar will be
> compiled against and available on the computed class path.

Maven allows to specify all versions of dependencies (including
trainsient) explicitly. It is not required though.

I agree that normally each included package should have a single
version. However ocasionally package authors decide to break backward
compatiblity while using the same import path (package name in Java).
If a project is "fortunate" enough to need these incompatible versions
(e.g. via transitive dependencies) something needs to be done. Java
developers use maven-shade-plugin. In Go it is possible to use nested
vendor directories or import path rewriting. This situation doesn't
happen often but if it happens I do not see any other solutions.

[Dave Cheney]

Just because it's possible does not in any way make it a good idea and I don't see why tools should continue to enable this.

[Peter Bourgon]

On Tue, Aug 9, 2016 at 12:07 AM, Dave Cheney <da...@cheney.net> wrote:
> The lesson that GB took from maven is that dependencies should specify only
> the things they depend on, not which version.

Just to clarify: are you saying that dependencies should not specify
•a specific• version of what they depend on, or •any• version spec at
all, including e.g. version ranges or major.*?

> It's up to the project that
> integrates all that code to decide on which single version of a jar will be
> compiled against and available on the computed class path.
>
> In go, we get the first part of this sorry for free, the import decl already
> tells us _what_ is the name of the package this code depends on, all that is
> missing is _which_ copy (or version if you're being imprecise) should be
> used.
>
>
> On Tue, 9 Aug 2016, 08:03 Konstantin Shaposhnikov <k.shapo...@gmail.com>
> wrote:
>>
>> >>
>> >> I also would like to point out that in some cases the current Go build
>> >> system supports dependencies with different versions via nested vendor
>> >> directories.
>> >
>> > Yes, this is a critical flaw that requires additional tooling to
>> > "flatten" various copies of dependencies into their parent.
>> >
>>
>> I don't think it is a flaw though. This can be used to include
>> incompatible dependencies (that use the same import path) without
>> import path rewriting. I understand that It is not always possible.
>

[Sam Boyer]

On Monday, August 8, 2016 at 12:08:08 PM UTC-4, atd...@gmail.com wrote:

No.

I think it would be helpful for dialogue if you refrained from these kind of absolutist statements. e.g., "No." -> "No, I don't think solvers are a good idea." You're not convincing anyone by asserting harder.

 

There is no solution that really scales and having this problem in the first place is not a positive thing to have. It involves unwarranted complexity.

A good write up on this: https://blogs.msdn.microsoft.com/eric_brechner/2015/06/30/diamond-dependencies/

That writeup is optimistic, but its suggestions about the costs of diamond dep resolution are based on some assumptions that don't necessarily hold in our case (version ranges can mitigate some of the problem, and that diamond dep support somehow entails huge source repos).

In general, I think you're conflating what's desirable - appropriately-sized packages, backwards compatibility, branching by abstraction - with what's allowable in the system. Also, crucially, that blog article is discussing the management of dependencies within an organization, where it is (more) possible to impose requirements like that. While you can encourage them in a community, enforcing them is difficult, and the Go ecosystem in particular has no readily available means to do it.

It's worth noting that many npm folks now regret the "always isolate dep versions" approach, and they're moving *more* towards flattening over time.

 

Using SAT solvers for this is not a scalable nor very good solution. We shouldn't get into this in the first place. It involves more upfront effort of course.

But for a cleaner, more maintainable overall architecture.

Sorry, but this seems to me an assertion without evidence.

[Sam Boyer]

It's perfectly possible - and yes, I think desirable - to allow dependencies to optionally indicate a version constraint on their own deps, and use that information to assist in narrowing the search space. As long as the root project retains full control (i.e., can override those constraints if need be), then that doesn't strike me as universally harmful. It might be suboptimal for some use cases, but I haven't yet seen an alternative proposal that doesn't optimize some use cases at the expense of others.

[Dave Cheney]

The way it worked in maven world is you would have a parent pom (your
project), child poms (if you built your project out of various
libraries), and the poms they depended on would specify the artefact
name and group id (basically a fully qualified name) but _not_ include
the version field. That field was provided by the parent pom by
specifying all three. In effect the project has to specify the fully
qualified name _and_ the version of every transitive dependency.

Which is exactly what gb does, but without the xml and brass band.

[atd...@gmail.com]

I stand by my statement that I disagree with the claims that were made. Feel free to disagree too without having to dictate how I express myself, thank you.

It's funny that you disagree because all the evidence you need is the Go tree.

SAT solvers are not needed. I could give the example of other ecosystems (canonical example being cabal).

If one of your attempted solution to the problem was to introduce SAT solving, indeed I understand that my wording may have been a bit hurtful and for this I apologize.

[atd...@gmail.com]

And by mentioning Cabal, I mean that they have introduced a SAT solver typically because of the lack of backward compatibility and 8 years in, in 2016, they are still not done trying to find a correct answer to the issue of dependency management...

[John Souvestre]

Hi Dave.

 

>> ... dependencies should specify only the things they depend on, not which version. It's up to the project that integrates all that code to decide on which single version of a jar will be compiled against and available on the computed class path.

 

OK on a single version per project, but how do you propose that the project manager tool select (or validate) the version to use (or being used)?

 

For example:  What if my project imports package A, but I need an older version (1.x) because a recent API change (2.x) of A isn't compatible with my project's code.  I would expect the project manager tool to verify that the version selected is "legal" per this constraint.  So where is the constraint stated by package A’s author?

 

John

    John Souvestre - New Orleans LA

--

[Dave Cheney]

In the maven world it is done by hand, by copying and pasting the xml stanza into your project pom.

In the gb world it is done by putting the source of the package in your $PROJECT/vendor/src folder.

[Sam Boyer]

On Monday, August 8, 2016 at 7:30:41 PM UTC-4, atd...@gmail.com wrote:

I stand by my statement that I disagree with the claims that were made. Feel free to disagree too without having to dictate how I express myself, thank you.

All I did was suggest.

 

It's funny that you disagree because all the evidence you need is the Go tree.

The Go tree, as in stdlib?

The organization that creates and manages a system matters, for the reasons I gave in my last response.. stdlib is not comparable to a 

 

SAT solvers are not needed. I could give the example of other ecosystems (canonical example being cabal).

If one of your attempted solution to the problem was to introduce SAT solving, indeed I understand that my wording may have been a bit hurtful and for this I apologize.

I appreciate that (though I really did just mean to suggest that your arguments might be better received). 

Yes, the work I've done is rooted in a solver - https://github.com/sdboyer/gps - but not full-blown SAT. To that end, is your issue with applying full-on SAT (which I tend to agree is excessive), or with any sort of constraint-solving approach at all?

[Sam Boyer]

err, forgot to finish the middle bit: stdlib isn't comparable, because the organization (the Go team) that produces it isn't comparable to the ecosystem. It's not just a question of size, but knowability - it's an ecosystem full of people operating without knowledge of, or any influence over, one another. This might not change the essential problem being solved, but I think it does change the expectations we can place, and the tools we need to give people to function comfortably in it.

[John Souvestre]

Hi Dave.

 

>> So where is the constraint stated by package A's author?

 

> In the gb world it is done by putting the source of the package in your $PROJECT/vendor/src folder.

 

I don’t understand.  The author of package A doesn't know anything about my project.  Where do you propose that he state the constraint (version number) for his package?

 

John

    John Souvestre - New Orleans LA

 

[Dave Cheney]

Two things

- go packages have no notion of version because we have no release process, so there no way for a library author to tell a downstream consumer which version they should use.
- gb doesn't have a solution to this at the moment either, because of the first point.

All you can do is get some source, and put it in your vendor folder and hope for the best.

[Dave Cheney]

On Tue, 9 Aug 2016, 09:56 Dave Cheney <da...@cheney.net> wrote:

Two things

- go packages have no notion of version because we have no release process, so there no way for a library author to tell a downstream consumer which version they should use.

Please understand, nobody is more tired of hearing this little stump speech than me. 

[John Souvestre]

Hi Dave.

 

> Please understand, nobody is more tired of hearing this little stump speech than me.

 

Sorry, but I don't think that you addressed the question I asked.  So forgive me if I try again.

 

> - go packages have no notion of version because we have no release process, so there no way for a library author to tell a downstream consumer which version they should use.

 

I'm not asking how a library author would tell a consumer which version to use.  I'm asking how the author would say what version his library is.

 

> All you can do is get some source, and put it in your vendor folder and hope for the best.

 

I believe that we can do better in the future.  :)

[Dave Cheney]

If I understand the question you asked, that is proposal/12302.

If I didn't answer you question, could you please ask it in a different way.

Thanks

[John Souvestre]

Hi Dave.

 

> If I understand the question you asked, that is proposal/12302.

 

Yes, thanks.  I see:

 

>> Source code is released by tagging (eg. git tag) the VCS repository with a string representing a SemVer compatible version number for that release.

 

OK.

 

> If I didn't answer you[r] question, could you please ask it in a different way.

 

You did.  But I have a follow-up.  :)

 

>> The lesson that GB took from maven is that dependencies should specify only the things they depend on, not which version.

 

Do I understand correctly that you feel that my project code should not include a constraint that only version 1.x of package A (from my previous example) is allowed?  No automatic alert should someone should later replace (in my project) package A version 1.x with 2.x?

[Dave Cheney]

On Tue, 9 Aug 2016, 10:55 John Souvestre <jo...@souvestre.com> wrote:

Hi Dave.

 

> If I understand the question you asked, that is proposal/12302.

 

Yes, thanks.  I see:

 

>> Source code is released by tagging (eg. git tag) the VCS repository with a string representing a SemVer compatible version number for that release.

 

OK.

 

> If I didn't answer you[r] question, could you please ask it in a different way.

 

You did.  But I have a follow-up.  :)

 

>> The lesson that GB took from maven is that dependencies should specify only the things they depend on, not which version.

 

Do I understand correctly that you feel that my project code should not include a constraint that only version 1.x of package A (from my previous example) is allowed?  No automatic alert should someone should later replace (in my project) package A version 1.x with 2.x?

No. I don't believe I have made that statement. If I have inadvertently said that I retact it with my apologies. 

[John Souvestre]

Hi Dave.

 

Sorry for my misinterpretation.  I'm having some trouble fully understanding what you said.

 

If code in my project would not specify the dependency (version 1.x of package A), then how would my project's package manager be aware of it?

 

John

    John Souvestre - New Orleans LA

 

From: Dave Cheney [mailto:da...@cheney.net]
Sent: 2016 August 08, Mon 20:17
To: John Souvestre; Go Package Management
Subject: Re: [go-pm] Re: Transitive dependencies

 

 

On Tue, 9 Aug 2016, 10:55 John Souvestre <jo...@souvestre.com> wrote:

Hi Dave.

 

> If I understand the question you asked, that is proposal/12302.

 

Yes, thanks.  I see:

 

>> Source code is released by tagging (eg. git tag) the VCS repository with a string representing a SemVer compatible version number for that release.

 

OK.

 

> If I didn't answer you[r] question, could you please ask it in a different way.

 

You did.  But I have a follow-up.  :)

 

>> The lesson that GB took from maven is that dependencies should specify only the things they depend on, not which version.

 

Do I understand correctly that you feel that my project code should not include a constraint that only version 1.x of package A (from my previous example) is allowed?  No automatic alert should someone should later replace (in my project) package A version 1.x with 2.x?

[Dave Cheney]

On Tue, Aug 9, 2016 at 11:38 AM, John Souvestre <jo...@souvestre.com> wrote:
> Hi Dave.
>
>
>

> Sorry for my misinterpretation. I'm having some trouble fully understanding
> what you said.
>
>
>
> If code in my project would not specify the dependency (version 1.x of
> package A), then how would my project's package manager be aware of it?
>

I guess it would parse the source of the .go files for import declarations.

>
>
> John
>
> John Souvestre - New Orleans LA
>
>
>
> From: Dave Cheney [mailto:da...@cheney.net]
> Sent: 2016 August 08, Mon 20:17
> To: John Souvestre; Go Package Management
> Subject: Re: [go-pm] Re: Transitive dependencies
>
>
>
>
>
> On Tue, 9 Aug 2016, 10:55 John Souvestre <jo...@souvestre.com> wrote:
>
> Hi Dave.
>
>
>
>> If I understand the question you asked, that is proposal/12302.
>
>
>
> Yes, thanks. I see:
>
>
>
>>> Source code is released by tagging (eg. git tag) the VCS repository with
>>> a string representing a SemVer compatible version number for that release.
>
>
>
> OK.
>
>
>
>> If I didn't answer you[r] question, could you please ask it in a different
>> way.
>
>
>
> You did. But I have a follow-up. :)
>
>
>
>>> The lesson that GB took from maven is that dependencies should specify
>>> only the things they depend on, not which version.
>
>
>
> Do I understand correctly that you feel that my project code should not
> include a constraint that only version 1.x of package A (from my previous
> example) is allowed? No automatic alert should someone should later replace
> (in my project) package A version 1.x with 2.x?
>
>
>
> No. I don't believe I have made that statement. If I have inadvertently said
> that I retact it with my apologies.
>

[John Souvestre]

Hi Dave.

>> If code in my project would not specify the dependency (version 1.x of
>> package A), then how would my project's package manager be aware of it?

> I guess it would parse the source of the .go files for import declarations.

Parse the project's (not package A's) .go files? OK, but the import declaration doesn't currently include a version dependency. So I still have the same question.

John

John Souvestre - New Orleans LA

[Dave Cheney]

Respectfully I think this conversation is going in circles and there is nothing more I can usefully add. Thank you for your understanding.

[John Souvestre]

Thank you for your time.

 

John

    John Souvestre - New Orleans LA

 

From: Dave Cheney [mailto:da...@cheney.net]
Sent: 2016 August 08, Mon 22:25
To: John Souvestre; Go Package Management
Subject: Re: [go-pm] Re: Transitive dependencies

 

Respectfully I think this conversation is going in circles and there is nothing more I can usefully add. Thank you for your understanding.

 

[Konstantin Shaposhnikov]

The way it worked in maven world is you would have a parent pom (your 
project), child poms (if you built your project out of various 
libraries), and the poms they depended on would specify the artefact 
name and group id (basically a fully qualified name) but _not_ include 
the version field. That field was provided by the parent pom by 
specifying all three. In effect the project has to specify the fully 
qualified name _and_ the version of every transitive dependency. 

Which is exactly what gb does, but without the xml and brass band. 

Dave, I am sorry if I am stating obvious things that you already know but I feel that I need to provide a clarification.

gb does require indeed that versions must be specified for all dependencies including transitive. This is done by placing the source code of these dependencies into src/vendor.

Maven however does not require to specify versions for all dependencies. It is possible (and sometimes encouraged) but not required. Versions of transitive dependencies are resolved based on poms of direct dependencies. The first version of Maven did not support transitive dependencies but this was 10 years ago.

 

 

[Dave Cheney]

On Tue, Aug 9, 2016 at 6:48 PM, Konstantin Shaposhnikov
<k.shapo...@gmail.com> wrote:
>
>> The way it worked in maven world is you would have a parent pom (your
>> project), child poms (if you built your project out of various
>> libraries), and the poms they depended on would specify the artefact
>> name and group id (basically a fully qualified name) but _not_ include
>> the version field. That field was provided by the parent pom by
>> specifying all three. In effect the project has to specify the fully
>> qualified name _and_ the version of every transitive dependency.
>>
>> Which is exactly what gb does, but without the xml and brass band.
>
>
> Dave, I am sorry if I am stating obvious things that you already know but I
> feel that I need to provide a clarification.
>
> gb does require indeed that versions must be specified for all dependencies
> including transitive. This is done by placing the source code of these
> dependencies into src/vendor.

False. Source code does not have versions, or even revisions, those
are properties of source code control systems.

>
> Maven however does not require to specify versions for all dependencies. It
> is possible (and sometimes encouraged) but not required. Versions of
> transitive dependencies are resolved based on poms of direct dependencies.
> The first version of Maven did not support transitive dependencies but this
> was 10 years ago.
>
>
>

[Konstantin Shaposhnikov]

>>
>> gb does require indeed that versions must be specified for all dependencies
>> including transitive. This is done by placing the source code of these
>> dependencies into src/vendor.
>
> False. Source code does not have versions, or even revisions, those
> are properties of source code control systems.

I don't think we disagree here. Sorry for being imprecise. Comparing a
build tool (gb) to a package manager that can also act as a build tool
(Maven) could be confusing ;)

[Dave Cheney]

Yes, I am sorry, that was unnecessarily blunt.

I'm uninterested in arguing about the taxonomy of which tool is a
package manage and which tool is a build tool, I don't think is
useful. gb takes some ideas from maven, some ideas from cargo, some
ideas from the go tool, and so on. It's fine to be inspired by another
tool even if it's not a direct replacement.