dependency conflicts

[Günter Zöchbauer]

I'm struggling a lot recently with conflicting dependency constraints.

There is also a discussion going on at http://dartbug.com/22054 about bin_dependencies

See also https://plus.google.com/+AndersHolmgren1/posts/5QQzfcfve2R

My situation is similar - Polymer on the client, Shelf and AppEngine on the server, Grinder in tools.

If I add a new dependency in dev_dependencies, the added dependency and transitive dependencies need still to be compatible with the existing normal dependencies.

When I for example add Grinder to automate some tasks and I need a set of dependencies which help to build the Grinder tasks, these dependencies still need to be compatible with the normal dependencies, right?

I think this unnecessarily complicates things.

If the Dart scripts in prj/tools don't import anything from prj/lib as it is usually the case with Grinder tasks and probably many others applicable for tools, then there is no reason to enforce such a dependency. 

For Grinder I add for example linter, dart_style, analyzer, ... which can operate on any code and shouldn't limit me in what dependencies (or versions of deps) I can use for code in prj/lib, prj/web, or prj/bin.

More subtle are dependencies in packages which contain client and server.

When some dependencies are only used on the client and others only on the server, they still need to be compatible.

One might argue to just split the package in three (shared, client, server) but is this really the intention?

It still doesn't solve the prj/tools case.

Maybe there are improvements possible with the new https://github.com/lrhn/dep-pkgspec

[Natalie Weizenbaum]

Sure, we could theoretically have entirely separate dependency trees for every top-level library, every tooling script, every piece of an application that could potentially not import any other piece of the application. But this would mean that the process of selecting and managing your dependencies gets correspondingly more and more complicated, and it opens a lot of room for mistakes. What happens if two previously-separate parts of your code start talking to one another? The process of ensuring that they use the same versions of everything is going to be difficult, and it will be next to impossible to verify that you got every case. And if you don't do it right, you're setting yourself up for subtle and confusing failures down the line.

An important thing to understand here is that imports aren't the only way that two pieces of code might need to communicate. A tool may write a file that a library needs to read, or it might start a server that a library connects to. Ensuring that everything has the same (or compatible) versions eliminates a vast class of potential bugs here. I have first-hand experience: when we were righting pub's barback integration, we didn't have that guarantee, and it was hugely painful.

We made the decision early on when designing pub's dependency model to keep it straightforward and avoid a proliferation of different dependencies for different contexts. In retrospect it would have been nice to have a notion of nested packages, but even then they likely would have been additive on the root dependency graph. Although I understand that it causes pain in some circumstances, I'm confident that having a single canonical version of each dependency is silently alleviating much more pain.

--
For other discussions, see https://groups.google.com/a/dartlang.org/
 
For HOWTO questions, visit http://stackoverflow.com/tags/dart
 
To file a bug report or feature request, go to http://www.dartbug.com/new

To unsubscribe from this group and stop receiving emails from it, send an email to misc+uns...@dartlang.org.

[Bob Nystrom]

On Mon, Mar 30, 2015 at 2:16 AM, Günter Zöchbauer <gzo...@gmail.com> wrote:

I'm struggling a lot recently with conflicting dependency constraints.

I think the key question here is are you struggling because pub's dependency system is too coarse-grained, or are you struggling because some of the packages you use aren't being good versioning citizens?

Even if we add all of the features in the world to pub, a few poorly managed packages can make any constraint solver go off the deep end. In all of the examples I've seen where pub is taking forever to find a solution, it's come down to a package or two that hadn't been scrupulous about its versioning.

I'd much rather either:

• Get those package maintainers to be a little more disciplined. In the case of the analyzer package, which was where we saw almost all of the pain come from, they have gotten a lot better. Of course, all of the old versions where they were less careful are still on pub and may still trip up the solver. :(
  
  
• Tweak pub's heuristics to handle cases like this better.
  

If we jump to "add features to pub to let users slice and dice their dependencies", we are throwing complexity on to every user of pub. Anyone who consumes packages will have to think about what flavor of dependencies they have and if they don't, they're right back to the solver going bad on them. If we do this, we've added complexity to their life, and we will never get that simplicity back.

 In other words, if pub did eventually find a set of versions given the current semantics and your app worked fine with those, then your dependencies are already expressive enough. You have expressed what you need and pub gave it to you. We just need to work on it giving them to you more efficiently.

You shouldn't have to change how you express your dependencies to make pub's life better. If you want to express your dependencies differently, it should be to make your life better.

- bob

[Günter Zöchbauer]

Thanks for your comprehensive responses @Nathan, @Bob.

All your arguments are reasonable and you are mostly right.

Nevertheless this causes me lots of troubles and because there are ongoing discussions about related topics (pkgspec, bin_dependencies) I think it's a good time to reconsider the current strategy before any concrete actions are taken in the related topics.

The argument with packages not being good citizens might seem to apply but when I take a closer look almost every maintainer is busy fixing and improving things. 

In practice there are always several packages which are just not there yet.

From finding out there is an issue or conflict to getting to "problem fixed" it usually takes a few days to a few weeks and sometimes months, depending on the issues. When just the dependency constraint was too narrow, this is fixed easily, but sometimes more refactoring is necessary to make a package work with a new version of a dependency and this might take weeks or months, depending on the time constraints of the maintainer.

An example (worst case): I made a PR for such a dependency recently and then found out that someone else already made the same PR a month ago but the maintainer seems not to have time or interest to maintain his package anymore. Sure I can fork it, but this is not the first natural reaction. Until you come to the conclusion you have to fork and maintain it yourself, weeks or months go by.

During the weeks your are busy making the dependencies work together, others are busy adding new features and fixing bugs and you might want to use them in your application because otherwise you have to code them yourself, which would be a waste of time of course.

To be able to use this new feature you start again your journey trying to get your dependencies work with the new version of this cool package.

This is a boring story and everybody knows it by heart. So if you want to be a developer just cope with it and accept that this job doesn't have only bright sides, right?

I'm willing to accept that, except when I see that the dependencies I work my ass off to make them work together are forced onto me because of limitations of the tools I'm using.

I'm aware that this is a complicated topic and there are trade-offs to be considered in each direction.

I read a bit about how dependencies are managed in Ruby because I saw it mentioned several times and from what I learned, I think we are already far better off with pub.

In my experience every additional dependency increases the problem in an exponential way and its worst when the dependencies are added for another part of the application.

If you build a server application and add a new dependency to use it on the server part, this causes usually less troubles than in other parts of the application like for example the web client. I guess this is because the maintainer of the dependency already put some effort into making related dependencies work together because they are already used in this package anyway.

I think the worst situation is, if some development workflow tools which work on any Dart code/package and are totally unrelated to my specific application implementation, enforce some dependency constraints on my application. This just doesn't make any sense.

I can imagine that there are situations where even here some dependencies need to be compatible but I'm pretty sure this is a rare exception and can still be managed by adding a version constraint for both parts (for example bin and lib, like in the example mentioned by Bob about mesh3d), to ensure compatible dependencies are used. And this requires only specific dependencies to be compatible, not the entire set of the entire application.

In my opinion the problem with tools is much worse than the discussed bin_dependencies. Of course tools are only related to a package/applications while bin_dependencies might also be inherited to packages (not sure I fully understand yet why) which depend on the package.

But packages are usually very specific and don't introduce as many dependencies as an application, where everything comes together.

And while it is possible to to split a package in shared, server, client, examples, and even tests, this seems much more cumbersome for tools, especially when I think about using them for Travis, Drone, or other CI tools.

Even though I singled out tools as worst case, I think that none of the parts (top-level directories) should force their specific constraints on the other parts. 

I think each top-level folder should be handled like a sub-project. It could be helpful to have some support from pub to configure versions constraints for a set of these sub-projects, but shared constraints are inherited from lib anyway because this will usually be imported into test, example, bin and web.

[Bob Nystrom]

On Tue, Mar 31, 2015 at 12:15 AM, Günter Zöchbauer <gzo...@gmail.com> wrote:

Thanks for your comprehensive responses @Nathan, @Bob.

All your arguments are reasonable and you are mostly right.

Nevertheless this causes me lots of troubles and because there are ongoing discussions about related topics (pkgspec, bin_dependencies) I think it's a good time to reconsider the current strategy before any concrete actions are taken in the related topics.

The argument with packages not being good citizens might seem to apply but when I take a closer look almost every maintainer is busy fixing and improving things. 

In practice there are always several packages which are just not there yet.

This is fine. There's always a froth of unstable packages and they don't generally cause problems. Where you run into trouble is when you have a package that is:

1. Changing quickly.

2. Depended on by multiple other packages.

3. That are in turn often used in concert.

You generally have to hit all three of these before things get bad. For example, last summer, analyzer was revving breaking changes daily. It's also depended on by polymer, angular, code_transformers and several other important middleware-ish packages. And, naturally, people use the latter together in web apps.

But if your app is using new_hotness which is still unstable and changing all the time, that's not really a problem. It's when that gets deeply embedded at the bottom of your dependency graph where it gets over constrained.

 

From finding out there is an issue or conflict to getting to "problem fixed" it usually takes a few days to a few weeks and sometimes months, depending on the issues. When just the dependency constraint was too narrow, this is fixed easily, but sometimes more refactoring is necessary to make a package work with a new version of a dependency and this might take weeks or months, depending on the time constraints of the maintainer.

This is the kind of scenario that dependency_overrides were added for. If you just need to get stuff done now, jam an override in and make pub shut up. Just make sure to test carefully.

 

I can imagine that there are situations where even here some dependencies need to be compatible but I'm pretty sure this is a rare exception and can still be managed by adding a version constraint for both parts (for example bin and lib, like in the example mentioned by Bob about mesh3d), to ensure compatible dependencies are used.

Right. For what it's worth, I'm not strongly opposed to bin_dependencies. I think they may be worth adding, we just need to be careful.

 

In my opinion the problem with tools is much worse than the discussed bin_dependencies.

I don't know much about Grinder yet, but in general, for things in tool/, you can just use dev_dependencies.

 

But packages are usually very specific and don't introduce as many dependencies as an application, where everything comes together.

And while it is possible to to split a package in shared, server, client, examples, and even tests, this seems much more cumbersome for tools, especially when I think about using them for Travis, Drone, or other CI tools.

Yeah, I agree this is painful. It may be that we should spend more time thinking about nested packages as a general solution for this. I don't want to keep adding new one-off dependencies flavors to the pubspec to patch around the problem.

 

Even though I singled out tools as worst case, I think that none of the parts (top-level directories) should force their specific constraints on the other parts. 

Yeah, something like nested packages would effectively do this.

Cheers!

- bob

[Alex Tatumizer]

@Bob: you remember we discussed similar issue briefly in another thread? I suggested that every configuration file is actually a program, and it's just a matter of time for it to start acquiring more and more traits of Turing machine? You argued against this thesis. But look: all those dev_dependencies, bin_dependencies are just hidden "IFs".
Had it been a dart script instead of yaml file, it could implement any logic and return a map to be used unconditionally. It's impossible to anticipate all factors affecting this map. Many of them are too ad hoc to be standardized.

The idea is not that new: Emacs configuration IS a (Lisp) script. 

[Bob Nystrom]

On Tue, Mar 31, 2015 at 11:57 AM, Alex Tatumizer <tatu...@gmail.com> wrote:

@Bob: you remember we discussed similar issue briefly in another thread? I suggested that every configuration file is actually a program, and it's just a matter of time for it to start acquiring more and more traits of Turing machine? You argued against this thesis. But look: all those dev_dependencies, bin_dependencies are just hidden "IFs".

Sure, but there's nothing new about targeting a non-Turing-complete language for your configuration. Doing that guarantees you can "execute" the script in constant time and without side effects. That's important for tools like the IDE that need to understand this configuration.

 

The idea is not that new: Emacs configuration IS a (Lisp) script. 

Yes, and Sublime Text configuration is JSON. IntelliJ and Visual Studio use XML.

We did consider making pubspecs just be .dart files (Gemfiles in Bundler are Ruby scripts after all), but we considered easy static analyzability to be more important than end user expressiveness, especially given that we didn't have concrete use cases for what they would want to express.

Cheers!

- bob

[Günter Zöchbauer]

On Tuesday, March 31, 2015 at 6:19:09 PM UTC+2, Bob wrote:

On Tue, Mar 31, 2015 at 12:15 AM, Günter Zöchbauer <gzo...@gmail.com> wrote:

Thanks for your comprehensive responses @Nathan, @Bob.

All your arguments are reasonable and you are mostly right.

Nevertheless this causes me lots of troubles and because there are ongoing discussions about related topics (pkgspec, bin_dependencies) I think it's a good time to reconsider the current strategy before any concrete actions are taken in the related topics.

The argument with packages not being good citizens might seem to apply but when I take a closer look almost every maintainer is busy fixing and improving things. 

In practice there are always several packages which are just not there yet.

This is fine. There's always a froth of unstable packages and they don't generally cause problems. Where you run into trouble is when you have a package that is:

1. Changing quickly.

2. Depended on by multiple other packages.

3. That are in turn often used in concert.

You generally have to hit all three of these before things get bad. For example, last summer, analyzer was revving breaking changes daily. It's also depended on by polymer, angular, code_transformers and several other important middleware-ish packages. And, naturally, people use the latter together in web apps.

But if your app is using new_hotness which is still unstable and changing all the time, that's not really a problem. It's when that gets deeply embedded at the bottom of your dependency graph where it gets over constrained.

 

From finding out there is an issue or conflict to getting to "problem fixed" it usually takes a few days to a few weeks and sometimes months, depending on the issues. When just the dependency constraint was too narrow, this is fixed easily, but sometimes more refactoring is necessary to make a package work with a new version of a dependency and this might take weeks or months, depending on the time constraints of the maintainer.

This is the kind of scenario that dependency_overrides were added for. If you just need to get stuff done now, jam an override in and make pub shut up. Just make sure to test carefully.

 

dependency_overrides is very limited. A package containing dependency_overrides can't be published

even when dependency_overrides is only used for a dev_dependencies even though dev_dependencies don't affect package users.

dependency_overrides also forces this constrain on normal dependencies even when it's only necessary for dev_dependencies.

 

I can imagine that there are situations where even here some dependencies need to be compatible but I'm pretty sure this is a rare exception and can still be managed by adding a version constraint for both parts (for example bin and lib, like in the example mentioned by Bob about mesh3d), to ensure compatible dependencies are used.

Right. For what it's worth, I'm not strongly opposed to bin_dependencies. I think they may be worth adding, we just need to be careful.

 

I think treating top-level folders like sub-packages would make dev_dependencies and bin_dependencies redundant  

and also tool_depenedencies, example_dependencies, .... 

 

In my opinion the problem with tools is much worse than the discussed bin_dependencies.

I don't know much about Grinder yet, but in general, for things in tool/, you can just use dev_dependencies.

 

But dev_dependencies need to be compatible with normal dependencies. 

When I want to use the linter package for grind tasks it requires analyzer 0.25.0-dev.x.

then I have to add

dev_dependencies: 

  analyzer: '>=0.25.0-dev.0 <0.26.0'

I have another dependency or just a dev_dependency which requires analyzer '>=0.23.0 <0.25.0'

When I run pub upgrade I get a dependency conflict

Package analyzer 0.25.0-dev.1 does not match >=0.25.0-dev.1 <0.25.0 derived from:

- linter 0.0.2 depends on version >=0.25.0-dev.1

- shelf_exception_handler 0.1.0 depends on version >=0.24.0 <0.26.0

- test 0.12.0-beta.4 depends on version >=0.23.0 <0.25.0

Then I also need 

dependency_overrides:   

  analyzer: '>=0.25.0-dev.0 <0.26.0'

Which results my lib code to work with an unstable dependency just because I want to use linter in tools

and I also am not allowed to publish the package anymore.

This is just a current example. I had similar issues recently with more stable packages as well.

But packages are usually very specific and don't introduce as many dependencies as an application, where everything comes together.

And while it is possible to to split a package in shared, server, client, examples, and even tests, this seems much more cumbersome for tools, especially when I think about using them for Travis, Drone, or other CI tools.

Yeah, I agree this is painful. It may be that we should spend more time thinking about nested packages as a general solution for this. I don't want to keep adding new one-off dependencies flavors to the pubspec to patch around the problem.

 

Even though I singled out tools as worst case, I think that none of the parts (top-level directories) should force their specific constraints on the other parts. 

Yeah, something like nested packages would effectively do this.

I think it's really important to avoid dependencies when it's not absolutely necessary.

Dependencies added only for test/tools/example/bin/web shouldn't pass additional constraints on lib or on each other, but they do, even when they are added as dev_dependencies.

Cheers!

- bob

[Günter Zöchbauer]

How about creating real subprojects in top-level folders like 'tool' or 'example'?

Would this cause troubles, for example when publishing to pub.dartlang.org? 

For 'test' and 'bin' this would need special tools support but at least for 'tool' and 'example' this could work.

[Sean Eagan]

I filed a bug for something like nested packages a while back:

http://dartbug.com/21920

I hadn't thought of the use case of having separate version solving / dependency graphs in a nested package.

--