Go Packaging: building a great packaging story

[Nathan Youngman]

A few fundamentals to building a great packaging story.

http://nathany.com/go-packages/

[Kyle Lemons]

"It would be great if the Go Authors specified and built a full-featured package manager for Go. It's not a requirement though."

On Fri, Jul 26, 2013 at 6:42 AM, Nathan Youngman <junk...@nathany.com> wrote:

A few fundamentals to building a great packaging story.

http://nathany.com/go-packages/

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Kyle Lemons]

whoops, the formatting button is way too close to the send button.

It would be great if the Go Authors specified and built a full-featured package manager for Go. It's not a requirement though.

The Go Authors is a really expansive list, I think you mean the Go Team :).  I have to say though that the Go Authors are reasonably likely to be the ones who make such a thing, because they're some of the most active members of the community.  In a way, anything the community makes is made by the Go Authors. 

[Lasana Murray]

"gopack" who is with me?

[Philippe Lafoucrière]

Go is really nice and fun, and I like playing with it. Anyway, it won't be enterprise-ready (and yes, I know google is using it in production) until this package manager is available...

This is really a blocking issue for us. "Go get" is fine for small programs, but bigger projects require reproducible builds.

This will lead soon to zombies projects on github, because the features scope will change, and it will break future build using these projects.

My 2 cents, but I really liked Mitchell Hashimoto's idea about gophervault. Maybe a proxy around "go get" to wrap URL with gophervault would do the trick, without modifying current source base.

[Jan Mercl]

On Sun, Jul 28, 2013 at 12:48 PM, Philippe Lafoucrière
<philippe.l...@tech-angels.com> wrote:
> Go is really nice and fun, and I like playing with it. Anyway, it won't be
> enterprise-ready (and yes, I know google is using it in production) until
> this package manager is available...
> This is really a blocking issue for us. "Go get" is fine for small programs,
> but bigger projects require reproducible builds.
> This will lead soon to zombies projects on github, because the features
> scope will change, and it will break future build using these projects.

I guestimate a utility collecting revision numbers of all
dependent packages into some "repro-xyz" text file - or checkouts
revisions from that list to make a "reproducible build" possible
later; would cost like few hours to write, test and document.

About every essential part for it is in the Go repository already,
ready to consume. Is there some magic in this I've missed?

-j

[Dan Kortschak]

Or even just use one already written, e.g. http://kylelemons.net/blog/2012/04/22-rx-for-go-headaches.article

[Philippe Lafoucrière]

Yes, I guess the point I'd not to rewrite one, but more to make one
official / part of core.

Sent from my iPhone

[andrey mirtchovski]

> Yes, I guess the point I'd not to rewrite one, but more to make one
> official / part of core.

i have a kitchen sink i desire to make part of "core". one size fits all.

[Philippe Lafoucrière]

On Sun, Jul 28, 2013 at 1:14 PM, andrey mirtchovski
<mirtc...@gmail.com> wrote:
> i have a kitchen sink i desire to make part of "core". one size fits all.

Is it an official sink ?

[Dan Kortschak]

On Sun, 2013-07-28 at 05:14 -0600, andrey mirtchovski wrote:
> i have a kitchen sink i desire to make part of "core". one size fits
> all.

Is that a generic kitchen sink?

[andrey mirtchovski]

> Is that a generic kitchen sink?

but of course! i hear they have top men working on it right now ¹ :)

--
1: http://www.youtube.com/watch?v=yoy4_h7Pb3M

[Philippe Lafoucrière]

:)


--
Philippe Lafoucrière - CEO
http://www.tech-angels.com
main : +33 (0) 970 444 643
mobile : +33 (0) 6 72 63 75 40
fax : +33 (0) 9 72 12 78 75

[Philippe Lafoucrière]

Actually: https://github.com/golang/groupcache/blob/master/sinks.go ;)

--
Philippe Lafoucrière - CEO
http://www.tech-angels.com
main : +33 (0) 970 444 643
mobile : +33 (0) 6 72 63 75 40
fax : +33 (0) 9 72 12 78 75

[John Waycott]

On Friday, July 26, 2013 6:42:07 AM UTC-7, Nathan Youngman wrote:

A few fundamentals to building a great packaging story.

http://nathany.com/go-packages/

I really like the idea of a consistent import path that Nathan mentions in his article. This is especially important in multi-year projects where source control systems come and go, source code rights get transferred, etc. I've never liked seeing author/company names as part of the import path for this reason. The maintenance of source code should be separate from the maintenance of its location.

[Gerard]

Thinking out loud.

Debian also has .dsc files, which are source files instead of binaries in the .deb files. See http://www.debian.org/doc/manuals/debian-faq/ch-pkg_basics

Maybe something like this and the removal of the website notation in the import string might do the job.

Then the placement of the files is like regular C/C++, for instance "/usr/src/go/packages" (except of course it's only the source).

[Gerard]

Or even better, just place the files in %GOPATH%

[William Kennedy]

I don't understand how someone would not use this language in a production application just because it lacks a convenient devops feature. If Go is the best tool for the job and you know this and you refuse to use it because you need to expend some thought on how best to build and package the code, you are performing an injustice.

The language is very young and more important features are being worked on. If you think this feature is absolutely necessary then go for it. Go is open source, make it happen.

We are using it for a consumer mobile app and are excited with everything we can accomplish. It is the best tool for what we are using it for. Taking the time to incorporate this into our build process has been worth every minimal moment we have expended.

I hope others who are reading this topic don't walk away from Go or consider it a hobby language. It is a very powerful tool. Use it for your production apps.

[Kevin D]

I just built a small script to put me into a GOPATH environment where all my packages are downloaded into my repository. So I have repeatable builds.

outroot@ubuntu:~/dev/go-projects/src/rxmg$ source env/activate 
goenv:rxmg outroot@ubuntu:~/dev/go-projects/src/rxmg$ ls env
activate  pkg  README.md  src

Works exactly like python's virtualenv.

[John Nagle]

On 7/28/2013 6:27 AM, John Waycott wrote:
>
>
> On Friday, July 26, 2013 6:42:07 AM UTC-7, Nathan Youngman wrote:
>>
>> A few fundamentals to building a great packaging story.
>>
>> http://nathany.com/go-packages/

He misses the point there. It's not the packaging mechanism
that matters. It's the quality control on the packages. This
is why CPAN is a success and PyPi (formerly Cheese Shop) is
full of junk. CPAN has tests and testing standards. PyPi is
just a link directory.

John Nagle

[Lars Seipel]

On Sun, Jul 28, 2013 at 03:48:25AM -0700, Philippe Lafoucrière wrote:
> Go is really nice and fun, and I like playing with it. Anyway, it won't be
> enterprise-ready (and yes, I know google is using it in production) until
> this package manager is available...

Yeah, that's nearly as bad as C or C++. No way anyone is using something
like that in the enterprise. Uhm …

> This is really a blocking issue for us. "Go get" is fine for small
> programs, but bigger projects require reproducible builds.

Go get is just a convenience wrapper. You don't have to use it. The only
thing that matters is that you lay out the files in your GOPATH
according to the import paths in your source code. Where those files
come from is in your hand. Versioned tarballs, VCS checkouts or
Debian/RPM packages — it doesn't matter.

To get reproducible builds just write out the versions/commit hashes of
your deps at buildtime. Like others said, there's already a program for
it.

[aries.m...@gmail.com]

2013. július 28., vasárnap 22:01:47 UTC+1 időpontban John Nagle a következőt írta:

    He misses the point there.  It's not the packaging mechanism
that matters.  It's the quality control on the packages.  This
is why CPAN is a success and PyPi (formerly Cheese Shop) is
full of junk.  CPAN has tests and testing standards.  PyPi is
just a link directory.

                                John Nagle

Personally I would never recommend to use external sources when you build a mission-critical or enterprise app. You can have different versions for different builds, you will never know what will be broken on the next build or if the external repo is down, your deployment will fail. Use internal repo when you cook your packages / deployment.

[GreatOdinsRaven]

On Monday, July 29, 2013 6:05:13 AM UTC-6, Lars Seipel wrote:

On Sun, Jul 28, 2013 at 03:48:25AM -0700, Philippe Lafoucrière wrote:
> Go is really nice and fun, and I like playing with it. Anyway, it won't be
> enterprise-ready (and yes, I know google is using it in production) until
> this package manager is available...

Yeah, that's nearly as bad as C or C++. No way anyone is using something
like that in the enterprise. Uhm …

Not exactly a fair comparison. C/C++ are not very opinionated languages, in the sense that they just give you a bazooka and 100 feet of rope and let you both hang *and* shoot yourself all you like, at the same time, sanity be damned. Go *is* opinionated, and the Go Team seem to have strong feelings about what is the right way to do certain things vs what isn't. And that's an excellent thing. The fact that they already wrote "gofmt"  leads to me to believe that a "gopkg" is not that crazy of an idea. 

One of my favorite things about Clojure, another *very* opinionated language I'm dabbling with, is Leiningen - *one* tool to get/build/deploy, *one* tool to rule them all. There are no questions, if's, but's, maybe's. How do you "get" a Clojure package? Leiningen. How do you build a JAR? WAR? Uber-JAR? Leiningen. How do you deploy it to the package repos? Leiningen. It's phenomenal. It works. It's great. Everyone uses it. Period. The community has already moved on to bigger, better things. 

Go doesn't have to compare itself to the joke that is the C/C++ eco-system and feel good about itself - that's setting the bar too low. "go get" is nice, but it can't be, nor should it be, the end-all, let's pat ourselves on the shoulder and move on, kind of deal. 

I say - +1 for a "standard", "blessed", "everyone-please-use-this", "go package" tool. 

[Henrik Johansson]

I think its less about packaging. Make with extra programs does that fine when it comes to bundled resources.

Still (i am sorry) it comes down to versioning. It can be solved but not without some hassle. The superior simplicity of the "go" command only worsens the impression.

I am not proposing that the conflicting trans deps problem be solved but only that non-conflicting trans deps works.

*end bimonthly rant*

/Henrik

[Rob Pike]

I am not convinced the dependency problem can be solved. I believe that in general it is intractable, although in practice the pathological cases might not arise often.

-rob

[Henrik Johansson]

No its only superficially simple i guess.

A tool like maven does resolution and selects the transitive dependency that is declared first. It simply ignores the possible ramifications for the dependency that gets a transparent bump on its dependency... It works compile time since the actual resolution happens runtime. This may have changed but I am fairly certain that it still says so in its docs.

This is of course not good and not something that I want at least.

If only the resolution part could be done I would be happy to have it tell me that something is wrong and I have to fix my dependencies.

I don't know really what would be needed in the go tool but if the sources are required then a simple tag in the import statement might suffice.

Perhaps the library files can be generated to have it as well.

From:

   go.importpath.labix.org/v2/mgo/bson

To:

   go.importpath.labix.org/v2/mgo/bson.v2.1 (or whatever makes sense and can be unambigously understood by all involved tools)

Then it seems like a simple matter of programming to get it to sing? ;-)

I am sure I am overlooking a whole slew of issues...

/ Henrik

[Philippe Lafoucrière]

Not to mention the potential risks of not locking versions:

- unexpected deps behavior
- licence change <= you really don't want the legal dept to come into
your office for that, I mean it, really
- incompatibilities <= will raise with the number of deps available
- vulnerabilities <= sometimes, more recent versions are vulnerable
- dead libraries detection (not maintained any more, so possibly
buggy, with security flows) <= Yes, we have tools for that too
- API locking <= A new API might expose a different API to consumers.
If your go programm is designed to work with a V2 API, and you fetch
the last version for V3, it won't even compile...

The Go ecosystem is still pretty young yet, so these issues are
probably not existing for now, but they will, soon.

[Kyle Lemons]

Sure they do.

I think the only way to approximately "solve" the dependency problem is the same way we "solve" the code formatting problem: provide tools to help where tools can help (and innovate on these outside the standard distribution until we find something we like), and then make a decision as a community to hold ourselves up to the highest standards we can; you'll notice that pretty much any code posted to the mailing list that isn't gofmt or that isn't go-gettable has this pointed out almost immediately.  For example, if we can decide as a community to make only backward-compatible API changes when possible and to provide gofix modules for as many other things as possible and call one another out when we violate this, we'll be well on our way to making the situation better.  One reason I see this problem being so rampant elsewhere is that people take "semantic versioning" too far and assume that because they can make a backward-incompatible API change and it will be safe if they bump the version number, this is an okay thing to do and don't think any more about it.

[Nathan Youngman]

> I don't understand how someone would not use this language in a production application just because it lacks a convenient devops feature.

Yet at the same time I've heard the Go Team boast on the strength of Go's tooling (and tools like gofmt are particularly good evidence).

"Entire developer ecosystems live and die by how dependency management works.", Charlie Robbins, nodejitsu blog

> It works. It's great. Everyone uses it. Period. The community has already moved on to bigger, better things.

:-)

I can't wait until that's true for Go.

I'm beginning to research packaging tools in other language ecosystems. Right now I have a pretty good handle on Pip/Virtualenv (Python) and Gem/Bundler (Ruby). I've been learning NPM (Node.js) and CocoaPods (Objective-C). I hope to dig into Leiningen (Clojure) and perhaps even Composer (PHP) or Cabal (Haskell)

Notice how each language mentioned has one tool (or a few tools for related tasks) vs. the 10 third-party package managers for Go. :facepalm:

Nathan.

[Jan Mercl]

On Fri, Aug 2, 2013 at 5:33 AM, Nathan Youngman <junk...@nathany.com> wrote:

> "Entire developer ecosystems live and die by how dependency management works.", Charlie Robbins, nodejitsu blog

Go has a perfect, correctly working dependency management tool.
Perhaps this is about versioning instead?

> Notice how each language mentioned has one tool (or a few tools for related
> tasks) vs. the 10 third-party package managers for Go. :facepalm:

Go doesn't have an "official" versioning management tool because that
problem is proven to be, in the general case, either automatic or
correct. Pick one.

-j

[Henrik Johansson]

Where is it proven to have these properties? Do you have any references?

[Jan Mercl]

On Fri, Aug 2, 2013 at 10:32 AM, Henrik Johansson <dahan...@gmail.com> wrote:
> Where is it proven to have these properties? Do you have any references?

Obviously infinite many of them exist. The most trivial form goes like:

a1->b1->c1
a1->d1->c2

Then:

- If package 'c' runs an init func doing something what can be done
only once (e.g. create O_EXCL "foo", listen @ :8080, ...), the game is
over.

- If package 'c' has global state, you have a set of two "global" states. Oops.

- If you infer in 'a' a type from 'c' transitively through 'b', it is
not assign compatible with the "same" type inferred transitively
through 'd'. What the ...?!?

- Etc. ad absurdum.

-j

[Philippe Lafoucrière]

Good point. I agree with you a version control tool worse it, to avoid
these issues then ;)

--
Philippe Lafoucrière - CEO
http://www.tech-angels.com
main : +33 (0) 970 444 643
mobile : +33 (0) 6 72 63 75 40
fax : +33 (0) 9 72 12 78 75

> --
> You received this message because you are subscribed to a topic in the Google Groups "golang-nuts" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/golang-nuts/sfshThQ_wrA/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to golang-nuts...@googlegroups.com.

[Henrik Johansson]

Sure. I know it is a basically NP complete problem to formally solve.

That does not mean it can't be handled and handled correctly.

The problem we have now is that the problem exists even if we do not have "semantic versions", since all code dependencies will over time diverge but we ignore it (or we handle it in all the different ways available).

I would prefer a tool that did all the checks you paint in your example and fails with a nice error message.

"A depends on C1 but B depends on C2, please fix your dependencies"

In the case where the dependency graph is valid we just use that.

This would in effect be the same state as today but with a nice failure when it does not compute.

Right? Would it not be better if it was possible to specify and handle?

[Philippe Lafoucrière]

Completely agree.
I guess most of this talk is sterile now, this has been discussed
again and again over time, languages, and software in general.
(I couldn't live without debs package in my debian, so it's the same for go).
I will find time to help on a project, or write a new one if needed,
and if the tool is good and useful, it will become a defacto standard
(like bundler for ruby apps).
end of story :)

[Rob Pike]

It is not NP-complete. In general it is impossible. NP-complete problems have solutions. As Jan's example shows, non-soluble examples are easy to construct.

-rob

[Graham Anderson]

On Friday 02 Aug 2013 11:05:25 Henrik Johansson wrote:
> This would in effect be the same state as today but with a nice failure
> when it does not compute.
> Right? Would it not be better if it was possible to specify and handle?

This pretty much sounds like the route my distro's package manager takes.

It uses a SAT solver (because as you say the problem is known/NP-complete) to
automatically, formally and safely handle the version and dependency
resolution until such a point where either; there's multiple possible paths
with which to proceed, it's not safe to automatically choose for the user, or
it's impossible to proceed because a repo is missing a required package
version. At this point the user is presented with a multiple choice of actions
on how to proceed.

The package manager is zypper and the library used is libzypp, links to source
(C++) and API docs:

http://en.opensuse.org/Portal:Libzypp

I would point out though, that when you make life easier for users and package
consumers by offering a safe and fast SAT solver based solution to package
management; you are shifting the burden and cost of maintenance (providing
meta data for the solver) to the repository and/or package maintainers. That's
preferable of course to having many individual user/consumers bear the cost
but I just wanted to remind folks that effective package management using
solvers, package managers and repos requires careful consideration at both
ends.


Cheers the noo,
Graham

[Henrik Johansson]

I just don't think that we can get around this somehow not counting ignoring it.

Real projects rot and decay but still have to be maintained and of course this is manageable in many ways but when even the Java world that has long resisted (and built third party solutions aplenty) finally realizes that it needs to go into the specification itself (http://openjdk.java.net/projects/jigsaw/) then I think we can at least deduce that there is _something_ to it at least.

--

You received this message because you are subscribed to the Google Groups "golang-nuts" group.

To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.

[tomwilde]

On Monday, July 29, 2013 11:29:01 PM UTC+2, Rob Pike wrote:

I am not convinced the dependency problem can be solved.

-rob

We're free to redefine the problem in a solvable way, though.

It's the ambiguity of what people are used to that makes it impossible.

Let's take a different approach.

- Tom

[Jan Mercl]

On Fri, Aug 2, 2013 at 9:44 PM, tomwilde <sedevel...@gmail.com> wrote:
> On Monday, July 29, 2013 11:29:01 PM UTC+2, Rob Pike wrote:
>>
>> I am not convinced the dependency problem can be solved.
>>
>> -rob
>
>
> We're free to redefine the problem in a solvable way, though.

If every problem could be reformulated in a solvable way then, of
course, 42 is the only correct solution ;-)

-j

[Henrik Johansson]

I have spent the evening reading like a madman and I think I will adopt the "version by sub directory" approach as done by mgo and suggested by many on the list for my api breaking changes.

It was a tough pill to swallow but I think it is not bad at all now that I have gotten over the initial chock. :)

One thing I am curious about though is how dynamic linking will be handled when and if it gets implemented? How does gccgo do it?

[Kyle Lemons]

On Fri, Aug 2, 2013 at 3:16 PM, Henrik Johansson <dahan...@gmail.com> wrote:

I have spent the evening reading like a madman and I think I will adopt the "version by sub directory" approach as done by mgo and suggested by many on the list for my api breaking changes.

Please don't.  This makes it impossible for the compiler to tell you when you've entered a situation like what Jan suggested, as it will happily compile v1/pkg and v2/pkg and link them in. If you update your/pkg and it breaks a dependency, the compiler will yell.

Design your API and don't make backward-incompatible changes without (a) advance notice and (b) a gofix-like tool to fix up your code automatically where possible and add TODOs where the programmer needs to investigate. 

It was a tough pill to swallow but I think it is not bad at all now that I have gotten over the initial chock. :)

One thing I am curious about though is how dynamic linking will be handled when and if it gets implemented? How does gccgo do it?

On Aug 2, 2013 10:17 PM, "Jan Mercl" <0xj...@gmail.com> wrote:

On Fri, Aug 2, 2013 at 9:44 PM, tomwilde <sedevel...@gmail.com> wrote:
> On Monday, July 29, 2013 11:29:01 PM UTC+2, Rob Pike wrote:
>>
>> I am not convinced the dependency problem can be solved.
>>
>> -rob
>
>
> We're free to redefine the problem in a solvable way, though.

If every problem could be reformulated in a solvable way then, of
course, 42 is the only correct solution ;-)

-j

--

[Henrik Johansson]

You mean in case of a diamond? Damn, its late ill think on it some more.
I implicitly meant to use ample notification time of course but in essence the entire problem still remains since no matter how much heads up and gofix provided there will always be someone who does not have the time to adjust and they will be left behind. And making needless api changes will not happen.

That is really my main argument, choosing when to upgrade should be left to both the implementors as well as the users independently.

[Rob Pike]

All the proposals to do the obvious fail because the problem is intractable. It's not only intractable in tricky cases, it's intractable in really simple, easy cases that arise very early in many systems.

The only way to deal with this problem is to explain what the constraints are and be prepared to fail, before compilation or at least before linking, if the constraints aren't met. That requires information outside the source code.

No solution to the problem exists if the only input to the tool is the source code.

It doesn't matter where you put the source or what you write in the source, no solution to the problem exists if the only input to the tool is the source code.

In fact, no solution exists in general. It is up to the user of the dependency tool to state what reduction of the possible state space is acceptable to the programmer, and to the dependency tool itself to guarantee that the program exists within that state space an that that state space guarantees the build can be satisfied.

Stop proposing stuff that doesn't take this into account.

-rob

[Henrik Johansson]

This sounds like what i used to say long before (other threads) but was turned down.

Roughly that each package declares its version and the versions of all its needed dependencies. Circular incompatible transitive dependencies should fail. I don't know why this has to be declared outside of the source code file(s) but it probably have to be in the compiled packages as well. I don't know the technical intricacies well enough wrt feks initialisation.

My last idea was mostly a somewhat exasperated grasping for straws. I really just want proper dependency management in line with the simplicity of the rest of the Go tools and idioms.

I get that this thing is a bit of hot potato but there are already several tools and workflows invented that does it to various degrees. What I want and I may very well be alone in this is a "go fmt" type "this is the tool for dependencies, period" tool to do exactly what you stated.

[Kyle Lemons]

On Fri, Aug 2, 2013 at 4:24 PM, Henrik Johansson <dahan...@gmail.com> wrote:

This sounds like what i used to say long before (other threads) but was turned down.

Roughly that each package declares its version and the versions of all its needed dependencies. Circular incompatible transitive dependencies should fail. I don't know why this has to be declared outside of the source code file(s) but it probably have to be in the compiled packages as well. I don't know the technical intricacies well enough wrt feks initialisation.

If I import a package that says

import "labix.org/mgo" // allow 1.0-2.2

then as soon as I import another package that says

import "labix.org/mgo" // allow 2.5-3.0

that breaks down.  Probably because 2.2 was the latest version when the first one was written, even if it happens to still work on 2.5 or even 3.0.  The package maintainer would have to go do very careful checking of their library under these versions and then update their source and then get everyone to pull the new version... and if they're willing to do that, you might as well just have them do that iff the build fails under 3.0 and ditch the versions in the source completely.

"But then the original maintainer should've said 1.0+!" I hear you saying... now assume that 3.0 *did* have a breaking change.  We're back to where we started.

If you're willing to go outside the source, there are many interesting things to consider.  Perhaps we could upload version tuples every time we compile a package, so other people can check and say 'Oh, cool, this package compiles with this version' and you might have a chance of finding a version that works with all of the packages.  To get something like this, though, you have to give something up, and (at least in the case of rx) that is probably the ability to upgrade beyond an incompatible change anywhere in the tree if a parent hasn't considered it.

My last idea was mostly a somewhat exasperated grasping for straws. I really just want proper dependency management in line with the simplicity of the rest of the Go tools and idioms.

The versioning problem exists outside of Go.  It has never been solved.  I've been unsatisfied with every package/version management tool I've ever used in one way or another.  As I think we've stated many times, there are not going to be any "solutions" to the problem of dependency management that live up to the simplicity of the language, because there are no simple solutions. 

I get that this thing is a bit of hot potato but there are already several tools and workflows invented that does it to various degrees. What I want and I may very well be alone in this is a "go fmt" type "this is the tool for dependencies, period" tool to do exactly what you stated.

You will never find one tool that handles the workflows that everyone wants.  I actually quite like that nobody has told me that I have to use goven, and I won't be telling anyone that they have to use rx.  I'm sure over time we'll converge on a few that we like as a community, but I doubt that we'll ever centralize on something like Maven or PIP (which do more than just dependencies). 

[Henrik Johansson]

You are right of course it breaks down at that point and that is fine, then i know and can go on the hunt for another version or upgrade or whatever. What if the first package really doesn't work with a newer mgo? Wildcard versions sounds bad.

The thing is that we have the exact same problem now only there is no early warning. Things may very well compile nicely but have incompatible behavior that prohibits its usage without changes. Of course this can be thought of as part of not breaking backwards compatibility but it does happen.

At least the library users gets control over what gets used, information about conflicting dependencies and the library developer gets the freedom to change as needed in a new version.

I know that its not a go specific problem. I have lived with Maven for a long time, maybe that is my problem. Maven 1 was better in the dependency department since you had to declare all your dependencies transitively which was a bit of a pain but at least you knew what you got. This is by no means a deal breaker for me and i can manage somewhat since i have to my knowledge not had any issues with conflicting dependencies.

Ill simply have to try to not make any breaking changes and revisit the issue if the need arises.

[Caleb Spare]

On Friday, August 2, 2013 1:49:22 AM UTC-7, Jan Mercl wrote:

On Fri, Aug 2, 2013 at 10:32 AM, Henrik Johansson <dahan...@gmail.com> wrote:
> Where is it proven to have these properties? Do you have any references?

Obviously infinite many of them exist. The most trivial form goes like:

a1->b1->c1
a1->d1->c2

Obviously bringing two different versions of the same code into the project is crazy. So we won't do that.

This is very simple to handle -- if the strict dependencies imply two different versions of a library, then the tool fails. There is no solution. This is how tools I'm familiar with work.

The key thing is that you generally don't rely on a particular version -- you use on a range of versions. So maybe you get:

a1 -> b{1, 2} -> c{1, 2, 3, 4}

a1 -> d{1, 2 3} -> c{2, 3}

And then the dependency resolution tool solves a constraint satisfaction problem and gets e.g. {d2, c3}. (After this you want to lock the dependencies in so anyone else working on the code gets the exact same versions -- this is what, for instance, Bundler uses Gemfile.lock for.)

Something like semantic versioning helps with this process -- if people follow the convention that they don't break APIs without major version changes, then you can generally specify dependencies like "at least 1.2.3 but in the same major version series".

-Caleb

[Kyle Lemons]

On Fri, Aug 2, 2013 at 6:05 PM, Caleb Spare <ces...@gmail.com> wrote:

On Friday, August 2, 2013 1:49:22 AM UTC-7, Jan Mercl wrote:

On Fri, Aug 2, 2013 at 10:32 AM, Henrik Johansson <dahan...@gmail.com> wrote:
> Where is it proven to have these properties? Do you have any references?

Obviously infinite many of them exist. The most trivial form goes like:

a1->b1->c1
a1->d1->c2

Obviously bringing two different versions of the same code into the project is crazy. So we won't do that.

This is very simple to handle -- if the strict dependencies imply two different versions of a library, then the tool fails. There is no solution. This is how tools I'm familiar with work.

You might want to read the blog post and the associated doc :).  The problems below (and many more) are already addressed.

 

The key thing is that you generally don't rely on a particular version -- you use on a range of versions. So maybe you get:

a1 -> b{1, 2} -> c{1, 2, 3, 4}

a1 -> d{1, 2 3} -> c{2, 3}

And then the dependency resolution tool solves a constraint satisfaction problem and gets e.g. {d2, c3}. (After this you want to lock the dependencies in so anyone else working on the code gets the exact same versions -- this is what, for instance, Bundler uses Gemfile.lock for.)

What if there is more than one solution?  Which one do you pick?  What if the other guy on your team ends up having chosen a different set of also-legal versions?  How do you figure out which package maintainer to bug if your dependency chains are long and weblike and there are multiple sets of constraint changes that could make the problem soluble?

 

Something like semantic versioning helps with this process -- if people follow the convention that they don't break APIs without major version changes, then you can generally specify dependencies like "at least 1.2.3 but in the same major version series".

I disagree, actually.  Semantic versioning frees developers (especially those who aren't closely tied to their users) from having to think long and hard about making a breaking API change, because they can "just bump the major version number" and not break anyone.  Well, it also means that every user of their package has to update now in order to get bug fixes, security patches, etc.  It takes a lot of discipline to avoid this.

 

-Caleb

[Caleb Spare]

On Friday, August 2, 2013 10:32:32 PM UTC-7, Kyle Lemons wrote:

On Fri, Aug 2, 2013 at 6:05 PM, Caleb Spare <ces...@gmail.com> wrote:

On Friday, August 2, 2013 1:49:22 AM UTC-7, Jan Mercl wrote:

On Fri, Aug 2, 2013 at 10:32 AM, Henrik Johansson <dahan...@gmail.com> wrote:
> Where is it proven to have these properties? Do you have any references?

Obviously infinite many of them exist. The most trivial form goes like:

a1->b1->c1
a1->d1->c2

Obviously bringing two different versions of the same code into the project is crazy. So we won't do that.

This is very simple to handle -- if the strict dependencies imply two different versions of a library, then the tool fails. There is no solution. This is how tools I'm familiar with work.

You might want to read the blog post and the associated doc :).

I did

 

The problems below (and many more) are already addressed.

they weren't 

 

The key thing is that you generally don't rely on a particular version -- you use on a range of versions. So maybe you get:

a1 -> b{1, 2} -> c{1, 2, 3, 4}

a1 -> d{1, 2 3} -> c{2, 3}

And then the dependency resolution tool solves a constraint satisfaction problem and gets e.g. {d2, c3}. (After this you want to lock the dependencies in so anyone else working on the code gets the exact same versions -- this is what, for instance, Bundler uses Gemfile.lock for.)

What if there is more than one solution?  Which one do you pick?

Doesn't matter, pick any one. It's nice to try to get later versions of packages rather than earlier, but any solution to the constraints is OK.

 

What if the other guy on your team ends up having chosen a different set of also-legal versions?

If you read the second sentence of my paragraph, you'll note that I mentioned locking in the dependencies once you've chosen a solution (like Gemfile.lock does). Check that into git -> done (until the next time the dependencies change).

 

How do you figure out which package maintainer to bug if your dependency chains are long and weblike and there are multiple sets of constraint changes that could make the problem soluble?

Huh? You don't have to bug anybody. It doesn't matter if there are multiple solutions.

 

 

Something like semantic versioning helps with this process -- if people follow the convention that they don't break APIs without major version changes, then you can generally specify dependencies like "at least 1.2.3 but in the same major version series".

I disagree, actually.  Semantic versioning frees developers (especially those who aren't closely tied to their users) from having to think long and hard about making a breaking API change, because they can "just bump the major version number" and not break anyone.  Well, it also means that every user of their package has to update now in order to get bug fixes, security patches, etc.  It takes a lot of discipline to avoid this.

Breaking changes are going to happen regardless. Not everyone has the same workflow you do. For instance, I like to experiment and play around with APIs, and then solidify them as I get more practice using them. This fits nicely with the pre-version-1 part of semver. You're not going to avoid people making breaking changes if we don't use some versioning scheme; it will just make it more painful when they do.

[Nathan Youngman]

I disagree, actually.  Semantic versioning frees developers (especially those who aren't closely tied to their users) from having to think long and hard about making a breaking API change, because they can "just bump the major version number" and not break anyone.  Well, it also means that every user of their package has to update now in order to get bug fixes, security patches, etc.  It takes a lot of discipline to avoid this.

Hey Kyle. I see where you are coming from, but I think this is more of a people problem than a tool problem. People can introduce breaking changes whether or not they have a good way to indicate that they have. What this says to me is that tools themselves are insufficient, we also need best practices.

I think it's awesome how Rx runs tests as part of package installation, but I think I'd rather use that to indicate a bug or unintentional breaking change, and still have versions (or some equivalent?) to indicate intention.

Breaking changes may be inevitable. Take another read of why mgo went v2: https://groups.google.com/forum/#!searchin/golang-nuts/Gustavo$20Niemeyer$20mgo/golang-nuts/EzDFpLN6eHw/8hIDMmLC2lEJ

That doesn't mean breaking changes should be taken lightly, or that versions should be used as an excuse to change APIs every year.

Nathan.

[Dave Cheney]

I don't care about semantic versioning, breaking changes,
unintentional upgrades, etc.

Here are my requirements

1. I want to be able to identify a particular revision for each
dependency of the Juju project, and a location to fetch said.
2. I want to be able to fetch the source for those revisions if they
are not included directly in the Juju source.
3. I want to be able to be able to reliably follow changes in the
dependency graph such that when my fellow developers, or I, change
that graph they can reliably reproduce it on their machines.
4. I want to have two (or more) working copies of Juju, stable and
devel, which have different dependency graphs such that building
stable doesn't accidentally use devel's deps and vice versa.

I think this is achievable, and the best model I've seen is the way
Bundler does it.

Cheers

Dave

[Nathan Youngman]

Even though I use Bundler practically everyday at my job, I'm still doing some research into it. It certainly might be nice to have a similar tool that puts everything in the right place so the Go compiler could just work as usual. I intend to summarize it and some other tools, with a particular focus on how they work when pulling source directly from a DVCS.

As part of my research, here is part of the transcript from a podcast with one of the Bundler maintainers:

Andre Arko, Ruby Rouges 045, March 14, 2012

http://rubyrogues.com/episode-guide/

@ 8:40

Andre: "If you think Bundler is too complex, then you probably don't understand the problem"

James: "Yes, can we talk about that? Isn't the problem of resolving those dependencies actually an NP hard problem?"

Andre: "It is in fact. It is NP-Complete, just like any other dependency graph problem. So we do the best we can, but every once in a while, someone manages to craft a Gemfile that has such a wide... In your Gemfile you can list a range of versions that you'll accept as a valid resolution. And some people just don't put version numbers in at all, they just list all their gems, and that of course is an implicit >= 0 version number. And if you put enough gems in your Gemfile with a >= 0 and there are enough versions of each of those gems -- for example, I hear there are a lot of versions of the Rails gem. If you don't require any specific version at all, and there are many, many, many versions of each of those gems that you listed in your Gemfile, of course this means that sometimes we get bug reports that Bundler has been attempting to resolve for the last hour, and will it ever finish, and why won't you fix this terrible, terrible bug?"

...

@ 10:55

Andre: "The high-level overview is that it tries as hard as it can to narrow down the problem space in advance, most people's Gemfiles actually do in fact resolve. The time that it takes to download and unpack all those Ruby gems, that occupies the vast majority of Bundler's time rather than actually running the resolver. It's a very rare Gemfile that manages to make it into that crazy problem space. And as far as I know, all of those people have been able to solve their problem by specifying a more narrow version in their Gemfile."

@ 19:00

Andre: "Funny story about why Bundler was so slow. It's almost never the resolver. Everyone's like, well it's an NP-Complete problem, it will be really slow, right? Well it turns out for the vast majority of Gemfiles, and honestly, if you have a Gemfile.lock, there is no problem left right, the graph has already been resolved. All of the slowness of Bundler came from having to download and sort through a vast amount of from RubyGems... RubyGems offered an index... on that server that lets you know the name and version number of every single gem that has ever been released in the history of gems.... Bundler 1.0 had to go download that entire thing. That index keeps getting bigger, and it's never getting smaller. Even gzipped... it's almost a megabyte."

@ 20:45

Andre: "Nick Quaranto came up with an API that he added to the RubyGems.org server... You can hit that API with a list of gems, and it will give you back only the information about those gems and those gems child dependencies. Since that's all the information that Bundler wanted, this is significantly faster than downloading everything about every gem ever."

@ 25:04

Andre: "Bundler resolves the graph in advance, it's not a runtime dependency resolver... Because the graph is already resolved, it then locks down all of the other gems that can be loaded to the pre-approved correct versions."

@ 36:25

Andre: "There are 3 levels of installation of gems with Bundler. Install is the most conservative, it grabs gems that meet your stated requirements, make sure none of them conflict with each other, and writes out your Gemfile.lock. Once you have a lock file, `bundle install` doesn't change anything, running install just gets you the gems that you already know work with your app.

"The next level is running `bundle update gemname`, that only allows that gem and child dependencies of that gem to update... Lastly, least conservatively is running `bundle update` which gives me the newest versions of everything, similar in function to just deleting the Gemfile.lock."

@ 51:00

Andre: "The initial development was driven by the necessity of Merb having so many gems, and such a complicated dependency graph, that human dependency resolution was no longer practical."

---

Nathan.

[Jason McVetta]

On Friday, August 2, 2013 10:32:32 PM UTC-7, Kyle Lemons wrote:

I disagree, actually.  Semantic versioning frees developers (especially those who aren't closely tied to their users) from having to think long and hard about making a breaking API change, because they can "just bump the major version number" and not break anyone.  Well, it also means that every user of their package has to update now in order to get bug fixes, security patches, etc.  It takes a lot of discipline to avoid this.

It would be nice if there were an automated way to know when one of my commits breaks a consumer of my package.  

GoDoc.org can list the packages that import a given package.  It's only based on what GoDoc has seen, but it's a good start.  Haven't looked around yet to see if there's a nice API to fetch that info.  If there is, it probably wouldn't be too hard to write a tool that fetches all the known consumers of my package, and runs all of their tests.  

Most developers probably don't want to break their consumers more than absolutely necessary.  Greater visibility into the downstream consequences of a commit would help a developer avoid such breakage.

[Dobrosław Żybort]

Would be even better to check http://go-search.org as it list also "main" programs.

My oauth2 package example:
http://go-search.org/view?id=bitbucket.org%2fgosimple%2foauth2#imported

And JSON with all data (search for: "Imported"):
http://go-search.org/api?action=package&id=bitbucket.org%2fgosimple%2foauth2

Best regards,
Dobrosław Żybort