[ANN] gonuts.io - centralized repository for versioned Go packages

[Alexey Palazhchenko]

Good news everyone!

I'm happy to announce a preview of http://gonuts.io/ – centralized repository for versioned Go packages.

Why do I think Go ecosystem need this? There are two problem "go get" can't solve.

First, it doesn't support package versioning – it always installs latest version. If someone wants to install previous one, he/she has to use git/hg/svn/bzr manually. Therefore, package authors are forced to maintain backward compatibility since first commit. If they want to remove some API, they should use a different repository.

Second, in practice many developers are moving their code to other places (for example, from Google Code to GitHub), renaming repositories (web.go become web) or just deleting them (at the time of this writing many links in Go Projects Dashboard and GoPkgDoc are dead). Yes, it's a social problem, but we still should deal with it.

So how can we solve those problems? Big companies typically have special repositories for third-party code. Once imported there, code is never deleted. And they have a budget to fix their world of dependencies. So, "go get" probably works okay for "google/..." packages. Smaller companies and individual developers are able to bundle third-party packages with their application and take pain of updating them only when needed. But what should package developers do if they want to use other packages?..

gonuts.io for Go, similar to PyPI for Python, RubyGems.org for Ruby and NPM for Node.js should solve those problems. Published packages (called "nuts") are never deleted, and versioning schema allows to install exact version. There are plans to allow to install a version matching pattern (like 2.*.*), but still be in control (similar to RubyGems' Bundler). And nut client tool was designed to work along with Go conventions: nuts are installed into workspace specified by GOPATH environment variable, and imported as "gonuts.io/test_nut1/0.0.1".

There are few more things I need to do before official launch. First of all, I want to provide a clear path for transition of well-known packages to gonuts.io without names being squatted. So for now gonuts.io works in preview mode, and all nuts will be removed before going into real production.

So – feel free to try Go Nuts, publish your packages, install other and post your comments into this thread or gonu...@googlegroups.com discussion group. You even may contribute. ;)

Thanks.

–-–
Alexey "AlekSi" Palazhchenko

[Jimmy Zelinskie]

There were rumors of Andrew Gerrand doing something with Gary Burd (gopkgdoc author) in order to create a central repo a while back. I know making a central repo has been something on the chopping block, but I don't know if any progress has been made. It's nice to see some initiative towards solving the problem. Even if this doesn't become the de facto repo, I'm sure its innovations will drive future solutions to be more fully featured. I'm looking forward to seeing the community response.

Best Wishes

[Andrew Gallant]

I also like this idea, and I'm glad you've taken the initiative to get
something rolling.

I do have one concern, and I think it's something that has to be
addressed sooner rather than later. That is, the import path naming.
The way I see it, there are two choices:

gonuts.io/username/package-name

Or

gonuts.io/package-name

From your post, it is my understanding that you have chosen the latter
scheme.

The latter is shorter and could be useful to represent "canonical"
packages. But there's a practical problem here: how is a package
determined to be canonical? There are various solutions, but I'm
personally not a big fan of any of them. (What happens when a project
dies? Or if another project becomes more popular?) With the first
scheme, a package can only become canonical through use and
popularity, which IMO, is the "right" way.

Secondly, I think the latter scheme hinders an important feature of
the Go tool: it is stupidly easy to fork an existing project, and add
your own changes. I think the latter scheme makes this more
inconvenient due to having to choose a different name. On the other
hand, the first scheme has the nice property of just having the
username change while the project name remains the same.

- Andrew

[Kevin Gillette]

Agreed. In my experience, the major failings of centralized package systems from other languages is they use a single flat namespace. To be effective, and exist within the social/collaborative development model, you must give each user their own namespace. Besides just the forking problem is the general name collision problem: there are no doubt many goutil packages out there, which more than likely have nothing to do with each other -- it would be detrimental (and drive potential users away) if you provisioned package names on a "first come first serve" basis.

[David DENG]

Good try!

But the name sounds a little bit confusing.

David

[Jesse McNelis]

On Mon, Dec 10, 2012 at 12:50 PM, Andrew Gallant <jam...@gmail.com> wrote:

The latter is shorter and could be useful to represent "canonical"
packages. But there's a practical problem here: how is a package
determined to be canonical?

I agree, this is a problem that exists in all other package systems.

Popularity isn't a good measure of canonical either. 

The fact that a package has the name "email" means that it's much harder for 

any other possibly better email package to get popular.

Bad early packages become popular and stay popular due to their names.

Developers that come later, in 5 years, will find that all the names are taken. 

So they have to make up nonsense names for straight forward packages.

pypi and gems are great examples of this.

I'm not a fan of explicit package versioning either. 

Explicit versioning allows packages to continue to justify depending on older versions of other packages.

The result is multiple packages that depend on many different versions of the same package with a python2/3 like divide being the result.

I'd rather developers acknowledge that dependencies are a hard social problem instead of pretending it's something simple that we can solve in software.

--
=====================
http://jessta.id.au

[Andrew Gerrand]

Not a central repo - just a way to make existing packages more discoverable. That something is GoPkgDoc, which now has a pretty good search engine: http://go.pkgdoc.org/

The community still needs a centralized directory of projects and other resources, rather than just packages. I have some ideas about how that might work, but no code. (I did have a half-written thing on my laptop before it was stolen. :/ Haven't worked up the energy to start it again yet.)

--
 
 

[Jan Mercl]

By accident replied only to Andrew, forwarding to list. Sorry.

---------- Forwarded message ----------
From: Jan Mercl <0xj...@gmail.com>
Date: Mon, Dec 10, 2012 at 8:11 AM
Subject: Re: [go-nuts] Re: [ANN] gonuts.io - centralized repository
for versioned Go packages

To: Andrew Gerrand <a...@golang.org>


Quoting `sgndave` at reddit
(http://www.reddit.com/r/golang/comments/14jrvf/gonutsio_centralized_repository_for_versioned_go/c7dv8ah):

----

> First, [go get] doesn't support package versioning – it always installs latest version.

> If someone wants to install previous one, he/she has to use git/hg/svn/bzr manually.
> Therefore, package authors are forced to maintain backward compatibility since first
> commit. If they want to remove some API, they should use a different repository.

This is only half-accurate. There is a contract between both the
package authors and the package users. If the upstream package API
changes, there needs to be a period of backward-compatibility in which
downstream users are expected to make the transition. It doesn't
necessarily have to work back to the first commit.

If it is completely impossible to resolve this API skew, then yes, it
becomes a different package altogether. Once you remove two legs and
nail a beak, webbed feet, and wings onto a dog, it's not a
dog_v2.01.07a. It is a duck.

In general, this is exactly the transitive dependency problem that go
tries so hard to avoid. If package A depends on B and C, and Balso
depends on C, then A and B had better agree upon which version of C
they will use. If this is impossible, they need to depend on different
targets. Calling it "versioning" simply side-steps the issue that A
and B cannot share a C.

Criticism aside, it's always good to see more options available in the
go community. Packages and APIs do change, so philosophy aside, I hope
this site makes go more useful to a broader audience of developers.

----

-j

[Alexey Palazhchenko]

Thanks for feedback!

About flat namespace first. Issues like this is why gonuts.io works in "preview" mode now.

I forgot to mention one more source of influence: Composer/Packagist for PHP. It uses names in format vendor/project.

Let's see use-cases:
1) Alice fixes a bug in Bob's nut. Bob is on vacation and can't apply fix and release a new version in 2 weeks. Meanwhile, Alice may fork repository, apply patch and use go get.
2) Alice proposes a new feature, but Bob doesn't like it. They discuss it. If Alice thinks that feature is useful only for her, she forks repository and uses go get. If Alice insist this killer-feature is must-have for everyone, she goes ahead and creates new nut.
3) Alice wants to maintain a neglected Bob's nut. She may ask him to add her as author so she can publish new versions, or she may create new nut.

So I think "canonical" packages are useful, and ability to fork code and use go get is useful too. Also I think (and hope) that flat namespace will lead to better packages, not just more packages.

But it's not a hard decision. Adding vendor names isn't hard. I would like to hear more opinions about this issue.

–-–
Alexey "AlekSi" Palazhchenko

[Dave Cheney]

Mate, I'm not going to quote urban dictionary to you, but you have to
find another name for a versioned package. Nut is not acceptable.

Dave

> --
>
>

[Alexey Palazhchenko]

About everything else.

> But the name sounds a little bit confusing.

Oops. Ruby has Gems, Python has Eggs (not quite equivalent, I know). I thought for Go to has Nuts (fruits or hardware, I not decided yet) would be okay, and a nice word play with "go nuts". Is it that bad?

> Mate, I'm not going to quote urban dictionary to you, but you have to find another name for a versioned package. Nut is not acceptable.

I wasn't aware about this, sorry. :( In my defense I may say that this part of human body is called eggs in Russian, and we don't have much jokes about Python eggs.

> The community still needs a centralized directory of projects and other resources, rather than just packages. I have some ideas about how that might work, but no code.

May you share them?

> Explicit versioning allows packages to continue to justify depending on older versions of other packages.

I plan to add support for "optimistic" version constraint like 1.*.* or 1.>=3.* and rewrite imports in code on install. It's not implemented yet.

> I'd rather developers acknowledge that dependencies are a hard social problem instead of pretending it's something simple that we can solve in software.

I 100% agree with this. It's social, and it's not simple. But when why the lucky stiff committed infosuicide and removed his code from everywhere, it was also a problem for me – I wasn't able to deploy easily. RubyGems.org did not existed at this time. If it were, life would be simpler.

–-–
Alexey "AlekSi" Palazhchenko

[Jan Mercl]

.On Mon, Dec 10, 2012 at 9:50 AM, Alexey Palazhchenko

<alexey.pa...@gmail.com> wrote:
>
> I plan to add support for "optimistic" version constraint like 1.*.* or
> 1.>=3.* and rewrite imports in code on install. It's not implemented yet.

A imports C, requires C 1.8.0
B imports C, requires C >= 2.*
main imports A, B

How is the versioning schema supposed to solve this situation?

-j

[Alexey Palazhchenko]

A imports C, requires C 1.8.0
B imports C, requires C >= 2.*
main imports A, B

How is the versioning schema supposed to solve this situation?

It will prevent you from doing this unless you really want to (something like -f flag). And in fact Go compiler will compile it: import paths are different.

–-–
Alexey "AlekSi" Palazhchenko

[Jan Mercl]

Then the versioning schema is completely broken. If C, for example,
happens to export some global state/entity, for instance like package
"log" does, than building a program with two different(ly named)
versions of C linked into the binary is a pretty clear show stopper.
This is never going to work.

-j

[Alexey Palazhchenko]

>> B imports C, requires C >= 2.*
>> main imports A, B
>>
>> How is the versioning schema supposed to solve this situation?
>
> It will prevent you from doing this unless you really want to (something
> like -f flag). And in fact Go compiler will compile it: import paths are
> different.

Then the versioning schema is completely broken. If C, for example,
happens to export some global state/entity, for instance like package
"log" does, than building a program with two different(ly named)
versions of C linked into the binary is a pretty clear show stopper.
This is never going to work.

 
If you do pass -f, it becomes your responsibility to deal with global state and other issues. Just don't.

[Andrew Gerrand]

Would you prefer gonads.io?

--

[José Carlos Nieto]

This is an interesting topic.

I actually like that Go does not have a centralized source code repository: I can fork an open source repo, make some changes and then use the same name to import the package into my application, that's just great and it's one of my favorite features. So I think that's an important concept you should keep in a centralized repo, vendor names.

I am not saying you should do this, but I'd like something like this could be done with the current Go buid and go get:

import "gonuts.io/vendor/package:1.1"

to fetch the "1.1" version/tag of a package or just:

import "gonuts.io/vendor/package"

to fetch the latest bleeding edge revision.

Of course the ":1.1" part should be done within the Go compiler but I just wanted to point it out.

I also think Go is missing is a good centralized directory, I know pkgdoc.org has a search engine and a directory but the problem is not solved yet, the directory does not seem to have a ranking for packages or at least a way to show most relevant stuff first, reading all packages descriptions is too boring, also we could use the help of a designer to fix the directory looks, it misses details but they're important details. These days I prefer reading opinions and the actual code before importing anything instead of the developer's description. I think gonuts.io can learn from how we use pkgdoc.org and how we use third party libraries.

Some centralized code repositories, like Linux distros, have package maintainers, it is not common to let the developers handle their own packages as the quality/popularity must be ensured by a trusted third party (the maintainer), what's your idea about package maintainers? what if some package has evil code in it?

> 3) Alice wants to maintain a neglected Bob's nut.

Well, Poor Bob, I didn't know Alice was that gross.

- Carlos

[tomwilde]

On Monday, December 10, 2012 10:46:02 AM UTC+1, Andrew Gerrand wrote:

Would you prefer gonads.io?

Made my day.

As to the idea; I think it's awesome but regarding versioning I would suggest keeping it simple and making it only part of the import path, not promoting it much with wildcard matching or whatever.

This way nobody gets confused (is it part of the language?) and no author gets too enthusiastic about it. I like the way Gustavo Niemeyer did it. Also, it should be optional.

- Tom

[Kevin Gillette]

On Monday, December 10, 2012 12:13:45 AM UTC-7, Jan Mercl wrote:

This is only half-accurate. There is a contract between both the
package authors and the package users. If the upstream package API
changes, there needs to be a period of backward-compatibility in which
downstream users are expected to make the transition. It doesn't
necessarily have to work back to the first commit.

I agree only to the extent that the package author has a contract with the package user due to a lack of tooling in this area. I do believe that a package author has a contract to keep previously public revisions available in their source tree, except for revisions that sensitive data managed to leak into.

In Go already, go get does not update packages unless you explicitly ask it to, and these are build-time, not run-time concerns. Additionally, many Go programmers are wary (for security reasons and otherwise) of fetching the tip/head of a repo when building in a new environment, and automated build systems for go seem to more frequently use a push-to-target deployment model, rather than a build-on-target model (which makes dependency management a relative non-issue even for massively-parallel deploys).

While I loathe manifest files and such that you see in gem's, egg's, etc., those are designed for a deployment and runtime model nearly opposite to how Go functions; if the go tool were extended with a 'freeze' command that dumped dependency versions (including the installed version of Go and the stdlib) into a .go-deps file within that package's directory, that file could later be used by other tools to conditionally fetch the exact dependency revisions (with some exceptions, like allowing newer stdlib/runtime releases). go get would also benefit from the ability to select a revision/branch via an option switch. If this tooling existed, then package users would not be so troubled when an upstream API is changed in a backwards incompatible way. A tool that dispatches to `go tool api` could even leverage rcs bisect capabilities to find and report on the latest revision supporting the (subset of the) foreign API used by an app or package.

With a complete core toolset, maintaining backwards compatibility should not be a mandate, though that's not to say it wouldn't be a good thing to maintain the old API on a separate branch if there are known to be existing users and bugs which need backported fixes.

[Kyle Lemons]

On Tue, Dec 11, 2012 at 6:41 PM, Kevin Gillette <extempor...@gmail.com> wrote:

On Monday, December 10, 2012 12:13:45 AM UTC-7, Jan Mercl wrote:

This is only half-accurate. There is a contract between both the
package authors and the package users. If the upstream package API
changes, there needs to be a period of backward-compatibility in which
downstream users are expected to make the transition. It doesn't
necessarily have to work back to the first commit.

I agree only to the extent that the package author has a contract with the package user due to a lack of tooling in this area. I do believe that a package author has a contract to keep previously public revisions available in their source tree, except for revisions that sensitive data managed to leak into.

In Go already, go get does not update packages unless you explicitly ask it to, and these are build-time, not run-time concerns. Additionally, many Go programmers are wary (for security reasons and otherwise) of fetching the tip/head of a repo when building in a new environment, and automated build systems for go seem to more frequently use a push-to-target deployment model, rather than a build-on-target model (which makes dependency management a relative non-issue even for massively-parallel deploys).

While I loathe manifest files and such that you see in gem's, egg's, etc., those are designed for a deployment and runtime model nearly opposite to how Go functions; if the go tool were extended with a 'freeze' command that dumped dependency versions (including the installed version of Go and the stdlib) into a .go-deps file within that package's directory, that file could later be used by other tools to conditionally fetch the exact dependency revisions (with some exceptions, like allowing newer stdlib/runtime releases). go get would also benefit from the ability to select a revision/branch via an option switch. If this tooling existed, then package users would not be so troubled when an upstream API is changed in a backwards incompatible way. A tool that dispatches to `go tool api` could even leverage rcs bisect capabilities to find and report on the latest revision supporting the (subset of the) foreign API used by an app or package.

You're describing something similar to what I started, rx : kylelemons.net/go/rx

The problem is what was described above.  If A depends on B v1 and C v2, and B depends on C v1, then no amount of tooling can really help you if you need a feature that's in C v2.

With a complete core toolset, maintaining backwards compatibility should not be a mandate, though that's not to say it wouldn't be a good thing to maintain the old API on a separate branch if there are known to be existing users and bugs which need backported fixes.

--
 
 

[Martin Angers]

A few thoughts about this:

I think gopkgdoc is great for package discovery. This part it handles well, in a decentralized way (don't know how many hosting providers are supported, but clearly most big ones), this is going in the right direction I believe (especially since it supports Go's experimental packages! Now my packages can be found... :).

As for versioning, node's npm has perfected this to an art form. Managing dependencies, like the A that uses B and C with a different B is trivial in node, and versions can be locked down using shrinkwrap, it's really neat. But this is in dynamic javascript-land, not in statically-linked Go-land, where it's a bit harder. The OP announcement is the first attempt that I'm aware of to tackle this problem, this is great news, but I'm not so sure about the centralized repo thing though. Couldn't this work using conventions, such as looking for a vM.m.p tag (Major, minor and patch, as prescribed by http://semver.org/) in source control? No need to publish to a separate site?

Finally, for the "moral contract" between package developers and users, the API compatiblity and all, this is why we have versions. The solution is version-based, not a fragile and implicit "contract"-based that puts a hell of a lot of pressure on the package developer ("yes, your production code will always work!"). This helps not only for breaking changes, but to explicitly announce bug fixes too (hey, here's a new patch-bumping version: bug fix).

[Kyle Lemons]

On Wed, Dec 12, 2012 at 9:25 AM, Martin Angers <martin....@gmail.com> wrote:

A few thoughts about this:

I think gopkgdoc is great for package discovery. This part it handles well, in a decentralized way (don't know how many hosting providers are supported, but clearly most big ones), this is going in the right direction I believe (especially since it supports Go's experimental packages! Now my packages can be found... :).

As for versioning, node's npm has perfected this to an art form. Managing dependencies, like the A that uses B and C with a different B is trivial in node, and versions can be locked down using shrinkwrap, it's really neat. But this is in dynamic javascript-land, not in statically-linked Go-land, where it's a bit harder.

To be clear, it's not "hard" to link Av1 -> {Bv1, Cv2}, Bv1 -> {Cv1} in Go.  It's not possible.  If you use different directories, then it's no longer the same package, so it "works," but you will have two instances of the package in memory whose values cannot be used interchangeably, not to mention the potential of having separate global state if the package employs it.

 

The OP announcement is the first attempt that I'm aware of to tackle this problem,

If you go back in the history of the mailing list, there have been many attempts to tackle this problem through a number of different mechanisms.  I, myself, have made two separate attempts which address one or more of the problems (one as a proxy for packages and one as a tool to help versioning), and neither were the first.

 

this is great news, but I'm not so sure about the centralized repo thing though. Couldn't this work using conventions, such as looking for a vM.m.p tag (Major, minor and patch, as prescribed by http://semver.org/) in source control? No need to publish to a separate site?

I don't think we should enforce versioning semantics on the entire community.  Sure, it's probably a good idea to tag versions in your repository, but (especially because of the linking problem above) I don't think it's as important a requirement as stabilizing your exported API.

 

Finally, for the "moral contract" between package developers and users, the API compatiblity and all, this is why we have versions. The solution is version-based, not a fragile and implicit "contract"-based that puts a hell of a lot of pressure on the package developer ("yes, your production code will always work!"). This helps not only for breaking changes, but to explicitly announce bug fixes too (hey, here's a new patch-bumping version: bug fix).

Again, unless you bump every package in a binary to the bug fix version, it simply won't work.  At that point, you've basically just sync'd to head (on the default branch) and don't really need to be concerned with version numbers.

 

Le dimanche 9 décembre 2012 04:23:45 UTC-5, Alexey Palazhchenko a écrit :

Good news everyone!

I'm happy to announce a preview of http://gonuts.io/ – centralized repository for versioned Go packages.

Why do I think Go ecosystem need this? There are two problem "go get" can't solve.

First, it doesn't support package versioning – it always installs latest version. If someone wants to install previous one, he/she has to use git/hg/svn/bzr manually. Therefore, package authors are forced to maintain backward compatibility since first commit. If they want to remove some API, they should use a different repository.

Second, in practice many developers are moving their code to other places (for example, from Google Code to GitHub), renaming repositories (web.go become web) or just deleting them (at the time of this writing many links in Go Projects Dashboard and GoPkgDoc are dead). Yes, it's a social problem, but we still should deal with it.

So how can we solve those problems? Big companies typically have special repositories for third-party code. Once imported there, code is never deleted. And they have a budget to fix their world of dependencies. So, "go get" probably works okay for "google/..." packages. Smaller companies and individual developers are able to bundle third-party packages with their application and take pain of updating them only when needed. But what should package developers do if they want to use other packages?..

gonuts.io for Go, similar to PyPI for Python, RubyGems.org for Ruby and NPM for Node.js should solve those problems. Published packages (called "nuts") are never deleted, and versioning schema allows to install exact version. There are plans to allow to install a version matching pattern (like 2.*.*), but still be in control (similar to RubyGems' Bundler). And nut client tool was designed to work along with Go conventions: nuts are installed into workspace specified by GOPATH environment variable, and imported as "gonuts.io/test_nut1/0.0.1".

There are few more things I need to do before official launch. First of all, I want to provide a clear path for transition of well-known packages to gonuts.io without names being squatted. So for now gonuts.io works in preview mode, and all nuts will be removed before going into real production.

So – feel free to try Go Nuts, publish your packages, install other and post your comments into this thread or gonu...@googlegroups.com discussion group. You even may contribute. ;)

Thanks.

–-–
Alexey "AlekSi" Palazhchenko

--
 
 

[Martin Angers]

To be clear, it's not "hard" to link Av1 -> {Bv1, Cv2}, Bv1 -> {Cv1} in Go.  It's not possible.  If you use different directories, then it's no longer the same package, so it "works," but you will have two instances of the package in memory whose values cannot be used interchangeably, not to mention the potential of having separate global state if the package employs it.

Of course, you're right. I agree, but I also disagree :) We don't talk from the same perspective. The reasons you mention to say it's not possible, I (implicitly) mention as part of the solution. A different directory is a different package from Go's perspective, of course, otherwise it would not work. But from a user's perspective, if we end up with a good enough tool, this is exactly what he wants. No, they won't share internal state, that's how it is in node too, or in .NET if you load different versions of the same assembly, there's a reason A and B don't use the same version of C, it's because they have separate needs. I believe this would rarely be a problem, but I may be terribly wrong, I have more experience in dynamically-linked languages.

What I see as a real problem (not that I have thought this through, though!) is that in the A-B-C case, assuming B is a general purpose package installed from a 3rd party (and not developed specifically for application A), it is probably not written to import a specific version of C. So the tool would need to rewrite the import (a-la Go Vet?). As I said, a bit harder, but impossible? It depends on what solution we are willing to accept.

I don't think we should enforce versioning semantics on the entire community.  Sure, it's probably a good idea to tag versions in your repository, but (especially because of the linking problem above) I don't think it's as important a requirement as stabilizing your exported API.

No need to enforce it, if there is no version, then fallback on Go1 tag or master. Obviously, these packages would be used on the presumption that they will maintain compatibility. As for stabilizing your exported API, this is only a small subset of the use of versions. If I deployed an app a month ago, and I need to deploy it on another server a month later, what guarantees do I have that your non-versioned, API-compatible package that I use is still stable for my use? With versions, I grab the exact same one. Without versions, leap of faith (or more realistically, try and find the commit hash to fetch this specific commit by hand).

 

[Kevin Gillette]

On Wednesday, December 12, 2012 11:07:58 AM UTC-7, Kyle Lemons wrote:

To be clear, it's not "hard" to link Av1 -> {Bv1, Cv2}, Bv1 -> {Cv1} in Go.  It's not possible.

I'm not convinced this is a major problem with Go. If it's that hard to update B to use Cv2, then either B or C was probably written quite poorly. The real issue comes with whether Cv2 has a disjoint feature-set with Cv1 (and where features in the symmetric difference are needed by A and B respectively. In this case, I'd say the author of C is either is taking a long time on v2 (and didn't have the good sense to delay releasing it until it was at least as capable as v1), or the author of A didn't have the good sense to notice this kind of issue.

In any case, I don't think this is a case that needs tooling. I was suggesting a dependency list, not for the each library, but for the app as a whole (if you can't build it with `go build`, then deps don't matter, and if the problem you describe above happens to occur, you can't `go build` anyway).

[Kyle Lemons]

On Wed, Dec 12, 2012 at 5:31 PM, Kevin Gillette <extempor...@gmail.com> wrote:

On Wednesday, December 12, 2012 11:07:58 AM UTC-7, Kyle Lemons wrote:

To be clear, it's not "hard" to link Av1 -> {Bv1, Cv2}, Bv1 -> {Cv1} in Go.  It's not possible.

I'm not convinced this is a major problem with Go.

Hmm. I didn't mean to imply that I thought it was a problem with Go itself, or even a language problem.  It seems like it exists in lots of languages, and I think it's a social problem that occurs with the changing of external APIs, which often have multiplicative effect on package consumers (or even design of the API without consideration for future updates).

 

If it's that hard to update B to use Cv2, then either B or C was probably written quite poorly.

That is quite true.  In fact, in the cases in which a package author does need to change their API, it would be really swell of them to provide a gofix module for it, after a suitable period of supporting the backward-compatible interface as well.

 

The real issue comes with whether Cv2 has a disjoint feature-set with Cv1 (and where features in the symmetric difference are needed by A and B respectively. In this case, I'd say the author of C is either is taking a long time on v2 (and didn't have the good sense to delay releasing it until it was at least as capable as v1), or the author of A didn't have the good sense to notice this kind of issue.

In any case, I don't think this is a case that needs tooling.

That's one of the reasons I stopped working on rx, actually.  It was conceptualized as a way to identify and track inter-repository dependencies in such a way that it could then pull the updates from a repository and check that nothing depending on it broke, and if it did, play games with (tagged) versions in between to see if one of them still works.  I'll also mention that I planned on having it run tests, as API changes aren't the only thing that could change, and a package author might not even realize that he made a semantic change or that he depended on an undocumented feature/bug in another package.

 

I was suggesting a dependency list, not for the each library, but for the app as a whole (if you can't build it with `go build`, then deps don't matter, and if the problem you describe above happens to occur, you can't `go build` anyway).

--
 
 

[Kevin Gillette]

On Wednesday, December 12, 2012 9:37:01 PM UTC-7, Kyle Lemons wrote:

If it's that hard to update B to use Cv2, then either B or C was probably written quite poorly.

That is quite true.  In fact, in the cases in which a package author does need to change their API, it would be really swell of them to provide a gofix module for it, after a suitable period of supporting the backward-compatible interface as well.

Intriguing. A quick search, however, seems to indicate that there's no "userland" support in gofix yet. I suppose limited changes could be bundled in a file of gofmt invocations that could probably be made to be sh/bat/rc compat all at once.

 

That's one of the reasons I stopped working on rx, actually.  It was conceptualized as a way to identify and track inter-repository dependencies in such a way that it could then pull the updates from a repository and check that nothing depending on it broke, and if it did, play games with (tagged) versions in between to see if one of them still works.  I'll also mention that I planned on having it run tests, as API changes aren't the only thing that could change, and a package author might not even realize that he made a semantic change or that he depended on an undocumented feature/bug in another package.

You've made me realize that I've now danced on both sides of the argument. Certainly with this awareness, I'm now leaning towards the "leverage go's hackability" as an inherent "tool". In something like Java, where you see a lot of copypasta, both library authors and library users may be unwilling to consider or adapt to incompatible changes without automated or batched tooling. In go, those habits may linger, yet I regularly find it's faster to figure out, fix, scrap, and then rewrite large portions of someone else's library than it is to wait for a response to an bug report, for better or worse. Sure, it promotes fragmentation, but only if the fragmenters don't post pull requests or the authors don't consider them.  Projects with merit but without stewardship are asking to get fragmented anyway.

[Dan Kortschak]

Some time ago I moved my project from using gb to the go tool and wrote
a gofix module for that purpose. This involved inserting a module into
the gofix in $GOROOT. I suspect this is the approach Kyle is suggesting,
though a userland plugin approach would be nice.

Dan

[Kyle Lemons]

On Wed, Dec 12, 2012 at 11:55 PM, Kevin Gillette <extempor...@gmail.com> wrote:

On Wednesday, December 12, 2012 9:37:01 PM UTC-7, Kyle Lemons wrote:

If it's that hard to update B to use Cv2, then either B or C was probably written quite poorly.

That is quite true.  In fact, in the cases in which a package author does need to change their API, it would be really swell of them to provide a gofix module for it, after a suitable period of supporting the backward-compatible interface as well.

Intriguing. A quick search, however, seems to indicate that there's no "userland" support in gofix yet. I suppose limited changes could be bundled in a file of gofmt invocations that could probably be made to be sh/bat/rc compat all at once.

No, gofix itself doesn't support userland (yet? maybe this would be worth trying to design), but the few times I've seen it done they basically copied the gofix source and inserted their own modules and made it a go-gettable binary in their repo.

 

That's one of the reasons I stopped working on rx, actually.  It was conceptualized as a way to identify and track inter-repository dependencies in such a way that it could then pull the updates from a repository and check that nothing depending on it broke, and if it did, play games with (tagged) versions in between to see if one of them still works.  I'll also mention that I planned on having it run tests, as API changes aren't the only thing that could change, and a package author might not even realize that he made a semantic change or that he depended on an undocumented feature/bug in another package.

You've made me realize that I've now danced on both sides of the argument. Certainly with this awareness, I'm now leaning towards the "leverage go's hackability" as an inherent "tool". In something like Java, where you see a lot of copypasta, both library authors and library users may be unwilling to consider or adapt to incompatible changes without automated or batched tooling. In go, those habits may linger, yet I regularly find it's faster to figure out, fix, scrap, and then rewrite large portions of someone else's library than it is to wait for a response to an bug report, for better or worse. Sure, it promotes fragmentation, but only if the fragmenters don't post pull requests or the authors don't consider them.  Projects with merit but without stewardship are asking to get fragmented anyway.

--
 
 

[Guillermo Estrada]

Kudos on the announcement!! If this becomes even half of what RubyGems.org is, I will be thrilled, i want to start using it now, any documentation on how to make/package some nuts? Some people might argue that the best approach is the one being used, because it gives you [bla, bla, bla], but really, the central repository approach with a package management tool is just too good to pass out on any language (npm, pypi, luarocks, rubygems, etc..) and I'm glad someone noticed it and made something about it. I would love to help testing and such, to make release come faster. BTW, consider submitting package information like repository, wiki, homepage, author, bug tracker, group/mailing list, etc... on each nut page (kinda like Rubygems.org does), of all the package managers it has the best design as a service.

It's been a while since I was looking for this, way before the go tool existed. There should be a "like" button on Google Groups, just so I can click on it right now.

[Dave Cheney]

I am curious. In this thread there is a common reference to the idea of a singular central repository for packages being highly desirable.

Personally I find the idea unattractive, a SPoF, overly authoritarian and potentially a political football.

Could someone explain what benefits they see in a central singular namespace for packages?

Dave

--
 
 

[Martin Angers]

Especially since I think it could be done in a decentralized way. Get the packages where they are, use a tag convention or some other means to manage versions, forks are easily gettable (they simply have a different path), so namespacing is automatic, etc. *Maybe* register your package in a central repo just so it knows you exist, but the rest is decentralized (pkgdoc could also be used for discovery, as it becomes more and more the de facto standard). So I'm curious too.

[Peter Kleiweg]

Op maandag 10 december 2012 09:18:56 UTC+1 schreef Dave Cheney het volgende:

Mate, I'm not going to quote urban dictionary to you, but you have to
find another name for a versioned package. Nut is not acceptable. 

You do know this is golang-nuts, right?

[bryanturley]

Only because golang-testicles was already taken?

Though I would agree for different reasons.  No matter the definite of nut I can't see it in a metaphorical relation with source code.

That said, I have seen some
( •_•)>⌐■-■

(⌐■_■)
hairy code in some of these packages...

[Martin Angers]

Gives a whole other meaning to code smell... 

[Kevin Gillette]

Python has eggs. Ruby has gems (read "family jewels"). Go programmers like to examine each others' packages. I'm beginning to see a trend here.

[Kyle Lemons]

On Thu, Dec 13, 2012 at 1:20 PM, Dave Cheney <da...@cheney.net> wrote:

I am curious. In this thread there is a common reference to the idea of a singular central repository for packages being highly desirable.

Personally I find the idea unattractive, a SPoF, overly authoritarian and potentially a political football.

Could someone explain what benefits they see in a central singular namespace for packages?

Do you mean something different from the URI-based namespace we have now?  I like the "singular namespace" as it exists, despite not being particularly keen on a single, canonical, central repository.

 

--
 
 

[Dave Cheney]

> Do you mean something different from the URI-based namespace we have now? I
> like the "singular namespace" as it exists, despite not being particularly
> keen on a single, canonical, central repository.

I don't think of them as URIs, more abstract paths that happen to map
neatly onto various code hosting services. I think the idea of a
single import namespace is more akin to the mount table on my
workstation, than a global URI hierarchy.

[Kamil Kisiel]

Yes this works in theory, but after you work with Python for some time you find out that there are many flaws in the whole pip / PyPI system. For example, not all packages can be installed with pip or setuptools, even prominent ones such as numpy. As such they are not on PyPI, and at that point the whole requirements.txt approach breaks down. The other problem is that until some time this year there were no mirrors for PyPI and as such there would be periods of time during which you wouldn't be able to install any software at all if that's the system you used. The Python community realized this is a problem and now there are mirrors of PyPI and even alternate services such as crate.io that provide the same packages.

Having deal with Python deployments for the last six years I've found Go's package management to be a breath of fresh air. I like the fact that packages come directly from code repositories and that there's no one central location for them. It also means that it's trivial to just clone the repository, host it locally, and change the import paths to point to your own version. That means you can always be using the version of the code you want, and the tools support it without any extra work.

Having a canonical source for packages makes more sense if every installation of your project needs to also install the dependencies, as is the case with the dynamic languages we've been discussing in this thread. Since Go is compiled  and statically linked none of the dependencies are necessary at runtime, the argument for a central repository with canonical sources is a lot less compelling.

On Thursday, December 13, 2012 12:22:33 PM UTC-8, David Pope wrote:

On Thursday, December 13, 2012 12:20:33 PM UTC-6, Dave Cheney wrote:

Could someone explain what benefits they see in a central singular namespace for packages?

This is a very good point.  Most of the discussion here is being done in abstract terms, which is fine to some extent.  But what would be even better would be to talk in terms of requirements, a.k.a. what developer problems are being solved.

To give an example of problems being solved:  I'm currently learning Python, coming from a C# background.  I'm discovering that PyPI and the 'pip' tool solve some real workflow problems for me.  Here's how pip and PyPI work together (apologies to everyone for whom this is well-understood, including possibly the parent poster).

I create a file "requirements.txt" containing (say)

    django

    south

    simplejson

    django-facebook

    urllib3

I run

    pip install -r requirements.txt

to have the latest version of everything I need installed from PyPI, including the listed projects' dependencies (also from PyPI), and start hacking.  I can re-run the install at any point to either refresh my packages or get new ones I've added to requirements.txt.

When it's time to nail down the specific set of libraries I'm using, e.g. to build a machine in a production environment, I run

    pip freeze

and it outputs the precise version of everything I'm currently using, in a form suitable for use as a requirements.txt file:

    Django==1.4.3

    South==0.7.6

    argparse==1.2.1

    distribute==0.6.24

    django-facebook==4.2.5

    simplejson==2.6.2

    urllib3==1.5

    wsgiref==0.1.2

Deploying to production is as simple as loading up my own code and running 'pip install -r' on the above list.

So, here's a list of problems being solved, a.k.a. possible requirements:

- supply well-known unambiguous names to aid discovery, discussion, and blogging

- zero-configuration download

- zero-configuration install into my project

- automatic dependency detection (so I can specify only what matters to me, and not risk getting it wrong for the package I'm bringing in)

- support imprecise versioning (for hacking)

- support precise versioning (for production deployment and/or QA)

- Related:  per-project download and install (isolate my different projects from local dependencies, where possible; this gets into python's virtualenv concept)

The problem of packages relying on different versions of the same dependency doesn't seem to be an issue; it's possible that they've solved the social aspects here, or that they don't emerge as frequently as we might think.  I'm new to Python so maybe I just haven't seen it yet.

I'm so new to Go that I'm nearly useless for analyzing these requirements for their intrinsic value or their difficulty in implementation, but nonetheless I'd be very interested to hear the experts discuss them.

It's entirely possible that the Go ecosystem just isn't ready for this, and that we need to wait for a few "obvious gold standard" projects to emerge before trying to centralize their distribution; for all I know that's how PyPI got its start.  It's also possible that the presence of a "benevolent dictator" in the Python world tends to make decisions like "which project gets to call itself the 'email' project" easier when it becomes an issue.

-- Dave

[Justin Israel]

http://pypi.python.org/pypi/numpy/1.6.2

pip install numpy

pip freeze

...

numpy==1.6.2

...

--
 
 

[Kamil Kisiel]

That does not always work and is not officially supported by numpy because there's no way to pass compilation flags via pip, as you can see there's no mention of them in requirements.txt. Furthermore packages like scipy require numpy to be already installed before they can begin installation, because they require access to numpy's C headers. However pip freeze just lists the package in alphabetical order, so that can fail. There are even more problems but I'm not going to go in to them at length here since it's a bit off topic and has nothing to do with centralized repositories but more to do with Python's tooling. If you're interested, check out some of the packaging related discussions on Python mailing lists to learn more about the problems. 

I maintain a large scientific software platform with nearly 200 Python packages, so I hope you can trust me when I say it's a nightmare :)

The Go tooling and deployment scenario is far better and simpler, even when compared to some of our smaller Python software deployments. For me it's one of Go's biggest advantages.

[Torrance Hodgson]

Hi,

I'm just wondering if someone can provide an update on gonuts.io? Is work still progressing on this, or have are alternatives being looked at instead?

The issue with dependency versioning with Go seems to be a fairly recurrent issue (eg. most recently listed as 'the bad' of Go development by one of the Ubuntu Juju developers).

Cheers,
Torrance

[Gustavo Niemeyer]

On Tue, Mar 5, 2013 at 3:01 AM, Torrance Hodgson <torra...@gmail.com> wrote:
> The issue with dependency versioning with Go seems to be a fairly recurrent
> issue (eg. most recently listed as 'the bad' of Go development by one of the
> Ubuntu Juju developers).

The issues seen within the juju development routine was consciously
introduced because some of the developers did not want to use the
convention of using a version number in the package URL, because they
felt it was too much trouble while they were at a fast pace developing
an external dependency. I disagree.. the convention works well in my
experience with mgo and others, and it can be used in those cases too.

If this is "the bad of Go", I'm honestly curious about what comes
after "the good of ...?".


gustavo @ http://niemeyer.net

[Alexey Palazhchenko]

Hi,

> I'm just wondering if someone can provide an update on gonuts.io? Is work still progressing on this, or have are alternatives being looked at instead?

Sorry for little news and updates. I'm working on it [1] [2]. I'm going to release a first stable version next week, it will include some breaking changes like vendors (usernames, gonuts.io/AlekSi/nut instead of gonuts.io/nut) suggested in this thread. All data at http://gonuts.io will be wiped at this time. It will not include some expected features, but provide a stable ground for their development.

> The issue with dependency versioning with Go seems to be a fairly recurrent issue (eg. most recently listed as 'the bad' of Go development by one of the Ubuntu Juju developers).

I will be very happy if Juju developers use gonuts.io as alternative repository for their packages, but it's up to them.

[1] https://github.com/AlekSi/nut/tree/vendors
[2] https://github.com/AlekSi/gonuts.io/tree/vendors

–-–
Alexey "AlekSi" Palazhchenko

[Jan Mercl]

On Tue, Mar 5, 2013 at 9:29 AM, Alexey Palazhchenko
<alexey.pa...@gmail.com> wrote:

It has been shown so many times why the versioning idea is broken. I
don't have to do it one more time, do I?

The correct solution to the versioning problem is undecidable (non
computable). If you prefer, call it the halting problem ;-)

-j

[Kyle Lemons]

While that seems like a reasonable way to embed metadata in a project, it still doesn't solve the fundamental problem: how do you resolve dependencies?  Given a package that depends indirectly on two versions of a third package, what do you do?

On Sun, Mar 17, 2013 at 10:50 AM, Ian MacLeod <ian.ne...@gmail.com> wrote:

So, I've got this crazy idea, and I'd like to know how crazy it really is.

The current path appears to be to place a json file in each repo to describe the project, have some metadata, and enumerate the package's dependencies.  Very similar to most other languages out there.

Now, one thing that I really enjoy about Go is the "code is configuration" mantra.  Similarly, I feel pretty strongly that documentation is part of your "code".  Smush those two ideas together, and you get:

README-defined project metadata.

With a simple set of conventions, a project readme can contain all the metadata any self respecting package manager ever needs.  Plus, it becomes very visible documentation, helps project authors not to repeat themselves, etc.

A format something along these lines: https://gist.github.com/nevir/5182712

-Ian

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Rory McGuire]

Surely the point of versions is to make it so you can't do that? Not to make it so you can.

If they are versioned and the version is easily visible it can hopefully become obvious why you cannot use a package that depends on a different version of a package that you already depend on.

[Kyle Lemons]

On Mon, Mar 18, 2013 at 12:26 AM, Rory McGuire <rjmc...@gmail.com> wrote:

Surely the point of versions is to make it so you can't do that? Not to make it so you can.

If they are versioned and the version is easily visible it can hopefully become obvious why you cannot use a package that depends on a different version of a package that you already depend on.

There are a number of problems with this.  Which version, precisely, does X.Y.Z refer to?  The first revision in the project history (on which branch?) which included that in the README?  The most recent?  When are these version numbers checked?  Do you also require tags to prevent having to linearly search through the revision history?  What happens when the versions aren't strictly increasing?  If they are only checked when you fetch the repository, it is exceedingly easy for the dependencies listed in your README to diverge from what someone else might get by looking for the same version in a remote repository, or for you to even forget to update it.  If any of these things happen, then it not only makes it possible for you to indirectly depend on two incompatible versions of the same package and put the tools in an impossible position, it makes it unclear precisely when that happened and how to fix it.

[Kevin Gillette]

On Tuesday, March 5, 2013 1:06:53 AM UTC-7, Gustavo Niemeyer wrote:

The issues seen within the juju development routine was consciously
introduced because some of the developers did not want to use the
convention of using a version number in the package URL, because they
felt it was too much trouble while they were at a fast pace developing
an external dependency. I disagree.. the convention works well in my
experience with mgo and others, and it can be used in those cases too.

I'm sure you mean major version (like "v1" and "v2"), not what many would have first come to mind when thinking of a "version", such as "1.3.6".

[Ian MacLeod]

Aha, here we go :)

There's a couple options for dealing with the issues you raise (thankfully, we've got a lot of examples of other languages/package managers dealing with them).  I'd like to push for a very opinionated approach, though:

• Assert that all projects strictly adhere to http://semver.org/
  
• When a project says it depends on version X.Y.Z, it is indicating that it depends on code that was introduced in that version.  This becomes an implicit ">= X.Y.Z && < (X+1).Y.Z" version requirement.
  
• Maybe we allow for more complex requirements expressions, but probably best to start simple.
  

On the identification side, I think it's necessary to support releases for package maintainers, for the same reasons the language started having explicit releases.  IMO it's a minimal burden on package maintainers to tag each release; and one that many are used to coming from other languages.  So:

• Package maintainers tag each release as "X.Y.Z" (or "vX.Y.Z", but the former seems more popular)
  
• Ideally, they tag at the same time they bump the README (I don't think it matters from the packaging point of view - the tag is the authority)
  

Having versioned releases that point to a specific commit is also a huge step forward for supporting things like mirrors, dependency freezers, package repos, and all those other goodies that people will start clamoring for.  Also, being in the business of searching a repo for a particular version smells really bad to me.

Now, when are tags used?  This is what I'm less clear on.  It seems like the Go community enjoys the current wild-west of everyone on tip/master.  One option for go get (or more likely a separate package manager, because this trends towards per-project package management):

1. First, grab tip/master of the referenced package - if the current version (indicated in the README) satisfies the dependency, stop there.
2. Otherwise, go grab the highest version tag that satisfies, and stop there.
3. Nothing satisfies, freak out.

On your unanswered questions:

• "What if dependencies diverge from the README".  Tooling could help here.  The draconian approach would be to refuse to compile a project that is missing a dependency line in the README (and stdlib imports are ignored).  Note that this approach is similar to other languages, but more explicit (Ruby/Python/Node/etc would happily check out and install a package, but users get a load error at runtime).

• "What happens when versions aren't strictly increasing?"  Having tags masks this to the point that it's probably a non-issue, I think.

-Ian

[Ian MacLeod]

Ack, addendum.  This:

This becomes an implicit ">= X.Y.Z && < (X+1).Y.Z" version requirement.

Should read as:

This becomes an implicit ">= X.Y.Z && < (X+1).0.0" version requirement.

-Ian

[Kyle Lemons]

On Mon, Mar 18, 2013 at 8:09 AM, Ian MacLeod <ian.ne...@gmail.com> wrote:

Aha, here we go :)

There's a couple options for dealing with the issues you raise (thankfully, we've got a lot of examples of other languages/package managers dealing with them).  I'd like to push for a very opinionated approach, though:

• Assert that all projects strictly adhere to http://semver.org/
  

• When a project says it depends on version X.Y.Z, it is indicating that it depends on code that was introduced in that version.  This becomes an implicit ">= X.Y.Z && < (X+1).0.0" version requirement.
  

This is precisely where it falls apart.  Unless you assert that you must build from a precise version that unambiguously identifies the state of code in that repository, different people can get different views of your dependencies, some of which may work and some of which may not.  You can "require" that APIs and dependencies not change unless the major version number changes, but the solution will eventually degenerate to requiring that you explicitly demarcate the allowable revision or range of revisions for every dependency.  Building from head, while annoying, is at the very least much easier to understand.

[Ian MacLeod]

Before I try to address your points, let me enumerate what I currently

understand about Go's package management; I feel like I must be missing

something:

1. A Go package is identified by its repository URI; with code being read from

   either `master`, or a branch specific to the current go version.

2. Any time a package maintainer makes a breaking change to their API, their

   only option is to fork the project (to have a new URI).

3. Packages are not locked into a given set of dependencies: imports are

   resolved at `go get` time, from whatever code was just checked out.

4. `go get` will no-op if a package already exists at the target `src` location.

5. `import` statements are effectively a statement indicating that your code

   works when run against a dependency _at the time the code was written_.  They

   do not indicate retroactive support for dependencies.

5a. I.e., if package A starts depending on package B today, but you previously

    checked out package B a week ago, there is no guarantee that package A will

    work with package B.  `go get` will not update package B for you in this

    case, either; you must update it manually.

5b. The "time the code was written" is actually not strictly true.  A package is

    really only supported on whatever revision of a dependency the author had

    checked out in their `$GOPATH` at the time.

6. If you encounter a breaking issue with a dependency, and you need a way to

   revert that package back to a known working version, your options are either

   to: fork that dependency, or tar it up and deploy it manually.  (This is to

   address issues before the maintainer has time to fix them)

Ok, on to your comment, assuming the above points are accurate:

> This is precisely where it falls apart.  Unless you assert that you must build

> from a precise version that unambiguously identifies the state of code in that

> repository, different people can get different views of your dependencies,

> some of which may work and some of which may not.

I'm not sure I understand this (from the way I'm reading it, I don't see any

difference from the current state of package management):

* Different people already get different views of a package's dependencies

  (points 3, 4, 5, 5a, 5b above) based on when they previously checked out a

  package and/or its dependencies.

* There's no guarantee that the dependencies you currently have in your $GOPATH

  will work with a newly `go gotten` package (points 5a, 5b above)

> You can "require" that APIs and dependencies not change unless the major

> version number changes, but the solution will eventually degenerate to

> requiring that you explicitly demarcate the allowable revision or range of

> revisions for every dependency.

From what I understand, Go already does demarcate allowable revisions in an

implicit manner.  A package supports a dependency that...

* ...was checked out in the author's `$GOPATH` at the time that they started

  depending on it. (points 5, 5a, 5b above)

* ...until that package URI is no longer maintained.

This is _almost exactly the same_ as saying you depend on version

`>= X.Y.Z && < (X+1).0.0` if packages were versioned.  "Y.Z" could just as

easily be a timestamp or revision (assuming linear history), and "X" could be

mapped to the repository URI, if you want to think about it like that.

One benefit of having version constraints is that they are expressed as a

guarantee - one that would obviate the issues introduced by 4, 5, 5a, 5b.

The tools would know when to bump a version of a stale package that you have in

your workspace.

> Building from head, while annoying, is at the very least much easier to

> understand.

Hard to diagnose logic issues (introduced by subtle behavior changes across

versions of a dependency), IMO, are worth having slightly more oppressive

tooling to avoid; in addition to all the other benefits that package management

provides (not enumerating here; this post is already far too long).

Finally, package versioning is a pretty common and well understood practice.  I

don't buy that package management as a concept is very confusing

Some of the really insane tooling that other communities have built around it

certainly is, but having a simple packaging system shouldn't be a burden (or,

IMO, it has failed at its job).

-Ian

[Kyle Lemons]

On Mon, Mar 18, 2013 at 7:55 PM, Ian MacLeod <ian.ne...@gmail.com> wrote:

Before I try to address your points, let me enumerate what I currently

understand about Go's package management; I feel like I must be missing

something:

1. A Go package is identified by its repository URI; with code being read from

   either `master`, or a branch specific to the current go version.

2. Any time a package maintainer makes a breaking change to their API, their

   only option is to fork the project (to have a new URI).

No, their other option is to break their clients.  Preferably they should warn them, have sane deprecation policies, and/or provide gofix-like tools to make migration easier, of course.

 

3. Packages are not locked into a given set of dependencies: imports are

   resolved at `go get` time, from whatever code was just checked out.

4. `go get` will no-op if a package already exists at the target `src` location.

No, `go get` can attempt to update existing packages (and their dependencies) as well.

 

5. `import` statements are effectively a statement indicating that your code

   works when run against a dependency _at the time the code was written_.  They

   do not indicate retroactive support for dependencies.

That's attaching a lot of meaning to them that doesn't really exist.  All an import statement does is pull in the symbols that were provided by that package the last time that package was built.  The `go build` tool is the one that provides file modification time checking and automatic dependency building.

 

5a. I.e., if package A starts depending on package B today, but you previously

    checked out package B a week ago, there is no guarantee that package A will

    work with package B.  `go get` will not update package B for you in this

    case, either; you must update it manually.

5b. The "time the code was written" is actually not strictly true.  A package is

    really only supported on whatever revision of a dependency the author had

    checked out in their `$GOPATH` at the time.

6. If you encounter a breaking issue with a dependency, and you need a way to

   revert that package back to a known working version, your options are either

   to: fork that dependency, or tar it up and deploy it manually.  (This is to

   address issues before the maintainer has time to fix them)

Well, no, you could revert the repository back to that version or update your code to the new semantics.

Ok, on to your comment, assuming the above points are accurate:

> This is precisely where it falls apart.  Unless you assert that you must build

> from a precise version that unambiguously identifies the state of code in that

> repository, different people can get different views of your dependencies,

> some of which may work and some of which may not.

I'm not sure I understand this (from the way I'm reading it, I don't see any

difference from the current state of package management):

* Different people already get different views of a package's dependencies

  (points 3, 4, 5, 5a, 5b above) based on when they previously checked out a

  package and/or its dependencies.

* There's no guarantee that the dependencies you currently have in your $GOPATH

  will work with a newly `go gotten` package (points 5a, 5b above)

That is precisely my point.  I don't think that the proposal, in its current form, provides enough benefits to outweigh the added burden, complexity, and possibly confusion that could come along with it.  I have first-hand experience trying to wrangle these issues: I tried to develop a tool to tackle this from a much stricter position, but using tools to identify dependencies and lock down (and publish) working versions of dependencies.  So far, I don't think it is much of a solution either.

 

> You can "require" that APIs and dependencies not change unless the major

> version number changes, but the solution will eventually degenerate to

> requiring that you explicitly demarcate the allowable revision or range of

> revisions for every dependency.

From what I understand, Go already does demarcate allowable revisions in an

implicit manner.  A package supports a dependency that...

* ...was checked out in the author's `$GOPATH` at the time that they started

  depending on it. (points 5, 5a, 5b above)

* ...until that package URI is no longer maintained.

This is _almost exactly the same_ as saying you depend on version

`>= X.Y.Z && < (X+1).0.0` if packages were versioned.  "Y.Z" could just as

easily be a timestamp or revision (assuming linear history), and "X" could be

mapped to the repository URI, if you want to think about it like that.

As I said; I don't think it's appreciably different than the status quo.

 

One benefit of having version constraints is that they are expressed as a

guarantee - one that would obviate the issues introduced by 4, 5, 5a, 5b.

The tools would know when to bump a version of a stale package that you have in

your workspace.

> Building from head, while annoying, is at the very least much easier to

> understand.

Hard to diagnose logic issues (introduced by subtle behavior changes across

versions of a dependency), IMO, are worth having slightly more oppressive

tooling to avoid; in addition to all the other benefits that package management

provides (not enumerating here; this post is already far too long).

Finally, package versioning is a pretty common and well understood practice.  I

don't buy that package management as a concept is very confusing

The problems are well-understood, certainly.  I don't think that there are any solutions that work well, however.

[Ian MacLeod]

Awesome, thanks for the thorough response.  Couple more questions:

> Well, no, you could revert the repository back to that version or update your code to the new semantics.

Are there any tools to help deal with this?  go get doesn't understand scm revisions, does it? (as part of the CLI at least).

How are people dealing with freezing versions and/or dealing with broken dependencies right now?  I can't imagine that waiting for the dependency author before trying to push again is reasonable?

The one thing that I hate the most is having to debug application code (due to some dependency) when rolling out new hosts or doing a hotfix against a known version of my code.

> As I said; I don't think it's appreciably different than the status quo.

Yeah, agree; and you've given a much better idea of the actual issues faced

-Ian

[Kyle Lemons]

On Tue, Mar 19, 2013 at 5:21 PM, Ian MacLeod <ian.ne...@gmail.com> wrote:

Awesome, thanks for the thorough response.  Couple more questions:

> Well, no, you could revert the repository back to that version or update your code to the new semantics.

Are there any tools to help deal with this?  go get doesn't understand scm revisions, does it? (as part of the CLI at least).

How are people dealing with freezing versions and/or dealing with broken dependencies right now?  I can't imagine that waiting for the dependency author before trying to push again is reasonable?

Go get does not help with this, and is deliberately very naive about revisions.  My attempt at an improvement upon this problem (not a full solution, even in concept) did add some knowledge of this.

 

The one thing that I hate the most is having to debug application code (due to some dependency) when rolling out new hosts or doing a hotfix against a known version of my code.

In general, I think people check out the code into their own project if they're worried.  I tend to use few enough third-party libraries in production code, and the ones I do use are from authors I trust to (a) only make breaking changes when it's absolutely necessary and (b) provide nice documentation and examples, if not a full-fledged gofix-like tool, to help in fixing it up.

I also find that a surprising number of Go packages don't seem to have transitive dependencies, and it seems like most that do depend on packages by the same author.  Maybe it's because the language is still young, maybe it's because of this issue, but I tend to think it's because the standard library is full featured enough that you can build straight on top of that to build out a package which can be directly used by the end application without a lot of middle men required.  This is purely anecdotal, however; I bet a more thorough analysis of this could be done with the nifty dependency graphs on godoc.org.

[Alexey Palazhchenko]

Hello,

http://gonuts.io was relaunched last week. As I said in the original announcement, all data was wiped. It will not happen in the future, and there will not be more breaking changes.

There are two major changes in this release.

First, I added vendors (user names, project names) as suggested in this thread. I think gonuts.io/aleksi/syslog is much better then gonuts.io/aleksi-super-duper-syslog. For now, each registered user reserves one vendor name for himself, but code actually supports many-to-many relationship between users and vendors, it just not in UI yet. If you want to register a vendor name for your project, please ask me in gonu...@googlegroups.com list.

Second, version is no longer a part of import path. My original idea was to allow several versions of nut to be in the same GOPATH and rewrite import paths on install. Looking at this mail list and IRC, I figured out that it takes time to "get" GOPATH. My original idea would drive people away from gonuts.io and Go forever. :) So, now there is only one nut per GOPATH, with import path like gonuts.io/vendor/nut. Recommended way is to have one GOPATH per project (think Bundler, virtualenv, npm without -g).

This version also omits dependency resolution for very simple reason – there is nothing to depend on yet. :) Next version will include it.

Also, next version will include ability to lock to specific revisions/commits of all "go get"able packages in GOPATH. This would allow your application to be the same in every environment. But will not allow your library package (not application) to depend on specific revision of other library – for this you will need nuts.

Few days ago there was an interesting discussion between Ian MacLeod and Kyle Lemons which I think crystalizes all arguments for and against versioning and dependency resolution. I would like to respond to this specific Kyle's statement:

> I also find that a surprising number of Go packages don't seem to have transitive dependencies, and it seems like most that do depend on packages by the same author.

I can't talk for everyone in our community, but let me express my fillings and fillings of people who agree with me. My company uses Go in production, we deploy new code several times per day. I feel insecure to depend on latest version of third-party code which can change anytime: anything that can go wrong will go wrong in the middle of deployment. So I fork third-party code. If I make a library package, I will make it depend only on my other packages or forks – for the same reason. And I think this hurts our community. Few years ago I participated in development of Ruby on Rails application which depended on 200+ gems from (our mirror of) RubyGems.org. We did not depend on our forks (it would be hard to maintain 200+ of them) yet we felt secure, and it was easy to update to new version (btw, how often do you see CHANGELOGs in Go packages? there are no releases, so no change logs). By adopting semantic versioning from the start and enforcing it with "go/types" we can do even better.

So, I'm asking you, Go package authors: allow your users to choose between installing your package with "go get" or "nut get". It's trivial to create a nut (or at least I'm trying to make it trivial). It's trivial to publish a new version. Please do it.

Thank you.

–-–
Alexey "AlekSi" Palazhchenko

[Jan Mercl]

On Mon, Mar 25, 2013 at 11:30 AM, Alexey Palazhchenko
<alexey.pa...@gmail.com> wrote:
> So, I'm asking you, Go package authors: allow your users to choose between installing your package with "go get" or "nut get". It's trivial to create a nut (or at least I'm trying to make it trivial). It's trivial to publish a new version. Please do it.

I'll not do it. I don't like the idea at all. Reasons were discussed
before. It's strictly a technical issue, nothing personal - but I'm
sorry anyway.

-j

[Aram Hăvărneanu]

> I'll not do it. I don't like the idea at all. Reasons were discussed
> before. It's strictly a technical issue, nothing personal - but I'm
> sorry anyway.

Likewise.

--
Aram Hăvărneanu

[Gustavo Niemeyer]

I feel sad to agree. It's a significant effort being pushed by a
community member, and I'd like to have good things to say about it and
collaborate towards its success, but I have to say I really like the
"go" tool model. Most languages use a centralized system for such
archives (pypi, cpan, etc), but with Go you can talk straight to the
archive, without such a middle-man entity. That's a well-crafted
feature that had significant thinking on. I'd rather not lose it.


gustavo @ http://niemeyer.net

[Rory McGuire]

Agreed the only thing that is kind of similar is the central documents page which you add your go get repo to and then it keeps itself updated.

imho that would be the model to follow with gonuts.io too. Just check master for changes every day or whatever and if its different automatically increment the gonut version number and

notify anyone that has subscribed interest in it.

A developer can then choose to commit their changes to a branch other than master and whenever they merge bam new version. I think that would be pretty cool. The end user developer could

then run `nut build -u` or something and gonuts would automatically update all dependencies for a project build.

Rory McGuire ClearFormat - Research and Development UK : 44 870 224 0424 USA : 1 877 842 6286 RSA : 27 21 466 9400 Email: rmcg...@clearformat.com Website: www.clearformat.com

--
You received this message because you are subscribed to a topic in the Google Groups "golang-nuts" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/golang-nuts/cyt-xteBjr8/unsubscribe?hl=en-US.
To unsubscribe from this group and all its topics, send an email to golang-nuts...@googlegroups.com.

[Gustavo Niemeyer]

On Mon, Mar 25, 2013 at 11:18 AM, Rory McGuire <rmcg...@clearformat.com> wrote:
> Agreed the only thing that is kind of similar is the central documents
> page which you add your go get repo to and then it keeps itself updated.

Strictly speaking, there's no central documents page. What exists is a
very nice service that fetches code and renders documentation on
demand. It's a consumer, rather than being on the pipeline.


gustavo @ http://niemeyer.net

[Rory McGuire]

Nice. Well that works equally well, then the only thing that would need to be stored would be the version and git revision number.

Rory McGuire ClearFormat - Research and Development UK : 44 870 224 0424 USA : 1 877 842 6286 RSA : 27 21 466 9400 Email: rmcg...@clearformat.com Website: www.clearformat.com

[Matt Joiner]

Cool idea, it's disappointing to see that nobody has tried out your ideas.

I'm having some trouble with this third-party packaging thing at my company now, for which I am strongly advocating the use of Go.

[Elazar Leibovich]

On Wednesday, December 12, 2012 5:29:26 AM UTC+2, Kyle Lemons wrote:

You're describing something similar to what I started, rx : kylelemons.net/go/rx

The problem is what was described above.  If A depends on B v1 and C v2, and B depends on C v1, then no amount of tooling can really help you if you need a feature that's in C v2.

"jar jar links"[0] gives a nice solution. If A depends on B v1 and C v2, it renames all elements in C v2 with "v2" prefix, and rename all references from A to C v2 to use this prefix, then you almost always do not have a problem. Basically you embed C v2 in A, and that's it.

Of course, it could be the case that, say, both library version requires opening the same file, or holding the same port at their init() function, you could also lose performance since B uses a global cache, but most of the time I think this is a decent solution.

I'm not saying it's a good idea for go, I'm just saying that the problem is not exactly totally unsolvable.

[0] https://code.google.com/p/jarjar/ Hey, I didn't come with the name!

[minux]

What if package C contains a global registry (for example, net/http handler
or crypto's hash registry)?

[Jan Mercl]

On Thu, Apr 11, 2013 at 2:06 PM, Elazar Leibovich <ela...@gmail.com> wrote:
> I'm not saying it's a good idea for go, I'm just saying that the problem is
> not exactly totally unsolvable.

No one in this thread (and other threads on the same topic), IIRC,
ever said the problem is totally unsolvable. Actually the problem is
totally solvable! Of course, manually, by careful inspection and
updating by an intelligent human being.

The claim actually always was that it cannot be solved both
automatically and correctly at the same time.
IOW: It cannot work automatically in _all_ cases and it cannot
automatically detect the cases where it doesn't work correctly.
IOW2: It's completely useless.

-j

[Elazar Leibovich]

On Thu, Apr 11, 2013 at 3:11 PM, minux <minu...@gmail.com> wrote:

What if package C contains a global registry (for example, net/http handler
or crypto's hash registry)?

I basically agree, I even mentioned similar problems. But note that in that case it'll be weird if both A and B depend on C's global registry wouldn't it? It's a bad form for a library to change your global net/http.DefaultServerMux under your back, so I guess it would be rare for that to happen in library code.

I don't see why crypto's hash registry won't work. A will register and use whatever it wishes with A_crypto, B will register and use whatever it wishes with B_crypto. You used some memory in vain, but everything should work.

Regarding Jan's comment. I agree (and even wrote an example in my message), there are edge cases where this won't work. I believe however that a solution which works for many cases is sometimes worth implementing, and probably so did some folks at Google thought, hence "jar jar links".

Last comment is, user intervention is not always sufficient. You sometimes can't keep both version around without massive change to the code.

I remind again that I don't think it's a good idea to do that as default in go get, but it's worth to keep this solution in mind.