non-invasive, mostly decentralized versioning of Go packages

[Jason McVetta]

I've been reading with interest the discussion around the relaunch of the gonuts.io versioning system.  The points I've taken away are:

• Not being able to version dependencies is a pain
• The existing 'go get' command is pleasantly simple
• Many people prefer the simplicity of 'go get' over the imperfect package managers used by other languages
• Some people don't like the idea of adding extra metadata files to a package
• People don't want to be forced to version their packages, or to use versioned packages
• People don't want to be forced to use some particular version numbering scheme
• People don't like the idea of relying on a central package repository

These all seem like pretty reasonable points.  So I threw together a little tool I hope will address at least some of these concerns.  It allow versioning of packages - as a web service! - based on git tags.  

Let's say you want to use a tag named 'sometag' from a repo named 'github.com/someuser/somerepo'.  You would just update your import statement to the following, and run 'go get' as normal:

import (
    "gopkg.herokuapp.com/github.com/someuser/somerepo/tag/sometag/somerepo.git"

)

When 'go get' clones what it thinks of as an ordinary repo, the gopkg app will init a new repo, fetch data from github, merge the specified tag into master, and serve up this new repo to the client.  This way we can install a package locked to a particular tag without modifying go get's behavior.

Benefits:

• Non-invasive, works with 'go get'
• No metadata file in packages - just 'git tag'
• Totally optional - if you don't like it, don't use it.  If you use a package that in turn uses it, you'll notice nothing more than 'go get' downloading a few longish package names.
• Mostly decentralized.  No persistent copy of any package, nor any index of packages, is kept.  The gopkg app is centralized if you choose to use it - but it's AGPL and deploys in a single command, so feel free to host your own.  
• Only need to change import statement; all other code remains the same.
• Doesn't care about or attempt to understand version numbering schemes

Drawbacks:

• 'go get' emits a complaint, because it tries (and fails) to connect with git:// before trying (and succeeding with) http://
• Long, slightly repetitive import names
• Can't push changes back to tagged repos (this may be a benefit)

Notes:

• Currently gopkg is a Ruby application based on grack, because there is no Git HTTP server implementation in Go.  
• It only supports github right now, but would not be hard to add other VCS.  
• No effort is made at sane caching - repos are created in  temp space, which is not persistent on heroku.  Does not attempt to clean up repos - but does recreate them if they are >120 seconds old.
• Might have performance issues for very large repos?

If folks like the idea, I'll register gopkg.io and host it there so the import names are shorter.

[Chris Howey]

I really like this idea! But I'm not a version-dependency fan, so I'm not sure it solves their need.

Would this work with mercurial as well, does it have tags similar to git that could support such a scheme?

I really like the decentralized nature of this, I always saw that as a bonus of the current "go get" way. Also, as a lib writer all you'd have do do is start taging, or perhaps not, I suppose you could depend on a "commit" as a version.

[Dmitri Shuralyov]

This is quite a cool solution.

Personally, I think it's a mistake in the long run to ignore versions of dependencies the way Go tries to now. Well, it might be ok of a system outside Go helps out, but it shouldn't be all manual work.

One potential chicken and the egg problem with this is that if you want people to use it, it should be quite reliable and unlikely to disappear soon.

[Kyle Lemons]

Importing a "versioned" path and the real thing (or another versioned path) from different places in a project or dependency tree will result in duplicate inits running, and you can't use values from one with values from the other.  Again, a non-starter for me.

 

Notes:

• Currently gopkg is a Ruby application based on grack, because there is no Git HTTP server implementation in Go.  
• It only supports github right now, but would not be hard to add other VCS.  
• No effort is made at sane caching - repos are created in  temp space, which is not persistent on heroku.  Does not attempt to clean up repos - but does recreate them if they are >120 seconds old.
• Might have performance issues for very large repos?

If folks like the idea, I'll register gopkg.io and host it there so the import names are shorter.

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Dave Cheney]

Kyle makes a good point. I've been wary of solutions that possibly introduce duplicate init sequences via import paths that have version strings in them, but I thought it would be if not manageable, at least explainable. But the problem of having types which have the same name, but come from different packages, and are therefor different has to rule out this class of solution. 

[Jason McVetta]

On Sat, Mar 30, 2013 at 8:07 AM, Chris Howey <how...@gmail.com> wrote:

Would this work with mercurial as well, does it have tags similar to git that could support such a scheme?

Mercurial does have tags, and does have transport over HTTPS, so it should be possible to add support.

 

I really like the decentralized nature of this, I always saw that as a bonus of the current "go get" way. Also, as a lib writer all you'd have do do is start taging, or perhaps not, I suppose you could depend on a "commit" as a version.

I thought about allowing specific revisions, but tentatively decided against it.  It seems like a pretty bad idea to make one's project dependent on a specific commit - just because there's nothing special about that commit, the author in no way said "hey, this is something you should use".  Also, there is no way to fix bugs for people who have a specific commit checked out.  A tag, however, can be moved to a different commit if necessary.

That's what I envision - the tag is the contract between package developer and package user.  I'm not really comfortable with the idea that master need always maintain backward compatibility, and if I want to change that I should fork a whole new version of the package.  That seems kinda burdensome, and also very unlikely to be reliable across a community of hundreds, eventually thousands, of package maintainers.  However if I tag one of my releases as 'asimov' (I'm more fond of alphabetical words than dotted version numbers), I think it's a pretty reasonable for my users to rely on that tag staying compatible, regardless where master goes.  

 

[Jason McVetta]

On Sun, Mar 31, 2013 at 3:44 AM, Kyle Lemons <kev...@google.com> wrote:

Importing a "versioned" path and the real thing (or another versioned path) from different places in a project or dependency tree will result in duplicate inits running, and you can't use values from one with values from the other.  Again, a non-starter for me. 

Can you help me understand the class of problems this could cause?  Typically an init() is used to prepare the internal state of the package.  It's appropriate that two similar inits run if two versions of package are imported - because these two packages are not the same, that's the whole point of versioning them - and so there's no assumption the inits work the same way.  

The only way I can see this being a problem is if a package's init() tries to do stuff outside its own package (e.g. configure some shared resource).  But maybe I am overlooking something?

[Jason McVetta]

On Sat, Mar 30, 2013 at 9:17 AM, Dmitri Shuralyov <shur...@gmail.com> wrote:

One potential chicken and the egg problem with this is that if you want people to use it, it should be quite reliable and unlikely to disappear soon.

One nifty feature of this solution is that gopkg doesn't keep or depend on any data that's not contained in the import path.  Thus any gopkg instance is exactly equivalent to any other instance.  If your company wants to use gopkg, but doesn't want to depend on gopkg.io (which I still need to register), they can just set up their own instance at gopkg.yourcompany.com.  Or if you're a package author, and the gopkg instance you've been depending on disappears from the net, all you need to do is change your import paths to some other gopkg instance.  

[Kamil Kisiel]

Suppose you have a package like pprof which registers an HTTP handler with the default http.ServeMux instance during its init(). If you have two versions of the package they will each register at the same location. Which one actually gets called during runtime is undetermined because the order of init is undetermined.

[Kyle Lemons]

actually, it will panic for multiple registration of the same path.

The same could happen with e.g. multiple versions of a database/sql binding which register themselves in init.

[Rory McGuire]

:) I like the idea, its limits seem to be that a package can't indirectly depend on multiple versions of any package. Sounds like a reasonable limitation.

With the goproj app that is being discussed elsewhere on this forum, you could install the required versions into the project specific gopath.

[Jason McVetta]

On Mon, Apr 1, 2013 at 6:27 PM, Kyle Lemons <kev...@google.com> wrote:

The same could happen with e.g. multiple versions of a database/sql binding which register themselves in init.

One way to avoid this problem when tagging a release is to add the tag name to the registration name.  Let's say I wanted to tag an 'v2' release of a postgresql driver.  In the tagged revision, instead of registering itself with package sql as 'postgres', it would register as 'postgres_v2'.  

In my code I would have to do sql.Open("postgres_v2", dsn) to use the versioned driver.  But I don't really see a problem with that.