[RFC] Idea for version-locking: GopherVault

[Mitchell Hashimoto]

Hi Gophers,

I know that version locking, tag referencing, etc. has been an issue brought up on the mailing list _many_ times before. For the purpose of this post, I'll re-explain the problem in the next paragraph, but this post is different in that I propose a solution that requires zero changes to the actual Go tooling and actually requires no external tooling in and of itself.

The problem (skip if you understand): External, non standard library imports are tied to GitHub, BitBucket, etc. repositories. When you run `go get`, it just gets the tip of each repository if it hasn't before. `go get -u` updates from the tip if you already have the repository downloaded. The idea is elegant: master should always be stable, APIs should be stable. In practice, however, this is not the case for a good amount of time as something is being developed, and the uncertainty of change is frightening in many business cases. I'd like a way to lock an import to a specific ref (commit, tag, branch, etc.). 

As stated earlier, this problem has been noted many times before. From what I can tell, there are a camp of people who don't think its a problem, a camp of people who believe it is a problem but don't have a good solution yet, and a far right group who identifies the problem and demands changes to Go itself immediately to handle it. The solution I'll propose will hopefully please (or at least compromise) with all of these groups.

I propose a free, open-source, and sponsored service called GopherVault. Due to the nature of the service, I wanted to ask for input prior to working on something like this. 

GopherVault is simple. You replace your imports that look like this:

    import "github.com/mitchellh/mapstructure"

With ones that look like this:

    import "gophervault.com/8b59089/github.com/mitchellh/mapstructure"

Where the "8b59089" is the ref. That can be a commit hash, a tag name, a branch name, etc.

GopherVault itself uses the HTTP <meta> tag that `go get` supports to redirect to the proper ref in the repository. The elegance in this solution is that it is just _plain old Go_. You don't need an external tool, you don't need to even care about GopherVault. But it is there if you want it.

I work with a lot of companies (primarily because of Vagrant: vagrantup.com), and I've proposed this solution and they're almost completely in favor of it. A few large companies I work with have explicitly said they can't use Go because the dependency situation is too much of a headache. They understand they can fork the repositories themselves, but the management and uncertainty involved is just too much compared to something like Maven. Something like GopherVault, especially if they can run a firewall (on-premise) version, is very attractive. 

Let me address some complaints or feedback I expect to receive:

* SPOF (single point of failure) - It really is no more of an SPOF than GitHub, BitBucket, etc. If you care about availability, you can self-host one. Also, you usually aren't updating (`go get -u`) that often, so the effect of downtime would be minimal.

* Ugly URLs - They're marginally longer than what they were previously. The "meat" of the URL is still the suffix of the URL, just like every other Go dependency. I think this is a fair tradeoff and doesn't add a lot of noise.

* Why don't you have an external file mapping dependencies to refs? - Because it would require tooling (in Go itself or external). I want there to be ZERO dependencies, so that if someone new to Go doesn't even know about GopherVault, they can still use the standard tooling and it'll *just work*. 

* Keeping all my dependencies locked at the same ref will be hard... - `gofmt -r` to the rescue.

* It isn't a problem! - I wish I could agree with this, but I've talked to enough businesses in the area that is most certainly is a point of contention. Its hard to put something in production on a wide-scale if you can't GUARANTEE what dependencies you're going to be getting. I realize to the people who are arguing this point, that that may sound silly, but I promise I'm just echoing what I've heard. I think my solution is a pragmatic solution to this, whether you like it or not.

Now, asking for comments like this prior to building something isn't typically my style. But because Git/Hg/etc. URLs dont support this sort of thing, GopherVault will have to cache these repositories, periodically updating them upstream. This requires a lot of bandwidth. If I bring something like GopheVault to fruition, I don't want it to ever go down, for obvious reasons. So I'm asking for comments on what people think prior to building it, so that I may go out and potentially get sponsorship SOLELY for hosting/bandwidth costs. This is a non-profit, free service to the community. And as I said earlier, it will be open source so people can run their own if they wished.

As for the ability to deliver: I have a reasonably strong background in being able to ship quality software, including open source software. My creds: 

https://github.com/mitchellh (my github)

http://oreilly.com/go/vagrant (my book)

http://mitchellh.com (my blog)

Thanks for your time. I appreciate it.

Best,

Mitchell

[Kamil Kisiel]

One point you haven't addressed yet: what about the imports made within the packages? Say I fetch gophervault.com/8b59089/github.com/mitchellh/mapstructure , but it imports github.com/lib/pq (just an example). Then that dependency will still be pulled from github. Also there is often the situation that gophervault.com/f00ba2/github.com/kisielk/something imports github.com/kisielk/something/sub and the project would be out of sync with itself.

[m...@paperlesspost.com]

I'm a big +1 on this. It:

* Solves the problem if you think you have one

* Doesn't impact you if you don't think you have the problem

* Doesn't require tooling changes.

Also I can vouch for Mitchell. I work at one of the MANY companies who are now more profitable and happy because of Vagrant.

Best,

mrb

[Henrik Johansson]

The first should not be a problem but transitive deps of the same library is not solved by this right?

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Mitchell Hashimoto]

Henrik,

Just as they're not solved with forking to lock. There is nothing lost here versus previous options.

Best,

Mitchell

--
You received this message because you are subscribed to a topic in the Google Groups "golang-nuts" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/golang-nuts/BMZDD6FM-QE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to golang-nuts...@googlegroups.com.

[Maxim Khitrov]

But then the question is what do you actually gain if this only works
for leaf packages?

[Mitchell Hashimoto]

Maxim,

On Wed, Jun 26, 2013 at 9:37 AM, Maxim Khitrov <m...@mxcrypt.com> wrote:

But then the question is what do you actually gain if this only works
for leaf packages?

Its true. :( A true dependency resolver all the way up is the best solution. 

[Mitchell Hashimoto]

Alright, responding to this early: 

I see and agree that transitive dependencies are clearly a huge issue and blocker for this sort of thing. *sigh*. 

The hunt for a decent solution goes on.

Best,

Mitchell

--

[youdont...@gmail.com]

I would still see value in a solution like this. No, it's not a 100% catch-all solution, but most of the projects so far in my experience, deal with only a handful of leafs anyways. The nature of Go, and the nature of the way that we use Go, yields a lot of small, one-off services, and locking in the handful of dependencies being used is really nice, IMHO.

[Robert Melton]

On Wed, Jun 26, 2013 at 12:40 PM, Mitchell Hashimoto <xmi...@gmail.com> wrote:

I see and agree that transitive dependencies are clearly a huge issue and blocker for this sort of thing. *sigh*. 

While, I agree that is a big problem, I wouldn't let it kill your idea.  It still a marginal improvement, and if GopherVault became wildly popular, it might be GopherVaults all the way down.  You might even be able to do some creative things in the middle (some way to freeze deps)?  

This dependency issue will hopefully continue to be discussed because the solutions so far are all various degrees of painful.  The "point and pray" system of just referencing a git repo someone else controls is never going to be OK for a shipping products.  The fork your own copy and only reference it seems to be the current "recommended policy", but even this requires tracking down all transitive dependencies and forking them as well.  

There are various shops (like mine) doing other hideous but functional things.  We don't fork unless we are going to modify.  If we are just using a library, we "go get" it -- then we delete the .git directories that end up under ./src -- so we end up checking in copies of all the libraries we depend on (even if they got put there transitively).  This is obviously ugly, and removes the ability to use "go get -u", but it has some upsides: hermetic checkouts always in the exact same state, no submodules or nonsense to worry about... we just git clone FOO && go install BAR .. works the same every time.  Also, because we never rename them, we still have there library source right in include pathing (github.com/OriginalAuthor/foo), so if we decide to update stuff, we delete it out then redo the go get.  This is the best solution we came up with -- so anyone looking to improve the state of dependencies without making them hopelessly complex gets my vote.  

[Arne Hormann]

Am Mittwoch, 26. Juni 2013 18:40:46 UTC+2 schrieb Mitchell Hashimoto:

Alright, responding to this early: 

I see and agree that transitive dependencies are clearly a huge issue and blocker for this sort of thing. *sigh*. 

The hunt for a decent solution goes on.

Mitchell, you wrote gophervault would have to cache the packages anyway. Would rewriting them be an obstacle? It's just the import statements at the very top of the files...

That's a little more work for the service, but the cached versions are stable, rewriting would only happen on download.

I can imagine rewriting could pose weird licensing issues due to changed source files, but that could probably be dealt with. Either by making it license dependent or by letting package maintainers opt in.

Great idea btw!

[Dobrosław Żybort]

+1 for that, thought about something similar few days ago.
It will be especially useful for custom installs, when original repo disappear we can still use `go get -u` to get last cached version.

One question: do you think it should support caching also `main` packages (tools)?
Maybe it could support caching them only in local installs? (Well, then we can always use it as simple backup for our internal tools/apps)

Best regards,
Dobrosław Żybort

[Dave Cheney]

I'm solidly in the "this is important, but no good solution has
presented itself camp".

Without trivialising the amount of thought and consideration you've
put into this proposal I'm not seeing a lot different from previous
suggestions like gonuts.io and gopkg.io, which, while providing
additional value, all leverage corruptions of the import statement to
insert some sort of version or tag. Credit where credit is due, this
original idea may very well be attributable to Gustavo Niemeyer, ie,
labix.org/v2/mgo.

Regrettably your solution, and the ones that come before it, do not
address the problem of transitive dependencies. If version A of a
logging package and version B and version C can be present at the same
time in by binary, insanity will ensue. Others have noted on previous
threads that this problem gets even more complex when you consider
side effect imports that register http handlers, crypto hashes, image
formats, etc.

> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an

[George Shammas]

On Wed, Jun 26, 2013 at 5:35 PM, Dave Cheney <da...@cheney.net> wrote:

 If version A of a
logging package and version B and version C can be present at the same
time in by binary, insanity will ensue. Others have noted on previous
threads that this problem gets even more complex when you consider
side effect imports that register http handlers, crypto hashes, image
formats, etc.

If some app depends on some package with version greater then one and another of its dependencies depends on the same package with a version less then one, then the package / build system should throw an error saying that and fail.

I am not truly sure I understand the problem and may be over simplifying it in my head. But isn't that how everyone else handles it?

If I try to apt-get something that depends on a library that is too new or too old, and would break an already installed package, apt throws an error saying that. Same with npm, cpan and others.

[Dave Cheney]

This OP wanted to achieve this goal without modifying the go tool. In his proposal different package versions have different import paths so there is no conflict from the POV of the go tool. 

[Daniel Theophanes]

Counter proposal:

For proposal reference:

govaultd: The service hosting the code.

govault: A command line tool on a client machine. Get locally with "go get govault.local/govault"

running "govault rewrite ." would rewrite all dependencies for the current package to the configured govaultd AND it would send the dependencies to govaultd for recursive evaluation and storage.

 (after this you may commit the rewrite "hg comm -m "Version deps."")

running "go get -u ." will then fetch all re-written dependencies for the given service.

running "govault store ." will store the current package in govault. It will output the version url.

It may be noted that "govault rewrite" performs "govault store" recursively on all dependencies.

it is REQUIRED that running "govault store" twice on the same code files will result in the same version url.

govault SHOULD inspect VCS systems to determine the best name. If a branch or tag is available, use it. If no VCS is available, then all non-hidden or ignored files should be hashed and a portion of the hash should be used.

govaultd should introduce a new <meta> tag to declare that it is a versioned repository. That way if dependency is discovered, the govault tool can hit the server to determine if it needs to be re-written or not. This answer can be cached. This allows for multiple govaultds to be used in a single tree. This also allows a tool like godoc.org to act in an intelligent manner to versioned repositories.

OPTIONALLY aliases may be configured for both govault and  go tool.

    sembit/ => dan...@192.168.200.10:1234/

This introduces a new command. I think that is alright.

...

Typically package maintainers would run "govault store" after tagging or branching a release. when releasing documentation they will then use the govault provided url.

govault can also be used to update a package from one dependency version to another dependency version.

"govault update . bitbucket.org/kardianos/service 1.2.0"

The above will update the current package to use the 1.2.0 version of the given bitbucket package. The resulting import line might look like:

import "govault.org/pkg/1.2.0/bitbucket.org/kardianos/service" (perhaps before it was unversioned or used an earlier version.

-Daniel

[Gerard]

Apt-get works very good. Why? For 2 reasons:

1) It works with version numbers

2) The apps/libs have a stable api during the version lifetime (only expanding, no changing). So there are (or should be) no surprises with updates.

Both can be easily applied with go get, just by adding a version number in the path, like 

github.com/myname/v1/myapp or github.com/myname/version.myapp

Maybe it's time for a package versioning best practice page on the go wiki where we can point to.

[Nathan Youngman]

Using the go-import META tag has a lot of promise. Seems like every solution has some challenges though.

I started a new Google Doc http://goo.gl/oqkvu to list some of the existing solutions and problems. Not new solutions yet, as I'm hoping to get a better idea of what people use/want first.

Your help fleshing it out would be greatly appreciated.

Nathan.

[George Shammas]

The problem with package/v1/ is multiple. There are only a few projects that can/will implement this correctly. Your either copying entire code base into a subdirectory of your project or your creating API wrappers around your code base so you can keep the API consistant. If your coping, branches and tags are a much better choice. If your creating API wrappers you are really locked in to never truly being able to make a breaking API change.  Not to mention, the wrappers are calling new code that still can behave differently then what you expect, so it solves nothing.

The only package that I have come across that seems to implement this well is mgo. I don't expect smaller packages to as good a job.

[Gerard]

The only package that I have come across that seems to implement this well is mgo. I don't expect smaller packages to as good a job.

That's exactly why a bit of teaching (with a wiki for instance) would be benefitial. 

I mean, that's how they did it with apt-get. So why shouldn't it work with Go?

[Gerard]

s/should/would/