Dependencies & vendoring

24,574 views
Skip to first unread message

Brad Fitzpatrick

unread,
Mar 2, 2015, 12:38:02 PM3/2/15
to golang-dev

Gophers,


One of the most frequent questions we’ve received since Go 1 was how to deal with dependencies and their versions. We’ve never recommended any particular answer.


In Google’s internal source tree, we vendor (copy) all our dependencies into our source tree and have at most one copy of any given external library. We have the equivalent of only one GOPATH and rewrite our imports to refer to our vendored copy. For example, Go code inside Google wanting to use “golang.org/x/crypto/openpgp” would instead import it as something like “google/third_party/golang.org/x/crypto/openpgp”. This has worked out very well for us and we get reproducible builds. If we ever want to update openpgp in our vendored directory, we update it, verify that all the affected code in the world still compiles and tests pass (making code changes as necessary), and then check it in. The world compiles and passes at all points in the version control system’s history, and if we bisect in time, we see which version of an external library was in use at any point.


We’re starting to see others in the Go community do the same, with tools like godep and nut.


We think it’s time to start addressing the dependency & vendoring issue, especially before too many conflicting tools arise and fragment best practices in the Go ecosystem, unnecessarily complicating tooling. It would be nice if the community could converge on a standard way to vendor.


Our proposal is that the Go project,


  1. officially recommends vendoring into an “internal” directory with import rewriting (not GOPATH modifications) as the canonical way to pin dependencies.

  2. defines a common config file format for dependencies & vendoring

  3. makes no code changes to cmd/go in Go 1.5. External tools such as “godep” or “nut” will implement 1) and 2). We can reevaluate including such a tool in Go 1.6+.


The important part is that as a community, we all do this the same way, so tooling can mature and interoperate.


In Go 1.5, the “internal” package mechanism introduced in Go 1.4 for the standard library will be extended to all go-gettable packages, so using the “internal” directory as the root of rewritten import paths makes sense (as opposed to “vendor” or “third_party”).


Consider an existing use of vendoring in the Go source tree: $GOROOT/src/cmd/internal currently contains copies of “rsc.io/x86/x86asm” and “rsc.io/arm/armasm” as $GOROOT/src/cmd/internal/rsc.io/x86/x86asm/ and $GOROOT/src/cmd/internal/rsc.io/arm/armasm/, respectively.  When we use those inside the Go tools, however, we import them “cmd/internal/rsc.io/x86/x86asm” and not “rsc.io/x86/x86asm”. (Although this example comes from the Go distribution repo, the effect is the same as a local project using $GOPATH/src/your.project/path instead of $GOROOT/src/cmd.)


We currently maintain those copies by hand. Instead, we want to write a file (filename and syntax to be determined), such as:


     src/cmd/internal/TBDCONFIG.CFG:

             “rsc.io/x86/x86asm” with revision af2970a7819d

             “rsc.io/arm/armasm” with revision 616aea947362


And then your vendoring tool (such as “godep” or “nut”) would read TBDCONFIG.CFG and write out,

     src/cmd/internal/rsc.io/x86/x86asm/*.go

     src/cmd/internal/rsc.io/arm/armasm/*.go


rewriting imports and import comments in these files for the new location.  It may also optionally change your source so any occurrence of


    import “rsc.io/x86/x86asm


becomes


    import “cmd/internal/rsc.io/x86/x86asm


The vendoring tool would be responsible for generating errors on conflicts or missing dependencies.


The thing that we as a community need to figure out is the recommended configuration file format.


We’d prefer something that the Go standard library can already parse easily. That includes XML, JSON, and Go.  Nobody likes XML, so that leaves JSON and Go.


godep already uses JSON, as do Go tools themselves (e.g. “go list -json fmt”), so we’ll probably want something like godep’s JSON format:


{

   "Deps": [

       {

           "ImportPath": "rsc.io/arm/armasm",

           "Rev": "616aea947362"

       },

       {

           "ImportPath": "rsc.io/x86/x86asm",

           "Rev": "af2970a7819d"

       }

   ]

}


We can start with that for discussion.


Note that we have rejected non-vendoring approaches that require modifications to GOPATH or new semantics inside the go command and toolchain. We believe it is important that the solution not require additional effort on the part of all the tools that already understand how to build, analyze, or modify code in the standard GOPATH hierarchy.


Aram Hăvărneanu

unread,
Mar 2, 2015, 12:47:03 PM3/2/15
to Brad Fitzpatrick, golang-dev
One of the nicest things about Go is that all the information required
to build Go programs is in the source code, a.i. in .go files, and not
in any other files (Makefiles, .cfg files, etc). With this proposal,
this would change. We'd have information in non-Go files
(TBDCONFIG.CFG).

--
Aram Hăvărneanu

Brendan Tracey

unread,
Mar 2, 2015, 12:49:12 PM3/2/15
to Brad Fitzpatrick, golang-dev
Who is the we in “Our proposal is that”?

--
You received this message because you are subscribed to the Google Groups "golang-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

cm...@golang.org

unread,
Mar 2, 2015, 12:51:30 PM3/2/15
to golan...@googlegroups.com, brad...@golang.org
As noted in the OP, TBDCONFIG.CFG could be TBDCONFIG.go (or TBDCONFIG.json). I'm not sure if that alleviates the problem of having "other files" but the other files can be other Go files.
 
--
Aram Hăvărneanu

Brad Fitzpatrick

unread,
Mar 2, 2015, 12:56:19 PM3/2/15
to Brendan Tracey, golang-dev
On Mon, Mar 2, 2015 at 9:49 AM, Brendan Tracey <tracey....@gmail.com> wrote:
Who is the we in “Our proposal is that”?

me, Rob, Russ, Andrew, Ian, David Crawshaw, David Symonds, Sameer, Alan Donovan, et al.

Russ Cox

unread,
Mar 2, 2015, 12:56:30 PM3/2/15
to Aram Hăvărneanu, Brad Fitzpatrick, golang-dev
On Mon, Mar 2, 2015 at 12:46 PM, Aram Hăvărneanu <ara...@mgk.ro> wrote:
One of the nicest things about Go is that all the information required
to build Go programs is in the source code, a.i. in .go files, and not
in any other files (Makefiles, .cfg files, etc).

Not really true. cmd/internal/gc has information in a .y file. go generate arranges to write out the files needed to conform to the usual setup, without teaching the usual setup about yacc. The same thing is going on here. The tool (godep, nut, whatever) would use the config file to write the files needed to conform to the usual setup. This config file would only be read by that tool, not by the go command.

Russ

Nathan Youngman

unread,
Mar 2, 2015, 1:37:16 PM3/2/15
to golan...@googlegroups.com, ara...@mgk.ro, brad...@golang.org

It's good to see this discussion happening.

Agreeing on a common file format and file layout below /internal/ would be a huge step forward.

There are some philosophical differences between nut and godep. Is the file generated from current imports or created manually? Does it only contain SHA commits or can it include branches or tags?

I think it's great to have multiple independent implementations (at first), to see what works better in the wild.

Nathan.

Rodrigo Kochenburger

unread,
Mar 2, 2015, 2:10:24 PM3/2/15
to Nathan Youngman, golan...@googlegroups.com, ara...@mgk.ro, brad...@golang.org
I really like the fact we are discussing this as the community has a lot to benefit from a standard approach.

One thing I love about Go is how the workflow flows. It's smooth. How do you guys envision the workflow for: a) adding a new dependency; b) updating a dependency?

Does the developer have to manually update the dependency file (similar to nut) or is the dependency file generated from the code (similar to godep)? In which step would the import path rewrite happen? 

Also, will library packages also have dependencies specified like that or only application/main packages? What happens to the dependencies of dependencies?  For example, my application (github.com/divoxx/app) imports "github.com/foo/foo which in turn imports "github.com/bar/bar", after the rewrite they are both gonna be "github.com/divoxx/app/internal/github.com/foo/foo" and "github.com/divoxx/app/internal/github.com/bar/bar" respectively?



--

Brad Fitzpatrick

unread,
Mar 2, 2015, 2:20:34 PM3/2/15
to Rodrigo Kochenburger, Nathan Youngman, golang-dev, Aram Hăvărneanu
On Mon, Mar 2, 2015 at 11:10 AM, Rodrigo Kochenburger <div...@gmail.com> wrote:
I really like the fact we are discussing this as the community has a lot to benefit from a standard approach.

One thing I love about Go is how the workflow flows. It's smooth. How do you guys envision the workflow for: a) adding a new dependency; b) updating a dependency?

Does the developer have to manually update the dependency file (similar to nut) or is the dependency file generated from the code (similar to godep)? In which step would the import path rewrite happen? 

We're not looking to specify either of those right now.  We only care about standardizing the configuration file format at this time.  (we might say "for Go 1.5" on accident, but this has nothing to do with Go 1.5's release cycle timing... this could happen tomorrow or in 5 months)
 
Also, will library packages also have dependencies specified like that or only application/main packages?

The mechanism should work for libraries too, but I think we'll discourage overuse of it for libraries. We imagine it'll be mostly used for package main.

What happens to the dependencies of dependencies?  For example, my application (github.com/divoxx/app) imports "github.com/foo/foo which in turn imports "github.com/bar/bar", after the rewrite they are both gonna be "github.com/divoxx/app/internal/github.com/foo/foo" and "github.com/divoxx/app/internal/github.com/bar/bar" respectively?

That's up to the tool.  There might also be a mechanism to declare packages which are safe for duplication (no internal state in package-level vars) vs those which are not.  Then the vendoring tool can error if it finds a problem.

Rodrigo Kochenburger

unread,
Mar 2, 2015, 2:28:30 PM3/2/15
to Brad Fitzpatrick, Nathan Youngman, golang-dev, Aram Hăvărneanu
Okay, gotcha. I also think Godeps seems to be the easiest and more comprehensive format out there. I like the fact that it also has the go version specified in it, which AFAIK is not enforced in any way but potentially could.

Brad Fitzpatrick

unread,
Mar 2, 2015, 2:29:36 PM3/2/15
to Rodrigo Kochenburger, Nathan Youngman, golang-dev, Aram Hăvărneanu
On Mon, Mar 2, 2015 at 11:28 AM, Rodrigo Kochenburger <div...@gmail.com> wrote:
Okay, gotcha. I also think Godeps seems to be the easiest and more comprehensive format out there. I like the fact that it also has the go version specified in it, which AFAIK is not enforced in any way but potentially could.

I removed it from the example because it wasn't clear what it even meant.
 

ben.d...@gmail.com

unread,
Mar 2, 2015, 2:38:37 PM3/2/15
to golan...@googlegroups.com
On Monday, March 2, 2015 at 12:38:02 PM UTC-5, Brad Fitzpatrick wrote:

In Google’s internal source tree, we vendor (copy) all our dependencies into our source tree and have at most one copy of any given external library. We have the equivalent of only one GOPATH and rewrite our imports to refer to our vendored copy. For example, Go code inside Google wanting to use “golang.org/x/crypto/openpgp” would instead import it as something like “google/third_party/golang.org/x/crypto/openpgp”. This has worked out very well for us and we get reproducible builds. If we ever want to update openpgp in our vendored directory, we update it, verify that all the affected code in the world still compiles and tests pass (making code changes as necessary), and then check it in. The world compiles and passes at all points in the version control system’s history, and if we bisect in time, we see which version of an external library was in use at any point.


There is a crucial difference between Google's use of import rewriting and how this would be used in the outside world: the third_party directory (which I assume would be renamed to "internal" under this proposal) exists in a scope that spans many projects, so google/gmail, google/plus, and google/selfdrivingcar can all share the same set of vendored packages. Outside of Google, there is no accessible common ancestor directory, so when two packages share a common dependency they must each import their own version of it into their own vendor directory.

For a concrete example, cockroachdb currently depends on etcd, and they both depend on golang.org/x/net/context. Etcd vendors their dependencies and so cockroach uses the unsightly import path "github.com/coreos/etcd/Godeps/_workspace/src/golang.org/x/net/context". If "Godeps/_workspace" were moved under "internal", we couldn't do that and we'd end up with two copies of the library (for net/context I think that would still work since the main thing used from that package is an interface, but it wouldn't work with e.g. glog).

Granted, this situation is a bit unusual since etcd is primarily intended to be an end-user executable rather than a reusable go library, and we intend to break out the parts that are used by cockroachdb (the 'raft' subpackage) into a separate package/repo, but that just moves the problem around. The new raft package would either have to vendor its dependencies and rewrite its imports, or use canonical import paths for everything with no way to control dependency versions even for its own tests. 

Personally, while I see the value of import-rewriting for corporate environments where you have one large meta-project, I think the better solution for the open-source world is to avoid rewriting imports and instead improve tooling support for a per-project GOPATH (I now let the emacs package go-projectile manage my GOPATH, which works very well for my workflow) and to use something like gpm or goop to pin dependency versions.

-Ben

Axel Wagner

unread,
Mar 2, 2015, 3:44:53 PM3/2/15
to Brad Fitzpatrick, golang-dev
Hi,

the gist: I strongly dislike vendoring because
1 You end up with more than one copy of the same code in your binary
(unless there is some magic I am not aware of possible)
2 It discourages contribution to upstream (because it is quicker to fix
it in the vendored copy, right?)
3 As a consequence it creates diverging versions of the same library
4 And possibly licensing issues (as observed with ruby)
5 It puts a lot of burden unto package maintainers, as distributions
normally disallow vendoring (for security- and maintability concerns)
At the very least I would want to discourage vendoring for the creators
of libraries.

The rant (I will probably not say a lot more about this, but I at least
wanted to put my concerns out there):

I am a big non-fan of vendoring. My main question with this proposal is:
What happens, when two different packages vendor the same dependency?
Will we end up with two copies of the dependency? Will that be a
problem? I could see problems both in codesize and with stuff like
sql/driver, where you would effectively end up with two disjunct sources
for drivers, if the package gets duplicated.

I dislike the comparison with Google's mechanism. A big difference is,
that nobody imports googles internal packages, so the above problem (or
any below) doesn't arrive. Google's approach works, because it is
basically the same as the approach in a distribution:
third_party/foo/bar corresponds to a package in a distribution with
distribution-specific patches. As you have one codebase with one
controlling authority, you can do global updates and more or less atomic
changes over all of the codebase. This stops working with an
open-source ecosystem, where you have in part little to no control over
upstreams. Instead of updating a library on my system and maybe fixing
up dependencies, if there are build-failures, I now have to wait for all
upstreams to update their vendored versions (or in turn have to vendor
*them* and take over some maintainership responsibilities).

Vendoring is the solution the ruby-community has chosen, apparently, and
at least as far as I can tell, that is the main reason, why jekyll in
debian has been broken for year(s?). At least I gave up trying to
package it after having to figure out bugs in all the transitive
vendored dependencies and having to deal with the licensing jungle
(because once stuff gets vendored anyway, you can just add random files
right? Without regard for what is licensed how. Or even annotating what
files are from where).

I think vendoring is the wrong solution for reproducible
builds. Reproducible build only requires all the version information of
all transitive dependencies used to build a binary plus the version of
the go toolchain and stdlib used (btw: Does godep or nut address that?
Just out of curiosity).

Therefore I feel that reproducible builds are a red herring, when
advocating for vendoring. I think the main reason, why vendoring became
popular is, that it makes it easier to deal with API-incompatibilities
of upstream and to make software go-gettable. But I think this is a bad
optimization goal. Again, gem/pip/npm are things chosen by other
languages and I at least will try everything I can *not* to install
something that depends on any of those. I prefer clean releases and
*one* Package manager. Not everyone wants to install a go toolchain,
just to install one tool, just as I don't want to install a node.js
toolchain just to use the keybase.io CLI :)

I fear, that with library, we are ultimately giving up on any
API-stability, because we don't even have to deal with an upstream that
changes their API all the time, we can just vendor and ship with a
frozen-in-time version…

minux

unread,
Mar 2, 2015, 3:51:14 PM3/2/15
to Russ Cox, Aram Hăvărneanu, Brad Fitzpatrick, golang-dev
Then this view suggests indirectly that the go tool will never gain the ability to use such file
to import new revisions?

Keith Rarick

unread,
Mar 2, 2015, 3:52:24 PM3/2/15
to Brad Fitzpatrick, golang-dev
This proposal sounds good to me.
Especially using "internal"; it makes sense.

I'll make any changes necessary in godep to
work with the format and file paths we agree on.

I don't care about the details of the config file
format. I just want it to be possible to generate
it automatically from scratch from an existing
set of go source code and dependency code,
if the user so desires.

Keith Rarick

unread,
Mar 2, 2015, 3:56:48 PM3/2/15
to Brad Fitzpatrick, Rodrigo Kochenburger, Nathan Youngman, golang-dev, Aram Hăvărneanu
>> Also, will library packages also have dependencies specified like that or
>> only application/main packages?
>
> The mechanism should work for libraries too, but I think we'll discourage
> overuse of it for libraries. We imagine it'll be mostly used for package
> main.

I support discouraging this for libraries.

If a library P does this with its dependency D,
then someone who wants to use both P and D
is required to also use a vendoring tool (unless
it happens to be ok to link in two copies of D).
That is annoying, especially for a person or
project that's just getting started.

It's much nicer to be able to 'go get P D' and
import P and D and get to work.

Keith Rarick

unread,
Mar 2, 2015, 4:04:52 PM3/2/15
to Axel Wagner, Brad Fitzpatrick, golang-dev
On Mon, Mar 2, 2015 at 12:10 PM, Axel Wagner
<axel.wa...@googlemail.com> wrote:
> Reproducible build only requires all the version information of
> all transitive dependencies used to build a binary plus the version of
> the go toolchain and stdlib used

Unfortunately, experience shows this is not true.

A reproducible build requires the source code needed to make
the build. With vendoring, you have the source code. Without it,
you need not only the version information but also a *means* to
acquire the source code—a network that is functional and fast
enough. You might be surprised how often that requirement is
not met.

> (btw: Does godep or nut address that?
> Just out of curiosity).

Godep records the output of "go version" when it generates
the file Godeps.json.

Nathan Youngman

unread,
Mar 2, 2015, 4:23:47 PM3/2/15
to Keith Rarick, Brad Fitzpatrick, Rodrigo Kochenburger, golang-dev, Aram Hăvărneanu

Libraries that depend on libraries other than the standard library do complicate things.

I've had good success with shallow dependencies:

main (Godep) -> library (gopkg.in or go get) -> standard library


So while the directory structure and revisions file may support vendoring within libraries, I would also discourage it.

Nathan.

--
Nathan Youngman 
Email: he...@nathany.com
Web: http://www.nathany.com

Sébastien Douche

unread,
Mar 2, 2015, 4:41:31 PM3/2/15
to golan...@googlegroups.com
On Mon, 2 Mar 2015, at 18:46, Aram Hăvărneanu wrote:
> One of the nicest things about Go is that all the information required
> to build Go programs is in the source code, a.i. in .go files, and not
> in any other files (Makefiles, .cfg files, etc).

Hmm, I would say "the nicest thing about Go is you don't need external
tool to build Go programs".


--
Sébastien Douche <s...@nmeos.net>
Twitter: @sdouche
http://douche.name

Owen Ou

unread,
Mar 2, 2015, 4:46:50 PM3/2/15
to golan...@googlegroups.com

It's good to see such discussion going on. I'm the author of nut so my comments might be bias :)

It sounds like there're two things nut does differently from Brad's suggestions:

  1. the format of the config file
  2. the file structure for vendored dependencies

For 1), I'm open to suggestions as long as the format is clear and concise. Currently nut adopts Tomlas the config format. An example of declaring depnednecies:

[dependencies]

"rsc.io/arm/armasm" = "616aea947362"
"rsc.io/x86/x86asm" = "af2970a7819d"

I personally think this is very clear and less noisy than JSON. But I understand the preference of using something Go standard library can parse. It wouldn't be hard for nut to support JSON as the config file. But I think the proposed JSON schema is a bit verbose: the keys of "ImportPath" and "Rev" seem unecessary. How about removing the keys?

{
    "Deps": {
        "rsc.io/arm/armasm": "616aea947362",
        "rsc.io/x86/x86asm": "af2970a7819d"
    }
}

For 2), I don't have a problem of renaming "vendor" as in nut to "internal": making vendored dependencies only accessible to current package makes sense.

Brendan Tracey

unread,
Mar 2, 2015, 4:52:54 PM3/2/15
to Brad Fitzpatrick, Rodrigo Kochenburger, Nathan Youngman, golang-dev, Aram Hăvărneanu

Also, will library packages also have dependencies specified like that or only application/main packages?

The mechanism should work for libraries too, but I think we'll discourage overuse of it for libraries. We imagine it'll be mostly used for package main.

What is the suggested behavior for libraries? I must be missing something, because this mentality does not seem in the spirit of Go. Go is excellent at supporting programming in the large. As programs get richer and as the ecosystem grows, there will inevitably be a hierarchy of libraries. The standard library will only implement so many ideas. I don’t intend to be critical, I just don’t see the intended vision.

Andrew Gerrand

unread,
Mar 2, 2015, 4:54:11 PM3/2/15
to ben.d...@gmail.com, golang-dev
On 3 March 2015 at 06:34, <ben.d...@gmail.com> wrote:
Granted, this situation is a bit unusual since etcd is primarily intended to be an end-user executable rather than a reusable go library, and we intend to break out the parts that are used by cockroachdb (the 'raft' subpackage) into a separate package/repo, but that just moves the problem around. The new raft package would either have to vendor its dependencies and rewrite its imports, or use canonical import paths for everything with no way to control dependency versions even for its own tests. 

Vendoring should be used only for the dependencies of binaries, not for the dependencies of packages. This particular problem could be solved by:
a) a policy of API stability adhered to by the raft subpackage and its dependent packages, 
b) testing infrastructure that is version-aware.

Personally, while I see the value of import-rewriting for corporate environments where you have one large meta-project, I think the better solution for the open-source world is to avoid rewriting imports and instead improve tooling support for a per-project GOPATH (I now let the emacs package go-projectile manage my GOPATH, which works very well for my workflow) and to use something like gpm or goop to pin dependency versions.

I think we can develop the tools to make it easier to test packages with different versions of their dependencies, and to make it easier for the ultimate consumer (a program binary) to vendor the correct dependencies (or provide diagnostics in the rare case of incompatible dependenct versions). Those tools may be [based on] projects like gpm or goop.

Right now I'm working on an API stability policy similar to the Go 1 compatibility promise for the gokit project and its dependencies. If that goes well, maybe the greater Go community can adopt the policy for their projects.

Andrew

Andrew Gerrand

unread,
Mar 2, 2015, 4:56:01 PM3/2/15
to Aram Hăvărneanu, Brad Fitzpatrick, golang-dev
This property doesn't change; you won't need non-Go files to *build* Go programs. You'll just need the metadata to update a program's dependencies, and that necessary metadata is not currently encoded in Go source code anywhere.

Andrew Gerrand

unread,
Mar 2, 2015, 4:58:02 PM3/2/15
to Owen Ou, golang-dev

On 3 March 2015 at 08:46, Owen Ou <jing...@gmail.com> wrote:
I personally think this is very clear and less noisy than JSON. But I understand the preference of using something Go standard library can parse. It wouldn't be hard for nut to support JSON as the config file. But I think the proposed JSON schema is a bit verbose: the keys of "ImportPath" and "Rev" seem unecessary. How about removing the keys?

One advantage to Brad's proposed syntax is that we can add extra fields (if need be) while maintaining backward compatibility with older tools.

 

Burcu Dogan

unread,
Mar 2, 2015, 5:16:26 PM3/2/15
to Andrew Gerrand, ben.d...@gmail.com, golang-dev
> Vendoring should be used only for the dependencies of binaries, not for the dependencies of packages.

Could you clarify what this exactly means? How am I supposed to vendor
a revision of package x, if that revision doesn't know which revision
of package y to depend on? In any case, a revision has to know about
the revision it is depending on.

And if the packages need to vendor, there is a critical problem with
this proposal.

> officially recommends vendoring into an “internal” directory with import rewriting (not GOPATH modifications) as the canonical way to pin dependencies.

How am I supposed to export a symbol from a vendored package?

package a

type Doer interface


package b

func X(doer a.Doer)


a's import path will be visible to package b only.

Burcu Dogan

unread,
Mar 2, 2015, 5:18:37 PM3/2/15
to Andrew Gerrand, ben.d...@gmail.com, golang-dev
Ah, Gmail quoted and wrapped the bottom half of my email. Resending that part.

And if the packages need to vendor, there is a critical problem with
this proposal.

How am I supposed to export a symbol from a vendored package?

package a

type Doer interface


package b

func X(doer a.Doer)


a's import path will be visible to package b only.

Owen Ou

unread,
Mar 2, 2015, 5:23:23 PM3/2/15
to golan...@googlegroups.com, jing...@gmail.com

On Monday, March 2, 2015 at 1:58:02 PM UTC-8, Andrew Gerrand wrote:

One advantage to Brad's proposed syntax is that we can add extra fields (if need be) while maintaining backward compatibility with older tools.


Ben Darnell

unread,
Mar 2, 2015, 5:24:41 PM3/2/15
to Andrew Gerrand, golang-dev
On Mon, Mar 2, 2015 at 4:53 PM, Andrew Gerrand <a...@golang.org> wrote:

On 3 March 2015 at 06:34, <ben.d...@gmail.com> wrote:
Granted, this situation is a bit unusual since etcd is primarily intended to be an end-user executable rather than a reusable go library, and we intend to break out the parts that are used by cockroachdb (the 'raft' subpackage) into a separate package/repo, but that just moves the problem around. The new raft package would either have to vendor its dependencies and rewrite its imports, or use canonical import paths for everything with no way to control dependency versions even for its own tests. 

Vendoring should be used only for the dependencies of binaries, not for the dependencies of packages. This particular problem could be solved by:
a) a policy of API stability adhered to by the raft subpackage and its dependent packages, 

The raft package was developed in tandem with the rest of etcd; it is only now reaching a point of API stability (after validating its interfaces with usage in two applications). In a world of rewritten imports (where at least some dependencies are like glog and must not be duplicated), reusing a piece of an application is a significant burden. Before we could even begin to use raft in cockroachdb, we would have had to move the master copy of the raft code out of etcd (into a new repo which would be developed under "package rules" instead of "binary rules") and vendor it back in to etcd. (we are in fact undertaking this work now, but only after the experiment has proven successful) I don't like the strict division of packages into "those which are so closely tied to a particular binary that they use its rewritten imports" and "those which use 'go get' for their own dependencies but should be vendored into any application that uses them"

 
b) testing infrastructure that is version-aware.

But if we had this version-aware infrastructure (which I admit is a harder problem), would we still want to rewrite imports? I claim that for purposes of reproducible builds it's better to place the entire GOPATH under version control (as long as you can have per-project GOPATHs) than to use an internal/vendor directory to ensure that all your dependencies share a common prefix deep in the source tree.
 

 

Personally, while I see the value of import-rewriting for corporate environments where you have one large meta-project, I think the better solution for the open-source world is to avoid rewriting imports and instead improve tooling support for a per-project GOPATH (I now let the emacs package go-projectile manage my GOPATH, which works very well for my workflow) and to use something like gpm or goop to pin dependency versions.

I think we can develop the tools to make it easier to test packages with different versions of their dependencies, and to make it easier for the ultimate consumer (a program binary) to vendor the correct dependencies (or provide diagnostics in the rare case of incompatible dependenct versions). Those tools may be [based on] projects like gpm or goop.

Yes. I agree that the needs of library developers are different from the needs of application developers. Libraries must be as broad as possible in their dependencies to minimize conflicts (hopefully just a minimum version; occasionally a maximum or range) while applications want to pin things down exactly. I'd just like for this difference to be as localized as possible (e.g. "use == instead of >= in your dependencies.json file") instead of completely changing the workflow.

-Ben

zel...@gmail.com

unread,
Mar 2, 2015, 5:49:03 PM3/2/15
to golan...@googlegroups.com, jing...@gmail.com
On Monday, March 2, 2015 at 1:58:02 PM UTC-8, Andrew Gerrand wrote:
On 3 March 2015 at 08:46, Owen Ou <jing...@gmail.com> wrote:
[...]I think the proposed JSON schema is a bit verbose: the keys of "ImportPath" and "Rev" seem unecessary. How about removing the keys?

One advantage to Brad's proposed syntax is that we can add extra fields (if need be) while maintaining backward compatibility with older tools.

Agreed. As an example, I could see particular tools adopting conventions for noting which (semver.org -style) version of dependencies you were aiming for, and trying to merge shared dependencies.
Like Brad's "safe for duplication declaration" mentioned above, this would require some kind of metadata convention too, so I'm glad we're not trying here to decide what color to paint that bikeshed.

Zellyn

Andrew Gerrand

unread,
Mar 2, 2015, 6:36:28 PM3/2/15
to Ben Darnell, golang-dev
On 3 March 2015 at 09:24, Ben Darnell <ben.d...@gmail.com> wrote:
On Mon, Mar 2, 2015 at 4:53 PM, Andrew Gerrand <a...@golang.org> wrote:

On 3 March 2015 at 06:34, <ben.d...@gmail.com> wrote:
Granted, this situation is a bit unusual since etcd is primarily intended to be an end-user executable rather than a reusable go library, and we intend to break out the parts that are used by cockroachdb (the 'raft' subpackage) into a separate package/repo, but that just moves the problem around. The new raft package would either have to vendor its dependencies and rewrite its imports, or use canonical import paths for everything with no way to control dependency versions even for its own tests. 

Vendoring should be used only for the dependencies of binaries, not for the dependencies of packages. This particular problem could be solved by:
a) a policy of API stability adhered to by the raft subpackage and its dependent packages, 

The raft package was developed in tandem with the rest of etcd; it is only now reaching a point of API stability (after validating its interfaces with usage in two applications). In a world of rewritten imports (where at least some dependencies are like glog and must not be duplicated), reusing a piece of an application is a significant burden. Before we could even begin to use raft in cockroachdb, we would have had to move the master copy of the raft code out of etcd (into a new repo which would be developed under "package rules" instead of "binary rules") and vendor it back in to etcd. (we are in fact undertaking this work now, but only after the experiment has proven successful) I don't like the strict division of packages into "those which are so closely tied to a particular binary that they use its rewritten imports" and "those which use 'go get' for their own dependencies but should be vendored into any application that uses them"

I think it's reasonable to expect some difficulty when re-using the internal packages of larger projects. Until those packages are officially supported by their authors, you're kind of in "here be dragons" territory. I don't think this proposal makes such issues better or worse. 
  
b) testing infrastructure that is version-aware.

But if we had this version-aware infrastructure (which I admit is a harder problem), would we still want to rewrite imports? I claim that for purposes of reproducible builds it's better to place the entire GOPATH under version control (as long as you can have per-project GOPATHs) than to use an internal/vendor directory to ensure that all your dependencies share a common prefix deep in the source tree.

There's an argument for keeping all the various mechanical pieces small and simple. It's definitely possible to imagine some larger infrastructure that manages everything for us, from end to end, but that's not the direction we have taken so far.

 

Personally, while I see the value of import-rewriting for corporate environments where you have one large meta-project, I think the better solution for the open-source world is to avoid rewriting imports and instead improve tooling support for a per-project GOPATH (I now let the emacs package go-projectile manage my GOPATH, which works very well for my workflow) and to use something like gpm or goop to pin dependency versions.

I think we can develop the tools to make it easier to test packages with different versions of their dependencies, and to make it easier for the ultimate consumer (a program binary) to vendor the correct dependencies (or provide diagnostics in the rare case of incompatible dependenct versions). Those tools may be [based on] projects like gpm or goop.

Yes. I agree that the needs of library developers are different from the needs of application developers. Libraries must be as broad as possible in their dependencies to minimize conflicts (hopefully just a minimum version; occasionally a maximum or range) while applications want to pin things down exactly. I'd just like for this difference to be as localized as possible (e.g. "use == instead of >= in your dependencies.json file") instead of completely changing the workflow.

What I'm talking about is codifying our existing workflows and formal documentation of the stability of packages, not changing workflows entirely.

Andrew Gerrand

unread,
Mar 2, 2015, 6:38:45 PM3/2/15
to Burcu Dogan, Ben Darnell, golang-dev
On 3 March 2015 at 09:16, Burcu Dogan <j...@google.com> wrote:
> Vendoring should be used only for the dependencies of binaries, not for the dependencies of packages.

Could you clarify what this exactly means? How am I supposed to vendor
a revision of package x, if that revision doesn't know which revision
of package y to depend on? In any case, a revision has to know about
the revision it is depending on.

And if the packages need to vendor, there is a critical problem with
this proposal.

I'm saying the maintainers of the binary should vendor all of its transitive dependencies, if they vendor at all.
 
The package maintainers should not vendor anything.

> officially recommends vendoring into an “internal” directory with import rewriting (not GOPATH modifications) as the canonical way to pin dependencies.

How am I supposed to export a symbol from a vendored package?

package a

type Doer interface


package b

func X(doer a.Doer)


a's import path will be visible to package b only.

I don't understand the question. As maintainer of the code en masse you have visibility into all import paths. Maybe my answer to your first question helps?

Keith Rarick

unread,
Mar 2, 2015, 6:46:50 PM3/2/15
to Andrew Gerrand, Owen Ou, golang-dev
For example: if the program has vendored a patch to one
of its dependencies and the patch is not yet merged upstream,
the author might want their tool to include an alternative URL
where the patch has been published.

Or: godep currently includes the output of "git describe --tags"
(or a similarly descriptive command for hg and bzr) for each
dependency, for the benefit of any human who is reading the file.

Russ Cox

unread,
Mar 2, 2015, 6:47:41 PM3/2/15
to Andrew Gerrand, Burcu Dogan, Ben Darnell, golang-dev
People seem to agree that libraries should not vendor other libraries without a good reason. That's actually beyond the scope here. There is some question about what happens if libraries *do* vendor other libraries, and that gets to the heart of the proposal. 

If there is an agreed-upon vendoring approach (import path rewriting) and data format that describes what vendoring did happen, then no matter what vendoring helper did it, another tool (or perhaps the same one) can come along and analyze the situation and either just fix it or help the programmer fix it with minimal interactions.

On the other hand, if the vendoring tools use different semantics or even just different config files, this kind of metatool becomes much more difficult.

The goal here is to (1) agree on import path rewriting (as opposed to dynamic GOPATH tweaking or other complications), and (2) agree on a config file format that records the rewriting that happened, so that different projects can use different tools and still interoperate, both for just building things and for using meta tools that provide things like deduplication and diamond crushing.

Russ

Ben Darnell

unread,
Mar 2, 2015, 6:59:26 PM3/2/15
to Andrew Gerrand, golang-dev
On Mon, Mar 2, 2015 at 6:35 PM, Andrew Gerrand <a...@golang.org> wrote:



The raft package was developed in tandem with the rest of etcd; it is only now reaching a point of API stability (after validating its interfaces with usage in two applications). In a world of rewritten imports (where at least some dependencies are like glog and must not be duplicated), reusing a piece of an application is a significant burden. Before we could even begin to use raft in cockroachdb, we would have had to move the master copy of the raft code out of etcd (into a new repo which would be developed under "package rules" instead of "binary rules") and vendor it back in to etcd. (we are in fact undertaking this work now, but only after the experiment has proven successful) I don't like the strict division of packages into "those which are so closely tied to a particular binary that they use its rewritten imports" and "those which use 'go get' for their own dependencies but should be vendored into any application that uses them"

I think it's reasonable to expect some difficulty when re-using the internal packages of larger projects. Until those packages are officially supported by their authors, you're kind of in "here be dragons" territory. I don't think this proposal makes such issues better or worse. 

This proposal does make some such issues a bit worse by requiring (or at least encouraging) the use of an "internal" namespace for vendored packages. Currently we can experiment in "here be dragons" territory by importing another application's vendored packages; that becomes more difficult (at least in some cases) if such packages become internal. But your point is taken; in the absence of a specific counter-proposal it is probably better to move forward with increased standardization in this area since vendoring with rewritten imports does solve at least some of the versioning problem.

-Ben

Burcu Dogan

unread,
Mar 2, 2015, 7:59:45 PM3/2/15
to Andrew Gerrand, Ben Darnell, golang-dev
> Maybe my answer to your first question helps?

Yes, the first answer clarifies both. The second question arises only
if the packages have to vendor their own dependencies.

(IMHO, vendoring libraries should be considered in the scope of this
proposal not to break whatever standardization may come out of this
proposal. What's your argument against the libs vendoring or blessing
the revisions of their dependencies? We need to develop some
ideas/conventions around the API compatibility and versioning before
tackling a configuration schema problem -- especially in a world where
a revision hash hardly ever can represent an API version.)

Kamil Kisiel

unread,
Mar 2, 2015, 8:26:05 PM3/2/15
to golan...@googlegroups.com, brad...@golang.org
I like the idea of having it be a .go file. It could be required to have a specific name, and would import a package which has an exported struct type used to describe a dependency. Over time when the requirements evolve, more fields could be added to the struct type. The syntax is familiar and we get strict checking for free via the compiler.

eg:

// TDBCONFIG.go
import "deps"

var Dependencies = []deps.D{
    {Path: "rsc.io/arm/armasm", Rev: "616aea947362"},
}


The dependency tool could either compile it or read the information using the ast package.

On Monday, March 2, 2015 at 9:51:30 AM UTC-8, cm...@golang.org wrote:


On Monday, March 2, 2015 at 9:47:03 AM UTC-8, Aram Hăvărneanu wrote:
One of the nicest things about Go is that all the information required
to build Go programs is in the source code, a.i. in .go files, and not
in any other files (Makefiles, .cfg files, etc). With this proposal,
this would change. We'd have information in non-Go files
(TBDCONFIG.CFG).


As noted in the OP, TBDCONFIG.CFG could be TBDCONFIG.go (or TBDCONFIG.json). I'm not sure if that alleviates the problem of having "other files" but the other files can be other Go files.
 
--
Aram Hăvărneanu

emi...@gmail.com

unread,
Mar 2, 2015, 11:14:01 PM3/2/15
to golan...@googlegroups.com
I would like to suggest an "alias" parameter as well, inside the configuration format. This would allow us to specify the folder that we would like the given dependency to be written out to, therefore also allowing us to simultaneously vendor multiple versions of one package.

For example :

{
  "Deps": [
    {
      "Alias": "armasm",
      "ImportPath": "rsc.io/arm/armasm",
      "Rev": "616aea947362"
    },
    {
      "Alias": "x86asm_1",
      "ImportPath": "rsc.io/x86/x86asm",
      "Rev": "af2970a7819d"
    },
    {
      "Alias": "x86asm_2",
      "ImportPath": "rsc.io/x86/x86asm",
      "Rev": "7e021db6589f"
    }
  ]
}


These would be written out to :

internal/armasm/
internal/x86asm_1/
internal/x86asm_2/



jul...@soundcloud.com

unread,
Mar 2, 2015, 11:14:15 PM3/2/15
to golan...@googlegroups.com, ben.d...@gmail.com
On Monday, March 2, 2015 at 10:54:11 PM UTC+1, Andrew Gerrand wrote:

On 3 March 2015 at 06:34, <ben.d...@gmail.com> wrote:
Granted, this situation is a bit unusual since etcd is primarily intended to be an end-user executable rather than a reusable go library, and we intend to break out the parts that are used by cockroachdb (the 'raft' subpackage) into a separate package/repo, but that just moves the problem around. The new raft package would either have to vendor its dependencies and rewrite its imports, or use canonical import paths for everything with no way to control dependency versions even for its own tests. 

Vendoring should be used only for the dependencies of binaries, not for the dependencies of packages.

So if I have a repo that has both libraries and binaries, should it be split up? Or should any binaries in it vendor the library parts of the same repo?

Actually we recently had the problem where binary B used library L (different repos), and L had a method which accepted protobufs defined in a third repo P. Since L vendored P (now P'), it was impossible for B to pass in protobuf types from P to L. So we had to un-vendor everything in L, which is all good according to your suggestion to not vendor dependencies for libraries. But the library repo also contains some sample binaries. Should these live somewhere else or simply not use vendoring?

Julius

michae...@gmail.com

unread,
Mar 2, 2015, 11:14:21 PM3/2/15
to golan...@googlegroups.com, brad...@golang.org
I am not sure what you are aiming for when saving the output requiring dependencies to the Dependencies variable. In the original post versioned dependencies have their own path in the vendored folder so implementing an entire new stdlib seems like overkill. This also allows for some odd things like being able to manage deps in any go file which could get very confusing and I see it being abused. Looking at the following JSON object I think this would be a much cleaner way. I can immediately tell by looking at this that if i want to import armasm into the project i am working on i would use vendor/folder/rsc.io/arm/armasm.

{

"destination": "vendor/folder/",

   "deps": [

       {

           "ImportPath": "rsc.io/arm/armasm",

           "Rev": "616aea947362"

       },

       {

           "ImportPath": "rsc.io/x86/x86asm",

           "Rev": "af2970a7819d"

       }

   ]

}

Jacek Masiulaniec

unread,
Mar 3, 2015, 12:01:56 AM3/3/15
to Brad Fitzpatrick, golang-dev
It will be daunting having to read long lines such as:


This can be ameliorated by running a vanity domain, for example: 


In fact, this is the exact approach taken by my recent project (http://opentsp.org/). I am overall quite happy with this approach. It has made me toy with an idea for a dependency upgrade tool that walks internal/ to obtain the dependency list (i.e. without resorting to a config file), and then syncs each to the latest version that doesn't introduce a build or test failure in any importing package. This idea is half-baked; having an explicit hash or tag is probably the way to go.

Jacek


On 2 March 2015 at 09:37, Brad Fitzpatrick <brad...@golang.org> wrote:

Gophers,


One of the most frequent questions we’ve received since Go 1 was how to deal with dependencies and their versions. We’ve never recommended any particular answer.


In Google’s internal source tree, we vendor (copy) all our dependencies into our source tree and have at most one copy of any given external library. We have the equivalent of only one GOPATH and rewrite our imports to refer to our vendored copy. For example, Go code inside Google wanting to use “golang.org/x/crypto/openpgp” would instead import it as something like “google/third_party/golang.org/x/crypto/openpgp”. This has worked out very well for us and we get reproducible builds. If we ever want to update openpgp in our vendored directory, we update it, verify that all the affected code in the world still compiles and tests pass (making code changes as necessary), and then check it in. The world compiles and passes at all points in the version control system’s history, and if we bisect in time, we see which version of an external library was in use at any point.


We’re starting to see others in the Go community do the same, with tools like godep and nut.


We think it’s time to start addressing the dependency & vendoring issue, especially before too many conflicting tools arise and fragment best practices in the Go ecosystem, unnecessarily complicating tooling. It would be nice if the community could converge on a standard way to vendor.


Our proposal is that the Go project,


  1. officially recommends vendoring into an “internal” directory with import rewriting (not GOPATH modifications) as the canonical way to pin dependencies.

  1. defines a common config file format for dependencies & vendoring

  1. makes no code changes to cmd/go in Go 1.5. External tools such as “godep” or “nut” will implement 1) and 2). We can reevaluate including such a tool in Go 1.6+.


The important part is that as a community, we all do this the same way, so tooling can mature and interoperate.


In Go 1.5, the “internal” package mechanism introduced in Go 1.4 for the standard library will be extended to all go-gettable packages, so using the “internal” directory as the root of rewritten import paths makes sense (as opposed to “vendor” or “third_party”).


Consider an existing use of vendoring in the Go source tree: $GOROOT/src/cmd/internal currently contains copies of “rsc.io/x86/x86asm” and “rsc.io/arm/armasm” as $GOROOT/src/cmd/internal/rsc.io/x86/x86asm/ and $GOROOT/src/cmd/internal/rsc.io/arm/armasm/, respectively.  When we use those inside the Go tools, however, we import them “cmd/internal/rsc.io/x86/x86asm” and not “rsc.io/x86/x86asm”. (Although this example comes from the Go distribution repo, the effect is the same as a local project using $GOPATH/src/your.project/path instead of $GOROOT/src/cmd.)


We currently maintain those copies by hand. Instead, we want to write a file (filename and syntax to be determined), such as:


     src/cmd/internal/TBDCONFIG.CFG:

             “rsc.io/x86/x86asm” with revision af2970a7819d

             “rsc.io/arm/armasm” with revision 616aea947362


And then your vendoring tool (such as “godep” or “nut”) would read TBDCONFIG.CFG and write out,

     src/cmd/internal/rsc.io/x86/x86asm/*.go

     src/cmd/internal/rsc.io/arm/armasm/*.go


rewriting imports and import comments in these files for the new location.  It may also optionally change your source so any occurrence of


    import “rsc.io/x86/x86asm


becomes


    import “cmd/internal/rsc.io/x86/x86asm


The vendoring tool would be responsible for generating errors on conflicts or missing dependencies.


The thing that we as a community need to figure out is the recommended configuration file format.


We’d prefer something that the Go standard library can already parse easily. That includes XML, JSON, and Go.  Nobody likes XML, so that leaves JSON and Go.


godep already uses JSON, as do Go tools themselves (e.g. “go list -json fmt”), so we’ll probably want something like godep’s JSON format:


{

   "Deps": [

       {

           "ImportPath": "rsc.io/arm/armasm",

           "Rev": "616aea947362"

       },

       {

           "ImportPath": "rsc.io/x86/x86asm",

           "Rev": "af2970a7819d"

       }

   ]

}


We can start with that for discussion.


Note that we have rejected non-vendoring approaches that require modifications to GOPATH or new semantics inside the go command and toolchain. We believe it is important that the solution not require additional effort on the part of all the tools that already understand how to build, analyze, or modify code in the standard GOPATH hierarchy.


Joshua Marsh

unread,
Mar 3, 2015, 12:29:11 AM3/3/15
to golan...@googlegroups.com
On Monday, March 2, 2015 at 10:38:02 AM UTC-7, Brad Fitzpatrick wrote:

We currently maintain those copies by hand. Instead, we want to write a file (filename and syntax to be determined), such as:


     src/cmd/internal/TBDCONFIG.CFG:

             “rsc.io/x86/x86asm” with revision af2970a7819d

             “rsc.io/arm/armasm” with revision 616aea947362




I would personally prefer using something within the source code and more familiar to what I'm used to in Go. I like the idea of using something like a build constraint/generate comment or a struct field tag. Some possible formats might be:

import (
     "rsc.io/x86/x86asm" `source:"rsc.io/x86/x86asm af2970a7819d"`
     "rsc.io/arm/armasm" // +source rsc.io/x86/x86asm 616aea947362 
     "golang.org/x/crypto/openpgp// go:source golang.org/x/crypto/openpgp af2970a7819d 
)

This appeals to me because:
  • It looks familiar so it doesn't raise the hair on my neck when I'd want to use it.
  • It's not external to the source so I immediately know when someones code is expecting something and I don't have to look elsewhere.
  • The commented format would already show up in the ast and the tag would just be an additional field to the ast.ImportSpec.
  • It could still be used by other projects with relative ease.

Kamil Kisiel

unread,
Mar 3, 2015, 12:36:41 AM3/3/15
to golan...@googlegroups.com, brad...@golang.org, michae...@gmail.com
I think you misunderstood, I wasn't suggesting that you'd be able to have dependencies in any .go file, just the one specially named for configuration.
Your dependencies would still ultimately go in to their own vendored path. I'm only discussing the format of the version definitions, using standard go code as an alternative to JSON syntax. I don't think the JSON is any cleaner, it's probably about on par with Go's syntax as far as verbosity goes. The advantage of using Go syntax is that it's a bit easier to validate via the languages type system. Maybe that's not compelling enough and/or parsing would be too complicated compared to JSON, that's why I'm asking what people think.

I'm not sure what you mean by "implementing an entire new stdlib".

Dmitri Shuralyov

unread,
Mar 3, 2015, 1:59:30 AM3/3/15
to golan...@googlegroups.com, brad...@golang.org, michae...@gmail.com
I have one observation I'd like to share, it's one of many aspects that should be considered.

The examples in discussion above seem to be all using import paths as keys.

Perhaps it's better to allow import path patterns (as defined at https://golang.org/cmd/go/#hdr-Description_of_package_lists), since they can represent a VCS repository in one line. It's less likely you'd use different revisions for packages within one VCS repo.

So, instead of...

"rsc.io/arm/armasm": "616aea947362"
"rsc.io/x86/x86asm": "af2970a7819d"
"github.com/gregjones/httpcache": "4fb00a8eec008ffa8ac9ec46a93684dcbb62fc09" "github.com/gregjones/httpcache/diskcache": "4fb00a8eec008ffa8ac9ec46a93684dcbb62fc09" "github.com/gregjones/httpcache/memcache": "4fb00a8eec008ffa8ac9ec46a93684dcbb62fc09"

It could be:

"rsc.io/arm/armasm/...": "616aea947362"
"rsc.io/x86/x86asm/...": "af2970a7819d"
"github.com/gregjones/httpcache/...": "4fb00a8eec008ffa8ac9ec46a93684dcbb62fc09"

All I ask is you consider this point (that a single repository may contain more than one Go package) and see what works better.

(Also note that normal import paths are valid import path patterns too, so it's backwards compatible.)

Matthew Sackman

unread,
Mar 3, 2015, 5:57:27 AM3/3/15
to golang-dev
On Mon, Mar 02, 2015 at 09:37:53AM -0800, Brad Fitzpatrick wrote:
> One of the most frequent questions <http://golang.org/doc/faq#get_version>
> we’ve received since Go 1 was how to deal with dependencies and their
> versions. We’ve never recommended any particular answer.

...

> Our proposal is that the Go project,
>
>
> 1.
>
> officially recommends vendoring into an “internal
> <https://docs.google.com/a/golang.org/document/d/1e8kOo3r51b2BWtTs_1uADIA5djfXhPT36s6eHVRIvaU/edit>”
> directory with import rewriting (not GOPATH modifications) as the canonical
> way to pin dependencies.

That doc talks about how to solve the issue of ensuring limited
import-ability of packages through import rewriting.

This seems to be a quite different issue than you stated at first - that
of dealing with "dependencies and versions". At best, restricted imports
is a subset of the wider issue at hand.

> 2.
>
> defines a common config file format for dependencies & vendoring

Fine. But I would support the various proposals elsewhere in this thread
that this should not be a static config file. It is always worth having
some ability to compute as well as declare such dependencies.

However, far more generally, I have yet to see (and apologies if I've
managed to miss it), a clear statement of which problems are in scope
for being solved, and which are not.

For example, import rewriting would allow you to hide packages, but as
seen elsewhere in this thread, if A imports B and C, and B imports C,
and A needs to use C to talk to B then import rewriting appears to ruin
that. Imports are resource discovery - they give known names to
resources. There are clearly use cases where you want to pin versions of
dependencies and make them easily discoverable by a dependee.

I do not understand why libraries should be treated differently to
"programs". Anything that requires me, when using a library, to manually
have to track down the transitive dependencies and decide for myself
which versions of X Y and Z some random library I'm using can work with,
I'm just not going to use. If the design of whatever-is-being-designed
here does not work in the general transitive case, then I don't
understand how it can be useful. I'm guess I must be missing something
here.

The question of bit-equal builds is another issue which I can't tell
whether or not it's in-scope for this or not. I have spent a fair amount
of time packaging Go programs for Nix (eg Serfdom) and Nix certainly
manages this perfectly well without needing any vendoring of packages.
If you have bit-equal builds then you have a proof that you know which
sources (up to equivalence) went into the compilation, in which case
build-farms and caching binary artifacts works well. In light of this,
I'm not convinced that "vendor, in case you don't have the internet
available" is a valid concern.

Matthew

pic...@gmail.com

unread,
Mar 3, 2015, 8:38:38 AM3/3/15
to golan...@googlegroups.com
Vendoring is going to be a complete mess to maintain in the long run, and way too complicated a solution. I think if you're Google you can make it work. But if you're a startup in a basement, you're going to have to switch to a language with proper dependency management. You just won't have the manpower to maintain the source and updates for 30 other libraries along with your app.

Why not instead just add support to Go for an optional tag or commit ID to the import string, along the lines of what Joshua Marsh is suggesting:

import "github.com/johndoe/mylib 1.0"
import "bitbucket.org/janedoe/another fe3b56a01ccde"

If it's not there, use the "master" branch.

import "github.com/old/mylib" // uses master branch

eric.t....@gmail.com

unread,
Mar 3, 2015, 8:38:46 AM3/3/15
to golan...@googlegroups.com, emi...@gmail.com
FWIW, I have something in my in-progress Grapnel tool (https://github.com/eanderton/grapnel) that does this Alias capability.  Instead of 'Alias', it optionally de-couples the import path from the URL that is used to obtain it.  I've found this approach to be both flexible and self-documenting:

# TOML format

[[dependencies]   # denotes a single dependency in the 'dependencies' struct array
import = "foobar"
url = "https://github.com/foo/bar"

What's nice about this is that if you have a fork of a commonly used library, you can easily aim Grapnel at that repo instead.

Aside: Grapnel is very under-documented at the moment, but it already does some other interesting things like sifting through tags for semantic versioning support, dynamically re-writing gopkg.in imports to direct github.com interactions, and recursive dependency solving.  Because of this thread, JSON config support and import re-writing are going to have to take priority next.

Mihai B

unread,
Mar 3, 2015, 9:14:01 AM3/3/15
to golan...@googlegroups.com
 package ax
 import  "golang.org/x/crypto/openpgp// internal af2970a7819d  - would keep the source of openpgp package with revision af2970a7819d in the internal directory of the package ax.

I would definitely vote to maintain the versioning information in the source code (i.e. special comments as in the proposal above). That way what you see is what you get. You don't need to look in a config file to find if the imported package actually has a different version than what you see on godoc. It just pops up when you read the import paths which you need to do anyway.  

 I've always liked that Go doesn't use config files like npm or rust to manage the dependencies/import packages. That may not be the case for some packages with unusual requirements  (as  cmd/internal/gc was mentioned) but such packages are "exceptions". 

Lars Seipel

unread,
Mar 3, 2015, 9:32:52 AM3/3/15
to golang-dev
On Tue, Mar 03, 2015 at 10:56:55AM +0000, Matthew Sackman wrote:
> I do not understand why libraries should be treated differently to
> "programs".

Because library packages are supposed to be combined with other Go code.
If a package vendors its dependencies, the user of that package also has
to use the vendored copies or deal with the consequences of having
multiple copies of a package inside a single binary. What if two
packages both include a copy of a third one, but using different
versions?

It would put the burden of reconciling the mess on the package user and,
in my opinion, massively hurt the Go package ecosystem if gaining
widespread use.

A package should be as widely useful as reasonably possible—don't
decrease its usefulness by restricting the environment it might be used
in. Being able to specify a commit hash is no substitute for choosing
your dependencies wisely.

Matthew Sackman

unread,
Mar 3, 2015, 9:48:57 AM3/3/15
to golang-dev
On Tue, Mar 03, 2015 at 03:32:42PM +0100, Lars Seipel wrote:
> If a package vendors its dependencies, the user of that package also has
> to use the vendored copies or deal with the consequences of having
> multiple copies of a package inside a single binary. What if two
> packages both include a copy of a third one, but using different
> versions?

Absolutely. This is pretty much the classic diamond inheritance problem.

> It would put the burden of reconciling the mess on the package user and,
> in my opinion, massively hurt the Go package ecosystem if gaining
> widespread use.

Huh? So the alternative is that packages aren't able to specify the
versions of things they depend on, and so the user of the package now
has to trawl through all their libraries, find the transitive
dependencies, and hope that none of the libraries implicitly depend on
specific versions? This isn't putting unacceptable burden on the package
user?

> A package should be as widely useful as reasonably possible—don't
> decrease its usefulness by restricting the environment it might be used
> in. Being able to specify a commit hash is no substitute for choosing
> your dependencies wisely.

I'm writing package X. It depends on some other package Y that I have no
control over. The most sensible and responsible thing I can do is to
specify that X can only use the exact version of Y that I've built it to
use. I have no ability to control what the author of Y does in the
future. So in terms of being able to deliver X with the behaviour and
semantics that I, and my users want, I *have* to pin it to the "current"
version of Y that I'm working with. When other versions of Y have been
verified to work correctly, they can be added to the set of acceptable
versions of Y. Surely, that makes it as widely useful as possible.

The end goal here is to deliver a package that has understood semantics
and behaviour which people can rely on. Having it build with as many
versions of Y as possible is very much a secondary goal, if that.

Matthew

pote...@gmail.com

unread,
Mar 3, 2015, 10:24:46 AM3/3/15
to golan...@googlegroups.com, emi...@gmail.com
> officially recommends vendoring into an “internal” directory with import rewriting (not GOPATH modifications) as the canonical way to pin dependencies.

Is there any reason to being so adamant about not changing the GOPATH? What benefits does import rewriting bring to the table over it? In my experience changing the GOPATH solves a pretty big part of the vendoring/dependency problem and it does so without having to change a single line of Go code, this is something that has been out there for a while and we can leverage immediately, why go for a more complex solution? Why hardcode something like "internal" or "vendor" when setting a per-project GOPATH can give us a flexible solution for free?

> defines a common config file format for dependencies & vendoring

I am obviously biased from working on gpm, but I think it's dependency file format is a good starting point: https://github.com/pote/gpm#the-godeps-file, it is almost verbatim what is described as what we need in the initial proposal.

Having package versions in .go files as part of imports doesn't communicate how Go dependencies work: it would imply that you can import different versions of the same package in different files, a package can only have one active version per application importing it so having it be a per-application metadata file seems like a more correct approach. I also don't think we should jump directly into JSON/Yaml without there being an actual need for them, particularly at such an early point in the discussion: complexity is easy to add and hard to get rid of.

mediocr...@gmail.com

unread,
Mar 3, 2015, 10:24:46 AM3/3/15
to golan...@googlegroups.com
I like the idea of using a .go file as opposed to a .json file for this. Another nice property of a .go file which wasn't mentioned is that it can be commented, something which could definitely be useful if you want to inform future generations why you're using a particular commit or branch.

One negative which we ought to address though: it's possible (though not terribly likely) that someone's project already exports a variable or type of the same name that we choose here, which would mean this would require changes to their existing package

Brad Fitzpatrick

unread,
Mar 3, 2015, 10:26:21 AM3/3/15
to mediocr...@gmail.com, golang-dev
On Tue, Mar 3, 2015 at 6:09 AM, <mediocr...@gmail.com> wrote:
I like the idea of using a .go file as opposed to a .json file for this. Another nice property of a .go file which wasn't mentioned is that it can be commented, something which could definitely be useful if you want to inform future generations why you're using a particular commit or branch.

Good point. That's probably the strongest argument in favor of Go instead of JSON. I'm always annoyed by the lack of comments in JSON.

Russ Cox

unread,
Mar 3, 2015, 10:31:58 AM3/3/15
to Brad Fitzpatrick, mediocr...@gmail.com, golang-dev
I like the idea of using a Go file, but note that it cannot be one that actually builds, because it needs to record the original import paths, not the vendored ones. I would expect that a Go config file would have a standard .go suffix (so that gofmt etc apply) but a build tag keeping it from building, a standard name (say, vendor.go), and a required package name (say, vendor):

---
// +build ignore

package vendor

import (
    // These are for the disassemblers.
    "rsc.io/arm/armasm" // 616aea947362
    "rsc.io/x86/x86asm" // af2970a7819d
)
---

Russ

r.w.jo...@gmail.com

unread,
Mar 3, 2015, 10:56:30 AM3/3/15
to golan...@googlegroups.com, a...@golang.org
On Monday, 2 March 2015 17:24:41 UTC-5, Ben Darnell wrote:
But if we had this version-aware infrastructure (which I admit is a harder problem), would we still want to rewrite imports? I claim that for purposes of reproducible builds it's better to place the entire GOPATH under version control (as long as you can have per-project GOPATHs) than to use an internal/vendor directory to ensure that all your dependencies share a common prefix deep in the source tree.


Is there a use-case for vendoring some dependencies, but pulling others from your GOPATH?   Otherwise, I don't see the advantage of import rewriting over pointing GOPATH at your vendored libraries.

chris dollin

unread,
Mar 3, 2015, 10:56:51 AM3/3/15
to Brad Fitzpatrick, mediocr...@gmail.com, golang-dev
On 3 March 2015 at 15:26, Brad Fitzpatrick <brad...@golang.org> wrote:

> Good point. That's probably the strongest argument in favor of Go instead of
> JSON. I'm always annoyed by the lack of comments in JSON.

{"comment": "this is a comment", "otherField": 1066, ...}

{"comment": ["lacking multiline strings", "use an array"]}

Would that not suffice despite its clunkiness?

Chris

--
Chris "or is it 'clunkyness'?" Dollin

Pablo Astigarraga

unread,
Mar 3, 2015, 10:59:59 AM3/3/15
to r.w.jo...@gmail.com, golan...@googlegroups.com, a...@golang.org
Is there a use-case for vendoring some dependencies, but pulling others from your GOPATH?   Otherwise, I don't see the advantage of import rewriting over pointing GOPATH at your vendored libraries.

Even if there was you can easily accomplish that with a GOPATH like "locally/vendored/dependencies:/global/go/dependencies", I am also struggling to understand the benefits of import rewriting.

--
You received this message because you are subscribed to a topic in the Google Groups "golang-dev" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/golang-dev/nMWoEAG55v8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to golang-dev+...@googlegroups.com.

Ian Davis

unread,
Mar 3, 2015, 11:03:52 AM3/3/15
to golan...@googlegroups.com
Could use the main package if we wanted to restrict vendoring only to applications rather than packages.
 
Ian
 

Lars Seipel

unread,
Mar 3, 2015, 11:07:50 AM3/3/15
to golang-dev
On Tue, Mar 03, 2015 at 02:48:27PM +0000, Matthew Sackman wrote:
> I'm writing package X. It depends on some other package Y that I have no
> control over. The most sensible and responsible thing I can do is to
> specify that X can only use the exact version of Y that I've built it to
> use. I have no ability to control what the author of Y does in the
> future.

If the author of the package you depend on introduces a bug which breaks
your package it should be fixed. Just pinning down some old version
might let that bug go unnoticed for a long time and some way down the
road maybe someone else starts to depend on that buggy behaviour. Now
it's no longer possible to combine your package and that other one. Not
good for the ecosystem.

If you don't trust package authors to not go nuts and break the API left
and right, you shouldn't use their packages at all. If they break it
inadvertently, it's simply a bug and should be fixed.

It's certainly not the most convenient way (and might be unpractical at
times) but it should pay off in the long run.

Ian Davis

unread,
Mar 3, 2015, 11:15:26 AM3/3/15
to golan...@googlegroups.com
On Tue, Mar 3, 2015, at 03:31 PM, Russ Cox wrote:
I suspect go generate could be used for vendoring.
 
//go:generate vendor rsc.io/arm/armasm 616aea947362
//go:generate vendor rsc.io/x86/x86asm af2970a7819d
 
Running go generate would check out those package versions, copy the source below an internal package and rewrite imports.
 
Ian
 
 
 
 
 
 
 
 
 
 
 
 

Matthew Sackman

unread,
Mar 3, 2015, 11:27:52 AM3/3/15
to golang-dev
On Tue, Mar 03, 2015 at 05:07:44PM +0100, Lars Seipel wrote:
> Now
> it's no longer possible to combine your package and that other one. Not
> good for the ecosystem.

Really? There's a technical reason why you can't, through rewriting, have:

A depends on B
A depends on C
B depends on D v1
C depends on D v2

? My guess (though I've not tried it) is that Go would be just fine with
that - Dv1 and Dv2 should exist in their own namespaces and not
interfere with each other at all. Maybe I've missed something.

> If you don't trust package authors to not go nuts and break the API left
> and right, you shouldn't use their packages at all. If they break it
> inadvertently, it's simply a bug and should be fixed.

Very much struggling to reply to that.

There's a very substantial difference between a minor version bump, for
which I would expect API stability, and a major version bump. semver is
quite useful in this regard IMO. I would not expect a package to be
renamed even if is completely rewritten, provided it solves the same
problem.

I would hope that, as a package author, I would be able to specify
constraints on the versions of libraries on which I depend. It is then
up to some other tool, perhaps yet to be written, to gather such
constraints, calculate the transitive closure, and determine whether or
not they can be satisfied.

Matthew

giacomo...@gmail.com

unread,
Mar 3, 2015, 11:44:44 AM3/3/15
to golan...@googlegroups.com, brad...@golang.org, mediocr...@gmail.com
Has this bee proposed and ruled out already?

package main

import (
    //go:vendor 616aea947362

    //go:vendor af2970a7819d
)

The imports reordering tools need to keep track of the "vendoring comment".


Giacomo

Michael Schuett

unread,
Mar 3, 2015, 11:49:53 AM3/3/15
to giacomo...@gmail.com, golan...@googlegroups.com, brad...@golang.org, mediocr...@gmail.com
I feel like allowing every import to specify a version ends up with people in "version hell". You would also have to specify the version everytime you wanted to use that package. If you changed one would an error be thrown until you changed it in the various other places that you are specifying the old version? 

--

Giacomo Tartari

unread,
Mar 3, 2015, 11:59:49 AM3/3/15
to Michael Schuett, golan...@googlegroups.com, brad...@golang.org, mediocr...@gmail.com
On 3Mar, 2015, at 17:49 , Michael Schuett <michae...@gmail.com> wrote:

I feel like allowing every import to specify a version ends up with people in "version hell". You would also have to specify the version everytime you wanted to use that package. If you changed one would an error be thrown until you changed it in the various other places that you are specifying the old version? 
I would assume so, but that is another easily automated task.
Could even be made interactive:
$go vendor
$go vendor found mismatching version for package X: v1, v2, v3.
do you wish to unify the versions or something? y,n
version [1], [2], [3]?


And it would be cool it tags are supported
//go:vendor rev:616aea947362
//go:vendor tag:616aea947362
Tags are easier to read.

Giacomo

giacomo...@gmail.com

unread,
Mar 3, 2015, 12:01:59 PM3/3/15
to golan...@googlegroups.com, michae...@gmail.com, brad...@golang.org, mediocr...@gmail.com, giacomo...@gmail.com


On Tuesday, 3 March 2015 17:59:49 UTC+1, Giacomo Tartari wrote:

On 3Mar, 2015, at 17:49 , Michael Schuett <michae...@gmail.com> wrote:

I feel like allowing every import to specify a version ends up with people in "version hell". You would also have to specify the version everytime you wanted to use that package. If you changed one would an error be thrown until you changed it in the various other places that you are specifying the old version? 
I would assume so, but that is another easily automated task.
Could even be made interactive:
$go vendor
$go vendor found mismatching version for package X: v1, v2, v3.
do you wish to unify the versions or something? y,n
version [1], [2], [3]?


And it would be cool it tags are supported
//go:vendor rev:616aea947362
//go:vendor tag:616aea947362
this was supposed to be  //go:vendor tag:v1.0
I need more coffe/sleep, sorry.

To unsubscribe from this group and all its topics, send an email to golang-dev+unsubscribe@googlegroups.com.

Michael Schuett

unread,
Mar 3, 2015, 12:15:11 PM3/3/15
to giacomo...@gmail.com, golan...@googlegroups.com, brad...@golang.org, mediocr...@gmail.com
Tags would be cool but I don't really know how many packages support them I am guessing a very small amount since go get works off master. Some of the larger projects do which may be reason of enough for tag support but commit tag support is really all I would be looking for and would simplify things. But I may bias I like the simplicity of one json or go file in the root of the projects that manages all the dependencies I think it would also be familiar with people coming from pretty much any other language not that we need to cater to them. I am leaning towards a .json file because the vendoring file should not be considered to be part of your go project it's just installing something that you will be using in your project.

just my 2c.

Joshua Marsh

unread,
Mar 3, 2015, 12:23:50 PM3/3/15
to golan...@googlegroups.com
On Tue, Mar 3, 2015 at 9:49 AM, Michael Schuett <michae...@gmail.com> wrote:
I feel like allowing every import to specify a version ends up with people in "version hell". You would also have to specify the version everytime you wanted to use that package. If you changed one would an error be thrown until you changed it in the various other places that you are specifying the old version? 


I think this might actually be no different than how a config might produce a "version hell". If one developer was working on one revision and another developer a different revision, once the two sources were brought together there would be a build failure that could be resolved. Using a config won't fix this. You could possibly catch it early if you looked at the config during the merge, but otherwise, you'd probably still find it during the testing after the merge. If the a revision was already in place local testing would catch it just as fast as the command that parses the config.

This way seems more inline with the go ecosystem as opposed to some external configuration file in my mind. Using a configuration feels like building an additional room to your house with some plywood and duct tape. Using something like a build constraint or struct tag like I suggested earlier feels like you called a contractor and got it done right.

Giacomo Tartari

unread,
Mar 3, 2015, 12:29:28 PM3/3/15
to Michael Schuett, golan...@googlegroups.com, brad...@golang.org, mediocr...@gmail.com
I don’t see tags as more complicated, to the contrary they would make it easier to distinguish form major versions(and backward incompatibilities):

//go:vendor rev:af2970a7819d

vs

//go:vendor tag:v2.1

As for the json file, I prefer to keep it to go.
Eventually a json file will be filled up with other configurations for other incompatible third party tools and stuff will break.
Keeping it in control of the go team is a more stable choice imho.

Giacomo

Govert Versluis

unread,
Mar 3, 2015, 12:29:36 PM3/3/15
to Michael Schuett, giacomo...@gmail.com, golang-dev, Brad Fitzpatrick, mediocr...@gmail.com
If one uses tags or commit hashes in the .go source file, how are conflicting versions stored in the gopath?
e.g. package A depends on package C tag 123, package B depends on package C tag 456.
They both have a magic comment in the source file to indicate the right tag to use.
However, since the path in the go source file is the same ("C"), the go build tool (and indeed other tools) will all look in $GOPATH/src/C.

I'm against storing this data in .go files, especially if it's in some form of magic comment.
The need for structure in directives such as these stands completely opposed to comments (which can be anything).

The primary approach to vendoring at the moment seems to be to place the vendored packages in a internal/, third_party/ or vendor/ sub-directory, thus import paths need to be rewritten.
As such it seems to make more sense to store the original package path and the revision to use in a manner separate from the actual source files, in structured format. allowing all imports (in multiple files) to be rewritten as needed by the appropriate tool.

I don't see an elegant way to do this in .go files.

Pieter Droogendijk

unread,
Mar 3, 2015, 12:43:53 PM3/3/15
to golan...@googlegroups.com, michae...@gmail.com, giacomo...@gmail.com, brad...@golang.org, mediocr...@gmail.com
On Tuesday, March 3, 2015 at 6:29:36 PM UTC+1, Govert Versluis wrote: 
I'm against storing this data in .go files, especially if it's in some form of magic comment.
The need for structure in directives such as these stands completely opposed to comments (which can be anything).

I agree with this, putting it in a comment may be the wrong move. But aren't import paths just strings? Don't they allow somewhat more freedom? Didn't we do that for a reason?

import `rsc.io/x86/x86asm, vendor, rev:"af2970a7819d"`

Or perhaps a tag string, like struct fields. Nicely backwards-compatible, and it nets us something like this:

import (
    "os"
    x86asm_v2 "rsx.io/x86/x86asm" `vendor, tag:"v2"`
)

Giacomo Tartari

unread,
Mar 3, 2015, 12:44:14 PM3/3/15
to Govert Versluis, Michael Schuett, golang-dev, Brad Fitzpatrick, mediocr...@gmail.com
Sorry for the noise did not reply to all.
> On 3Mar, 2015, at 18:29 , Govert Versluis <gov...@ver.slu.is> wrote:
>
> If one uses tags or commit hashes in the .go source file, how are conflicting versions stored in the gopath?
They are not, the comment is for a vendoring tool that will checkout the right version in the internal dir.

I would also add a go version comment to get completely reproducible builds, //go:version 1.5.

leise...@gmail.com

unread,
Mar 3, 2015, 12:46:00 PM3/3/15
to golan...@googlegroups.com
I like the idea of vendoring.
Doing it only on the main package level is fine.

But what I dislike is leaving the external library authors
out of the game.

If a library has no non standard lib as dependency we do not have a problem.

But if library A depends on B,
the maintainer should have a mechanism to tell his users which version(s) of B are fine, and which are not.
He knows that from its own test, and it would be an immense waste of effort if we do not share this knowledge and even worse we do not have a way of sharing it.
If we now think of a third package C depending on B as well, we may use the imformation provided by the the maintainers of A and C to make our vendoring decision.

But how to supply this kind of information must be a second proposal, separate from the first.

Martin

Eric Myhre

unread,
Mar 3, 2015, 1:23:47 PM3/3/15
to golang-dev
Whatever comes of this discussion: Distancing any proposed model from how Google works internally is going to be difficult, but necessary.

The open source community as a whole has been burned before by the behaviors that emerge from a company culture that attempts open source but maintains things only in their personal vendorized ecosystem: remember the release of thrift.... followed by the release of thrift? This is -- and there's no shame in this, but I remember a number of central gophers admitting their workplace basically means they never use 'go get' -- a large part of the reason 'go get' is in such a dilapidated state compared to the rest of the otherwise charming and excellent go tool.

I don't want to see go getting hung up on root misunderstandings of how a needs of a global collaboration culture may differ from those of a single company with a single dictatorial source tree with a single snapshot view available for a single point in time. The latter is easier, to be sure. Let's focus on moving beyond that. Planet Earth does not have a single shared source tree.

Solutions need to account for usage *without prior coordination* between package authors.

... And that's where a whole bunch of the proposed stuff fails. Badly.

This whole discussion started with vendoring presumed. (Why is it even in the thread title?) Import rewriting is then proposed as a workaround.

There's still huge trouble with this when extended to more than *literally one* project, so it's proposed (with *colossal* handwaving) that libraries and programs should be different.

These are both ridiculous duct tape workarounds. No: import rewriting is a ridiculous ducktape workaround, and claiming that libraries and programs are clearly distinct is just lying back and thinking of England.

I propose we take a moment and clearly define the goalposts again: everyone's concerns are oriented around isolation, repeatability, and composability. Are there better ways we can serve these goals, especially with scaling across many repositories and many uncoordinated authors?


### isolation

Notice how I didn't say "import rewriting". What everyone wants is isolation. We should discuss how to implement that concept. "import rewriting" is one possible way to implement isolation; let's phrase the conversation around the concept first.

I second every single person (there have been many) who suggest either using per-project GOPATH settings, or -- if this hasn't been suggested outloud yet, I'd like to make the proposal -- creating a new system with similar semantics. Having wholey separate directory trees for each project is, in fact, absolutely the semantic that I want. Global variables are bad; global library heaps are equally bad, particularly when shared between many different programs of wildly unrelated authorship. (Anyone from the maven world -- remember how it feels to `rm -rf ~/.m2`? Tension just flows out of the body. Tension that never should have existed.)

Much like we gophers value our statically linked single file binaries for their ease of shipping, so too do I value a project folder that is clearly that project, only that project, and all of that project. This is what happens when I set `GOPATH=$project/.gopath/`. Much like the whole truth and nothing but the truth, it's great! This means I can sync severable parts of my work to various computers easily, etc, and is super empowering.

I regret to say I cannot buy suggestions that this would be an overwhelmingly disruptive chance to the go ecosystem. Currently doing so with every single go library and program I've ever touched in fact seems like rather concrete proof that this is doable, since it is in fact, done.

Import rewriting makes sense in the situation that I have a diamond dependency (A -> B -> D.v1 & A -> C -> D.v2) and I'm willing to duplicate the two different versions of that transitively required library in order to isolate them while retaining both. This is by far the exception case, not the norm. Imagine if every package that used something as common as "fmt" from the standard library re-wrote and re-vendored it!

Note that I have little care for the size of my working tree in an active development environment -- duplicated libraries on disk there per project is fine; disk is cheap. Duplication that ends up in version control is where concern lies, because this duplication has troublesome impacts over deep time, and the trouble can be pushed to other people who are then powerless to address it.


### repeatability

Notice how I didn't say "vendoring". What everyone wants is repeatability. "vendoring" is one possible way to implement repeatability; let's phrase the conversation around the concept first.

As pointed out earlier in the threat, repeatability is also maintained by systems such as Nix: hashes do wonders. Similarly, I've been using git submodules to get perfectly reproducible builds. Bower can pin dependencies by hashes. There's a huge list of concrete examples of repeatability without vendoring. Hash-based resource references solve repeatability.

Vendoring may score points in the immediate offline mode. And I'm deeply pleased with importance placed on offline operation -- I share this priority the extent that I label my business cards "sneakernet advocate" -- but offline operation is orthogonal to vendoring and also available through other choices. It's good to have a command after which the build is guaranted not to need network; that command need not be `git clone`, and could just as easily be some other subcommand of the go tool. I think folk coming from other languages would be perfectly unsurprised by such a feature in the go tool, and we could have very satisfying outcomes from a command that does all resource acquisition and then stops.

Vendoring has gained traction largely because it works without additional tooling, and that makes it the most contributor-friendly thing in the current landscape where the go tool has no chosen stance and thus the community as a whole is uncommitted. In discussing additions to the go tool, this impetus is made irrelevant.

Vendoring has a variety of serious drawbacks. It gets repeatability right and *everything else* wrong. It's not transparent to libraries; it results in globally multiple histories for files; and it results in permanent bloat to a project source repo. (This was discussed before at length in a particular golang project forum and this table of comparisons was made back in 2013, which remains fully accurate today: https://gist.github.com/heavenlyhash/6343783 )

The most major threat to repeatability when using hash-based resource references is network available in the deep time sense: what happens when the organization behind mycompany.net goes out of business and drops their domain, or library bananapancakes is renamed to something more user-centric, etc. In tracking recent changes, this is not frequently an issue, but doing a bisect across a well-aged repo can become troublesome. Fortunately, this is solvable: The system must have a way to replace old network addresses with updated aliases. We can do this.


### composability

I'm not sure this is in the discussion as such yet, but it's been mentioned in passing, and so I figured I might as well give it a name and a heading.

As a user of go, when examining a new library for potential use, I regularly want to see if it builds. Thereafter, I want to see if its tests pass. This requires that the library specify it's dependencies; specifically, it means the library must have repeatability just like a "real" "program".

Bundler and rubygems actually got this fairly right: specifying a semver dependency range in one file, and specifying the "locked", pre-resolved versions of everything depended on (including transitively) is a great example of unifying both composability and repeatability goals. Bundler fell short in that it still only resolves down to a precise semver string, where it should really resolve to a hash. In the javascript world, Bower did similar things, and also carried the resolve results all the way to a hash: this truly satisfies repeatability. We should take a page from these books.

While it is possible to ship executables at the end of the day without tackling this (and that is the most important thing), as Matthew Sackman has pointed out, it's a severe cramp to developer usability if we ignore this.

On the plus side, this is also possible to do later, since isolation and repeatability are solvable without a solution to composability -- but it may be worth thinking about it now, when discussing data formats.


### aside: this "libraries and programs are different" thing *doesn't work* in the field

Treating libraries and programs differently has a hidden presumption: that I have any control over other authors.

Regrettably, I lack total control over all programmers. (Imagine! We could forgo this entire discussion! World conquest by friday! Etc.) And even if everyone in the world reoriented their views to ask all present and future users whether a package should be a library or not, there's no guarantee we'd agree. And a time dimension is indeed a part of that discussion, unless we break free of VCS repository layouts entirely, which has not yet come to pass.

As one example, Docker has some code that I'd like to use as a library. But they don't consider themselves a library, and as a result, they've already vendored about 5 megs of other sources (and if including history, a much larger amount even after compression). This situation is understandable -- they consider themselves a "program", and rightly so -- but nonetheless, it's become very difficult for me to reuse their code without forking (and I mean manually, by copy-paste, a highly undesirable outcome). And then AFTER copy-paste forking the sections of code I want, I have to rewrite all their imports again to fit in my project, since I'm already using several of the same libraries at the same versions. This entire process would be nonsensical made-work that actively distracts from real development, and with all due respect to all the authors in this scenario who are all understandably acting in their best interests, I want no part of this to be in my future for other golang proje
cts.


Chris Hines

unread,
Mar 3, 2015, 1:48:46 PM3/3/15
to golan...@googlegroups.com, brad...@golang.org, mediocr...@gmail.com
It could be valuable to have the vendoring metadata available at runtime. I am thinking of asking a binary to dump out its version and the versions of its build dependencies. This would be trivial if the metadata was compiled into the binary in an accessible Go data structure. I don't think this is a slam dunk argument, but something to consider.

Chris

Sean Bowman

unread,
Mar 3, 2015, 2:06:46 PM3/3/15
to Michael Schuett, giacomo...@gmail.com, golan...@googlegroups.com, brad...@golang.org, mediocr...@gmail.com
Aren't we already at the gates of version hell?  I feel like the only reason the import system has worked so far is because we don't have a lot of "v2" libraries out there yet. The ecosystem is too young. 

I'm already used to repeating myself with Go. It's not the DRYist language in the world. And yes, I'd expect the compiler to generate an error if I use conflicting versions in my code. 

shado...@gmail.com

unread,
Mar 3, 2015, 2:42:42 PM3/3/15
to golan...@googlegroups.com
I think a better solution is this:

import "import/path/that/has/some/slashes#commit-like"

This has the advantage of also being reproducable in the filesystem:

$GOPATH/src/import/path/that/has/some/slashes#commit-like"

Thoughts? This is much better than introducing the same versioning hell that Gem and NPM suffer from (not to mention Haskell's Cabal).

agro...@digitalocean.com

unread,
Mar 3, 2015, 2:42:42 PM3/3/15
to golan...@googlegroups.com
A problem with this approach is that if my project imports package A, and package A imports package B, by rewriting imports, I will modify the source of package A from importing `package.com/b` to become the vendored `internal/third_party/package.com/b`.

Consider a bug is found in package A. Since we modified the imports, we can't just `cd` into the repo, create a change commit and send contributions back upstream. It's possible to do it by restoring the original source file, but not as easy as if package A was vendored verbatim.

When we investigated how we wanted to deal with our third party dependencies, being able to contribute changes to our vendored code back to upstream was deemed important. We found that a good way to distinguish between our project code and third party code was to create 2 gopaths, appended together in the GOPATH variable.

export GOPATH:third_party/:owncode/

Since `go get` fetches package into the first path it finds in GOPATH, this means we can use `go get` on third party deps and the repos are inserted under the right path. We also get to vendor all of our code in one repository, achieving the 1 GOPATH that is mentioned above.  We wrote a blog post about how we deal with dependencies:

https://www.digitalocean.com/company/blog/taming-your-go-dependencies/#whats-a-mono-repo

We find that this is a clean and simple way to maintain and vendor all of our code. Import paths are not changed yet we get reproducible builds at any commit. We can modify parts of a project and immediately know all the thing we broke and fix them, since we can `go build ./...` to compile all of our code. The one downside is that Git will try to make submodules from third party repos, but the same would likely be true if we used the technique described here with import rewriting.


On Monday, March 2, 2015 at 12:38:02 PM UTC-5, Brad Fitzpatrick wrote:

Gophers,


One of the most frequent questions we’ve received since Go 1 was how to deal with dependencies and their versions. We’ve never recommended any particular answer.


In Google’s internal source tree, we vendor (copy) all our dependencies into our source tree and have at most one copy of any given external library. We have the equivalent of only one GOPATH and rewrite our imports to refer to our vendored copy. For example, Go code inside Google wanting to use “golang.org/x/crypto/openpgp” would instead import it as something like “google/third_party/golang.org/x/crypto/openpgp”. This has worked out very well for us and we get reproducible builds. If we ever want to update openpgp in our vendored directory, we update it, verify that all the affected code in the world still compiles and tests pass (making code changes as necessary), and then check it in. The world compiles and passes at all points in the version control system’s history, and if we bisect in time, we see which version of an external library was in use at any point.


We’re starting to see others in the Go community do the same, with tools like godep and nut.


We think it’s time to start addressing the dependency & vendoring issue, especially before too many conflicting tools arise and fragment best practices in the Go ecosystem, unnecessarily complicating tooling. It would be nice if the community could converge on a standard way to vendor.


Our proposal is that the Go project,


  1. officially recommends vendoring into an “internal” directory with import rewriting (not GOPATH modifications) as the canonical way to pin dependencies.

  2. defines a common config file format for dependencies & vendoring

  3. makes no code changes to cmd/go in Go 1.5. External tools such as “godep” or “nut” will implement 1) and 2). We can reevaluate including such a tool in Go 1.6+.


The important part is that as a community, we all do this the same way, so tooling can mature and interoperate.


In Go 1.5, the “internal” package mechanism introduced in Go 1.4 for the standard library will be extended to all go-gettable packages, so using the “internal” directory as the root of rewritten import paths makes sense (as opposed to “vendor” or “third_party”).


Consider an existing use of vendoring in the Go source tree: $GOROOT/src/cmd/internal currently contains copies of “rsc.io/x86/x86asm” and “rsc.io/arm/armasm” as $GOROOT/src/cmd/internal/rsc.io/x86/x86asm/ and $GOROOT/src/cmd/internal/rsc.io/arm/armasm/, respectively.  When we use those inside the Go tools, however, we import them “cmd/internal/rsc.io/x86/x86asm” and not “rsc.io/x86/x86asm”. (Although this example comes from the Go distribution repo, the effect is the same as a local project using $GOPATH/src/your.project/path instead of $GOROOT/src/cmd.)


We currently maintain those copies by hand. Instead, we want to write a file (filename and syntax to be determined), such as:


     src/cmd/internal/TBDCONFIG.CFG:

             “rsc.io/x86/x86asm” with revision af2970a7819d

             “rsc.io/arm/armasm” with revision 616aea947362


And then your vendoring tool (such as “godep” or “nut”) would read TBDCONFIG.CFG and write out,

     src/cmd/internal/rsc.io/x86/x86asm/*.go

     src/cmd/internal/rsc.io/arm/armasm/*.go


rewriting imports and import comments in these files for the new location.  It may also optionally change your source so any occurrence of


    import “rsc.io/x86/x86asm


becomes


    import “cmd/internal/rsc.io/x86/x86asm


The vendoring tool would be responsible for generating errors on conflicts or missing dependencies.


The thing that we as a community need to figure out is the recommended configuration file format.


We’d prefer something that the Go standard library can already parse easily. That includes XML, JSON, and Go.  Nobody likes XML, so that leaves JSON and Go.


godep already uses JSON, as do Go tools themselves (e.g. “go list -json fmt”), so we’ll probably want something like godep’s JSON format:


{

   "Deps": [

       {

           "ImportPath": "rsc.io/arm/armasm",

           "Rev": "616aea947362"

       },

       {

           "ImportPath": "rsc.io/x86/x86asm",

           "Rev": "af2970a7819d"

       }

   ]

}


We can start with that for discussion.


Note that we have rejected non-vendoring approaches that require modifications to GOPATH or new semantics inside the go command and toolchain. We believe it is important that the solution not require additional effort on the part of all the tools that already understand how to build, analyze, or modify code in the standard GOPATH hierarchy.


s...@box.com

unread,
Mar 3, 2015, 2:42:42 PM3/3/15
to golan...@googlegroups.com, brad...@golang.org, mediocr...@gmail.com
A JSON file is something that nearly anyone can read and modify trivially. Trying to understand and make changes to a .go file like this is far more challenging and would reduce the audience of people who could build tools to automatically maintain dependencies and resolve issues. I don't believe the lack of comments is enough to jettison such a machine- and human-friendly format.
 

Russ

Ian Davis

unread,
Mar 3, 2015, 3:15:32 PM3/3/15
to golan...@googlegroups.com
 
 
 
On Tue, Mar 3, 2015, at 07:20 PM, shado...@gmail.com wrote:
I think a better solution is this:
 
import "import/path/that/has/some/slashes#commit-like"
 
This cant work in general. Does the commit hash have to be on every import in every file that uses the package? What happens if two files in the same project specify different commits?
 
 
 
This has the advantage of also being reproducable in the filesystem:o
 
$GOPATH/src/import/path/that/has/some/slashes#commit-like"
 
Thoughts? This is much better than introducing the same versioning hell that Gem and NPM suffer from (not to mention Haskell's Cabal).


--
You received this message because you are subscribed to the Google Groups "golang-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+...@googlegroups.com.

Evan Shaw

unread,
Mar 3, 2015, 3:21:22 PM3/3/15
to Brad Fitzpatrick, golang-dev
I like the spirit of the proposal, but I don't like using "internal"
as the vendored code directory. I'd like to be able to separate my own
internal packages from vendored packages, and creating a "vendor"
subdirectory under "internal" makes an already long import path even
longer.

I understand and agree that we want to disallow importing vendored
dependencies from other projects, but it seems like "internal" is
being used because it already exists; not because it's the best name.

Peter Collingbourne

unread,
Mar 3, 2015, 3:44:09 PM3/3/15
to Russ Cox, Brad Fitzpatrick, mediocr...@gmail.com, golang-dev
Maybe something like

 import (
  “cmd/internal/rsc.io/arm/armasm” // "rsc.io/arm/armasm" @ 616aea947362
)

would work?

On Tue, Mar 3, 2015 at 7:31 AM, Russ Cox <r...@golang.org> wrote:
I like the idea of using a Go file, but note that it cannot be one that actually builds, because it needs to record the original import paths, not the vendored ones. I would expect that a Go config file would have a standard .go suffix (so that gofmt etc apply) but a build tag keeping it from building, a standard name (say, vendor.go), and a required package name (say, vendor):

---
// +build ignore

package vendor

import (
    // These are for the disassemblers.
    "rsc.io/arm/armasm" // 616aea947362
    "rsc.io/x86/x86asm" // af2970a7819d
)
---

Russ

Sam Dodrill

unread,
Mar 3, 2015, 3:48:58 PM3/3/15
to golan...@googlegroups.com
On Tue, Mar 03, 2015 at 08:15:25PM +0000, Ian Davis wrote:
>
>
>
>
> On Tue, Mar 3, 2015, at 07:20 PM, shado...@gmail.com wrote:
> > I think a better solution is this:
> >
> > import "import/path/that/has/some/slashes#commit-like"
>
> This cant work in general. Does the commit hash have to be on every
> import in every file that uses the package? What happens if two files in
> the same project specify different commits?

It's a commit-like so you get the advantage of specifying git tags, git
commits, svn commits, bzr commits/tags etc.

--

Sam Dodrill
shado...@gmail.com / xe...@yolo-swag.com
+1 425 221 7761

Dmitri Savintsev

unread,
Mar 3, 2015, 3:59:20 PM3/3/15
to golan...@googlegroups.com, brad...@golang.org, mediocr...@gmail.com
It looks rather clunky and ugly.  My vote would be to use Go or INI (http://en.wikipedia.org/wiki/INI_file format is well-known and should be easy to parse) and to avoid JSON (mainly due to the lack of comments but also because of the syntax generally not very friendly to a human reader or writer).

On Tuesday, March 3, 2015 at 4:56:51 PM UTC+1, ehedgehog wrote:
On 3 March 2015 at 15:26, Brad Fitzpatrick <brad...@golang.org> wrote:

> Good point. That's probably the strongest argument in favor of Go instead of
> JSON. I'm always annoyed by the lack of comments in JSON.

{"comment": "this is a comment", "otherField": 1066, ...}

{"comment": ["lacking multiline strings", "use an array"]}

Would that not suffice despite its clunkiness?

Chris

--
Chris "or is it 'clunkyness'?" Dollin

Michael Schuett

unread,
Mar 3, 2015, 4:14:46 PM3/3/15
to Dmitri Savintsev, golan...@googlegroups.com, brad...@golang.org, Brian Picciano
It seems like INI is coming up a lot as well as Go. I think I and a few others are pushing JSON but seem to be in the minority. I am curious to see what comes of this poll though if anyone is interested in participating as it's hard to get a great idea since some people in favor of a specific file type post a lot more than others. https://docs.google.com/forms/d/1y9jmJBHk2-mml1vwTvVVTYkSM-YgxkzmLU9A1zuL26w/viewform?usp=send_form thanks.

--
You received this message because you are subscribed to a topic in the Google Groups "golang-dev" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/golang-dev/nMWoEAG55v8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to golang-dev+...@googlegroups.com.

Ian Davis

unread,
Mar 3, 2015, 4:18:20 PM3/3/15
to golan...@googlegroups.com
On Tue, Mar 3, 2015, at 08:48 PM, Sam Dodrill wrote:
> > On Tue, Mar 3, 2015, at 07:20 PM, shado...@gmail.com wrote:
> > > I think a better solution is this:
> > >
> > > import "import/path/that/has/some/slashes#commit-like"
> >
> > This cant work in general. Does the commit hash have to be on every
> > import in every file that uses the package? What happens if two files in
> > the same project specify different commits?
>
> It's a commit-like so you get the advantage of specifying git tags, git
> commits, svn commits, bzr commits/tags etc.
>

I don't think I made my point clear. Surely the commit-like would have
to be specified every time the package is imported which could be spread
across dozens of source files. What happens when they get out of sync,
or are missed?

Ian

Ian Davis

unread,
Mar 3, 2015, 4:53:26 PM3/3/15
to Justin Scheiber, golan...@googlegroups.com
 
On Tue, Mar 3, 2015, at 09:46 PM, Justin Scheiber wrote:
I believe that's where tooling comes in.  Also, see Russ's comment earlier in the thread.  From what I guess, vendoring would look like this:
 
Yep I saw that, see my reply suggesting using go:generate to achieve the same effect.
 
Ian

Gustavo Niemeyer

unread,
Mar 3, 2015, 5:55:57 PM3/3/15
to Brad Fitzpatrick, golang-dev
It's great to see the approaches being successfully put to good use by the community in the last few years being commended as first-class solutions. Having a standard file format for that will be a very welcome improvement to these conventions.

At the same time, it's sad to see this being approached as an answer to questions about dependency management, when it doesn't solve any of the weak points of existing solutions. In fact, it's remarkably odd that "approaches that require [...] new semantics inside the go command and toolchain" were rejected on the path of answering questions about "how to deal with dependencies and their versions", when the former is the implementation of the latter.

The subject of this thread should likely have been "standardizing on a vendoring configuration file" instead.


--
You received this message because you are subscribed to the Google Groups "golang-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.



--

Karan Misra

unread,
Mar 3, 2015, 7:05:19 PM3/3/15
to golan...@googlegroups.com
Brad,

I had asked a very similar question to Dave and Francesc during the Q&A session at GopherConIndia 2015. So you can imagine my happiness on seeing this discussion thread. Thanks for bringing this up.

First of all, I have to say, that vendoring is truly the most pragmatic solution for reproducible builds in Go. Our compiler is super fast and we can afford to check in the source code of whatever open source libraries we want to incorporate into our product.

However, to truly come up with a solution which can be accepted by everyone, we must address the problem of versioning and vendoring separately, both for programs (the ones who should be VENDORING their dependencies' code) and libraries (who also are equal citizens in the Go world and are allowed to develop against a particular API of other open source libraries.)

Also, the solution to both these problems should ideally be handled by the proposed "go deps" tool so that the community can just move on from this long hotly debated argument and focus on generics instead (I kid! I kid!)

I am trying to draw inspiration from how other eco systems have approached this problem. I have worked extensively with both the Ruby/node.js ecosystems so have a feel for what works, and what doesn't. I might get lynched for saying this, but I liked the "package.json" concept followed in node.js. And my proposal is based around that idea.

Common Ground
  • We keep the version information in a .go file (why bring in a json/toml/ini file when we can use our beloved Go syntax to do the same). Lets call it deps.go
  • The format is standardized so that other tools can start coming up to better the dev experience even further
  • We adopt semver (works pretty well, and is sensible)
Libraries
  • Are allowed to have a deps.go file and specify the versions of their dependencies
  • Also specify their own version*
  • Are not allowed to actually vendor their dependencies. Instead they rely on the consuming programs (at the root of the graph) doing the vendoring / rewriting to hook things up
* creating tags to reflect own version should work

Programs
  • Have a deps.go file (if they want vendoring) which specifies the versions of dependencies they can work with
  • Vendor their dependencies (I quite like third_party, but can make peace with internal/)
The "go deps" command would be responsible for a few things:
  • Resolve the versions starting from the root deps.go file
  • Call out conflicts which could potentially cause multiple versions of the same package to be included as errors
  • Rewrite* code so that a single flat list of vendored libraries can be written out to internal/
The conflict resolution would take care of stopping situations like so:
  • Program includes pq
  • Program includes go-dockerclient
  • pq includes logrus v1 (a pretty awesome logging library, check it out!)
  • go-dockerclient includes logrus v2
Its natural for something like this to arise and the proper way to get this fixed would be to then send a PR upstream to pq to get its dependency updated to logrus v2. This also highlights the need to have proper versions (instead of just tags or hashes.) With versions, a situation like this could still work (with the Go compiler giving the final go ahead if everything compiles):
  • Program includes pq
  • Program includes go-dockerclient
  • Program includes logrus 2.1.4
  • pq includes logrus 2.0.0 (a pretty awesome logging library, check it out!)
  • go-dockerclient includes logrus 2.0.0
Semantic versioning dictates that (ideally) all v2.x.y versions of logrus should have a stable public API. So, "go deps" could resolve this by vendoring the actual specific version specified by the program (i.e 2.1.4) and counting on the deps.go file in both pq/go-dockerclient correctly specifying that they are okay with logrus v2.x.y. A "go build" would deliver the final verdict, but that is something which the dev would be doing anyway (as part of ensuring that all tests are still running green.)

As someone said earlier, Go is reaching that point where a rush of v2 libraries is imminent and to prevent chaos, it is important to have a solution which gives the ability of "versioning" dependencies to libraries as well.

I know I have glossed over a lot of details (how do you specify sliding dependencies, what format of deps.go, etc.) but I wanted to get the idea across beyond a REM sleep cycle wiped it clean from my head! And irrespective of what eventual solution emerges from this, I am really glad that this discussion is happening now.

On Rewriting? Could the go toolchain not automatically augment the GOPATH to include the current root deps.go "vendor" folder? With all the rewriting, the novelty of bringing in a new version of a dependency and seeing "what has he changed" sort of gets diminished. Its not the end of the world, but something to consider.

Regards,
Karan Misra

bit...@gmail.com

unread,
Mar 3, 2015, 7:19:56 PM3/3/15
to golan...@googlegroups.com
I like a few things about this proposal:

1. That we're settling on something for the community.

2. That it uses the internal mechanism.

Unfortunately I feel like there's several issues with this proposal:

1. The idea of using an external JSON file for configuration.
JSON was never meant for configs and it continues to be a poor choice.
The reasons should be obvious. For an example irritating trailing commas that unlike Go struct literals are not uniform in their
placement but one always has to be left off the last entry, and surrounding keys in quotations to be conformant.

Please consider TOML or the best choice: Use the Go source files. Everything in Go so far uses magic comments to achieve
it's goals (build tags, documentation) and I'm think it's best to not change this. As several have pointed out simply having the
commit hash/tag/branch at the end of the import path seems reasonable. It would require a backwards incompatible change to
how imports are defined but it very much fits the theme of struct tags etc.

2. That it leaves libraries on it's own, if we're going to half solve the problem and fragment how we write code today, I think
it's simply not a very good solution. "Are you writing package main? Do this! Are you writing anything else? Good luck!"
I guess I don't see the problem with retracting this proposal completely until something that satisfies all areas of Go development
is on the table while tools like godep and nut continue to hold the fort in a non-official way.

3. The overly long import paths. They're already an eyesore without adding another 4 directories of depth to them. Some proposals
here liken to shortening them. I think that if they have a version number attached to them ie:
Then they are automatically referred to from the vendored root of "currentpackage/internal". It is sort of magic which I agree is
bad in Go but the entire Go tool is convention over configuration so far, so a little more couldn't hurt.

pre...@gmail.com

unread,
Mar 3, 2015, 7:19:56 PM3/3/15
to golan...@googlegroups.com, pic...@gmail.com


On Tuesday, March 3, 2015 at 5:38:38 AM UTC-8, pic...@gmail.com wrote:
Why not instead just add support to Go for an optional tag or commit ID to the import string, along the lines of what Joshua Marsh is suggesting:

    import "github.com/johndoe/mylib 1.0"
    import "bitbucket.org/janedoe/another fe3b56a01ccde"

If it's not there, use the "master" branch.

    import "github.com/old/mylib" // uses master branch


Suppose those imports are in package A. Now package B imports "github.com/johndoe/mylib 0.9". And Package C depends on A and B. Which version of "github.com/johndoe/mylib" should be chosen when building package C?

Or let's say, B imports version 1.0.1. So, in theory, the dependency management tool should automatically pick v1.0.1 of mylib. But why? Can it be proven that 1.0.1 version of mylib is less buggy, and compatible with 1.0 clients? It can't be. Versions are fickle.

That's pretty much why not.

Justin Scheiber

unread,
Mar 3, 2015, 7:20:07 PM3/3/15
to Ian Davis, golan...@googlegroups.com
I believe that's where tooling comes in.  Also, see Russ's comment earlier in the thread.  From what I guess, vendoring would look like this:

A central file defining the dependencies: 

// +build ignore

package vendor

import (
    // These are for the disassemblers.
    "rsc.io/arm/armasm" // 616aea947362
    "rsc.io/x86/x86asm" // af2970a7819d
)

​Then use these packages like normal in your source:

import (
)

The tooling would be responsible for ensuring consistency, that the correct version is checked out, etc..

Mihai

unread,
Mar 3, 2015, 7:46:24 PM3/3/15
to Karan Misra, golan...@googlegroups.com
I don't think there is a common ground yet. Has the special directive (i.e. like go:generate or the vanity import) proposal already been ruled out in favour of a config file? 
   IMHO the config file adds friction compared with the comments/directives just like go:generate commands or the vanity import would do if they were placed in a build/config file. We already have the vanity import which enforces a custom import path so why don't we specify a specific version (be it commit hash or tag) the same way? It totally makes sense to me.

  Either way it would be great if someone with authority would make it clear if this thread is only about a potential config file and its specifications or if go:generate like directives are considered as well.


--
You received this message because you are subscribed to a topic in the Google Groups "golang-dev" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/golang-dev/nMWoEAG55v8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to golang-dev+...@googlegroups.com.

Peter Vessenes

unread,
Mar 3, 2015, 8:10:07 PM3/3/15
to golan...@googlegroups.com
Of all the proposals, the #commit-like append makes the most sense to me, so I'll try and answer your question:

It's easy enough to pick a behavior: 
   Allow them and import the two commits into separate directories on go get if they're not there.
   Alternately, deprecate this as a possibility, and require all be at the same commit level without a build directive.
   Alternately, only allow one commit-like across the package without renaming the import, the others just use the standard URL to refer

In the second and third case, go vet and goimports could help do the work of keeping things in sync. 

As a middlingly experienced Go Dev, I really hate the idea of adding cognitive load to the build process. If a build doesn't work for some reason, the idea that I now need to know where a special file is, how to check a special file, etc.. Seems evil to me. Go's cognitive load for compile/build/import is very, very low right now. I think it's a feature of the language. 

Adding a file like this seems like it will be a constant temptation to abuse into adding another layer to the process. 

Consider when adding the proposed file that every go debugging session and tutorial will need to mention that the wrong commit may be specified in a totally different place than your main code. Annoyances like that add up, in my opinion. They are easiest on the very smartest developers and hardest on the beginners and less smart. 

Andrew Gerrand

unread,
Mar 3, 2015, 8:40:21 PM3/3/15
to Peter Vessenes, golang-dev

On 4 March 2015 at 12:10, Peter Vessenes <pe...@coinlab.com> wrote:
As a middlingly experienced Go Dev, I really hate the idea of adding cognitive load to the build process. If a build doesn't work for some reason, the idea that I now need to know where a special file is, how to check a special file, etc.. Seems evil to me. Go's cognitive load for compile/build/import is very, very low right now. I think it's a feature of the language. 

I don't think anything about the build process changes. The go tool needn't know the file exists. This is just formalising something that people are already doing. With a standard file format we actually reduce complexity in the ecosystem.

jean...@gmail.com

unread,
Mar 4, 2015, 12:25:12 AM3/4/15
to golan...@googlegroups.com
Greetings,

one thing I did like in Go apart from others like Ruby or JS is, I don't need to write package.json or Gemfile. I like go get ./... and everything just work naturally 

jonath...@gmail.com

unread,
Mar 4, 2015, 12:25:25 AM3/4/15
to golan...@googlegroups.com, brad...@golang.org, mediocr...@gmail.com
If we're going to call the file vendor.go, the package vendor, and we're going to be using "vendored" as a verb so often, I can't help but think that we might as well be consistent and use "vendor" in the path instead of "internal."


On Tuesday, March 3, 2015 at 10:31:58 AM UTC-5, rsc wrote:
I like the idea of using a Go file, but note that it cannot be one that actually builds, because it needs to record the original import paths, not the vendored ones. I would expect that a Go config file would have a standard .go suffix (so that gofmt etc apply) but a build tag keeping it from building, a standard name (say, vendor.go), and a required package name (say, vendor):

---
// +build ignore

package vendor

import (
    // These are for the disassemblers.
    "rsc.io/arm/armasm" // 616aea947362
    "rsc.io/x86/x86asm" // af2970a7819d
)
---

Russ

cudmo...@gmail.com

unread,
Mar 4, 2015, 12:25:26 AM3/4/15
to golan...@googlegroups.com
We have had go get and in-source build flags since the beginning, and we have already seen the addition of go:generate meta commands in comments. I don't see why go get is not to be enhanced, and why a new config file will be added to the mix since in-source comments are already significant.

aos...@gmail.com

unread,
Mar 4, 2015, 12:35:42 AM3/4/15
to golan...@googlegroups.com
I'm late to this party but I've been doing this my own way for quite some time:

alias go='GOPATH=`pwd`:$GOPATH go'

This means you need a local src and pkg directory but at least everything is self contained.

Nathan Youngman

unread,
Mar 4, 2015, 1:32:34 AM3/4/15
to golan...@googlegroups.com, brad...@golang.org, mediocr...@gmail.com

I'm not sold on using a Go file to store SHA commit hashes. Perhaps this is just the way I use godep currently.

When working on an app, I use godep to save the state of the world inside my main source tree. I'm not manually editing the config file, just like I never would manually edit a Gemfile.lock in Ruby-land (excepting merge conflicts :-( ).

When patching dependencies, I leave my hermetically sealed bubble and work on a feature branch of that library. This allows me to push my changes upstream for hopeful inclusion (yay, collaboration, win!). I treat the Godeps subdirectory (now internal/github.com/*) as "DO NOT EDIT", simply a snapshot of work I did out in the world. This particular workflow makes "godep restore" quite necessary (implying bidirectional rewriting of import paths).

But the point is, I'm not mucking about with SHA commit hashes, that's what we have tools for. I'm also not providing tags or branches in some config file. If I want to vendor a particular branch, I use commands like "cd" and "git checkout" and then take a snapshot. 

For me, this is the simplest thing that could possibly work, but no simpler.

Mind you, I'm relatively new to this whole concept of copying third-party libraries into an internal/ folder. I'm sure I have much yet to learn, especially when applied in large teams.

Nathan.

Brian Picciano

unread,
Mar 4, 2015, 1:40:39 AM3/4/15
to Nathan Youngman, brad...@golang.org, golan...@googlegroups.com

> But the point is, I'm not mucking about with SHA commit hashes, that's what we have tools for

My understanding is that we're primarily discussing the format of the file the vendor information is stored in and the basic details of how vendoring will work. It's likely that tools will sprout up around this that will allow for what you describe, regardless of what format is chosen.

zond...@gmail.com

unread,
Mar 4, 2015, 1:43:40 AM3/4/15
to golan...@googlegroups.com
I also really like using the string in the import declaration to define version, but why add non path compatible characters?

Why not use

import (
"github.com/a/b/ver-1."
)

Then you could use 'go get', or a completely new tool that:

- Downloads the latest version of 'github.com/a/b' to with a tag that begins with 'ver-1.' to '$GOPATH/src/github.com/a/b/ver-1.'.
- Rewrites all internal imports inside it to use the imported path.
- Uses itself to download all of the missing deps of the new package (thus downloading all missing deps, with correct version, of the new package).
- Is fully compatible with regular import paths to support packages depending on regular 'go get'able paths.

This would solve the problem of A depending on B depending on ver 2 of C, and A depending on D depending on version 1 of C, because they would, for the go toolchain, be two separate packages.

It would also allow package maintainers to fix non API breaking bugs in new minor versions, while introducing API breakage in new major versions.

It would not, of course, remove the problem of this package being included twice on drive and in the binary, but no other suggestion solves that either.
It is loading more messages.
0 new messages