Vendoring edge case, critical problem

[Peter Bourgon]

This is a successor to a previous unresolved post on golang-dev[0]. I
originally posted my problem on golang-nuts[1].

Concretely, I can't implement an interface[2] from coreos/etcd in my
own library, because the interface includes a type that the repo
vendors, and it's impossible for me to create an instance of that
type, because the vendored package is inaccessible to me. Here[3] is a
self-contained demonstration of the problem.

One possible solution is to add vendoring to my repo, vendor
coreos/etcd, and flatten all of its transitive dependencies. But
well-behaved libraries should not vendor their deps, so this isn't
viable.

Another possible solution is to ask etcd to change their repo somehow.
They could stop vendoring the context package, or define their own
context package that I can import, or split coreos/etcd into separate
etcdbin (vendored) and etcdlib (not vendored) repos. But these seem
like nuclear options, unwarranted given that coreos/etcd has a
reasonable layout, i.e. several binaries and libraries that are
closely related.

It might appear that another possible solution could be having lib/
and cmd/ subdirectories for libraries and binaries respectively, and
moving the vendor/ folder to cmd/vendor/. But this doesn't work, see
[4] for a self-contained demonstration.

I'm not aware of any other possible solution. Thus, this seems like a
critical problem.

I've done [5] in order to make forward progress on my library, and
I'll have to recommend users of my library vendor+flatten to use it.
But this is extremely awkward. Frankly, it's enough to put people off
of the language and its ecosystem.

I think Go must allow packages to vendor code, and other packages to
import them without also having to vendor. Or, said differently: I
should be able to import a package from a repo that happens to have a
vendor folder, without also requiring my repository to have a vendor
folder & flatten the transitive vendored dependencies.

Thoughts appreciated.

[0] https://groups.google.com/d/msg/golang-dev/WebP4dLV1b0/Lhk4hpwJEgAJ
[1] https://groups.google.com/forum/#!topic/golang-nuts/AnMr9NL6dtc
[2] https://github.com/coreos/etcd/blob/b324735/etcdserver/etcdserverpb/rpc.pb.go#L1749
[3] https://github.com/peterbourgon/wtf
[4] https://github.com/zellyn/wtf2
[5] https://github.com/weaveworks/mesh/commit/8aabe08

Cheers,
Peter.

[Brad Fitzpatrick]

Wow, that does look broken.

--
You received this message because you are subscribed to the Google Groups "golang-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[David Crawshaw]

The repository coreos/etcd is home to both binaries and libraries. It
vendors its dependencies, which is correct for its binaries, but
incorrect for its libraries.

Here is a modification of approach [4] that is bizarre at first glance
but I believe will work: the repository's binaries can vendor its own
libraries. That is,

cmd/main.go
cmd/vendor/golang.org/x/net/context
cmd/vendor/lib # copy of lib
lib/lib.go

[David Symonds]

I think the coreos/etcd package is at fault. It would be in the same
situation if its external API mentioned types from an internal/
package, or mentioned unexported types. Names from vendored packages
are in the same situations as those, and should be avoided in APIs.

[Dave Cheney]

This sounds like a very hard rule to explain to newcomers.

For example, it _is_ ok to write a function

package p

import "golang.org/x/net/context"

func NewContext() context.Context

Iff the repo that contains package p _does not_ have a copy of
golang.org/x/net/context in it's vendor/.

How are newcomers supposed to know which repositories are ok to vendor
and which are not ? How can they stop themselves from moving from
having a floating dependency on golang.org/x/net/context to vendoring
that dependency without realising how it will affect the downstream
consumers of package p ?

Will go vet spot this kind of hazzard ?

[David Symonds]

I think one needs to be aware of the types in one's public API, and
that's the responsibility of whoever owns that API. As you point out,
the vendoring structure is a little sneaky in that it'll switch what
import statements mean without any visible effect on the source files.
That means that you need to factor in any vendoring you've done to
your package when considering your package's API.

I guess go vet could spot this and flag it as a problem. Golint flags
exported APIs using unexported types, so it could be done there
instead. It is a "gotcha" as far as vendoring is concerned.

[Brad Fitzpatrick]

On Tue, Apr 5, 2016 at 4:41 PM, David Crawshaw <craw...@golang.org> wrote:

The repository coreos/etcd is home to both binaries and libraries. It
vendors its dependencies, which is correct for its binaries, but
incorrect for its libraries.

Here is a modification of approach [4] that is bizarre at first glance
but I believe will work: the repository's binaries can vendor its own
libraries. That is,

cmd/main.go
cmd/vendor/golang.org/x/net/context
cmd/vendor/lib  # copy of lib
lib/lib.go

Ugh, so I can't hack away in one repo?

I think Camlistore also gets this wrong, since we have context in our APIs, and our pkg/ and cmd/ are both beside our vendor/.

And I'm always afraid to use symlinks with cmd/go, but maybe it's improved.

[Brad Fitzpatrick]

Yeah, there's definitely some room for tooling here to tell people they're doing the wrong thing.

[Peter Waller]

On 6 April 2016 at 05:09, Brad Fitzpatrick <brad...@golang.org> wrote:

Ugh, so I can't hack away in one repo?

This is my reaction too. 

To reiterate a point I made on the nuts thread: there are many more consumers of go code than producers. The "choice" of whether to get this right lies with the producer, and asking the producer to fix the problem is the nuclear solution in many cases, as Peter B suggested. This has already blown up in my face once.

It means that my choice as an importer wanting to use git submodules and the standard tools without code rewriting is restricted to those packages which happen to get this right. There are plenty of high profile projects which get this wrong.

[Daniel Theophanes]

The solution coreos/etcd came up with involves symlinks and can be found here: https://github.com/coreos/etcd/pull/4950 .

Of course their solution will result in the same package being compiled twice, once in the context of vendor, the other in the context of just GOPATH.

I agree there is probably room to improve the diagnostics. A part of me would also like to explore ways to enable the above coreos/etcd solution without symlinks.

[David Crawshaw]

On Wed, Apr 6, 2016 at 10:48 AM, Paul Jolly <pa...@myitcv.org.uk> wrote:
>> ...which is correct for its binaries, but
>> incorrect for its libraries.
>
>
> I believe this is the key statement.
>
> David - is this guidance on vendoring libraries vs binaries formalised
> anywhere?

Not that I know of. I don't think there are definitive answers yet.

[Daniel Theophanes]

I also don't know of any official guidance. 

For govendor users, this is the guidance I have: https://github.com/kardianos/govendor/blob/master/doc/dev-guide.md

--
You received this message because you are subscribed to a topic in the Google Groups "golang-dev" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/golang-dev/4FfTBfN2YaI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to golang-dev+...@googlegroups.com.

[minux]

On Mon, Apr 4, 2016 at 1:52 PM, Peter Bourgon <pe...@bourgon.org> wrote:

This is a successor to a previous unresolved post on golang-dev[0]. I
originally posted my problem on golang-nuts[1].

Concretely, I can't implement an interface[2] from coreos/etcd in my
own library, because the interface includes a type that the repo
vendors, and it's impossible for me to create an instance of that
type, because the vendored package is inaccessible to me. Here[3] is a
self-contained demonstration of the problem.

 

[0] https://groups.google.com/d/msg/golang-dev/WebP4dLV1b0/Lhk4hpwJEgAJ

[1] https://groups.google.com/forum/#!topic/golang-nuts/AnMr9NL6dtc
[2] https://github.com/coreos/etcd/blob/b324735/etcdserver/etcdserverpb/rpc.pb.go#L1749
[3] https://github.com/peterbourgon/wtf

What if we make cmd/go allowing import of "path/to/package/vendor/pkg"?
That will solve the problem without modifying the source code organization of

etcd in this case.

I know that cmd/go explicitly disallows such imports, but what would be the

implications if we allow it?

My thinking is that, unlike internal packages, vendored packages are not

private (unless it's placed in internal/vendor directory, which means it's

privately vendored packages.)

[Daniel Theophanes]

The problem that I see with that is that then all clients would need to use the same path. If would be highly susceptible for a single package or even a single file to use the wrong import path. If you did what you mentioned, you use a lib that used an un-prefixed version. Even worse you would be back to needing to rewrite import paths... (I don't hold that import path rewriting is bad by itself, but I don't relish it in combination of the vendor folder.)

--

[Paul Jolly]

...which is correct for its binaries, but
incorrect for its libraries.

I believe this is the key statement. 

David - is this guidance on vendoring libraries vs binaries formalised anywhere?

Here is a modification of approach [4] that is bizarre at first glance
but I believe will work: the repository's binaries can vendor its own
libraries. That is,

cmd/main.go
cmd/vendor/golang.org/x/net/context
cmd/vendor/lib  # copy of lib
lib/lib.go

We tried exactly this approach (worth noting none of our binaries are public). But then ended up in a rather messy state with the many **/cmd/*/vendor directories all over the place which:

a) were hard to manage and

b) contained much duplication

At which point we shifted to augment our GOPATH when working within our source code tree:

$ pwd

/path/to/gopath/src/ourdomain.com/x

$ echo $GOPATH

/path/to/gopath/src/ourdomain.com/x/_vendor:/path/to/gopath

We're writing some tooling to handle the "vendoring" in the _vendor directory.

For binaries which are publicly distributed, the vendor approach works well (and is the only option if you want things to be `go get`able, i.e. the GOPATH solution is not guaranteed to work)

The two approaches are clearly not mutually exclusive so if any of our binaries were ever to be made publicly available we could vendor those.

[samuel....@gmail.com]

I worked up a little example of something like this (I believe) here: https://github.com/sdboyer/nesttest

Yep - a happy little jaunt on the road to absurdity.

[roger peppe]

Part of the issue I think is that vendoring becomes viral.

The idea of the vendor spec as designed is that, in Russ's
words [0], "you as the consumer of the library can use any vendor-spec-aware
tool to hoist those dependencies up higher."

Assuming they use a standard vendor spec, it should
be possible to get around Peter's problems by vendoring
everything - if you want to use etcd, you would vendor
the entirety of it, removing all its vendor directories and
moving them directly into your own vendor directory.

So we end up in a situation where in order to use any [1]
package that contains a vendor directory, we must
vendor it ourselves.

[0] https://groups.google.com/d/msg/golang-dev/xuDb2ulFgjU/LZWdV7ynEgAJ
[1] well, almost any - packages that don't expose any vendored types in their
API can sometimes get away with it.

[samuel....@gmail.com]

I, for one, think this issue just reinforces that tooling assistance is really the only way that use of the vendor directory makes any sense. If you're not vendor-hoisting (or flattening, as we've tended to call it) deps from nested vendor directories up into the topmost vendor 'owned' by you're own project, then this problem is just one of several; you're probably going to have a bad time.

In theory, this could be helped by changes to go/build that allow only a single level of vendoring...but I suspect there'd be plenty of gotchas hiding in there, as well. Thus, avoiding these problems is pretty much tooling's job. (In fact, while this problem sucks in the short term, changes that disallowed vendor nesting would actually cut off some useful choices to tooling in the longer term.)

And I know I'm not alone in working fervently towards improving tooling.

[l...@leetrout.com]

I'm quite interested in this discussion. I'm pretty new to Go and I don't want to drag this off topic but it appears I hit the same or a similar issue.

Specifically I was experimenting with vendoring and I added it to 2 projects, let's just call them A & B, and A also vendored B. Both projects vendored lib/pq and lib/pq's init was called twice resulting in a panic. It appears the flattening approach would work and also seems that the prevailing though here is that I have misused vendoring. A comment on the PR I submitted to lib/pq[1] made a lot of sense and lines up with that is being said here about vendoring for the binaries not the libraries.

As a Go noob spoiled by all the great tooling I've encountered so far, here's a big +1 for tooling or better docs / guidance on vendoring properly.

[1] https://github.com/lib/pq/pull/455

[Dan Kortschak]

What we really have is a form of the cosmological censorship principle (interestingly for similar reasons). It's not just that we can't expose vendored types, but rather that we may not have any externally observable behaviour from a vendored package - external resource side effects also need to be considered.

[Paul Jolly]

At which point we shifted to augment our GOPATH when working within our source code tree:

$ pwd

/path/to/gopath/src/ourdomain.com/x

$ echo $GOPATH

/path/to/gopath/src/ourdomain.com/x/_vendor:/path/to/gopath

We're writing some tooling to handle the "vendoring" in the _vendor directory.

For binaries which are publicly distributed, the vendor approach works well (and is the only option if you want things to be `go get`able, i.e. the GOPATH solution is not guaranteed to work)

The following repo:

https://github.com/myitcv/go-vendoring

attempts the following:

1. vendor (in the Go 1.5 Vendor Experiment definition of the word) for each public binary (there is only one in this case, github.com/myitcv/go-vendoring/cmd/a) such that they are `go get`-able
   
2. a "vendor" (not the Go 1.5 Vendor Experiment definition of the word) of all external dependencies (of both binaries and libraries) for developers of the repo (allow for reproducible builds etc)
   
3. no duplication of copies of external dependencies, i.e. 1 and 2 share code (easier to maintain)
4. allows the "vendor"-ed code to be hoisted if required by a consumer of github.com/myitcv/go-vendoring/{mylib1,mylib2}

This is achieved by augmenting `$GOPATH` when working within the repo to:

/my/go/path/src/github.com/myitcv/go-vendoring/_vendor:/my/go/path

and the following directory structure (with symlinks):

https://github.com/myitcv/go-vendoring/wiki/Directory-Structure

Unfortunately, with this augmented `$GOPATH` we see the following panic in cmd/go when running `go test ./...` in /my/go/path/src/github.com/myitcv/go-vendoring:

https://github.com/golang/go/issues/15201

But I wonder whether this sort of structure doesn't give us the best of both worlds (ignoring for one second how `$GOPATH` is augmented, I use smartcd)

Symlinks on a `$GOPATH` appear to work, and well; it's just in this situation there is effectively a cycle which is I guess what gives rise to this panic.

Does this approach hold water?

Is it even valid to symlink in this way?

[Konstantin Shaposhnikov]

I wonder if this approach with symlinks will work on Windows.

[Paul Jolly]

On 9 Apr 2016 09:43, "Konstantin Shaposhnikov" <k.shapo...@gmail.com> wrote:
>
> I wonder if this approach with symlinks will work on Windows.

*sigh* this totally slipped my mind

[Ian Davis]

This is a very common occurrence though. For example the recently
released Doorman project from youtube[1] vendors grpc[2] and exposes its
types in public APIs, e.g. setting options for a connection [3]

[1]https://github.com/youtube/doorman
[2]https://github.com/youtube/doorman/tree/master/vendor/google.golang.org/grpc
[3]https://github.com/youtube/doorman/blob/master/go/connection/connection.go#L97

[atd...@gmail.com]

Isn't it possible to expose, in the vendoring package, a type constructor ?

Seems that it's the generic case of exporting a type transitively (not necessarily a vendoring issue)

[Mateusz Czapliński]

On Thursday, April 7, 2016 at 1:09:39 AM UTC+2, Paul Jolly wrote:

At which point we shifted to augment our GOPATH when working within our source code tree:

$ pwd

/path/to/gopath/src/ourdomain.com/x

$ echo $GOPATH

/path/to/gopath/src/ourdomain.com/x/_vendor:/path/to/gopath

We're writing some tooling to handle the "vendoring" in the _vendor directory.

If you care enough, have a look if the following tool could possibly help you (and save time) with managing the _vendor/ directory:
  https://github.com/zpas-lab/vendo
Sorry that it doesn't have a proper README at this point. We managed to open-source the tool on a brink of us getting buried under tons of more important work (still applies), and also got a kind of a pushback after realizing the "kardianos/vendor-spec" repo (a revision of which we dutifully adhered to) became a moving target at some point. But I quickly dumped the output of --help now, to give you some idea how to use it:
  https://gist.github.com/akavel/dba956bc5595356d67c08759c41962d0
plus there's a design doc which outlines the use cases:
  https://github.com/zpas-lab/vendo/blob/master/use-cases.md

We use it internally with success. Some quick overview:
- the initial command to use is e.g.: "vendo recreate -platforms=linux_amd64,windows_amd64"
- walks your dependency tree and copies all git/hg/bzr repos from regular GOPATH into your repo, into "_vendor/src", automatically performing "git add" on them, skipping (ignoring) .git/.hg/.bzr dirs, as well as detecting when you stop using a dependency;
- tracks upstream revision IDs and dates in "vendor.json";
- allows updating selected packages ("vendo update" - analogue of "go get -u");
- "vendo check" (intended to be used as a git pre-commit hook) is Not Implemented Yet;
- it supports patching vendored repos (described in "use-cases.md", p.7).

In case of questions, you're more than welcome to contact me at czap...@gmail.com.

Also note it is not a response to the main problem mentioned in this thread ("vendor" contents interlocking), just to one sentence by Paul Jolly (managing _vendor/ dir); sorry for OT, but wanted to keep it public in case anyone else got interested.

/Mateusz.