proposal: external packages

14398 views
Skip to first unread message

Keith Rarick

unread,
Oct 4, 2014, 1:44:59 AM10/4/14
to golang-dev
Here is a pseudo-proposal to change the behavior of the go tool. I
realize this is coming out of left field, so I won't be offended if
the change itself not given serious consideration. However, I'm hoping
the underlying problem will be, and I'd really like to get some fresh
perspective and feedback on how best to address/solve/avoid it.

https://docs.google.com/document/d/1CJnU6ZKvsp21B0lQwbJlKFt8Zz4EWscaCRy_EwK8ja8

There's a similar document at
https://github.com/tools/godep/issues/127 that does not delve into the
possibility of changing the go tool.

Keith Rarick

unread,
Oct 4, 2014, 2:16:52 AM10/4/14
to golang-dev
(Observant readers will note that the proposed behavior is essentially
what godep itself does today as a wrapper for the go tool. The
existing behavior in godep is bad because it requires end users to
have godep installed to build commands reproducibly. It is preferable
for commands to be "go-gettable" with the stock go toolchain, while
still keeping reproducibility. Moving godep's search path-altering
behavior into the go tool would fix that objection, if only by brute
force.)

Viktor Kojouharov

unread,
Oct 6, 2014, 5:15:08 AM10/6/14
to golan...@googlegroups.com
Would it be possible to instead create a new comment, similar to the build constants or go:generate, which specifies a package version, possibly in a manner similar to bower. Maybe the go get tool could also be extended with that syntax as well.
Something like: // go:dep code.google.com/p/go.net/context#~1.2.3
That way you don't need to drag dependency code around.

Keith Rarick

unread,
Oct 10, 2014, 7:04:44 PM10/10/14
to golang-dev
Dave Cheney asked, doesn't godep solve u's problem, by "rolling up" or
"flattening" the dependencies to avoid paths of the form
".../ext/.../ext/.../ext/..."? It does, if u is using godep. But I
think it's unreasonable to require u to use godep.

For one thing, u might be a non-main package. For another, even if u
is a command, this problem makes it hard to get started. I can't ask
the author to run 'godep save' to vendor their dependencies before
they've written the first line of code.

Keith Rarick

unread,
Jan 18, 2015, 11:38:18 PM1/18/15
to golang-dev
A while ago, Andrew suggested to me that asking about this
at the beginning of October was poor timing, since everyone
was busy with Go 1.4, and that I might have better luck once
general work on 1.5 started. So I'd like to try to reopen this
discussion now.

I'd love to get feedback:

- Do you think that the problem described in the linked
Google doc is really a problem?

- Do you think my suggestion is reasonable?

- Do you have any other ideas?

Egon Elbre

unread,
Jan 19, 2015, 2:52:20 AM1/19/15
to golan...@googlegroups.com
On Monday, 19 January 2015 06:38:18 UTC+2, Keith Rarick wrote:
A while ago, Andrew suggested to me that asking about this
at the beginning of October was poor timing, since everyone
was busy with Go 1.4, and that I might have better luck once
general work on 1.5 started. So I'd like to try to reopen this
discussion now.

I'd love to get feedback:

- Do you think that the problem described in the linked
Google doc is really a problem?

Yes and no.

Yes, in the sense that is happening and will happen in the future.
No, that I don't know how common it is. 
Or whether leaving it "unsolved" and requiring people to think about the problem and find the best solution is a better approach?


- Do you think my suggestion is reasonable?

It seems reasonable; but it is confusing.
 
- Do you have any other ideas?

The "flattening" seems simpler, but of course carries it's own problems:

(by "flattening" I mean) go build will always implicitly add "external" to $GOPATH. And you have a command "go vendor" to pull in all the dependencies into "external" (by flattening and removing any nested "external"-s). Any conflicts must be solved manually.

This approach has the problem that when you have imported a package "foo" that has imported "foo/external/bar" and you have "bar" in your GOPATH, then your code may not build - (i.e. even if there isn't an actual conflict of packages.)

+ Egon

Daniel Theophanes

unread,
Jan 19, 2015, 1:58:36 PM1/19/15
to golan...@googlegroups.com
Hi Keith,

I use Godep at work; I like how it works and the concept of external packages. For us it solves a very real problem.

In your proposal the external folder must be under the command folder. I have a project that builds multiple commands using the same vendor packages:
/proj/cmd/cmdA
/proj/cmd/cmdB
proj/cmd/cmdC
/proj/projPkgA
/proj/external/otherPkgA
proj/external/otherPkgB

When building cmdA, the go command (and other tooling) need to walk down the folder tree once to see if it contained a magically named "external" folder. This would need to happen not only for building main packages, but also for any and every package that is analyzed with tools or build system. It also follows that any external package that is built within the "external" folder should also search for packages first in the "external" folder (after GOROOT).

The cost would be:
 1. Stat many non-existing "external" folders.
 2. Tools would be more complex.
 3. Knowing what code an import refers to becomes more complex.

However: (1) the disk cost could cache information each run. (3) Projects that don't live in a global source control system where code gets vendored somehow, there is going to be an extra complexity regardless of whether the tooling supports it or not.

My only other idea is to somehow support a source and binary archives that can be linked in (think Java jars or .Net DLLs).

So as a developer, I support this proposal.
-Daniel

Chris Hines

unread,
Jan 20, 2015, 10:37:02 AM1/20/15
to golan...@googlegroups.com
Generally speaking I agree with the goal of making it easier to have reproducible builds.

Have you given any thought to how this feature would impact the predictability of `go test`? I use the package names from the document.

My reading of the document implies that `go test mypkg/p` would import foo. Even if all the tests pass we cannot feel confident that when we run `go install mypkg/c` because that will import mypkg/external/foo. We cannot assume that `go test mypkg/c` will fully test mypkg/p, therefore we must have a way to test mypkg/p when linked to mypkg/external/foo.

On the other hand, if `go test mypkg/p` imports mypkg/external/foo, then we cannot test mypkg/p when linked with foo?

Do we add a flag to `go test` to enable or disable searching for external imports?

Chris

Andrew Gerrand

unread,
Jan 22, 2015, 10:29:23 PM1/22/15
to Keith Rarick, golang-dev

On 19 January 2015 at 15:37, Keith Rarick <k...@xph.us> wrote:
- Do you think that the problem described in the linked
Google doc is really a problem?

I am worried about this problem; the preferred behaviour is context-dependent. I don't think there is a good solution. It does seem very weird that two packages within the same program could see different versions of a package "foo".

I am also worried that "external" adds more unpredictability to something people are already having some trouble with (resolving package names across multiple GOPATHs).

Russ Cox

unread,
Jun 11, 2015, 9:21:06 AM6/11/15
to Keith Rarick, golang-dev

Thanks, Keith. I agree with you that the go command should do this for Go 1.5, as an experiment guarded by a -vendor flag.


I apologize for the delay in responding. I wanted to let both this discussion and the broader conversation proceed, especially since the community understands and encounters these problems more than we on the Go team at Google do. This has happened: we now have much more information from the community about the problem being solved, both in descriptions and in code, like godep, nut, and gb.


We have been rereading the discussions here and looking at the landscape of both Go vendoring tools and large projects that use them. One fact is clear: people very much want to copy (“vendor”) source code into their projects without modifying import paths. Godep and similar tools do this with GOPATH manipulation, and gb does it by reimplementing the build system. In both cases, the result is not compatible with “go get”: people using standard tools cannot easily install projects managed with tools like gb and godep. The proposal in this thread can help the go command meet these tools halfway, so that, perhaps along with minor changes to gb, godep, and friends, users will be able to “go get” those projects.


Specifically, we propose that, as an experiment for Go 1.5, we add a temporary “-vendor” flag that causes the go command to add these semantics:

If there is a source directory d/vendor, then, when compiling a source file within the subtree rooted at d, import "p" is interpreted as import "d/vendor/p" if that exists.


When there are multiple possible resolutions, the most specific (longest) path wins.


The short form must always be used: no import path can contain “/vendor/” explicitly.


Import comments are ignored in vendored packages.


This is a limited form of the original proposal in this thread.


In the original proposal (as I understand it), the interpretation of an import depends both on where the source code containing that import sits in the tree and why the package is being compiled. The meaning of an import in package X is different when X is imported by Y than when X is imported by Z. Therefore the compilation would have to happen multiple times. This doesn’t scale and possibly leads to inconsistent constraints.


In the revised proposal, the interpretation of an import depends only on where the source code containing that import sits in the tree.


The revised proposal also uses the word “vendor” instead of “external”, because (1) there is at least one popular vendoring tool (gb) that uses “vendor” and none that we know of that use “external”; (2) “external” sounds like the opposite of “internal”, which is not the right meaning; and (3) in discussions, everyone calls the broader topic vendoring. It would be nice not to bikeshed the name.


As an aside, the terms “internal vendoring” and “external vendoring” have been introduced into some discussions, to make the distinction between systems that rewrite import paths and systems that do not. With the addition of vendor directories to the go command, we hope that this distinction will fade into the past. There will just be vendoring.


Example


The gb project ships an example project called gsftp. It has a gsftp program with three dependencies outside the standard library: golang.org/x/crypto/ssh, golang.org/x/crypto/ssh/agent, and github.com/pkg/sftp.


Adjusting that example to use the new vendor directory, the source tree would look like:


$GOPATH

| src/

| | github.com/constabulary/example-gsftp/

| | | cmd/

| | | | gsftp/

| | | | | main.go

| | | vendor/

| | | | github.com/pkg/sftp/

| | | | golang.org/x/crypto/ssh/

| | | | | agent/


The file github.com/constabulary/example-gsftp/cmd/gsftp/main.go says:


import (

...

"golang.org/x/crypto/ssh"

"golang.org/x/crypto/ssh/agent"

"github.com/pkg/sftp"

)


Because github.com/constabulary/example-gsftp/vendor/golang.org/x/crypto/ssh exists and the file being compiled is within the subtree rooted at github.com/constabulary/example-gsftp (the parent of the vendor directory), the source line:


import "golang.org/x/crypto/ssh"


is compiled as if it were:


import "github.com/constabulary/example-gsftp/vendor/golang.org/x/crypto/ssh"


(but this longer form is never written).


So the source code in github.com/constabulary/example-gsftp depends on the vendored copy of golang.org/x/crypto/ssh, not one elsewhere in $GOPATH.


In this example, all the dependencies needed by gsftp are (recursively) supplied in the vendor directory, so “go install” does not read any source files outside the gsftp Git checkout. Therefore the gsftp build is reproducible given only the content of the gsftp Git repo and not any other code. And the dependencies need not be edited when copying them into the gsftp repo. And potential users can run “go get github.com/constabulary/example-gsftp/cmd/gsftp” without needing to have an additional vendoring tool installed or special GOPATH configuration.


The point is that adding just the vendor directory mechanism to the go command allows other tools to achieve their goals of reproducible builds and not modifying vendored source code while still remaining compatible with plain “go get”.


Discussion


There are a few points to note about this.


The first, most obvious, and most serious is that the resolution of an import must now take into account the location where that import path was found. This is a fundamental implication of supporting vendoring that does not modify source code. However, the resolution of an import already needed to take into account the current GOPATH setting, so import paths were never absolute. This proposal allows the Go community to move from builds that require custom GOPATH configuration beforehand to builds that just work, because the (more limited) configuration is inferred from the conventions of the source file tree. This approach is also in keeping with the rest of the go command.


The second is that this does not attempt to solve the problem of vendoring resulting in multiple copies of a package being linked into a single binary. Sometimes having multiple copies of a library is not a problem; sometimes it is. At least for now, it doesn’t seem that the go command should be in charge of policing or solving that problem.


The final point is that existing tools like godep, nut, and gb will need to change their file tree layouts if they want to take advantage of compatibility with “go get”. However, compatibility with “go get” used to be impossible. Also, combined with eventual agreement on the vendor-spec, it should be possible for the tools themselves to interoperate.


Deployment


The signals from the Go community are clear: the standard go command must support building source trees in which dependencies have been vendored (copied) without modifications to import paths.


We are well into the Go 1.5 cycle, so caution is warranted. If we put off making any changes, then vendoring tools and “go get” will remain incompatible for the next eight months. On the other hand, if we can make a small, targeted change, then vendoring tools can spend the next eight months experimenting and innovating and possibly converging on a common file tree layout compatible with the go command.

We believe that the experimental proposal above is that small, targeted change. It seems to be the minimal adjustment necessary to support fetching and building vendored, unmodified source code with “go get”. There are many possible extensions or complications we might consider, but for Go 1.5 we want to do as little as possible while remaining useful.


The use of an explicit -vendor flag should help contain the experiment now and ease the transition later. For Go 1.5, users who want to participate in the experiment can opt in by using “go get -vendor”, “go install -vendor”, and so on.


The new semantics changes the meaning of (breaks) source trees containing directories already named “vendor”. Of the over 60,000 listed on godoc.org, there are fewer than 50 such examples. Putting the new semantics behind the -vendor flag avoids breaking those trees for now.


If we decide that the -vendor behavior is correct, then in a later release (possibly Go 1.6) we would make the -vendor flag default to true. Projects containing “vendor” directories could still use “-vendor=false” to get the old behavior while they convert their code. In a still later release (possibly Go 1.7) we would remove the -vendor flag, locking in the vendoring semantics.


Comments welcome here. I just sent CLs 10922 and 10923 as a prototype.

Thanks very much for this proposal.
Russ

C Cirello

unread,
Jun 11, 2015, 10:38:02 AM6/11/15
to golan...@googlegroups.com, k...@xph.us
Hi Russ,


I am not 100% sure if I am mixing up ideas - but especially in the case of gb, not only it addressed vendoring, but also multi-target compilation (such having several cmd/ files all compiled in a single `gb build` call), and also added plugins to facilitate source code download (gb vendor fetch|update|purge etc).

Are these extra functionalities going to be added some time in the future? 


Regards,
Carlos

Brad Fitzpatrick

unread,
Jun 11, 2015, 11:00:16 AM6/11/15
to C Cirello, golang-dev, Keith Rarick
You can already do multi-target compilations with the go tool:

$ go install ./cmd/... 

As for go tool subcommands to update 'vendor' directories: those could come later (Go 1.6+), but not in Go 1.5.


--
You received this message because you are subscribed to the Google Groups "golang-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Nate Finch

unread,
Jun 11, 2015, 11:09:33 AM6/11/15
to golan...@googlegroups.com
I think this is exactly the right approach.

Brendan Tracey

unread,
Jun 11, 2015, 11:17:36 AM6/11/15
to golan...@googlegroups.com, k...@xph.us
Sorry, accidentally sent off list:

As you say, this means the resolution of an import now depends on where the path is found. A second implication is that it removes one of my favorite properties about Go; in the current environment I can read an import statement and know exactly where the code is. This issue is clearly very important for the community, which may merit breaking this invariant. I would be sad to see more changes that obscure the link between import path and file location.

Toward this end, many users use goimports, which automatically orders and formats imports. Could goimports help legibility by organizing the imports?

import (
   “fmt”
   “log”

   // vendored packages (comment could be explicit or not)
   “github.com/pkg/sftp
   “github.com/x/crypto/ssh

   // non-vendored
   sftp2 “github.com/pkg/sftp
  “github.com/gonum/floats
)
 

Why is this style of vendoring chosen? I don’t wish to bike shed, but this proposal seems to take per-package vendoring as a given and discusses how that should happen. For example, why not specify a specific version of a package (by commit hash or something), and then have a package with that hash. Then there could be a $GOPATH/vendor that contains all of the vendored packages. This would scale better when there is a single vendored package imported by many other packages, and could help (though not solve) the duplicitous-import issue.

The above is clearly not a full-fledged proposal, but there are lots of ways to accomplish vendoring, and there isn’t much discussion of the alternatives. Is the decision being made because of the way the current tools work?

Thanks

Brad Fitzpatrick

unread,
Jun 11, 2015, 11:21:24 AM6/11/15
to Brendan Tracey, golang-dev, Keith Rarick
On Thu, Jun 11, 2015 at 8:17 AM, Brendan Tracey <tracey....@gmail.com> wrote:
Sorry, accidentally sent off list:

As you say, this means the resolution of an import now depends on where the path is found. A second implication is that it removes one of my favorite properties about Go; in the current environment I can read an import statement and know exactly where the code is.

As Russ wrote: "However, the resolution of an import already needed to take into account the current GOPATH setting, so import paths were never absolute."

That is, a GOPATH can already contain multiple entries. So if GOPATH is /foo:/bar, import "x" might be in /foo/src/x or /bar/src/x.
 
This issue is clearly very important for the community, which may merit breaking this invariant. I would be sad to see more changes that obscure the link between import path and file location.

Toward this end, many users use goimports, which automatically orders and formats imports. Could goimports help legibility by organizing the imports?

import (
   “fmt”
   “log”

   // vendored packages (comment could be explicit or not)
   “github.com/pkg/sftp
   “github.com/x/crypto/ssh

   // non-vendored
   sftp2 “github.com/pkg/sftp
  “github.com/gonum/floats
)

I don't foresee users mixing vendored and non-vendored packages. It'll typically be all or none. So I also don't foresee any changes to goimports.


Chris Hines

unread,
Jun 11, 2015, 11:25:22 AM6/11/15
to golan...@googlegroups.com, k...@xph.us
On Thursday, June 11, 2015 at 11:17:36 AM UTC-4, Brendan Tracey wrote:
...

Why is this style of vendoring chosen? I don’t wish to bike shed, but this proposal seems to take per-package vendoring as a given and discusses how that should happen. For example, why not specify a specific version of a package (by commit hash or something), and then have a package with that hash. Then there could be a $GOPATH/vendor that contains all of the vendored packages. This would scale better when there is a single vendored package imported by many other packages, and could help (though not solve) the duplicitous-import issue.

The goal is to achieve repeatable builds where all of the code (other than the stdlib) is in one source code repository. Putting "vendored" code under $GOPATH/vendor does not make that easy.

Daniel Theophanes

unread,
Jun 11, 2015, 11:37:17 AM6/11/15
to Russ Cox, Keith Rarick, golang-dev
I'm happy with this proposal.

From the proposed CLs, it looks like tools that re-write import paths should not have the "/vendor/" in the path. I think that's fine; existing tools don't currently place them there anyway.

I think having a flag as currently proposed would allow existing tools to easily opt in. If people wanted to use the go command directly, it might be prudent to support an environment variable as well "GO_VENDOR=1".

Do you foresee modifications to other tools should this experiment go well, such as tools that rename symbols across a GOPATH and oracle?

Thank you,
-Daniel


--
You received this message because you are subscribed to a topic in the Google Groups "golang-dev" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/golang-dev/74zjMON9glU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to golang-dev+...@googlegroups.com.

Brad Fitzpatrick

unread,
Jun 11, 2015, 11:53:49 AM6/11/15
to Brendan Tracey, golang-dev, Keith Rarick
... that is, to the goimports output. goimports will still need to learn how to find packages within vendor directories.


tho...@gmail.com

unread,
Jun 11, 2015, 12:42:25 PM6/11/15
to golan...@googlegroups.com
Hi Russ,

This looks like a real step forward. It is not clear to me what you intend to happen if I vendor a project that itself has a vendor/ directory with its own deps.

Should it be an error? Or will the dep's deps just be ignored? Or is this truly a recursive definition?

Tim

Jens Frederich

unread,
Jun 11, 2015, 12:47:36 PM6/11/15
to Russ Cox, Keith Rarick, golang-dev
I'm also happy with this proposal.

Jens

--
You received this message because you are subscribed to the Google Groups "golang-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+...@googlegroups.com.

Florin Patan

unread,
Jun 11, 2015, 1:16:04 PM6/11/15
to golan...@googlegroups.com
Hi,

Unrelated to this change, but I'd like to suggest a small change. How about introducing the -x.  prefix for flags that are experimental? It should allow you to flag when things are experimental, just like golang.org/x/ and still be able to "promote" them to non-experimental later on. For this example, -x.vendor and if it turns out to be a good choice then change it in 1.6 to -vendor? 

What do you think?


Kind regards,
Florin

Andrew Gerrand

unread,
Jun 11, 2015, 1:19:30 PM6/11/15
to Florin Patan, golang-dev

On 11 June 2015 at 10:06, Florin Patan <flori...@gmail.com> wrote:
Unrelated to this change, but I'd like to suggest a small change. How about introducing the -x.  prefix for flags that are experimental? It should allow you to flag when things are experimental, just like golang.org/x/ and still be able to "promote" them to non-experimental later on. For this example, -x.vendor and if it turns out to be a good choice then change it in 1.6 to -vendor? 

What do you think?

Interesting suggestion. I can see a couple of issues, though:

The -x flag already exists in the Go tool, and does something else. Overloading it seems confusing.

When we want to enable vendoring for everyone  we can just set the -vendor flag's default to 'true'. This means all users (including scripts) that are already setting -vendor=true or -vendor=false will continue to see the behavior they expect. If we changed the flag then we'd either need to break people's scripts or maintain two flags.

Andrew

Brad Fitzpatrick

unread,
Jun 11, 2015, 1:19:31 PM6/11/15
to Florin Patan, golang-dev
On Thu, Jun 11, 2015 at 10:06 AM, Florin Patan <flori...@gmail.com> wrote:
Hi,

Unrelated to this change, but I'd like to suggest a small change. How about introducing the -x.  prefix for flags that are experimental? It should allow you to flag when things are experimental, just like golang.org/x/ and still be able to "promote" them to non-experimental later on. For this example, -x.vendor and if it turns out to be a good choice then change it in 1.6 to -vendor? 

What do you think?

That would mean people need to update their scripts as the flag name changes.
 
Also, the -x flag already exists in the go command.

Also, the /x/ in golang.org/x/foo doesn't mean experimental.


Alan Donovan

unread,
Jun 11, 2015, 1:24:20 PM6/11/15
to golan...@googlegroups.com, r...@golang.org, k...@xph.us
On Thursday, 11 June 2015 11:37:17 UTC-4, Daniel Theophanes wrote:
I'm happy with this proposal. [...] Do you foresee modifications to other tools should this experiment go well, such as tools that rename symbols across a GOPATH and oracle?

Yes, I will make sure tools such as gorename and the oracle work well under the new scheme. 

Nate Finch

unread,
Jun 11, 2015, 1:40:48 PM6/11/15
to golan...@googlegroups.com, tho...@gmail.com
It would not be an error, but it would also not get found by an import statement.  You'd have to copy the vendored packages into your project's vendor directory.  The proposal is not recursive.

Nathan Youngman

unread,
Jun 11, 2015, 1:43:03 PM6/11/15
to golan...@googlegroups.com, tho...@gmail.com
Hi Tim

Copying the code to the vendor/ folder is still handled outside of the go tool, whether manually or through another tool.

Nathan.

Nathan Youngman

unread,
Jun 11, 2015, 1:43:03 PM6/11/15
to golan...@googlegroups.com, r...@golang.org, k...@xph.us

An environment variable to set the -vendor flag or, more generally, to set default arguments to go get/install would be nice.
+1

Nathan.
...

Tim Hockin

unread,
Jun 11, 2015, 1:50:49 PM6/11/15
to Nate Finch, golan...@googlegroups.com

If that is the case then this is insufficient for our needs. Without recursive vendoring or deep rewriting, I don't see how to have two versions of github.com/me/prj that are used for different binaries in a repo.

Moving these things to different git repos would be unpleasant, so I am really saving that as a very last resort.

Tim

Andrew Gerrand

unread,
Jun 11, 2015, 1:54:08 PM6/11/15
to Tim Hockin, Nate Finch, golang-dev

On 11 June 2015 at 10:50, Tim Hockin <tho...@gmail.com> wrote:

If that is the case then this is insufficient for our needs. Without recursive vendoring or deep rewriting, I don't see how to have two versions of github.com/me/prj that are used for different binaries in a repo.


Sure you can:

repo/a/foo
repo/a/vendor/github.com/me/prj
repo/b/bar
repo/b/vendor/github.com/me/prj

Where foo and bar are both binaries that import "github.com/me/prj"

Nathan Youngman

unread,
Jun 11, 2015, 1:56:41 PM6/11/15
to golan...@googlegroups.com, tho...@gmail.com, nate....@gmail.com
Tim,

I don't know the full details of your project, but I think it should be possible to have multiple vendor/ directories in the same repo. 

repo
|
+-- project1/cmd
|
+-- project1/vendor
|
+-- project2/cmd
|
+-- project2/vendor

Tim Hockin

unread,
Jun 11, 2015, 2:19:06 PM6/11/15
to Andrew Gerrand, Nate Finch, golan...@googlegroups.com

Hmm, then I don't understand the heuristic for where the build looks for a vendor dir. I assumed it was just at the root of the project repo.

Can you clarify that aspect for me? Given a source file $GOPATH/github.com/me/prj/pkg/foo/bar/bar.go, which imports github.com/you/other, what are the places where an vendor/ may exist and be found by the build?

Thanks

Tim

Andrew Gerrand

unread,
Jun 11, 2015, 2:23:31 PM6/11/15
to Tim Hockin, Nate Finch, golang-dev
Given a source file $GOPATH/src/github.com/me/prj/pkg/foo/bar/bar.go, which imports "github.com/you/other", I believe that possible vendor locations are:

$GOPATH/src/github.com/me/prj/pkg/foo/vendor/github.com/you/other
$GOPATH/src/vendor/github.com/you/other

The longest one that exists is used.

What I'm not sure about is whether


is also a valid search path.

Nathan Youngman

unread,
Jun 11, 2015, 2:33:23 PM6/11/15
to Andrew Gerrand, Tim Hockin, Nate Finch, golang-dev

Finding vendor should follow the same rules as internal/ packages introduced in Go 1.4, correct?




--
You received this message because you are subscribed to a topic in the Google Groups "golang-dev" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/golang-dev/74zjMON9glU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to golang-dev+...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.



--
Nathan Youngman 
Email: he...@nathany.com
Web: http://www.nathany.com

Tim Hockin

unread,
Jun 11, 2015, 2:33:36 PM6/11/15
to Andrew Gerrand, Nate Finch, golan...@googlegroups.com

Ah, so it is recursive. If I vendor a project that has valid vendor/ does then those deps will be preserved, right?

This does imply that a single binary might have multiple copies of the same or different versions of a single dep, but that is for the import tooling to resolve. If the build supports it, the tooling can choose when to use or not use it.

Am I getting it?

Why "vendor" and not "_vendor" ?

Andrew Gerrand

unread,
Jun 11, 2015, 2:37:05 PM6/11/15
to Tim Hockin, Nate Finch, golang-dev
On 11 June 2015 at 11:33, Tim Hockin <tho...@gmail.com> wrote:

Ah, so it is recursive. If I vendor a project that has valid vendor/ does then those deps will be preserved, right?

What I described is not recursive. I just expanded the path search algorithm.

But yes, I do believe that the rules described by Russ allow you to have a vendor directory inside a vendored sub-tree. 

This does imply that a single binary might have multiple copies of the same or different versions of a single dep, but that is for the import tooling to resolve. If the build supports it, the tooling can choose when to use or not use it.

Am I getting it?

I think so. 

Why "vendor" and not "_vendor" ?

Why "_vendor" and not "vendor" ? :-) The latter is easier to type and is consistent with "internal".

Andrew

Tim Hockin

unread,
Jun 11, 2015, 2:57:45 PM6/11/15
to Andrew Gerrand, Nate Finch, golan...@googlegroups.com

"internal" dirs are still yours, and so you would want them included in things like go test ./... but I would assume one would NOT want vendored deps to have tests run. Doesn't leading underscore have that property?

Andrew Gerrand

unread,
Jun 11, 2015, 3:00:48 PM6/11/15
to Tim Hockin, Nate Finch, golang-dev

On 11 June 2015 at 11:57, Tim Hockin <tho...@gmail.com> wrote:
"internal" dirs are still yours, and so you would want them included in things like go test ./... but I would assume one would NOT want vendored deps to have tests run.

I would not assume that, and this proposal does not implement that.

With this proposal, "go test ...", "go list ...", etc will match packages inside vendor directories, just as they do with internal directories.

Nate Finch

unread,
Jun 11, 2015, 3:01:20 PM6/11/15
to golan...@googlegroups.com, nate....@gmail.com, tho...@gmail.com
Yeah, sorry. I meant that import "github.com/other/pkg" at the root of the repo won't import vendor/github.com/foo/bar/vendor/github.com/other/pkg

And yes, _vendor is unnecessarily ugly.

Also, I think a vendor directory in the same directory as a file must be checked, otherwise you wouldn't be able to have go files in the root of your repo that import from a vendor directory, e.g.


main.go needs to be able to import "github.com/spf13/cobra" from the vendor directory.

Andrew Gerrand

unread,
Jun 11, 2015, 3:04:49 PM6/11/15
to Nate Finch, golang-dev, Tim Hockin

On 11 June 2015 at 12:01, Nate Finch <nate....@gmail.com> wrote:
Also, I think a vendor directory in the same directory as a file must be checked

Yes. On further reflection, it seems natural that "b/b.go" importing "foo" will see the package in "b/vendor/foo", and that is what Russ' specification allows.

Tim Hockin

unread,
Jun 11, 2015, 3:09:23 PM6/11/15
to Andrew Gerrand, Nate Finch, golan...@googlegroups.com

Why do I want to run test under my deps on any regular basis? My own tests are slow enough. If my deps are properly carried, their test results don't change when my code does...

Tim Hockin

unread,
Jun 11, 2015, 3:09:23 PM6/11/15
to Andrew Gerrand, Nate Finch, golan...@googlegroups.com

Why do I want to run test under my deps on any regular basis? My own tests are slow enough. If my deps are properly carried, their test results don't change when my code does...

On Jun 11, 2015 12:00 PM, "Andrew Gerrand" <a...@golang.org> wrote:

Andrew Gerrand

unread,
Jun 11, 2015, 3:17:30 PM6/11/15