proposal: external packages

[Keith Rarick]

Here is a pseudo-proposal to change the behavior of the go tool. I
realize this is coming out of left field, so I won't be offended if
the change itself not given serious consideration. However, I'm hoping
the underlying problem will be, and I'd really like to get some fresh
perspective and feedback on how best to address/solve/avoid it.

https://docs.google.com/document/d/1CJnU6ZKvsp21B0lQwbJlKFt8Zz4EWscaCRy_EwK8ja8

There's a similar document at
https://github.com/tools/godep/issues/127 that does not delve into the
possibility of changing the go tool.

[Keith Rarick]

(Observant readers will note that the proposed behavior is essentially
what godep itself does today as a wrapper for the go tool. The
existing behavior in godep is bad because it requires end users to
have godep installed to build commands reproducibly. It is preferable
for commands to be "go-gettable" with the stock go toolchain, while
still keeping reproducibility. Moving godep's search path-altering
behavior into the go tool would fix that objection, if only by brute
force.)

[Viktor Kojouharov]

Would it be possible to instead create a new comment, similar to the build constants or go:generate, which specifies a package version, possibly in a manner similar to bower. Maybe the go get tool could also be extended with that syntax as well.

Something like: // go:dep code.google.com/p/go.net/context#~1.2.3

That way you don't need to drag dependency code around.

[Keith Rarick]

Dave Cheney asked, doesn't godep solve u's problem, by "rolling up" or
"flattening" the dependencies to avoid paths of the form
".../ext/.../ext/.../ext/..."? It does, if u is using godep. But I
think it's unreasonable to require u to use godep.

For one thing, u might be a non-main package. For another, even if u
is a command, this problem makes it hard to get started. I can't ask
the author to run 'godep save' to vendor their dependencies before
they've written the first line of code.

[Keith Rarick]

A while ago, Andrew suggested to me that asking about this
at the beginning of October was poor timing, since everyone
was busy with Go 1.4, and that I might have better luck once
general work on 1.5 started. So I'd like to try to reopen this
discussion now.

I'd love to get feedback:

- Do you think that the problem described in the linked
Google doc is really a problem?

- Do you think my suggestion is reasonable?

- Do you have any other ideas?

[Egon Elbre]

On Monday, 19 January 2015 06:38:18 UTC+2, Keith Rarick wrote:

A while ago, Andrew suggested to me that asking about this
at the beginning of October was poor timing, since everyone
was busy with Go 1.4, and that I might have better luck once
general work on 1.5 started. So I'd like to try to reopen this
discussion now.

I'd love to get feedback:

- Do you think that the problem described in the linked
Google doc is really a problem?

Yes and no.

Yes, in the sense that is happening and will happen in the future.

No, that I don't know how common it is. 

Or whether leaving it "unsolved" and requiring people to think about the problem and find the best solution is a better approach?

- Do you think my suggestion is reasonable?

It seems reasonable; but it is confusing.

 

- Do you have any other ideas?

The "flattening" seems simpler, but of course carries it's own problems:

(by "flattening" I mean) go build will always implicitly add "external" to $GOPATH. And you have a command "go vendor" to pull in all the dependencies into "external" (by flattening and removing any nested "external"-s). Any conflicts must be solved manually.

This approach has the problem that when you have imported a package "foo" that has imported "foo/external/bar" and you have "bar" in your GOPATH, then your code may not build - (i.e. even if there isn't an actual conflict of packages.)

+ Egon

[Daniel Theophanes]

Hi Keith,

I use Godep at work; I like how it works and the concept of external packages. For us it solves a very real problem.

In your proposal the external folder must be under the command folder. I have a project that builds multiple commands using the same vendor packages:

/proj/cmd/cmdA

/proj/cmd/cmdB

proj/cmd/cmdC

/proj/projPkgA

/proj/external/otherPkgA

proj/external/otherPkgB

When building cmdA, the go command (and other tooling) need to walk down the folder tree once to see if it contained a magically named "external" folder. This would need to happen not only for building main packages, but also for any and every package that is analyzed with tools or build system. It also follows that any external package that is built within the "external" folder should also search for packages first in the "external" folder (after GOROOT).

The cost would be:

 1. Stat many non-existing "external" folders.

 2. Tools would be more complex.

 3. Knowing what code an import refers to becomes more complex.

However: (1) the disk cost could cache information each run. (3) Projects that don't live in a global source control system where code gets vendored somehow, there is going to be an extra complexity regardless of whether the tooling supports it or not.

My only other idea is to somehow support a source and binary archives that can be linked in (think Java jars or .Net DLLs).

So as a developer, I support this proposal.

-Daniel

[Chris Hines]

Generally speaking I agree with the goal of making it easier to have reproducible builds.

Have you given any thought to how this feature would impact the predictability of `go test`? I use the package names from the document.

My reading of the document implies that `go test mypkg/p` would import foo. Even if all the tests pass we cannot feel confident that when we run `go install mypkg/c` because that will import mypkg/external/foo. We cannot assume that `go test mypkg/c` will fully test mypkg/p, therefore we must have a way to test mypkg/p when linked to mypkg/external/foo.

On the other hand, if `go test mypkg/p` imports mypkg/external/foo, then we cannot test mypkg/p when linked with foo?

Do we add a flag to `go test` to enable or disable searching for external imports?

Chris

[Andrew Gerrand]

On 19 January 2015 at 15:37, Keith Rarick <k...@xph.us> wrote:

- Do you think that the problem described in the linked
Google doc is really a problem?

I am worried about this problem; the preferred behaviour is context-dependent. I don't think there is a good solution. It does seem very weird that two packages within the same program could see different versions of a package "foo".

I am also worried that "external" adds more unpredictability to something people are already having some trouble with (resolving package names across multiple GOPATHs).

[Russ Cox]

Thanks, Keith. I agree with you that the go command should do this for Go 1.5, as an experiment guarded by a -vendor flag.

I apologize for the delay in responding. I wanted to let both this discussion and the broader conversation proceed, especially since the community understands and encounters these problems more than we on the Go team at Google do. This has happened: we now have much more information from the community about the problem being solved, both in descriptions and in code, like godep, nut, and gb.

We have been rereading the discussions here and looking at the landscape of both Go vendoring tools and large projects that use them. One fact is clear: people very much want to copy (“vendor”) source code into their projects without modifying import paths. Godep and similar tools do this with GOPATH manipulation, and gb does it by reimplementing the build system. In both cases, the result is not compatible with “go get”: people using standard tools cannot easily install projects managed with tools like gb and godep. The proposal in this thread can help the go command meet these tools halfway, so that, perhaps along with minor changes to gb, godep, and friends, users will be able to “go get” those projects.

Specifically, we propose that, as an experiment for Go 1.5, we add a temporary “-vendor” flag that causes the go command to add these semantics:

If there is a source directory d/vendor, then, when compiling a source file within the subtree rooted at d, import "p" is interpreted as import "d/vendor/p" if that exists.

When there are multiple possible resolutions, the most specific (longest) path wins.

The short form must always be used: no import path can contain “/vendor/” explicitly.

Import comments are ignored in vendored packages.

This is a limited form of the original proposal in this thread.

In the original proposal (as I understand it), the interpretation of an import depends both on where the source code containing that import sits in the tree and why the package is being compiled. The meaning of an import in package X is different when X is imported by Y than when X is imported by Z. Therefore the compilation would have to happen multiple times. This doesn’t scale and possibly leads to inconsistent constraints.

In the revised proposal, the interpretation of an import depends only on where the source code containing that import sits in the tree.

The revised proposal also uses the word “vendor” instead of “external”, because (1) there is at least one popular vendoring tool (gb) that uses “vendor” and none that we know of that use “external”; (2) “external” sounds like the opposite of “internal”, which is not the right meaning; and (3) in discussions, everyone calls the broader topic vendoring. It would be nice not to bikeshed the name.

As an aside, the terms “internal vendoring” and “external vendoring” have been introduced into some discussions, to make the distinction between systems that rewrite import paths and systems that do not. With the addition of vendor directories to the go command, we hope that this distinction will fade into the past. There will just be vendoring.

Example

The gb project ships an example project called gsftp. It has a gsftp program with three dependencies outside the standard library: golang.org/x/crypto/ssh, golang.org/x/crypto/ssh/agent, and github.com/pkg/sftp.

Adjusting that example to use the new vendor directory, the source tree would look like:

$GOPATH

| src/

| | github.com/constabulary/example-gsftp/

| | | cmd/

| | | | gsftp/

| | | | | main.go

| | | vendor/

| | | | github.com/pkg/sftp/

| | | | golang.org/x/crypto/ssh/

| | | | | agent/

The file github.com/constabulary/example-gsftp/cmd/gsftp/main.go says:

import (

...

"golang.org/x/crypto/ssh"

"golang.org/x/crypto/ssh/agent"

"github.com/pkg/sftp"

)

Because github.com/constabulary/example-gsftp/vendor/golang.org/x/crypto/ssh exists and the file being compiled is within the subtree rooted at github.com/constabulary/example-gsftp (the parent of the vendor directory), the source line:

import "golang.org/x/crypto/ssh"

is compiled as if it were:

import "github.com/constabulary/example-gsftp/vendor/golang.org/x/crypto/ssh"

(but this longer form is never written).

So the source code in github.com/constabulary/example-gsftp depends on the vendored copy of golang.org/x/crypto/ssh, not one elsewhere in $GOPATH.

In this example, all the dependencies needed by gsftp are (recursively) supplied in the vendor directory, so “go install” does not read any source files outside the gsftp Git checkout. Therefore the gsftp build is reproducible given only the content of the gsftp Git repo and not any other code. And the dependencies need not be edited when copying them into the gsftp repo. And potential users can run “go get github.com/constabulary/example-gsftp/cmd/gsftp” without needing to have an additional vendoring tool installed or special GOPATH configuration.

The point is that adding just the vendor directory mechanism to the go command allows other tools to achieve their goals of reproducible builds and not modifying vendored source code while still remaining compatible with plain “go get”.

Discussion

There are a few points to note about this.

The first, most obvious, and most serious is that the resolution of an import must now take into account the location where that import path was found. This is a fundamental implication of supporting vendoring that does not modify source code. However, the resolution of an import already needed to take into account the current GOPATH setting, so import paths were never absolute. This proposal allows the Go community to move from builds that require custom GOPATH configuration beforehand to builds that just work, because the (more limited) configuration is inferred from the conventions of the source file tree. This approach is also in keeping with the rest of the go command.

The second is that this does not attempt to solve the problem of vendoring resulting in multiple copies of a package being linked into a single binary. Sometimes having multiple copies of a library is not a problem; sometimes it is. At least for now, it doesn’t seem that the go command should be in charge of policing or solving that problem.

The final point is that existing tools like godep, nut, and gb will need to change their file tree layouts if they want to take advantage of compatibility with “go get”. However, compatibility with “go get” used to be impossible. Also, combined with eventual agreement on the vendor-spec, it should be possible for the tools themselves to interoperate.

Deployment

The signals from the Go community are clear: the standard go command must support building source trees in which dependencies have been vendored (copied) without modifications to import paths.

We are well into the Go 1.5 cycle, so caution is warranted. If we put off making any changes, then vendoring tools and “go get” will remain incompatible for the next eight months. On the other hand, if we can make a small, targeted change, then vendoring tools can spend the next eight months experimenting and innovating and possibly converging on a common file tree layout compatible with the go command.

We believe that the experimental proposal above is that small, targeted change. It seems to be the minimal adjustment necessary to support fetching and building vendored, unmodified source code with “go get”. There are many possible extensions or complications we might consider, but for Go 1.5 we want to do as little as possible while remaining useful.

The use of an explicit -vendor flag should help contain the experiment now and ease the transition later. For Go 1.5, users who want to participate in the experiment can opt in by using “go get -vendor”, “go install -vendor”, and so on.

The new semantics changes the meaning of (breaks) source trees containing directories already named “vendor”. Of the over 60,000 listed on godoc.org, there are fewer than 50 such examples. Putting the new semantics behind the -vendor flag avoids breaking those trees for now.

If we decide that the -vendor behavior is correct, then in a later release (possibly Go 1.6) we would make the -vendor flag default to true. Projects containing “vendor” directories could still use “-vendor=false” to get the old behavior while they convert their code. In a still later release (possibly Go 1.7) we would remove the -vendor flag, locking in the vendoring semantics.

Comments welcome here. I just sent CLs 10922 and 10923 as a prototype.

Thanks very much for this proposal.

Russ

[C Cirello]

Hi Russ,

I am not 100% sure if I am mixing up ideas - but especially in the case of gb, not only it addressed vendoring, but also multi-target compilation (such having several cmd/ files all compiled in a single `gb build` call), and also added plugins to facilitate source code download (gb vendor fetch|update|purge etc).

Are these extra functionalities going to be added some time in the future? 

Regards,

Carlos

[Brad Fitzpatrick]

You can already do multi-target compilations with the go tool:

$ go install ./cmd/... 

$ go install example.com/cmd/foo example.com/cmd/bar

As for go tool subcommands to update 'vendor' directories: those could come later (Go 1.6+), but not in Go 1.5.

--
You received this message because you are subscribed to the Google Groups "golang-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Nate Finch]

I think this is exactly the right approach.

[Brendan Tracey]

Sorry, accidentally sent off list:

As you say, this means the resolution of an import now depends on where the path is found. A second implication is that it removes one of my favorite properties about Go; in the current environment I can read an import statement and know exactly where the code is. This issue is clearly very important for the community, which may merit breaking this invariant. I would be sad to see more changes that obscure the link between import path and file location.

Toward this end, many users use goimports, which automatically orders and formats imports. Could goimports help legibility by organizing the imports?

import (
   “fmt”
   “log”

   // vendored packages (comment could be explicit or not)
   “github.com/pkg/sftp”
   “github.com/x/crypto/ssh”

   // non-vendored
   sftp2 “github.com/pkg/sftp”
  “github.com/gonum/floats”
)
 

Why is this style of vendoring chosen? I don’t wish to bike shed, but this proposal seems to take per-package vendoring as a given and discusses how that should happen. For example, why not specify a specific version of a package (by commit hash or something), and then have a package with that hash. Then there could be a $GOPATH/vendor that contains all of the vendored packages. This would scale better when there is a single vendored package imported by many other packages, and could help (though not solve) the duplicitous-import issue.

The above is clearly not a full-fledged proposal, but there are lots of ways to accomplish vendoring, and there isn’t much discussion of the alternatives. Is the decision being made because of the way the current tools work?

Thanks

[Brad Fitzpatrick]

On Thu, Jun 11, 2015 at 8:17 AM, Brendan Tracey <tracey....@gmail.com> wrote:

Sorry, accidentally sent off list:

As you say, this means the resolution of an import now depends on where the path is found. A second implication is that it removes one of my favorite properties about Go; in the current environment I can read an import statement and know exactly where the code is.

As Russ wrote: "However, the resolution of an import already needed to take into account the current GOPATH setting, so import paths were never absolute."

That is, a GOPATH can already contain multiple entries. So if GOPATH is /foo:/bar, import "x" might be in /foo/src/x or /bar/src/x.

 

This issue is clearly very important for the community, which may merit breaking this invariant. I would be sad to see more changes that obscure the link between import path and file location.

Toward this end, many users use goimports, which automatically orders and formats imports. Could goimports help legibility by organizing the imports?

import (
   “fmt”
   “log”

   // vendored packages (comment could be explicit or not)
   “github.com/pkg/sftp”
   “github.com/x/crypto/ssh”

   // non-vendored
   sftp2 “github.com/pkg/sftp”
  “github.com/gonum/floats”
)

I don't foresee users mixing vendored and non-vendored packages. It'll typically be all or none. So I also don't foresee any changes to goimports.

[Chris Hines]

On Thursday, June 11, 2015 at 11:17:36 AM UTC-4, Brendan Tracey wrote:

...

Why is this style of vendoring chosen? I don’t wish to bike shed, but this proposal seems to take per-package vendoring as a given and discusses how that should happen. For example, why not specify a specific version of a package (by commit hash or something), and then have a package with that hash. Then there could be a $GOPATH/vendor that contains all of the vendored packages. This would scale better when there is a single vendored package imported by many other packages, and could help (though not solve) the duplicitous-import issue.

The goal is to achieve repeatable builds where all of the code (other than the stdlib) is in one source code repository. Putting "vendored" code under $GOPATH/vendor does not make that easy.

[Daniel Theophanes]

I'm happy with this proposal.

From the proposed CLs, it looks like tools that re-write import paths should not have the "/vendor/" in the path. I think that's fine; existing tools don't currently place them there anyway.

I think having a flag as currently proposed would allow existing tools to easily opt in. If people wanted to use the go command directly, it might be prudent to support an environment variable as well "GO_VENDOR=1".

Do you foresee modifications to other tools should this experiment go well, such as tools that rename symbols across a GOPATH and oracle?

Thank you,

-Daniel

--
You received this message because you are subscribed to a topic in the Google Groups "golang-dev" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/golang-dev/74zjMON9glU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to golang-dev+...@googlegroups.com.

[Brad Fitzpatrick]

... that is, to the goimports output. goimports will still need to learn how to find packages within vendor directories.

[tho...@gmail.com]

Hi Russ,

This looks like a real step forward. It is not clear to me what you intend to happen if I vendor a project that itself has a vendor/ directory with its own deps.

Should it be an error? Or will the dep's deps just be ignored? Or is this truly a recursive definition?

Tim

[Jens Frederich]

I'm also happy with this proposal.

Jens

--

You received this message because you are subscribed to the Google Groups "golang-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+...@googlegroups.com.

[Florin Patan]

Hi,

Unrelated to this change, but I'd like to suggest a small change. How about introducing the -x.  prefix for flags that are experimental? It should allow you to flag when things are experimental, just like golang.org/x/ and still be able to "promote" them to non-experimental later on. For this example, -x.vendor and if it turns out to be a good choice then change it in 1.6 to -vendor? 

What do you think?

Kind regards,

Florin

[Andrew Gerrand]

On 11 June 2015 at 10:06, Florin Patan <flori...@gmail.com> wrote:

Unrelated to this change, but I'd like to suggest a small change. How about introducing the -x.  prefix for flags that are experimental? It should allow you to flag when things are experimental, just like golang.org/x/ and still be able to "promote" them to non-experimental later on. For this example, -x.vendor and if it turns out to be a good choice then change it in 1.6 to -vendor? 

What do you think?

Interesting suggestion. I can see a couple of issues, though:

The -x flag already exists in the Go tool, and does something else. Overloading it seems confusing.

When we want to enable vendoring for everyone  we can just set the -vendor flag's default to 'true'. This means all users (including scripts) that are already setting -vendor=true or -vendor=false will continue to see the behavior they expect. If we changed the flag then we'd either need to break people's scripts or maintain two flags.

Andrew

[Brad Fitzpatrick]

On Thu, Jun 11, 2015 at 10:06 AM, Florin Patan <flori...@gmail.com> wrote:

Hi,

Unrelated to this change, but I'd like to suggest a small change. How about introducing the -x.  prefix for flags that are experimental? It should allow you to flag when things are experimental, just like golang.org/x/ and still be able to "promote" them to non-experimental later on. For this example, -x.vendor and if it turns out to be a good choice then change it in 1.6 to -vendor? 

What do you think?

That would mean people need to update their scripts as the flag name changes.

 

Also, the -x flag already exists in the go command.

Also, the /x/ in golang.org/x/foo doesn't mean experimental.

[Alan Donovan]

On Thursday, 11 June 2015 11:37:17 UTC-4, Daniel Theophanes wrote:

I'm happy with this proposal. [...] Do you foresee modifications to other tools should this experiment go well, such as tools that rename symbols across a GOPATH and oracle?

Yes, I will make sure tools such as gorename and the oracle work well under the new scheme. 

[Nate Finch]

It would not be an error, but it would also not get found by an import statement.  You'd have to copy the vendored packages into your project's vendor directory.  The proposal is not recursive.

[Nathan Youngman]

Hi Tim

Copying the code to the vendor/ folder is still handled outside of the go tool, whether manually or through another tool.

Nathan.

[Nathan Youngman]

An environment variable to set the -vendor flag or, more generally, to set default arguments to go get/install would be nice.

+1

Nathan.

...

[Tim Hockin]

If that is the case then this is insufficient for our needs. Without recursive vendoring or deep rewriting, I don't see how to have two versions of github.com/me/prj that are used for different binaries in a repo.

Moving these things to different git repos would be unpleasant, so I am really saving that as a very last resort.

Tim

[Andrew Gerrand]

On 11 June 2015 at 10:50, Tim Hockin <tho...@gmail.com> wrote:

If that is the case then this is insufficient for our needs. Without recursive vendoring or deep rewriting, I don't see how to have two versions of github.com/me/prj that are used for different binaries in a repo.

Sure you can:

repo/a/foo

repo/a/vendor/github.com/me/prj

repo/b/bar

repo/b/vendor/github.com/me/prj

Where foo and bar are both binaries that import "github.com/me/prj"

[Nathan Youngman]

Tim,

I don't know the full details of your project, but I think it should be possible to have multiple vendor/ directories in the same repo. 

repo

|

+-- project1/cmd

|

+-- project1/vendor

|

+-- project2/cmd

|

+-- project2/vendor

[Tim Hockin]

Hmm, then I don't understand the heuristic for where the build looks for a vendor dir. I assumed it was just at the root of the project repo.

Can you clarify that aspect for me? Given a source file $GOPATH/github.com/me/prj/pkg/foo/bar/bar.go, which imports github.com/you/other, what are the places where an vendor/ may exist and be found by the build?

Thanks

Tim

[Andrew Gerrand]

Given a source file $GOPATH/src/github.com/me/prj/pkg/foo/bar/bar.go, which imports "github.com/you/other", I believe that possible vendor locations are:

$GOPATH/src/github.com/me/prj/pkg/foo/vendor/github.com/you/other

$GOPATH/src/github.com/me/prj/pkg/vendor/github.com/you/other

$GOPATH/src/github.com/me/prj/vendor/github.com/you/other

$GOPATH/src/github.com/me/vendor/github.com/you/other

$GOPATH/src/github.com/vendor/github.com/you/other

$GOPATH/src/vendor/github.com/you/other

The longest one that exists is used.

What I'm not sure about is whether

$GOPATH/src/github.com/me/prj/pkg/foo/bar/vendor/github.com/you/other

is also a valid search path.

[Nathan Youngman]

Finding vendor should follow the same rules as internal/ packages introduced in Go 1.4, correct?

http://golang.org/doc/go1.4#internalpackages

--
You received this message because you are subscribed to a topic in the Google Groups "golang-dev" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/golang-dev/74zjMON9glU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to golang-dev+...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Nathan Youngman 
Email: he...@nathany.com
Web: http://www.nathany.com

[Tim Hockin]

Ah, so it is recursive. If I vendor a project that has valid vendor/ does then those deps will be preserved, right?

This does imply that a single binary might have multiple copies of the same or different versions of a single dep, but that is for the import tooling to resolve. If the build supports it, the tooling can choose when to use or not use it.

Am I getting it?

Why "vendor" and not "_vendor" ?

[Andrew Gerrand]

On 11 June 2015 at 11:33, Tim Hockin <tho...@gmail.com> wrote:

Ah, so it is recursive. If I vendor a project that has valid vendor/ does then those deps will be preserved, right?

What I described is not recursive. I just expanded the path search algorithm.

But yes, I do believe that the rules described by Russ allow you to have a vendor directory inside a vendored sub-tree. 

This does imply that a single binary might have multiple copies of the same or different versions of a single dep, but that is for the import tooling to resolve. If the build supports it, the tooling can choose when to use or not use it.

Am I getting it?

I think so. 

Why "vendor" and not "_vendor" ?

Why "_vendor" and not "vendor" ? :-) The latter is easier to type and is consistent with "internal".

Andrew

[Tim Hockin]

"internal" dirs are still yours, and so you would want them included in things like go test ./... but I would assume one would NOT want vendored deps to have tests run. Doesn't leading underscore have that property?

[Andrew Gerrand]

On 11 June 2015 at 11:57, Tim Hockin <tho...@gmail.com> wrote:

"internal" dirs are still yours, and so you would want them included in things like go test ./... but I would assume one would NOT want vendored deps to have tests run.

I would not assume that, and this proposal does not implement that.

With this proposal, "go test ...", "go list ...", etc will match packages inside vendor directories, just as they do with internal directories.

[Nate Finch]

Yeah, sorry. I meant that import "github.com/other/pkg" at the root of the repo won't import vendor/github.com/foo/bar/vendor/github.com/other/pkg

And yes, _vendor is unnecessarily ugly.

Also, I think a vendor directory in the same directory as a file must be checked, otherwise you wouldn't be able to have go files in the root of your repo that import from a vendor directory, e.g.

github.com/natefinch/foo/vendor/github.com/spf13/cobra

github.com/natefinch/foo/main.go

main.go needs to be able to import "github.com/spf13/cobra" from the vendor directory.

[Andrew Gerrand]

On 11 June 2015 at 12:01, Nate Finch <nate....@gmail.com> wrote:

Also, I think a vendor directory in the same directory as a file must be checked

Yes. On further reflection, it seems natural that "b/b.go" importing "foo" will see the package in "b/vendor/foo", and that is what Russ' specification allows.

[Tim Hockin]

Why do I want to run test under my deps on any regular basis? My own tests are slow enough. If my deps are properly carried, their test results don't change when my code does...

[Tim Hockin]

Why do I want to run test under my deps on any regular basis? My own tests are slow enough. If my deps are properly carried, their test results don't change when my code does...

On Jun 11, 2015 12:00 PM, "Andrew Gerrand" <a...@golang.org> wrote:

[Andrew Gerrand]

On 11 June 2015 at 12:09, Tim Hockin <tho...@gmail.com> wrote:

Why do I want to run test under my deps on any regular basis? My own tests are slow enough. If my deps are properly carried, their test results don't change when my code does...

We should definitely make testing faster, but there are better solutions to this problem that aren unrelated to vendoring. instance, there is work in progress to more reliably detect whether Go source has changed between builds; with that done, we can cache test results to avoid redundant test runs. That way you would run the tests of your vendored packages only when you update them. 

In the meantime, you can always skip running vendored tests with:

  go test $(go list ./... | grep -v /vendor/)

It's worth reiterating that Russ' goal with this change is to provide "the minimal adjustment necessary to support fetching and building vendored, unmodified source code with “go get”." and that this is a last-minute change for Go 1.5. In Go 1.6 I expect we will make other improvements to the go tool to make working with vendored dependencies easier.

[Tim Hockin]

I think _vendor provides the (more) correct semantics out of the box without pushing the extra work to attain correctness on the user and without depending on hypothetical other features.

And it is just as minimal if not more :)

Go chose to make _dir have special meaning, and that special meaning is correct here. I am sorry if people find it ugly take that up with the designers. Most existing vendoring tools do exactly this, and changing it would be silly, IMO of course.

[Tim Hockin]

Also, _vendor seems far less likely to collide with legit users of that name today.

[Karan Misra]

I was actually hoping for "third_party" (gotten used to it from various repos I study) ! But "vendor" will do just fine.

[Monnand]

If go test and go list are not affected by this change, then which tools will be affected (in 1.5)?

I can see that go build and go install will be clearly support -vendor flag. But how about other tools? More precisely:

- go get: Will it refrain to fetch packages which has been put in the vendor directory? Once the vendor/ directory is allowed, I am not sure if people would have some crazy ideas like "vendor/we_renamed_some_btree_pkg". In this case, go get may not be able to find the corresponding repo on any hosting service.

- go doc: Will the generated documents contain the correct hyperlinks to the vendor's documents?

I'm glad to see that this problem will finally be solved in Go 1.5. But I think I will probably see some of my development tools stop working due to this change. (gocode, goimports, etc.)

[Andrew Gerrand]

The -vendor flag only applies to building code (the build, install, get, and test commands). It has a very simple effect; it'll make the compiler and linker see vendored dependencies instead of trying to resolve paths from the workspace root. The rest of the tools are unchanged, and will see vendored packages using their full paths (including /vendor/). Tools like goimports, gorename, etc will need to be updated to know about the /vendor/ directory. 

go get will support -vendor. The result is that you can fetch a repository that has vendored dependencies.

go doc will not support -vendor. I'm not sure what it would mean even if it did. It doesn't produce hyperlinks to other packages (are you thinking of godoc?).

--

You received this message because you are subscribed to the Google Groups "golang-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+...@googlegroups.com.

[tran...@gmail.com]

Russ,

I have some concerns about the proposal. 

First, it feels a bit like a fem-fatal affair with deep directory structure.

    "$GOPATH/src/github.com/constabulary/example-gsftp/vendor/golang.org/x/crypto/ssh/agent/..."

It's like the Limbo, how low can you go? Which brings me the my next concern. Perhaps I missed it, but what about the vendored libraries of vendored libraries? Are they going to be yet further down the tree?

    "$GOPATH/src/github.com/trans/foo-gstfp/vendor/githubc.com/constabulary/example-gsftp/vendor/golang.org/x/crypto/ssh/agent/..."

Lastly, when you speak of a "small, targeted change", I can't help but read "kludge". Rather than solving the issue full-on it's an attempt just to muddle through so as to effect existing `go` functionality as little as possible.

[brainman]

You cannot have paths longer then about 250 on windows (see http://golang.org/issue/3358). Even more, given the way "go test" arranges its files your path will get doubled sometimes. So you really cannot make paths more then about 130.

Alex

[yiyus]

On Thursday, 11 June 2015 15:21:06 UTC+2, rsc wrote:

Specifically, we propose that, as an experiment for Go 1.5, we add a temporary “-vendor” flag that causes the go command to add these semantics:

If there is a source directory d/vendor, then, when compiling a source file within the subtree rooted at d, import "p" is interpreted as import "d/vendor/p" if that exists.

When there are multiple possible resolutions, the most specific (longest) path wins.

The short form must always be used: no import path can contain “/vendor/” explicitly.

Import comments are ignored in vendored packages.

Once this gets implemented, what is the need for internal packages? We could just put them in the vendor directory. Using an internal subdirectory may be a convention, but we could get rid of that special case (so, for example, you would import "internal/mypackage", which would be in vendor/internal/mypackage).

I am not sure which is the best way to combine both features, but I think that adding both internal and vendor directories with this overlapping of functionality may be a mistake in the long term.

[andrewc...@gmail.com]

I saw _vendor in a project previously and immediately started crying and banging my head on the desk. I don't know If I will be able to recover from that sort of shock again..

[Tim Hockin]

I apologize by proxy for the original design which dictates _whatever is special, but the specialness is warranted for vendored packages. If I 'go test ./...' I do NOT want it kicking off test for vendors deps. Etc.

[Brad Fitzpatrick]

On Thu, Jun 11, 2015 at 5:58 PM, Tim Hockin <tho...@hockin.org> wrote:

I apologize by proxy for the original design which dictates _whatever is special, but the specialness is warranted for vendored packages. If I 'go test ./...' I do NOT want it kicking off test for vendors deps. Etc.

If you update from Go 1.N to Go 1.N+1, you don't want to know whether your vendored dependencies still pass?

If a user reports a bug on random $GOOS and you start debugging on that OS, you don't want to know whether your vendored dependencies still pass there?

If you really don't want to be bothered with test failures, just instruct your vendoring tool to delete the test files.

[Tim Hockin]

We run vendored tests ourselves, NOT as part of every 'go test' . But I am clearly on the losing side of this one, so I will just shut up now.

[Dave Cheney]

I agree with Tim. I also suggested _vendor (although I wasn't that insistent)

[Dave Cheney]

Hyperbole won't help the situation. The go tool already treats directories starting with an underscore as invisible. Given that property, _vendor is consistent and appropriate.

[andrewc...@gmail.com]

Well, In all honesty, when I saw what godep does with Godeps/_workspace I was instantly turned off. Something about the capitalization and underscore just made me think it was hacky, I know it is entirely in my head, but I am probably not the only person.

I don't even really like the word "vendor" tbh, but it is bikeshedding so, meh - Apologies and withdrawn.

[Henrik Johansson]

The thread has quickly become long and maybe it has been answered but did the transitive dependencies "which of the same dependency to choose" get settled?

Would it not be good if the topmost vendored dependenciy took priority? That way I as a user of a dependency can more easilly choose which version of a dependency to use.

[bdtin...@gmail.com]

On Thursday, June 11, 2015 at 8:21:24 AM UTC-7, Brad Fitzpatrick wrote:

On Thu, Jun 11, 2015 at 8:17 AM, Brendan Tracey <tracey....@gmail.com> wrote:

Sorry, accidentally sent off list:

As you say, this means the resolution of an import now depends on where the path is found. A second implication is that it removes one of my favorite properties about Go; in the current environment I can read an import statement and know exactly where the code is.

As Russ wrote: "However, the resolution of an import already needed to take into account the current GOPATH setting, so import paths were never absolute."

That is, a GOPATH can already contain multiple entries. So if GOPATH is /foo:/bar, import "x" might be in /foo/src/x or /bar/src/x.

On my machine, GOPATH is a single directory so I don't need to hunt across GOPATHs
 

 

This issue is clearly very important for the community, which may merit breaking this invariant. I would be sad to see more changes that obscure the link between import path and file location.

Toward this end, many users use goimports, which automatically orders and formats imports. Could goimports help legibility by organizing the imports?

import (
   “fmt”
   “log”

   // vendored packages (comment could be explicit or not)
   “github.com/pkg/sftp”
   “github.com/x/crypto/ssh”

   // non-vendored
   sftp2 “github.com/pkg/sftp”
  “github.com/gonum/floats”
)

I don't foresee users mixing vendored and non-vendored packages. It'll typically be all or none. So I also don't foresee any changes to goimports.

I don't see why users wouldn't mix. If there are a set of packages that are useful and provide a credible statement about APIs (like the "Go 1 compatibility promise), is it still necessary to vendor those packages into every repository? There are already tools which assist stable APIs, for example, gopkg.in. Users may wish to have a single version of those packages.

[Yongjian Xu]

One issue that has not been covered in the spec, suppose you have:

b/vendor/foo

b/vendor/bar/vendor/foo (so pkg bar also imports foo but these two versions of foo might well be different)

b/main.go

we probably don't want to import "foo" from the vendored pkg's vendor dir, even if thats longer in path length. That is, vender search path can only contain one level of vendor directory rooted from the pkg itself. Otherwise, this will become very confusing as which vendored "foo" gets used. Accidentally use the vendored pkg's vendored one may not be what the user wants at all.

The simple "longest applies" does not seem to address this issue, and I think the spec should cover this.

[Mateusz Czapliński]

I seem to recall that some earlier semi-official discussion/proposal explored an idea of vendoring only "command" packages (i.e. ones with "package main"). Do I understand correctly, that this proposal above by Russ dismisses that, and as such is expected to work both for "cmds" and "pkgs" exactly the same? Please confirm or deny for clarification.

Other than that, developing quite much on Windows, I'm also worried with regards to path limits mentioned by Alex Brainman, especially if this is expected to work recursively.

/M.

[Ian Lance Taylor]

On Fri, Jun 12, 2015 at 2:22 AM, Mateusz Czapliński <czap...@gmail.com> wrote:
>
> I seem to recall that some earlier semi-official discussion/proposal
> explored an idea of vendoring only "command" packages (i.e. ones with
> "package main").

I don't recall that, and to be honest it doesn't seem useful.

> Do I understand correctly, that this proposal above by Russ
> dismisses that, and as such is expected to work both for "cmds" and "pkgs"
> exactly the same? Please confirm or deny for clarification.

That is correct: Russ's proposal applies to both commands and
packages.

> Other than that, developing quite much on Windows, I'm also worried with
> regards to path limits mentioned by Alex Brainman, especially if this is
> expected to work recursively.

That sounds like a very annoying problem, but people are already
vendoring libraries within libraries. I don't see how this proposal
makes the problem worse.

Ian

[Gustavo Niemeyer]

Vendoring and gopkg.in are orthogonal and complementary concepts.

Using gopkg.in allows people to provide a public statement of API stability, and also to release and maintain new versions which break said API stability in a convenient and expectable way. Vendoring allows people to lock down the exact code that is being used by an application, and to update the code in a controlled manner.

One benefits from API stability even when vendoring, and applications benefit from code lock down even with API stability.

Russ' proposal looks great to me.

--

You received this message because you are subscribed to the Google Groups "golang-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

gustavo @ http://niemeyer.net

[Brendan Tracey]

On Jun 12, 2015, at 6:17 AM, Gustavo Niemeyer <gus...@niemeyer.net> wrote:

Vendoring and gopkg.in are orthogonal and complementary concepts.

Using gopkg.in allows people to provide a public statement of API stability, and also to release and maintain new versions which break said API stability in a convenient and expectable way. Vendoring allows people to lock down the exact code that is being used by an application, and to update the code in a controlled manner.

One benefits from API stability even when vendoring, and applications benefit from code lock down even with API stability.

I agree. Brad’s comment was that imports will only fall into two categories {std library, vendored}, or {std library, no vendored}. 

[Nate Finch]

It sounds like the major limiting factor here is the "tests may double the length of your paths" ... perhaps we can fix it so go test doesn't do that?  That seems like a bug worth fixing regardless of vendoring.

[Gustavo Niemeyer]

What Brad means is that when someone locks down an application's dependencies, in general all dependencies will be locked down rather than half of them. I agree with that.

That said, there's also room for partial vendoring, even for libraries. Think about the case where you publish a library that depends on a third-party library whose maintainer is remarkably unreliable in terms of API breakage. Even if you are working in the unstable in-development version of your library, there would be value in locking down that library with an unreliable API.

[Daniel Theophanes]

Hi Alex,

I currently vendor with path rewrites on Windows. In practice this just isn't an issue for me. If this does become an issue there is a known change we can make of adding "\\?\" to paths if longer then 250 chars. If you think it is an issue then we should address this in go1.6 separately.

-Daniel

On Thu, Jun 11, 2015 at 4:39 PM brainman <alex.b...@gmail.com> wrote:

You cannot have paths longer then about 250 on windows (see http://golang.org/issue/3358). Even more, given the way "go test" arranges its files your path will get doubled sometimes. So you really cannot make paths more then about 130.

Alex

--
You received this message because you are subscribed to a topic in the Google Groups "golang-dev" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/golang-dev/74zjMON9glU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to golang-dev+...@googlegroups.com.

[Russ Cox]

In addition to what Brad wrote:

On Thu, Jun 11, 2015 at 7:04 AM, Brendan Tracey <tracey....@gmail.com> wrote:

As you say, this means the resolution of an import now depends on where the path is found. A second implication is that it removes one of my favorite properties about Go; in the current environment I can read an import statement and know exactly where the code is. This issue is clearly very important for the community, which may merit breaking this invariant. I would be sad to see more changes that obscure the link between import path and file location.

Yes, absolutely. I agree with you 100%.

 

Toward this end, many users use goimports, which automatically orders and formats imports. Could goimports help legibility by organizing the imports?

There is a more general point about making sure that tools help us understand how imports resolve. We will need to see how this gets used to understand what mechanical help we will need to understand the trees being created. I hope that having the functionality behind a flag for the Go 1.5 release will give the community time to adapt higher-level tooling as appropriate.

For the specific case of goimports, it may not work well to try to add comments about what is vendored and what is not. That is a property of the directory tree that surrounds the code. If the code is moved, those comments would become stale, and all of this is motivated by not wanting to need to modify code at all when it moves.

For what it's worth, until there is more tooling, at least 'go list' can help tell you what is going on:

$ cat x.go

package main

import (

"z"

"y"

)

func main() {

y.F()

z.F()

}

$ 

$ go list -vendor -f '{{join .Imports "\n"}}' x.go

test/vendor/y

vendor/z

$  

Why is this style of vendoring chosen? I don’t wish to bike shed, but this proposal seems to take per-package vendoring as a given and discusses how that should happen. For example, why not specify a specific version of a package (by commit hash or something), and then have a package with that hash. Then there could be a $GOPATH/vendor that contains all of the vendored packages. This would scale better when there is a single vendored package imported by many other packages, and could help (though not solve) the duplicitous-import issue.

I hope that as much as possible this proposal does not limit people to a particular style of vendoring. It creates a mechanism to define a source location-dependent search path for imports other than "modify GOPATH between commands". What happens next is really up to the vendoring tools.

I understand that people use GOPATH in a variety of ways, the intended use of GOPATH is that you set it once and never worry about it again. For example, I set GOPATH=$HOME in my profile and never modify it explicitly. I install many different projects in it. If there are two commands that vendor (and need) different versions of a package, letting them have different locations for the vendored code keeps them independent. I can build both, get both binaries, and everything works. If there must be one $GOPATH/vendor, then those two commands collide, and now I can't build one or the other. Also we check things out into $GOPATH/src/<repopath>. If vendored code went somewhere outside $GOPATH/src, we'd have to define how its gets there, how it gets updated, and so on.

The main concern is that we want independent projects that vendor all their dependencies to be able to coexist in GOPATH. We separate the non-vendored code using the directory name space. It made sense to do the same for vendored code.

Russ

[Russ Cox]

On Thu, Jun 11, 2015 at 8:45 AM, <tho...@gmail.com> wrote:

Hi Russ,

This looks like a real step forward.  It is not clear to me what you intend to happen if I vendor a project that itself has a vendor/ directory with its own deps.

Should it be an error?  Or will the dep's deps just be ignored? Or is this truly a recursive definition?

You and Andrew seem to have figured this out, but I wanted to say it one more time for the archives.

When there are multiple possible resolutions, the one deepest in the tree wins. I wouldn't call this "recursive" but I think it's what you meant. Suppose you have:

test/

| code/

| subdir/

| | code/

| | vendor/

| | | pkg/

| vendor/

| | pkg/

Then an import "pkg" in the source files in test/code expands to import "test/vendor/pkg", but the same import in the source files in test/subdir/code expans to import "test/subdir/vendor/pkg".

Note, however, that this proposal only defines how the build system interprets a source tree. It does not define the process for preparing that source tree. The tool you use to copy test/subdir into your tree might also move the vendored things up into test/vendor and do some kind of intelligent deduplication. The goal today is to get everyone on one page regarding directory layout, and then metadata layout (github.com/kardianos/vendor-spec), to encourage these kinds of higher-level tools.

Russ

[Russ Cox]

On Thu, Jun 11, 2015 at 11:33 AM, Tim Hockin <tho...@gmail.com> wrote:

Why "vendor" and not "_vendor" ?

The most important reason is that we've documented that "[d]irectories beginning with . or _ are ignored by the go tool." That means all the time, not just some of the time.

I understand your point about testing, but as Andrew said that's really an unrelated problem. We need to (and will) make it easy to test only things that need retesting in general, not just for this specific case.

Russ

[Russ Cox]

On Thu, Jun 11, 2015 at 2:07 PM, <tran...@gmail.com> wrote:

    "$GOPATH/src/github.com/constabulary/example-gsftp/vendor/golang.org/x/crypto/ssh/agent/..."

It's like the Limbo, how low can you go? Which brings me the my next concern. Perhaps I missed it, but what about the vendored libraries of vendored libraries? Are they going to be yet further down the tree?

See my reply to Tim just above this one. Yes, if you arrange your tree that way, that's what it will look like. But I expect there will be tools to help hoist everything up into one vendor directory for your particular project.

The paths are certainly long, but remember the constraints: we are dealing with a global, decentralized name space, hence the URLs. And we need to express "this is github.com/constabulary/example-gsftp's copy of golang.org/x/crypto/ssh/agent". The path is only as long as it needs to be to express that.

 

    "$GOPATH/src/github.com/trans/foo-gstfp/vendor/githubc.com/constabulary/example-gsftp/vendor/golang.org/x/crypto/ssh/agent/..."

Lastly, when you speak of a "small, targeted change", I can't help but read "kludge". Rather than solving the issue full-on it's an attempt just to muddle through so as to effect existing `go` functionality as little as possible.

I'm sorry you feel that way, but I disagree. I'd like to explain how I see the bigger picture behind this proposal. If you have constructive suggestions about what we should do instead, please feel free to share them in this thread or to email them to me directly.

Broadly, there are two parts to vendoring: you have to copy the source code into your repository, and you have to make your code use the vendored copy during the build. 

The copying step need not be part of the go command: there are very thorny issues about how to specify what to copy, how to resolve conflicting specifications, and so on. External tools still seem to me the right place to tackle this problem. There are many, which is great: let's experiment and figure out what works and what does not. 

The "make your code use it" step is the topic today. In the past, we the Go team at Google have advocated for rewriting import paths, so that when the code was copied in, you'd replace:

    import "golang.org/x/crypto/ssh/agent"

with

    import "github.com/constabulary/example-gsftp/vendor/golang.org/x/crypto/ssh/agent"

The Go community has spoken essentially unanimously in saying that they do not want to rewrite import paths this way. Instead, they found a kludge that involved reconfiguring GOPATH every time you wanted to switch to a different project. This avoids modifying copied sources but makes these tools incompatible with 'go get'.

Today's proposal (really Keith's proposal from October) provides a different way to allow the vendored copies to be used unmodified, one that is compatible with 'go get'. I believe it does solve the "make your code use the vendored copy during the build" problem full-on. But if you see a reason it does not, again I would like to know about that, so please share details.

Thanks.

Russ

[Russ Cox]

On Thu, Jun 11, 2015 at 7:39 PM, brainman <alex.b...@gmail.com> wrote:

You cannot have paths longer then about 250 on windows (see http://golang.org/issue/3358). Even more, given the way "go test" arranges its files your path will get doubled sometimes. So you really cannot make paths more then about 130.

We definitely need to fix this problem, but it's not caused by this proposal. I believe the directory layouts in this proposal end up with shorter names than they would in godep or nut, and roughly the same as in gb.

Russ

[Russ Cox]

On Thu, Jun 11, 2015 at 7:45 PM, yiyus <yiyu...@gmail.com> wrote:

Once this gets implemented, what is the need for internal packages? We could just put them in the vendor directory. Using an internal subdirectory may be a convention, but we could get rid of that special case (so, for example, you would import "internal/mypackage", which would be in vendor/internal/mypackage).

I am not sure which is the best way to combine both features, but I think that adding both internal and vendor directories with this overlapping of functionality may be a mistake in the long term.

It might be, but we've already introduced internal, so we can't easily take it back now. Also, despite the similar visibility, they have very different meanings, and the meanings matter: code in vendor was copied from somewhere else, but code in internal probably wasn't. Code in internal is also imported under the clearer, longer path, and it would be inappropriate to manage with vendoring tools.

Russ

[Russ Cox]

On Thu, Jun 11, 2015 at 8:37 PM, Yongjian Xu <i3dm...@gmail.com> wrote:

One issue that has not been covered in the spec, suppose you have:

b/vendor/foo

b/vendor/bar/vendor/foo (so pkg bar also imports foo but these two versions of foo might well be different)

b/main.go

we probably don't want to import "foo" from the vendored pkg's vendor dir, even if thats longer in path length. That is, vender search path can only contain one level of vendor directory rooted from the pkg itself. Otherwise, this will become very confusing as which vendored "foo" gets used. Accidentally use the vendored pkg's vendored one may not be what the user wants at all.

The simple "longest applies" does not seem to address this issue, and I think the spec should cover this.

This scenario is definitely confusing. But the root cause is the ambiguity caused by the complex directory structure, not the interpretation by the go command. The solution is not to create directory trees like that. I believe that once there is general agreement on the directory structure and the vendor spec, vendoring tools will be written that deduplicate situations like this, hoisting the vendoring up as high as makes sense (to the top of the source control repo, probably).

Russ

[Nathan Youngman]

I do get the sense that this proposal is a specific case of project-relative import paths using "vendor" as a convention (magic) much like the internal folder is a convention for packages that can't be imported from other projects.

Which isn't too say I see any need for importing the packages within my own project with a shorter path.

For now this proposal is the most pragmatic option, and I'm glad to see it. Placing it behind a flag does mean it could change if there is some reason to do so in Go 1.6. Thanks.

--

You received this message because you are subscribed to a topic in the Google Groups "golang-dev" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/golang-dev/74zjMON9glU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to golang-dev+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Nathan Youngman 
Email: he...@nathany.com
Web: http://www.nathany.com

[Matthew Sackman]

On Fri, Jun 12, 2015 at 02:30:03PM -0400, Russ Cox wrote:
> This scenario is definitely confusing. But the root cause is the ambiguity
> caused by the complex directory structure, not the interpretation by the go
> command. The solution is not to create directory trees like that. I believe
> that once there is general agreement on the directory structure and the
> vendor spec, vendoring tools will be written that deduplicate situations
> like this, hoisting the vendoring up as high as makes sense (to the top of
> the source control repo, probably).

I believe you're assuming that it always makes sense to "deduplicate"
such situations. Are there not cases where it's critical to be able to
have library X depend on v1 of library Y, but application Z, which uses
library X and (directly) library Y use v2 of library Y? Do you consider
accomodating such cases as overengineering and/or YAGNI?

Matthew

[Brendan Tracey]

On Friday, June 12, 2015 at 11:30:06 AM UTC-7, rsc wrote:

On Thu, Jun 11, 2015 at 8:37 PM, Yongjian Xu <i3dm...@gmail.com> wrote:

One issue that has not been covered in the spec, suppose you have:

b/vendor/foo

b/vendor/bar/vendor/foo (so pkg bar also imports foo but these two versions of foo might well be different)

b/main.go

we probably don't want to import "foo" from the vendored pkg's vendor dir, even if thats longer in path length. That is, vender search path can only contain one level of vendor directory rooted from the pkg itself. Otherwise, this will become very confusing as which vendored "foo" gets used. Accidentally use the vendored pkg's vendored one may not be what the user wants at all.

The simple "longest applies" does not seem to address this issue, and I think the spec should cover this.

I was confused by this too initially, but the spec does cover it. If we import "foo" from main, it will resolve to b/vendor/foo. The spec says  " import 'p' is interpreted as import 'd/vendor/p' if that exists". bar's version of the package will not be imported because it is /bar/vendor/foo from main, not just foo. Additionally, main cannot import bar/vendor/foo because "vendor" is a prohibited import.

[Tim Hockin]

I parse the meaning of _dir differently than you, but it's not interpretation that matters. I read it as ... will never match.

I think this overall proposal is a step forward and makes the vendoring ecosystem one step closer to consistent.

[Dmitri Shuralyov]

On Friday, June 12, 2015 at 11:21:12 AM UTC-7, rsc wrote:

On Thu, Jun 11, 2015 at 11:33 AM, Tim Hockin <tho...@gmail.com> wrote:

Why "vendor" and not "_vendor" ?

The most important reason is that we've documented that "[d]irectories beginning with . or _ are ignored by the go tool." That means all the time, not just some of the time.

To clarify, it's actually:

> Directory and file names that begin with "." or "_" are ignored by the go tool, as are directories named "testdata".

I've had to make many PRs trying to help other people make their tools that work on Go packages respect the above rule. For example, testdata may contain invalid .go files and that's valid to do (it's done in some packages for testing the Go parser, for example). But then the tools that don't respect those rules end up complaining. E.g., see https://github.com/tools/godep/issues/140 still remains an open issue after more than year.

[Brendan Tracey]

Thanks Russ, Brad, Chris and Gustavo for the comments. I better understand now what this change does and does not imply. I still think there will be a mix of vendoring and not, but I now better understand the complexities of using goimports to help fix that.

It is a nice specification, especially with how close it is to "internal".

On Friday, June 12, 2015 at 11:20:28 AM UTC-7, rsc wrote:

In addition to what Brad wrote:

On Thu, Jun 11, 2015 at 7:04 AM, Brendan Tracey <tracey....@gmail.com> wrote:

As you say, this means the resolution of an import now depends on where the path is found. A second implication is that it removes one of my favorite properties about Go; in the current environment I can read an import statement and know exactly where the code is. This issue is clearly very important for the community, which may merit breaking this invariant. I would be sad to see more changes that obscure the link between import path and file location.

Yes, absolutely. I agree with you 100%.

I'm glad.

[Joe Shaw]

Hi,

For what it's worth, I agree with you Tim that running vendored tests as part of "go test ./..." is definitely not what I want as the default.  While vendored code will change as dependencies are updated this is a much less frequent occurrence than updating my own code.  Vendored code does not need to be tested for every change in my own project.

"go test ./..." is very convenient.  If to test only my own code I must "go test $(go list ./... | grep -v vendor)" or whatever the command is, then I might as well start using a Makefile and "make test" instead.

It should still be possible to test vendored code with something like "go test ./_vendor/..." and I would want to do that when I update dependencies.

On the aesthetics of "vendor" versus "_vendor", the former is nicer but the latter isn't so ugly as to warrant changing better default behavior.

Joe

On Thursday, June 11, 2015 at 9:52:01 PM UTC-4, Tim Hockin wrote:

We run vendored tests ourselves, NOT as part of every 'go test' . But I am clearly on the losing side of this one, so I will just shut up now.

On Jun 11, 2015 6:39 PM, "Brad Fitzpatrick" <brad...@golang.org> wrote:

On Thu, Jun 11, 2015 at 5:58 PM, Tim Hockin <tho...@hockin.org> wrote:

I apologize by proxy for the original design which dictates _whatever is special, but the specialness is warranted for vendored packages. If I 'go test ./...' I do NOT want it kicking off test for vendors deps. Etc.

If you update from Go 1.N to Go 1.N+1, you don't want to know whether your vendored dependencies still pass?

If a user reports a bug on random $GOOS and you start debugging on that OS, you don't want to know whether your vendored dependencies still pass there?

If you really don't want to be bothered with test failures, just instruct your vendoring tool to delete the test files.

[matt....@gmail.com]

If package a imported packages b and c where b and c both vendored a different version of package d what would happen?

On Thursday, June 11, 2015 at 9:21:06 AM UTC-4, rsc wrote:

Thanks, Keith. I agree with you that the go command should do this for Go 1.5, as an experiment guarded by a -vendor flag.

I apologize for the delay in responding. I wanted to let both this discussion and the broader conversation proceed, especially since the community understands and encounters these problems more than we on the Go team at Google do. This has happened: we now have much more information from the community about the problem being solved, both in descriptions and in code, like godep, nut, and gb.

We have been rereading the discussions here and looking at the landscape of both Go vendoring tools and large projects that use them. One fact is clear: people very much want to copy (“vendor”) source code into their projects without modifying import paths. Godep and similar tools do this with GOPATH manipulation, and gb does it by reimplementing the build system. In both cases, the result is not compatible with “go get”: people using standard tools cannot easily install projects managed with tools like gb and godep. The proposal in this thread can help the go command meet these tools halfway, so that, perhaps along with minor changes to gb, godep, and friends, users will be able to “go get” those projects.

Specifically, we propose that, as an experiment for Go 1.5, we add a temporary “-vendor” flag that causes the go command to add these semantics:

If there is a source directory d/vendor, then, when compiling a source file within the subtree rooted at d, import "p" is interpreted as import "d/vendor/p" if that exists.

When there are multiple possible resolutions, the most specific (longest) path wins.

The short form must always be used: no import path can contain “/vendor/” explicitly.

Import comments are ignored in vendored packages.

This is a limited form of the original proposal in this thread.

In the original proposal (as I understand it), the interpretation of an import depends both on where the source code containing that import sits in the tree and why the package is being compiled. The meaning of an import in package X is different when X is imported by Y than when X is imported by Z. Therefore the compilation would have to happen multiple times. This doesn’t scale and possibly leads to inconsistent constraints.

In the revised proposal, the interpretation of an import depends only on where the source code containing that import sits in the tree.

The revised proposal also uses the word “vendor” instead of “external”, because (1) there is at least one popular vendoring tool (gb) that uses “vendor” and none that we know of that use “external”; (2) “external” sounds like the opposite of “internal”, which is not the right meaning; and (3) in discussions, everyone calls the broader topic vendoring. It would be nice not to bikeshed the name.

As an aside, the terms “internal vendoring” and “external vendoring” have been introduced into some discussions, to make the distinction between systems that rewrite import paths and systems that do not. With the addition of vendor directories to the go command, we hope that this distinction will fade into the past. There will just be vendoring.

Example

The gb project ships an example project called gsftp. It has a gsftp program with three dependencies outside the standard library: golang.org/x/crypto/ssh, golang.org/x/crypto/ssh/agent, and github.com/pkg/sftp.

Adjusting that example to use the new vendor directory, the source tree would look like:

$GOPATH

| src/

| | github.com/constabulary/example-gsftp/

| | | cmd/

| | | | gsftp/

| | | | | main.go

| | | vendor/

| | | | github.com/pkg/sftp/

| | | | golang.org/x/crypto/ssh/

| | | | | agent/

The file github.com/constabulary/example-gsftp/cmd/gsftp/main.go says:

import (

...

"golang.org/x/crypto/ssh"

"golang.org/x/crypto/ssh/agent"

"github.com/pkg/sftp"

)

Because github.com/constabulary/example-gsftp/vendor/golang.org/x/crypto/ssh exists and the file being compiled is within the subtree rooted at github.com/constabulary/example-gsftp (the parent of the vendor directory), the source line:

import "golang.org/x/crypto/ssh"

is compiled as if it were:

import "github.com/constabulary/example-gsftp/vendor/golang.org/x/crypto/ssh"

(but this longer form is never written).

So the source code in github.com/constabulary/example-gsftp depends on the vendored copy of golang.org/x/crypto/ssh, not one elsewhere in $GOPATH.

In this example, all the dependencies needed by gsftp are (recursively) supplied in the vendor directory, so “go install” does not read any source files outside the gsftp Git checkout. Therefore the gsftp build is reproducible given only the content of the gsftp Git repo and not any other code. And the dependencies need not be edited when copying them into the gsftp repo. And potential users can run “go get github.com/constabulary/example-gsftp/cmd/gsftp” without needing to have an additional vendoring tool installed or special GOPATH configuration.

The point is that adding just the vendor directory mechanism to the go command allows other tools to achieve their goals of reproducible builds and not modifying vendored source code while still remaining compatible with plain “go get”.

Discussion

There are a few points to note about this.

The first, most obvious, and most serious is that the resolution of an import must now take into account the location where that import path was found. This is a fundamental implication of supporting vendoring that does not modify source code. However, the resolution of an import already needed to take into account the current GOPATH setting, so import paths were never absolute. This proposal allows the Go community to move from builds that require custom GOPATH configuration beforehand to builds that just work, because the (more limited) configuration is inferred from the conventions of the source file tree. This approach is also in keeping with the rest of the go command.

The second is that this does not attempt to solve the problem of vendoring resulting in multiple copies of a package being linked into a single binary. Sometimes having multiple copies of a library is not a problem; sometimes it is. At least for now, it doesn’t seem that the go command should be in charge of policing or solving that problem.

The final point is that existing tools like godep, nut, and gb will need to change their file tree layouts if they want to take advantage of compatibility with “go get”. However, compatibility with “go get” used to be impossible. Also, combined with eventual agreement on the vendor-spec, it should be possible for the tools themselves to interoperate.

Deployment

The signals from the Go community are clear: the standard go command must support building source trees in which dependencies have been vendored (copied) without modifications to import paths.

We are well into the Go 1.5 cycle, so caution is warranted. If we put off making any changes, then vendoring tools and “go get” will remain incompatible for the next eight months. On the other hand, if we can make a small, targeted change, then vendoring tools can spend the next eight months experimenting and innovating and possibly converging on a common file tree layout compatible with the go command.

We believe that the experimental proposal above is that small, targeted change. It seems to be the minimal adjustment necessary to support fetching and building vendored, unmodified source code with “go get”. There are many possible extensions or complications we might consider, but for Go 1.5 we want to do as little as possible while remaining useful.

The use of an explicit -vendor flag should help contain the experiment now and ease the transition later. For Go 1.5, users who want to participate in the experiment can opt in by using “go get -vendor”, “go install -vendor”, and so on.

The new semantics changes the meaning of (breaks) source trees containing directories already named “vendor”. Of the over 60,000 listed on godoc.org, there are fewer than 50 such examples. Putting the new semantics behind the -vendor flag avoids breaking those trees for now.

If we decide that the -vendor behavior is correct, then in a later release (possibly Go 1.6) we would make the -vendor flag default to true. Projects containing “vendor” directories could still use “-vendor=false” to get the old behavior while they convert their code. In a still later release (possibly Go 1.7) we would remove the -vendor flag, locking in the vendoring semantics.

Comments welcome here. I just sent CLs 10922 and 10923 as a prototype.

Thanks very much for this proposal.

Russ

[Nicolas Grilly]

On Thursday, June 11, 2015 at 3:21:06 PM UTC+2, rsc wrote:

Specifically, we propose that, as an experiment for Go 1.5, we add a temporary “-vendor” flag that causes the go command to add these semantics:

If there is a source directory d/vendor, then, when compiling a source file within the subtree rooted at d, import "p" is interpreted as import "d/vendor/p" if that exists.

When there are multiple possible resolutions, the most specific (longest) path wins.

The short form must always be used: no import path can contain “/vendor/” explicitly.

Import comments are ignored in vendored packages.

This is great improvement to the go tool.

I'm wondering what would happen if:

- I have a package in `$GOPATH/src/myproject` that depends on `somedependency` stored in `$GOPATH/src/myproject/vendor/somedependency`.

- I build my project.

- I copy a different version of `somedependency` in `$GOPATH/src/somedependency`.

- I delete the vendored version in `$GOPATH/src/myproject/vendor/somedependency`.

- I rebuilt my project.

Will the go tool detect that `somedependency` is stale and rebuild it using the version in `$GOPATH/src/somedependency`?

Thanks,

- Nicolas Grilly

[Mateusz Czapliński]

Ah, and by the way, talking of sense: the deduplication may actually be not that trivial and obvious: if pkg foo has some internal state (for example, something like database/sql's Register()), then dedupping it (even if both are identical) may change behavior; arguably, we can say now that "that'd be the package's fault", etc, and: "undetectable without solving the stop problem". But, at least good to be aware, probably.

/M.

[Dan Kortschak]

On Fri, 2015-06-12 at 06:21 -0700, Ian Lance Taylor wrote:
> On Fri, Jun 12, 2015 at 2:22 AM, Mateusz Czapliński
> <czap...@gmail.com> wrote:
> >
> > I seem to recall that some earlier semi-official discussion/proposal
> > explored an idea of vendoring only "command" packages (i.e. ones
> with
> > "package main").
>
> I don't recall that, and to be honest it doesn't seem useful.

I think these comments are those in question:

adg: https://groups.google.com/forum/#!msg/golang-dev/nMWoEAG55v8/GXh7DT7HknoJ
rsc: https://groups.google.com/forum/#!msg/golang-dev/nMWoEAG55v8/C1S8bhKoCrEJ

[Matthew Sackman]

On Fri, Jun 12, 2015 at 02:59:56PM -0700, Mateusz Czapliński wrote:
> Ah, and by the way, talking of sense: the deduplication may actually be not
> that trivial and obvious: if pkg foo has some internal state (for example,
> something like database/sql's Register()), then dedupping it (even if both
> are identical) may change behavior; arguably, we can say now that "that'd
> be the package's fault", etc, and: "undetectable without solving the stop
> problem". But, at least good to be aware, probably.

Oh come on: surely everyone writes perfectly pure, referentially
transparent, side-effect-free code all the time? Surely!

Matthew

[Russ Cox]

On Thu, Jun 11, 2015 at 8:09 AM, Nate Finch <nate....@gmail.com> wrote:

I think this is exactly the right approach.

Thanks.

On Thu, Jun 11, 2015 at 8:37 AM, Daniel Theophanes <kard...@gmail.com> wrote:

I'm happy with this proposal.

Thanks.

 

From the proposed CLs, it looks like tools that re-write import paths should not have the "/vendor/" in the path. I think that's fine; existing tools don't currently place them there anyway.

I think having a flag as currently proposed would allow existing tools to easily opt in. If people wanted to use the go command directly, it might be prudent to support an environment variable as well "GO_VENDOR=1".

I think you are probably right.

Russ

[Russ Cox]

I'm not making any assumption about this being a valid general transformation. If it were, there would be a stronger case for disallowing complex trees like in the original example. In fact, what I believe is that deduplicating in a sensible way is a very difficult problem, best left to other tools, without the go command's build rules imposing additional constraints on the result.

Russ

[Russ Cox]

On Fri, Jun 12, 2015 at 12:44 PM, <matt....@gmail.com> wrote:

If package a imported packages b and c where b and c both vendored a different version of package d what would happen?

The final binary using package a would end up with both copies of package d in it. They would have different completed import paths, so they would be kept separate like any other two packages. (There would not be name collisions or any other kinds of conflicts.) Whether the duplication is harmless depends on what package d is. I expect other tools to help analyze these situations and figure out what, if anything, needs to be done.

Russ

[Russ Cox]

On Fri, Jun 12, 2015 at 12:09 PM, Joe Shaw <j...@joeshaw.org> wrote:

"go test ./..." is very convenient.  If to test only my own code I must "go test $(go list ./... | grep -v vendor)" or whatever the command is, then I might as well start using a Makefile and "make test" instead.

As I said in my reply to Tim, there is a more general problem here: go test doesn't know which tests merit being rerun. That's getting a bit off topic for this thread, but it's something I've talked to Tim about in the past in the context of Kubernetes (a fairly large tree) and definitely something we want to address. It won't be in Go 1.5, but I plan to try to address it with a separate tool for now and, with some experience, maybe we can get something into Go 1.6.

Vendored code is just a special case of "code I didn't touch recently that can't possibly have started breaking".

I've been thinking about this for a few months but hadn't created a GitHub issue. I just did: golang.org/issue/11193. If anyone is interested in this, please star the issue to follow along, or create a separate golang-dev thread.

Russ

[Russ Cox]

On Fri, Jun 12, 2015 at 10:50 AM, Nicolas Grilly <nic...@vocationcity.com> wrote:

I'm wondering what would happen if:

- I have a package in `$GOPATH/src/myproject` that depends on `somedependency` stored in `$GOPATH/src/myproject/vendor/somedependency`.

- I build my project.

- I copy a different version of `somedependency` in `$GOPATH/src/somedependency`.

- I delete the vendored version in `$GOPATH/src/myproject/vendor/somedependency`.

- I rebuilt my project.

Will the go tool detect that `somedependency` is stale and rebuild it using the version in `$GOPATH/src/somedependency`?

Yes.

Russ

[Trans]

Thanks for this well considered response. In reply let me first back
up and address `internal` since that is clearly the pattern that this
`vendor` idea is mimicking.

The idea of `internal` is essentially to create *private* packages.
Whenever the possibility of making something private comes into play,
whatever the perceived benefits, there are also serious negative
consequences that follow. Coders can (and will) start making
everything but the API for their particular perceived public use case
private. Consequently code reuse is increasingly limited. Coders can
also hide poor code behind the curtain, so to speak. A good analogy to
this are Java class private members -- the bane of reuse. Another
persons Java API may contain a class that is almost what you need, but
alas b/c of the extensive use of private members it is not possible to
subclass and adjust the behavior, hence reuse defeated. Ruby is the
antitheses of this, and the language that (IMHO) got it right. There
is next to nothing that can't be subclassed and reused. I feel the
same about `internal`, it will turn out to be a hindrance more than a
benefit. If you think about it, the idea of `internal` is to create
packages that aren't packages, and then it is obvious that it is an
oxymoron. If there is code that needs to be shared by separate
packages, then just accept that is so and share it as another package.
If it not something for general consumption, because it might change
or what have you, then just document it as such. Perhaps even have a
warning system. But preventing people from access to code beyond
copy-and-paste is an unnecessary limitation --it is more a perceived
problem than a real one.

So given that, now we have this new `vendor` proposal built on the
premise of `internal`. Vendoring is more than a fancy means of
copy-and-paste and not having to modify the import lines. There are
also questions of version conflict and recursion. You want to leave
those issues to other tools while defining the minimum to get just as
far along as you need at the moment, but that approach can have
unintended consequences. As you said about `internal`, now that you
have introduced it you can hardly take it back. (Though I remind you
it was supposed to be a "test" in 1.4.) Given this, whatever you
choose to do, I caution you to take it slow and think it all through
carefully. There is no need to rush.

As for my own druthers, I much rather see a standard dependency file
that maps an import name to a package name. This way the import name
is always the same regardless of the location -- and it can be nice
and short. Of course, this would give people the freedom to be rather
wonky with import names, but good coders will stick to standards. Then
have a toplevel `ext` directory for the all vendored packages -- no
need for deep directory structures. And note, *vendored packages
should not be checked into the scm!*

[Andrew Gerrand]

This was my initial concern with "internal", and I pushed back when the concept was introduced. Having now watched it in action in the core and other repositories I think our concern is more theoretical than practical, and that the internal mechanism provides a nice way to encapsulate weirdness that would be unfortunate to expose to the outside world.

I caution you to take it slow and think it all through
carefully. There is no need to rush.

We've been watching and thinking about it for three years. :-) 

 

As for my own druthers, I much rather see a standard dependency file
that maps an import name to a package name. This way the import name
is always the same regardless of the location -- and it can be nice
and short. Of course, this would give people the freedom to be rather
wonky with import names, but good coders will stick to standards. Then
have a toplevel `ext` directory for the all vendored packages -- no
need for deep directory structures. And note, *vendored packages
should not be checked into the scm!*

You may be interested in this effort,

https://github.com/kardianos/vendor-spec 

which seeks to provide a standard file format for dependency management metadata. That, and the change proposed by this thread, should provide a clean base on which to experiment with various compatible dependency management approaches.

Andrew

 

[Nicolas Grilly]

On Sat, Jun 13, 2015 at 1:25 AM, Russ Cox <r...@golang.org> wrote:

Will the go tool detect that `somedependency` is stale and rebuild it using the version in `$GOPATH/src/somedependency`?

Yes.

That's a very clear answer. Thanks!

[Nicolas Grilly]

On Thursday, June 11, 2015 at 3:21:06 PM UTC+2, rsc wrote:

Specifically, we propose that, as an experiment for Go 1.5, we add a temporary “-vendor” flag that causes the go command to add these semantics:

If there is a source directory d/vendor, then, when compiling a source file within the subtree rooted at d, import "p" is interpreted as import "d/vendor/p" if that exists.

When there are multiple possible resolutions, the most specific (longest) path wins.

The short form must always be used: no import path can contain “/vendor/” explicitly.

Import comments are ignored in vendored packages.

What are the consequences of this change for godoc?

If godoc is started with the -vendor flag, I would expect hyperlinks to target the vendored versions of packages if they exist (by applying the import rules defined in your proposal).

- Nicolas

[Russ Cox]

Probably godoc will not know about vendoring for Go 1.5. Part of the experiment is learning what tools need to be updated for when we make it official. Godoc is certainly one of them. I expect it would (be made to) get the hyperlinks in import statements correct.

Russ

[Nicolas Grilly]

On Thu, Jun 18, 2015 at 5:46 PM, Russ Cox <r...@golang.org> wrote:

Probably godoc will not know about vendoring for Go 1.5. Part of the experiment is learning what tools need to be updated for when we make it official. Godoc is certainly one of them. I expect it would (be made to) get the hyperlinks in import statements correct.

Russ, thanks for the explanation. It sounds reasonable.

[Russ Cox]

On Thu, Jun 11, 2015 at 9:21 AM, Russ Cox <r...@golang.org> wrote:

Thanks, Keith. I agree with you that the go command should do this for Go 1.5, as an experiment guarded by a -vendor flag.

This is now committed. Instead of a -vendor flag you set GO15VENDOREXPERIMENT=1 in your environment.

Please feel free to try it out and report problems; note that the Go tree at tip is not terribly stable right now, so it shouldn't be used for building production code. But experimenting with the vendor support should be fine.

Thanks.

Russ

[chai2010]

Failed on windows :(

go run hello.go

invalid vendoredImportPath: dir="D:\\apps\\go\\gopkg\\src\\mydev\\hello\\a" root="d:\\apps\\go\\gopkg" separator="\\"

// dir="D:\\...", root="d:\\..."

go version

go version devel +3cab476 Sun Jun 21 03:11:01 2015 +0000 windows/amd64

go env

set GOARCH=amd64

set GOBIN=

set GOEXE=.exe

set GOHOSTARCH=amd64

set GOHOSTOS=windows

set GOOS=windows

set GOPATH=d:\apps\go\gopkg

set GORACE=

set GOROOT=d:\apps\Go\gotip

set GOTOOLDIR=d:\apps\Go\gotip\pkg\tool\windows_amd64

set CC=gcc

set GOGCCFLAGS=-m64 -mthreads -fmessage-length=0

set CXX=g++

set CGO_ENABLED=1

--
You received this message because you are subscribed to the Google Groups "golang-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

https://github.com/golang-china

https://github.com/chai2010

[Mateusz Czapliński]

On Friday, June 19, 2015 at 9:28:37 PM UTC+2, rsc wrote:

Please feel free to try it out and report problems; note that the Go tree at tip is not terribly stable right now, so it shouldn't be used for building production code. But experimenting with the vendor support should be fine.

Related to "_vendor" and "./..." topics: I'm now somewhat often doing something like below:

  $ cd $GOPATH/src/mycompany.com
  $ go install ./...

When doing that, I'm not interested in all random apps that may lie somewhere in my vendor/ subdir (i.e. $GOPATH/src/mycompany.com/vendor), which can be present there if we assume that vendor/ subdirs have all the contents of corresponding Git(/hg/bzr/...) repositories. I've tested with the current tip (go version devel +d16c7f8 Thu Jul 2 03:43:34 2015 +0000 linux/amd64) and they all seem to get goinstalled with the above commandline, unfortunately for me. I especially don't like that it introduces a risk if multiple cmds in vendor/ have the same names (I wouldn't know which one got installed, and also they could override some apps I explicitly goinstalled earlier on purpose). I'd like it very much if I could still use a command like above, and not have to write the likes of `go install $(go list ./... | grep -v vendor)`, which would also be especially troublesome on Windows.

Thinking about solutions, I considered whether the "vendoring tool" could maybe delete any cmds in the vendor/ tree; but a huge disadvantage of this would be that I couldn't store full repos in the vendor/ subdir. If the dir was named _vendor, it seems it would solve this issue for me. Also, a flag in `go install` could do that, I suppose. Don't have any ideas for other solutions as of now.

Thanks,
/Mateusz.