how to avoid breaking build with GO15VENDOREXPERIMENT

[Matt Farina]

If you use the vendor experiment to put external packages into the vendor/ folder and then others pull your package into theirs it's easy to run into a problem. For example see https://github.com/mattfarina/golang-broken-vendor.

I've not seen instruction in how to avoid that situation. And, it becomes complex for some projects such as k8s. k8s is an application in its own right and others can include it (or parts of it) for their SDK to k8s. Applications that have a dual purpose can cause problems.

I'm curious what thought has been put into this situation.

[Russ Cox]

If you want to share code with others and can't be sure they all set GO15VENDOREXPERIMENT, then don't use vendor directories yet. Wait until vendor support is on by default in Go 1.6 (and you can be sure those you want to share with are all up to 1.6).

Russ

 

--
You received this message because you are subscribed to the Google Groups "golang-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Matt Farina]

Unfortunately, waiting for 1.6 isn't going to solve this problem. It may only get worse. Let me explain.

Quite often you don't know who you're going to share with. The code goes up on GitHub and none or many people may use it. If package authors don't know about this behavior they may store the packages they use in their vendor/ folder. Then, when someone else uses that package and has their own dependencies in their vendor/ folder this problem can arise. In that case, what's someone to do?

I found this problem in practice by using packages where everyone expected the GO15VENDOREXPERIMENT to be on. People were not aware of this behavior.

Once the vendor experiment is on by default a larger audience will be open to this problem.

[Russ Cox]

On Tue, Nov 3, 2015 at 2:48 PM, Matt Farina <matt....@gmail.com> wrote:

Unfortunately, waiting for 1.6 isn't going to solve this problem. It may only get worse. Let me explain.

Quite often you don't know who you're going to share with. The code goes up on GitHub and none or many people may use it. If package authors don't know about this behavior they may store the packages they use in their vendor/ folder. Then, when someone else uses that package and has their own dependencies in their vendor/ folder this problem can arise. In that case, what's someone to do?

I found this problem in practice by using packages where everyone expected the GO15VENDOREXPERIMENT to be on. People were not aware of this behavior.

Once the vendor experiment is on by default a larger audience will be open to this problem.

I am not sure I understand the problem. I thought the problem was that some people had vendor semantics enabled and some did not. That's a short term problem: eventually close enough to everyone will have them enabled (because they're using Go 1.6 or later). 

But it sounds like you are describing a problem that arises even when everyone involved agrees that vendor semantics are enabled. What is the problem in that case?

Thanks.

Russ

[Matt Farina]

@rsc did you look at the example I linked off to? When the same package is in 2 vendor/ locations. It could be more but the linked example only has 2. In case you missed it, there example is at https://github.com/mattfarina/golang-broken-vendor

It has to do with how, in practice, people use the vendor/ directories. This issue has been seen in the wild.

[minux]

On Tue, Nov 3, 2015 at 3:47 PM, Matt Farina <matt....@gmail.com> wrote:

@rsc did you look at the example I linked off to? When the same package is in 2 vendor/ locations. It could be more but the linked example only has 2. In case you missed it, there example is at https://github.com/mattfarina/golang-broken-vendor

It has to do with how, in practice, people use the vendor/ directories. This issue has been seen in the wild.

It's working as intended. Two vendored copies of the same package are not

the same package (their import paths are different).

The vendor experiment merely provides a mechanism, the vendor policy is

not and shouldn't be enforced by the go tool.

[Dave Cheney]

So here's the problem,

The way the vendor rule works at the moment is vendor/x is imported as "x", but linked as " vendor/x"

This introduces all the problems of putting version numbers in imports, aka gopkg.in. specifically

* duplicate calls to init routines to register drivers and loggers
* types that fail type assertion checks.

However, to do the opposite, vendor/x linking as x would conflict with other x's, possibly from GOPATH or other nested, or sibling vendor structures.

This would cause linking failures as symbols wouldn't match, or worse, no linking failures and structures being the wrong size at run time.

Either way is problematic, and I don't see a way to resolve Matt's concerns.

Thanks

Dave

--

[Matt Farina]

@dave If this is what we have I see two possibilities of ways to use it in practice. Something we'll need to teach people.

1. No one stores their dependences in the vendor/ in their VCS (unless you really know what you're doing... an exception). Then you use tooling to populate the vendor/ folder when needed (dev, test, and build environments). The tooling uses config files (and nested ones in dependencies) to populate it. This allows you to use it and have different versions of packages in different applications. This is basically the model used by PHP, node.js, and many other languages. Let the tooling manage your vendor/ folder based on config so you can avoid bad situations.
2. Tooling walks through all the dependencies and moves vendor dependencies to the top level. This may be opt-in and would need to rely on some config (to handle knowledge about versions so you can deal with the same package multiple times but different versions).

Really, the first case it tooling to manage you vendor/ folder to avoid the problems. The second it tooling to fix the problems.

I personally prefer the former because it's an easier problem to solve and is generally known because so many other languages tooling operate this way. It's familiar and we can learn from their struggles.

Without tooling to "manage" this it quickly becomes painful.

Unless you have a better idea (and I'm looking for one).

- Matt

[Dave Cheney]

On Wed, Nov 4, 2015 at 10:28 AM, Matt Farina <matt....@gmail.com> wrote:

@dave If this is what we have I see two possibilities of ways to use it in practice. Something we'll need to teach people.

1. No one stores their dependences in the vendor/ in their VCS (unless you really know what you're doing... an exception). Then you use tooling to populate the vendor/ folder when needed (dev, test, and build environments). The tooling uses config files (and nested ones in dependencies) to populate it. This allows you to use it and have different versions of packages in different applications. This is basically the model used by PHP, node.js, and many other languages. Let the tooling manage your vendor/ folder based on config so you can avoid bad situations.

This wound nullify what I see as the main value of the vendor directory; vendoring code into your repository. If this cannot be done, then what is the point of vendor ? Why not just have tools that manage GOPATH directly, like https://launchpad.net/godeps ?

 

1. Tooling walks through all the dependencies and moves vendor dependencies to the top level. This may be opt-in and would need to rely on some config (to handle knowledge about versions so you can deal with the same package multiple times but different versions).

This sounds like a better option, but how do you handle conflicts between a deps that live in a (flattened) vendor/ folder, and raw, or transitively (see below)  in GOPATH ?

github.com/

        /foo/bar/

              /vendor/

                    /logger

        /quxx/frob

             /vendor/

                    /logger

If quxx/frob imports "logger" and "foo/bar", which copy of logger is "foo/bar" going to link against ? I don't think flattening can solve this.

[Dave Cheney]

Replying to myself on the second point. It would appear that if you vendor/ anything, you must therefore vendor everything as a partial solution introduces ambiguity. This is probably something tooling can help with.

[Matt Farina]

Dave,

In ref to #1, what if you're working on more than one application and they use different versions of the same dependency? How do you handle that with one gopath?

Oh, and I understand how a project specific workspace solves that. I'm attempting to look at this through the go tool.

To flatten multiple projects to the top level and handle versions you'd need a record of versions. Ideally, to handle mild differences they'd be versions as semver constraints and the tooling picks the best version to meet those (if one is available).

If you need that record of versions somewhere and to have to restructure a codebase what benefit is there to vendoring?

- Matt

--

Matt Farina

mattfarina.com

Go in Practice - A book of Recipes for the Go programming language.

Engineered Web - A blog on cloud computing and web technologies.

[Dave Cheney]

On Wed, Nov 4, 2015 at 10:58 AM, Matt Farina <matt....@gmail.com> wrote:

Dave,

In ref to #1, what if you're working on more than one application and they use different versions of the same dependency?

This has never been possible, the linker prevents it.

 

How do you handle that with one gopath?

If you have one entry in your gopath, then there can only be one revision of the code for that package there. If you have multiple, then the first found wins, for the same reason you cannot have a package called "net" in GOPATH, GOROOT always preceeds it.

 

Oh, and I understand how a project specific workspace solves that. I'm attempting to look at this through the go tool.

To flatten multiple projects to the top level and handle versions you'd need a record of versions. Ideally, to handle mild differences they'd be versions as semver constraints and the tooling picks the best version to meet those (if one is available).

If you need that record of versions somewhere and to have to restructure a codebase what benefit is there to vendoring?

Indeed.

[Dave Cheney]

On Wed, Nov 4, 2015 at 11:01 AM, Dave Cheney <da...@cheney.net> wrote:

On Wed, Nov 4, 2015 at 10:58 AM, Matt Farina <matt....@gmail.com> wrote:

Dave,

In ref to #1, what if you're working on more than one application and they use different versions of the same dependency?

This has never been possible, the linker prevents it.

Sorry that response was incomplete. In the example I work on daily in Juju, every time we git checkout a branch, we must remember to run godeps -u dependencies.tsv to adjust the rest of the GOPATH to match the expectations of this branch. This approach is error prone, and a key driver behind the project based approach that gb uses.

[minux]

On Tue, Nov 3, 2015 at 5:09 PM, Dave Cheney <da...@cheney.net> wrote:

So here's the problem,

The way the vendor rule works at the moment is vendor/x is imported as "x", but linked as " vendor/x"

This is documented in the design docs for the vendoring experiment.

To put it simply, it is a way to rewrite import paths without physically rewriting the import paths in source

code, but all the caveats from the latter apply.

[minux]

On Tue, Nov 3, 2015 at 6:28 PM, Matt Farina <matt....@gmail.com> wrote:

@dave If this is what we have I see two possibilities of ways to use it in practice. Something we'll need to teach people.

1. No one stores their dependences in the vendor/ in their VCS (unless you really know what you're doing... an exception). Then you use tooling to populate the vendor/ folder when needed (dev, test, and build environments). The tooling uses config files (and nested ones in dependencies) to populate it. This allows you to use it and have different versions of packages in different applications. This is basically the model used by PHP, node.js, and many other languages. Let the tooling manage your vendor/ folder based on config so you can avoid bad situations.
2. Tooling walks through all the dependencies and moves vendor dependencies to the top level. This may be opt-in and would need to rely on some config (to handle knowledge about versions so you can deal with the same package multiple times but different versions).

Really, the first case it tooling to manage you vendor/ folder to avoid the problems. The second it tooling to fix the problems.

The way I see the problem is that, library packages should not vendor packages unless

absolutely necessary. And if they do, they must make sure that the API interfaces from

the vendored packages are not leaked to the outside of the library package. (I.e. if you

make a private copy of net/http in your package, it's fine if its use is purely internal to

the package.)

Only applications should vendor packages.

To this end, I actually like the original recommendation of vendoring by rewriting import

paths. At least people know that vendored packages are different from the original one.

[Matt Farina]

Dave,

With the vendor/ directory each application can have their own versions of a packages. They just need to be in the top level vendor/ directory and the linker works. That's what makes the GO15VENDOREXPERIMENT interesting. Not because of the ability to store outside pacakges in your vcs repo. It's interesting because it provides the first sane way to handle multiple versions of packages. At least that's why it's interesting to me.

- Matt

 

- Matt

To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Matt Farina]

@minux what's an application? For example, Kubernetes is both an application and internally there's an SDK that others should import. Docker is an application and many people import it to leverage it. They are both. It's not always cut and dry.

I would not expect most people to read the design document for vendor/. Instead, I would expect them to think it operates the same way this directory does for PHP, node.js, and virtually every other language that has something comparable. Most people don't read the docs... especially the design docs. They first go with their intuition and experience. Experience with other languages has trained a lot of devs how something like this should work.

[Daniel Theophanes]

Hi Matt,

govendor is a tool to manage dependencies. It moves dependencies to the top level. It tries to use "vendor-spec" to see if there is a real difference in names in nested dependencies. If there is one, it asks the user what it should do.

-Daniel

[Yongjian Xu]

I think what minux@ meant by "application" is that it is a binary containing a main package. For a library (e.g. as you said it could be something common abstracted out as a library from an application), the question of how to resolve multiple versions of the same vendored package seems unavoidable... (which is a hard problem I think and the original spec also chosen to leave it out)

--

[Matt Farina]

Hi Daniel,

I understand how 3rd party tools can help with this. Glide flattens to the top level and works with Godep, gb, and GPM in addition to Glide config files to help resolve versions. In fact, we're open to supporting vendor-spec as an import. It's just that no one has asked us for it and we've not needed to support it, yet. Pull requests welcome.

But, what about those not using 3rd party tools? And, what's the theoretical best solution, why, and how should that be communicated? That's what I'm interested in here.

- Matt

[Matt Farina]

@Jimxu I'm not sure version handling and packages is too hard to solve. Every other modern language ecosystem has it solved. That includes Rust, PHP, JavaScript, Java, Ruby, Python, Objective-C, Swift, and others. Not only do they have this problem solved, they don't run into issues like the one I originally outlined here.

[Yongjian Xu]

On Wed, Nov 4, 2015 at 1:05 PM Matt Farina <matt....@gmail.com> wrote:

@Jimxu I'm not sure version handling and packages is too hard to solve. Every other modern language ecosystem has it solved. That includes Rust, PHP, JavaScript, Java, Ruby, Python, Objective-C, Swift, and others. Not only do they have this problem solved, they don't run into issues like the one I originally outlined here.

Sure. I was just suspecting that this would be a real issue under the current implementation of the Go toolchain. I am not familiar with how other languages had solved this issue. Are they just pick one of the copies and compile that or their toolchains natively support compile/link multiple versions into one binary? Or they rely on some kind of config system input?

On Wednesday, November 4, 2015 at 12:27:05 PM UTC-5, Jimxu wrote:

I think what minux@ meant by "application" is that it is a binary containing a main package. For a library (e.g. as you said it could be something common abstracted out as a library from an application), the question of how to resolve multiple versions of the same vendored package seems unavoidable... (which is a hard problem I think and the original spec also chosen to leave it out)

--

[Matt Farina]

@jimxu There are two ways other languages handle this. In some cases they can cleanly handle multiple versions. In other cases they flatten the dependency tree. In both cases they have tooling that basically does this for them. It's mostly automated.

The timing of your question is great because I just wrote http://engineeredweb.com/blog/2015/pkg-mgr-overview/ for an entirely different purpose that talks about how the package managers for other languages handle packages. Maybe it will be useful here as well.

[sna...@gmail.com]

This sort of issue around the vendor experiment is one I've been concerned with for some time. I'm glad it's finally being discussed. I have a fairly specific opinion:

On Tuesday, November 3, 2015 at 4:39:37 PM UTC-8, minux wrote:

Only applications should vendor packages.

 ^ This. And I believe a way to do this is to only have the vendor/ pull from a location relative to the "main" package of an application (rather than relative to the current package). This will prevent impossible diamond dependencies (package a vendors package c, package b vendors packages c, c.type returned by a.func is incompatible with c.type accepted by b.func), I illustrate this issue in a simple application here: https://github.com/conslo/vendor_problem. However by leaving the dynamic nature of the vendor folder location, one can still share a vendor folder among multiple projects (github.com/vendor as a silly example for anything from github)

I don't personally agree with rewriting import paths, and I think the community has shown it doesn't either (hence this experiment). And as for letting external tooling solve these problems, I feel that goes against the idea that anything should be "go get-able", requiring tooling outside what's provided by the language to interact with projects seems... silly. Other languages solve these problems by having package managers: a canonical source with canonical names, and version numbers for packages. Neither of these are being provided or enforced, and expecting a tool to do some sort of versioning resolution without any sort of in-built version identification (semver, anyone?) seems... also silly.

In addition, the general theme I've seen, including with this vendor experiment, seems to assume "a future version of package x will always be compatible with a previous version of package x". When you work in a totally vendored environment where there's only one version of any package at any time (sound familiar?) this works fine; breaking changes are made atomically, everyone is compatible with each other, and bad versions are fixed and never seen again. However, this is not what open source is like; when a package needs to make breaking changes, it has to maintain the old version for a while so people have time to transition, people try to pin to versions, but we're not all pinned to the same one. There are multiple versions (some/many incompatible with each other) of any one dependency in use. This is why communities have, mostly, learned to use semantic versioning: I need a feature in 1.5, and you need 1.6, so above 1.6 and below 2.0 should work for both of us, if we find a version that doesn't work there should be a patch version after it that fixes the break. As of right now there is not widely accepted mechanism for declaring versions of dependencies, just a canonical URL and an expectation that that URL will never have changes that break me, there's gopkg.in (which is awesome!), but if a package has internal packages it doesn't work unless the developer specifically includes it (because, you know, no relative imports ;)). Some things enable pinning, but they generally pin to a commit hash, not a "version", so any dependency resolution isn't really enabled. Perhaps a discussion of versioning as a whole is in order? An endorsed mechanism for declaring versions (tags/branches? I know git/mercurial but not bazaar or others), and/or an endorsed method of declaring versions of dependencies would be incredible (for the later, I actually think a language addition for suffixing the version in the import would be awesome: import name "path" "version").

[Matt Farina]

I think what you're saying is we need:

1. A place to store dependencies other than the GOPATH for a main package
   
2. A way to record what versions of a dependency a package relies on. Then the tooling can try to resolve the right one to put in the store for #1.

For anyone wonder why a per application store, rather than the GOPATH, that's because different applications will resolve to different versions for different applications.

For versioning there is already a proposal to use SemVer at https://github.com/golang/go/issues/12302.

While I like gopkg.in it's limited to public projects on GitHub. There are just so many cases where that won't work that Go needs to support.

I'm wary of putting the version in the GOPATH because, like you said, it's a version range. How would you put ">= 1.2.3, < 2.0.0, != 1.4.5" in the import path? If it's just a single version you have a problem of resolution in a complex project.

And, this is why I feel this is stuck.

Given the way Google works with code I wouldn't ask them to handle #2. People who use the tooling should be the ones to write it. As for a #1 store, is a vendor/ directory in main the best option or should there be something better?

[sna...@gmail.com]

On Thursday, November 5, 2015 at 3:46:52 AM UTC-8, Matt Farina wrote:

I'm wary of putting the version in the GOPATH because, like you said, it's a version range. How would you put ">= 1.2.3, < 2.0.0, != 1.4.5" in the import path? If it's just a single version you have a problem of resolution in a complex project.

This isn't precisely what I suggested. It wouldn't be in the import path, but rather an optional piece of metadata:

import (

    name "github.com/some/package" ">=1.2.3,<2.0.0,!=1.4.5"

)

It could also be a comment on the same line, rather than part of the syntax (so it's just for external tooling), though on second thought requiring a version specification in every file that uses a package seems excessive.

I agree with your two pointed list. And personally I feel a vendor/ director in main is an excellent option (motivated by the acceptance so far) for #1. I'll have to disagree with you on #2 a bit though, assuming 12302 gets accepted in some form, I feel it *is* google's responsibility to give us a canonical way to deal with these things, solving problems like this early on (rather than letting someone make a most-of-the-way solution get widely accepted after it's become a large enough problem) I feel are intensely important to the health of the ecosystem. I don't expect them to make the tool itself (though accepting one into the core tools later would be nice, ie: pip), but they should give us something to build on top of, a standardized method for declaration for version and dependencies (ie: dist_utils). It can be simple, I rather like the Go vendor file spec, and combined with a canonical acceptance of semver it would give us everything we need to make our own tools.

[Matt Farina]

The problem with having the version in the import path is three fold:

• It's impractical to update on large projects
• It's annoying to get right every place you import a package
• There are times you need to pin or lock to a specific version rather than keep to a SemVer range

These problems have been solved by every other modern programming language community in just about the same way. I recently wrote about how they do it and why we should care.

At first glance the vendor-spec appears to work. But, in the past year of working on Glide I've encountered a number of ways to get packages it doesn't account for. I crafted the pkg spec which is an alternative in the same vein as all the other modern package managers that handles the straight forward cases and the more complicated ones we've run into. It can be just as simple or move up in complexity as required.

Package management has been solved by every other modern programming language/platform and they do it in similar ways. There are clear patterns.

[Matt Farina]

Also, I've come up with a hole in the idea of only having a vendor/ directory on main projects.

It's a practical issue. If you're working on a library and require external dependencies where do you put them? What if they are a different version of what's in your GOPATH? Or, there is a fork you need to use in place of the canonical package? How do you automate handling that?

What we really need is the concept of a project in addition to a package. The project can have a vendor/ space and there is only one in any given project. That means if you depend on another library/project only your top level one is included when looking for packages.

On Thursday, November 5, 2015 at 5:18:32 AM UTC-7, sna...@gmail.com wrote:

[Travis Johnson]

Your issue was addressed earlier, but I didn't quote it: libraries shouldn't vendor dependencies. If you're working on the library as part of a project, that projects vendor/ directory will suffice, if you're just working on the library itself, then your testing setup can be utilized, or the vendor/ directory could also be allowed for running tests (though this ambiguity between tests and use would concern me), or this could bring up a point that we should work in virtualenv-style environments instead of single global one.

But I feel those are problems we can rely on external tooling to solve, it'd be nice if we all used the same ones, but it doesn't actually harm much if different developers use different ones, so experimentation and a variety of tools is ok.

Whereas if everyone starts using differing methods of version and dependency declaration, it could make things quite difficult: my package manager uses the vendor-spec file, but I want to use a library that expects your pkg file format. So using different tools is ok, so long as the method by which they communicate is common, this is why I feel the decision/proposal on how this is done needs to come from "on high" as it were, because during the time spent deciding communally what is best will be slow and frustrating as we have many packages using (or worse, not using) many different methods of versioning/dependency declaration, and several tools develop that are incompatible.

I agree with your synopsis of my import suggestion, it was an off-the-top suggestion to get things going. I pointed to the vendor-spec as a launching point, agree with a lot of the decisions in your package spec, both could be utilized as good examples.

--

-Travis

617-549-3856

[Matt Farina]

@Travis That solution was my original thought and that's how Glide operates in practice. IIRC, the other tools do as well. But, that's convention laid on top of a feature. Many developers don't read the docs and don't follow conventions. Let me illustrate with two examples,

1. I've run into cases where nested vendor/ directories exist and people blame the tools rather than understand the process
2. Where people don't understand why, even when given examples, you should only have one vendor directory. So, they don't follow it and share that.

Have you read the book "JavaScript The Good Parts"? If you haven't, it describes using a subset of JavaScript and avoiding others. It appears this is building that same kind of thing in Go. Shouldn't we solve this before the experiment is turned on for everyone? So it's not just a convention.

Working on Glide we now support 3 spec formats. I see more coming in the not too distant future. This is complicated. I believe a spec needs to be use case driven rather than following the opinions of a handful of developers. We have package management use cases and we need a spec that contains enough details for tools to implement them. I started that at https://github.com/mattfarina/pkg/tree/master/use_cases and I think I need to revisit it with what I've learned.

Before we have any kind of spec we need to get the SemVer proposal in. It's going to need to be a multi-step process to get there.

[sna...@gmail.com]

On Thursday, November 5, 2015 at 5:34:55 AM UTC-8, Matt Farina wrote:

Shouldn't we solve this before the experiment is turned on for everyone? So it's not just a convention.

 

Yes yes yes. This is one of the points I'm trying to make; restricting the vendor/ directory to only "main" packages was my attempt at a fix, and I meant by changing out the vendor experiment works, not just by convention.

Before we have any kind of spec we need to get the SemVer proposal in. It's going to need to be a multi-step process to get there.

 

Yep. But I believe the plan was for the vendor experiment to be on by default in 1.6, and I think this thread has done a decent job at highlighting that this should not be the case.

I believe a spec needs to be use case driven rather than following the opinions of a handful of developers.

 

Agreed. The proposal for a spec should come from someone who's dealt with these sorts of problems, and should draw heavily from existing solutions for other languages. But until some central body with enough attention and pull comes out and says "this is what you should use" then we'll have to keep dealing with a slew of competing standards. Once we have an endorsed standard we can iterate on it for improvement, and it could be designed with extensions in mind so use cases not 100% covered could be supported by tools until if/when it gets moved into the spec.

[Russ Cox]

This is a good discussion to have, but please keep in mind that it's all about how to use the vendor/ behavior. This is going to be decided by convention and 3rd-party vendor-management tools, not by restrictions in the go command itself. The way you find it natural to use vendor/ and the way another person finds it natural to use vendor/ may be different, both can be valid, and we don't want to force one or the other.

Thanks.

Russ

--

[sna...@gmail.com]

I feel like this discussion has pointed out a problem with using the GO15VENDOREXPERIMENT. While I agree that 3rd-party vendor-management tools could solve this, if the experiment becomes enabled by default (as is planned for 1.6) then some interactions with packages that currently work (because the vendor is off by default, so everything uses my global versions instead of the vendored one), won't anymore.

Packages I could (and presently do) previously use with only the core tools will become only usable if I use a 3rd-party tool.

I was also trying to make the point that the lack of agreement in how to declare dependency versions impedes the creation of such a tool. As Matt said, he's had to implement reading 3 different types of requirements files.

Is this something we're ok with? As previously stated, people don't read design docs, or follow directions, so simply deciding how to use vendor by convention won't solve the problem. Are we ok with giving the ecosystem and on-by-default mechanism that will force people downstream to use a 3rd-party tool?

[Kyle Wolfe]

On Wednesday, November 11, 2015 at 3:25:31 PM UTC-5, sna...@gmail.com wrote:

I feel like this discussion has pointed out a problem with using the GO15VENDOREXPERIMENT. While I agree that 3rd-party vendor-management tools could solve this, if the experiment becomes enabled by default (as is planned for 1.6) then some interactions with packages that currently work (because the vendor is off by default, so everything uses my global versions instead of the vendored one), won't anymore.

Packages I could (and presently do) previously use with only the core tools will become only usable if I use a 3rd-party tool.

I was also trying to make the point that the lack of agreement in how to declare dependency versions impedes the creation of such a tool. As Matt said, he's had to implement reading 3 different types of requirements files.

Is this something we're ok with? As previously stated, people don't read design docs, or follow directions, so simply deciding how to use vendor by convention won't solve the problem. Are we ok with giving the ecosystem and on-by-default mechanism that will force people downstream to use a 3rd-party tool?

I would say adding in a warning during build -v to show that a single import path is being resolved to two different real paths in the same project may be a helpful start to this issue. What if a new rule were added in to use the lowest level vendor path for the project? In the case of your test project, vendor/a.go would be used in place of vendor/b/vendor/a.go. This is definitely not something I'd suggest doing without the user knowing about it during build, though.

 

[Dave Cheney]

If the argument is that people don't read design docs, or follow
directions, then it seems unlikely they will heed a warning.

[Kyle Wolfe]

On Wednesday, November 11, 2015 at 4:21:43 PM UTC-5, Dave Cheney wrote:

If the argument is that people don't read design docs, or follow
directions, then it seems unlikely they will heed a warning.

 So keep things hidden behind the scenes, then? While I agree, people should do their homework, there's no reason that things like this should not be presented to them.

[sna...@gmail.com]

On Wednesday, November 11, 2015 at 1:20:35 PM UTC-8, Kyle Wolfe wrote:

I would say adding in a warning during build -v to show that a single import path is being resolved to two different real paths in the same project may be a helpful start to this issue. What if a new rule were added in to use the lowest level vendor path for the project? In the case of your test project, vendor/a.go would be used in place of vendor/b/vendor/a.go. This is definitely not something I'd suggest doing without the user knowing about it during build, though.

This seems like a good idea at first glance, but if we allow different packages in the same executable to use different versions of the same dependencies (as the vendor experiment does), then there *will* be projects that depend on this, which means that legitimate projects will have a useless warning displayed on every build. I'm against this sort of noise (Like when I build C projects and watch the compiler warnings just fly past, no one pays attention to those, and thus they don't help).

So keep things hidden behind the scenes, then? While I agree, people should do their homework, there's no reason that things like this should not be presented to them.

I agree. But if the behavior warrants a warning, perhaps the behavior should be changed/removed?

[Daniel Theophanes]

Kyle, snafex,

Now that go1.5 has been out for a while, what projects have you used it on and what tools did you use? Was your experience confusing or troubling and if so, what aspects were?

Thanks, -Daniel

 

--
You received this message because you are subscribed to a topic in the Google Groups "golang-dev" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/golang-dev/WebP4dLV1b0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to golang-dev+...@googlegroups.com.

[Travis Johnson]

I use vendor/ for my "main" packages, but never with libraries, I do this by modifying my GOPATH to include the vendor/ directory of whatever project I'm presently working on.

If I try to enable the vendor experiment instead of modifying my GOPATH, my projects break, because I use other people's libraries who also vendor with the vendor/ directory, and diamond dependencies immediately break the build.

--

-Travis

617-549-3856

[Daniel Theophanes]

Hi Travis,

May I recommend using a tool to resolve that? If you used a tool like github.com/kardianos/govendor, it would take care of those diamond dependencies. I think other tools would do that as well.

-Daniel

[Travis Johnson]

Lovely of you to advertise your tool, and I'm sure it works great, but it doesn't solve the problem. So I'll restate something I said earlier:

Are we ok with *requiring* users to use a 3rd party tool for things made only using the standard tools?

As it stands, if I am only using the standard toolchains (single global GOPATH, no manipulation etc), and things work, and I "go get" a library that uses the vendor/ directory (but no custom build process), I will be *required* to use a 3rd party tool (such as govendor) as of Go 1.6.

--

-Travis

617-549-3856

[Daniel Theophanes]

Unless go get fetches into the vendor folder, which I don't think it does, nothing really changes unless you copy it in, probably with a tool.

[Travis Johnson]

It doesn't, but it does download the vendor/ folder as well, and since the experiment is scheduled to be on by default in 1.6, "go build" will use that libraries vendor/ directory for imports.

--

-Travis

617-549-3856

[Matt Farina]

Daniel,

The issue is with the default behavior. Defaults are important because a lot of people are only going to do that. In this case that's what we're getting at. The default behavior for a feature that's going to be on by default in 1.6 provides ample opportunity to create build breaking situations.

This isn't good for developer experience. It's going to cause a lot of headaches.

The fact that we talk about 3rd party tools to correct a common problem brought on by a default behavior isn't good. I'm not naming tools because it's not about using a tool to correct the behavior. We need to shine a light on the problems from the default behavior.

Matt

[Daniel Theophanes]

Hi Matt and Travis,

I think what you are saying is that this allows packages to vendor their own dependencies. Thus if you go get a package into GOPATH that has a vendor folder, you could run into duplicates or conflicts.

For example, package "dbmigrate/vendor/pgx" is imported by a main package that already imports "pgx". This would cause a runtime panic when they both try to register.

Is this the concern you are raising? -Daniel

[Travis Johnson]

I agree with your first paragraph, but the second one doesn't quite make sense. If I import "dbmigrate" which imports "dbmigrate/vendor/pgx" then my "pgx" is incompatible with the types it uses/returns, however this is a compile time error, not a runtime one.

--

-Travis

617-549-3856

[Daniel Theophanes]

We agree that there would be an error. There may be compile time errors, if dbmigrate exposes the vendor/pgx package, but the panic would come from registering twice in "database/sql". We both agree that errors could occur in situations like this.

I'm fairly certain this could occur if package copied dependencies locally with import path rewriting, but that is neither here nor there.

I think this simply points to packages need to be careful what they vendor. Here is an example I am familiar with (many others exist I'm sure): package rdb vendors two packages:

https://bitbucket.org/kardianos/rdb/src/9cb23bea5ac5ee09d98e06b1e30657ec6e93dae9/vendor/?at=default "x/net/context" and two packages from vitess. In this case the vitess packages are obviously safe; due to their construction even if there "were another copy" (unlikely) no harm would come of it and vitess has changed rapidly in the past that these packages need to be pulled out. "x/net/context" on the other hand make a good case to *not* be copied into the local "vendor" folder; it is both common, and directly used by other packages.

I think we can agree on the above. Let me know if I'm catching something wrong.

My take away is we need to focus on *what* message we want to send package and command authors. Is that also your take away? Or do you think that something with the vendor folder should be changed? What would you message package and command authors or what would you change with the vendor folder?

[Russ Cox]

On Wed, Nov 11, 2015 at 3:25 PM, <sna...@gmail.com> wrote:

I feel like this discussion has pointed out a problem with using the GO15VENDOREXPERIMENT. While I agree that 3rd-party vendor-management tools could solve this, if the experiment becomes enabled by default (as is planned for 1.6) then some interactions with packages that currently work (because the vendor is off by default, so everything uses my global versions instead of the vendored one), won't anymore.

It is true that people who happen to have named a directory "vendor" and use it expecting the old behavior will find their code broken. We explicitly took this into account: there were very few instances of this in public code that we could find, and the point of the Go 1.5 transition period was to give people time to name that directory something else. If a source tree has no directory named "vendor" there is no difference between default Go 1.5 and default Go 1.6 behavior.

That said, I don't quite understand what you mean here. If I didn't reply to your actual point, can you give a detailed example? In particular I don't know what you mean by "everything" and by "my global versions" and "the vendored one".

Thanks.

Russ

[Konstantin Shaposhnikov]

I've tried to recreate the problem how I understand it here:
https://github.com/kostya-sh/sandbox/tree/master/vendor-problem

GO15VENDOREXPERIMENT=0 go get github.com/kostya-sh/sandbox/vendor-problem

works, but

GO15VENDOREXPERIMENT=1 go get github.com/kostya-sh/sandbox/vendor-problem

fails with the following error:

./main.go:14: cannot use p.Status() (type
*"github.com/derekparker/delve/vendor/golang.org/x/sys/unix".WaitStatus)
as type *"golang.org/x/sys/unix".WaitStatus in assignment

Basically with vendor experiment enabled by default a project that
doesn't vendor its dependencies
(github.com/kostya-sh/sandbox/vendor-problem in the example above)
cannot be built if it depends on both
1. a package that vendors its dependencies (github.com/derekparker/delve/proc)
2 a package that is vendored by 1 (golang.org/x/sys/unix)

I think the fix for this should make golang.org/x/sys/unix import path
be resolved to github.com/derekparker/delve/vendor/golang.org/x/sys/unix
in such case.

[Russ Cox]

On Thu, Nov 12, 2015 at 12:10 PM, Konstantin Shaposhnikov <k.shapo...@gmail.com> wrote:

I've tried to recreate the problem how I understand it here:
https://github.com/kostya-sh/sandbox/tree/master/vendor-problem

GO15VENDOREXPERIMENT=0 go get github.com/kostya-sh/sandbox/vendor-problem

works, but

GO15VENDOREXPERIMENT=1 go get github.com/kostya-sh/sandbox/vendor-problem

fails with the following error:

./main.go:14: cannot use p.Status() (type
*"github.com/derekparker/delve/vendor/golang.org/x/sys/unix".WaitStatus)
as type *"golang.org/x/sys/unix".WaitStatus in assignment

Basically with vendor experiment enabled by default a project that
doesn't vendor its dependencies
(github.com/kostya-sh/sandbox/vendor-problem in the example above)
cannot be built if it depends on both
1. a package that vendors its dependencies (github.com/derekparker/delve/proc)
2 a package that is vendored by 1 (golang.org/x/sys/unix)

It's true, you can't do this. It looks like Delve does not intend for people to import pieces of it, and therefore you can't.

 

I think the fix for this should make golang.org/x/sys/unix import path
be resolved to github.com/derekparker/delve/vendor/golang.org/x/sys/unix
in such case.

I understand you to be saying you want github.com/kostya-sh/sandbox/vendor-problem's import "golang.org/x/sys/unix" to resolve to "github.com/derekparker/delve/vendor/golang.org/x/sys/unix" because it also happens to import "github.com/derekparker/delve/proc".

That does seem the obvious fix, but it breaks down once there are multiple choices. What if vendor-problem also imported "rsc.io/foo/bar" and there was an "rsc.io/foo/vendor/golang.org/x/sys/unix"? Then what should import "golang.org/x/sys/unix" resolve to? What if you don't want derekparker's copy of sys/unix? How do you get the real one? The options seem to be "stop importing delve/proc" (seems unrelated) or "make Derek delete his copy of sys/unix" (seems excessive).

The resolution of imports depends on the source location and not what the source imports precisely to avoid these problems. 

It really looks like Derek does not intend you or anyone else to import and use delve/proc directly, so he's doing everything right. If he did want you to import delve/proc directly, then he would need to use the real sys/unix types or else publish his copy of sys/unix (outside the vendor directory) for you to import.

Also, vendored code is a private copy of something. That is, "vendor" is explicitly "internal" as well. We very much do not want one project importing another project's vendored copy of a third project.

Russ

[Kyle Wolfe]

On Thursday, November 12, 2015 at 12:35:22 PM UTC-5, rsc wrote:

Also, vendored code is a private copy of something. That is, "vendor" is explicitly "internal" as well. We very much do not want one project importing another project's vendored copy of a third project.

Russ, this is currently not the behaviour, as shown by the example project. My suggestion earlier was the the lowest level path should be resolved. In the meantime, I've opened a propsoal to push back the milestones for vendoring by one release until more of a consensus is reached (https://github.com/golang/go/issues/13218).

[Konstantin Shaposhnikov]

I used delve/proc only as example to demonstrate the described issue.
I agree that it is probably not intended to be used as a package.
However I can imagine that putting applications (that usually will
have vendored dependencies) and library packages in the same
repository is quite common.

I don't think that this particular issue is a big problem though. As
you demonstrated there are potentially many ways to resolve a package
in the described scenario so go get cannot do it automatically. The
only valid solution in this case is to make a developer vendor the
required dependencies manually.

[Russ Cox]

I know this thread is already very long, but I am going to try to reply again now that I've had a chance to look at the exact details of the GitHub repo.

On Tue, Nov 3, 2015 at 12:28 PM, Matt Farina <matt....@gmail.com> wrote:

If you use the vendor experiment to put external packages into the vendor/ folder and then others pull your package into theirs it's easy to run into a problem. For example see https://github.com/mattfarina/golang-broken-vendor.

Yes, that source code is broken. The directory tree here looks like:

    - $GOPATH/src/github.com/mattfarina/golang-broken-vendor

      - foo.go

      - vendor/

        - a/

        - b/

            - b.go

            - vendor/a/

And gbv/vendor/b.go says (paraphrasing)

    package b

    import "a"

    func Do(a.A)

And gbv/foo.go says

    package foo

    import "a"

    import "b"

   

    func _() {
        var x a.A

        b.Do(x)

    }

The problem here is that import "a" means something different in gbv/vendor/b.go as it does in gbv/foo.go. In gbv/vendor/b.go it means "gbv/vendor/b/vendor/a". In gbv/foo.go it means "gbv/vendor/a". 

The fact that the imports mean different things is working as designed. That lets b supply its own copy of a even if it's being incorporated into a larger program that needs a different copy. That is, keeping those two a's separate makes it possible, when you need two different versions of the code in the same program, to do that. This possibility is called out in golang.org/s/go15vendor. As the doc says, the vendor support "does not attempt to solve the problem of vendoring resulting in multiple copies of a package being linked into a single binary. Sometimes having multiple copies of a library is not a problem; sometimes it is. At least for now, it doesn’t seem that the go command should be in charge of policing or solving that problem."

The code as in the above example is just incorrect. Creating gbv/vendor/b/vendor/a effectively says "this is the copy of 'a' that 'b' and nearby code should use", and exposing a.A in b's API effectively says "you have to be able to import b's copy of a to use b.Do". The implication is that b.Do can only be used by code within the tree rooted at gbv/vendor/b. (If there were subdirectories, they could use it, and b itself can use it.) Since gbv itself (as implemented in gbv/foo.go) is not covered by those statements, it cannot use b.Do. An import "a" would not compile, except for the creation of gbv/vendor/a, which says "this is the copy of 'a' that gbv and nearby code should use". That makes the import "a" compile but it is a different "a".

What the program and the placement of source files says means gbv/foo.go cannot use b.Do. If that's not what is meant, then the solution is not to redesign vendoring; it is to place the source files differently. 

As others have noted, tools might help you avoid creating this state or resolve it. You responded that you don't want to have to use tools. You don't. It's perfectly fine to build these trees by hand, provided you realize what the creation of these directories says and make sure not to say something that doesn't work (or take responsibility for fixing it when you do).

For what it's worth, one can construct very similar situations without vendoring. For example, consider this tree:

    - $GOPATH/src/github.com/mattfarina/golang-broken-vendor

      - foo.go

      - vendor/

        - a/

        - b/

            - b.go

            - internal/a/

I've changed the inner vendor to internal, and assume that b.go now imports "b/internal/a" which resolves to "gbv/vendor/b/internal/a". Now foo.go still doesn't work, for exactly the same reason: b.go's a.A is different from foo.go's a.A. But maybe it makes more sense to see it in this case.

Similarly, we might get rid of all the a's entirely and just write, in b.go:

    package b

    type hiddenA ...

    func Do(hiddenA)

Then foo.go cannot call Do either, at least not using an explicit reference to hiddenA. Like in the other two examples, the program has been constructed in a way that makes the necessary type impossible to name. The resolution is not that the rules for exported vs unexported are wrong; it's that you must write the program a different way.

I do believe we need to make this clearer to people. The point of the vendor/ tree is to support having multiple copies when that makes sense. But if you have multiple copies you have multiple copies. If you need to have just one copy, that's up to you. The vendor support does not just pick one and force it on the whole tree. As I wrote earlier, "The way you find it natural to use vendor/ and the way another person finds it natural to use vendor/ may be different, both can be valid, and we don't want to force one or the other."

Russ

[Kyle Wolfe]

Thank you for your post. I gather that as of now, the build tool should allow for recursive vendoring and NOT provide a mechanism for fixing potential issues in references between different versions. You state that this should be made clearer. Where can this be documented to draw attention to it? Perhaps a dedicated page within godoc?

So, I'd like to talk about Go 1.7. In my recently closed issue, I brought up a scenerio where one may vendor a third party package, and that third party package also has vendoring. Russ, taking your structure again:

 - $GOPATH/src/github.com/mattfarina/golang-broken-vendor

     - foo.go

     - vendor/

       - a/

       - b/

           - b.go

           - vendor/a/

Assuming foo uses both a and b/vendor/a and breaks. As of Go 1.7, because I do not have an option to disable Go vendoring, I must either 1) convince the owner of a to remove vendoring 2) host my own fork of a 3) save the repo statically, without a symlink.

Do you feel that providing a mechanism to disable vendoring within the build tool, or better yet, a mechanism to disable recursive vendoring (would force package b to resolve a as github.com/mattfarina/golang-broken-vendor/vendor/a) would go against the idea of the build tool as it stands now?

[Dave Cheney]

@rsc, replying to your request for comment from
https://github.com/golang/go/issues/13218#issuecomment-156194968

I am concerned that the vendor experiment has side effects that are
not well understood, specifically:

- the interaction of vendoring at multiple levels of a project.
- the above but combined with transitive vendoring.
- the unintended confusion where 'native' (unvendored) packages in
GOPATH superseded, or are superseded by vendored ones.

These problems will hurt Go users with the most complex package
dependency problems, the target market the vendor experiment was
supposed to help.

I am *very* concerned about the issues of duplicate packages appearing
in the final binary. I worry that this will be the typed nil interface
WTF of the next generation of Go programmers.

There are also the issues of unexpected interactions with go test
./... (and go list). I know these issues are marked resolved, but I am
one of the number who thinks the resolution is incorrect.

I agree with Kyle that serious consideration should be given to
delaying making the vendor experiment the default in Go 1.6 -- I just
think the complex corner cases are not well enough understood, and
_disabling_ vendor support in a future Go release sounds vanishingly
unlikely. We're going to be stuck with this behaviour for a long time,
I think it's prudent to wait.

At a minimum there should be clear guidelines published for Go
developers. I suggest

- vendoring *must* only occur at the top of the repository, thus
providing a single location for packages in the repository to override
ones in the greater GOPATH.
- the go tool *must* make it a compile error if it detects vendoring
at any other level of the repository*.

Thank you for your time.

Dave

* I have no support for a warning or optional flag.

[Kyle Wolfe]

On Thursday, November 12, 2015 at 6:20:18 PM UTC-5, Dave Cheney wrote:

I agree with Kyle that serious consideration should be given to
delaying making the vendor experiment the default in Go 1.6 -- I just
think the complex corner cases are not well enough understood, and
_disabling_ vendor support in a future Go release sounds vanishingly
unlikely. We're going to be stuck with this behaviour for a long time,
I think it's prudent to wait.

I think I was hasty in opening that issue, and also suggesting that the 1.6 milestone be delayed. I think there are sure to be issues. To Russ' point earlier, the idea is to change behavior to suite the new feature. My main concern at this point is that the complex corner cases won't have a chance to be addressed come 1.7, when it can no longer be turned off.

- vendoring *must* only occur at the top of the repository, thus
providing a single location for packages in the repository to override
ones in the greater GOPATH.

Are you completely opposed to the idea of allowing recursive vendoring as long as there is an option to disable it, allowing an attempt at a build where packages are resolved to the parent repositories vendor directory as I described in my last? I think build options for --disable-recursive-vendor and --disable-vendor gives many options for a fix in these corner cases.

I also think it's imperative that the build tool be verbose when it comes to vendoring. It should alert me when a vendored path is being used over my gopath, or when a recursive vendor has been detected (and used or fallen back to a parent).

 

[Dave Cheney]

> Are you completely opposed to the idea of allowing recursive vendoring as
> long as there is an option to disable it, allowing an attempt at a build
> where packages are resolved to the parent repositories vendor directory as I
> described in my last? I think build options for --disable-recursive-vendor
> and --disable-vendor gives many options for a fix in these corner cases.

I'm completely opposed to any sort of flags like that.

[Travis Johnson]

On Thu, Nov 12, 2015 at 3:20 PM Dave Cheney <da...@cheney.net> wrote:

- vendoring *must* only occur at the top of the repository, thus
providing a single location for packages in the repository to override
ones in the greater GOPATH.

I agree with this. Actually, this was something I suggested earlier in the thread, but rather than by convention/documentation by changing the vendor experiment to only work this way (instead of the current behavior (as your second item would do).

[Matt Farina]

It's worth noting that the dependency tool authors on this thread have each expressed agreement in one vendor/ directory per repo.

Personally, I learned this the hard way. I ran across issues in practice.

I've also studied how people use dependencies in other languages. This tells us how people expect things to work. Nested dependencies are going to be annoying in practice because they don't work as the long tail of developers will expect them to.

For what it's worth (very little), I happen to agree with Daves proposal.

[Kyle Wolfe]

On Thursday, November 12, 2015 at 7:15:57 PM UTC-5, Matt Farina wrote:

It's worth noting that the dependency tool authors on this thread have each expressed agreement in one vendor/ directory per repo.

Right, but this is not enforced by the tool. It is a recommendation for repo design.

[Kyle Wolfe]

I'd be okay with this, and if something like this where in place by Go 1.7, my concerns would be addressed.

[Russ Cox]

On Thu, Nov 12, 2015 at 6:20 PM, Dave Cheney <da...@cheney.net> wrote:

I agree with Kyle that serious consideration should be given to
delaying making the vendor experiment the default in Go 1.6 -- I just
think the complex corner cases are not well enough understood, and
_disabling_ vendor support in a future Go release sounds vanishingly
unlikely.

The problem with waiting is that no one can use it for real until it's the default. I have for example refrained from using it in my own rsc.io packages (instead using internal and long import paths) because otherwise people cannot 'go get rsc.io/whatever' without setting an environment variable first. The current bifurcation hurts the community. We need to merge back together.

At a minimum there should be clear guidelines published for Go
developers. I suggest

- vendoring *must* only occur at the top of the repository, thus
providing a single location for packages in the repository to override
ones in the greater GOPATH. 

- the go tool *must* make it a compile error if it detects vendoring
at any other level of the repository*.

These restriction still don't fix the problem of possibly duplicated code. If someone vendors net/http and there's only one vendor/net/http, there are still probably two net/http's in the binary. Unless you disallow vendoring standard packages. But the compiler vendors math/big precisely so that it can supply its own independent of the Go version, so we know there are good use cases for this. Even in the compiler, we'd almost certainly use $GOROOT/src/cmd/vendor/ instead of $GOROOT/src/vendor/, precisely so to control the visibility of the vendored packages. It's important that only the cmd/ tree use them. I can see that consideration being important in other large source trees as well. Maybe some package depends on a buggy behavior of Go 1.2's text/template, and the expedient thing is to put a vendor directory under it to hold that old copy. Certainly that can't be lifted out, and there's no harm in letting that one package use a different text/template. If we impose the restrictions you suggest, then the only solution is to copy the code into the tree and rename all the imports, which is exactly the previous approach that was rejected.

Given all that, I think these are both mistakes. As I've said multiple times, people will find different equally valid ways to use vendor directories. Forcing them into the model you think best is added complexity when that's not the right choice. I'm happy for people to make tentative suggestions about best practices, but I don't want the go command limiting the use cases.

The fact is, we need to turn this on by default in order to learn more, because people can't use it at scale until it's on. The Go 1.5 period served two purposes: (1) shaking out bugs and (2) giving people time to rename "vendor" directories that do intend to have the new meaning. We probably need to advertise the latter a bit more, but it's a trivial update. I don't see any need to wait another 6 months.

Russ

[Edward Muller]

Could the compiler at least error when it determines that a vendored package's vendored types escapes the vendored package?

That doesn't fix a lot, but it sends a clear signal as to the purpose of vendor/.

As one of those 3rd party tool maintainers I'm equally concerned with the things Dave and Matt are concerned about. So +1 wrt their concerns. I've seen all of these issues and more during the 1.5 period.

--

You received this message because you are subscribed to the Google Groups "golang-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Edward Muller
@freeformz

[Edward Muller]

PS: s/compiler/go tool chain/

--

Edward Muller
@freeformz

[Russ Cox]

On Thu, Nov 12, 2015 at 4:47 PM, Kyle Wolfe <kyle.a...@gmail.com> wrote:

Thank you for your post. I gather that as of now, the build tool should allow for recursive vendoring and NOT provide a mechanism for fixing potential issues in references between different versions. You state that this should be made clearer. Where can this be documented to draw attention to it?

It is documented explicitly in golang.org/s/go15vendor, which is linked from 'go doc cmd/go'. There is documentation.

So, I'd like to talk about Go 1.7. In my recently closed issue, I brought up a scenerio where one may vendor a third party package, and that third party package also has vendoring. Russ, taking your structure again:

 - $GOPATH/src/github.com/mattfarina/golang-broken-vendor

     - foo.go

     - vendor/

       - a/

       - b/

           - b.go

           - vendor/a/

Assuming foo uses both a and b/vendor/a and breaks.

Just to back up a step, "recursive" vendoring is not actually essential to this example, and I think it is distracting from the main point. To get to the state above, it must be that the author of b/ published the directories b and b/vendor/a and then the author of foo copied them into another vendor tree. But suppose the author of foo just wrote foo and did 'go get b', without introducing a second layer of vendoring:

    $GOPATH/src

        foo/

            foo.go

        b/ 

            b.go

            vendor/a/

Again suppose that package b imports b/vendor a as simply import "a" and exposes a.A in its exported API. Then the author of b has created an API that cannot be used outside the b/... tree. Just as if the directory were named b/internal/a instead of b/vendor/a or if b's API were entirely unexported, foo cannot use b's API. The author of b has arranged things so that only b/... can use it. That might have been intentional or might not. Either way, it's working as written by b's author.

If foo/ vendors the b/ tree, that doesn't change anything about the fact that b/ cannot be used from outside the b/... tree. The b/... tree moved, but the effect is still the same.

If foo/ adds its own foo/vendor/a, that's a different a, so b/ still cannot be used from outside the b/... tree. Recursive vendoring is still not the problem. The problem is b's author didn't write an API foo can use.

 

As of Go 1.7, because I do not have an option to disable Go vendoring, I must either 1) convince the owner of a to remove vendoring 2) host my own fork of a 3) save the repo statically, without a symlink.

Yes, it is true that for foo to use b's API, you must either (1) convince b's author to publish code foo can use or (2) make your own edited copy that foo can use.

Russ 

[Russ Cox]

On Thu, Nov 12, 2015 at 8:44 PM, Edward Muller <edwa...@interlix.com> wrote:

Could the compiler at least error when it determines that a vendored package's vendored types escapes the vendored package?

That doesn't fix a lot, but it sends a clear signal as to the purpose of vendor/.

As one of those 3rd party tool maintainers I'm equally concerned with the things Dave and Matt are concerned about. So +1 wrt their concerns. I've seen all of these issues and more during the 1.5 period.

Thank you for writing a 3rd party tool. The goal of the vendor/ support is to add to the toolchain/compiler the minimum functionality required to support exploration by these 3rd party tools, by making it possible for import "x" to resolve to a package other than "x" (namely, to "something/else/vendor/x"). I am sure we will learn a lot about what works and what does not, but we are still exploring. I believe it is a mistake to try to restrict that exploration based on the limited current ideas we have.

I do understand that many people are confused. Let's write tutorials and improve diagnostics. For example, it would be perfectly appropriate for a 3rd party tool to refuse to operate on a tree with nested vendoring and shadowed packages. Users for whom that is not a reasonable restriction can look for another 3rd party tool. The same is not true of the Go distribution itself. It's important we don't force these restrictions on everyone.

Russ

[Henrik Johansson]

Having lurked on this discussion on and off I feel this is bad news. The simplicity of pretty much everything else in the Go ecosystem will make this the really odd duck. Having different tools work in different principal ways seems really bad. I have no solution but I just felt I should give a future users view.

In hindsight I wish we would have done it the same way as Rust have done with Cargo. It has notions of both projects and versions and afaik noone is inclined to build other tools for project and dependency management. 2¢

--

[Travis Johnson]

Also, vendored code is a private copy of something. That is, "vendor" is explicitly "internal" as well. We very much do not want one project importing another project's vendored copy of a third project.

But isn't this what internal/ is for? Would it make sense to have internal/ behave like vendor/ does for imports, and then eliminate vendor/ (this would also give us the relative imports that's been discussed in the past)?

Though this may be seen as a breaking change, so maybe instead just expressly make vendor/ restricted like internal/ is (potentially with the additional compiler refusal to export types (I'd make an exception for interfaces), but there may be legitimate uses for this, and it sounds like a rabbit hole of complexity, so I'm hesitant to suggest it)? This would fall in line with this claim that vendor/ is "explicitly" internal (which while this may have been the intention, it isn't the pattern I see in projects).

I do however believe that turning this on by default will break a lot of builds, but then people will very quickly realize incorrect usage and fix (not without backlash though), though previously stable versions of these libraries will be incompatible with 1.6 (not sure I'd call this a breaking change, as it's functionality depending on an off-by-default "experiment"), that is if you consider the vendor/ directory as "part of the project" (I do, as I think do many others).

[Daniel Theophanes]

Travis, The only way a package could access a type in the vendor folder *today* is through indirect access, just like the "internal" folder. Russ wasn't stating a policy, he was stating the technical fact as it stands today.

--
You received this message because you are subscribed to a topic in the Google Groups "golang-dev" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/golang-dev/WebP4dLV1b0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to golang-dev+...@googlegroups.com.

[Russ Cox]

On Fri, Nov 13, 2015 at 3:43 AM, Henrik Johansson <dahan...@gmail.com> wrote:

Having lurked on this discussion on and off I feel this is bad news. The simplicity of pretty much everything else in the Go ecosystem will make this the really odd duck. Having different tools work in different principal ways seems really bad. I have no solution but I just felt I should give a future users view.

The vendor definition is simple. It's nearly trivial. golang.org/s/go15vendor states the complete definition in four short sentences. Everything about vendor is contained in those sentences:

1. If there is a source directory d/vendor, then, when compiling a source file within the subtree rooted at d, import "p" is interpreted as import "d/vendor/p" if that exists.
2. When there are multiple possible resolutions, the most specific (longest) path wins.
3. The short form must always be used: no import path can contain “/vendor/” explicitly.
4. Import comments are ignored in vendored packages.
   

That's everything there is to know about vendor. These rules are fewer and less text than the rules in the Go spec for assignability, or for struct literals, or for selectors, or for index operators, or for equality, or for conversions.

What's complicated is all the preconceptions people are bringing to the discussions, such as that the goal is to recreate an existing system, such as Cargo or NPM or the package manager du jour. No. The goal is to enable a variety of uses, including but not limited to those, by making the smallest, simplest change we can in the core toolchain. Specifically, that change is to make it possible to copy code from one place in the tree to another without having to change the import statements using it or within it, by applying the four rules above.

Russ

[Henrik Johansson]

I know it is simple but perhaps too simple given this thread? They do sound consistent but what with shadowing and linking issues exposed above perhaps they need amending.

I and everyone else of course have preconceptions. That is unavoidable. However what I meant about Cargo was not that it works in a certain way and we should replicate it but that the Rust community seems to have settled on both 1 way and even 1 tool to do it. And they didn't do it by years of darwinian trial and error but by choice. This by the way is something the Java world has been unable to do over the decades.
This is normally what I associate with "the Go way" highly opinionated _solutions_ to things that are generally contested.

Why is it so important all of a sudden to not force a particular way on everyone? We have done so repeatedly before.

[Andrew Gerrand]

On 14 November 2015 at 16:43, Henrik Johansson <dahan...@gmail.com> wrote:

Why is it so important all of a sudden to not force a particular way on everyone?

Because this is an incremental step. If we knew the best definite way to do it, we'd just do that. Instead, we'd like to canonicalize this small part of the solution to make the ecosystem better today. Then over time we can see which other parts of the story we can canonize, if any.

[Henrik Johansson]

Ok I buy that. I do however think we went the wrong way about it. One of the coolest things that happened during the early days of Go was how breaking changes was handled via programmatic rewriting. We could have done the same thing here, collaborative development of one tool with automatic restructuring of people's code as changes were implemented.

[yiyus]

On Saturday, 14 November 2015 01:09:25 UTC+1, Travis Johnson wrote:

Also, vendored code is a private copy of something. That is, "vendor" is explicitly "internal" as well. We very much do not want one project importing another project's vendored copy of a third project.

But isn't this what internal/ is for? Would it make sense to have internal/ behave like vendor/ does for imports, and then eliminate vendor/ (this would also give us the relative imports that's been discussed in the past)?

Indeed, there is some overlapping. Instead of removing the vendor directory, what may make sense once the vendor experiment is on is to use vendor/internal for internal packages. This would not require support by any tool, and it would shorten the import paths (which would not need to be rewritten when the repo is cloned). The downside is that vendor/internal/ is less intuitive than internal/.

I hope the vendor experiment is enabled by default in 1.6, then users will decide how vendor and internal come along together.