--
You received this message because you are subscribed to the Google Groups "golang-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Unfortunately, waiting for 1.6 isn't going to solve this problem. It may only get worse. Let me explain.Quite often you don't know who you're going to share with. The code goes up on GitHub and none or many people may use it. If package authors don't know about this behavior they may store the packages they use in their vendor/ folder. Then, when someone else uses that package and has their own dependencies in their vendor/ folder this problem can arise. In that case, what's someone to do?I found this problem in practice by using packages where everyone expected the GO15VENDOREXPERIMENT to be on. People were not aware of this behavior.Once the vendor experiment is on by default a larger audience will be open to this problem.
@rsc did you look at the example I linked off to? When the same package is in 2 vendor/ locations. It could be more but the linked example only has 2. In case you missed it, there example is at https://github.com/mattfarina/golang-broken-vendorIt has to do with how, in practice, people use the vendor/ directories. This issue has been seen in the wild.
So here's the problem,
The way the vendor rule works at the moment is vendor/x is imported as "x", but linked as " vendor/x"
This introduces all the problems of putting version numbers in imports, aka gopkg.in. specifically
* duplicate calls to init routines to register drivers and loggers
* types that fail type assertion checks.
However, to do the opposite, vendor/x linking as x would conflict with other x's, possibly from GOPATH or other nested, or sibling vendor structures.
This would cause linking failures as symbols wouldn't match, or worse, no linking failures and structures being the wrong size at run time.
Either way is problematic, and I don't see a way to resolve Matt's concerns.
Thanks
Dave
--
@dave If this is what we have I see two possibilities of ways to use it in practice. Something we'll need to teach people.
- No one stores their dependences in the vendor/ in their VCS (unless you really know what you're doing... an exception). Then you use tooling to populate the vendor/ folder when needed (dev, test, and build environments). The tooling uses config files (and nested ones in dependencies) to populate it. This allows you to use it and have different versions of packages in different applications. This is basically the model used by PHP, node.js, and many other languages. Let the tooling manage your vendor/ folder based on config so you can avoid bad situations.
- Tooling walks through all the dependencies and moves vendor dependencies to the top level. This may be opt-in and would need to rely on some config (to handle knowledge about versions so you can deal with the same package multiple times but different versions).
Dave,In ref to #1, what if you're working on more than one application and they use different versions of the same dependency?
How do you handle that with one gopath?
Oh, and I understand how a project specific workspace solves that. I'm attempting to look at this through the go tool.To flatten multiple projects to the top level and handle versions you'd need a record of versions. Ideally, to handle mild differences they'd be versions as semver constraints and the tooling picks the best version to meet those (if one is available).
If you need that record of versions somewhere and to have to restructure a codebase what benefit is there to vendoring?
On Wed, Nov 4, 2015 at 10:58 AM, Matt Farina <matt....@gmail.com> wrote:Dave,In ref to #1, what if you're working on more than one application and they use different versions of the same dependency?This has never been possible, the linker prevents it.
So here's the problem,
The way the vendor rule works at the moment is vendor/x is imported as "x", but linked as " vendor/x"
@dave If this is what we have I see two possibilities of ways to use it in practice. Something we'll need to teach people.
- No one stores their dependences in the vendor/ in their VCS (unless you really know what you're doing... an exception). Then you use tooling to populate the vendor/ folder when needed (dev, test, and build environments). The tooling uses config files (and nested ones in dependencies) to populate it. This allows you to use it and have different versions of packages in different applications. This is basically the model used by PHP, node.js, and many other languages. Let the tooling manage your vendor/ folder based on config so you can avoid bad situations.
- Tooling walks through all the dependencies and moves vendor dependencies to the top level. This may be opt-in and would need to rely on some config (to handle knowledge about versions so you can deal with the same package multiple times but different versions).
Really, the first case it tooling to manage you vendor/ folder to avoid the problems. The second it tooling to fix the problems.
- Matt
To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+unsubscribe@googlegroups.com.
--
@Jimxu I'm not sure version handling and packages is too hard to solve. Every other modern language ecosystem has it solved. That includes Rust, PHP, JavaScript, Java, Ruby, Python, Objective-C, Swift, and others. Not only do they have this problem solved, they don't run into issues like the one I originally outlined here.
On Wednesday, November 4, 2015 at 12:27:05 PM UTC-5, Jimxu wrote:I think what minux@ meant by "application" is that it is a binary containing a main package. For a library (e.g. as you said it could be something common abstracted out as a library from an application), the question of how to resolve multiple versions of the same vendored package seems unavoidable... (which is a hard problem I think and the original spec also chosen to leave it out)
--
Only applications should vendor packages.
I'm wary of putting the version in the GOPATH because, like you said, it's a version range. How would you put ">= 1.2.3, < 2.0.0, != 1.4.5" in the import path? If it's just a single version you have a problem of resolution in a complex project.
Shouldn't we solve this before the experiment is turned on for everyone? So it's not just a convention.
Before we have any kind of spec we need to get the SemVer proposal in. It's going to need to be a multi-step process to get there.
I believe a spec needs to be use case driven rather than following the opinions of a handful of developers.
--
I feel like this discussion has pointed out a problem with using the GO15VENDOREXPERIMENT. While I agree that 3rd-party vendor-management tools could solve this, if the experiment becomes enabled by default (as is planned for 1.6) then some interactions with packages that currently work (because the vendor is off by default, so everything uses my global versions instead of the vendored one), won't anymore.Packages I could (and presently do) previously use with only the core tools will become only usable if I use a 3rd-party tool.I was also trying to make the point that the lack of agreement in how to declare dependency versions impedes the creation of such a tool. As Matt said, he's had to implement reading 3 different types of requirements files.Is this something we're ok with? As previously stated, people don't read design docs, or follow directions, so simply deciding how to use vendor by convention won't solve the problem. Are we ok with giving the ecosystem and on-by-default mechanism that will force people downstream to use a 3rd-party tool?
If the argument is that people don't read design docs, or follow
directions, then it seems unlikely they will heed a warning.
I would say adding in a warning during build -v to show that a single import path is being resolved to two different real paths in the same project may be a helpful start to this issue. What if a new rule were added in to use the lowest level vendor path for the project? In the case of your test project, vendor/a.go would be used in place of vendor/b/vendor/a.go. This is definitely not something I'd suggest doing without the user knowing about it during build, though.
So keep things hidden behind the scenes, then? While I agree, people should do their homework, there's no reason that things like this should not be presented to them.
--
You received this message because you are subscribed to a topic in the Google Groups "golang-dev" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/golang-dev/WebP4dLV1b0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to golang-dev+...@googlegroups.com.
Unless go get fetches into the vendor folder, which I don't think it does, nothing really changes unless you copy it in, probably with a tool.
I feel like this discussion has pointed out a problem with using the GO15VENDOREXPERIMENT. While I agree that 3rd-party vendor-management tools could solve this, if the experiment becomes enabled by default (as is planned for 1.6) then some interactions with packages that currently work (because the vendor is off by default, so everything uses my global versions instead of the vendored one), won't anymore.
I've tried to recreate the problem how I understand it here:
https://github.com/kostya-sh/sandbox/tree/master/vendor-problem
GO15VENDOREXPERIMENT=0 go get github.com/kostya-sh/sandbox/vendor-problem
works, but
GO15VENDOREXPERIMENT=1 go get github.com/kostya-sh/sandbox/vendor-problem
fails with the following error:
./main.go:14: cannot use p.Status() (type
*"github.com/derekparker/delve/vendor/golang.org/x/sys/unix".WaitStatus)
as type *"golang.org/x/sys/unix".WaitStatus in assignment
Basically with vendor experiment enabled by default a project that
doesn't vendor its dependencies
(github.com/kostya-sh/sandbox/vendor-problem in the example above)
cannot be built if it depends on both
1. a package that vendors its dependencies (github.com/derekparker/delve/proc)
2 a package that is vendored by 1 (golang.org/x/sys/unix)
I think the fix for this should make golang.org/x/sys/unix import path
be resolved to github.com/derekparker/delve/vendor/golang.org/x/sys/unix
in such case.
Also, vendored code is a private copy of something. That is, "vendor" is explicitly "internal" as well. We very much do not want one project importing another project's vendored copy of a third project.
If you use the vendor experiment to put external packages into the vendor/ folder and then others pull your package into theirs it's easy to run into a problem. For example see https://github.com/mattfarina/golang-broken-vendor.
- $GOPATH/src/github.com/mattfarina/golang-broken-vendor - foo.go - vendor/ - a/ - b/ - b.go - vendor/a/
github.com/mattfarina/golang-broken-vendor/vendor/a)
would go against the idea of the build tool as it stands now?I agree with Kyle that serious consideration should be given to
delaying making the vendor experiment the default in Go 1.6 -- I just
think the complex corner cases are not well enough understood, and
_disabling_ vendor support in a future Go release sounds vanishingly
unlikely. We're going to be stuck with this behaviour for a long time,
I think it's prudent to wait.
- vendoring *must* only occur at the top of the repository, thus
providing a single location for packages in the repository to override
ones in the greater GOPATH.
- vendoring *must* only occur at the top of the repository, thus
providing a single location for packages in the repository to override
ones in the greater GOPATH.
Personally, I learned this the hard way. I ran across issues in practice.
I've also studied how people use dependencies in other languages. This tells us how people expect things to work. Nested dependencies are going to be annoying in practice because they don't work as the long tail of developers will expect them to.
For what it's worth (very little), I happen to agree with Daves proposal.
It's worth noting that the dependency tool authors on this thread have each expressed agreement in one vendor/ directory per repo.
I agree with Kyle that serious consideration should be given to
delaying making the vendor experiment the default in Go 1.6 -- I just
think the complex corner cases are not well enough understood, and
_disabling_ vendor support in a future Go release sounds vanishingly
unlikely.
At a minimum there should be clear guidelines published for Go
developers. I suggest
- vendoring *must* only occur at the top of the repository, thus
providing a single location for packages in the repository to override
ones in the greater GOPATH.
- the go tool *must* make it a compile error if it detects vendoring
at any other level of the repository*.
Could the compiler at least error when it determines that a vendored package's vendored types escapes the vendored package?
That doesn't fix a lot, but it sends a clear signal as to the purpose of vendor/.
As one of those 3rd party tool maintainers I'm equally concerned with the things Dave and Matt are concerned about. So +1 wrt their concerns. I've seen all of these issues and more during the 1.5 period.
--
You received this message because you are subscribed to the Google Groups "golang-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Thank you for your post. I gather that as of now, the build tool should allow for recursive vendoring and NOT provide a mechanism for fixing potential issues in references between different versions. You state that this should be made clearer. Where can this be documented to draw attention to it?
So, I'd like to talk about Go 1.7. In my recently closed issue, I brought up a scenerio where one may vendor a third party package, and that third party package also has vendoring. Russ, taking your structure again:
- $GOPATH/src/github.com/mattfarina/golang-broken-vendor- foo.go- vendor/- a/- b/- b.go- vendor/a/
Assuming foo uses both a and b/vendor/a and breaks.
As of Go 1.7, because I do not have an option to disable Go vendoring, I must either 1) convince the owner of a to remove vendoring 2) host my own fork of a 3) save the repo statically, without a symlink.
Could the compiler at least error when it determines that a vendored package's vendored types escapes the vendored package?
That doesn't fix a lot, but it sends a clear signal as to the purpose of vendor/.
As one of those 3rd party tool maintainers I'm equally concerned with the things Dave and Matt are concerned about. So +1 wrt their concerns. I've seen all of these issues and more during the 1.5 period.
--
Also, vendored code is a private copy of something. That is, "vendor" is explicitly "internal" as well. We very much do not want one project importing another project's vendored copy of a third project.
--
You received this message because you are subscribed to a topic in the Google Groups "golang-dev" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/golang-dev/WebP4dLV1b0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to golang-dev+...@googlegroups.com.
Having lurked on this discussion on and off I feel this is bad news. The simplicity of pretty much everything else in the Go ecosystem will make this the really odd duck. Having different tools work in different principal ways seems really bad. I have no solution but I just felt I should give a future users view.
I know it is simple but perhaps too simple given this thread? They do sound consistent but what with shadowing and linking issues exposed above perhaps they need amending.
I and everyone else of course have preconceptions. That is unavoidable. However what I meant about Cargo was not that it works in a certain way and we should replicate it but that the Rust community seems to have settled on both 1 way and even 1 tool to do it. And they didn't do it by years of darwinian trial and error but by choice. This by the way is something the Java world has been unable to do over the decades.
This is normally what I associate with "the Go way" highly opinionated _solutions_ to things that are generally contested.
Why is it so important all of a sudden to not force a particular way on everyone? We have done so repeatedly before.
Why is it so important all of a sudden to not force a particular way on everyone?
Ok I buy that. I do however think we went the wrong way about it. One of the coolest things that happened during the early days of Go was how breaking changes was handled via programmatic rewriting. We could have done the same thing here, collaborative development of one tool with automatic restructuring of people's code as changes were implemented.
Also, vendored code is a private copy of something. That is, "vendor" is explicitly "internal" as well. We very much do not want one project importing another project's vendored copy of a third project.But isn't this what internal/ is for? Would it make sense to have internal/ behave like vendor/ does for imports, and then eliminate vendor/ (this would also give us the relative imports that's been discussed in the past)?