[go] cmd/go: add support for vendoring in workspace mode
Michael Matloob (Gerrit)
Attention is currently required from: Bryan Mills.
To view, visit change 495801. To unsubscribe, or for help writing mail filters, visit settings.
Michael Matloob (Gerrit)
Attention is currently required from: Bryan Mills.
Michael Matloob uploaded patch set #2 to this change.
cmd/go: add support for vendoring in workspace mode
In most cases this change removes assumptions that there is a single
main module in vendor mode and iterates over the workspace modules
when doing checks. The go mod vendor command will now, if in workspace
mode, create a vendor directory in the same directory as the go.work
file, containing the packages (and modules in modules.txt) loaded from
the workspace. When reassembling the module graph from the vendor
directory, an edges are added from each of the main modules to their
requirements, plus additionally to a fake 'vendor/modules.txt' module
with edges to all the modules listed in vendor/modules.txt.
For #60056
Change-Id: I4a485bb39836e7ab35cdc7726229191c6599903e
---
M src/cmd/go/internal/modcmd/vendor.go
M src/cmd/go/internal/modload/buildlist.go
M src/cmd/go/internal/modload/import.go
M src/cmd/go/internal/modload/init.go
M src/cmd/go/internal/modload/load.go
M src/cmd/go/internal/modload/modfile.go
M src/cmd/go/internal/modload/search.go
M src/cmd/go/internal/modload/vendor.go
M src/cmd/go/testdata/script/work.txt
A src/cmd/go/testdata/script/work_vendor_modules_txt_consistent.txt
A src/cmd/go/testdata/script/work_vendor_prune.txt
A src/cmd/go/testdata/script/work_vendor_prune_all.txt
12 files changed, 709 insertions(+), 128 deletions(-)
To view, visit change 495801. To unsubscribe, or for help writing mail filters, visit settings.
Michael Matloob (Gerrit)
Attention is currently required from: Bryan Mills, Michael Matloob.
Michael Matloob uploaded patch set #3 to this change.
The following approvals got outdated and were removed: Run-TryBot+1 by Michael Matloob, TryBot-Result+1 by Gopher Robot
cmd/go: add support for vendoring in workspace mode
In most cases this change removes assumptions that there is a single
main module in vendor mode and iterates over the workspace modules
when doing checks. The go mod vendor command will now, if in workspace
mode, create a vendor directory in the same directory as the go.work
file, containing the packages (and modules in modules.txt) loaded from
the workspace. When reassembling the module graph from the vendor
directory, an edges are added from each of the main modules to their
requirements, plus additionally to a fake 'vendor/modules.txt' module
with edges to all the modules listed in vendor/modules.txt.
For #60056
Change-Id: I4a485bb39836e7ab35cdc7726229191c6599903e
---
M src/cmd/go/internal/modcmd/vendor.go
M src/cmd/go/internal/modload/buildlist.go
M src/cmd/go/internal/modload/import.go
M src/cmd/go/internal/modload/init.go
M src/cmd/go/internal/modload/load.go
M src/cmd/go/internal/modload/modfile.go
M src/cmd/go/internal/modload/search.go
M src/cmd/go/internal/modload/vendor.go
A src/cmd/go/internal/workcmd/vendor.go
M src/cmd/go/internal/workcmd/work.go
M src/cmd/go/testdata/script/work.txt
A src/cmd/go/testdata/script/work_vendor_modules_txt_consistent.txt
A src/cmd/go/testdata/script/work_vendor_prune.txt
A src/cmd/go/testdata/script/work_vendor_prune_all.txt
14 files changed, 782 insertions(+), 129 deletions(-)
To view, visit change 495801. To unsubscribe, or for help writing mail filters, visit settings.
Michael Matloob (Gerrit)
Attention is currently required from: Bryan Mills, Michael Matloob.
Michael Matloob uploaded patch set #4 to this change.
The following approvals got outdated and were removed: Run-TryBot+1 by Michael Matloob
cmd/go: add support for vendoring in workspace mode
14 files changed, 780 insertions(+), 129 deletions(-)
To view, visit change 495801. To unsubscribe, or for help writing mail filters, visit settings.
Michael Matloob (Gerrit)
Attention is currently required from: Bryan Mills, Michael Matloob.
Michael Matloob uploaded patch set #5 to this change.
The following approvals got outdated and were removed: Run-TryBot+1 by Michael Matloob, TryBot-Result-1 by Gopher Robot
cmd/go: add support for vendoring in workspace mode
In most cases this change removes assumptions that there is a single
main module in vendor mode and iterates over the workspace modules
when doing checks. The go mod vendor command will now, if in workspace
mode, create a vendor directory in the same directory as the go.work
file, containing the packages (and modules in modules.txt) loaded from
the workspace. When reassembling the module graph from the vendor
directory, an edges are added from each of the main modules to their
requirements, plus additionally to a fake 'vendor/modules.txt' module
with edges to all the modules listed in vendor/modules.txt.
For #60056
Change-Id: I4a485bb39836e7ab35cdc7726229191c6599903e
---
M src/cmd/go/internal/modcmd/vendor.go
M src/cmd/go/internal/modload/buildlist.go
M src/cmd/go/internal/modload/import.go
M src/cmd/go/internal/modload/init.go
M src/cmd/go/internal/modload/load.go
M src/cmd/go/internal/modload/modfile.go
M src/cmd/go/internal/modload/search.go
M src/cmd/go/internal/modload/vendor.go
A src/cmd/go/internal/workcmd/vendor.go
M src/cmd/go/internal/workcmd/work.go
M src/cmd/go/testdata/script/work.txt
A src/cmd/go/testdata/script/work_vendor_main_module_replaced.txt
A src/cmd/go/testdata/script/work_vendor_modules_txt_consistent.txt
A src/cmd/go/testdata/script/work_vendor_prune.txt
A src/cmd/go/testdata/script/work_vendor_prune_all.txt
15 files changed, 814 insertions(+), 129 deletions(-)
To view, visit change 495801. To unsubscribe, or for help writing mail filters, visit settings.
Michael Matloob (Gerrit)
Attention is currently required from: Bryan Mills, Michael Matloob.
Michael Matloob uploaded patch set #6 to this change.
15 files changed, 816 insertions(+), 129 deletions(-)
To view, visit change 495801. To unsubscribe, or for help writing mail filters, visit settings.
Michael Matloob (Gerrit)
Attention is currently required from: Bryan Mills, Michael Matloob.
Michael Matloob uploaded patch set #7 to this change.
15 files changed, 824 insertions(+), 129 deletions(-)
To view, visit change 495801. To unsubscribe, or for help writing mail filters, visit settings.
Michael Matloob (Gerrit)
Attention is currently required from: Bryan Mills, Michael Matloob.
Michael Matloob uploaded patch set #8 to this change.
The following approvals got outdated and were removed: Run-TryBot+1 by Michael Matloob, TryBot-Result-1 by Gopher Robot
cmd/go: add support for vendoring in workspace mode
In most cases this change removes assumptions that there is a single
main module in vendor mode and iterates over the workspace modules
when doing checks. The go mod vendor command will now, if in workspace
mode, create a vendor directory in the same directory as the go.work
file, containing the packages (and modules in modules.txt) loaded from
the workspace. When reassembling the module graph from the vendor
directory, an edges are added from each of the main modules to their
requirements, plus additionally to a fake 'vendor/modules.txt' module
with edges to all the modules listed in vendor/modules.txt.
For #60056
Change-Id: I4a485bb39836e7ab35cdc7726229191c6599903e
---
M src/cmd/go/alldocs.go
M src/cmd/go/internal/modcmd/vendor.go
M src/cmd/go/internal/modload/buildlist.go
M src/cmd/go/internal/modload/import.go
M src/cmd/go/internal/modload/init.go
M src/cmd/go/internal/modload/load.go
M src/cmd/go/internal/modload/modfile.go
M src/cmd/go/internal/modload/search.go
M src/cmd/go/internal/modload/vendor.go
A src/cmd/go/internal/workcmd/vendor.go
M src/cmd/go/internal/workcmd/work.go
M src/cmd/go/testdata/script/work.txt
A src/cmd/go/testdata/script/work_vendor_main_module_replaced.txt
A src/cmd/go/testdata/script/work_vendor_modules_txt_consistent.txt
A src/cmd/go/testdata/script/work_vendor_prune.txt
A src/cmd/go/testdata/script/work_vendor_prune_all.txt
16 files changed, 846 insertions(+), 129 deletions(-)
To view, visit change 495801. To unsubscribe, or for help writing mail filters, visit settings.
Michael Matloob (Gerrit)
Attention is currently required from: Bryan Mills, Michael Matloob.
Michael Matloob uploaded patch set #9 to this change.
16 files changed, 830 insertions(+), 121 deletions(-)
To view, visit change 495801. To unsubscribe, or for help writing mail filters, visit settings.
Michael Matloob (Gerrit)
Attention is currently required from: Bryan Mills, Michael Matloob.
Michael Matloob uploaded patch set #10 to this change.
The following approvals got outdated and were removed: Run-TryBot+1 by Michael Matloob
cmd/go: add support for vendoring in workspace mode
16 files changed, 829 insertions(+), 120 deletions(-)
To view, visit change 495801. To unsubscribe, or for help writing mail filters, visit settings.
Michael Matloob (Gerrit)
Attention is currently required from: Bryan Mills, Michael Matloob.
Michael Matloob uploaded patch set #11 to this change.
The following approvals got outdated and were removed: Run-TryBot+1 by Michael Matloob, TryBot-Result-1 by Gopher Robot
cmd/go: add support for vendoring in workspace mode
Bryan Mills (Gerrit)
Attention is currently required from: Michael Matloob.
3 comments:
File src/cmd/go/internal/modcmd/vendor.go:
// Question for Bryan: why do wo check for m.Version == "". Do we want to
// vendor packages in the main module / workspace module at different versions?No idea. Looks like that part of the condition was added in CL 334932, which I don't seem to have reviewed. 😅
As far as I can tell, just `m.Path == "" || modload.MainModules.Contains(m.Path)` would suffice.
File src/cmd/go/internal/modload/init.go:
Patch Set #11, Line 575:
return ""The empty case here surprises me — I would not expect `VendorDir` to be called unless we have `-mod=vendor` (explicitly or implicitly), and in that case I would expect the command to fail immediately if there is no corresponding `vendor` directory.
If there are cases where this is needed, a comment would be helpful.
Patch Set #11, Line 1392:
panic(fmt.Errorf("outside workspace mode, but have %v modRoots", modRoots))Is it possible that we have no `modRoots` here (in `NoRoot` mode)?
To view, visit change 495801. To unsubscribe, or for help writing mail filters, visit settings.
Michael Matloob (Gerrit)
Attention is currently required from: Bryan Mills, Michael Matloob.
Michael Matloob uploaded patch set #12 to this change.
The following approvals got outdated and were removed: Run-TryBot+1 by Michael Matloob, TryBot-Result+1 by Gopher Robot
16 files changed, 831 insertions(+), 121 deletions(-)
To view, visit change 495801. To unsubscribe, or for help writing mail filters, visit settings.
Michael Matloob (Gerrit)
Attention is currently required from: Bryan Mills, Michael Matloob.
// Question for Bryan: why do wo check for m.Version == "". Do we want to
// vendor packages in the main module / workspace module at different versions?
No idea. […]
Done
File src/cmd/go/internal/modload/init.go:
Patch Set #11, Line 575:
return ""
The empty case here surprises me — I would not expect `VendorDir` to be called unless we have `-mod= […]
We have a test case where -mod=vendor works with no modroot (as long as there are no dependencies). See script/vendor_outside_module.txt
Added a comment.
Patch Set #11, Line 1392:
panic(fmt.Errorf("outside workspace mode, but have %v modRoots", modRoots))Is it possible that we have no `modRoots` here (in `NoRoot` mode)?
We check at line 1372 that len(modRoots) >= 1. (Before this change, we checked that it was 1, so we excluded the no modroots case previously too)
To view, visit change 495801. To unsubscribe, or for help writing mail filters, visit settings.
Bryan Mills (Gerrit)
Attention is currently required from: Michael Matloob.
4 comments:
File src/cmd/go/internal/modcmd/vendor.go:
Patch Set #12, Line 189:
for _, r := range modload.MainModules.ModFile(m).Replace {(nit) In the baseline we always wrote unused replacements in the order in which they appeared in the original `go.mod` file.
However, since in workspace mode we are now intermixing `replace` directives from multiple sources (potentially the `go.work` file and multiple `go.mod` files), we should probably enforce some kind of ordering on them. I think the ordering right now ends up a little haphazard, because we iterate over the `go.mod` files first but write out the replacement from the `go.work` file instead if one is present.
Perhaps we should list the unused `go.work` entries first, followed by the entries for each `go.mod` file? Or perhaps we should sort the unused entries by path.
File src/cmd/go/internal/modload/import.go:
Patch Set #12, Line 343:
mods = append(mods, vendorPkgModule[path])Mind adding a TODO here? I think there is an existing bug.
If someone manually added `.go` source files to `vendor/example.com/foo`,
this would result in `import "example.com/foo"` succeeding but reporting an empty module path for the imported package, which violates all kinds of internal invariants and could perhaps cause a panic somewhere down the line.(#45649 is related.)
Patch Set #12, Line 345:
roots = append(roots, filepath.Dir(vendorDir))The call to `filepath.Dir` here looks a little suspect.
That return value appears to only be used here:
https://cs.opensource.google/go/go/+/master:src/cmd/go/internal/modload/load.go;l=1805;drc=ce8146ed3361f584ba79427ac6c6d6fe9c297beaIt is fed as an input to `modload.scanDir`, which passes it to `modindex.GetPackage`. `GetPackage` calls `GetModule`, which then [returns an error](https://cs.opensource.google/go/go/+/refs/heads/master:src/cmd/go/internal/modindex/read.go;l=156-161;drc=98617fd23fa799173c33741987d41ee64cbb2a4f), causing `GetPackage` to fall back to per-package indexing. It isn't clear to me what the per-package indexer does with that information or whether there may be a bug there.
At any rate: I think I would find this less confusing if the path added to `roots` is the same as the third (`mdir`) argument passed to `dirInModule`, since that is the case for all of the other calls to `dirInModule` here.
File src/cmd/go/internal/modload/init.go:
Patch Set #11, Line 575:
return ""
We have a test case where -mod=vendor works with no modroot (as long as there are no dependencies). […]
I'm worried about missing checks on the caller here.
Before this change, we only call `VendorDir` when either creating the directory itself (which requires a main module or workspace), or processing a `*load.Package` package with a non-nil `Module` field (which implies that the package was loaded from either a module directory or a module recorded in `vendor/modules.txt`). In both of those cases, the location of the vendor directory is well-defined.
I would like to either maintain the invariant that `VendorDir` is only called when the vendor directory is well-defined (implying that we should panic if it isn't), or change the `VendorDir` signature to return an explicit boolean instead of the empty string (so that we don't accidentally try to work with an invalid directory on the caller side).
To view, visit change 495801. To unsubscribe, or for help writing mail filters, visit settings.
Michael Matloob (Gerrit)
Attention is currently required from: Bryan Mills, Michael Matloob.
Michael Matloob uploaded patch set #13 to this change.
16 files changed, 844 insertions(+), 121 deletions(-)
To view, visit change 495801. To unsubscribe, or for help writing mail filters, visit settings.
Michael Matloob (Gerrit)
Attention is currently required from: Bryan Mills, Michael Matloob.
Patch Set #12, Line 189:
for _, r := range modload.MainModules.ModFile(m).Replace {
(nit) In the baseline we always wrote unused replacements in the order in which they appeared in the […]
Does this work? I moved the workfile replacements first, and then we list the replacements that are not overridden from each of the go.mod files.
File src/cmd/go/internal/modload/import.go:
Patch Set #12, Line 343:
mods = append(mods, vendorPkgModule[path])
Mind adding a TODO here? I think there is an existing bug. […]
Added.
Patch Set #12, Line 345:
roots = append(roots, filepath.Dir(vendorDir))The call to `filepath.Dir` here looks a little suspect. […]
That's VendorDir(), right?
File src/cmd/go/internal/modload/init.go:
Patch Set #11, Line 575:
return ""
I'm worried about missing checks on the caller here. […]
How's this: I added a panic to VendorDir() and a function vendorDirExists that is called in cases where the vendor directory is not guaranteed to exist.
To view, visit change 495801. To unsubscribe, or for help writing mail filters, visit settings.
Bryan Mills (Gerrit)
Attention is currently required from: Michael Matloob.
5 comments:
File src/cmd/go/internal/modcmd/vendor.go:
Patch Set #12, Line 118:
if gv := modload.MainModules.GoStatement(); gv != nil {It's not obvious to me why we need both `(*MainModuleSet).GoStatement` and `(*MainModuleSet).GoVersion` — they produce the same result when a `go` version is present (which by now is “basically always”).
If it's important that we know whether the Go version was explicit, perhaps `GoVersion` should return a `(string, bool)` to indicate that (instead of just a `string`)?
File src/cmd/go/internal/modload/buildlist.go:
for _, m := range MainModules.Versions() {
reqs, _ := rootsFromModFile(m, MainModules.ModFile(m))
mg.g.Require(m, append(reqs, vendorMod))
}
mg.g.Require(vendorMod, vendorList)We previously had the invariant that all of the modules in `rs.rootModules` are present in `mg.g` even in `vendor` mode. I think we have lost that property here.
In particular, I believe we have lost the `go` and/or` toolchain` roots added from the `go.work` file (although I would not want to fix that by hard-coding those specific targets here).
Perhaps in workspace mode we need to call `mvs.NewGraph` with `rs.rootModules` instead of `MainModules.Versions()`?
File src/cmd/go/internal/modload/init.go:
Patch Set #11, Line 1392:
panic(fmt.Errorf("outside workspace mode, but have %v modRoots", modRoots))We check at line 1372 that len(modRoots) >= 1. […]
Acknowledged
File src/cmd/go/internal/modload/init.go:
// Add explicit go and toolchain versions, inferring as needed.
roots = append(roots, module.Version{Path: "go", Version: goVersion})
direct["go"] = true // Every module directly uses the language and runtime.
if toolchain != "" {
roots = append(roots, module.Version{Path: "toolchain", Version: toolchain})
// Leave the toolchain as indirect: nothing in the user's module directly
// imports a package from the toolchain, and (like an indirect dependency in
// a module without graph pruning) we may remove the toolchain line
// automatically if the 'go' version is changed so that it implies the exact
// same toolchain.
}I wonder if the `go` and `toolchain` roots should be included in the slice returned by `rootsFromModFile`. That would provide a somewhat more realistic graph with workspace vendoring, although it's not obvious to me whether that realism is strictly necessary.
Patch Set #12, Line 1315:
roots = make([]module.Version, 0, 2+len(modFile.Require))A comment on the `2+` here would be helpful here — this extra padding is for the `go` and/or `toolchain` requirements added in `requirementsFromModFiles`, but that fact is no local local within the same scope.
To view, visit change 495801. To unsubscribe, or for help writing mail filters, visit settings.
Bryan Mills (Gerrit)
File src/cmd/go/internal/modcmd/vendor.go:
Patch Set #12, Line 189:
for _, r := range modload.MainModules.ModFile(m).Replace {Does this work? I moved the workfile replacements first, and then we list the replacements that are […]
Yep, I think that's good.
File src/cmd/go/internal/modload/import.go:
Patch Set #12, Line 345:
roots = append(roots, filepath.Dir(vendorDir))That's VendorDir(), right?
Yep, I think it's `vendorDir` exactly in this case.
File src/cmd/go/internal/modload/init.go:
Patch Set #13, Line 585:
vendorDirExists`vendorDirExists` to me carries a connotation of `Stat(VendorDir())` succeeding.
Is `vendorDirExists()` equivalent to `HasModRoot()`? (Could we use the latter instead?)
File src/cmd/go/internal/modload/vendor.go:
if len(modFiles) < 1 {
panic("checkVendorConsistency called with zero modfiles")
}FWIW, I think it would also be ok to just consider the check to be trivially passed if there are no modules to be checked. (That might simplify the `NoRoot` case somewhat? Not sure.)
Might at least be worth leaving a comment here about whether there are any specific concerns.
Patch Set #13, Line 229:
replacementSource(nit) If we are in workspace mode, each replacement comes from either a specific `go.mod` file or the `go.work` file. It would be helpful to indicate which file we found it in.
(I think that information could be obtained from the `modRoots` slice, perhaps shortened with `base.ShortPath`?)
To view, visit change 495801. To unsubscribe, or for help writing mail filters, visit settings.
Bryan Mills (Gerrit)
Attention is currently required from: Michael Matloob.
6 comments:
File src/cmd/go/testdata/script/work_vendor_main_module_replaced.txt:
Patch Set #13, Line 7:
go list all # make sure the consistency checks passThis is a nice control case, but it needs an experiment case too.
If we drop the `replace` directive in `a/go.mod`, does that cause the consistency check to fail? (Why or why not?)
File src/cmd/go/testdata/script/work_vendor_modules_txt_consistent.txt:
stderr 'example.com/p...@v1.0.0: is replaced by ../p in the workspace, but marked as replaced by r in vendor/modules.txt'
stderr 'example.com/p...@v1.0.0: is marked as replaced by r in vendor/modules.txt, but replaced by p in the workspace'Probably we should log only one of these two lines. 😅
(Perhaps we should also use `cmp stderr` instead of `stderr`, so that we can check the complete error messages — and detect unnecessary duplication — for all of these cases?)
This seems potentially problematic, or at least a little bit confusing — there is no indication that the `p` here is a file path rather than a module path.
I think this is a previously-buried bug in `canonicalizeReplacePath`: perhaps it should use `ToDirectoryPath` on the resulting path to ensure that the `Path` field in a `module.Version` always refers unambiguously to exactly one of a file path or a module path?
(I notice that `canonicalizeReplacePath` always returns the path unchanged when `WorkFilePath()` is empty, so this probably would not have been visible before outside of `go list`. Probably better to fix it now before it leaks into files that may be committed to version control.)
File src/cmd/go/testdata/script/work_vendor_prune.txt:
replace example.com/q v1.0.0 => ../q1_0_0
replace example.com/q v1.1.0 => ../q1_1_0These replacements have no effect since they're not in the workspace, right?
File src/cmd/go/testdata/script/work_vendor_prune_all.txt:
go list -m -f '{{.Version}}' example.com/w
stdout '^v1.0.0$'
go list -m -f '{{.Version}}' example.com/b
stdout '^v1.0.0$'
go list -m -f '{{.Version}}' example.com/q
stdout '^v1.1.0$'
go list -m -f '{{.Version}}' example.com/z
stdout '^v1.1.0$'(nit) Maybe `go list -f '{{with .Module}}{{.Path}} {{.Version}}{{end}}' all` instead of listing the modules individually?
Patch Set #13, Line 35:
go run example.com/p(nit) `go run` might be a little overkill, given that we're already check the `Dir` for all of these dependencies. (Or maybe use `go list -deps example.com/p` instead of listing the dependency packages individually above?)
To view, visit change 495801. To unsubscribe, or for help writing mail filters, visit settings.
Michael Matloob (Gerrit)
Attention is currently required from: Bryan Mills, Michael Matloob.
Michael Matloob uploaded patch set #14 to this change.
16 files changed, 909 insertions(+), 150 deletions(-)
To view, visit change 495801. To unsubscribe, or for help writing mail filters, visit settings.
Michael Matloob (Gerrit)
Attention is currently required from: Bryan Mills, Michael Matloob.
13 comments:
File src/cmd/go/internal/modcmd/vendor.go:
Patch Set #12, Line 118:
if gv := modload.MainModules.GoStatement(); gv != nil {It's not obvious to me why we need both `(*MainModuleSet).GoStatement` and `(*MainModuleSet). […]
I don't know if it's important, but this seems to be one of the few places we have different behavior if there's no go version. Since go versions are present most of the time, is it ok to just assume it exists? That would simplify things for sure.
File src/cmd/go/internal/modload/buildlist.go:
for _, m := range MainModules.Versions() {
reqs, _ := rootsFromModFile(m, MainModules.ModFile(m))
mg.g.Require(m, append(reqs, vendorMod))
}
mg.g.Require(vendorMod, vendorList)
We previously had the invariant that all of the modules in `rs.rootModules` are present in `mg. […]
would this be fixed if we had `rootsFromModFile` add in the `go` and `toolchain` roots?
File src/cmd/go/internal/modload/init.go:
// Add explicit go and toolchain versions, inferring as needed.
roots = append(roots, module.Version{Path: "go", Version: goVersion})
direct["go"] = true // Every module directly uses the language and runtime.
if toolchain != "" {
roots = append(roots, module.Version{Path: "toolchain", Version: toolchain})
// Leave the toolchain as indirect: nothing in the user's module directly
// imports a package from the toolchain, and (like an indirect dependency in
// a module without graph pruning) we may remove the toolchain line
// automatically if the 'go' version is changed so that it implies the exact
// same toolchain.
}
I wonder if the `go` and `toolchain` roots should be included in the slice returned by `rootsFromMod […]
I tried adding them here. The code is pretty awkward because we also want to add them in the workspace case.
Patch Set #12, Line 1315:
roots = make([]module.Version, 0, 2+len(modFile.Require))
A comment on the `2+` here would be helpful here — this extra padding is for the `go` and/or `toolch […]
Done
File src/cmd/go/internal/modload/init.go:
Patch Set #13, Line 585:
vendorDirExists
`vendorDirExists` to me carries a connotation of `Stat(VendorDir())` succeeding. […]
Done
File src/cmd/go/internal/modload/vendor.go:
if len(modFiles) < 1 {
panic("checkVendorConsistency called with zero modfiles")
}
FWIW, I think it would also be ok to just consider the check to be trivially passed if there are no […]
Added a comment.
Patch Set #13, Line 229:
replacementSource
(nit) If we are in workspace mode, each replacement comes from either a specific `go. […]
How's this?
One potential issue is that in the error below (line 232), if we say it's replaced in the go.mod, we have to choose between giving a path relative to the go.mod (which is probably less confusing because we're saying it's replaced by X in go.mod) or a path relative to the workspace (what we're actually comparing it to)
File src/cmd/go/testdata/script/work_vendor_main_module_replaced.txt:
Patch Set #13, Line 7:
go list all # make sure the consistency checks pass
This is a nice control case, but it needs an experiment case too. […]
Added a test case.
File src/cmd/go/testdata/script/work_vendor_modules_txt_consistent.txt:
stderr 'example.com/p...@v1.0.0: is replaced by ../p in the workspace, but marked as replaced by r in vendor/modules.txt'
stderr 'example.com/p...@v1.0.0: is marked as replaced by r in vendor/modules.txt, but replaced by p in the workspace'
Probably we should log only one of these two lines. 😅 […]
Done
This seems potentially problematic, or at least a little bit confusing — there is no indication that […]
Done
File src/cmd/go/testdata/script/work_vendor_prune.txt:
replace example.com/q v1.0.0 => ../q1_0_0
replace example.com/q v1.1.0 => ../q1_1_0These replacements have no effect since they're not in the workspace, right?
Ah right! Done.
File src/cmd/go/testdata/script/work_vendor_prune_all.txt:
go list -m -f '{{.Version}}' example.com/w
stdout '^v1.0.0$'
go list -m -f '{{.Version}}' example.com/b
stdout '^v1.0.0$'
go list -m -f '{{.Version}}' example.com/q
stdout '^v1.1.0$'
go list -m -f '{{.Version}}' example.com/z
stdout '^v1.1.0$'
(nit) Maybe `go list -f '{{with .Module}}{{.Path}} {{. […]
Done
(nit) `go run` might be a little overkill, given that we're already check the `Dir` for all of these […]
Removed.
To view, visit change 495801. To unsubscribe, or for help writing mail filters, visit settings.
Michael Matloob (Gerrit)
Attention is currently required from: Bryan Mills, Michael Matloob.
Michael Matloob uploaded patch set #15 to this change.
The following approvals got outdated and were removed: Run-TryBot+1 by Michael Matloob, TryBot-Result-1 by Gopher Robot
To view, visit change 495801. To unsubscribe, or for help writing mail filters, visit settings.
Bryan Mills (Gerrit)
Attention is currently required from: Michael Matloob, triciu...@appspot.gserviceaccount.com.
10 comments:
File src/cmd/go/internal/modload/buildlist.go:
for _, m := range MainModules.Versions() {
reqs, _ := rootsFromModFile(m, MainModules.ModFile(m))
mg.g.Require(m, append(reqs, vendorMod))
}
mg.g.Require(vendorMod, vendorList)
would this be fixed if we had `rootsFromModFile` add in the `go` and `toolchain` roots?
I think this is still a problem: I think we're missing the roots from the `go.work` file in the graph, and those wouldn't be included in the `rootsFromModFile` for any one module.
File src/cmd/go/internal/modload/import.go:
Patch Set #15, Line 345:
suceed"suceed" is a possible misspelling of "succeed".
Please fix.
File src/cmd/go/internal/modload/init.go:
Patch Set #11, Line 575:
return ""How's this: I added a panic to VendorDir() and a function vendorDirExists that is called in cases wh […]
SGTM.
File src/cmd/go/internal/modload/init.go:
// Add explicit go and toolchain versions, inferring as needed.
roots = append(roots, module.Version{Path: "go", Version: goVersion})
direct["go"] = true // Every module directly uses the language and runtime.
if toolchain != "" {
roots = append(roots, module.Version{Path: "toolchain", Version: toolchain})
// Leave the toolchain as indirect: nothing in the user's module directly
// imports a package from the toolchain, and (like an indirect dependency in
// a module without graph pruning) we may remove the toolchain line
// automatically if the 'go' version is changed so that it implies the exact
// same toolchain.
}
I tried adding them here. […]
Hmm. I might have led you wrong here — it looks like in the baseline, in workspace mode we only added the `go` and `toolchain` requirements for the workspace proper in workspace mode.
That might be an existing bug, too. Probably in module mode we ought to add the `go` version for each module if (and only if) that version is at least `gover.GoStrictVersion`, and omit the per-module `toolchain` version (the `toolchain` line in the `go.work` file wins, I think). 🤔
File src/cmd/go/internal/modload/init.go:
Patch Set #15, Line 570:
filepath.Join(MainModules.ModRoot(MainModules.mustGetSingleMainModule()), "vendor")(nit) shorten to `filepath.Join(modRoot, "vendor")`
File src/cmd/go/internal/modload/modfile.go:
if foundModRoot != "" {
r = canonicalizeReplacePath(r, foundModRoot)
}(nit) This `if modRoot != ""` pattern appears a couple of times. Maybe change `canonicalizeReplacePath` to be a no-op if `modRoot` is empty, so that you can make the calls unconditional.?
File src/cmd/go/internal/modload/vendor.go:
Patch Set #13, Line 229:
replacementSourceHow's this? […]
Seems reasonable.
File src/cmd/go/testdata/script/work_vendor_main_module_replaced.txt:
Patch Set #13, Line 7:
go list all # make sure the consistency checks pass
Added a test case.
Done
File src/cmd/go/testdata/script/work_vendor_prune.txt:
replace example.com/q v1.0.0 => ../q1_0_0
replace example.com/q v1.1.0 => ../q1_1_0
Ah right! Done.
Acknowledged
File src/cmd/go/testdata/script/work_vendor_prune_all.txt:
Patch Set #13, Line 35:
go run example.com/pRemoved.
Acknowledged
To view, visit change 495801. To unsubscribe, or for help writing mail filters, visit settings.
Michael Matloob (Gerrit)
Attention is currently required from: Bryan Mills, Michael Matloob, triciu...@appspot.gserviceaccount.com.
Michael Matloob uploaded patch set #16 to this change.
The following approvals got outdated and were removed: Run-TryBot+1 by Michael Matloob, TryBot-Result+1 by Gopher Robot
16 files changed, 915 insertions(+), 153 deletions(-)
To view, visit change 495801. To unsubscribe, or for help writing mail filters, visit settings.
Michael Matloob (Gerrit)
Attention is currently required from: Bryan Mills, Michael Matloob, triciu...@appspot.gserviceaccount.com.
Patch Set #15, Line 345:
suceed
> "suceed" is a possible misspelling of "succeed". […]
Done
File src/cmd/go/internal/modload/init.go:
Patch Set #15, Line 570:
filepath.Join(MainModules.ModRoot(MainModules.mustGetSingleMainModule()), "vendor")
(nit) shorten to `filepath. […]
Done
File src/cmd/go/internal/modload/modfile.go:
if foundModRoot != "" {
r = canonicalizeReplacePath(r, foundModRoot)
}
(nit) This `if modRoot != ""` pattern appears a couple of times. […]
Done
To view, visit change 495801. To unsubscribe, or for help writing mail filters, visit settings.
Michael Matloob (Gerrit)
Attention is currently required from: Bryan Mills, Michael Matloob, triciu...@appspot.gserviceaccount.com.
2 comments:
File src/cmd/go/internal/modload/buildlist.go:
for _, m := range MainModules.Versions() {
reqs, _ := rootsFromModFile(m, MainModules.ModFile(m))
mg.g.Require(m, append(reqs, vendorMod))
}
mg.g.Require(vendorMod, vendorList)
I think this is still a problem: I think we're missing the roots from the `go. […]
Changed to call mvs.NewGraph with rs.rootModules
File src/cmd/go/internal/modload/init.go:
// Add explicit go and toolchain versions, inferring as needed.
roots = append(roots, module.Version{Path: "go", Version: goVersion})
direct["go"] = true // Every module directly uses the language and runtime.
if toolchain != "" {
roots = append(roots, module.Version{Path: "toolchain", Version: toolchain})
// Leave the toolchain as indirect: nothing in the user's module directly
// imports a package from the toolchain, and (like an indirect dependency in
// a module without graph pruning) we may remove the toolchain line
// automatically if the 'go' version is changed so that it implies the exact
// same toolchain.
}
Hmm. […]
Updated rootsFromModFile to conditionally add toolchainRoot
To view, visit change 495801. To unsubscribe, or for help writing mail filters, visit settings.
Michael Matloob (Gerrit)
Attention is currently required from: Bryan Mills, Michael Matloob, triciu...@appspot.gserviceaccount.com.
Michael Matloob uploaded patch set #17 to this change.
The following approvals got outdated and were removed: Run-TryBot+1 by Michael Matloob
cmd/go: add support for vendoring in workspace mode
16 files changed, 919 insertions(+), 153 deletions(-)
To view, visit change 495801. To unsubscribe, or for help writing mail filters, visit settings.
Bryan Mills (Gerrit)
Attention is currently required from: Michael Matloob, triciu...@appspot.gserviceaccount.com.
Patch set 17:Code-Review +2
5 comments:
File src/cmd/go/internal/modcmd/vendor.go:
Patch Set #12, Line 118:
if gv := modload.MainModules.GoStatement(); gv != nil {
I don't know if it's important, but this seems to be one of the few places we have different behavio […]
Unfortunately I don't think we can just assume it exists — there may be tools out in the world that check out a copy of a module, run `go mod vendor`, and then push the code elsewhere. (I think Google Cloud Run might do that? Not sure.)
But if there is no explicit Go version, I think `modload.MainModules.GoVersion()` returns a default — is that default low enough not to matter for the checks here?
File src/cmd/go/internal/modload/buildlist.go:
for _, m := range MainModules.Versions() {
reqs, _ := rootsFromModFile(m, MainModules.ModFile(m))
mg.g.Require(m, append(reqs, vendorMod))
}
mg.g.Require(vendorMod, vendorList)
Changed to call mvs.NewGraph with rs. […]
Acknowledged
File src/cmd/go/internal/modload/buildlist.go:
Patch Set #17, Line 223:
doNotAddToolchainRoot := false(nit) It's not quite clear to me whether this means “do add toolchain root” or “don't add toolchain root” — can we define an enum-like type for this instead, with constants like `withToolchainRoot` and `omitToolchainRoot`?
File src/cmd/go/internal/modload/init.go:
// Add explicit go and toolchain versions, inferring as needed.
roots = append(roots, module.Version{Path: "go", Version: goVersion})
direct["go"] = true // Every module directly uses the language and runtime.
if toolchain != "" {
roots = append(roots, module.Version{Path: "toolchain", Version: toolchain})
// Leave the toolchain as indirect: nothing in the user's module directly
// imports a package from the toolchain, and (like an indirect dependency in
// a module without graph pruning) we may remove the toolchain line
// automatically if the 'go' version is changed so that it implies the exact
// same toolchain.
}
Updated rootsFromModFile to conditionally add toolchainRoot
Acknowledged
File src/cmd/go/internal/modload/vendor.go:
if len(modFiles) < 1 {
panic("checkVendorConsistency called with zero modfiles")
}
Added a comment.
Acknowledged
To view, visit change 495801. To unsubscribe, or for help writing mail filters, visit settings.
Michael Matloob (Gerrit)
Attention is currently required from: Bryan Mills, Michael Matloob, triciu...@appspot.gserviceaccount.com.
Patch Set #12, Line 118:
if gv := modload.MainModules.GoStatement(); gv != nil {
Unfortunately I don't think we can just assume it exists — there may be tools out in the world that […]
The default is gover.DefaultGoModVersion, which is Go 1.16 We care about 1.14 here so the default is not low enough to not matter for the checks here.
Should I add back GoStatement so we can check for the existence of the version again?
File src/cmd/go/internal/modload/buildlist.go:
Patch Set #17, Line 223:
doNotAddToolchainRoot := false
(nit) It's not quite clear to me whether this means “do add toolchain root” or “don't add toolchain […]
How's this
To view, visit change 495801. To unsubscribe, or for help writing mail filters, visit settings.
Michael Matloob (Gerrit)
Attention is currently required from: Bryan Mills, Michael Matloob, triciu...@appspot.gserviceaccount.com.
Michael Matloob uploaded patch set #18 to this change.
The following approvals got outdated and were removed: Run-TryBot+1 by Michael Matloob, TryBot-Result+1 by Gopher Robot
cmd/go: add support for vendoring in workspace mode
16 files changed, 924 insertions(+), 153 deletions(-)
To view, visit change 495801. To unsubscribe, or for help writing mail filters, visit settings.
Michael Matloob (Gerrit)
Attention is currently required from: Bryan Mills, Michael Matloob, triciu...@appspot.gserviceaccount.com.
Michael Matloob uploaded patch set #21 to this change.
The following approvals got outdated and were removed: Run-TryBot+1 by Michael Matloob, TryBot-Result+1 by Gopher Robot
The change is no longer submittable: TryBots-Pass is unsatisfied now.
A src/cmd/go/testdata/script/work_vendor_empty.txt
A src/cmd/go/testdata/script/work_vendor_main_module_replaced.txt
A src/cmd/go/testdata/script/work_vendor_modules_txt_consistent.txt
A src/cmd/go/testdata/script/work_vendor_prune.txt
A src/cmd/go/testdata/script/work_vendor_prune_all.txt
17 files changed, 940 insertions(+), 153 deletions(-)
To view, visit change 495801. To unsubscribe, or for help writing mail filters, visit settings.
Michael Matloob (Gerrit)
Michael Matloob submitted this change.
18 is the latest approved patch-set.
The change was submitted with unreviewed changes in the following files:
```
The name of the file: src/cmd/go/testdata/script/work_vendor_empty.txt
Insertions: 16, Deletions: 0.
The diff is too large to show. Please review the diff.
```
```
The name of the file: src/cmd/go/internal/modcmd/vendor.go
Insertions: 1, Deletions: 1.
The diff is too large to show. Please review the diff.
```
cmd/go: add support for vendoring in workspace mode
In most cases this change removes assumptions that there is a single
main module in vendor mode and iterates over the workspace modules
when doing checks. The go mod vendor command will now, if in workspace
mode, create a vendor directory in the same directory as the go.work
file, containing the packages (and modules in modules.txt) loaded from
the workspace. When reassembling the module graph from the vendor
directory, an edges are added from each of the main modules to their
requirements, plus additionally to a fake 'vendor/modules.txt' module
with edges to all the modules listed in vendor/modules.txt.
For #60056
Change-Id: I4a485bb39836e7ab35cdc7726229191c6599903e
Reviewed-on: https://go-review.googlesource.com/c/go/+/495801
Reviewed-by: Bryan Mills <bcm...@google.com>
TryBot-Result: Gopher Robot <go...@golang.org>
Run-TryBot: Michael Matloob <mat...@golang.org>
---
M src/cmd/go/alldocs.go
M src/cmd/go/internal/modcmd/vendor.go
M src/cmd/go/internal/modload/buildlist.go
M src/cmd/go/internal/modload/import.go
M src/cmd/go/internal/modload/init.go
M src/cmd/go/internal/modload/load.go
M src/cmd/go/internal/modload/modfile.go
M src/cmd/go/internal/modload/search.go
M src/cmd/go/internal/modload/vendor.go
A src/cmd/go/internal/workcmd/vendor.go
M src/cmd/go/internal/workcmd/work.go
M src/cmd/go/testdata/script/work.txt
A src/cmd/go/testdata/script/work_vendor_empty.txt
A src/cmd/go/testdata/script/work_vendor_main_module_replaced.txt
A src/cmd/go/testdata/script/work_vendor_modules_txt_consistent.txt
A src/cmd/go/testdata/script/work_vendor_prune.txt
A src/cmd/go/testdata/script/work_vendor_prune_all.txt
17 files changed, 940 insertions(+), 153 deletions(-)
diff --git a/src/cmd/go/alldocs.go b/src/cmd/go/alldocs.go
index bb28756..45de1cc 100644
--- a/src/cmd/go/alldocs.go
+++ b/src/cmd/go/alldocs.go
@@ -1504,6 +1504,7 @@
// init initialize workspace file
// sync sync workspace build list to modules
// use add modules to workspace file
+// vendor make vendored copy of dependencies
//
// Use "go help work <command>" for more information about a command.
//
@@ -1652,6 +1653,27 @@
// See the workspaces reference at https://go.dev/ref/mod#workspaces
// for more information.
//
+// # Make vendored copy of dependencies
+//
+// Usage:
+//
+// go work vendor [-e] [-v] [-o outdir]
+//
+// Vendor resets the workspace's vendor directory to include all packages
+// needed to build and test all the workspace's packages.
+// It does not include test code for vendored packages.
+//
+// The -v flag causes vendor to print the names of vendored
+// modules and packages to standard error.
+//
+// The -e flag causes vendor to attempt to proceed despite errors
+// encountered while loading packages.
+//
+// The -o flag causes vendor to create the vendor directory at the given
+// path instead of "vendor". The go command can only use a vendor directory
+// named "vendor" within the module root directory, so this flag is
+// primarily useful for other tools.
+//
// # Compile and run Go program
//
// Usage:
diff --git a/src/cmd/go/internal/modcmd/vendor.go b/src/cmd/go/internal/modcmd/vendor.go
index 1a0d69e..82a2675 100644
--- a/src/cmd/go/internal/modcmd/vendor.go
+++ b/src/cmd/go/internal/modcmd/vendor.go
@@ -66,6 +66,14 @@
}
func runVendor(ctx context.Context, cmd *base.Command, args []string) {
+ modload.InitWorkfile()
+ if modload.WorkFilePath() != "" {
+ base.Fatalf("go: 'go mod vendor' cannot be run in workspace mode. Run 'go work vendor' to vendor the workspace or set 'GOWORK=off' to exit workspace mode.")
+ }
+ RunVendor(ctx, vendorE, vendorO, args)
+}
+
+func RunVendor(ctx context.Context, vendorE bool, vendorO string, args []string) {
if len(args) != 0 {
base.Fatalf("go: 'go mod vendor' accepts no arguments")
}
@@ -98,7 +106,7 @@
modpkgs := make(map[module.Version][]string)
for _, pkg := range pkgs {
m := modload.PackageModule(pkg)
- if m.Path == "" || m.Version == "" && modload.MainModules.Contains(m.Path) {
+ if m.Path == "" || modload.MainModules.Contains(m.Path) {
continue
}
modpkgs[m] = append(modpkgs[m], pkg)
@@ -107,21 +115,25 @@
includeAllReplacements := false
includeGoVersions := false
isExplicit := map[module.Version]bool{}
- if gv := modload.ModFile().Go; gv != nil {
- if gover.Compare(gv.Version, "1.14") >= 0 {
- // If the Go version is at least 1.14, annotate all explicit 'require' and
- // 'replace' targets found in the go.mod file so that we can perform a
- // stronger consistency check when -mod=vendor is set.
- for _, r := range modload.ModFile().Require {
- isExplicit[r.Mod] = true
+ gv := modload.MainModules.GoVersion()
+ if gover.Compare(gv, "1.14") >= 0 && (modload.FindGoWork(base.Cwd()) != "" || modload.ModFile().Go != nil) {
+ // If the Go version is at least 1.14, annotate all explicit 'require' and
+ // 'replace' targets found in the go.mod file so that we can perform a
+ // stronger consistency check when -mod=vendor is set.
+ for _, m := range modload.MainModules.Versions() {
+ if modFile := modload.MainModules.ModFile(m); modFile != nil {
+ for _, r := range modFile.Require {
+ isExplicit[r.Mod] = true
+ }
}
- includeAllReplacements = true
+
}
- if gover.Compare(gv.Version, "1.17") >= 0 {
- // If the Go version is at least 1.17, annotate all modules with their
- // 'go' version directives.
- includeGoVersions = true
- }
+ includeAllReplacements = true
+ }
+ if gover.Compare(gv, "1.17") >= 0 {
+ // If the Go version is at least 1.17, annotate all modules with their
+ // 'go' version directives.
+ includeGoVersions = true
}
var vendorMods []module.Version
@@ -143,9 +155,11 @@
w = io.MultiWriter(&buf, os.Stderr)
}
+ replacementWritten := make(map[module.Version]bool)
for _, m := range vendorMods {
replacement := modload.Replacement(m)
line := moduleLine(m, replacement)
+ replacementWritten[m] = true
io.WriteString(w, line)
goVersion := ""
@@ -173,17 +187,41 @@
// Record unused and wildcard replacements at the end of the modules.txt file:
// without access to the complete build list, the consumer of the vendor
// directory can't otherwise determine that those replacements had no effect.
- for _, r := range modload.ModFile().Replace {
- if len(modpkgs[r.Old]) > 0 {
- // We we already recorded this replacement in the entry for the replaced
- // module with the packages it provides.
- continue
- }
+ for _, m := range modload.MainModules.Versions() {
+ if workFile := modload.MainModules.WorkFile(); workFile != nil {
+ for _, r := range workFile.Replace {
+ if replacementWritten[r.Old] {
+ // We already recorded this replacement.
+ continue
+ }
+ replacementWritten[r.Old] = true
- line := moduleLine(r.Old, r.New)
- buf.WriteString(line)
- if cfg.BuildV {
- os.Stderr.WriteString(line)
+ line := moduleLine(r.Old, r.New)
+ buf.WriteString(line)
+ if cfg.BuildV {
+ os.Stderr.WriteString(line)
+ }
+ }
+ }
+ if modFile := modload.MainModules.ModFile(m); modFile != nil {
+ for _, r := range modFile.Replace {
+ if replacementWritten[r.Old] {
+ // We already recorded this replacement.
+ continue
+ }
+ replacementWritten[r.Old] = true
+ rNew := modload.Replacement(r.Old)
+ if rNew == (module.Version{}) {
+ // There is no replacement. Don't try to write it.
+ continue
+ }
+
+ line := moduleLine(r.Old, rNew)
+ buf.WriteString(line)
+ if cfg.BuildV {
+ os.Stderr.WriteString(line)
+ }
+ }
}
}
}
@@ -367,7 +405,7 @@
return false
}
if info.Name() == "go.mod" || info.Name() == "go.sum" {
- if gv := modload.ModFile().Go; gv != nil && gover.Compare(gv.Version, "1.17") >= 0 {
+ if gv := modload.MainModules.GoVersion(); gover.Compare(gv, "1.17") >= 0 {
// As of Go 1.17, we strip go.mod and go.sum files from dependency modules.
// Otherwise, 'go' commands invoked within the vendor subtree may misidentify
// an arbitrary directory within the vendor tree as a module root.
diff --git a/src/cmd/go/internal/modload/buildlist.go b/src/cmd/go/internal/modload/buildlist.go
index 8d3af08..3908e85 100644
--- a/src/cmd/go/internal/modload/buildlist.go
+++ b/src/cmd/go/internal/modload/buildlist.go
@@ -174,17 +174,20 @@
// requirements.
func (rs *Requirements) initVendor(vendorList []module.Version) {
rs.graphOnce.Do(func() {
+ roots := MainModules.Versions()
+ if inWorkspaceMode() {
+ // Use rs.rootModules to pull in the go and toolchain roots
+ // from the go.work file and preserve the invariant that all
+ // of rs.rootModules are in mg.g.
+ roots = rs.rootModules
+ }
mg := &ModuleGraph{
- g: mvs.NewGraph(cmpVersion, MainModules.Versions()),
+ g: mvs.NewGraph(cmpVersion, roots),
}
- if MainModules.Len() != 1 {
- panic("There should be exactly one main module in Vendor mode.")
- }
- mainModule := MainModules.Versions()[0]
-
if rs.pruning == pruned {
- // The roots of a pruned module should already include every module in the
+ mainModule := MainModules.mustGetSingleMainModule()
+ // The roots of a single pruned module should already include every module in the
// vendor list, because the vendored modules are the same as those needed
// for graph pruning.
//
@@ -215,8 +218,18 @@
// graph, but still distinguishes between direct and indirect
// dependencies.
vendorMod := module.Version{Path: "vendor/modules.txt", Version: ""}
- mg.g.Require(mainModule, append(rs.rootModules, vendorMod))
- mg.g.Require(vendorMod, vendorList)
+ if inWorkspaceMode() {
+ for _, m := range MainModules.Versions() {
+ reqs, _ := rootsFromModFile(m, MainModules.ModFile(m), omitToolchainRoot)
+ mg.g.Require(m, append(reqs, vendorMod))
+ }
+ mg.g.Require(vendorMod, vendorList)
+
+ } else {
+ mainModule := MainModules.mustGetSingleMainModule()
+ mg.g.Require(mainModule, append(rs.rootModules, vendorMod))
+ mg.g.Require(vendorMod, vendorList)
+ }
}
rs.graph.Store(&cachedGraph{mg, nil})
diff --git a/src/cmd/go/internal/modload/import.go b/src/cmd/go/internal/modload/import.go
index 83b9ad4..cc6a482 100644
--- a/src/cmd/go/internal/modload/import.go
+++ b/src/cmd/go/internal/modload/import.go
@@ -318,30 +318,41 @@
mods = append(mods, module.Version{})
}
// -mod=vendor is special.
- // Everything must be in the main module or the main module's vendor directory.
+ // Everything must be in the main modules or the main module's or workspace's vendor directory.
if cfg.BuildMod == "vendor" {
- mainModule := MainModules.mustGetSingleMainModule()
- modRoot := MainModules.ModRoot(mainModule)
var mainErr error
- if modRoot != "" {
- mainDir, mainOK, err := dirInModule(path, MainModules.PathPrefix(mainModule), modRoot, true)
- mainErr = err
- if mainOK {
- mods = append(mods, mainModule)
- dirs = append(dirs, mainDir)
- roots = append(roots, modRoot)
+ for _, mainModule := range MainModules.Versions() {
+ modRoot := MainModules.ModRoot(mainModule)
+ if modRoot != "" {
+ dir, mainOK, err := dirInModule(path, MainModules.PathPrefix(mainModule), modRoot, true)
+ if mainErr == nil {
+ mainErr = err
+ }
+ if mainOK {
+ mods = append(mods, mainModule)
+ dirs = append(dirs, dir)
+ roots = append(roots, modRoot)
+ }
}
- vendorDir, vendorOK, _ := dirInModule(path, "", filepath.Join(modRoot, "vendor"), false)
+ }
+
+ if HasModRoot() {
+ vendorDir := VendorDir()
+ dir, vendorOK, _ := dirInModule(path, "", vendorDir, false)
if vendorOK {
- readVendorList(mainModule)
+ readVendorList(vendorDir)
+ // TODO(#60922): It's possible for a package to manually have been added to the
+ // vendor directory, causing the dirInModule to succeed, but no vendorPkgModule
+ // to exist, causing an empty module path to be reported. Do better checking
+ // here.
mods = append(mods, vendorPkgModule[path])
- dirs = append(dirs, vendorDir)
- roots = append(roots, modRoot)
+ dirs = append(dirs, dir)
+ roots = append(roots, vendorDir)
}
}
if len(dirs) > 1 {
- return module.Version{}, modRoot, "", nil, &AmbiguousImportError{importPath: path, Dirs: dirs}
+ return module.Version{}, "", "", nil, &AmbiguousImportError{importPath: path, Dirs: dirs}
}
if mainErr != nil {
@@ -349,7 +360,7 @@
}
if len(dirs) == 0 {
- return module.Version{}, modRoot, "", nil, &ImportMissingError{Path: path}
+ return module.Version{}, "", "", nil, &ImportMissingError{Path: path}
}
return mods[0], roots[0], dirs[0], nil, nil
diff --git a/src/cmd/go/internal/modload/init.go b/src/cmd/go/internal/modload/init.go
index 1c6f7d9..5ab46d5 100644
--- a/src/cmd/go/internal/modload/init.go
+++ b/src/cmd/go/internal/modload/init.go
@@ -200,6 +200,10 @@
return mms.modFiles[m]
}
+func (mms *MainModuleSet) WorkFile() *modfile.WorkFile {
+ return mms.workFile
+}
+
func (mms *MainModuleSet) Len() int {
if mms == nil {
return 0
@@ -553,7 +557,17 @@
}
func VendorDir() string {
- return filepath.Join(MainModules.ModRoot(MainModules.mustGetSingleMainModule()), "vendor")
+ if inWorkspaceMode() {
+ return filepath.Join(filepath.Dir(WorkFilePath()), "vendor")
+ }
+ // Even if -mod=vendor, we could be operating with no mod root (and thus no
+ // vendor directory). As long as there are no dependencies that is expected
+ // to work. See script/vendor_outside_module.txt.
+ modRoot := MainModules.ModRoot(MainModules.mustGetSingleMainModule())
+ if modRoot == "" {
+ panic("vendor directory does not exist when in single module mode outside of a module")
+ }
+ return filepath.Join(modRoot, "vendor")
}
func inWorkspaceMode() bool {
@@ -914,23 +928,28 @@
setDefaultBuildMod() // possibly enable automatic vendoring
rs := requirementsFromModFiles(ctx, workFile, modFiles, opts)
+ if cfg.BuildMod == "vendor" {
+ readVendorList(VendorDir())
+ var indexes []*modFileIndex
+ var modFiles []*modfile.File
+ var modRoots []string
+ for _, m := range MainModules.Versions() {
+ indexes = append(indexes, MainModules.Index(m))
+ modFiles = append(modFiles, MainModules.ModFile(m))
+ modRoots = append(modRoots, MainModules.ModRoot(m))
+ }
+ checkVendorConsistency(indexes, modFiles, modRoots)
+ rs.initVendor(vendorList)
+ }
+
if inWorkspaceMode() {
- // We don't need to do anything for vendor or update the mod file so
- // return early.
+ // We don't need to update the mod file so return early.
requirements = rs
return rs, nil
}
mainModule := MainModules.mustGetSingleMainModule()
- if cfg.BuildMod == "vendor" {
- readVendorList(mainModule)
- index := MainModules.Index(mainModule)
- modFile := MainModules.ModFile(mainModule)
- checkVendorConsistency(index, modFile)
- rs.initVendor(vendorList)
- }
-
if rs.hasRedundantRoot() {
// If any module path appears more than once in the roots, we know that the
// go.mod file needs to be updated even though we have not yet loaded any
@@ -1243,44 +1262,69 @@
var roots []module.Version
direct := map[string]bool{}
var pruning modPruning
- var goVersion, toolchain string
if inWorkspaceMode() {
pruning = workspace
roots = make([]module.Version, len(MainModules.Versions()), 2+len(MainModules.Versions()))
copy(roots, MainModules.Versions())
- goVersion = gover.FromGoWork(workFile)
+ goVersion := gover.FromGoWork(workFile)
+ var toolchain string
if workFile.Toolchain != nil {
toolchain = workFile.Toolchain.Name
}
+ roots = appendGoAndToolchainRoots(roots, goVersion, toolchain, direct)
} else {
pruning = pruningForGoVersion(MainModules.GoVersion())
if len(modFiles) != 1 {
panic(fmt.Errorf("requirementsFromModFiles called with %v modfiles outside workspace mode", len(modFiles)))
}
modFile := modFiles[0]
- roots = make([]module.Version, 0, 2+len(modFile.Require))
- mm := MainModules.mustGetSingleMainModule()
- for _, r := range modFile.Require {
- if index := MainModules.Index(mm); index != nil && index.exclude[r.Mod] {
- if cfg.BuildMod == "mod" {
- fmt.Fprintf(os.Stderr, "go: dropping requirement on excluded version %s %s\n", r.Mod.Path, r.Mod.Version)
- } else {
- fmt.Fprintf(os.Stderr, "go: ignoring requirement on excluded version %s %s\n", r.Mod.Path, r.Mod.Version)
- }
- continue
- }
-
- roots = append(roots, r.Mod)
- if !r.Indirect {
- direct[r.Mod.Path] = true
- }
- }
- goVersion = gover.FromGoMod(modFile)
- if modFile.Toolchain != nil {
- toolchain = modFile.Toolchain.Name
- }
+ roots, direct = rootsFromModFile(MainModules.mustGetSingleMainModule(), modFile, withToolchainRoot)
}
+ gover.ModSort(roots)
+ rs := newRequirements(pruning, roots, direct)
+ return rs
+}
+
+type addToolchainRoot bool
+
+const (
+ omitToolchainRoot addToolchainRoot = false
+ withToolchainRoot = true
+)
+
+func rootsFromModFile(m module.Version, modFile *modfile.File, addToolchainRoot addToolchainRoot) (roots []module.Version, direct map[string]bool) {
+ direct = make(map[string]bool)
+ padding := 2 // Add padding for the toolchain and go version, added upon return.
+ if !addToolchainRoot {
+ padding = 1
+ }
+ roots = make([]module.Version, 0, padding+len(modFile.Require))
+ for _, r := range modFile.Require {
+ if index := MainModules.Index(m); index != nil && index.exclude[r.Mod] {
+ if cfg.BuildMod == "mod" {
+ fmt.Fprintf(os.Stderr, "go: dropping requirement on excluded version %s %s\n", r.Mod.Path, r.Mod.Version)
+ } else {
+ fmt.Fprintf(os.Stderr, "go: ignoring requirement on excluded version %s %s\n", r.Mod.Path, r.Mod.Version)
+ }
+ continue
+ }
+
+ roots = append(roots, r.Mod)
+ if !r.Indirect {
+ direct[r.Mod.Path] = true
+ }
+ }
+ goVersion := gover.FromGoMod(modFile)
+ var toolchain string
+ if addToolchainRoot && modFile.Toolchain != nil {
+ toolchain = modFile.Toolchain.Name
+ }
+ roots = appendGoAndToolchainRoots(roots, goVersion, toolchain, direct)
+ return roots, direct
+}
+
+func appendGoAndToolchainRoots(roots []module.Version, goVersion, toolchain string, direct map[string]bool) []module.Version {
// Add explicit go and toolchain versions, inferring as needed.
roots = append(roots, module.Version{Path: "go", Version: goVersion})
direct["go"] = true // Every module directly uses the language and runtime.
@@ -1293,19 +1337,16 @@
// automatically if the 'go' version is changed so that it implies the exact
// same toolchain.
}
-
- gover.ModSort(roots)
- rs := newRequirements(pruning, roots, direct)
- return rs
+ return roots
}
// setDefaultBuildMod sets a default value for cfg.BuildMod if the -mod flag
// wasn't provided. setDefaultBuildMod may be called multiple times.
func setDefaultBuildMod() {
if cfg.BuildModExplicit {
- if inWorkspaceMode() && cfg.BuildMod != "readonly" {
- base.Fatalf("go: -mod may only be set to readonly when in workspace mode, but it is set to %q"+
- "\n\tRemove the -mod flag to use the default readonly value,"+
+ if inWorkspaceMode() && cfg.BuildMod != "readonly" && cfg.BuildMod != "vendor" {
+ base.Fatalf("go: -mod may only be set to readonly or vendor when in workspace mode, but it is set to %q"+
+ "\n\tRemove the -mod flag to use the default readonly value, "+
"\n\tor set GOWORK=off to disable workspace mode.", cfg.BuildMod)
}
// Don't override an explicit '-mod=' argument.
@@ -1327,7 +1368,7 @@
// to work in buggy situations.
cfg.BuildMod = "mod"
return
- case "mod vendor":
+ case "mod vendor", "work vendor":
cfg.BuildMod = "readonly"
return
}
@@ -1340,25 +1381,47 @@
return
}
- if len(modRoots) == 1 && !inWorkspaceMode() {
- index := MainModules.GetSingleIndexOrNil()
- if fi, err := fsys.Stat(filepath.Join(modRoots[0], "vendor")); err == nil && fi.IsDir() {
+ if len(modRoots) >= 1 {
+ var goVersion string
+ var versionSource string
+ if inWorkspaceMode() {
+ versionSource = "go.work"
+ if wfg := MainModules.WorkFile().Go; wfg != nil {
+ goVersion = wfg.Version
+ }
+ } else {
+ versionSource = "go.mod"
+ index := MainModules.GetSingleIndexOrNil()
+ if index != nil {
+ goVersion = index.goVersion
+ }
+ }
+ vendorDir := ""
+ if workFilePath != "" {
+ vendorDir = filepath.Join(filepath.Dir(workFilePath), "vendor")
+ } else {
+ if len(modRoots) != 1 {
+ panic(fmt.Errorf("outside workspace mode, but have %v modRoots", modRoots))
+ }
+ vendorDir = filepath.Join(modRoots[0], "vendor")
+ }
+ if fi, err := fsys.Stat(vendorDir); err == nil && fi.IsDir() {
modGo := "unspecified"
- if index != nil && index.goVersion != "" {
- if gover.Compare(index.goVersion, "1.14") >= 0 {
+ if goVersion != "" {
+ if gover.Compare(goVersion, "1.14") >= 0 {
// The Go version is at least 1.14, and a vendor directory exists.
// Set -mod=vendor by default.
cfg.BuildMod = "vendor"
- cfg.BuildModReason = "Go version in go.mod is at least 1.14 and vendor directory exists."
+ cfg.BuildModReason = "Go version in " + versionSource + " is at least 1.14 and vendor directory exists."
return
} else {
- modGo = index.goVersion
+ modGo = goVersion
}
}
// Since a vendor directory exists, we should record why we didn't use it.
// This message won't normally be shown, but it may appear with import errors.
- cfg.BuildModReason = fmt.Sprintf("Go version in go.mod is %s, so vendor directory was not used.", modGo)
+ cfg.BuildModReason = fmt.Sprintf("Go version in "+versionSource+" is %s, so vendor directory was not used.", modGo)
}
}
diff --git a/src/cmd/go/internal/modload/load.go b/src/cmd/go/internal/modload/load.go
index a993fe8..9b4cb19 100644
--- a/src/cmd/go/internal/modload/load.go
+++ b/src/cmd/go/internal/modload/load.go
@@ -571,7 +571,7 @@
return "", fmt.Errorf("without -mod=vendor, directory %s has no package path", absDir)
}
- readVendorList(mainModule)
+ readVendorList(VendorDir())
if _, ok := vendorPkgModule[pkg]; !ok {
return "", fmt.Errorf("directory %s is not a package listed in vendor/modules.txt", absDir)
}
@@ -1354,6 +1354,15 @@
// In workspace mode / workspace pruning mode, the roots are the main modules
// rather than the main module's direct dependencies. The check below on the selected
// roots does not apply.
+ if cfg.BuildMod == "vendor" {
+ // In workspace vendor mode, we don't need to load the requirements of the workspace
+ // modules' dependencies so the check below doesn't work. But that's okay, because
+ // checking whether modules are required directly for the purposes of pruning is
+ // less important in vendor mode: if we were able to load the package, we have
+ // everything we need to build the package, and dependencies' tests are pruned out
+ // of the vendor directory anyway.
+ continue
+ }
if mg, err := rs.Graph(ctx); err != nil {
return false, err
} else if _, ok := mg.RequiredBy(dep.mod); !ok {
diff --git a/src/cmd/go/internal/modload/modfile.go b/src/cmd/go/internal/modload/modfile.go
index d6c395f..8107b23 100644
--- a/src/cmd/go/internal/modload/modfile.go
+++ b/src/cmd/go/internal/modload/modfile.go
@@ -318,15 +318,22 @@
// module.Version is relative it's relative to the single main module outside
// workspace mode, or the workspace's directory in workspace mode.
func Replacement(mod module.Version) module.Version {
+ r, foundModRoot, _ := replacementFrom(mod)
+ return canonicalizeReplacePath(r, foundModRoot)
+}
+
+// replacementFrom returns the replacement for mod, if any, the modroot of the replacement if it appeared in a go.mod,
+// and the source of the replacement. The replacement is relative to the go.work or go.mod file it appears in.
+func replacementFrom(mod module.Version) (r module.Version, modroot string, fromFile string) {
foundFrom, found, foundModRoot := "", module.Version{}, ""
if MainModules == nil {
- return module.Version{}
+ return module.Version{}, "", ""
} else if MainModules.Contains(mod.Path) && mod.Version == "" {
// Don't replace the workspace version of the main module.
- return module.Version{}
+ return module.Version{}, "", ""
}
if _, r, ok := replacement(mod, MainModules.WorkFileReplaceMap()); ok {
- return r
+ return r, "", workFilePath
}
for _, v := range MainModules.Versions() {
if index := MainModules.Index(v); index != nil {
@@ -335,13 +342,13 @@
if foundModRoot != "" && foundFrom != from && found != r {
base.Errorf("conflicting replacements found for %v in workspace modules defined by %v and %v",
mod, modFilePath(foundModRoot), modFilePath(modRoot))
- return canonicalizeReplacePath(found, foundModRoot)
+ return found, foundModRoot, modFilePath(foundModRoot)
}
found, foundModRoot = r, modRoot
}
}
}
- return canonicalizeReplacePath(found, foundModRoot)
+ return found, foundModRoot, modFilePath(foundModRoot)
}
func replaceRelativeTo() string {
@@ -355,7 +362,7 @@
// are relative to the workspace directory (in workspace mode) or to the module's
// directory (in module mode, as they already are).
func canonicalizeReplacePath(r module.Version, modRoot string) module.Version {
- if filepath.IsAbs(r.Path) || r.Version != "" {
+ if filepath.IsAbs(r.Path) || r.Version != "" || modRoot == "" {
return r
}
workFilePath := WorkFilePath()
@@ -364,11 +371,11 @@
}
abs := filepath.Join(modRoot, r.Path)
if rel, err := filepath.Rel(filepath.Dir(workFilePath), abs); err == nil {
- return module.Version{Path: rel, Version: r.Version}
+ return module.Version{Path: ToDirectoryPath(rel), Version: r.Version}
}
// We couldn't make the version's path relative to the workspace's path,
// so just return the absolute path. It's the best we can do.
- return module.Version{Path: abs, Version: r.Version}
+ return module.Version{Path: ToDirectoryPath(abs), Version: r.Version}
}
// resolveReplacement returns the module actually used to load the source code
@@ -549,7 +556,7 @@
module: module.Version{Path: m.Path},
}
- readVendorList(MainModules.mustGetSingleMainModule())
+ readVendorList(VendorDir())
if vendorVersion[m.Path] != m.Version {
// This module is not vendored, so packages cannot be loaded from it and
// it cannot be relevant to the build.
diff --git a/src/cmd/go/internal/modload/search.go b/src/cmd/go/internal/modload/search.go
index cb03b69..aea301a 100644
--- a/src/cmd/go/internal/modload/search.go
+++ b/src/cmd/go/internal/modload/search.go
@@ -164,10 +164,13 @@
}
if cfg.BuildMod == "vendor" {
- mod := MainModules.mustGetSingleMainModule()
- if modRoot := MainModules.ModRoot(mod); modRoot != "" {
- walkPkgs(modRoot, MainModules.PathPrefix(mod), pruneGoMod|pruneVendor)
- walkPkgs(filepath.Join(modRoot, "vendor"), "", pruneVendor)
+ for _, mod := range MainModules.Versions() {
+ if modRoot := MainModules.ModRoot(mod); modRoot != "" {
+ walkPkgs(modRoot, MainModules.PathPrefix(mod), pruneGoMod|pruneVendor)
+ }
+ }
+ if HasModRoot() {
+ walkPkgs(VendorDir(), "", pruneVendor)
}
return
}
diff --git a/src/cmd/go/internal/modload/vendor.go b/src/cmd/go/internal/modload/vendor.go
index ffc79bb..b2cb441 100644
--- a/src/cmd/go/internal/modload/vendor.go
+++ b/src/cmd/go/internal/modload/vendor.go
@@ -37,13 +37,13 @@
}
// readVendorList reads the list of vendored modules from vendor/modules.txt.
-func readVendorList(mainModule module.Version) {
+func readVendorList(vendorDir string) {
vendorOnce.Do(func() {
vendorList = nil
vendorPkgModule = make(map[string]module.Version)
vendorVersion = make(map[string]string)
vendorMeta = make(map[module.Version]vendorMetadata)
- vendorFile := filepath.Join(MainModules.ModRoot(mainModule), "vendor/modules.txt")
+ vendorFile := filepath.Join(vendorDir, "modules.txt")
data, err := os.ReadFile(vendorFile)
if err != nil {
if !errors.Is(err, fs.ErrNotExist) {
@@ -140,15 +140,31 @@
// checkVendorConsistency verifies that the vendor/modules.txt file matches (if
// go 1.14) or at least does not contradict (go 1.13 or earlier) the
// requirements and replacements listed in the main module's go.mod file.
-func checkVendorConsistency(index *modFileIndex, modFile *modfile.File) {
- readVendorList(MainModules.mustGetSingleMainModule())
+func checkVendorConsistency(indexes []*modFileIndex, modFiles []*modfile.File, modRoots []string) {
+ // readVendorList only needs the main module to get the directory
+ // the vendor directory is in.
+ readVendorList(VendorDir())
+
+ if len(modFiles) < 1 {
+ // We should never get here if there are zero modfiles. Either
+ // we're in single module mode and there's a single module, or
+ // we're in workspace mode, and we fail earlier reporting that
+ // "no modules were found in the current workspace".
+ panic("checkVendorConsistency called with zero modfiles")
+ }
pre114 := false
- if gover.Compare(index.goVersion, "1.14") < 0 {
- // Go versions before 1.14 did not include enough information in
- // vendor/modules.txt to check for consistency.
- // If we know that we're on an earlier version, relax the consistency check.
- pre114 = true
+ if !inWorkspaceMode() { // workspace mode was added after Go 1.14
+ if len(indexes) != 1 {
+ panic(fmt.Errorf("not in workspace mode but number of indexes is %v, not 1", len(indexes)))
+ }
+ index := indexes[0]
+ if gover.Compare(index.goVersion, "1.14") < 0 {
+ // Go versions before 1.14 did not include enough information in
+ // vendor/modules.txt to check for consistency.
+ // If we know that we're on an earlier version, relax the consistency check.
+ pre114 = true
+ }
}
vendErrors := new(strings.Builder)
@@ -163,18 +179,20 @@
// Iterate over the Require directives in their original (not indexed) order
// so that the errors match the original file.
- for _, r := range modFile.Require {
- if !vendorMeta[r.Mod].Explicit {
- if pre114 {
- // Before 1.14, modules.txt did not indicate whether modules were listed
- // explicitly in the main module's go.mod file.
- // However, we can at least detect a version mismatch if packages were
- // vendored from a non-matching version.
- if vv, ok := vendorVersion[r.Mod.Path]; ok && vv != r.Mod.Version {
- vendErrorf(r.Mod, fmt.Sprintf("is explicitly required in go.mod, but vendor/modules.txt indicates %s@%s", r.Mod.Path, vv))
+ for _, modFile := range modFiles {
+ for _, r := range modFile.Require {
+ if !vendorMeta[r.Mod].Explicit {
+ if pre114 {
+ // Before 1.14, modules.txt did not indicate whether modules were listed
+ // explicitly in the main module's go.mod file.
+ // However, we can at least detect a version mismatch if packages were
+ // vendored from a non-matching version.
+ if vv, ok := vendorVersion[r.Mod.Path]; ok && vv != r.Mod.Version {
+ vendErrorf(r.Mod, fmt.Sprintf("is explicitly required in go.mod, but vendor/modules.txt indicates %s@%s", r.Mod.Path, vv))
+ }
+ } else {
+ vendErrorf(r.Mod, "is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt")
}
- } else {
- vendErrorf(r.Mod, "is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt")
}
}
}
@@ -190,42 +208,77 @@
// don't directly apply to any module in the vendor list, the replacement
// go.mod file can affect the selected versions of other (transitive)
// dependencies
- for _, r := range modFile.Replace {
- vr := vendorMeta[r.Old].Replacement
- if vr == (module.Version{}) {
- if pre114 && (r.Old.Version == "" || vendorVersion[r.Old.Path] != r.Old.Version) {
- // Before 1.14, modules.txt omitted wildcard replacements and
- // replacements for modules that did not have any packages to vendor.
- } else {
- vendErrorf(r.Old, "is replaced in go.mod, but not marked as replaced in vendor/modules.txt")
+ seenrep := make(map[module.Version]bool)
+ checkReplace := func(replaces []*modfile.Replace) {
+ for _, r := range replaces {
+ if seenrep[r.Old] {
+ continue // Don't print the same error more than once
}
- } else if vr != r.New {
- vendErrorf(r.Old, "is replaced by %s in go.mod, but marked as replaced by %s in vendor/modules.txt", describe(r.New), describe(vr))
+ seenrep[r.Old] = true
+ rNew, modRoot, replacementSource := replacementFrom(r.Old)
+ rNewCanonical := canonicalizeReplacePath(rNew, modRoot)
+ vr := vendorMeta[r.Old].Replacement
+ if vr == (module.Version{}) {
+ if rNewCanonical == (module.Version{}) {
+ // r.Old is not actually replaced. It might be a main module.
+ // Don't return an error.
+ } else if pre114 && (r.Old.Version == "" || vendorVersion[r.Old.Path] != r.Old.Version) {
+ // Before 1.14, modules.txt omitted wildcard replacements and
+ // replacements for modules that did not have any packages to vendor.
+ } else {
+ vendErrorf(r.Old, "is replaced in %s, but not marked as replaced in vendor/modules.txt", base.ShortPath(replacementSource))
+ }
+ } else if vr != rNewCanonical {
+ vendErrorf(r.Old, "is replaced by %s in %s, but marked as replaced by %s in vendor/modules.txt", describe(rNew), base.ShortPath(replacementSource), describe(vr))
+ }
}
}
+ for _, modFile := range modFiles {
+ checkReplace(modFile.Replace)
+ }
+ if MainModules.workFile != nil {
+ checkReplace(MainModules.workFile.Replace)
+ }
for _, mod := range vendorList {
meta := vendorMeta[mod]
if meta.Explicit {
- if _, inGoMod := index.require[mod]; !inGoMod {
- vendErrorf(mod, "is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod")
+ // in workspace mode, check that it's required by at least one of the main modules
+ var foundRequire bool
+ for _, index := range indexes {
+ if _, inGoMod := index.require[mod]; inGoMod {
+ foundRequire = true
+ }
}
+ if !foundRequire {
+ article := ""
+ if inWorkspaceMode() {
+ article = "a "
+ }
+ vendErrorf(mod, "is marked as explicit in vendor/modules.txt, but not explicitly required in %vgo.mod", article)
+ }
+
}
}
for _, mod := range vendorReplaced {
r := Replacement(mod)
+ replacementSource := "go.mod"
+ if inWorkspaceMode() {
+ replacementSource = "the workspace"
+ }
if r == (module.Version{}) {
- vendErrorf(mod, "is marked as replaced in vendor/modules.txt, but not replaced in go.mod")
+ vendErrorf(mod, "is marked as replaced in vendor/modules.txt, but not replaced in %s", replacementSource)
continue
}
- if meta := vendorMeta[mod]; r != meta.Replacement {
- vendErrorf(mod, "is marked as replaced by %s in vendor/modules.txt, but replaced by %s in go.mod", describe(meta.Replacement), describe(r))
- }
+ // If both replacements exist, we've already reported that they're different above.
}
if vendErrors.Len() > 0 {
- modRoot := MainModules.ModRoot(MainModules.mustGetSingleMainModule())
- base.Fatalf("go: inconsistent vendoring in %s:%s\n\n\tTo ignore the vendor directory, use -mod=readonly or -mod=mod.\n\tTo sync the vendor directory, run:\n\t\tgo mod vendor", modRoot, vendErrors)
+ subcmd := "mod"
+ if inWorkspaceMode() {
+ subcmd = "work"
+ }
+ base.Fatalf("go: inconsistent vendoring in %s:%s\n\n\tTo ignore the vendor directory, use -mod=readonly or -mod=mod.\n\tTo sync the vendor directory, run:\n\t\tgo %s vendor", filepath.Dir(VendorDir()), vendErrors, subcmd)
}
}
diff --git a/src/cmd/go/internal/workcmd/vendor.go b/src/cmd/go/internal/workcmd/vendor.go
new file mode 100644
index 0000000..f9f0cc0
--- /dev/null
+++ b/src/cmd/go/internal/workcmd/vendor.go
@@ -0,0 +1,55 @@
+// Copyright 2022 The Go Authors. All rights reserved.
+// Use of this source code is governed by a BSD-style
+// license that can be found in the LICENSE file.
+
+package workcmd
+
+import (
+ "cmd/go/internal/base"
+ "cmd/go/internal/cfg"
+ "cmd/go/internal/modcmd"
+ "cmd/go/internal/modload"
+ "context"
+)
+
+var cmdVendor = &base.Command{
+ UsageLine: "go work vendor [-e] [-v] [-o outdir]",
+ Short: "make vendored copy of dependencies",
+ Long: `
+Vendor resets the workspace's vendor directory to include all packages
+needed to build and test all the workspace's packages.
+It does not include test code for vendored packages.
+
+The -v flag causes vendor to print the names of vendored
+modules and packages to standard error.
+
+The -e flag causes vendor to attempt to proceed despite errors
+encountered while loading packages.
+
+The -o flag causes vendor to create the vendor directory at the given
+path instead of "vendor". The go command can only use a vendor directory
+named "vendor" within the module root directory, so this flag is
+primarily useful for other tools.`,
+
+ Run: runVendor,
+}
+
+var vendorE bool // if true, report errors but proceed anyway
+var vendorO string // if set, overrides the default output directory
+
+func init() {
+ cmdVendor.Flag.BoolVar(&cfg.BuildV, "v", false, "")
+ cmdVendor.Flag.BoolVar(&vendorE, "e", false, "")
+ cmdVendor.Flag.StringVar(&vendorO, "o", "", "")
+ base.AddChdirFlag(&cmdVendor.Flag)
+ base.AddModCommonFlags(&cmdVendor.Flag)
+}
+
+func runVendor(ctx context.Context, cmd *base.Command, args []string) {
+ modload.InitWorkfile()
+ if modload.WorkFilePath() == "" {
+ base.Fatalf("go: no go.work file found\n\t(run 'go work init' first or specify path using GOWORK environment variable)")
+ }
+
+ modcmd.RunVendor(ctx, vendorE, vendorO, args)
+}
diff --git a/src/cmd/go/internal/workcmd/work.go b/src/cmd/go/internal/workcmd/work.go
index c99cc2a..bfbed83 100644
--- a/src/cmd/go/internal/workcmd/work.go
+++ b/src/cmd/go/internal/workcmd/work.go
@@ -74,5 +74,6 @@
cmdInit,
cmdSync,
cmdUse,
+ cmdVendor,
},
}
diff --git a/src/cmd/go/testdata/script/work.txt b/src/cmd/go/testdata/script/work.txt
index e229ab6..69391ef 100644
--- a/src/cmd/go/testdata/script/work.txt
+++ b/src/cmd/go/testdata/script/work.txt
@@ -32,7 +32,7 @@
# -mod can only be set to readonly in workspace mode
go list -mod=readonly all
! go list -mod=mod all
-stderr '^go: -mod may only be set to readonly when in workspace mode'
+stderr '^go: -mod may only be set to readonly or vendor when in workspace mode'
env GOWORK=off
go list -mod=mod all
env GOWORK=
diff --git a/src/cmd/go/testdata/script/work_vendor_empty.txt b/src/cmd/go/testdata/script/work_vendor_empty.txt
new file mode 100644
index 0000000..3c0c7ed
--- /dev/null
+++ b/src/cmd/go/testdata/script/work_vendor_empty.txt
@@ -0,0 +1,16 @@
+go work vendor
+stderr 'go: no dependencies to vendor'
+! exists vendor/modules.txt
+! go list .
+stderr 'go: no modules were found in the current workspace'
+mkdir vendor
+mv bad_modules.txt vendor/modules.txt
+! go list .
+stderr 'go: no modules were found in the current workspace'
+
+-- bad_modules.txt --
+# a/module
+a/package
+-- go.work --
+go 1.21
+
diff --git a/src/cmd/go/testdata/script/work_vendor_main_module_replaced.txt b/src/cmd/go/testdata/script/work_vendor_main_module_replaced.txt
new file mode 100644
index 0000000..70446c7
--- /dev/null
+++ b/src/cmd/go/testdata/script/work_vendor_main_module_replaced.txt
@@ -0,0 +1,46 @@
+# This is a test that if one of the main modules replaces the other
+# the vendor consistency checks still pass. The replacement is ignored
+# because it is of a main module, but it is still recorded in
+# vendor/modules.txt.
+
+go work vendor
+go list all # make sure the consistency checks pass
+! stderr .
+
+# Removing the replace causes consistency checks to fail
+cp a_go_mod_no_replace a/go.mod
+! go list all # consistency checks fail
+stderr 'example.com/b...@v0.0.0: is marked as replaced in vendor/modules.txt, but not replaced in the workspace'
+
+
+-- a_go_mod_no_replace --
+module example.com/a
+
+go 1.21
+
+require example.com/b v0.0.0
+-- go.work --
+go 1.21
+
+use (
+ a
+ b
+)
+-- a/go.mod --
+module example.com/a
+
+go 1.21
+
+require example.com/b v0.0.0
+
+replace example.com/b => ../b
+-- a/a.go --
+package a
+
+import _ "example.com/b"
+-- b/go.mod --
+module example.com/b
+
+go 1.21
+-- b/b.go --
+package b
\ No newline at end of file
diff --git a/src/cmd/go/testdata/script/work_vendor_modules_txt_consistent.txt b/src/cmd/go/testdata/script/work_vendor_modules_txt_consistent.txt
new file mode 100644
index 0000000..038e1a5
--- /dev/null
+++ b/src/cmd/go/testdata/script/work_vendor_modules_txt_consistent.txt
@@ -0,0 +1,135 @@
+go work vendor
+cmp modules.txt.want vendor/modules.txt
+go list example.com/a example.com/b
+
+# Module required in go.mod but not marked explicit in modules.txt
+cp modules.txt.required_but_not_explicit vendor/modules.txt
+! go list example.com/a example.com/b
+cmpenv stderr required_but_not_explicit_error.txt
+
+# Replacement in go.mod but no replacement in modules.txt
+cp modules.txt.missing_replacement vendor/modules.txt
+! go list example.com/a example.com/b
+cmpenv stderr missing_replacement_error.txt
+
+# Replacement in go.mod but different replacement target in modules.txt
+cp modules.txt.different_replacement vendor/modules.txt
+! go list example.com/a example.com/b
+cmpenv stderr different_replacement_error.txt
+
+# Module marked explicit in modules.txt but not required in go.mod
+cp modules.txt.extra_explicit vendor/modules.txt
+! go list example.com/a example.com/b
+cmpenv stderr extra_explicit_error.txt
+
+# Replacement in modules.txt but not in go.mod
+cp modules.txt.extra_replacement vendor/modules.txt
+! go list example.com/a example.com/b
+cmpenv stderr extra_replacement_error.txt
+
+-- modules.txt.want --
+# example.com/p v1.0.0 => ./p
+## explicit; go 1.21
+# example.com/q v1.0.0 => ./q
+## explicit; go 1.21
+-- modules.txt.required_but_not_explicit --
+# example.com/p v1.0.0 => ./p
+## go 1.21
+# example.com/q v1.0.0 => ./q
+## explicit; go 1.21
+-- required_but_not_explicit_error.txt --
+go: inconsistent vendoring in $GOPATH${/}src:
+ example.com/p...@v1.0.0: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
+
+ To ignore the vendor directory, use -mod=readonly or -mod=mod.
+ To sync the vendor directory, run:
+ go work vendor
+-- modules.txt.missing_replacement --
+# example.com/p v1.0.0
+## explicit; go 1.21
+# example.com/q v1.0.0 => ./q
+## explicit; go 1.21
+-- missing_replacement_error.txt --
+go: inconsistent vendoring in $GOPATH${/}src:
+ example.com/p...@v1.0.0: is replaced in a${/}go.mod, but not marked as replaced in vendor/modules.txt
+
+ To ignore the vendor directory, use -mod=readonly or -mod=mod.
+ To sync the vendor directory, run:
+ go work vendor
+-- modules.txt.different_replacement --
+# example.com/p v1.0.0 => ./r
+## explicit; go 1.21
+# example.com/q v1.0.0 => ./q
+## explicit; go 1.21
+-- different_replacement_error.txt --
+go: inconsistent vendoring in $GOPATH${/}src:
+ example.com/p...@v1.0.0: is replaced by ../p in a${/}go.mod, but marked as replaced by ./r in vendor/modules.txt
+
+ To ignore the vendor directory, use -mod=readonly or -mod=mod.
+ To sync the vendor directory, run:
+ go work vendor
+-- modules.txt.extra_explicit --
+# example.com/p v1.0.0 => ./p
+## explicit; go 1.21
+# example.com/q v1.0.0 => ./q
+## explicit; go 1.21
+# example.com/r v1.0.0
+example.com/r
+## explicit; go 1.21
+-- extra_explicit_error.txt --
+go: inconsistent vendoring in $GOPATH${/}src:
+ example.com/r...@v1.0.0: is marked as explicit in vendor/modules.txt, but not explicitly required in a go.mod
+
+ To ignore the vendor directory, use -mod=readonly or -mod=mod.
+ To sync the vendor directory, run:
+ go work vendor
+-- modules.txt.extra_replacement --
+# example.com/p v1.0.0 => ./p
+## explicit; go 1.21
+# example.com/q v1.0.0 => ./q
+## explicit; go 1.21
+# example.com/r v1.0.0 => ./r
+example.com/r
+## go 1.21
+-- extra_replacement_error.txt --
+go: inconsistent vendoring in $GOPATH${/}src:
+ example.com/r...@v1.0.0: is marked as replaced in vendor/modules.txt, but not replaced in the workspace
+
+ To ignore the vendor directory, use -mod=readonly or -mod=mod.
+ To sync the vendor directory, run:
+ go work vendor
+-- go.work --
+go 1.21
+
+use (
+ ./a
+ ./b
+)
+-- a/go.mod --
+module example.com/a
+
+go 1.21
+
+require example.com/p v1.0.0
+
+replace example.com/p v1.0.0 => ../p
+-- a/a.go --
+package p
+-- b/go.mod --
+module example.com/b
+
+go 1.21
+
+require example.com/q v1.0.0
+
+replace example.com/q v1.0.0 => ../q
+-- b/b.go --
+package b
+-- p/go.mod --
+module example.com/p
+
+go 1.21
+-- q/go.mod --
+module example.com/q
+
+go 1.21
diff --git a/src/cmd/go/testdata/script/work_vendor_prune.txt b/src/cmd/go/testdata/script/work_vendor_prune.txt
new file mode 100644
index 0000000..5972cc7
--- /dev/null
+++ b/src/cmd/go/testdata/script/work_vendor_prune.txt
@@ -0,0 +1,115 @@
+# This test exercises that vendoring works properly using the workspace in the
+# the work_prune test case.
+
+go work vendor
+cmp vendor/modules.txt modules.txt.want
+cmp vendor/example.com/b/b.go b/b.go
+cmp vendor/example.com/q/q.go q1_1_0/q.go
+go list -m -f '{{.Version}}' example.com/q
+stdout '^v1.1.0$'
+
+go list -f '{{.Dir}}' example.com/q
+stdout $GOPATH[\\/]src[\\/]vendor[\\/]example.com[\\/]q
+go list -f '{{.Dir}}' example.com/b
+stdout $GOPATH[\\/]src[\\/]vendor[\\/]example.com[\\/]b
+
+[short] skip
+
+rm b
+rm q1_0_0
+rm q1_1_0
+go run example.com/p
+stdout 'version 1.1.0'
+
+-- modules.txt.want --
+# example.com/b v1.0.0 => ./b
+## explicit; go 1.18
+example.com/b
+# example.com/q v1.0.0 => ./q1_0_0
+## explicit; go 1.18
+# example.com/q v1.1.0 => ./q1_1_0
+## go 1.18
+example.com/q
+-- go.work --
+go 1.18
+
+use (
+ ./a
+ ./p
+)
+-- a/go.mod --
+module example.com/a
+
+go 1.18
+
+require example.com/b v1.0.0
+
+replace example.com/b v1.0.0 => ../b
+-- a/foo.go --
+package main
+
+import "example.com/b"
+
+func main() {
+ b.B()
+}
+-- b/go.mod --
+module example.com/b
+
+go 1.18
+
+require example.com/q v1.1.0
+-- b/b.go --
+package b
+
+func B() {
+}
+-- b/b_test.go --
+package b
+
+import "example.com/q"
+
+func TestB() {
+ q.PrintVersion()
+}
+-- p/go.mod --
+module example.com/p
+
+go 1.18
+
+require example.com/q v1.0.0
+
+replace example.com/q v1.0.0 => ../q1_0_0
+replace example.com/q v1.1.0 => ../q1_1_0
+-- p/main.go --
+package main
+
+import "example.com/q"
+
+func main() {
+ q.PrintVersion()
+}
+-- q1_0_0/go.mod --
+module example.com/q
+
+go 1.18
+-- q1_0_0/q.go --
+package q
+
+import "fmt"
+
+func PrintVersion() {
+ fmt.Println("version 1.0.0")
+}
+-- q1_1_0/go.mod --
+module example.com/q
+
+go 1.18
+-- q1_1_0/q.go --
+package q
+
+import "fmt"
+
+func PrintVersion() {
+ fmt.Println("version 1.1.0")
+}
diff --git a/src/cmd/go/testdata/script/work_vendor_prune_all.txt b/src/cmd/go/testdata/script/work_vendor_prune_all.txt
new file mode 100644
index 0000000..b004afd
--- /dev/null
+++ b/src/cmd/go/testdata/script/work_vendor_prune_all.txt
@@ -0,0 +1,200 @@
+# This test exercises that vendoring works properly using the workspace in the
+# the work_prune test case.
+
+go work vendor
+cmp vendor/modules.txt modules.txt.want
+go list -f '{{with .Module}}{{.Path}}@{{.Version}}{{end}}' all
+cmp stdout want_versions
+
+go list -f '{{.Dir}}' example.com/q
+stdout $GOPATH[\\/]src[\\/]vendor[\\/]example.com[\\/]q
+go list -f '{{.Dir}}' example.com/b
+stdout $GOPATH[\\/]src[\\/]vendor[\\/]example.com[\\/]b
+go list -f '{{.Dir}}' example.com/w
+stdout $GOPATH[\\/]src[\\/]vendor[\\/]example.com[\\/]w
+go list -f '{{.Dir}}' example.com/z
+stdout $GOPATH[\\/]src[\\/]vendor[\\/]example.com[\\/]z
+
+cmp $GOPATH/src/vendor/example.com/q/q.go q1_1_0/q.go
+
+-- modules.txt.want --
+# example.com/b v1.0.0 => ./b
+## explicit; go 1.18
+example.com/b
+# example.com/q v1.0.0 => ./q1_0_0
+## explicit; go 1.18
+# example.com/q v1.1.0 => ./q1_1_0
+## go 1.18
+example.com/q
+# example.com/w v1.0.0 => ./w
+## go 1.18
+example.com/w
+# example.com/z v1.0.0 => ./z1_0_0
+## explicit; go 1.18
+# example.com/z v1.1.0 => ./z1_1_0
+## go 1.18
+example.com/z
+# example.com/q v1.0.5 => ./q1_0_5
+# example.com/r v1.0.0 => ./r
+# example.com/x v1.0.0 => ./x
+# example.com/y v1.0.0 => ./y
+-- want_versions --
+example.com/a@
+example.com/b...@v1.0.0
+example.com/p@
+example.com/q...@v1.1.0
+example.com/w...@v1.0.0
+example.com/z...@v1.1.0
+-- go.work --
+go 1.18
+
+use (
+ ./a
+ ./p
+)
+
+replace example.com/b v1.0.0 => ./b
+replace example.com/q v1.0.0 => ./q1_0_0
+replace example.com/q v1.0.5 => ./q1_0_5
+replace example.com/q v1.1.0 => ./q1_1_0
+replace example.com/r v1.0.0 => ./r
+replace example.com/w v1.0.0 => ./w
+replace example.com/x v1.0.0 => ./x
+replace example.com/y v1.0.0 => ./y
+replace example.com/z v1.0.0 => ./z1_0_0
+replace example.com/z v1.1.0 => ./z1_1_0
+
+-- a/go.mod --
+module example.com/a
+
+go 1.18
+
+require example.com/b v1.0.0
+require example.com/z v1.0.0
+-- a/foo.go --
+package main
+
+import "example.com/b"
+
+func main() {
+ b.B()
+}
+-- b/go.mod --
+module example.com/b
+
+go 1.18
+
+require example.com/q v1.1.0
+-- b/b.go --
+package b
+
+func B() {
+}
+-- p/go.mod --
+module example.com/p
+
+go 1.18
+
+require example.com/q v1.0.0
+
+replace example.com/q v1.0.0 => ../q1_0_0
+replace example.com/q v1.1.0 => ../q1_1_0
+-- p/main.go --
+package main
+
+import "example.com/q"
+
+func main() {
+ q.PrintVersion()
+}
+-- q1_0_0/go.mod --
+module example.com/q
+
+go 1.18
+-- q1_0_0/q.go --
+package q
+
+import "fmt"
+
+func PrintVersion() {
+ fmt.Println("version 1.0.0")
+}
+-- q1_0_5/go.mod --
+module example.com/q
+
+go 1.18
+
+require example.com/r v1.0.0
+-- q1_0_5/q.go --
+package q
+
+import _ "example.com/r"
+-- q1_1_0/go.mod --
+module example.com/q
+
+require example.com/w v1.0.0
+require example.com/z v1.1.0
+
+go 1.18
+-- q1_1_0/q.go --
+package q
+
+import _ "example.com/w"
+import _ "example.com/z"
+
+import "fmt"
+
+func PrintVersion() {
+ fmt.Println("version 1.1.0")
+}
+-- r/go.mod --
+module example.com/r
+
+go 1.18
+
+require example.com/r v1.0.0
+-- r/r.go --
+package r
+-- w/go.mod --
+module example.com/w
+
+go 1.18
+
+require example.com/x v1.0.0
+-- w/w.go --
+package w
+-- w/w_test.go --
+package w
+
+import _ "example.com/x"
+-- x/go.mod --
+module example.com/x
+
+go 1.18
+-- x/x.go --
+package x
+-- x/x_test.go --
+package x
+import _ "example.com/y"
+-- y/go.mod --
+module example.com/y
+
+go 1.18
+-- y/y.go --
+package y
+-- z1_0_0/go.mod --
+module example.com/z
+
+go 1.18
+
+require example.com/q v1.0.5
+-- z1_0_0/z.go --
+package z
+
+import _ "example.com/q"
+-- z1_1_0/go.mod --
+module example.com/z
+
+go 1.18
+-- z1_1_0/z.go --
+package z
To view, visit change 495801. To unsubscribe, or for help writing mail filters, visit settings.