[go] cmd/go: download fewer dependencies in 'go mod download'
225 views
Skip to first unread message
Bryan C. Mills (Gerrit)
Nov 4, 2021, 4:42:43 PM11/4/21
to Bryan C. Mills, goph...@pubsubhelper.golang.org, golang-...@googlegroups.com, Go Bot, Michael Matloob, Zik Aeroh, golang-co...@googlegroups.com
Bryan C. Mills submitted this change.
3 is the latest approved patch-set.
The change was submitted with unreviewed changes in the following files:
```
The name of the file: src/cmd/go/testdata/script/mod_download.txt
Insertions: 5, Deletions: 12.
@@ -133,11 +133,11 @@
# (without arguments) should only download the modules explicitly required in
# the go.mod file, not (presumed-irrelevant) transitive dependencies.
#
-# If the go.mod file is inconsistent, the version downloaded should be the
-# one from the go.mod file, not the selected version from the broader graph.
-# (If the go.mod file is inconsistent, the build can only succeed with the
-# default -mod=readonly if lazy loading prevents the graph from being loaded —
-# and in that case it will use the explicit version.)
+# (If the go.mod file is inconsistent, the version downloaded should be the
+# selected version from the broader graph, but the go.mod file will also be
+# updated to list the correct versions. If at some point we change 'go mod
+# download' to stop updating for consistency, then it should fail if the
+# requirements are inconsistent.)
rm go.sum
cp go.mod.orig go.mod
@@ -212,10 +212,3 @@
golang.org/x/text v0.0.0-20170915032832-14c0d48ead0c/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
rsc.io/quote v1.5.2/go.mod h1:LzX7hefJvL54yjefDEDHNONDjII0t9xZLPXsUe+TKr0=
rsc.io/sampler v1.3.0/go.mod h1:T1hPZKmBbMNahiBKFy5HrXp6adAjACjK9JXDnKaTXpA=
--- narrow/go.mod.orig --
-module m
-
-go 1.17
-
-require rsc.io/quote v1.5.2
--- narrow/go.sum.narrow --
```
```
The name of the file: doc/go1.18.html
Insertions: 4, Deletions: 2.
@@ -67,12 +67,13 @@
<p><!-- golang.org/issue/37475 -->
The <code>go</code> command now embeds version control information in
- binaries including the currently checked-out revision and a flag indicating
- whether edited or untracked files are present. Version control information
- is embedded if the <code>go</code> command is invoked in a directory within a
- Git or Mercurial repository, and the <code>main</code> package and its
- containing main module are in the same repository. This information may be
- omitted using the flag <code>-buildvcs=false</code>.
+ binaries including the currently checked-out revision, commit time, and a
+ flag indicating whether edited or untracked files are present. Version
+ control information is embedded if the <code>go</code> command is invoked in
+ a directory within a Git, Mercurial, Fossil, or Bazaar repository, and the
+ <code>main</code> package and its containing main module are in the same
+ repository. This information may be omitted using the flag
+ <code>-buildvcs=false</code>.
</p>
<p><!-- golang.org/issue/37475 -->
@@ -95,8 +96,10 @@
or higher, <code>go</code> <code>mod</code> <code>download</code> without
arguments now downloads source code for only the modules
explicitly <a href="/ref/mod#go-mod-file-require">required</a> in the main
- module's <code>go.mod</code> file. To also download source code for transitive
- dependencies, use
+ module's <code>go.mod</code> file. (In a <code>go</code> <code>1.17</code> or
+ higher module, that set already includes all dependencies needed to build the
+ packages and tests in the main module.)
+ To also download source code for transitive dependencies, use
<code>go</code> <code>mod</code> <code>download</code> <code>all</code>.
</p>
@@ -109,7 +112,7 @@
<p><!-- https://golang.org/issue/43566 -->
<code>gofmt</code> now reads and formats input files concurrently, with a
memory limit proportional to <code>GOMAXPROCS</code>. On a machine with
- multiple CPUs, gofmt should now be significantly faster.
+ multiple CPUs, <code>gofmt</code> should now be significantly faster.
</p>
@@ -133,10 +136,33 @@
<h2 id="library">Core library</h2>
+<h3>TODO</h3>
<p>
TODO: complete this section
</p>
+<h3 id="netip">New <code>net/netip</code> package</h3>
+<p>
+ The new <a href="/pkg/net/netip/"><code>net/netip</code></a>
+ package defines a new IP address type, <a href="/pkg/net/netip/#Addr"><code>Addr</code></a>.
+ Compared to the existing
+ <a href="/pkg/net/#IP"><code>net.IP</code></a> type, the <code>netip.Addr</code> type takes less
+ memory, is immutable, and is comparable so it supports <code>==</code>
+ and can be used as a map key.
+</p>
+<p>
+ In addition to <code>Addr</code>, the package defines
+ <a href="/pkg/net/netip/#AddrPort"><code>AddrPort</code></a>, representing
+ an IP and port, and
+ <a href="/pkg/net/netip/#Prefix"><code>Prefix</code></a>, representing
+ a network CIDR prefix.
+</p>
+<p>
+ The <code>net</code> package now has methods to send and receive UDP packets
+ using <code>netip.Addr</code> values instead of the relatively heavy
+ <code>*net.UDPAddr</code> values.
+</p>
+
<h3 id="minor_library_changes">Minor changes to the library</h3>
<p>
```
Approvals:
Michael Matloob: Looks good to me, approved
Bryan C. Mills: Trusted; Run TryBots
Go Bot: TryBots succeeded
cmd/go: download fewer dependencies in 'go mod download'
In modules that specify 'go 1.17' or higher, the go.mod file
explicitly requires modules for all packages transitively imported by
the main module. Users tend to use 'go mod download' to prepare for
testing the main module itself, so we should only download those
relevant modules.
In 'go 1.16' and earlier modules, we continue to download all modules
in the module graph (because we cannot in general tell which ones are
relevant without loading the full package import graph).
'go mod download all' continues to download every module in
'go list all', as it did before.
Fixes #44435
Change-Id: I3f286c0e2549d6688b3832ff116e6cd77a19401c
Reviewed-on: https://go-review.googlesource.com/c/go/+/357310
Trust: Bryan C. Mills <bcm...@google.com>
Run-TryBot: Bryan C. Mills <bcm...@google.com>
TryBot-Result: Go Bot <go...@golang.org>
Reviewed-by: Michael Matloob <mat...@golang.org>
---
M src/cmd/go/internal/modcmd/download.go
M src/cmd/go/testdata/script/mod_download.txt
M src/cmd/go/internal/modload/buildlist.go
M doc/go1.18.html
M src/cmd/go/alldocs.go
M src/cmd/go/internal/modload/modfile.go
6 files changed, 142 insertions(+), 20 deletions(-)
diff --git a/doc/go1.18.html b/doc/go1.18.html
index 171cc3c..385a1ae 100644
--- a/doc/go1.18.html
+++ b/doc/go1.18.html
@@ -90,6 +90,19 @@
package.
</p>
+<p><!-- https://golang.org/issue/44435 -->
+ If the main module's <code>go.mod</code> file
+ specifies <a href="/ref/mod#go-mod-file-go"><code>go</code> <code>1.17</code></a>
+ or higher, <code>go</code> <code>mod</code> <code>download</code> without
+ arguments now downloads source code for only the modules
+ explicitly <a href="/ref/mod#go-mod-file-require">required</a> in the main
+ module's <code>go.mod</code> file. (In a <code>go</code> <code>1.17</code> or
+ higher module, that set already includes all dependencies needed to build the
+ packages and tests in the main module.)
+ To also download source code for transitive dependencies, use
+ <code>go</code> <code>mod</code> <code>download</code> <code>all</code>.
+</p>
+
<p>
TODO: complete this section, or delete if not needed
</p>
diff --git a/src/cmd/go/alldocs.go b/src/cmd/go/alldocs.go
index 0dcb317..4420073 100644
--- a/src/cmd/go/alldocs.go
+++ b/src/cmd/go/alldocs.go
@@ -1074,8 +1074,11 @@
//
// Download downloads the named modules, which can be module patterns selecting
// dependencies of the main module or module queries of the form path@version.
-// With no arguments, download applies to all dependencies of the main module
-// (equivalent to 'go mod download all').
+//
+// With no arguments, download applies to the modules needed to build and test
+// the packages in the main module: the modules explicitly required by the main
+// module if it is at 'go 1.17' or higher, or all transitively-required modules
+// if at 'go 1.16' or lower.
//
// The go command will automatically download modules as needed during ordinary
// execution. The "go mod download" command is useful mainly for pre-filling
diff --git a/src/cmd/go/internal/modcmd/download.go b/src/cmd/go/internal/modcmd/download.go
index 5ea56c3..f252133 100644
--- a/src/cmd/go/internal/modcmd/download.go
+++ b/src/cmd/go/internal/modcmd/download.go
@@ -16,6 +16,7 @@
"cmd/go/internal/modload"
"golang.org/x/mod/module"
+ "golang.org/x/mod/semver"
)
var cmdDownload = &base.Command{
@@ -24,8 +25,11 @@
Long: `
Download downloads the named modules, which can be module patterns selecting
dependencies of the main module or module queries of the form path@version.
-With no arguments, download applies to all dependencies of the main module
-(equivalent to 'go mod download all').
+
+With no arguments, download applies to the modules needed to build and test
+the packages in the main module: the modules explicitly required by the main
+module if it is at 'go 1.17' or higher, or all transitively-required modules
+if at 'go 1.16' or lower.
The go command will automatically download modules as needed during ordinary
execution. The "go mod download" command is useful mainly for pre-filling
@@ -87,13 +91,8 @@
// Check whether modules are enabled and whether we're in a module.
modload.ForceUseModules = true
modload.ExplicitWriteGoMod = true
- if !modload.HasModRoot() && len(args) == 0 {
- base.Fatalf("go: no modules specified (see 'go help mod download')")
- }
haveExplicitArgs := len(args) > 0
- if !haveExplicitArgs {
- args = []string{"all"}
- }
+
if modload.HasModRoot() {
modload.LoadModFile(ctx) // to fill MainModules
@@ -102,16 +101,50 @@
}
mainModule := modload.MainModules.Versions()[0]
- targetAtUpgrade := mainModule.Path + "@upgrade"
- targetAtPatch := mainModule.Path + "@patch"
- for _, arg := range args {
- switch arg {
- case mainModule.Path, targetAtUpgrade, targetAtPatch:
- os.Stderr.WriteString("go: skipping download of " + arg + " that resolves to the main module\n")
+ if haveExplicitArgs {
+ targetAtUpgrade := mainModule.Path + "@upgrade"
+ targetAtPatch := mainModule.Path + "@patch"
+ for _, arg := range args {
+ switch arg {
+ case mainModule.Path, targetAtUpgrade, targetAtPatch:
+ os.Stderr.WriteString("go: skipping download of " + arg + " that resolves to the main module\n")
+ }
+ }
+ } else {
+ modFile := modload.MainModules.ModFile(mainModule)
+ if modFile.Go == nil || semver.Compare("v"+modFile.Go.Version, modload.ExplicitIndirectVersionV) < 0 {
+ if len(modFile.Require) > 0 {
+ args = []string{"all"}
+ }
+ } else {
+ // As of Go 1.17, the go.mod file explicitly requires every module
+ // that provides any package imported by the main module.
+ // 'go mod download' is typically run before testing packages in the
+ // main module, so by default we shouldn't download the others
+ // (which are presumed irrelevant to the packages in the main module).
+ // See https://golang.org/issue/44435.
+ //
+ // However, we also need to load the full module graph, to ensure that
+ // we have downloaded enough of the module graph to run 'go list all',
+ // 'go mod graph', and similar commands.
+ _ = modload.LoadModGraph(ctx, "")
+
+ for _, m := range modFile.Require {
+ args = append(args, m.Mod.Path)
+ }
}
}
}
+ if len(args) == 0 {
+ if modload.HasModRoot() {
+ os.Stderr.WriteString("go: no module dependencies to download\n")
+ } else {
+ base.Errorf("go: no modules specified (see 'go help mod download')")
+ }
+ base.Exit()
+ }
+
downloadModule := func(m *moduleJSON) {
var err error
m.Info, err = modfetch.InfoFile(m.Path, m.Version)
diff --git a/src/cmd/go/internal/modload/buildlist.go b/src/cmd/go/internal/modload/buildlist.go
index 4634ad0..27cab0b 100644
--- a/src/cmd/go/internal/modload/buildlist.go
+++ b/src/cmd/go/internal/modload/buildlist.go
@@ -236,7 +236,7 @@
// A ModuleGraph represents the complete graph of module dependencies
// of a main module.
//
-// If the main module is lazily loaded, the graph does not include
+// If the main module supports module graph pruning, the graph does not include
// transitive dependencies of non-root (implicit) dependencies.
type ModuleGraph struct {
g *mvs.Graph
diff --git a/src/cmd/go/internal/modload/modfile.go b/src/cmd/go/internal/modload/modfile.go
index 87e8a5e..1672d56 100644
--- a/src/cmd/go/internal/modload/modfile.go
+++ b/src/cmd/go/internal/modload/modfile.go
@@ -33,13 +33,13 @@
// tests outside of the main module.
narrowAllVersionV = "v1.16"
- // explicitIndirectVersionV is the Go version (plus leading "v") at which a
+ // ExplicitIndirectVersionV is the Go version (plus leading "v") at which a
// module's go.mod file is expected to list explicit requirements on every
// module that provides any package transitively imported by that module.
//
// Other indirect dependencies of such a module can be safely pruned out of
// the module graph; see https://golang.org/ref/mod#graph-pruning.
- explicitIndirectVersionV = "v1.17"
+ ExplicitIndirectVersionV = "v1.17"
// separateIndirectVersionV is the Go version (plus leading "v") at which
// "// indirect" dependencies are added in a block separate from the direct
@@ -123,7 +123,7 @@
)
func pruningForGoVersion(goVersion string) modPruning {
- if semver.Compare("v"+goVersion, explicitIndirectVersionV) < 0 {
+ if semver.Compare("v"+goVersion, ExplicitIndirectVersionV) < 0 {
// The go.mod file does not duplicate relevant information about transitive
// dependencies, so they cannot be pruned out.
return unpruned
diff --git a/src/cmd/go/testdata/script/mod_download.txt b/src/cmd/go/testdata/script/mod_download.txt
index 89e58a2..154e683 100644
--- a/src/cmd/go/testdata/script/mod_download.txt
+++ b/src/cmd/go/testdata/script/mod_download.txt
@@ -128,6 +128,50 @@
go mod download all
cmp go.mod.update go.mod
grep '^rsc.io/sampler v1.3.0 ' go.sum
+
+# https://golang.org/issue/44435: At go 1.17 or higher, 'go mod download'
+# (without arguments) should only download the modules explicitly required in
+# the go.mod file, not (presumed-irrelevant) transitive dependencies.
+#
+# (If the go.mod file is inconsistent, the version downloaded should be the
+# selected version from the broader graph, but the go.mod file will also be
+# updated to list the correct versions. If at some point we change 'go mod
+# download' to stop updating for consistency, then it should fail if the
+# requirements are inconsistent.)
+
+rm go.sum
+cp go.mod.orig go.mod
+go mod edit -go=1.17
+cp go.mod.update go.mod.go117
+go mod edit -go=1.17 go.mod.go117
+
+go clean -modcache
+go mod download
+cmp go.mod go.mod.go117
+
+go list -e -m all
+stdout '^rsc.io/quote v1.5.2$'
+exists $GOPATH/pkg/mod/cache/download/rsc.io/quote/@v/v1.5.2.zip
+stdout '^rsc.io/sampler v1.3.0$'
+! exists $GOPATH/pkg/mod/cache/download/rsc.io/sampler/@v/v1.2.1.zip
+exists $GOPATH/pkg/mod/cache/download/rsc.io/sampler/@v/v1.3.0.zip
+stdout '^golang\.org/x/text v0.0.0-20170915032832-14c0d48ead0c$'
+! exists $GOPATH/pkg/mod/cache/download/golang.org/x/text/@v/v0.0.0-20170915032832-14c0d48ead0c.zip
+cmp go.mod go.mod.go117
+
+# However, 'go mod download all' continues to download the selected version
+# of every module reported by 'go list -m all'.
+
+cp go.mod.orig go.mod
+go mod edit -go=1.17
+go clean -modcache
+go mod download all
+exists $GOPATH/pkg/mod/cache/download/rsc.io/quote/@v/v1.5.2.zip
+! exists $GOPATH/pkg/mod/cache/download/rsc.io/sampler/@v/v1.2.1.zip
+exists $GOPATH/pkg/mod/cache/download/rsc.io/sampler/@v/v1.3.0.zip
+exists $GOPATH/pkg/mod/cache/download/golang.org/x/text/@v/v0.0.0-20170915032832-14c0d48ead0c.zip
+cmp go.mod go.mod.go117
+
cd ..
# allow go mod download without go.mod
To view, visit change 357310. To unsubscribe, or for help writing mail filters, visit settings.
Reply all
Reply to author
Forward
0 new messages