Care and feeding of standard-library modules

[Bryan C. Mills]

Hey, folks!

If you've been following #30228, you'll notice that we're about to turn module mode by default (CL 162698). At this point, all tests within the go repository should be passing in both GOPATH mode and module mode, and the only mode-based skip calls should be for behaviors specific to one mode or the other.

I'd like to draw your attention to two of the related changes, CL 164622 and CL 164623. Those changes bring module mode to the standard library itself, adding two special modules: std and cmd. The std and cmd modules each contain the packages from the respective package pattern.

Since the std and cmd modules are tied to the specific Go toolchain in use, neither of those modules can appear in a require or replace directive, and they do not currently participate in minimum version selection.

The std and cmd modules exist in order to manage their respective vendor directories (GOROOT/src/vendor and GOROOT/src/cmd/vendor), which are now populated using go mod vendor. To update the vendored dependencies of those modules, first make sure your current working directory is inside the module. Then, employ the usual module-mode commands:

• go get -m $module@$version adds or modifies the required version of a dependency.

• go mod tidy ensures that the module requirements are complete and minimal.

• go mod vendor refreshes the vendor directory to match the current requirements.

• go mod graph and go mod why examine the module dependency graph.

There is one caveat: since we have biased the go command to avoid network lookups within GOROOT, go list -m all doesn't quite work yet. It lists the direct module dependencies of the standard modules, but not their transitive dependencies. (I plan to have that fixed by the 1.13 code freeze, ideally by a more general improvement such as #30240.)

As always, please file any issues you encounter in the tracker (or report them here), and tag me (@bcmills) and @jayconrod on issues relating to modules and the go command.

Happy modulating!

—Bryan

[Ian Lance Taylor]

On Mon, Mar 11, 2019 at 1:22 PM 'Bryan C. Mills' via golang-dev
<golan...@googlegroups.com> wrote:
>
> The std and cmd modules exist in order to manage their respective vendor directories (GOROOT/src/vendor and GOROOT/src/cmd/vendor), which are now populated using go mod vendor. To update the vendored dependencies of those modules, first make sure your current working directory is inside the module. Then, employ the usual module-mode commands:
>
>
> go get -m $module@$version adds or modifies the required version of a dependency.
>
> go mod tidy ensures that the module requirements are complete and minimal.
>
> go mod vendor refreshes the vendor directory to match the current requirements.
>
> go mod graph and go mod why examine the module dependency graph.
>
>
> There is one caveat: since we have biased the go command to avoid network lookups within GOROOT, go list -m all doesn't quite work yet. It lists the direct module dependencies of the standard modules, but not their transitive dependencies. (I plan to have that fixed by the 1.13 code freeze, ideally by a more general improvement such as #30240.)

Can you add this information to vendor/README and cmd/vendor/README,
or similar, so that there is one reference point for how to update
vendored source code in GOROOT? Thanks.

Ian

[Matthew Dempsky]

On Mon, Mar 11, 2019 at 1:22 PM 'Bryan C. Mills' via golang-dev <golan...@googlegroups.com> wrote:

As always, please file any issues you encounter in the tracker (or report them here), and tag me (@bcmills) and @jayconrod on issues relating to modules and the go command.

Did this break bootstrapping from master?

[Matthew Dempsky]

Repro:

    $ git clone https://go.googlesource.com/go /tmp/go1

    $ git clone /tmp/go1 /tmp/go2

    $ cd /tmp/go1/src && ./make.bash

    $ cd /tmp/go2/src && GOROOT_BOOTSTRAP=/tmp/go1 ./make.bash

Last command fails early on with:

Building Go cmd/dist using /tmp/go1.

Building Go toolchain1 using /tmp/go1.

go: cannot find main module, but found .git/config in /tmp/go2

to create a module there, run:

cd ../.. && go mod init

go tool dist: FAILED: /tmp/go1/bin/go install -gcflags=-l -tags=math_big_pure_go compiler_bootstrap bootstrap/cmd/...: exit status 1

[Bryan C. Mills]

Thanks for the report. Reverted in CL 166985, and I'll re-test those steps explicitly before the next attempt.

[Matthew Dempsky]

Sgtm, thanks!

[Nicolas Mailhot]

Le 2019-03-11 21:22, 'Bryan C. Mills' via golang-dev a écrit :

Hi,

> There is one caveat: since we have biased the go command to avoid
> network lookups within GOROOT, go list -m all doesn't quite work yet.
> It lists the direct module dependencies of the standard modules, but
> not their transitive dependencies.

Funny, that's almost exactly the behaviour we need Fedora-side, for
pretty much the same technical reasons, just not limited to GOROOT.

Regards,

--
Nicolas Mailhot

[Bryan C. Mills]

The revert was reverted in CL 167087, so we're back to GO111MODULE=on by default, with a new test (in misc/reboot) to catch regressions in the bootstrap process.

[Bryan C. Mills]

I looked into adding this, but realized:

(a) Since `go mod vendor` currently removes the entire vendor subtree, the README would be deleted every time (and require a manual `git checkout HEAD vendor/README` to restore).

(b) All of the steps for standard-library vendoring are now the same as for vendoring in any other module, so if the documentation is lacking within GOROOT, it is also lacking externally, and we should add it someplace where everyone can discover it.

So it seems that the main thing we need to communicate specifically for the standard library is the existence of the `std` and `cmd` modules, and the existence of the `go.mod` files communicates that fact.

[Austin Clements]

This is neat. I love that modules are flexible enough to capture std and cmd.

This may be true for someone up to their ears in modules, but because I live almost entirely in std, my knowledge of modules is (currently) severely limited. If I happen to ls $GOROOT/src and happen to notice the go.mod there, I'll know that we're using modules, but I wouldn't know that means I should use go mod vendor to update the vendor directory. Perhaps more to the point, if my goal is "update the vendor directory", I'm definitely not going to know my next step is "look for a go.mod".

So I would personally really appreciate a sentence in a readme, even if it can't be in vendor/README. :) It's fine if it points to some standard documentation for how to add new vendor dependencies, etc.

[Bryan C. Mills]

Is there a specific other file you'd like it to go in? (I could put it in the go.mod file itself, but that has the same discoverability problem — would src/README be discoverable enough?)

[Austin Clements]

It's a little weird that this would be the first thing in src/README, but I suppose that's where I would be inclined to put it. Maybe src/README.vendor?

[Michael Jones]

General observation: 

The role of mod files will henceforth be front and center for every developer. Just as each “welcome to Go” blog now starts with “start your program with ‘package main’ and then...” we will now preface every Go introduction with “start your project with ‘go mod init’“

This new world will put mod files in the minds of developers. Unlike today, where they are a new thing for those who care. After building 1.13 dev yesterday, and starting a new task, my first compile bailed with weird-to-me error about vendoring. 

Some investigation showed that I could no longer ignore vendoring or mod files, that the idea of no file means no vendoring is not a thing, and so it is now a core issue to any attempt to compile. That’s ok. Times change. But I doubt the next generation of Go users will feel the confusion I did, because I imagine each hello world will also include hello mod. 

--
You received this message because you are subscribed to the Google Groups "golang-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Michael T. Jones
michae...@gmail.com

[Bryan C. Mills]

On Fri, Mar 15, 2019 at 12:48 PM Michael Jones <michae...@gmail.com> wrote:

General observation: 

The role of mod files will henceforth be front and center for every developer. Just as each “welcome to Go” blog now starts with “start your program with ‘package main’ and then...” we will now preface every Go introduction with “start your project with ‘go mod init’“

This new world will put mod files in the minds of developers. Unlike today, where they are a new thing for those who care. After building 1.13 dev yesterday, and starting a new task, my first compile bailed with weird-to-me error about vendoring. 

Out of curiously, what was the exact error? (Was it telling you that it found a vendor.conf and wanted you to run 'go mod init'?)

At this point, module mode is still mostly independent of vendoring, so an error message about vendoring specifically would be unexpected.

[Michael Jones]

celeste:src mtj$ mkdir zzzz

celeste:src mtj$ cd zzzz

celeste:zzzz mtj$ cat > hello.go

package main

^D

celeste:zzzz mtj$ go build

go: cannot find main module; see 'go help modules'

celeste:zzzz mtj$ go help modules

A module is a collection of related Go packages.

Modules are the unit of source code interchange and versioning.

The go command has direct support for working with modules,

including recording and resolving dependencies on other modules.

Modules replace the old GOPATH-based approach to specifying

which source files are used in a given build.

Module support

Go 1.13 includes official support for Go modules,

including a module-aware 'go get' command.

Module-aware mode is active by default.

For more fine-grained control, Go 1.13 continues to respect

a temporary environment variable, GO111MODULE, which can be set to one

of three string values: off, auto, or on (the default).

If GO111MODULE=on or is unset, then the go command requires the use of

modules, never consulting GOPATH. We refer to this as the command

being module-aware or running in "module-aware mode".

If GO111MODULE=auto, then the go command enables or disables module

support based on the current directory. Module support is enabled only

when the current directory is outside GOPATH/src and itself contains a

go.mod file or is below a directory containing a go.mod file.

If GO111MODULE=off, then the go command never uses

module support. Instead it looks in vendor directories and GOPATH

to find dependencies; we now refer to this as "GOPATH mode."

In module-aware mode, GOPATH no longer defines the meaning of imports

during a build, but it still stores downloaded dependencies (in GOPATH/pkg/mod)

and installed commands (in GOPATH/bin, unless GOBIN is set).

Defining a module

A module is defined by a tree of Go source files with a go.mod file

in the tree's root directory. The directory containing the go.mod file

is called the module root. Typically the module root will also correspond

to a source code repository root (but in general it need not).

The module is the set of all Go packages in the module root and its

subdirectories, but excluding subtrees with their own go.mod files.

The "module path" is the import path prefix corresponding to the module root.

The go.mod file defines the module path and lists the specific versions

of other modules that should be used when resolving imports during a build,

by giving their module paths and versions.

For example, this go.mod declares that the directory containing it is the root

of the module with path example.com/m, and it also declares that the module

depends on specific versions of golang.org/x/text and gopkg.in/yaml.v2:

module example.com/m

require (

golang.org/x/text v0.3.0

gopkg.in/yaml.v2 v2.1.0

)

The go.mod file can also specify replacements and excluded versions

that only apply when building the module directly; they are ignored

when the module is incorporated into a larger build.

For more about the go.mod file, see 'go help go.mod'.

To start a new module, simply create a go.mod file in the root of the

module's directory tree, containing only a module statement.

The 'go mod init' command can be used to do this:

go mod init example.com/m

In a project already using an existing dependency management tool like

godep, glide, or dep, 'go mod init' will also add require statements

matching the existing configuration.

Once the go.mod file exists, no additional steps are required:

go commands like 'go build', 'go test', or even 'go list' will automatically

add new dependencies as needed to satisfy imports.

The main module and the build list

The "main module" is the module containing the directory where the go command

is run. The go command finds the module root by looking for a go.mod in the

current directory, or else the current directory's parent directory,

or else the parent's parent directory, and so on.

The main module's go.mod file defines the precise set of packages available

for use by the go command, through require, replace, and exclude statements.

Dependency modules, found by following require statements, also contribute

to the definition of that set of packages, but only through their go.mod

files' require statements: any replace and exclude statements in dependency

modules are ignored. The replace and exclude statements therefore allow the

main module complete control over its own build, without also being subject

to complete control by dependencies.

The set of modules providing packages to builds is called the "build list".

The build list initially contains only the main module. Then the go command

adds to the list the exact module versions required by modules already

on the list, recursively, until there is nothing left to add to the list.

If multiple versions of a particular module are added to the list,

then at the end only the latest version (according to semantic version

ordering) is kept for use in the build.

The 'go list' command provides information about the main module

and the build list. For example:

go list -m              # print path of main module

go list -m -f={{.Dir}}  # print root directory of main module

go list -m all          # print build list

Maintaining module requirements

The go.mod file is meant to be readable and editable by both

programmers and tools. The go command itself automatically updates the go.mod file

to maintain a standard formatting and the accuracy of require statements.

Any go command that finds an unfamiliar import will look up the module

containing that import and add the latest version of that module

to go.mod automatically. In most cases, therefore, it suffices to

add an import to source code and run 'go build', 'go test', or even 'go list':

as part of analyzing the package, the go command will discover

and resolve the import and update the go.mod file.

Any go command can determine that a module requirement is

missing and must be added, even when considering only a single

package from the module. On the other hand, determining that a module requirement

is no longer necessary and can be deleted requires a full view of

all packages in the module, across all possible build configurations

(architectures, operating systems, build tags, and so on).

The 'go mod tidy' command builds that view and then

adds any missing module requirements and removes unnecessary ones.

As part of maintaining the require statements in go.mod, the go command

tracks which ones provide packages imported directly by the current module

and which ones provide packages only used indirectly by other module

dependencies. Requirements needed only for indirect uses are marked with a

"// indirect" comment in the go.mod file. Indirect requirements are

automatically removed from the go.mod file once they are implied by other

direct requirements. Indirect requirements only arise when using modules

that fail to state some of their own dependencies or when explicitly

upgrading a module's dependencies ahead of its own stated requirements.

Because of this automatic maintenance, the information in go.mod is an

up-to-date, readable description of the build.

The 'go get' command updates go.mod to change the module versions used in a

build. An upgrade of one module may imply upgrading others, and similarly a

downgrade of one module may imply downgrading others. The 'go get' command

makes these implied changes as well. If go.mod is edited directly, commands

like 'go build' or 'go list' will assume that an upgrade is intended and

automatically make any implied upgrades and update go.mod to reflect them.

The 'go mod' command provides other functionality for use in maintaining

and understanding modules and go.mod files. See 'go help mod'.

The -mod build flag provides additional control over updating and use of go.mod.

If invoked with -mod=readonly, the go command is disallowed from the implicit

automatic updating of go.mod described above. Instead, it fails when any changes

to go.mod are needed. This setting is most useful to check that go.mod does

not need updates, such as in a continuous integration and testing system.

The "go get" command remains permitted to update go.mod even with -mod=readonly,

and the "go mod" commands do not take the -mod flag (or any other build flags).

If invoked with -mod=vendor, the go command assumes that the vendor

directory holds the correct copies of dependencies and ignores

the dependency descriptions in go.mod.

Pseudo-versions

The go.mod file and the go command more generally use semantic versions as

the standard form for describing module versions, so that versions can be

compared to determine which should be considered earlier or later than another.

A module version like v1.2.3 is introduced by tagging a revision in the

underlying source repository. Untagged revisions can be referred to

using a "pseudo-version" like v0.0.0-yyyymmddhhmmss-abcdefabcdef,

where the time is the commit time in UTC and the final suffix is the prefix

of the commit hash. The time portion ensures that two pseudo-versions can

be compared to determine which happened later, the commit hash identifes

the underlying commit, and the prefix (v0.0.0- in this example) is derived from

the most recent tagged version in the commit graph before this commit.

There are three pseudo-version forms:

vX.0.0-yyyymmddhhmmss-abcdefabcdef is used when there is no earlier

versioned commit with an appropriate major version before the target commit.

(This was originally the only form, so some older go.mod files use this form

even for commits that do follow tags.)

vX.Y.Z-pre.0.yyyymmddhhmmss-abcdefabcdef is used when the most

recent versioned commit before the target commit is vX.Y.Z-pre.

vX.Y.(Z+1)-0.yyyymmddhhmmss-abcdefabcdef is used when the most

recent versioned commit before the target commit is vX.Y.Z.

Pseudo-versions never need to be typed by hand: the go command will accept

the plain commit hash and translate it into a pseudo-version (or a tagged

version if available) automatically. This conversion is an example of a

module query.

Module queries

The go command accepts a "module query" in place of a module version

both on the command line and in the main module's go.mod file.

(After evaluating a query found in the main module's go.mod file,

the go command updates the file to replace the query with its result.)

A fully-specified semantic version, such as "v1.2.3",

evaluates to that specific version.

A semantic version prefix, such as "v1" or "v1.2",

evaluates to the latest available tagged version with that prefix.

A semantic version comparison, such as "<v1.2.3" or ">=v1.5.6",

evaluates to the available tagged version nearest to the comparison target

(the latest version for < and <=, the earliest version for > and >=).

The string "latest" matches the latest available tagged version,

or else the underlying source repository's latest untagged revision.

A revision identifier for the underlying source repository,

such as a commit hash prefix, revision tag, or branch name,

selects that specific code revision. If the revision is

also tagged with a semantic version, the query evaluates to

that semantic version. Otherwise the query evaluates to a

pseudo-version for the commit.

All queries prefer release versions to pre-release versions.

For example, "<v1.2.3" will prefer to return "v1.2.2"

instead of "v1.2.3-pre1", even though "v1.2.3-pre1" is nearer

to the comparison target.

Module versions disallowed by exclude statements in the

main module's go.mod are considered unavailable and cannot

be returned by queries.

For example, these commands are all valid:

go get github.com/gorilla/mux@latest    # same (@latest is default for 'go get')

go get github.com/gorilla/m...@v1.6.2    # records v1.6.2

go get github.com/gorilla/mux@e3702bed2 # records v1.6.2

go get github.com/gorilla/mux@c856192   # records v0.0.0-20180517173623-c85619274f5d

go get github.com/gorilla/mux@master    # records current meaning of master

Module compatibility and semantic versioning

The go command requires that modules use semantic versions and expects that

the versions accurately describe compatibility: it assumes that v1.5.4 is a

backwards-compatible replacement for v1.5.3, v1.4.0, and even v1.0.0.

More generally the go command expects that packages follow the

"import compatibility rule", which says:

"If an old package and a new package have the same import path,

the new package must be backwards compatible with the old package."

Because the go command assumes the import compatibility rule,

a module definition can only set the minimum required version of one

of its dependencies: it cannot set a maximum or exclude selected versions.

Still, the import compatibility rule is not a guarantee: it may be that

v1.5.4 is buggy and not a backwards-compatible replacement for v1.5.3.

Because of this, the go command never updates from an older version

to a newer version of a module unasked.

In semantic versioning, changing the major version number indicates a lack

of backwards compatibility with earlier versions. To preserve import

compatibility, the go command requires that modules with major version v2

or later use a module path with that major version as the final element.

For example, version v2.0.0 of example.com/m must instead use module path

example.com/m/v2, and packages in that module would use that path as

their import path prefix, as in example.com/m/v2/sub/pkg. Including the

major version number in the module path and import paths in this way is

called "semantic import versioning". Pseudo-versions for modules with major

version v2 and later begin with that major version instead of v0, as in

v2.0.0-20180326061214-4fc5987536ef.

As a special case, module paths beginning with gopkg.in/ continue to use the

conventions established on that system: the major version is always present,

and it is preceded by a dot instead of a slash: gopkg.in/yaml.v1

and gopkg.in/yaml.v2, not gopkg.in/yaml and gopkg.in/yaml/v2.

The go command treats modules with different module paths as unrelated:

it makes no connection between example.com/m and example.com/m/v2.

Modules with different major versions can be used together in a build

and are kept separate by the fact that their packages use different

import paths.

In semantic versioning, major version v0 is for initial development,

indicating no expectations of stability or backwards compatibility.

Major version v0 does not appear in the module path, because those

versions are preparation for v1.0.0, and v1 does not appear in the

module path either.

Code written before the semantic import versioning convention

was introduced may use major versions v2 and later to describe

the same set of unversioned import paths as used in v0 and v1.

To accommodate such code, if a source code repository has a

v2.0.0 or later tag for a file tree with no go.mod, the version is

considered to be part of the v1 module's available versions

and is given an +incompatible suffix when converted to a module

version, as in v2.0.0+incompatible. The +incompatible tag is also

applied to pseudo-versions derived from such versions, as in

v2.0.1-0.yyyymmddhhmmss-abcdefabcdef+incompatible.

In general, having a dependency in the build list (as reported by 'go list -m all')

on a v0 version, pre-release version, pseudo-version, or +incompatible version

is an indication that problems are more likely when upgrading that

dependency, since there is no expectation of compatibility for those.

See https://research.swtch.com/vgo-import for more information about

semantic import versioning, and see https://semver.org/ for more about

semantic versioning.

Module code layout

For now, see https://research.swtch.com/vgo-module for information

about how source code in version control systems is mapped to

module file trees.

Module downloading and verification

The go command checks downloads against known checksums,

to detect unexpected changes in the content of any specific module

version from one day to the next. See 'go help module-auth' for details.

The go command can fetch modules from a proxy instead of connecting

to source control systems directly, according to the setting of the GOPROXY

environment variable.

See 'go help goproxy' for details about the proxy and also the format of

the cached downloaded packages.

Modules and vendoring

When using modules, the go command completely ignores vendor directories.

By default, the go command satisfies dependencies by downloading modules

from their sources and using those downloaded copies (after verification,

as described in the previous section). To allow interoperation with older

versions of Go, or to ensure that all files used for a build are stored

together in a single file tree, 'go mod vendor' creates a directory named

vendor in the root directory of the main module and stores there all the

packages from dependency modules that are needed to support builds and

tests of packages in the main module.

To build using the main module's top-level vendor directory to satisfy

dependencies (disabling use of the usual network sources and local

caches), use 'go build -mod=vendor'. Note that only the main module's

top-level vendor directory is used; vendor directories in other locations

are still ignored.

celeste:zzzz mtj$