go modules have landed

14705 views
Skip to first unread message

Russ Cox

unread,
Jul 12, 2018, 5:20:40 PM7/12/18
to golang-dev, Paul Jolly, Bryan C. Mills, not...@gmail.com
As of CL 123580, submitted a few minutes ago, the development copy of the go command, in the main go repository, has support for Go modules. There are various minor known issues, but quite a lot works very well. 

There's still more to be done but this is a key milestone: if you run on the bleeding edge, you can play with modules just by using "go". Work on the module support will now be done exclusively in the main repo. We will automatically export the main repo to the x/vgo repo when we have a snapshot we want to make available to people using "vgo".

The open issues have moved from the vgo milestone to the Go 1.11 milestone, but labeled with the new 'modules' label.

A note on terminology: Go 1.11 is not shipping with "vgo support". It is shipping with "Go module support". Vgo was only ever the prototype/demo of that module support. Just like all the other features we've introduced to the go command over the past ten years, the goal for this one is to fit so well into the rest of the go command fabric that the seams are invisible - module support is just part of "go", the same way that, say, race detector support or GOPATH support is.

Speaking of GOPATH, the go command automatically chooses between using modules and using GOPATH depending on where it is run. If a go command runs outside GOPATH/src and there is a go.mod file in the current directory or one of its parent directories, then modules are enabled (otherwise not):

$ mkdir /tmp/hello
$ cd /tmp/hello
$ echo module hello >go.mod
$ cat >hello.go <<EOF
package main 

import "rsc.io/quote"

func main() {
println(quote.Glass())
}
EOF
$ go run .
I can eat glass and it doesn't hurt me.
$ cat go.mod
module hello

require rsc.io/quote v1.5.2
$ go list -m all
hello
golang.org/x/text v0.0.0-20170915032832-14c0d48ead0c

For more information, see 'go help modules', which I've included at the end of this mail.

Many thanks to everyone in the community who has helped us get to this point, but especially to Paul Jolly for really hammering on the module support and filing tons of great bugs, and to Baokun Lee for a steady sequence of helpful CLs. Thanks also to Bryan Mills for helping work out a lot of important details in the past few weeks. 

Thanks again, and keep the bug reports coming.

Best,
Russ




go help modules

A module is a collection of related Go packages.
Modules are the unit of source code interchange and versioning.
The go command has direct support for working with modules,
including recording and resolving dependencies on other modules.
Modules replace the old GOPATH-based approach to specifying
which source files are used in a given build.

Experimental module support

Go 1.11 includes experimental support for Go modules,
including a new module-aware 'go get' command.
We intend to keep revising this support, while preserving compatibility,
until it can be declared official (no longer experimental),
and then at a later point we may remove support for work
in GOPATH and the old 'go get' command.

The quickest way to take advantage of the new Go 1.11 module support
is to check out your repository into a directory outside GOPATH/src,
create a go.mod file (described in the next section) there, and run
go commands from within that file tree.

For more fine-grained control, the module support in Go 1.11 respects
a temporary environment variable, GO111MODULE, which can be set to one
of three string values: off, on, or auto (the default).
If GO111MODULE=off, then the go command never uses the
new module support. Instead it looks in vendor directories and GOPATH
to find dependencies; we now refer to this as "GOPATH mode."
If GO111MODULE=on, then the go command requires the use of modules,
never consulting GOPATH. We refer to this as the command being
module-aware or running in "module-aware mode".
If GO111MODULE=auto or is unset, then the go command enables or
disables module support based on the current directory.
Module support is enabled only when the current directory is outside
GOPATH/src and itself contains a go.mod file or is below a directory
containing a go.mod file.

Defining a module

A module is defined by a tree of Go source files with a go.mod file
in the tree's root directory. The directory containing the go.mod file
is called the module root. Typically the module root will also correspond
to a source code repository root (but in general it need not).
The module is the set of all Go packages in the module root and its
subdirectories, but excluding subtrees with their own go.mod files.

The "module path" is the import path prefix corresponding to the module root.
The go.mod file defines the module path and lists the specific versions
of other modules that should be used when resolving imports during a build,
by giving their module paths and versions.

For example, this go.mod declares that the directory containing it is the root
of the module with path example.com/m, and it also declares that the module
depends on specific versions of golang.org/x/text and gopkg.in/yaml.v2:

require (
)

The go.mod file can also specify replacements and excluded versions
that only apply when building the module directly; they are ignored
when the module is incorporated into a larger build.
For more about the go.mod file, see https://research.swtch.com/vgo-module.

To start a new module, simply create a go.mod file in the root of the
module's directory tree, containing only a module statement.
The 'go mod' command can be used to do this:

go mod -init -module example.com/m

In a project already using an existing dependency management tool like
godep, glide, or dep, 'go mod -init' will also add require statements
matching the existing configuration.

Once the go.mod file exists, no additional steps are required:
go commands like 'go build', 'go test', or even 'go list' will automatically
add new dependencies as needed to satisfy imports.

The main module and the build list

The "main module" is the module containing the directory where the go command
is run. The go command finds the module root by looking for a go.mod in the
current directory, or else the current directory's parent directory,
or else the parent's parent directory, and so on.

The main module's go.mod file defines the precise set of packages available
for use by the go command, through require, replace, and exclude statements.
Dependency modules, found by following require statements, also contribute
to the definition of that set of packages, but only through their go.mod
files' require statements: any replace and exclude statements in dependency
modules are ignored. The replace and exclude statements therefore allow the
main module complete control over its own build, without also being subject
to complete control by dependencies.

The set of modules providing packages to builds is called the "build list".
The build list initially contains only the main module. Then the go command
adds to the list the exact module versions required by modules already
on the list, recursively, until there is nothing left to add to the list.
If multiple versions of a particular module are added to the list,
then at the end only the latest version (according to semantic version
ordering) is kept for use in the build.

The 'go list' command provides information about the main module
and the build list. For example:

go list -m              # print path of main module
go list -m -f={{.Dir}}  # print root directory of main module
go list -m all          # print build list

Maintaining module requirements

The go.mod file is meant to be readable and editable by both
programmers and tools. The go command itself automatically updates the go.mod file
to maintain a standard formatting and the accuracy of require statements.

Any go command that finds an unfamiliar import will look up the module
containing that import and add the latest version of that module
to go.mod automatically. In most cases, therefore, it suffices to
add an import to source code and run 'go build', 'go test', or even 'go list':
as part of analyzing the package, the go command will discover
and resolve the import and update the go.mod file.

Any go command can determine that a module requirement is
missing and must be added, even when considering only a single
package from the module. On the other hand, determining that a module requirement
is no longer necessary and can be deleted requires a full view of
all packages in the module, across all possible build configurations
(architectures, operating systems, build tags, and so on).
The 'go mod -sync' command builds that view and then
adds any missing module requirements and removes unnecessary ones.

As part of maintaining the require statements in go.mod, the go command
tracks which ones provide packages imported directly by the current module
and which ones provide packages only used indirectly by other module
dependencies. Requirements needed only for indirect uses are marked with a
"// indirect" comment in the go.mod file. Indirect requirements are
automatically removed from the go.mod file once they are implied by other
direct requirements. Indirect requirements only arise when using modules
that fail to state some of their own dependencies or when explicitly
upgrading a module's dependencies ahead of its own stated requirements.

Because of this automatic maintenance, the information in go.mod is an
up-to-date, readable description of the build.

The 'go get' command updates go.mod to change the module versions used in a
build. An upgrade of one module may imply upgrading others, and similarly a
downgrade of one module may imply downgrading others. The 'go get' command
makes these implied changes as well. If go.mod is edited directly, commands
like 'go build' or 'go list' will assume that an upgrade is intended and
automatically make any implied upgrades and update go.mod to reflect them.

The 'go mod' command provides other functionality for use in maintaining
and understanding modules and go.mod files. See 'go help mod'.

Pseudo-versions

The go.mod file and the go command more generally use semantic versions as
the standard form for describing module versions, so that versions can be
compared to determine which should be considered earlier or later than another.
A module version like v1.2.3 is introduced by tagging a revision in the
underlying source repository. Untagged revisions can be referred to
using a "pseudo-version" of the form v0.0.0-yyyymmddhhmmss-abcdefabcdef,
where the time is the commit time in UTC and the final suffix is the prefix
of the commit hash. The time portion ensures that two pseudo-versions can
be compared to determine which happened later, the commit hash identifes
the underlying commit, and the v0.0.0- prefix identifies the pseudo-version
as a pre-release before version v0.0.0, so that the go command prefers any
tagged release over any pseudo-version.

Pseudo-versions never need to be typed by hand: the go command will accept
the plain commit hash and translate it into a pseudo-version (or a tagged
version if available) automatically. This conversion is an example of a
module query.

Module queries

The go command accepts a "module query" in place of a module version
both on the command line and in the main module's go.mod file.
(After evaluating a query found in the main module's go.mod file,
the go command updates the file to replace the query with its result.)

A fully-specified semantic version, such as "v1.2.3",
evaluates to that specific version.

A semantic version prefix, such as "v1" or "v1.2",
evaluates to the latest available tagged version with that prefix.

A semantic version comparison, such as "<v1.2.3" or ">=v1.5.6",
evaluates to the available tagged version nearest to the comparison target
(the latest version for < and <=, the earliest version for > and >=).

The string "latest" matches the latest available tagged version,
or else the underlying source repository's latest untagged revision.

A revision identifier for the underlying source repository,
such as a commit hash prefix, revision tag, or branch name,
selects that specific code revision. If the revision is
also tagged with a semantic version, the query evaluates to
that semantic version. Otherwise the query evaluates to a
pseudo-version for the commit.

All queries prefer release versions to pre-release versions.
For example, "<v1.2.3" will prefer to return "v1.2.2"
instead of "v1.2.3-pre1", even though "v1.2.3-pre1" is nearer
to the comparison target.

Module versions disallowed by exclude statements in the
main module's go.mod are considered unavailable and cannot
be returned by queries.

For example, these commands are all valid:

go get github.com/gorilla/mux@latest    # same (@latest is default for 'go get')
go get github.com/gorilla/m...@v1.6.2    # records v1.6.2
go get github.com/gorilla/mux@e3702bed2 # records v1.6.2
go get github.com/gorilla/mux@c856192   # records v0.0.0-20180517173623-c85619274f5d
go get github.com/gorilla/mux@master    # records current meaning of master


Module compatibility and semantic versioning

The go command requires that modules use semantic versions and expects that
the versions accurately describe compatibility: it assumes that v1.5.4 is a
backwards-compatible replacement for v1.5.3, v1.4.0, and even v1.0.0.
More generally the go command expects that packages follow the
"import compatibility rule", which says:

"If an old package and a new package have the same import path, 
the new package must be backwards compatible with the old package."

Because the go command assumes the import compatibility rule,
a module definition can only set the minimum required version of one 
of its dependencies: it cannot set a maximum or exclude selected versions.
Still, the import compatibility rule is not a guarantee: it may be that
v1.5.4 is buggy and not a backwards-compatible replacement for v1.5.3.
Because of this, the go command never updates from an older version
to a newer version of a module unasked.

In semantic versioning, changing the major version number indicates a lack
of backwards compatibility with earlier versions. To preserve import
compatibility, the go command requires that modules with major version v2
or later use a module path with that major version as the final element.
For example, version v2.0.0 of example.com/m must instead use module path
example.com/m/v2, and packages in that module would use that path as
their import path prefix, as in example.com/m/v2/sub/pkg. Including the
major version number in the module path and import paths in this way is
called "semantic import versioning". Pseudo-versions for modules with major
version v2 and later begin with that major version instead of v0, as in
v2.0.0-20180326061214-4fc5987536ef.

The go command treats modules with different module paths as unrelated:
it makes no connection between example.com/m and example.com/m/v2.
Modules with different major versions can be used together in a build
and are kept separate by the fact that their packages use different
import paths.

In semantic versioning, major version v0 is for initial development,
indicating no expectations of stability or backwards compatibility.
Major version v0 does not appear in the module path, because those
versions are preparation for v1.0.0, and v1 does not appear in the
module path either.

As a special case, for historical reasons, module paths beginning with
gopkg.in/ continue to use the conventions established on that system:
the major version is always present, and it is preceded by a dot 
instead of a slash: gopkg.in/yaml.v1 and gopkg.in/yaml.v2, not

for more information.

Module verification

The go command maintains, in the main module's root directory alongside
go.mod, a file named go.sum containing the expected cryptographic checksums
of the content of specific module versions. Each time a dependency is
used, its checksum is added to go.sum if missing or else required to match
the existing entry in go.sum.

The go command maintains a cache of downloaded packages and computes
and records the cryptographic checksum of each package at download time.
In normal operation, the go command checks these pre-computed checksums
against the main module's go.sum file, instead of recomputing them on
each command invocation. The 'go mod -verify' command checks that
the cached copies of module downloads still match both their recorded
checksums and the entries in go.sum.

Modules and vendoring

When using modules, the go command completely ignores vendor directories.

By default, the go command satisfies dependencies by downloading modules
from their sources and using those downloaded copies (after verification,
as described in the previous section). To allow interoperation with older
versions of Go, or to ensure that all files used for a build are stored
together in a single file tree, 'go mod -vendor' creates a directory named
vendor in the root directory of the main module and stores there all the
packages from dependency modules that are needed to support builds and
tests of packages in the main module.

To build using the main module's top-level vendor directory to satisfy
dependencies (disabling use of the usual network sources and local
caches), use 'go build -getmode=vendor'. Note that only the main module's
top-level vendor directory is used; vendor directories in other locations
are still ignored.



Sameer Ajmani

unread,
Jul 12, 2018, 5:50:10 PM7/12/18
to Russ Cox, golang-dev, Paul Jolly, Bryan C. Mills, not...@gmail.com
I'm thrilled to see this milestone. Thank you to Russ and all the contributors, testers, and other people who have helped inform the design and implementation of Go modules.
S

--
You received this message because you are subscribed to the Google Groups "golang-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

samuel....@gmail.com

unread,
Jul 12, 2018, 7:35:12 PM7/12/18
to golang-dev
i'll be speaking at GoSF next week with a tl;dr of why i believe this is a sad milestone. The upside here is that it is indeed still experimental.

Russ Cox

unread,
Jul 12, 2018, 8:48:40 PM7/12/18
to Filippo Valsorda, golang-dev
On Thu, Jul 12, 2018 at 5:57 PM Filippo Valsorda <fil...@golang.org> wrote:
Hi Russ, a quick note. My $GOPATH is $HOME, and I develop all my code in $HOME/src.

If I don't set GO111MODULE, I will have to put code somewhere else, which I'd rather not do. But if I set GO111MODULE=on globally, all non-module aware code will break. So my only option is to set GO111MODULE=on on a case-by-case basis, which is not great.

Why not make auto mode work within GOPATH as well when a go.mod file is present? I feel like it would also avoid another layer of GOPATH confusion, given "we don't want the repository location to matter" is a common complaint that module support is meant to address.

Hi Filippo,

It's a great question, and I've added golang-dev back because I think lots of people will have this question, so it's worth sharing the answer. I like to use GOPATH=HOME too, and keeping that case in mind is why the GOPATH carveout when GO111MODULE=on is only GOPATH/src not all of GOPATH. Even so, I think it's also important to remember that most of our users don't set GOPATH at all. They take the default HOME/go, and most of them just want to be able to git clone anywhere they want (probably NOT under HOME/go) and get to work. And with modules now they can.

The logic in your first paragraph is imprecise. It's simply not the case that your only option is to set GO111MODULE=on on a case-by-case basis. Even though you'd rather not, the other option is to work outside HOME/src - the rest of HOME is available. It's up to you which option you prefer, but you do have two options, not one. (That's exactly why the GOPATH carve-out is GOPATH/src not GOPATH - so that if GOPATH=HOME we aren't disallowing modules in your entire home directory.)

We did consider allowing automatic module-aware directories even inside GOPATH/src, but I think that ends up more confusing than not. In the current meaning of GO111MODULE=auto, there are just two worlds: directories in GOPATH/src, which build in GOPATH mode, and directories outside, which build in module mode. The worlds are completely disconnected: code from one world simply never connects to code from the other world. Suppose that weren't the case, and suppose you had GOPATH/src/A with no go.mod and importing B, GOPATH/src/B with a go.mod and importing C, and no GOPATH/src/C. If we did the more fine-grained flipping of module mode based on current directory within GOPATH/src, a work session might look like:

    $ cd GOPATH/src/A
    $ go build
    # B
    ../B/b.go:10: import "C": C not found
    $ 

Hmm, that's weird, let's try building B directly:

    $ go build B
    ../B/b.go:10: import "C": C not found
    $

Odd. I thought I was just working on B. Let's see what's going on.

    $ cd ../B
    $ go build
    $

Wait, what? It succeeded?

    $ go build B
    $ go build C
    $ 

Um, OK.

    $ cd ../A
    $ go build
    ../B/b.go:10: import "C": C not found
    $ 

Things get even worse if the build of B is simply silently using a different version of C than the build of A is, and the error is something like "undefined: C.foo" instead of "C not found". Or maybe just a crash in B that only happens with one version of C and not the other, so when you cd ../B to run 'go test' the problem goes away. Those optical illusions that look like one thing from far away but something completely different when you get up close are fun on a wall but I think less so in source code. I think that would be incredibly confusing.

We could try to fix this by saying that B/b.go has a single, fixed meaning - module meaning because B/go.mod exists - and when A imports B and B has a go.mod then B is still interpreted as using modules. But what if A imports C too, and B/go.mod says one version of C and GOPATH/src/C is a different version? A's direct import of C must mean GOPATH/src/C, right? But then B's import of C means the module version in go.mod? So import "C" means different things to different halves of the build? That can't be right either. And anyway in auto mode we're trying not to break the existing meaning of working in the non-module-aware GOPATH/src/A: it's no good to redefine what the B half of it means.

Disconnecting the two worlds avoids all this confusion while still giving us a way to support both uses: directories in GOPATH world cannot accidentally refer to directories in module world, and directories in module world cannot accidentally refer to directories in GOPATH world. (You could play funny games with replace statements in go.mod to point into GOPATH directories and cross the two worlds in that direction, but at least then the confusion is kind of your fault at that point.)

Now that modules are in the "go" command, I intend to try running with GO111MODULE=on and see how it goes. Each time I want to start working in a different checkout in GOPATH/src I'll run 'go mod -init' in the root of the repo and go from there. If you want keep working in $HOME/src, that's what I would do. Otherwise work in something like $HOME/project1, $HOME/project2, etc., or change GOPATH to $HOME/gopath.

Best,
Russ

Uli Kunitz

unread,
Jul 13, 2018, 3:41:53 AM7/13/18
to golang-dev
Since go.mod appears to become now a part of the official go tool, I gave it a run. It appeared to work well, but there are three observations that I would like to share:
  1. I can't do rm -fr $GOPATH/src/mod because the directory with the @<hexstring> (in my case golang.org/x/crypto@<hexstring>) is not writable for the user. 
  2. If GOPATH is not set $HOME/go/src/mod is created, which is not nice, when my go command comes actually from $HOME/go1.11 and there is another version in $HOME/go. I guess in that scenario I will have to keep GOPATH.
  3. goimports will have to support go.mod. Since I'm using an older version, some work might have already gone into it.


Jakub Cajka

unread,
Jul 13, 2018, 5:27:36 AM7/13/18
to Russ Cox, golang-dev, Paul Jolly, Bryan C. Mills, not...@gmail.com
Hello,

this looks really cool. It seems that it will make really easy to follow good engineering practices regarding creating releases. Also it seems that this will make vendoring effectively obsoleted/not needed any more, which I perceive as really good.

Also do you plan to tackle the issues with API stability/compatibility i.e. will vgo be able to detect and report API breakages?

On side note, is there a place that one can follow and see the development progress and development discussions of the vgo/dep and contribute to them? IIRC I have read somewhere sometime ago that you are using some private slack for the discussions, right?

Thanks for great work on Go modules,

JC

Sameer Ajmani

unread,
Jul 13, 2018, 9:16:08 AM7/13/18
to Uli Kunitz, Alan and Leila and Sabrina and Penelope Donovan-Kazemi, Ian Cottrell, golang-dev
We are actively working on updating goimports and other popular Go tools to support Go modules.  Ian Cottrell and Alan Donovan are leading this work. If you are a tool or IDE maintainer and have questions about this, please contact them directly.

Russ Cox

unread,
Jul 13, 2018, 9:19:37 AM7/13/18
to Uli Kunitz, golang-dev
On Fri, Jul 13, 2018 at 3:41 AM, Uli Kunitz <uli.k...@gmail.com> wrote:
Since go.mod appears to become now a part of the official go tool, I gave it a run. It appeared to work well, but there are three observations that I would like to share:
  1. I can't do rm -fr $GOPATH/src/mod because the directory with the @<hexstring> (in my case golang.org/x/crypto@<hexstring>) is not writable for the user. 
Yes, this is really annoying: when I made the directories unwritable I thought that rm -rf would still work on them, and I'm disappointed that it doesn't. Even so, I do think it's important to keep them unwritable to prevent accidental modification. I intend to add 'go clean -modcache' or something like that to blow away that tree for you (chmod + rm), so even though rm -rf fails, there will be an easy command. 
  1. If GOPATH is not set $HOME/go/src/mod is created, which is not nice, when my go command comes actually from $HOME/go1.11 and there is another version in $HOME/go. I guess in that scenario I will have to keep GOPATH.
Yes, that's true. The downloaded modules have to be kept somewhere, and since the file names show up in stack traces and the like, it didn't seem like $HOME/.cache/whatever was a terribly nice place to put them. (Also didn't want them to be deleted by some random cron job.) $GOPATH/src/mod seemed reasonable. But if the default GOPATH is not the right setting for your setup, then yes you do need to keep it set where you do want files to go. (GOPATH/bin is also the default place that 'go install' will write, in the absence of a GOBIN setting, so that's another thing you might not want to mix into some other Go checkout.)
  1. goimports will have to support go.mod. Since I'm using an older version, some work might have already gone into it.
Unfortunately not yet. That will be one of the things we have to make work really well before we can stop calling modules "experimental". For now, the latest goimports understands not to walk around in (the possibly very large) GOPATH/src/mod, but that's all.

I think goimports will actually get quite a bit faster with modules, because it can reliably assume that the versioned dependencies are not changing. But we haven't done that yet.

Russ

Russ Cox

unread,
Jul 13, 2018, 9:26:41 AM7/13/18
to Jakub Cajka, golang-dev, Paul Jolly, Bryan C. Mills, Lee Baokun
On Fri, Jul 13, 2018 at 5:27 AM, Jakub Cajka <jca...@redhat.com> wrote:
Hello,

  this looks really cool. It seems that it will make really easy to follow good engineering practices regarding creating releases. Also it seems that this will make vendoring effectively obsoleted/not needed any more, which I perceive as really good.

  Also do you plan to tackle the issues with API stability/compatibility i.e. will vgo be able to detect and report API breakages?

Yes, we intend to have a tool that can at least report type-level API changes. Of course sometimes unintentional API changes are made in the semantics of an implementation, and that's a little bit more difficult to report in an automated tool. :-)

  On side note, is there a place that one can follow and see the development progress and development discussions of the vgo/dep and contribute to them? IIRC I have read somewhere sometime ago that you are using some private slack for the discussions, right?

Discussion of Go modules happens mostly on the issue tracker. Until now we were using a vgo milestone, but now we're using the 'modules' label. (In general I prefer to do open source work in public view, not on walled-garden forums like Slack.)

Best,
Russ

Jakub Cajka

unread,
Jul 13, 2018, 10:01:56 AM7/13/18
to Russ Cox, golang-dev, Paul Jolly, Bryan C. Mills, Lee Baokun
> <https://github.com/golang/go/labels/modules>' label. (In general I prefer
> to do open source work in public view, not on walled-garden forums like
> Slack.)
>
> Best,
> Russ
>

Awesome to hear that :). So you are now under go not vgo under golang@github? And the CL workflow is the same as for Go, right?

JC

Russ Cox

unread,
Jul 13, 2018, 11:29:59 AM7/13/18
to Jakub Cajka, golang-dev, Paul Jolly, Bryan C. Mills, Lee Baokun
On Fri, Jul 13, 2018 at 10:01 AM, Jakub Cajka <jca...@redhat.com> wrote:
Awesome to hear that :). So you are now under go not vgo under golang@github?  And the CL workflow is the same as for Go, right?

Yes. We've been using the golang/go issue tracker from the start, we just changed from the vgo milestone to the Go1.11 milestone with a label.
The CL workflow is same as Go; before you used go.googlesource.com/vgo and now you use /go like everything else.

Best,
Russ

Nate Finch

unread,
Jul 13, 2018, 3:47:54 PM7/13/18
to golang-dev
I couldn't follow your response to Fillipo... what exactly are the problems for people who have GOPATH=$HOME, and what steps do we need to take to keep working?  I think a lot of gophers still set GOPATH this way (myself included). 

Bryan C. Mills

unread,
Jul 13, 2018, 4:02:39 PM7/13/18
to Russ Cox, fil...@golang.org, golang-dev, nate....@gmail.com
I've been thinking some more about $GOPATH and modules, and came up with a proposal that I *think* addresses these concerns (https://golang.org/issue/26377).

It is quite possible that I have missed some important detail — or that the proposal is not worth its cost — but perhaps it is worth considering.

--

Russ Cox

unread,
Jul 13, 2018, 4:17:52 PM7/13/18
to Nate Finch, golang-dev
On Fri, Jul 13, 2018 at 3:47 PM, Nate Finch <nate....@gmail.com> wrote:
I couldn't follow your response to Fillipo... what exactly are the problems for people who have GOPATH=$HOME, and what steps do we need to take to keep working?  I think a lot of gophers still set GOPATH this way (myself included). 

If you want to use modules, you can. You just have to create them outside GOPATH/src like everyone else. So for example it's fine to put your module-using project in $HOME/project,. If you use $HOME/src/project, modulesjust won't kick in.

Russ

Filippo Valsorda

unread,
Jul 13, 2018, 4:39:40 PM7/13/18
to Russ Cox, nate....@gmail.com, golang-dev
The problem is that we just finished telling people that "it just builds because it uses vendoring, but you have to clone it within GOPATH/src", which broke an assumption about the location of a repository being irrelevant, and now we have to tell people that "it just builds because it uses modules, but you have to clone it outside GOPATH/src", adding confusion instead of solving that very common adoption issue. The two answers will have to coexist for a while, too, as projects migrate at different paces. And if they never heard of GOPATH, then add "ah, then maybe you cloned it within ~/go, which is the default GOPATH, and this project doesn't build under that location".

Also, this will require most CI systems to be reconfigured, as they clone in GOPATH and run "go build" by default.

I don't think I understand the nuances enough to judge the tradeoff or the alternatives, but I am very familiar with the off-putting impact GOPATH has on new developers, and this only barely helps.

In practice, I'm afraid horrible symlink hacks provided for a better README#Install section than modules will in 1.11. 

Filippo Valsorda

unread,
Jul 13, 2018, 6:58:29 PM7/13/18
to Russ Cox, nate....@gmail.com, golang-dev
I realized the tone was off from my intention. To be clear, I believe modules are a better solution to this and other issues than what we had so far (definitely better than symlinks!), and I'm happy they are bringing the answer directly to the go tool.

I was only trying to convey how the transition might interact with an experience I've seen often, but I do believe I don't yet understand the nuances of the tradeoff that might make it necessary.

Uli Kunitz

unread,
Jul 14, 2018, 3:55:24 AM7/14/18
to golang-dev
Sameer, I appreciate that work on goimports has already started. Currently I'm only interested because I'm a heavy goimports user via the vim-go package.

  1. I can't do rm -fr $GOPATH/src/mod because the directory with the @<hexstring> (in my case golang.org/x/crypto@<hexstring>) is not writable for the user. 
Yes, this is really annoying: when I made the directories unwritable I thought that rm -rf would still work on them, and I'm disappointed that it doesn't. Even so, I do think it's important to keep them unwritable to prevent accidental modification. I intend to add 'go clean -modcache' or something like that to blow away that tree for you (chmod + rm), so even though rm -rf fails, there will be an easy  command. 

Ross, at least we agree the feature to be annoying. But it is really necessary? So far Go didn't require anything to be user-read-only. Neither .cache/go-build or anything under $GOPATH or $GOROOT. Accidental modification of binaries or source were always possible, but I have seen not a lot of complaints or issues about that. I have also trouble to name any other tool that sets files or directories user-read-only in the home directory by default. Not even security tools like openssh or gpg are doing this.

It appears also that the feature has a bug, because subdirectories of the module are still writable, even though the module directory and files aren't.

Russ Cox

unread,
Jul 16, 2018, 11:36:28 AM7/16/18
to Filippo Valsorda, Nate Finch, golang-dev
On Fri, Jul 13, 2018 at 4:39 PM, Filippo Valsorda <fil...@golang.org> wrote:
The problem is that we just finished telling people that "it just builds because it uses vendoring, but you have to clone it within GOPATH/src", which broke an assumption about the location of a repository being irrelevant, and now we have to tell people that "it just builds because it uses modules, but you have to clone it outside GOPATH/src", adding confusion instead of solving that very common adoption issue.

I don't quite understand what you mean by "we just finished telling people", since code has had to be built in GOPATH since Go 1 (March 2012). Nothing has changed since then, except maybe the introduction of vendoring in 2015, but that didn't affect the necessity of GOPATH. If you've been suggesting unsupported hacks involving symlinks to your users, so that they're fighting against the go command instead of working with it, and if that in turn somehow complicates their adoption of module-aware code, then that's unfortunate, but it does not create a constraint on the evolution of our tooling. However, this is not even the main consideration.

My priority here is to ensure a smooth transition for *late* adopters, who make up the bulk of our users and do not live and breathe golang-dev. A common mistake we all make in software development is to forget that users both know and care less than you do about the ins and outs of what you are shipping to them, whether that's a package or a tool. When we forget that fact, we write documentation, APIs, and tools that can't be understood without becoming an expert in the implementation. When we forget that fact, we badly underestimate the pain caused to end users by a breaking change and the amount of time and effort it will take them to figure out how to recover from it. 

The first rule of shipping something in a new Go release is the same as in medicine: "first, do no harm". Don't break people's existing Go usages, especially not without warning. This is why we pre-announce operating system and architecture deprecations in an earlier release. This is why we rolled back the change that made regexp.Regexp a little faster in concurrent usage but broke reflect.DeepEqual on regexp.Regexp. This is why we rolled back the change that made rand.Intn faster but return different numbers for a given source. This is why we add explicit transition support when we make significant changes to the way the go command finds source code ($GO15VENDOREXPERIMENT, and now $GO111MODULE).

This is the first release with module support. It is explicitly experimental, and to make sure we do not break users who are happily using GOPATH on the day they update to Go 1.11, we must keep GOPATH working by default exactly as it does today. That's why using modules is opt-in. Some later release will make it opt-out, and then some later release will take away the option entirely. Keeping GOPATH working by default means that we cannot redefine what work inside GOPATH means by default, not in this release. If you "go get A" in GOPATH mode and that downloads B into GOPATH/src/B, and then you cd into GOPATH/src/B, you need to get the GOPATH interpretation of the code - or else we've potentially broken you - and not the module interpretation. Again it is critical for late adopters - who may be interested in modules but haven't been following every last detail really just want all the commands they type to keep their current meanings until they're ready to move over - that we keep GOPATH working exactly as it did in the last release. Every time you break a user you give them an opportunity to consider whether finding a different language might be a better use of time than fixing the latest breakage.

Thankfully, work outside GOPATH has not been supported at all, so we can redefine work outside GOPATH to be module aware by default in this release, without breaking existing users. When we did vendoring, you couldn't try it without setting an environment variable. You can still do that here, but you can also just work in a different directory, which avoids forcing all usage in all cases into one mode or the other. This is a real improvement. At the same time, I do understand that some people want modules in GOPATH/src. That's where we're headed, and if you want to get there early, you can set the environment variable.

Also, this will require most CI systems to be reconfigured, as they clone in GOPATH and run "go build" by default.

I haven't used most CI systems but I can only assume they allow setting environment variables. Users who want to opt in to trying their code in module-aware mode can tell the CI system to set GO111MODULE=on. The rest of the system does not need to be reconfigured, and in particular it can keep using GOPATH. It might be reasonable for CI systems to set this automatically if they see a go.mod in the root directory, but I would probably caution against that and say that they should just let users who want module mode explicitly opt in by setting the environment variable themselves. Note that it would be entirely reasonable to start using modules and Go 1.11 but also run 'go mod -vendor' to populate the vendor directory for users on older Go releases. In that case you'd want the CI system to keep testing the vendor directory. This kind of consideration is why I think the CI systems should not be changing the default on their own. Let the users keep control, and let everything share the same default (the one from the go toolchain being used).
 
I don't think I understand the nuances enough to judge the tradeoff or the alternatives, but I am very familiar with the off-putting impact GOPATH has on new developers, and this only barely helps.

First, this is only the next step toward eliminating GOPATH confusion, not the end result. We will get to the point where users can check out code anywhere - inside GOPATH or out - and everything just works. But we will do it deliberately, to avoid breaking users. This release is not the one that is meant to solve all the problems with GOPATH. It just moves us a little closer to that goal.

Second, I think you are underestimating how much this does help new developers. New developers don't have GOPATH set at all, so it defaults to $HOME/go. That means they can use modules anywhere in $HOME except $HOME/go/src. It seems very likely to me that a developer would naturally choose to check out a project in $HOME/project or $HOME/src/project before stumbling into $HOME/go/src/project, and the first two will work fine with Go 1.11 and modules by default.

I certainly do appreciate that everyone on this list is excited to use modules as much as possible, myself included. We all just need to remember that not all our users are like us. In fact most aren't. They just want to get work done and adopt new potentially-breaking features on their schedule, not ours.

Best,
Russ

carolyn....@microsoft.com

unread,
Jul 16, 2018, 12:43:43 PM7/16/18
to golang-dev
I am quite excited to try this out! 🎉

A couple other people and I are learning how to be upstream Go contributors so that we can provide feedback and shape the UX for Go modules. If anyone else is interested in joining us, we are collaborating in the Go forums:

Russ Cox

unread,
Jul 16, 2018, 12:49:11 PM7/16/18
to Filippo Valsorda, golang-dev
Sorry to belabor this thread, but two quick followups.

On Thu, Jul 12, 2018 at 8:48 PM, Russ Cox <r...@golang.org> wrote:
On Thu, Jul 12, 2018 at 5:57 PM Filippo Valsorda <fil...@golang.org> wrote:
If I don't set GO111MODULE, I will have to put code somewhere else, which I'd rather not do. But if I set GO111MODULE=on globally, all non-module aware code will break. So my only option is to set GO111MODULE=on on a case-by-case basis, which is not great.

It's a great question, and I've added golang-dev back because I think lots of people will have this question, so it's worth sharing the answer. I like to use GOPATH=HOME too, and keeping that case in mind is why the GOPATH carveout when GO111MODULE=on is only GOPATH/src not all of GOPATH. ...

1. This is an unfortunate on my part: the GOPATH carveout only applies when GO111MODULE=auto. If instead GO111MODULE=on, then even code in GOPATH is assumed to be modules, so there's no "carveout".

2. It's not true that "all non-module aware code will break." In fact most code will work fine. If you set GO111MODULE=on and cd into a project in GOPATH/src that is using any of the nine known previous dependency management systems, the go command will convert that system's config to a go.mod and then proceed with what you asked it to do. Similarly, if the go command can find the root of a VCS checkout, it will create a go.mod with only a module line and then add dependencies as needed. This automatic initialization probably needs a few more heuristics before we can make GO111MODULE=on the default in a later release, but it already works quite well. In my experiments, about 90% of the most commonly imported go-gettable Go projects work unmodified using modules with auto-initialization. And among the go-gettable Go projects with the largest dependency counts, about 60% work unmodified. Dave Cheney's post about Go modules and CI integration shows that you can set GO111MODULE=on in the CI config for non-module-aware trees to test compatibility with the auto-initialization.

Best,
Russ

Andrew Gerrand

unread,
Jul 17, 2018, 5:39:50 AM7/17/18
to Russ Cox, golang-dev
Today, while debugging, I make some changes to a couple of dependent packages: cloud.google.com/go/pubsub and google.golang.org/grpc. Added some logging, changed a bit of error handling logic. The usual.

How do I do the same thing with modules?
The code in $GOPATH/src/mod is read-only, so apparently not there.

One thing I tried that works is "go mod -vendor", making my changes to the deps in the vendor directory, and then running go build/test with "-getmode=vendor". Is that the only supported way of doing this? This is a bit awkward, and makes it somewhat cumbersome to contribute fixes to upstream projects. 

Thanks,
Andrew

Paul Jolly

unread,
Jul 17, 2018, 7:14:45 AM7/17/18
to Andrew Gerrand, Russ Cox, golan...@googlegroups.com
> Today, while debugging, I make some changes to a couple of dependent packages: cloud.google.com/go/pubsub and google.golang.org/grpc. Added some logging, changed a bit of error handling logic. The usual.
>
> How do I do the same thing with modules?

You want to be using a replace directive for this.

Assuming you already have a require directive for
cloud.google.com/go/pubsub vX, then you can add a version-less replace
directive (i.e. applies to any version requirement of that module):

replace cloud.google.com/go/pubsub => github.com/adg/pubsub

or if you want to work locally:

replace cloud.google.com/go/pubsub => ../pubsub

Thanks,


Paul

Dan Kortschak

unread,
Jul 17, 2018, 7:47:31 AM7/17/18
to Russ Cox, golang-dev
The documentation for go mod falls down in this area. I was looking
through `go help mod` today trying to figure out how to resolve an
issue filed by one of our users and came to the conclusion that the
easiest way to deal with the problem was just manually edit the go.mod
file and then hope. This is not what I usually expect from Go docs.

Dan

roger peppe

unread,
Jul 17, 2018, 9:09:10 AM7/17/18
to Andrew Gerrand, Russ Cox, golang-dev
I agree that this is an interesting question. There's also the issue
of temporary
changes made just to track down an issue when debugging, something I
do quite a bit.

I've been wondering about making some kind of "gohack" command, so
a command like:

gohack golang.org/x/net

would copy the currently specified version of the golang.org/x/net module
into a local directory (e.g. ./_hack/golang.org/x/net), then add an appropriate
replace directive to the go.mod file.

It might also be useful to have a -diff flag to output any changes that
have been made in _hack, but more generally, perhaps gohack could initialize
a git repo with the current contents of the module, so git diff, git checkout,
etc would work. Better still might be to clone the existing repo when available,
because it's not uncommon for temporary hack changes to other modules
to get turned into PRs later.

Perhaps others have developed different ways of going about this kind of
cross-module development. Anyone?

komu wairagu

unread,
Jul 17, 2018, 9:59:22 AM7/17/18
to golang-dev
Hi Paul
Thanks for this, I did not know you can use a replace directive pointing to a local package.
I tried this and it failed, and then worked with a minor manual fix. 
I have filed, https://github.com/golang/go/issues/26417 because of that 
Thanks,


Paul

Russ Cox

unread,
Jul 17, 2018, 10:01:18 AM7/17/18
to roger peppe, Andrew Gerrand, golang-dev
On Tue, Jul 17, 2018 at 5:39 AM, Andrew Gerrand <a...@golang.org> wrote:
Today, while debugging, I make some changes to a couple of dependent packages: cloud.google.com/go/pubsub and google.golang.org/grpc. Added some logging, changed a bit of error handling logic. The usual.

How do I do the same thing with modules?
The code in $GOPATH/src/mod is read-only, so apparently not there.

Good question, as others have said. I agree that more tooling is needed here, to make it trivial to do this. Roger's gohack sketch sounds reasonable, and I'm sure we will learn more about exactly what's needed as we use all this in anger.

Russ

 


--
You received this message because you are subscribed to the Google Groups "golang-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+unsubscribe@googlegroups.com.

Russ Cox

unread,
Jul 17, 2018, 10:04:50 AM7/17/18
to Dan Kortschak, golang-dev
On Tue, Jul 17, 2018 at 7:47 AM, Dan Kortschak <dan.ko...@adelaide.edu.au> wrote:
On Mon, 2018-07-16 at 11:36 -0400, Russ Cox wrote:
> My priority here is to ensure a smooth transition for *late*
> adopters, ... 

The documentation for go mod falls down in this area. I was looking
through `go help mod` today trying to figure out how to resolve an
issue filed by one of our users and came to the conclusion that the
easiest way to deal with the problem was just manually edit the go.mod
file and then hope. This is not what I usually expect from Go docs.

Thanks for the feedback. I'll be the first to admit the docs are rough and that more docs are needed. Can you say a bit more about what problem you were trying to solve, so we can make the docs more helpful?

Also remember that right now you are an early adopter, not a late one. The way we're making the transition smooth for late adopters is not forcing them into modules in Go 1.11 at all, because we know there are various rough edges to sand down first.

Best,
Russ

Dan Kortschak

unread,
Jul 17, 2018, 6:53:52 PM7/17/18
to Russ Cox, golang-dev
The issue arose from a problem with a buggy dependency that was left in
gonum/gonum and how that interacted with gonum/plot which has
gonum/gonum as a dep.

We have a rarely used testing build tag that runs the mat package tests
using the CGo netlib blas and lapack implementations as the backing
implementation. Unfortunately we rarely run this test and when we moved
from github.com/gonum/... to gonum.org import paths, an error was made
in updating the import in that test file.

Now the problem arises that the go mod dep graph traversal seems to be
ignorant of build tags (this makes sense) and when it tried to traverse
this incorrect import the go mod invocation fails. I fixed the import
path in gonum/mat, but the gonum/plot go.mod still pointed to the bad
commit. The user attempted to fix this problem (which I didn't see,
because I'm actually not an early adopter - I'm doing this because they
are) but failed, adding a the netlib line that I don't want in the
go.mod and leaving gonum/plot's go.mod. It seemed reasonably clear to
me that what I wanted to do was just update the go.mod in gonum/plot to
point to a now fixed import. I read the docs, but (possibly due to
school holidays and the associated tiredness that comes with that) did
not find an easy way to do this (again, I'm not an early adopter, and
our model for managing deps in gonum is to generally just not have them
outside the project - there are a few cases in gonum/plot where this is
not true, and they have been a source of pain, so hopefully modules
will help here, but I digress). So, I just deleted the line pointing to
gonum/gonum and re-ran `go mod -sync` with the correct gonum/gonum
commit checked out. It seems to me that this is likely not the intended
way to do it, but it did work.

Part of the issue that I also faced is that I use GOPATH=$HOME, which
gave me "go mod: cannot use outside module", which I only was able to
decipher and resolve with the help of users from golang-nuts. This is
golang.org/issue/26365 filed by Dave.

thanks
Dan

Russ Cox

unread,
Jul 17, 2018, 8:05:35 PM7/17/18
to Dan Kortschak, golang-dev
Thanks for all the information. For future reference
(after solving the 'cannot use outside module' error)
the way to update those two known deps to latest versions is:



Dan Kortschak

unread,
Jul 17, 2018, 9:46:53 PM7/17/18
to Russ Cox, golang-dev
Thanks, Russ.

Dan Kortschak

unread,
Jul 18, 2018, 6:21:20 PM7/18/18
to Russ Cox, golang-dev
This does not work since there is no Go package at either of those
locations; they both have packages below the root.

Also, not that this is entirely directly related, but with the -u
option to go get (which would be the ideal way of handling the
dependency pull here) is not possible in the case
of gonum.org/v1/gonum/... since there is an "// import" comment in our
packages and they are checked out from git: rather than https://. I
know that softening the restriction on the scheme used has been
rejected in the past, but this is likely to be an increasing (though
already existing) point of irritation and friction when dealing with
dependency upgrading. At the moment it becomes a, sometimes tedious,
manual process of following deps and pulling them for updates. I'm
still not entirely clear why this restriction on scheme exists since
the // import comment does not specify the scheme anyway.

Dan

On Tue, 2018-07-17 at 20:05 -0400, Russ Cox wrote:

Dan Kortschak

unread,
Jul 19, 2018, 12:09:28 AM7/19/18
to Russ Cox, golang-dev
Correcting myself here - I did not set GO111MODULE=on for the
invocation. Doing that allows this to work. The remainder of the
previous mail stands.

This modification of go get by GO111MODULE=on is surprisingly non-
orthogonal.

Dan

Dan Kortschak

unread,
Jul 21, 2018, 1:26:09 AM7/21/18
to Russ Cox, golang-dev
An additional data point on this highlighting the non-orthonality; I
now have GO111MODULE=on as part of my env to avoid the problems that I
saw before, but this breaks just being able to do go get -u somepkg.

I wanted to update present for preparing some lectures and I get the
following:

```
~ $ go get -u golang.org/x/tools/cmd/present
go: cannot find main module root; see 'go help modules'
```

ISTM that this is due to overloading old go get function of getting go
packages with the new function of updating modules. I can fix the
behaviour here by unsetting GO111MODULE, but that won't work when it's
no longer an experiment and is always on as seems likely to happen in
the future.

I'm I doing this wrong?

Dan

Lars Seipel

unread,
Jul 21, 2018, 9:07:39 AM7/21/18
to Dan Kortschak, Russ Cox, golang-dev
On Sat, Jul 21, 2018 at 02:55:45PM +0930, Dan Kortschak wrote:
> ```
> ~ $ go get -u golang.org/x/tools/cmd/present
> go: cannot find main module root; see 'go help modules'
> ```
>
> ISTM that this is due to overloading old go get function of getting go
> packages with the new function of updating modules. I can fix the
> behaviour here by unsetting GO111MODULE, but that won't work when it's
> no longer an experiment and is always on as seems likely to happen in
> the future.
>
> I'm I doing this wrong?

No, this is how it's intended to work for now. The behaviour for a
future where module support is no longer experimental still needs to be
fleshed out.

In the interim, you may want to make it convenient for yourself to
toggle the GO111MODULE variable. Define a command that executes the go
program with the env var set to on. You could name it vgo.☺

Dan Kortschak

unread,
Jul 21, 2018, 7:32:25 PM7/21/18
to Lars Seipel, Russ Cox, golang-dev
Thanks. I'll do that.

Russ Cox

unread,
Jul 23, 2018, 10:33:52 AM7/23/18
to Dan Kortschak, golang-dev
On Sat, Jul 21, 2018 at 1:25 AM, Dan Kortschak <dan.ko...@adelaide.edu.au> wrote:
An additional data point on this highlighting the non-orthonality; I
now have GO111MODULE=on as part of my env to avoid the problems that I
saw before, but this breaks just being able to do go get -u somepkg.

Unless you are trying to be a very early (bleeding edge) adopter, I would suggest not setting GO111MODULE at all, not even in a shell script like Lars described. Instead, put your module-aware checkouts outside GOPATH/src.

Russ

Jan Mercl

unread,
Jul 23, 2018, 11:17:22 AM7/23/18
to Russ Cox, Dan Kortschak, golang-dev
On Mon, Jul 23, 2018 at 4:33 PM Russ Cox <r...@golang.org> wrote:

> Unless you are trying to be a very early (bleeding edge) adopter, I would suggest not setting GO111MODULE at all, not even in a shell script like Lars described.
> Instead, put your module-aware checkouts outside GOPATH/src.

I believe the suggestion make sense. However, so far I've studied almost nothing about the upcoming Go modules - shame on me. Is there a ELI5 summary why one has to move outside of $GOPATH? TBH, I'm increasingly concerned about this requirement (move out of GOPATH). I saw already elsewhere, asked a particular question (how to find a package's repo location at run time), but IIRC, never got an answer.

When vendoring came, I have no use direct for it, but it was easy to just ignore it. With modules this seems no more to be an option. The property, that import paths can be mechanically translated to file paths under $GOPATH/src and vice versa is IMHO a valuable one. Is it really unavoidable to lose this property (while providing and/or consuming Go modules in 1.11)? Or am I, which is far more probable, completely missing something again?

Thanks in advance for any enlightenment. It will be very appreciated.

--

-j

Sam Whited

unread,
Jul 23, 2018, 12:32:17 PM7/23/18
to golan...@googlegroups.com
On Mon, Jul 23, 2018, at 09:33, Russ Cox wrote:
> Unless you are trying to be a very early (bleeding edge) adopter, I would
> suggest not setting GO111MODULE at all, not even in a shell script like
> Lars described. Instead, put your module-aware checkouts outside GOPATH/src.

Unfortunately most other tools don't support modules yet, so if you want to be an early adopter and test out modules, but also want to use gosec or similar you'll need to do some GO111MODULE magic.
I've struggled with this quite a lot in CI.

—Sam

Dan Kortschak

unread,
Jul 23, 2018, 7:51:16 PM7/23/18
to Russ Cox, golang-dev
Thanks Russ,

Yeah, I don't want to be an early adopter, but was forced to be by a
Gonum issue. Moving things outside GOPATH/src for dealing with
occasional issues is not really a palatable solution. I will hope that
none of these issue arise in the future.

Dan

Axel Wagner

unread,
Jul 24, 2018, 5:49:53 AM7/24/18
to Jan Mercl, golang-dev
Hi Jan,

On Mon, Jul 23, 2018 at 5:17 PM Jan Mercl <0xj...@gmail.com> wrote:
The property, that import paths can be mechanically translated to file paths under $GOPATH/src and vice versa is IMHO a valuable one.

I don't think this property ever held in any sense of generality. $GOPATH could have mulitple elements, $GOPATH can change between build-time and runtime (and usually™ does), vendoring can break the mapping and the source might not even be available at runtime. So you already had to make a bunch of assumptions to make this work, now you just have to slightly modify them.

Personally I'd hope that https://godoc.org/golang.org/x/tools/go/packages provides a more robust way to solve this usecase more robustly though.

Is it really unavoidable to lose this property (while providing and/or consuming Go modules in 1.11)? Or am I, which is far more probable, completely missing something again?

Thanks in advance for any enlightenment. It will be very appreciated.

--

-j

--
You received this message because you are subscribed to the Google Groups "golang-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+...@googlegroups.com.

Brendan Tracey

unread,
Jul 26, 2018, 4:33:31 PM7/26/18
to Axel Wagner, Jan Mercl, golang-dev

On Jul 24, 2018, at 4:49 AM, 'Axel Wagner' via golang-dev <golan...@googlegroups.com> wrote:

Hi Jan,

On Mon, Jul 23, 2018 at 5:17 PM Jan Mercl <0xj...@gmail.com> wrote:
The property, that import paths can be mechanically translated to file paths under $GOPATH/src and vice versa is IMHO a valuable one.

I don't think this property ever held in any sense of generality. $GOPATH could have mulitple elements, $GOPATH can change between build-time and runtime (and usually™ does), vendoring can break the mapping and the source might not even be available at runtime. So you already had to make a bunch of assumptions to make this work, now you just have to slightly modify them.

If I want to make tools that work for everyone’s code, this is true, but if I just want to know about code that I’m running on my computer, it’s easy to control $GOPATH has one value and thus how to find source code from an import statement.

Ewan Chou

unread,
Jul 26, 2018, 9:58:25 PM7/26/18
to golang-dev
I tried to build a project but failed because it depends on a repo that has some of the generated code ignored in git.

In GOPATH mode, I can generate the code first, then build, but in modules mode, I can't do that.

Will `go modules` support `go:generate` in the future?

Dave Cheney

unread,
Jul 26, 2018, 10:39:17 PM7/26/18
to Ewan Chou, golang-dev
> Will `go modules` support `go:generate` in the future?

No, for the same reason that go get does not execute go generate as
part of checking out a repo, I imagine that go modules will not
support this either.

IMO go:generate code should be checked into the repo by the author. It
should not be the responsibilty of those who depend on that code as
clients to be able to execute their go:generate steps.

Rob Pike

unread,
Jul 26, 2018, 11:01:11 PM7/26/18
to Dave Cheney, Ewan Chou, golang-dev
Dave, thanks for your support. That to check in in the results of go generate is the way to work is not just your opinion but part of the design and documentation. The client of your code might not have the tools to run the command that generates it, so the design and intent is to assume that they never do, to avoid problems.

-rob





> For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "golang-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+unsubscribe@googlegroups.com.

coocood

unread,
Jul 26, 2018, 11:06:26 PM7/26/18
to r...@golang.org, Dave Cheney, golan...@googlegroups.com
Ok, thanks for the clarification. 

On Fri, Jul 27, 2018 at 11:01 AM Rob Pike <r...@golang.org> wrote:
Dave, thanks for your support. That to check in in the results of go generate is the way to work is not just your opinion but part of the design and documentation. The client of your code might not have the tools to run the command that generates it, so the design and intent is to assume that they never do, to avoid problems.

-rob



On Fri, Jul 27, 2018 at 12:39 PM, Dave Cheney <da...@cheney.net> wrote:
> Will `go modules` support `go:generate` in the future?

No, for the same reason that go get does not execute go generate as
part of checking out a repo, I imagine that go modules will not
support this either.

IMO go:generate code should be checked into the repo by the author. It
should not be the responsibilty of those who depend on that code as
clients to be able to execute their go:generate steps.

On 27 July 2018 at 11:58, Ewan Chou <coo...@gmail.com> wrote:
> I tried to build a project but failed because it depends on a repo that has
> some of the generated code ignored in git.
>
> In GOPATH mode, I can generate the code first, then build, but in modules
> mode, I can't do that.
>
> Will `go modules` support `go:generate` in the future?
>
> --
> You received this message because you are subscribed to the Google Groups
> "golang-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "golang-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+...@googlegroups.com.

Ian Davis

unread,
Jul 27, 2018, 3:30:33 AM7/27/18
to golan...@googlegroups.com
On Fri, 27 Jul 2018, at 3:39 AM, Dave Cheney wrote:
> > Will `go modules` support `go:generate` in the future?
>
> No, for the same reason that go get does not execute go generate as
> part of checking out a repo, I imagine that go modules will not
> support this either.
>

I think this is confusing for the OP. I'm sure that go generate will continue to work with go modules but the go mod command will not automatically run go generate.

Ian

Filippo Valsorda

unread,
Aug 14, 2018, 8:52:48 PM8/14/18
to Russ Cox, golang-dev
On Mon, Jul 16, 2018 at 11:36 AM Russ Cox <r...@golang.org> wrote:
[...] Note that it would be entirely reasonable to start using modules and Go 1.11 but also run 'go mod -vendor' to populate the vendor directory for users on older Go releases. In that case you'd want the CI system to keep testing the vendor directory. [...]

I wanted to followup on this in a variation of "I told you so" I'll call "you told me so".

The hybrid workflow using GO111MODULE=on locally (and maybe in a Makefile for off-GOPATH clones), and "go mod vendor" to support older versions, CI, and users who just "go get" in 1.11 is working very well for me. No need to tell my users to do anything differently, whichever way they do it it works.

👏

Olivier Mengué

unread,
Aug 29, 2018, 10:23:04 AM8/29/18
to golang-dev
Here is an issue I have about Go modules in combination with "go generate".

TL;DR: all go dependencies of go code inside a module are merged in a single section of go.mod and those dependencies are propagated from module to module down to the main module.

There are 3 kinds of Go files in a directory:
- main code of the package (including pure documentation file such as the idiomatic "doc.go")
- test code
- code generation tools, usually marked with "//+build ignore" in order to be launched with "go run" and triggered by a "go generate" command

When I run "GO111MODULE=on go generate" in my package github.com/dolmen-go/endian, a dependency on github.com/dolmen-go/codegen is added to go.mod.
I like the new behaviour that the precise version of that dependency is recorded in order to be able to reproduce the "go generate" later using the same toolchain (or decide to explicitely upgrade it) [1]. 
But this dependency should not be a concern for the users of github.com/dolmen-go/endian. Those dependencies injected during "go generate" are not useful to build the final binary.

In go.mod, there is no way to distinguish if a dependency is used for the main code, for testing or for code generation. The consequence is that the "go generate" dependency is just propagated by build tools down to the main module while the code is not used during the build. This dependency is useless for the main binary.

So the dependencies for "go generate" are currently in a dark corner that should be addressed before a stable release of go modules.

Olivier.

[1] But the dependency is marked "indirect" and so "GO111MODULE=on go mod tidy" removes it, which is inconsistent

Paul Jolly

unread,
Aug 29, 2018, 10:43:50 AM8/29/18
to olivier...@gmail.com, golan...@googlegroups.com
Hi Olivier,

Users of github.com/dolmen-go/endian will not be influenced/bothered
by its program (main) dependencies, because they (the program
dependencies) are not part of the transitive set of (test) package
dependencies of an import of github.com/dolmen-go/endian/... packages.

Hence the existence of such dependencies in
github.com/dolmen-go/endian's go.mod is harmless to users of
github.com/dolmen-go/endian, but, as you point out, extremely
beneficial to developers of github.com/dolmen-go/endian.


Paul

Olivier Mengué

unread,
Aug 29, 2018, 11:05:39 AM8/29/18
to golang-dev
I disagree that users will not be bothered: the "go generate" dependency is propagated (as "indirect") in the go.mod of the main package. So the maintainer who updates dependencies when new releases appear has to check that dependency while it is not involved in the build. 

Here is a sample project:

$ cat main.go
cat main.go
package main

import (
"fmt"
)

func main() {
fmt.Println(endian.Native)
}

$ GO111MODULE=on go mod init main
go: creating new go.mod: module main
$ GO111MODULE=on go mod tidy
go: finding github.com/dolmen-go/endian v1.0.0
go: downloading github.com/dolmen-go/endian v1.0.0
$ cat go.mod
module main

require (
github.com/dolmen-go/codegen v1.0.0 // indirect
)

Paul Jolly

unread,
Aug 29, 2018, 11:14:10 AM8/29/18
to olivier...@gmail.com, golan...@googlegroups.com
Hi Oliver,

This happens because github.com/dolmen-go/endian contains the
build-tag-"ignored"
https://github.com/dolmen-go/endian/blob/master/generate.go which
imports github.com/dolmen-go/codegen. However, the total set of
dependencies is agnostic of build tags, hence
github.com/dolmen-go/codegen is a dependency of
github.com/dolmen-go/endian.

If instead https://github.com/dolmen-go/endian/blob/master/generate.go
were a main package of its own (which imported
github.com/dolmen-go/codegen) then you would not be seeing this.

Thanks,


Paul

Axel Wagner

unread,
Aug 29, 2018, 11:52:59 PM8/29/18
to golang-dev
It's interesting though, that if you'd ignore build tags, the code wouldn't build, because of different package names. In a way, build tags are both ignored and not ignored in this case.

FWIW, I use the same pattern (having a foo.go with package main and +build ignore) relatively often. I'd be okay with putting this in a separate directory in the future. But I also think there is an argument to be made about the inherent contradiction of having different package names. And that maybe this should work as-is.

jerome....@gmail.com

unread,
Sep 10, 2018, 10:48:01 AM9/10/18
to golang-dev
Hello all,
On our inner gitlab, I have just try go mod with linux alpine with docker golang:1.11-alpine

I run this:
$ (cd module_path; go install -v -mod vendor ./...)
net
# net
exec: "gcc": executable file not found in $PATH

I don't have this error before I use go modules

Le jeudi 12 juillet 2018 23:20:40 UTC+2, rsc a écrit :
As of CL 123580, submitted a few minutes ago, the development copy of the go command, in the main go repository, has support for Go modules. There are various minor known issues, but quite a lot works very well. 

There's still more to be done but this is a key milestone: if you run on the bleeding edge, you can play with modules just by using "go". Work on the module support will now be done exclusively in the main repo. We will automatically export the main repo to the x/vgo repo when we have a snapshot we want to make available to people using "vgo".

The open issues have moved from the vgo milestone to the Go 1.11 milestone, but labeled with the new 'modules' label.

A note on terminology: Go 1.11 is not shipping with "vgo support". It is shipping with "Go module support". Vgo was only ever the prototype/demo of that module support. Just like all the other features we've introduced to the go command over the past ten years, the goal for this one is to fit so well into the rest of the go command fabric that the seams are invisible - module support is just part of "go", the same way that, say, race detector support or GOPATH support is.

Speaking of GOPATH, the go command automatically chooses between using modules and using GOPATH depending on where it is run. If a go command runs outside GOPATH/src and there is a go.mod file in the current directory or one of its parent directories, then modules are enabled (otherwise not):

$ mkdir /tmp/hello
$ cd /tmp/hello
$ echo module hello >go.mod
$ cat >hello.go <<EOF
package main 

import "rsc.io/quote"

func main() {
println(quote.Glass())
}
EOF
$ go run .
I can eat glass and it doesn't hurt me.
$ cat go.mod
module hello

require rsc.io/quote v1.5.2
$ go list -m all
hello
golang.org/x/text v0.0.0-20170915032832-14c0d48ead0c

For more information, see 'go help modules', which I've included at the end of this mail.

Many thanks to everyone in the community who has helped us get to this point, but especially to Paul Jolly for really hammering on the module support and filing tons of great bugs, and to Baokun Lee for a steady sequence of helpful CLs. Thanks also to Bryan Mills for helping work out a lot of important details in the past few weeks. 

Thanks again, and keep the bug reports coming.

Best,
Russ




go help modules

A module is a collection of related Go packages.
Modules are the unit of source code interchange and versioning.
The go command has direct support for working with modules,
including recording and resolving dependencies on other modules.
Modules replace the old GOPATH-based approach to specifying
which source files are used in a given build.

Experimental module support

Go 1.11 includes experimental support for Go modules,
including a new module-aware 'go get' command.
We intend to keep revising this support, while preserving compatibility,
until it can be declared official (no longer experimental),
and then at a later point we may remove support for work
in GOPATH and the old 'go get' command.

The quickest way to take advantage of the new Go 1.11 module support
is to check out your repository into a directory outside GOPATH/src,
create a go.mod file (described in the next section) there, and run
go commands from within that file tree.

For more fine-grained control, the module support in Go 1.11 respects
a temporary environment variable, GO111MODULE, which can be set to one
of three string values: off, on, or auto (the default).
If GO111MODULE=off, then the go command never uses the
new module support. Instead it looks in vendor directories and GOPATH
to find dependencies; we now refer to this as "GOPATH mode."
If GO111MODULE=on, then the go command requires the use of modules,
never consulting GOPATH. We refer to this as the command being
module-aware or running in "module-aware mode".
If GO111MODULE=auto or is unset, then the go command enables or
disables module support based on the current directory.
Module support is enabled only when the current directory is outside
GOPATH/src and itself contains a go.mod file or is below a directory
containing a go.mod file.

Defining a module

A module is defined by a tree of Go source files with a go.mod file
in the tree's root directory. The directory containing the go.mod file
is called the module root. Typically the module root will also correspond
to a source code repository root (but in general it need not).
The module is the set of all Go packages in the module root and its
subdirectories, but excluding subtrees with their own go.mod files.

The "module path" is the import path prefix corresponding to the module root.
The go.mod file defines the module path and lists the specific versions
of other modules that should be used when resolving imports during a build,
by giving their module paths and versions.

For example, this go.mod declares that the directory containing it is the root
of the module with path example.com/m, and it also declares that the module
depends on specific versions of golang.org/x/text and gopkg.in/yaml.v2:

require (
)

The go.mod file can also specify replacements and excluded versions
that only apply when building the module directly; they are ignored
when the module is incorporated into a larger build.
For more about the go.mod file, see https://research.swtch.com/vgo-module.

To start a new module, simply create a go.mod file in the root of the
module's directory tree, containing only a module statement.
The 'go mod' command can be used to do this:

go mod -init -module example.com/m

In a project already using an existing dependency management tool like
godep, glide, or dep, 'go mod -init' will also add require statements
matching the existing configuration.

Once the go.mod file exists, no additional steps are required:
go commands like 'go build', 'go test', or even 'go list' will automatically
add new dependencies as needed to satisfy imports.

The main module and the build list

The "main module" is the module containing the directory where the go command
is run. The go command finds the module root by looking for a go.mod in the
current directory, or else the current directory's parent directory,
or else the parent's parent directory, and so on.

The main module's go.mod file defines the precise set of packages available
for use by the go command, through require, replace, and exclude statements.
Dependency modules, found by following require statements, also contribute
to the definition of that set of packages, but only through their go.mod
files' require statements: any replace and exclude statements in dependency
modules are ignored. The replace and exclude statements therefore allow the
main module complete control over its own build, without also being subject
to complete control by dependencies.

The set of modules providing packages to builds is called the "build list".
The build list initially contains only the main module. Then the go command
adds to the list the exact module versions required by modules already
on the list, recursively, until there is nothing left to add to the list.
If multiple versions of a particular module are added to the list,
then at the end only the latest version (according to semantic version
ordering) is kept for use in the build.

The 'go list' command provides information about the main module
and the build list. For example:

go list -m              # print path of main module
go list -m -f={{.Dir}}  # print root directory of main module
go list -m all          # print build list

Maintaining module requirements

The go.mod file is meant to be readable and editable by both
programmers and tools. The go command itself automatically updates the go.mod file
to maintain a standard formatting and the accuracy of require statements.

Any go command that finds an unfamiliar import will look up the module
containing that import and add the latest version of that module
to go.mod automatically. In most cases, therefore, it suffices to
add an import to source code and run 'go build', 'go test', or even 'go list':
as part of analyzing the package, the go command will discover
and resolve the import and update the go.mod file.

Any go command can determine that a module requirement is
missing and must be added, even when considering only a single
package from the module. On the other hand, determining that a module requirement
is no longer necessary and can be deleted requires a full view of
all packages in the module, across all possible build configurations
(architectures, operating systems, build tags, and so on).
The 'go mod -sync' command builds that view and then
adds any missing module requirements and removes unnecessary ones.

As part of maintaining the require statements in go.mod, the go command
tracks which ones provide packages imported directly by the current module
and which ones provide packages only used indirectly by other module
dependencies. Requirements needed only for indirect uses are marked with a
"// indirect" comment in the go.mod file. Indirect requirements are
automatically removed from the go.mod file once they are implied by other
direct requirements. Indirect requirements only arise when using modules
that fail to state some of their own dependencies or when explicitly
upgrading a module's dependencies ahead of its own stated requirements.

Because of this automatic maintenance, the information in go.mod is an
up-to-date, readable description of the build.

The 'go get' command