go build, go list, go test - stop updating go.mod by default?

[Russ Cox]

Hi all,

I don't want to intrude on this list (which I'm not on), but I wonder if I could get your opinion on something.

As you probably know, we're trying to make Go 1.16 the release when there are no blockers remaining for anyone to adopt modules, including things like "go get -b" which Jay has been working on.

I'm wondering, as part of this polishing of modules, if we should change the default behavior and stop doing so much updating of go.mod. 

Obviously I can justify the current behavior at length, and have in the past, but I think the past thinking falls apart a bit when users make mistakes. 

For example, I was working in a module recently (let's call it m) and had a package (let's call it p), and I decided to rename the package (to q). I caught most of the m/p import paths and updated them to m/q. But not all, and the next time I ran "go build", the go command went off trying to find some other module to provide the no-longer-existent m/p. When m/p is a real thing that exists, automatically updating go.mod to use it makes some sense. When m/p is a mistake that needs updating, doing network work just slows down the real fix. 

Another example is adding an import you didn't actually want. You might copy a file into a module m from somewhere else, and when you "go build" it adds something to go.mod silently, perhaps even changing the versions of existing require statements. In this case - since you don't want the import - it would be better for the tool to stop and tell you about it instead of doing additional damage that you will have to undo.

I suspect there are many more of these. In general, while the automatic additions to go.mod seemed good on paper, my impression is that they haven't delivered the vision we wanted: they do the wrong thing too often, they slow down the compile-edit-debug cycle for typos, and they are confusing.

We already have a mode that reports errors instead of making changes, namely -mod=readonly. That flag does not apply to "go get" or "go mod" - those commands' job is to update go.mod and they would still do that - but it stops other commands like "go list", "go build", "go test" from changing go.mod. If we made this mode the default and also cleaned up some of its rough edges (go list can probably do a better job reporting partial results, for example), that might go a long way to fixing the problems I listed above. 

If we do change the default, it would probably make sense to change -mod=mod (the name for the current automatic updating) to something like -mod=autoupdate, leaving -mod=mod as an alias for that of course.

The command "go get" (with no arguments) has always done a build of the package in the current directory, including downloading any necessary dependencies. With a default -mod=readonly, "go get" would become the way to do a one-time fix-up of go.mod. If "go build" or "go list" fails because of a missing requirement or other problem with go.mod, "go get" would be the way to fix it. If "go test" fails due to an import specific to the test, we'd need to provide a way to deal with that, probably adding back "go get -t", which was the pre-modules way to download the dependencies for a test.

Along with this I think you'd want to change goimports to have a way to say what new requirements it thinks go along with the import paths. Then tooling invoking goimports could present the new imports and the new go.mod requirements together. Users opting in to that kind of automation would still have the option of both. It just wouldn't happen automatically in commands like "go build". (This listing of new go.mod requirements only matters when goimports has exhausted your module's current requirements and is grubbing around in your module cache. When it finds a match, especially a v0, you probably do want to get the version it found and not whatever the go command decides is latest from the internet. So letting goimports specify the new go.mod requirements would correct a sort of race in the current behavior anyway.)

I haven't heard many people suggesting we should default to -mod=readonly before, so maybe others haven't seen the current behavior as too much of a problem. But maybe people just figure it's not worth bringing up because it can't change at this point. I don't believe that's true - if the behavior is wrong, then Go 1.16 would be the time to get it right (and soon).

What do you all think? Thanks for your time and input.

Best,

Russ

[Marwan Sulaiman]

Hi Russ,

Thank you for the suggestion. Overall, I think this is a good idea

I would like to add a couple of scenarios that are worth mentioning: 

1. Most people I've worked with aren't aware of the -mod=readonly option in the first place therefore their build pipelines don't include this flag. I'm guilty of not including -mod=readonly in most of my projects as well. Therefore, having -mod=readonly by default is probably a good security decision as well. 

2. Accidentally adding a new import path could affect your entire build list through its transitive dependencies and removing that one import path directly from the go.mod file may not be enough. If your main module A depends on a go.mod-less module, say B@v1.2, and you accidentally introduce module C which depends on B@v1.3, then your own go.mod file will now have upgraded module B from 1.2 to 1.3. Therefore, simply removing module C from go.mod does not put you in the same reproducible build from before the accidental import path addition. You can possibly git-revert a go.mod file, assuming you are working in a git repository, but if you were already working on a feature where you had a go.mod file altered, it becomes a lot more difficult. 

3. It's also worth noting that "gopls" suggests adding an import path to your go.mod file when you add an import path to a Go file. Therefore, the developer flow (at least for gopls users) should already prevent you from accidentally calling "go build" once you have pasted a new path in a Go file. 

4. On the other hand, I have gotten used to putting a git branch name in my "go.mod" file for a specific import path and running "go build" to let the Go command take care of module resolution and updating the go.mod file into the proper pseudo semver. I'll just have to break that habit and switch to "go get" instead. 

Finally, it would be ideal if the Go command gave the user clear output as to why the build failed and maybe even suggest a copy-pastable fix so that they can run the fix commands and try building again. 

Marwan,

--
You received this message because you are subscribed to the Google Groups "golang-tools" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-tools...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-tools/e9f5fb8b-ba47-4235-8f09-41c70b97561en%40googlegroups.com.

--

Marwan Sulaiman

Software Developer

Mobile: 203-722-5772

[roger peppe]

Big +1 from here. I would have suggested this before except that it seemed like such a strongly justified assumption that there would be no chance of it being changed. The long wait while it tries to find a nonexistent submodule when you've spelled a package name wrong is indeed often annoying. Also, the fact that it often reaches out to the network unbidden is not great if disconnected or on a slow network connection.

I think that dependencies shouldn't be lightly added and this seems like a good way to go in that respect. A plain "go get" to fetch any dependencies that you might want also seems like a good idea.

Having arbitrary side-effects from apparently read-only operations such as "go list" has always seemed somewhat odd to me. This change would make me much happier.

  cheers,

    rog.

--

[Mathieu Lonjaret]

Hello,

Not many constructive arguments to add here, but like the others a
"yes please" overall.
On enough occasions to annoy me has go.mod been modified without me
understanding why, or being happy with the result, that to me the less
often it gets touched, the better.
And in any case, I think it's better to err on the side of caution (go
tools not doing "enough", and the user having to finish the work
manually), rather than the go tools trying harder, and getting it
wrong.

Thanks,
Mathieu

[Elias Naur]

Yes, please do stop automatic updates to go.* files without explicit `go get`. As an experience report, I have "GOFLAGS=-mod=readonly` in my shell profile to avoid surprising updates.

FWIW, convincing arguments for your proposal can be found in your own https://research.swtch.com/deps article about being careful and deliberate when adding dependencies. `go build`, `go list` etc. automatically adding dependencies goes counter to your suggestions.

Elias

[Jay Conrod]

Since we're on topic of changing 'go install', I wonder what everyone thinks about moving the build-and-install functionality from 'go get' into 'go install'. For example, that might mean:

• 'go install example.com/pkg@version' would build and install an executable in module mode and would ignore the go.mod file in the working directory (or parent directories) if there is one. It would have the same semantics proposed for 'go get -b' in #40276.
• 'go install example.com/pkg' (without a version suffix) would not change meaning, except for defaulting to -mod=readonly as suggested above.
• The 'go get -d' flag would always be on. 'go get' would not build anything.

The actual spelling of these commands might change; I'm more curious if folks want a change like this. It's an incompatible change to the 'go get' CLI, but as long as we're making changes to the CLI, we should try to batch them all in 1.16. It would be great to get a CLI we're happy with for the long term.

--

You received this message because you are subscribed to the Google Groups "golang-tools" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-tools...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/golang-tools/0ee7295f-1e77-4be6-9d35-6a73c623ab32n%40googlegroups.com.

[Marwan Sulaiman]

I could have sworn that the idea of "go install" becoming a "go get -b" was suggested in the past (can't find the issue). I think Jay that's probably why you mentioned that "go get <path>" without an "@version" would remain the same as before? 

In any case, I always thought "go install" made more sense to be "go get -b" mainly because the name feels right.

However, if I understand your suggestion correctly, does that mean if I run "go install <path>" (without "@version") from within a go.mod and a -mod=readonly, that it would fail? Or would it update the go.mod? If it fails (with a good error message), I think it's a good decision because it would force the developer to use the correct semantics: "either use go get if you wanna update your go.mod, or go install <path>@<version> if you wanna just install the binary outside of your go.mod environment". 

However, if it just updates the "go.mod", I think that's going to be pretty confusing to developers because why "@version" doesn't update the go.mod file while omitting "@version" updates the go.mod file. It's easily missed and it feels more like a side-effect than an explicit feature. 

On another thought, can't we make "go install" work like "go get -b" with or without "@version" in module mode? Here are all the scenarios I can think of: 

1. Running "go install" from within a main-package directory without any additional <path> arguments would work the same as today. It's quite common for people to do that during development. For example, I very frequently install gopls from master that way. 

2. Running "go install <path>" from within a go.mod where <path> is a filesystem path: this should work the same as the above. 

3. Running "go install <path>" from within a go.mod project where <path> is an import path, then this should be a "go get -b". If a user wanted to have their go.mod updated AND install the binary, they can just do "go get <path>". Hopefully I'm not missing something here.

4. Running "go install <path>@<version>" from within a go.mod project, would be the same as above except you want a specific version. 

All of this would avoid us having to tell users "make sure you add @latest" to your "go install" so that it doesn't affect your go.mod file.

Thanks!

To view this discussion on the web visit https://groups.google.com/d/msgid/golang-tools/CAGCADbafzPA3M6uhASZz2o-vDfwjJtRwkKfvK5iQHN0HKRGNrw%40mail.gmail.com.

[Jay Conrod]

> I could have sworn that the idea of "go install" becoming a "go get -b" was suggested in the past (can't find the issue). I think Jay that's probably why you mentioned that "go get <path>" without an "@version" would remain the same as before?

For sure, I take no credit for this idea. I've seen a lot of variations on this, but I didn't think they were viable since it's a pretty big breaking change.

> However, if I understand your suggestion correctly, does that mean if I run "go install <path>" (without "@version") from within a go.mod and a -mod=readonly, that it would fail?

What I was suggesting is that the @version suffix (which is an error right now) would cause 'go install' to operate in a global mode, where it builds and installs a package from a module at a specific version, ignoring the module in the working directory. 'go install' without a version suffix would still run in the context of the current module. It would be an error to install a package not provided by any module in the build list (because -mod=readonly). You'd be able to run 'go install' without arguments or with relative paths to install packages from the main module.

I'm not sure this is the right interface, but I think there should be some explicit marker on the command line so we know what will happen without knowing the contents of the working directory. That makes it easier to write commands in documentation and in scripts. For that reason, I think 'go install pkg' outside of a module without a @version suffix should be an error: it would have a different meaning inside a module, which is a problem we now have with 'go get'.

Your suggestion of 'go install pkg' (where pkg is a full package path, not a local path) satisfies that. It changes the meaning of an existing command, but maybe that's okay? It makes it more difficult to reproducibly build and install tools depended on by the current module. For example, a build script that runs 'go install google.golang.org/protobuf/cmd/protoc-gen-go' would install the latest version, rather than the version of 'google.golang.org/protobuf' required by go.mod.

Paul Jolly suggested an alternative in the #tools channel on Slack: the 'go install -g' flag would enable a "global" mode, like 'go get -b'. With that flag, the @version suffix wouldn't be necessary. We talked about -g last year in #30515; this is a somewhat narrower interpretation (only for 'go install').

[Russ Cox]

On Fri, Aug 7, 2020 at 1:52 PM 'Jay Conrod' via golang-tools <golang...@googlegroups.com> wrote:

> I could have sworn that the idea of "go install" becoming a "go get -b" was suggested in the past (can't find the issue). I think Jay that's probably why you mentioned that "go get <path>" without an "@version" would remain the same as before?

For sure, I take no credit for this idea. I've seen a lot of variations on this, but I didn't think they were viable since it's a pretty big breaking change.

I think it's important to separate out the steps here a little bit more. Please correct me if I've gotten any of this wrong.

1. If we make go operations not update go.mod automatically, we need a clear way to say "fix up go.mod".

The obvious answer is "go get" (with no args), which almost works for that today.

That command does fix up go.mod, but it also does a build of the package, and for a package main does an install into a bin directory ($GOBIN / $GOPATH/bin / $HOME/go/bin).

That's an unfortunate overloading, so Jay suggests maybe it would make sense to do a one-time breaking of command lines and make get stop installing binaries ever.

Then "go get" (with no args) is unambiguously the "fix up go.mod" command, and more generally "go get" only ever modifies go.mod; it does not build and does not install anything.

2. If we stop making "go get" install to bin directories, then we should double-check:
is "go get -b" still the right spelling for the "isolated install of binary" operation?

Jay suggests maybe instead we introduce "go install path@version" to mean what we had been calling "go get -b path@version".

3. Marwan suggests maybe when path is not a relative path, "go install path" can mean what we had been calling "go get -b path@latest".

4. Paul suggests perhaps "go install -g path@version" would be the new spelling of "go get -b path@version", with "go install -g path" meaning "go get -b path@latest".

Having done that, a few thoughts.

Regarding (3), I would be reluctant to start changing the behavior of go commands based on the form used to name a package. 

If I'm in the root directory of module m containing a foo subdirectory, then I really expect these command pairs to do the same things:

    go test ./foo

    go test m/foo

    go list ./foo

    go list m/foo

    go build ./foo

    go build m/foo

    go install ./foo

    go install m/foo

It would be very strange for that last one to mean "m/foo@latest". Same thing if we're talking about dependencies in other modules known to go.mod.

It would be weird for "go test my/dep/foo" to test the version implied by go.mod but "go install my/dep/foo" implicitly means @latest.

Redefining the behavior of existing "go install" commands seems to me unwise.

I don't see any way to special case "go install path" without introducing some very sharp jagged edges in the command-line behavior.

In particular, today it's always the case that "go anything path" and "go anything $(go list path)" do the same thing,

except for "go get", whose job is basically to break that rule.

I would be reluctant to break that rule in other commands, especially "go install".

On the other hand, "go install path@version" is not valid today and exists on a separate syntactic plane from these other commands,

so there's no breakage, nor are there jagged egdges with the existing commands. 

We already allow "go list -m path@version" (naming a module).

If we add "go install path@version", we should probably also add "go list path@version" (naming a package).

Regarding (4), the flag -g ends up mostly redundant compared to having the flag-free "go install path@version".

The one time it is not is "go install -g path", which would otherwise have to be written "go install path@latest".

At that point the difference between "go install path" and "go install -g path" is "-g means latest version".

(I'm oversimplifying, but so will users who don't understand all these details.)

The argument for both (3) and (4) over (2) seems to be to shorten the "go install path@latest"

down to something where you needn't type the seven characters "@latest".

But comparing "go install path@latest" to "go install -g path",

I appreciate how much the former makes clear that the specific version is time-dependent.

To the extent that blindly grabbing the latest version of something

is a non-reproducible and somewhat risky operation, it's nice to have that more clearly spelled out.

One more thing that occurred to me as I wrote this: as far as (1) is concerned,

we could consider doing a gradual roll-out for that change, something like:

 - Go 1.16:

    - add go install path@version (or whatever the final spelling is)

    - make go get path print a warning when path is a main package:

        go get: in future Go versions, "go get" will not install binaries; use "go install" for that

 - Go 1.17: 

    - go get stops printing the warning & stops installing the binary

Go 1.16 (in this example) would serve as a transition period when both the old and new

"get me a binary" commands work, so that people have a six-month window

to learn the new commands and update scripts instead of a flag day.

Best,

Russ

[Patrick Baxter]

I am also in the "yes please" camp for both making go get the only way to update the mod file and for making `go get -b path@version` just be `go install path@version`. I think cleanly separating go get and go install to be for managing go.mod and installing binaries respectively is a massive ux win. 

Agreed with Russ that `go install path@latest is better then `go install -g path`, makes the action being done more clear. Also I like Jay's thinking that go install without a version is only useful for building a local module from inside it. Otherwise, it requires a version for its global install function.

Very excited about this development.

[Sean Liao]

On Friday, August 7, 2020 at 9:06:44 PM UTC+2 Russ Cox wrote:

1. If we make go operations not update go.mod automatically, we need a clear way to say "fix up go.mod".

The obvious answer is "go get" (with no args), which almost works for that today.

A minor point, but I thought the obvious command to fix go.mod would be "go mod tidy"

[Jay Conrod]

'go mod tidy' will likely be what most people use, but it can delete requirements, so it may not always be the best tool to reach for.

'go get' (equivalent to 'go get .') will ensure that all packages needed to build the package in the current directory are provided by *some* module. 'go build' and other build commands would no longer have that effect, but 'go get' still would.

--

You received this message because you are subscribed to the Google Groups "golang-tools" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-tools...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/golang-tools/7590f7fd-2978-4441-9b23-6b2172de9481n%40googlegroups.com.

[Paul Jolly]

Thanks for the summary, Russ.

My point (which is arguably minor) about go install -g path[@version] was that the -g feels more like a clear instruction "install globally" (or whatever language we want to adopt) than the implicit "if there is a version specified then that means install globally". Particularly given go get (in a module context) works to modify the main module both with/without a version specified. 

--
You received this message because you are subscribed to the Google Groups "golang-tools" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-tools...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/golang-tools/CAA8EjDTNjrmfJFf4MoisdXDUJ5EKij7gcBKoMSmdJPpejEw74A%40mail.gmail.com.

[Bryan C. Mills]

'go mod tidy' is also in general much more expensive than 'go get': 'go mod tidy' loads all of the transitive dependencies of the main module, whereas 'go get' only needs to load the dependencies of the named package(s) (by default, the package in the current directory).

To view this discussion on the web visit https://groups.google.com/d/msgid/golang-tools/CAGCADbZCvYdkXCruebQS2aGt8EWa0sMPTtV%2BSRCvgRKNsS7Now%40mail.gmail.com.

[Russ Cox]

On Mon, Aug 10, 2020 at 10:04 AM Paul Jolly <pa...@myitcv.io> wrote:

Thanks for the summary, Russ.

My point (which is arguably minor) about go install -g path[@version] was that the -g feels more like a clear instruction "install globally" (or whatever language we want to adopt) than the implicit "if there is a version specified then that means install globally". Particularly given go get (in a module context) works to modify the main module both with/without a version specified. 

For what it's worth, although I realize that you and some others have a clear meaning in your heads for "g = global", I do not. I can't say exactly what "global" means or why g would suggest "global" (as opposed to some other Go word). For example, global could mean in /usr/bin instead of $HOME/bin. Or it could mean in $HOME/bin instead of some more ephemeral place? But I can't see why it would mean what it actually seems to, namely "bypass any go.mod on my computer". In contrast to my confusion about -g = global = ???, "path@latest" seems like a clear instruction to bypass any versions listed in any go.mod.

(Maybe there is some meaning of "-g = global" in a different language's tool that I'm unaware of?)

Best,

Russ

[Marwan Sulaiman]

(Maybe there is some meaning of "-g = global" in a different language's tool that I'm unaware of?)

The one I know of and is probably most common is "npm install -g <path>" https://docs.npmjs.com/downloading-and-installing-packages-globally

As for: 

"path@latest" seems like a clear instruction to bypass any versions listed in any go.mod.

I'm still having a hard time wrapping my head around how "go install <path>" would affect the go.mod but "go install <path>@latest" wouldn't. Why would specifying a particular version should mean "bypass go.mod"?

But if I recall from earlier in this thread, "go install <path>" should actually fail in module mode right? If it did and Go outputted a nice human friendly message such as:

"go install <path> is disallowed in Module mode. If you'd like to install a Go program without affecting your go.mod file, run go install <path>@latest". 

Then this would make the argument for "@latest" a lot better I think. Because now there is only one way to do this in Module mode, and Go clearly states the behavior for it. 

Best,

Marwan

PS. Thanks again for the discussion and allowing us to engage in this feedback :)

[Patrick Baxter]

The way I understand it, go install is always building/installing a main package with the dependencies of that package's module's go.mod with this design. Its never modifying a local go.mod as far as I understand. The only difference using it without a version is that it requires you to be locally in the module directory of the specified package to build and would allow reletive paths. Not specifying a version, being outside a module, and in module mode would be an error I think.

So distinguishing between building from a local source and a source from the proxy seems better distinguished by inclusion of a version then a global flag. Or, at least, that seems more intuitive to me.

[Jay Conrod]

> The way I understand it, go install is always building/installing a main package with the dependencies of that package's module's go.mod with this design. Its never modifying a local go.mod as far as I understand. The only difference using it without a version is that it requires you to be locally in the module directory of the specified package to build and would allow reletive paths. Not specifying a version, being outside a module, and in module mode would be an error I think.

Not quite what I meant with my earlier message. There would still be a difference between 'go install' running within a module and "globally". I don't think we should change the current behavior within a module (except for defaulting to -mod=readonly). It would likely break a lot of workflows and scripts in very subtle ways.

To be clear, what I'm proposing is:

• 'go install pkg@version' would install pkg at version. It would ignore the go.mod in the current directory, if there is one. It would always run in module or would report an error if GO111MODULE=off.
• This takes the place of 'go get -b' in #40276. I've updated that proposal to use 'go install', and added examples and rationale.
• There are several more restrictions: all arguments must refer to packages in the same module at the same version, and its go.mod must not contain replace directives or anything else that would cause it to be interpreted differently than if it were the main module.
• 'go install pkg' within a module would install pkg at the version required in the module's go.mod. If no required module provides pkg, an error would be reported. This is close to the current behavior, except that -mod=readonly would be the default.
• 'go install pkg' outside of a module would run in GOPATH mode by default. If GO111MODULE=on, then 'go install pkg' will report an error.
• Note: 'go install pkg' is not equivalent to 'go install pkg@latest' to avoid the ambiguity we have today with 'go get'.

The current behavior is:

• 'go install pkg@version' is an error: versions aren't allowed.
• 'go install pkg' within a module installs pkg at the version required in the module's go.mod. If the required module provides pkg, a requirement is added to go.mod at the latest version.
• 'go install pkg' outside a module runs in GOPATH mode by default. If GO111MODULE=on, then 'go install pkg' reports an error.

On Wed, Aug 12, 2020 at 6:02 PM Patrick Baxter <patr...@gmail.com> wrote:

The way I understand it, go install is always building/installing a main package with the dependencies of that package's module's go.mod with this design. Its never modifying a local go.mod as far as I understand. The only difference using it without a version is that it requires you to be locally in the module directory of the specified package to build and would allow reletive paths. Not specifying a version, being outside a module, and in module mode would be an error I think.

So distinguishing between building from a local source and a source from the proxy seems better distinguished by inclusion of a version then a global flag. Or, at least, that seems more intuitive to me.

--

You received this message because you are subscribed to the Google Groups "golang-tools" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-tools...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/golang-tools/2e794026-91f1-4069-8d3f-008fbcad5d54n%40googlegroups.com.