Trying to use a tool in my build (a friction log)

[Tim Hockin]

First: go version go1.16 linux/amd64

In one of my side-projects, I build a container image. It turns out
to be useful to people, but I got a request to put the license files
for all code used in the binary into the container. Lawyers, what can
you do?

So these people sent me a PR to gather the licenses with `go install
github.com/google/go-licenses` in the `make container` rule. Here's
the friction.

1) Their PR includes dozens of new lines in go.sum, which makes me
grumpy. I set out to see if I can do better.

2) When I run that command on my workstation I get:

```
$ go install github.com/google/go-licenses
cannot find package "." in:
/home/thockin/src/go/src/k8s.io/git-sync/vendor/github.com/google/go-licenses
```

I don't know what to make of that message. It's not in vendor/ yet -
I want to put it there (I think?).

So I am left wondering - what is the normal flow for "I want to use
this Go tool in my build" ?

I tried `-mod=mod`:

```
$ (unset GOPATH; go install -mod=mod github.com/google/go-licenses)
go: writing go.mod cache: mkdir /home/thockin/go: not a directory
go: writing go.mod cache: mkdir /home/thockin/go: not a directory
go: writing go.mod cache: mkdir /home/thockin/go: not a directory
go: writing go.mod cache: mkdir /home/thockin/go: not a directory
go: writing go.mod cache: mkdir /home/thockin/go: not a directory
go: writing go.mod cache: mkdir /home/thockin/go: not a directory
go: writing go.mod cache: mkdir /home/thockin/go: not a directory
go: writing go.mod cache: mkdir /home/thockin/go: not a directory
go: writing go.mod cache: mkdir /home/thockin/go: not a directory
go: writing go.mod cache: mkdir /home/thockin/go: not a directory
go: writing go.mod cache: mkdir /home/thockin/go: not a directory
go: writing go.mod cache: mkdir /home/thockin/go: not a directory
go: finding module for package github.com/google/go-licenses
go: downloading github.com/google/go-licenses v0.0.0-20201026145851-73411c8fa237
mkdir /home/thockin/go: not a directory
```

I keep hearing that GOPATH is dead (and frankly good riddance) but
this tries to write to GOPATH or ~/go (which is also annoying). I
don't want this project's build spilling over into my homedir.

I tried `go get` but it's the same.

```
$ (unset GOPATH; go get github.com/google/go-licenses)
go: writing go.mod cache: mkdir /home/thockin/go: not a directory
go: writing go.mod cache: mkdir /home/thockin/go: not a directory
go: writing go.mod cache: mkdir /home/thockin/go: not a directory
go: writing go.mod cache: mkdir /home/thockin/go: not a directory
go: writing go.mod cache: mkdir /home/thockin/go: not a directory
go: writing go.mod cache: mkdir /home/thockin/go: not a directory
go: writing go.mod cache: mkdir /home/thockin/go: not a directory
go: writing go.mod cache: mkdir /home/thockin/go: not a directory
go: writing go.mod cache: mkdir /home/thockin/go: not a directory
go: writing go.mod cache: mkdir /home/thockin/go: not a directory
go: writing go.mod cache: mkdir /home/thockin/go: not a directory
go: writing go.mod cache: mkdir /home/thockin/go: not a directory
go: downloading github.com/google/go-licenses v0.0.0-20201026145851-73411c8fa237
go get github.com/google/go-licenses: mkdir /home/thockin/go: not a directory
```

What do I *want*? I want to download a tool repo, build it, drop the
binary into ./bin/tools (or something) and then either vendor it or
have no artifacts.

Since GOPATH seems to be a requirement, I thought maybe I can fake it:

```
$ (export GOPATH="`pwd`/.gopath"; export GOBIN=`pwd`/bin/tools; mkdir
-p "$GOBIN"; go get github.com/google/go-licenses)
go: downloading github.com/google/go-licenses v0.0.0-20201026145851-73411c8fa237
go: downloading github.com/otiai10/copy v1.2.0
go: downloading github.com/spf13/cobra v0.0.5
go: downloading github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b
go: downloading github.com/google/licenseclassifier
v0.0.0-20190926221455-842c0d70d702
go: downloading golang.org/x/tools v0.0.0-20191118222007-07fc4c7f2b98
go: downloading gopkg.in/src-d/go-git.v4 v4.13.1
go: downloading github.com/spf13/pflag v1.0.5
go: downloading github.com/inconshreveable/mousetrap v1.0.0
go: downloading github.com/sergi/go-diff v1.0.0
go: downloading gopkg.in/src-d/go-billy.v4 v4.3.2
go: downloading golang.org/x/crypto v0.0.0-20191117063200-497ca9f6d64f
go: downloading golang.org/x/sys v0.0.0-20191119060738-e882bf8e40c2
go: downloading github.com/emirpasic/gods v1.12.0
go: downloading github.com/jbenet/go-context v0.0.0-20150711004518-d14ea06fba99
go: downloading github.com/src-d/gcfg v1.4.0
go: downloading github.com/kevinburke/ssh_config
v0.0.0-20190725054713-01f96b0aa0cd
go: downloading github.com/mitchellh/go-homedir v1.1.0
go: downloading github.com/xanzy/ssh-agent v0.2.1
go: downloading golang.org/x/net v0.0.0-20191119073136-fc4aabc6c914
go: downloading gopkg.in/warnings.v0 v0.1.2
go get: added github.com/google/go-licenses v0.0.0-20201026145851-73411c8fa237
```

This does actually build my binary (and leaves me one more thing to
`make clean`), but it also writes to go.mod and go.sum. I thought I'd
complete the task and vendor the code. `go mod vendor` adds a line to
vendor/modules.txt:

```
# github.com/google/go-licenses v0.0.0-20201026145851-73411c8fa237
```

...but doesn't actually vendor the code. OK. What if I just clean
up? `go mod tidy`. This leaves that vendor/modules.txt change but
removes almost all of the other changes to go.sum. Curiously, it
leaves some:

```
$ git diff
diff --git a/go.mod b/go.mod
index ced314c..cfe7bca 100644
--- a/go.mod
+++ b/go.mod
@@ -5,6 +5,7 @@ require (
github.com/go-logr/logr v0.1.0 // indirect
github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b // indirect
github.com/prometheus/client_golang v0.9.2
+ golang.org/x/sync v0.0.0-20190423024810-112230192c58 // indirect
)

go 1.15
diff --git a/go.sum b/go.sum
index b9f0527..4e159c8 100644
--- a/go.sum
+++ b/go.sum
@@ -19,5 +19,6 @@ github.com/prometheus/common
v0.0.0-20181126121408-4724e9255275/go.mod h1:daVV7q
github.com/prometheus/procfs v0.0.0-20181204211112-1dc9a6cbc91a
h1:9a8MnZMP0X2nLJdBg+pBmGgkJlSaKC2KaQmTCk1XDtE=
github.com/prometheus/procfs
v0.0.0-20181204211112-1dc9a6cbc91a/go.mod
h1:c3At6R/oaqEKCNdg8wHV1ftS6bRYblBhIjjI8uT2IGk=
golang.org/x/net v0.0.0-20181201002055-351d144fa1fc/go.mod
h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
-golang.org/x/sync v0.0.0-20181108010431-42b317875d0f
h1:Bl/8QSvNqXvPGPGXa2z5xUTmV7VDcZyvRZ+QQXkXTZQ=
golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod
h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
+golang.org/x/sync v0.0.0-20190423024810-112230192c58
h1:8gQV6CLnAEikrhgkHFbMAEhagSSnXWGV915qUMm9mrU=
+golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod
h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
```

If I reset the whole repo and run `go mod tidy` it does not make this change.

Googling around I found a suggestion to use a different modfile. That
sounds like a hack, but I'll try it.

I made a new, nearly-empty `go.tools.mod`:

```
$ (export GOPATH="`pwd`/.gopath"; export GOBIN=`pwd`/bin/tools; mkdir
-p "$GOBIN"; go get -modfile=go.tools.mod
github.com/google/go-licenses)
```

This works and wrote to go.tools.mod a created a new go.tools.sum. I
still can't figure how to vendor the tool. I guess I can just
.gitignore those, but this is starting to feel like I made a wrong
turn at Albuquerque.

There has to be a "normal" path here, right? I feel like I shouldn't
have to fight so hard to make a more self-contained build.

Can someone clue me in?

Tim

[Jay Conrod]

Hey Tim, thanks for writing this up. These kinds of reports are helpful. I'll try to respond to a few problems individually. 

1. Command installation with "go install"

There are basically two ways to do this, depending on whether you want to install the command within the context of your current module or "globally" ignoring go.mod, go.sum, and vendor.

To install a command globally (you can replace @latest with a specific version like @v1.2.3):

go install github.com/google/go-licenses@latest

This form (with the @version suffix) is new in Go 1.16.

Installing a command within the context of the current module is more complicated. I'd suggest this:

# Add a dependency on the tool (only need to do this once)

go get -d github.com/google/go-licenses

# Import the command from tools.go

# See below

# Sync vendor directory (only if vendoring; only after changing go.mod with 'go get' above)

go mod vendor

# Install the command

go install github.com/google/go-licenses

# Alternatively, build and write to some other directory

go build -o bin/go-licenses github.com/google/go-licenses

The problem with this workflow is that 'go mod tidy' removes the requirement on the command's module because nothing imports it. 'go mod vendor' won't vendor the package for the same reason. There is a workaround documented on the Wiki: import the command package from a file (usually named tools.go) that is excluded by a build constraint (usually "tools").

// +build tools

package tools

import _ "github.com/google/go-licenses"

This works, but personally, I don't find this to be an intuitive solution. At the moment, we don't have a concrete proposal to improve it though. There's some discussion at #25922.

2. "writing go.mod cache" error messages

This error message should be a lot better. Sorry about that.

Most module-aware commands need to download files into the module cache. The location of the cache can be set by setting the GOMODCACHE environment variable (or 'go env -w GOMODCACHE=/some/path'). That defaults to $GOPATH/pkg/mod, and GOPATH defaults to $HOME/go. So by default, the module cache is at $HOME/go/pkg/mod. It can't create that directory though. 

The error message should explain what went wrong and how to fix it, and it really shouldn't be repeated. I've opened #45113 for this.

3. 'go get' updating go.mod and go.sum

This is an unfortunate consequence of 'go get' being overloaded for downloading, installing, and managing package versions. Ideally, we want to separate these roles into different commands: 'go get' should be used to manage dependencies (changing requirements in go.mod and go.sum), and 'go install' should be used to install commands.

We've taken a couple steps toward this in 1.16. The 'go install cmd@version' form is new. Also, 'go install', 'go build', and other commands no longer change go.mod or go.sum automatically when something is missing.

We plan to deprecate 'go get' for installing commands. In Go 1.17, 'go get' will print a warning when the -d flag is not used and the arguments are main packages. In Go 1.18, we plan to remove that functionality entirely (the -d flag will always be on). #40276 is where that deprecation was proposed.

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CAO_Rewb_5vBW4fuEa6i-XLLRiHqgEibxXTce3hHmOHfgAsRpfQ%40mail.gmail.com.

[Tim Hockin]

Thanks for feedback, comments below

On Thu, Mar 18, 2021 at 3:35 PM Jay Conrod <jayc...@google.com> wrote:

Installing a command within the context of the current module is more complicated. I'd suggest this:

# Add a dependency on the tool (only need to do this once)

go get -d github.com/google/go-licenses

# Import the command from tools.go

# See below

# Sync vendor directory (only if vendoring; only after changing go.mod with 'go get' above)

go mod vendor

# Install the command

go install github.com/google/go-licenses

# Alternatively, build and write to some other directory

go build -o bin/go-licenses github.com/google/go-licenses

I bent this a bit, and it seems to work:

1) add tools.go and import the tool

2) go mod vendor

3) go build

IOW, I skipped `go get -d` and it all seems OK.

Now it builds.  If I run `go mod tidy` it removes a line from my go.mod (`github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b // indirect`) and then refuses to build:

```

go: inconsistent vendoring in /src:
github.com/golang/gl...@v0.0.0-20160126235308-23def4e6c14b: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod

run 'go mod vendor' to sync, or use -mod=mod or -mod=readonly to ignore the vendor directory

```

Now I run `go mod tidy` again and it adds a line to `vendor/modules.txt` and happily compiles again.

Subsequent runs of either command seem to be no-op at this point.

Is it by design that these two commands are unstable like that?

 

Now that it builds, extra fun.  This tool in particular seems to try to read a DB file from a subdir of the directory its source code came from (as per `runtime.Caller(0)`) which ...doesn't work at all with vendoring, since the dir doesn't contain Go code.

AFAIK there's not a way to ask `go mod vendor` to include the whole repo is there?  I can't find a spec for modules.txt - it would be super cool if I could add some tag in there...  As it stands, all this was for naught.

 

2. "writing go.mod cache" error messages

This error message should be a lot better. Sorry about that.

I mean, "mkdir /home/thockin/go: not a directory" is pretty concise - it's not a directory (on purpose) because I really don't want Go randomly deciding to write stuff to my homedir (which I suspected was happening, and that's why I made it a non-directory :)

 

3. 'go get' updating go.mod and go.sum

This is an unfortunate consequence of 'go get' being overloaded for downloading, installing, and managing package versions. Ideally, we want to separate these roles into different commands: 'go get' should be used to manage dependencies (changing requirements in go.mod and go.sum), and 'go install' should be used to install commands.

We've taken a couple steps toward this in 1.16. The 'go install cmd@version' form is new. Also, 'go install', 'go build', and other commands no longer change go.mod or go.sum automatically when something is missing.

From where I site, I like the distinction, and I might even argue for explicitness.  E.g.  if I said `go get -mod=vendor` I feel like I am unambiguously asking the tool to fetch the code and put it in ./vendor and update the module files.  If I said just `go get` I (not so unambiguously) mean "global" (for me).  I find the automatic-detection of module mode very confusing, personally.

Tim

[Manlio Perillo]

Il giorno venerdì 19 marzo 2021 alle 23:20:49 UTC+1 Tim Hockin ha scritto:

Thanks for feedback, comments below

On Thu, Mar 18, 2021 at 3:35 PM Jay Conrod <jayc...@google.com> wrote:

> [...] 

 

2. "writing go.mod cache" error messages

This error message should be a lot better. Sorry about that.

I mean, "mkdir /home/thockin/go: not a directory" is pretty concise - it's not a directory (on purpose) because I really don't want Go randomly deciding to write stuff to my homedir (which I suspected was happening, and that's why I made it a non-directory :)

 

If you don't want the go tool to write files to your home directory, you can just define a custom GOPATH or GOMODCACHE.  As an example, on my sysytem I have set GOPATH to ~/.local/lib/go, and this directory is excluded from the backup.

Manlio 

[Jay Conrod]

> Is it by design that these two commands ['go mod tidy' and 'go mod vendor'] are unstable like that?

That is more or less by design, but the behavior has evolved over time and it may be worth revisiting.

This was discussed in #29058. The rationale was that because 'go mod vendor' deletes and rebuilds the entire vendor directory, it would be dangerous to do that implicitly inside another command like 'go mod tidy', since people frequently make local changes to vendor, like when preparing a patch for upstream.

That discussion was a couple years ago, before vendoring was enabled automatically. At the time, you had to specify -mod=vendor with build commands.

A better user experience might be 1) support for incremental changes to the vendor directory via 'go get', 'go mod tidy', 2) detection of local changes to the vendor directory so we can avoid overwriting them.

I've opened #45161 to restart that discussion.

> This tool in particular seems to try to read a DB file from a subdir of the directory its source code came from (as per `runtime.Caller(0)`) which ...doesn't work at all with vendoring, since the dir doesn't contain Go code.

> AFAIK there's not a way to ask `go mod vendor` to include the whole repo is there?  I can't find a spec for modules.txt - it would be super cool if I could add some tag in there...  As it stands, all this was for naught.

Is there another way to include and locate that file? Even if it were vendored, runtime.Caller(0) may be unreliable when -trimpath is used, and the files will be read-only in the module cache when vendoring is not used.

'go mod vendor' only pulls in directories for packages needed to build packages in the main module and their tests ('go list -test -deps ./...', essentially). I don't think we even include testdata or packages imported by those packages' tests.

We don't have a spec for vendor/modules.txt. it basically contains enough information for the go command to know which packages are vendored and which modules they came from.

--

You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/8002edf8-1a9e-44a6-a427-3fbc2b4e1dc1n%40googlegroups.com.