go install an/import/path@revision and /vendor in Go 1.16

[Konstantin Khomoutov]

Hi!

We have recently turned one of our internal Go projects into a Go module.
The project was using vendoring and we intend to continue doing so - just
rather with the natural support from `go mod` instead of govendor as before.

An interesting issue came up when we've decided to try the recently added
ability of `go install` to grab a named module and build an executable from
it: we would expect that when we do

$ go install an/module/import/path@latest

the command would grab the module's source code (it's hosted by a GitLab
instance, so we'd expect the source code to be `git clone`-d) and then behave
as if one would clone the module's repository by hand and then run
`go install` in the local clone's work tree - that is, we'd expect the command
to actually make use of the vendored packages and not attempt to download
anything except the source code of the specified module.

Instead, the command appears to read the module's go.mod file and then
download all the dependencies from the network - completely ignoring the
"vendor" directory of the module.

I have tried to locate a related issue in the bug tracker, but failed;
[1] is the closest I was able to come up with, but it appears to not touch the
very problem I have described.

The documentation also does not seem to specify the expected behaviour.

Hence my question is: is this a problem worth reporting/discussing or just an
undocumented behaviour which is "by design"?

My take on why this behaviour is confusing is that it makes `go install`
behave differently when it's apparently told to do the same thing to the same
module:

- When I have the module's source code on my filesystem, and I run
`go install` in that directory, the command uses the vendored dependencies,
builds and installs the executable provided by the module.

- When I run `go install import/path/of/that/module@revision`
I expect the command to just add one extra step - fetch the module's
source code and then run `go install` in the (temporary) directory
with the fetched source code.

1. https://github.com/golang/go/issues/44841

[Sean Liao]

This is intentional and sort of covered by the statement that no module is considered the main module (where replace directives and vendor dirs take effect)

https://github.com/golang/go/issues/40276 has some more context

[Brian Candler]

The key point I just learned from #40276:

"Vendor directories are not included in module zip files."

I found this surprising because -mod=vendor is now the default (here) - therefore, I expected the vendor directory, if present in the source, always to be used unless explicitly disabled.

It would be possible (although not a good idea) for someone to tweak their vendor directory contents directly, as an ad-hoc way of forking dependent packages.  This would work fine for local checkout and build, but fail for "go install".  Presumably they should import the code into a directory called something other than "vendor" in that case?

I think it would be helpful to flag this up at https://golang.org/ref/mod#vendoring

It might also be worth a mention at https://golang.org/ref/mod#zip-files that vendor directories are excluded from module zip files (although I wouldn't be reading that section if I was only interested in how vendoring works)

[Konstantin Khomoutov]

On Fri, Aug 06, 2021 at 12:35:16AM -0700, Brian Candler wrote:

> The key point I just learned from #40276:
> "Vendor directories are not included in module zip files."
>
> I found this surprising because -mod=vendor is now the default (here

> <https://golang.org/ref/mod#vendoring>) - therefore, I expected the vendor

> directory, if present in the source, always to be used unless explicitly
> disabled.

An interesting point regarding our particular case - which I sadly failed to
mention - is that we do not use any module proxies for own internal projects,
and the hostname of our GitLab instance is listed in the GONOPROXY environment
variable, so technically `go install` did not download a "classic" module,
which is a ZIP archive file, but has rather performed the usual sequence of
issuing a HTTP GET request with the "?go-get=1" query parameter followed by a
shallow clone of the target Git repository.
Hence technically the "vendor" directory with its full contents was available.

But yes, that point of yours is a sensilbe one; it sheds some light on the
reasoning behind the behaviour have observed.

[...]

> It might also be worth a mention at https://golang.org/ref/mod#zip-files
> that vendor directories are excluded from module zip files (although I
> wouldn't be reading that section if I was only interested in how vendoring
> works)

I would say the doc could make use of some cross-referencing: this fact could
be mentioned either there or in the section on vendoring, and the remaining
section could contain a single statement about the fact with the link to a
full explanation in the other section.

What do you think?

[...]

[Konstantin Khomoutov]

On Thu, Aug 05, 2021 at 04:39:00PM -0700, Sean Liao wrote:

> This is intentional and sort of covered by the statement that no module is
> considered the main module (where replace directives and vendor dirs take
> effect)

I must admit that sentence regarding the main module has caught my attention
when I was reading that part of the docs but it looks like I have not paid
enough attention to the concept of a main module earlier in the manual.
To be honest, I do not remember I have even seen it, no idea why.

> https://github.com/golang/go/issues/40276 has some more context

Thanks for your explanations.

[...]

[Brian Candler]

On Friday, 6 August 2021 at 11:46:24 UTC+1 Konstantin Khomoutov wrote:

On Fri, Aug 06, 2021 at 12:35:16AM -0700, Brian Candler wrote:

> The key point I just learned from #40276:
> "Vendor directories are not included in module zip files."
>
> I found this surprising because -mod=vendor is now the default (here
> <https://golang.org/ref/mod#vendoring>) - therefore, I expected the vendor
> directory, if present in the source, always to be used unless explicitly
> disabled.

An interesting point regarding our particular case - which I sadly failed to
mention - is that we do not use any module proxies for own internal projects,
and the hostname of our GitLab instance is listed in the GONOPROXY environment
variable, so technically `go install` did not download a "classic" module,
which is a ZIP archive file, but has rather performed the usual sequence of
issuing a HTTP GET request with the "?go-get=1" query parameter followed by a
shallow clone of the target Git repository.
Hence technically the "vendor" directory with its full contents was available.

However according to #40276:

"Even when go connects directly to a repository instead of a

proxy, it still generates zip files so that builds work consistently no matter

how modules are fetched. Those zip files don't contain nested modules or vendor

directories."

[Jay Conrod]

Indeed, this should be better documented. I've mailed CL 340509 to mention in a couple places in /ref/mod that vendor directories are omitted from zip files.

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/9f4d48bd-fecf-46bf-aa85-5115f0513f53n%40googlegroups.com.