go mod tidy pulls in too much

[Harmen]

Hi all,

I've been converting some work projects to use go.mod.
Today I ran into a problem that after importing the sub-package
github.com/hashicorp/consul/api `go mod tidy` pulled in everything imported
anywhere in the hashicorp/consul repo, even if it's never imported in my
project.

I described the problem on https://github.com/golang/go/issues/27920, which got
closed within three minutes as being "documented", and "works as
expected" (which I assume also means "works as intended").
Is this really the intented behaviour? It seems unexpected to me. Or should I
simply stay away from `go mod tidy`?

Thanks!
Harmen

[Paul Jolly]

Hi Harmen

I described the problem on https://github.com/golang/go/issues/27920, which got
closed within three minutes as being "documented", and "works as
expected" (which I assume also means "works as intended").
Is this really the intented behaviour? It seems unexpected to me. Or should I
simply stay away from `go mod tidy`?

I replied to your issue earlier. But could arguably have been slightly more detailed in my response beyond simply linking to https://github.com/golang/go/wiki/Modules#why-does-go-mod-tidy-record-indirect-and-test-dependencies-in-my-gomod

As you note, github.com/hashicorp/consul/api is a package. It is a package in the module github.com/hashicorp/consul. Despite there being no go.mod in github.com/hashicorp/consul, the go tool simulates it as a module.

Given there are no go.mod files in any subdirectories below 

github.com/hashicorp/consul, then github.com/hashicorp/consul is the module for all the packages github.com/hashicorp/consul/...

Hence go mod tidy (per the link in my response to your issue) is pulling in the transitive dependencies needed for tests in github.com/hashicorp/consul/... The reason you see all of these in your go.mod is that github.com/hashicorp/consul has not yet been converted to a module as yet.

Hopefully that gives a bit more colour on what's going on here.

Thanks,

Paul

[Harmen]

Hi Paul,

thanks for your extended reply.

So if consul adds a go.mod file in the root, then `mod tidy` will suddenly
behave as I would expect (i.e. not pull in unused dependencies)? And until they
add it (if ever) I either better not run `mod tidy`, or simply go back to the
old dep system?

[thepud...@gmail.com]

 > "So if consul adds a go.mod file in the root, then `mod tidy` will suddenly 

behave as I would expect (i.e. not pull in unused dependencies)? "

Hi Harmen,

Just in case this isn't already clear-- note that even in your current situation, 'go build' is still pulling in exactly what it needs (and not pulling in unused dependencies).

I think there is not a significant harm in your go.mod containing these indirect dependencies (aside from of course seeing that longer list, etc.).

And there is some benefit -- this behavior is part of what provides for 100% reproducible builds and tests. The modules system records precise dependency version information, and in your case, that precise dependency version information for some of your indirect dependencies is being recorded in your go.mod (given that it is not yet recorded in the non-existent go.mod of some of your direct dependencies).  

As an example, this behavior helps make sure that `go test all` is 100% reproducible for you (where `go test all` runs tests for both your module, your direct dependencies, and your indirect dependencies, which is valuable as one way of validating that the currently selected packages versions are compatible -- the number of possible version combinations is exponential in the number of modules, so in general you cannot expect your dependencies to have tested against all possible combinations of *their* dependencies).

--thepudds

[thepud...@gmail.com]

Hi Harmen,

And my first sentence might not have been clear. When I said "even in your current situation, 'go build' is still pulling in exactly what it needs", I was trying to reference the actual compilation process.

In other words, I was just trying to make it clear that even if you have "extra" dependencies appearing in your go.mod file, the resulting binary produced by 'go build' does not have anything extra or any unused dependencies.

In any event, I wanted to share at least my personal understanding, but of course happy to learn more...

--thepudds

[Harmen]

On Fri, Sep 28, 2018 at 10:19:50AM -0700, thepud...@gmail.com wrote:
> Hi Harmen,
>
> And my first sentence might not have been clear. When I said "even in your
> current situation, 'go build' is still pulling in exactly what it needs", I
> was trying to reference the actual compilation process.
>
> In other words, I was just trying to make it clear that even if you have
> "extra" dependencies appearing in your go.mod file, the resulting binary
> produced by 'go build' does not have anything extra or any unused
> dependencies.

Hi thepudds,
thanks for your anwers.

It doesn't hurt per-se, but all those unused modules will be vendored in the
repo (or the CI will have to download them every time). Maybe the consul repo
is a particularly unlucky repo, since the /api package is small compared to the
rest of the repo, but still.

As for the argument that it's for test reproducibility, I do not follow that.
Everything to test /api is there, and the stuff I don't import doesn't need to
be tested in the first place.

> In any event, I wanted to share at least my personal understanding, but of
> course happy to learn more...

I'm trying to do the same :)
Thanks!

> --
> You received this message because you are subscribed to the Google Groups "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Scott Cotton]

Hi all,

I think this is related to https://github.com/golang/go/issues/26913 which is closed but still has commentary that seems unresolved after it was closed .

Scott

[thepud...@gmail.com]

Hi Scott,

Regarding your comment, a related issue (which is still open) is this one:

#26955: “cmd/go: provide straightforward way to see non-test dependencies”
https://github.com/golang/go/issues/26955

—thepudds

[Scott Cotton]

As a followup,

I have been informed that there is a related open issue which seems like a good place to direct followup

https://github.com/golang/go/issues/26955

Best,

Scott