Dependency hell with Go modules

Francis Chuang

unread,

Jan 17, 2019, 6:35:56 AM1/17/19

to golang-nuts

Hey everyone,

I was wondering if I can get some ideas on how to solve this dependency problem with Go modules. I am using Go 1.11.4 on Linux (Windows Subsystem for Linux to be exact).

The first part of the problem is that for one of my libraries, I am importing github.com/hashicorp/vault/api, which is an api client to access Vault servers. This package is pretty thin and does not contain a lot of dependencies. However, the root project at github.com/hashicorp/vault has quite a few dependencies.

If I delete the current go.mod and go.sum files in my library and start from scratch:

- I run go mod init github.com/username/mylibrary

- I then run go mod tidy to add missing packages to my go.mod and generate a go.sum

Go mod imports the Vault repository at its root (github.com/hashicorp/vault) as expected. Unfortunately, this pulls in a lot of subdependencies and I have a ton of indirect dependencies in my go.mod and go.sum. This is not unexpected of course as this is how go modules work. Unfortunately, this does slow down continuous integration builds and is probably not great for Github and VCS hosts as each build pulls in all of these unneeded dependencies.

The second problem is that 2 of Vault's subdependencies (not sure how deep) is labix.org/v2/mgo and launchpad.net/gocheck. These are bazaar repos, so they necessitate the installation of the bzr tool. In my library, I added the following to my go.mod to force it to pull from the Github repos:

replace labix.org/v2/mgo => github.com/go-mgo/mgo v0.0.0-20160801194620-b6121c6199b7

replace launchpad.net/gocheck => github.com/go-check/check v0.0.0-20180628173108-788fd7840127

This works great for running tests in the username/mylibrary repo as I no longer need to install bzr when running my CI builds. However, if I create another project such as github.com/username/some-project and import username/mylibrary, go mod will attempt to download launchpad.net/gocheck and labix.org/v2/mgo from the original bazaar repos when running go test and other commands.These causes errors such as `go: labix.org/v2/m...@v0.0.0-20140701140051-000000000287: bzr branch --use-existing-dir https://launchpad.net/mgo/v2 . in /go/pkg/mod/cache/vcs/ca61c737a32b1e09a0919e15375f9c2b6aa09860cc097f1333b3c3d29e040ea8: exec: "bzr": executable file not found in $PATH `. This is quite annoying and I'd like to avoid having to add those `replace` statements in the go.mod of projects that consume the mylibrary package. In addition, consumers of my library would need to install bazaar (which most people probably won't have installed).

The 2 options I have thought of are:

1. Ask the Vault team to extract the api package into a separate repository. I am not sure how likely they would be interested in doing this, but I am guessing it would be quite low.

2. Make a copy of the api package and copy it directly into my library. This is something I am hoping to avoid as much as possible, because I'd like to use go modules to manage my dependencies.

Are there any other options I've missed?

Cheers,

Francis

Justin Israel

unread,

Jan 17, 2019, 1:19:15 PM1/17/19

to Francis Chuang, golang-nuts

I'm not yet well versed in modules since I use them in a very simple way and I vendor. But is it also supported as an option to make /api a "sub module" by adding another go.mod at that level?

If that is possible then it is way less intrusive of a change than splitting it into another repo.

That aside, I don't know if that would help too much in this case because I don't think Go modules differentiates between test and non test when collecting dependencies. If the api tests have deps, they will get collected I think. Even the non test files have some deps anyways. Might be nice if there were a Go modules option to skip scanning tests.

Anyways just throwing that out there until someone more informed that me responds.

Cheers,
Francis

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Harmen

unread,

Jan 17, 2019, 1:47:55 PM1/17/19

to Justin Israel, Francis Chuang, golang-nuts

Yes: don't run `go mod tidy`
See for example: https://github.com/golang/go/issues/27920
(which was also with a hashicorp project)

Francis Chuang

unread,

Jan 17, 2019, 5:26:02 PM1/17/19

to golang-nuts

Thanks, Justin and Harmen.

I deleted my go.mod and go.sum, then ran "go mod init ..." and "go test". The list of dependencies in go.mod and go.sum is now a lot slimmer. I tend to run my test suites in a bunch of docker containers, so "go test" fails for me when blindly executed from the root of the project. I think it would be nice if there is a command to "populate go.mod and go.sum with just the things we need to build and run tests on the current platform" without having to run tests.

Space A.

unread,

Jan 17, 2019, 6:01:45 PM1/17/19

to golang-nuts

go list ./...

пятница, 18 января 2019 г., 1:26:02 UTC+3 пользователь Francis Chuang написал:

David Collier-Brown

unread,

Jan 17, 2019, 7:04:58 PM1/17/19

to golang-nuts

You're playing in a space where there are NP-complete problems, as described in part by https://leaflessca.wordpress.com/2017/02/12/dll-hell-and-avoiding-an-np-complete-problem/

Run screaming from the building (;-))

roger peppe

unread,

Jan 18, 2019, 10:06:18 AM1/18/19

to Francis Chuang, golang-nuts

This is a good illustration of something I've been concerned about for a while: the fact that pulling in a package also pulls in all testing dependencies of that package recursively.

To take github.com/hashicorp/vault/api as an example, I made this little module:

-- main.go --

package x

import (

_ "github.com/hashicorp/vault/api"

)

func main() {

}

-- go.mod --

package x

I ran `go build` on this. It took about a minute and fetched 13 dependencies.

`go list -m all` showed 17 modules.