Intentionally omitting requires in go.mod - alternative to multi-module repository?

[Rodolfo Carvalho]

Hello fellow Gophers!

I help maintain an SDK module that includes packages that provide middleware to different web frameworks (e.g. echo).

The SDK started out as a single module in a Git repository, and we have considered and hesitated splitting it into multiple modules in a single repository because of the implications that it has on maintenance, development, testing and release processes.

The root package implements the core functionality of the SDK and has no external dependencies, it only needs the standard library. That's what most people need. The other packages each depend on a web framework module, and downstream users typically only need to use one of those packages and do not want to bother about the others.

The current `/go.mod` file is littered with dependencies to many web frameworks and their dependencies, and that has caused problems and confusion to our downstream users, especially when one of those dependencies end up flagged by tools like Dependabot as containing a CVE. (Code security scanners seem to typically operate on the Go module level and not at the package level, so they often report problems even though the affected module/package is not part of the final build list of the main package.)

I've been trying to find a way to eliminate the problems of the current single-module-per-repo setup while avoiding the implications of going multi-module-per-repo, as we have limited resources to maintain the SDK.

The recent idea I had was to simply omit `require` entries in the `/go.mod` file, essentially making the module "untidy" (as in `go mod tidy` would re-introduce the missing requires), but I haven't found any mention to that approach on the Internet.

That idea seems hacky and possibly not conforming to the spec, but perhaps not totally invalid.

As I read https://go.dev/ref/mod#go-mod-file and https://go.dev/ref/mod#go-mod-file-updates, I understand that the important `go.mod` file is the one of the main module (i.e. the `go.mod` file of our downstream users), and as long as the main module requires the same web framework as expected by the SDK middleware (when used), the Go tool should be able to determine versions and complete the build.

I'd like to hear thoughts from others -- has anyone tried something similar?

Should I expect obvious problems for downstream consumers, like failed builds?

Is there an alternative solution to be considered?

Thanks,

Rodolfo Carvalho

PS: the module graph pruning from Go 1.17 has been of great help addressing another problem of the single-module-per-repo setup, making it such that our users that are on Go 1.17 do not need to download go.mod files from modules they don't depend on. Many thanks to Bryan C. Mills and the rest of the Go team!

[Bryan C. Mills]

On Thursday, January 13, 2022 at 12:45:13 PM UTC-5 rhcar...@gmail.com wrote:

Hello fellow Gophers!

I help maintain an SDK module that includes packages that provide middleware to different web frameworks (e.g. echo).

The SDK started out as a single module in a Git repository, and we have considered and hesitated splitting it into multiple modules in a single repository because of the implications that it has on maintenance, development, testing and release processes.

The root package implements the core functionality of the SDK and has no external dependencies, it only needs the standard library. That's what most people need. The other packages each depend on a web framework module, and downstream users typically only need to use one of those packages and do not want to bother about the others.

The current `/go.mod` file is littered with dependencies to many web frameworks and their dependencies, and that has caused problems and confusion to our downstream users, especially when one of those dependencies end up flagged by tools like Dependabot as containing a CVE. (Code security scanners seem to typically operate on the Go module level and not at the package level, so they often report problems even though the affected module/package is not part of the final build list of the main package.)

As you noted, this is one of the use-cases we had in mind for module graph pruning in Go 1.17.

So one option is to run 'go mod tidy -go=1.17' to upgrade your module to support pruning, so that the consumers of your module won't themselves need to download irrelevant dependencies. That shouldn't harm any of your users on older Go versions (we designed it to be backward-compatible), although they won't see the benefits of pruning until they upgrade.

Independently, if your users have automated tools that are issuing false-positive CVE warnings based on the module graph instead of the package-import graph, you may want to suggest that they file issues against those tools. Ideally automated tools ought to be using the package-import graph instead of the module graph; however, if that's too involved they could still get a better approximation by consulting the `go.sum` file for the main module. (Modules that contribute packages relevant to the build will have source-code checksums in the `go.sum` file; modules that are in the module graph but not otherwise relevant will have only `/go.mod` checksums.)

I've been trying to find a way to eliminate the problems of the current single-module-per-repo setup while avoiding the implications of going multi-module-per-repo, as we have limited resources to maintain the SDK.

The recent idea I had was to simply omit `require` entries in the `/go.mod` file, essentially making the module "untidy" (as in `go mod tidy` would re-introduce the missing requires), but I haven't found any mention to that approach on the Internet.

We have tests of cmd/go behavior in that kind of configuration, but we don't recommend it. Leaving your module untidy makes builds of your packages less reproducible for users, and in some cases can also cause extra work for the `go` command to resolve the latest versions of those dependencies. If you leave dependencies out of your own `go.mod` file, then those dependencies will be resolved and added to your users' `go.mod` files when they run `go mod tidy` — but they are resolved to the latest release of each missing dependency, which means a bit more latency to identify what the latest release is, and an increased risk of breakage from incompatible changes in those dependencies.

We designed module graph pruning specifically to address use-cases like yours, so I would suggest leaning on that feature — and please do file issues (or feel free to shoot me an email!) if you run into bugs or rough edges.

[Leigh McCulloch]

I'm a user of the sentry-go SDK who has been impacted by the large number of dependencies in the past.

>Independently, if your users have automated tools that are issuing false-positive CVE warnings based on the module graph instead of the package-import graph, you may want to suggest that they file issues against those tools.

Does using the package-import graph capture the full picture of how a module impacts importers of the module. My understanding – maybe incorrect or out-dated – is that importing a module still brings that modules complete dependency graph into the dependency resolution process. Anyone using dependencies imported will be limited by the minimum versions defined in sentry-go SDK or any transitive dependency through sentry-go. Even if their go.mod is simpler on the surface, an SDK with a large and far reaching dependency graph can still impact other projects.

Bryan, is there a good example of how to use the go tool to provide details of the package-import graph? I would normally use go mod graph and go mod why which as I understand both are limited to the module graph.

Thanks!

Leigh

[Bryan C. Mills]

On Monday, January 31, 2022 at 1:22:09 AM UTC-5 leig...@gmail.com wrote:

I'm a user of the sentry-go SDK who has been impacted by the large number of dependencies in the past.

>Independently, if your users have automated tools that are issuing false-positive CVE warnings based on the module graph instead of the package-import graph, you may want to suggest that they file issues against those tools.

Does using the package-import graph capture the full picture of how a module impacts importers of the module.

No. However, it does capture many of the properties relevant to build times, source code downloads, binary sizes, and security exposure, especially given https://go.dev/issue/44435 (a `go mod download` change coming in Go 1.18).

 

My understanding – maybe incorrect or out-dated – is that importing a module still brings that modules complete dependency graph into the dependency resolution process. Anyone using dependencies imported will be limited by the minimum versions defined in sentry-go SDK or any transitive dependency through sentry-go. Even if their go.mod is simpler on the surface, an SDK with a large and far reaching dependency graph can still impact other projects.

That is no longer the case for modules at `go 1.17` or higher, due to module graph pruning taking effect at that version. If your module is at `go 1.17` or higher, then importing a package from a `go 1.17` or higher module only brings in one layer of that module's dependencies, plus the dependencies of any other packages that are added to your module's package-import graph as a result of that import.

Bryan, is there a good example of how to use the go tool to provide details of the package-import graph? I would normally use go mod graph and go mod why which as I understand both are limited to the module graph.

`go mod why` does inspect the package-import graph — not the module graph — but admittedly only in a fairly limited way: it gives a path of package imports from your main module to the named package or (with the `-m` flag) module.

In addition, there are some third-party tools built on top of `go list` (such as goda) that can perform more detailed queries on the package-import graph.