Module dependency cycles

[Andy Bursavich]

I’m trying to get rid of a transient dependency on k8s.io/clie...@v12.0.0+incompatible and I’ve tracked it down to the co-dependency of Thanos and Cortex... Presumably, there isn't a dependency cycle in their packages, but there is in their modules (at different versions). The two update their versions of each other independently so there’s a trampoline leading all the way back to the k8s.io/client-go problem a year ago:

github.com/thanos-io/tha...@v0.18.0 github.com/cortexproject/cor...@v1.5.1-0.20201111110551-ba512881b076
github.com/cortexproject/cor...@v1.5.1-0.20201111110551-ba512881b076 github.com/thanos-io/tha...@v0.13.1-0.20201030101306-47f9a225cc52
github.com/thanos-io/tha...@v0.13.1-0.20201030101306-47f9a225cc52 github.com/cortexproject/cor...@v1.4.1-0.20201030080541-83ad6df2abea
github.com/cortexproject/cor...@v1.4.1-0.20201030080541-83ad6df2abea github.com/thanos-io/tha...@v0.13.1-0.20201019130456-f41940581d9a
github.com/thanos-io/tha...@v0.13.1-0.20201019130456-f41940581d9a github.com/cortexproject/cor...@v1.3.1-0.20200923145333-8587ea61fe17
github.com/cortexproject/cor...@v1.3.1-0.20200923145333-8587ea61fe17 github.com/thanos-io/tha...@v0.13.1-0.20200807203500-9b578afb4763
github.com/thanos-io/tha...@v0.13.1-0.20200807203500-9b578afb4763 github.com/cortexproject/cor...@v1.2.1-0.20200805064754-d8edc95e2c91
github.com/cortexproject/cor...@v1.2.1-0.20200805064754-d8edc95e2c91 github.com/thanos-io/tha...@v0.13.1-0.20200731083140-69b87607decf
github.com/thanos-io/tha...@v0.13.1-0.20200731083140-69b87607decf github.com/cortexproject/cor...@v0.6.1-0.20200228110116-92ab6cbe0995
github.com/cortexproject/cor...@v0.6.1-0.20200228110116-92ab6cbe0995 github.com/thanos-io/tha...@v0.8.1-0.20200109203923-552ffa4c1a0d
github.com/cortexproject/cor...@v0.6.1-0.20200228110116-92ab6cbe0995 github.com/prometheus/alertm...@v0.19.0
github.com/prometheus/alertm...@v0.19.0 github.com/prometheus/prome...@v0.0.0-20190818123050-43acd0e2e93f
github.com/prometheus/prome...@v0.0.0-20190818123050-43acd0e2e93f k8s.io/clie...@v12.0.0+incompatible

My initial thought was try to work with Thanos and Cortex to either (a) detangle their modules or (b) coordinate releases to start a new history for their trampoline, but now I'm wondering if this is something that should be addressed in the go tool. Perhaps the module "cycle" should be detected and newer versions of modules in the cycle should block older versions from being traversed?

[Andy Bursavich]

s/transient/transitive/

[Andy Bursavich]

To summarize the concrete issue:

An old version of Prometheus pulled in k8s.io/clie...@v12.0.0+incompatible.

An old version of Alertmanager pulled in that version of Prometheus.
An old version of Cortex pulled in that version of Alertmanager.
An old version of Thanos pulled in that version of Cortex
A newer version of Cortex pulled in that version of Thanos
A newer version of Thanos pulled in that version of Cortex

...
A newer version of Cortex pulled in that version of Thanos

A newer version of Thanos pulled in that version of Cortex

A recent version of Cortex pulled in that version of Thanos
A recent version of Thanos pulled in that version of Cortex

Newer versions of Prometheus have moved to k8s.io/clie...@v0.20.2.
Newer versions of Alertmanager don't depend on Prometheus.
Newer versions of Cortex have moved to newer versions of Alertmanager.
Newer versions of Thanos have moved to newer versions of Cortex.
Newer versions of Cortex have moved to newer versions of Thanos.

Because the k8s release tagging conventions predate go modules, go treats k8s.io/clie...@v12.0.0+incompatible as newer than k8s.io/clie...@v0.20.2, and the cycle between Thanos and Cortex keeps their oldest history together alive in the dependency graph, if you depend on Thanos then you get stuck with  k8s.io/clie...@v12.0.0+incompatible and must use replace directives.

[Bryan C. Mills]

In general the way to cut cycles is to cut a set of versions of the affected modules such that each module in the set requires only the other versions in that set. (So, a new `thanos` v0.19.0, a new `cortex` v1.8.0, and a new `alertmanager` v0.22.0, I think?)

The trick is to publish all of them more-or-less simultaneously. The new version of the first module to be published will be broken until the other versions in the set are published, but once everything is published those versions will settle out again, and you can leave behind all of the cruft underneath them.

That said, that kind of coordinated release is really unpleasant, so this is one of the motivating problems for https://golang.org/issue/36460 (which I've been actively working on since basically the start of 2020).

I'm fairly confident that it will finally land in Go 1.17. After that — when the dependencies of the affected modules become lazy, all of the irrelevant dependencies of older-than-selected versions will just melt away! (A gleaming future lies ahead!)

We also made some more targeted fixes to `exclude` directives in Go 1.16, so that the excluded dependencies are completely expunged from the module dependency graph rather than dynamically upgraded to the next-higher release. So for now, the best workaround is probably: (1) upgrade to Go 1.16, (2) notch out the problematic dependency with an `exclude` directive, and (3) wait for the long-term fix to (finally!) land in the `go` command.

I understand that this is all very inconvenient, and I'm sorry that we weren't able to land the long-term fix in 1.16. 😞

[Bart Grantham]

If I understand the post here, it seems we're also struggling with this issue with our private repos.  For us it means that in the go.sum of the highest level repos there's references for everything, all the way back to the initial v0.0.0 versions of every package.  AFAICT, this is happening with public modules as well.  I tried using modgraphviz to visualize our graph, it was hilariously large and it even generated a warning that it had to be scaled down in order for cairo to draw it.

If I can make an observation, and I may be totally wrong, this is a symptom of the linker's dependency graph and the module dependency graph being defined differently.  For the purposes of keeping go.sum tidy the linker's relaxed dependency graph is what I would expect, but I haven't really thought that through.  Looking at the proposal, it is clear that this is a very hairy problem.

In the meantime our go.sum's are growing monotonically.  Does this mean our ~/go/pkg/mod/ directories are also growing monotonically?  Mine is looking pretty heavy these days.

Bart

[Bryan C. Mills]

On Thu, Feb 25, 2021 at 7:43 PM Bart Grantham <ba...@bartgrantham.com> wrote:

If I understand the post here, it seems we're also struggling with this issue with our private repos.  For us it means that in the go.sum of the highest level repos there's references for everything, all the way back to the initial v0.0.0 versions of every package.  AFAICT, this is happening with public modules as well.  I tried using modgraphviz to visualize our graph, it was hilariously large and it even generated a warning that it had to be scaled down in order for cairo to draw it.

If I can make an observation, and I may be totally wrong, this is a symptom of the linker's dependency graph and the module dependency graph being defined differently.  For the purposes of keeping go.sum tidy the linker's relaxed dependency graph is what I would expect, but I haven't really thought that through.  Looking at the proposal, it is clear that this is a very hairy problem.

In the meantime our go.sum's are growing monotonically.  Does this mean our ~/go/pkg/mod/ directories are also growing monotonically?  Mine is looking pretty heavy these days.

Yes, if you go.sum file is growing monotonically then your module cache is probably also growing monotonically.

That said, for requirement cycles the only checksums (and the only module cache contents) that should actually be needed are for the …/go.mod paths, which contain the checksums for the go.mod files only (not the complete source code). So while it's true that your module cache is “growing monotonically”, the coefficient of growth should be small.

(Even a small coefficient can be a problem over the long term, of course — that's why we're also working on a long-term fix in the `go` command. 🙂)

[Bart Grantham]

Our solution to this was, against advice otherwise, to put go.sum in .gitignore and to rev all the co-dependent modules together simultaneously.  As you said, it was really unpleasant.  In the 3 days since I posted I've since had to do it again.

Is there a "Go modules therapy group" (or at least "Go modules haters club")?  I struggled to get our adoption of modules finalized for almost 9 straight days.  Now I'm exhausted and have a lot of very unhealthy negative energy that I need to vent regarding modules.  Its cast quite a shadow on my enjoyment of the language.  If I'm honest with myself, I wince every time I use it now.

Today I went to do a basic 'go run foobar.go' of a piece of prototype test code in a bare directory I was working with before upgrading to 1.16 and it gave me a "working directory is not part of a module" error because it imports an external package.  I nearly lost my mind.

Bart

[Andy Bursavich]

On Wednesday, February 24, 2021 at 7:42:24 PM UTC-8 Bryan C. Mills wrote:

That said, that kind of coordinated release is really unpleasant, so this is one of the motivating problems for https://golang.org/issue/36460 (which I've been actively working on since basically the start of 2020). I'm fairly confident that it will finally land in Go 1.17. After that — when the dependencies of the affected modules become lazy, all of the irrelevant dependencies of older-than-selected versions will just melt away! (A gleaming future lies ahead!)

I just saw that the new lazy code paths were enabled in tip. I'm sure there's still more to do before 1.17, but I know it's been a lot of work already and I wanted to say congrats and thanks... So, Bryan, congrats on landing this ~18 month behemoth and thanks for the gleaming future!