x/vgo: a brief note about a non-optimal experience

453 views
Skip to first unread message

Peter Waller

unread,
Jun 15, 2018, 9:42:14 AM6/15/18
to golang-nuts
Hi All,

I've made an example repository here which captures some thoughts on some poor experience I've had with vgo:

https://github.com/pwaller/vgo-use-dockerclient

I'd be interested to know if it could/should work better, or not, if there is something wrong with what I'm doing, and whether this is a case vgo is meant to address. The issue might also be that docker is not yet using go.mod files, or having straightforward semantic version numbers?

Apologies that they put together in a rush, I have limited time! Hopefully the issue is clear from the example and easy to reproduce. I have had the same issue with a couple of projects now, trying them out with vgo. On my co-working space WiFi, it takes a very long time - hours to download - starting vgo without a go.mod, I think because of this issue.

Thoughts appreciated,

- Peter


Ian Lance Taylor

unread,
Jun 15, 2018, 10:12:54 AM6/15/18
to Peter Waller, Bryan C. Mills, Russ Cox, golang-nuts
[ +bcmills, rsc ]
> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

Daniel Theophanes

unread,
Jun 15, 2018, 1:48:10 PM6/15/18
to golang-nuts
Hi Peter,

I think there are two issues at play here. 

When I tested this, and in your logs you posted, there is the following error:
fatal: unable to access 'https://bitbucket.org/ww/goautoneg/': GnuTLS recv error (-110): The TLS connection was non-properly terminated.

This happens when I try to git clone from bitbucket.org from https (rather then ssh) on my laptop as well. I'm unsure what this is stemming from; I was unable to resolve it at this time.

This is then behaving poorly (I'm somewhat guessing) with the prefix lookup code in "gitrepo/fetch.go", as it continues to retry.

I suspect there might be an action item with vgo, so that when it sees such an error ("unable to access" text), it fails hard and just stops.

Bryan C. Mills

unread,
Jun 15, 2018, 5:43:15 PM6/15/18
to pe...@pdftables.com, Russ Cox, golang-nuts, Ian Lance Taylor
Thanks for the feedback: it's great to have concrete examples that we can test against.

Some specific points from your readme:

> I think this behaviour can be explained in part because docker/docker isn't yet a vgo package, so the pain below may only be a teething pain which won't exist in the end. But I'm unsure on that point.

If docker/docker/client has substantially fewer dependencies than docker in geneneral, it should ideally be defined as a separate vgo module. I suspect that would mitigate most of the problems from extraneous dependencies.


> 1. Why does vgo take 8 minutes to run the first time?

Vgo doesn't do many things in parallel at the moment. We know that's not ideal, and it will get better.

https://golang.org/cl/119055 (just committed) should help quite a bit. How bad is the situation with a vgo built from commit 0a6cdd775a6812169f1b0e5c6bfb27d35d50cbef or later? (I just timed it at 3m18s on my machine, but YMMV.)


> 2. Why is vgo producing so many exit status 1 errors? Is this expected?

I'm not sure why it's doing that. I didn't see any “exit status 1” when I tried a `vgo build` on your example.

At the very least, the fact that the output looks buggy is cause to investigate. Could you file a bug in the Go issue tracker? (Use the prefix “x/vgo:” and please provide the versions of go, vgo, and the git command. We've had some trouble before with older versions of git that ship with some distros.)


> 3. Should vgo be looking at all of the transitive dependencies of docker/docker?

That's an interesting question.

When vgo first runs on a non-vgo repository, it generates a module definition for the root directory of that repository.
That module definition applies to the whole module: if some package imported by docker/docker/client comes from a package that depends on `foo v1.0.0`, and docker/docker/somethingelse imports a package whose module depends on `foo v1.2.0`, then the entire docker/docker module depends on `foo v1.2.0`.

If we didn't find the transitive requirements of the entire module, then when someone else incorporates your package into their program (that happens to also import docker/docker/somethingelse), they would get `foo v1.2.0` instead of the `foo v1.0.0` that you had tested against, even though they are using the same version of the docker/docker module that you were.

So we have to download at least enough of the transitive dependencies to be able to see the whole module graph. If our sparse `git fetch` logic is working correctly, we shouldn't have to download entire repos for those dependencies: just the go.mod files (or supported legacy lock files) should suffice.

Bryan C. Mills

unread,
Jun 15, 2018, 6:49:55 PM6/15/18
to Peter Waller, Russ Cox, golang-nuts, Ian Lance Taylor
One more detail:
> Furthermore, vgo picks up docker v1.13.1 and not the latest, v17.05.0-ce

That's a consequence of semantic import versioning. If you just say “github.com/docker/docker/client”, vgo assumes that you mean major version v0 or v1 (because you didn't say otherwise in the import path), and it assumes that v17 is not compatible with v1 (because the major version is different).

You should still be able to choose an arbitrary tag explicitly and have vgo resolve it to a pseudo-version. Unfortunately, the simple way to do that (using `vgo get`) doesn't seem to work for the docker repository: I've filed https://golang.org/issue/25917 to track that.

Peter Waller

unread,
Jun 19, 2018, 2:55:19 PM6/19/18
to Bryan C. Mills, Russ Cox, golang-nuts, Ian Lance Taylor
The latest commit (fixing https://golang.org/issues/25919) improves things fairly dramatically.

Some measurements for anyone following along at home:

  Without a go.mod, and empty `.../mod`: 1m40s (was > 8m on decent connection, > 1h on flaky WiFi).
  With a go.mod, and empty `$GOPATH/src/mod`: 1m5s. (Was > 8m on decent wired connection).
  Without a go.mod, with populated `.../mod` cache: 15s.
  With go.mod, with populated `.../mod` cache: 2s.

I haven't tested on the flaky WiFi yet but I'm guessing it will be much better than it was.

> > 3. Should vgo be looking at all of the transitive dependencies of docker/docker?
> That's an interesting question.
> When vgo first runs on a non-vgo repository, it generates a module definition for the root directory of that repository.
> That module definition applies to the whole module [snip]

Ah. OK, now I understand the behaviour at least.

If we didn't find the transitive requirements of the entire module, then when someone else incorporates your package into their program (that happens to also import docker/docker/somethingelse), they would get `foo v1.2.0` instead of the `foo v1.0.0` that you had tested against, even though they are using the same version of the docker/docker module that you were.

Hm. I'm not sure I quite follow this. Are you saying that, in the scenario where docker doesn't have any `go.mod` files, the `go.mod` generated for my project depends on the content of stuff I haven't imported?

If docker had a top-level go.mod (and therefore we weren't generating one), would anything be different? Would vgo still visit all the transitive dependencies of docker while generating deps for my app? My hope would be that it only need visit those imported by my app at that point.

Bryan C. Mills

unread,
Jun 19, 2018, 3:39:14 PM6/19/18
to Peter Waller, Russ Cox, golang-nuts, Ian Lance Taylor
On Tue, Jun 19, 2018 at 2:55 PM Peter Waller <pe...@pdftables.com> wrote:
The latest commit (fixing https://golang.org/issues/25919) improves things fairly dramatically.

Some measurements for anyone following along at home:

  Without a go.mod, and empty `.../mod`: 1m40s (was > 8m on decent connection, > 1h on flaky WiFi).
  With a go.mod, and empty `$GOPATH/src/mod`: 1m5s. (Was > 8m on decent wired connection).
  Without a go.mod, with populated `.../mod` cache: 15s.
  With go.mod, with populated `.../mod` cache: 2s.

I haven't tested on the flaky WiFi yet but I'm guessing it will be much better than it was.

> > 3. Should vgo be looking at all of the transitive dependencies of docker/docker?
> That's an interesting question.
> When vgo first runs on a non-vgo repository, it generates a module definition for the root directory of that repository.
> That module definition applies to the whole module [snip]

Ah. OK, now I understand the behaviour at least.

If we didn't find the transitive requirements of the entire module, then when someone else incorporates your package into their program (that happens to also import docker/docker/somethingelse), they would get `foo v1.2.0` instead of the `foo v1.0.0` that you had tested against, even though they are using the same version of the docker/docker module that you were.

Hm. I'm not sure I quite follow this. Are you saying that, in the scenario where docker doesn't have any `go.mod` files, the `go.mod` generated for my project depends on the content of stuff I haven't imported?

Yes: it reflects the versions listed in your module's transitive requirements.

Today it is all of the modules with packages that your module imports, less all of the modules for which the same version is implied by your transitive dependencies.
(I was talking with Russ about it this morning in relation to https://golang.org/cl/119575, and we might make the implied-requirement pruning less aggressive in order to avoid some surprising behaviors when upgrading packages.)

If docker had a top-level go.mod (and therefore we weren't generating one), would anything be different? Would vgo still visit all the transitive dependencies of docker while generating deps for my app? My hope would be that it only need visit those imported by my app at that point.

That is an interesting question. If we do make the pruning less aggressive, then we might not need to scan everything after all, but only if we can trust that the versions in your go.mod correctly reflect the versions of your dependencies. That's a pretty subtle invariant to assume, but on the other hand the benefits of pruning the search could be large.

Peter Waller

unread,
Jun 19, 2018, 5:57:44 PM6/19/18
to Bryan C. Mills, Russ Cox, golang-nuts, Ian Lance Taylor
On 19 June 2018 at 20:38, Bryan C. Mills <bcm...@google.com> wrote:
If docker had a top-level go.mod (and therefore we weren't generating one), would anything be different? Would vgo still visit all the transitive dependencies of docker while generating deps for my app? My hope would be that it only need visit those imported by my app at that point.

That is an interesting question. If we do make the pruning less aggressive, then we might not need to scan everything after all, but only if we can trust that the versions in your go.mod correctly reflect the versions of your dependencies. [snip]

When I said "while generating deps for my app", i.e. when "my app" doesn't yet have a go.mod. Or alternatively, if I have a go.mod but I have only just imported `docker/docker/client`: docker/docker doesn't yet exist in my go.mod, and I want vgo to 'figure it out'.

I want this to be as fast as it can be, so that I can depend on such things easily, and so that a newcomer using vgo for the first time doesn't feel that it is doing a surprising amount of work. We don't all always have super-awesome reliable low latency links :(

In my case, for example, vgo went off and seemingly explored hundreds of dependencies, when I added `docker/docker/client`. In fact, it only adds 3 modules to the go.mod. If `docker/docker/go.mod` exists, presumably vgo shouldn't need to go and explore the whole tree of dependencies, since the requisite information would be in that file.

If I understand correctly, this implies, I hope, that actually docker need not have a `docker/docker/client/go.mod`, only a `docker/docker/go.mod`. Then my UX should be acceptable when I first import `docker/docker/client`: it would only need to fetch those three modules, in principle. Does that sound plausible?

Then lastly the question I suppose is: what is different if there is no `docker/docker/go.mod`, and can that case be improved, too? I'm uncertain, from the conversation so far. My gut tells me it should be OK. But I suspect not from what you're saying.

Bryan C. Mills

unread,
Jun 19, 2018, 6:19:04 PM6/19/18
to Peter Waller, Russ Cox, golang-nuts, Ian Lance Taylor
On Tue, Jun 19, 2018 at 5:57 PM Peter Waller <pe...@pdftables.com> wrote:
On 19 June 2018 at 20:38, Bryan C. Mills <bcm...@google.com> wrote:
If docker had a top-level go.mod (and therefore we weren't generating one), would anything be different? Would vgo still visit all the transitive dependencies of docker while generating deps for my app? My hope would be that it only need visit those imported by my app at that point.

That is an interesting question. If we do make the pruning less aggressive, then we might not need to scan everything after all, but only if we can trust that the versions in your go.mod correctly reflect the versions of your dependencies. [snip]

When I said "while generating deps for my app", i.e. when "my app" doesn't yet have a go.mod. Or alternatively, if I have a go.mod but I have only just imported `docker/docker/client`: docker/docker doesn't yet exist in my go.mod, and I want vgo to 'figure it out'.

I want this to be as fast as it can be, so that I can depend on such things easily, and so that a newcomer using vgo for the first time doesn't feel that it is doing a surprising amount of work. We don't all always have super-awesome reliable low latency links :(

In my case, for example, vgo went off and seemingly explored hundreds of dependencies, when I added `docker/docker/client`. In fact, it only adds 3 modules to the go.mod. If `docker/docker/go.mod` exists, presumably vgo shouldn't need to go and explore the whole tree of dependencies, since the requisite information would be in that file.

Remember, `vgo build` builds not only your package, but also the packages it depends on. That's a fundamental property of how Go packages work: each package needs the compiled export data from its dependencies.

So we do need to fetch at least the dependencies of your package, the packages it imports, the packages they import, &c. And in order to do that, we have to figure out which versions of those packages to fetch, which in turn requires that we look at the whole module requirement graph: any requirement for a newer version of a package, no matter how deep in the module graph, impacts the version of that package that we should choose for your whole module.

On the other hand, we don't need to actually fetch the code for all of those packages up front: just the go.mod files suffice. With just the go.mod files, we can compute versions for all of the packages, and then we can fetch the appropriate versions lazily at build time.

This gets back to my point about “if we can trust that the versions in your go.mod correctly reflect the versions of your dependencies.” If your go.mod file lists modules for all of the packages it imports, and the versions it lists for those modules are the same as the versions implied by your transitive requirements, then we only need to look at immediate requirements to start the build. But those properties don't hold today: the go.mod file today only includes requirements that are not already implied (transitively) by the other requirements.

Peter Waller

unread,
Jun 20, 2018, 5:52:21 PM6/20/18
to Bryan C. Mills, Russ Cox, golang-nuts, Ian Lance Taylor
Thanks for your patient and helpful explanation.

On 19 June 2018 at 23:17, Bryan C. Mills <bcm...@google.com> wrote:
If your go.mod file lists modules for all of the packages it imports, and the versions it lists for those modules are the same as the versions implied by your transitive requirements, then we only need to look at immediate requirements to start the build. But those properties don't hold today: the go.mod file today only includes requirements that are not already implied (transitively) by the other requirements.

I think I'm confused on this point by the tooling, let me build up my model so you can poke holes in it:


* I depend only on docker/docker/client, which fortunately has a modest transitive dependency graph, if I understand correctly.
* I start with no go.mod and run vgo install.
* vgo synthesises a docker/docker/go.mod, since it doesn't have one. Likewise for many transitive dependencies of docker.
* ???

I'm a little confused about the result there, from what you've written above, how am I ending up with this (below) in my go.mod?

require (
github.com/docker/docker v1.13.1
github.com/docker/go-units v0.3.3
github.com/opencontainers/runc v1.0.0-rc5
)

This seems in conflict with the statement "the go.mod file today only includes requirements that are not already implied (transitively) by the other requirements", unless I misunderstand something. My project only depends on a single package, so I might have thought by definition from what you've written I should get only one require in my go.mod.

I think the significant complication here is that we live in a world where go.mod files don't yet exist and get synthesised in the background. It might be that a greater understanding of the principles of the synthesis would illuminate things for me.

At this point I'm just thinking aloud - clearly these ideas are going to take some time to settle to a correct model, so please bear with me. I've had a quick skim over what's been published looking specifically for the go.mod synthesis algorithm but didn't stumble on anything.

Does the case of synthesis for absent go.mod files simply look for the latest versions of things? In which case, I can't easily construct a scenario where any of the transitive dependencies of the "whole module docker/docker" can affect the decision, compared with looking at the dependencies of "docker/docker/client". In both cases the answer would be "most recent for each dep". At this point in time my first assumption is basically that go.mod files don't exist in the majority of cases, so MVS doesn't seem in play yet.

I guess rather, were some go.mod files present, synthesis might run MVS on the entire module tree to make the decision. Something like "choose max of mins, or max of published if absent from all go.mods in module dep graph". I don't know.

Bryan C. Mills

unread,
Jun 20, 2018, 6:34:47 PM6/20/18
to Peter Waller, Russ Cox, golang-nuts, Ian Lance Taylor
Vgo knows how to convert from a number of existing lockfile formats (see this list).

In this case, you're probably getting translations of vendor.conf files. I suspect that the converter is having trouble with this line of the Docker vendor.conf (probably a variation on issue 25556), so it's pulling in the latest tagged release of that package instead and adding an explicit requirement on it.

I'm not sure how you're ending up with an explicit requirement for github.com/docker/go-units: the vendor.conf for github.com/opencontainers/runc v1.0.0-rc5 specifies v0.2.0, and it's not obvious to me why vgo would choose to upgrade beyond that. (It is almost certainly a bug, but I can't identify a likely cause. Perhaps there is another converted lock file deeper in the tree with a similarly missed replacement.)

Once vgo has this go.mod file, it scans the rest of the module graph to make sure it has all of the requirements that may affect your module.

Peter Waller

unread,
Jul 1, 2018, 5:36:12 PM7/1/18
to Bryan C. Mills, Russ Cox, golang-nuts, Ian Lance Taylor
I just gave things another go, and the experience is much better again. I guess, owing to "cmd/go/internal/modfetch: disable go.mod conversion for transitive deps" and alike?

Looking good. Now building https://github.com/pwaller/vgo-use-dockerclient without a go.mod works within 30 seconds and results in 55 MiB downloaded. I think this is considerably better than it was.
Reply all
Reply to author
Forward
0 new messages