Go += Package Versioning

[Russ Cox]

Hi,

For Go 1.11 I am thinking about doing this.

https://research.swtch.com/vgo-intro.

Feedback welcome.

:-)

Best,

Russ

[Daniel Martí]

Congrats on getting this out, Russ!

The vgo-examples post, linked in a few places, is a 404 at the moment.
Is that on purpose?

[roger peppe]

Looks like the link in https://research.swtch.com/vgo should point to
https://research.swtch.com/vgo-tour not vgo-examples.

> --
> You received this message because you are subscribed to the Google Groups "golang-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Filippo Valsorda]

Hi Russ, I'm excited to read this, and enthusiastic to see GOPATH-less development on the horizon.

I will have to play with the API promise and minimal version selection in my head for a bit, in particular around what they will encourage in terms of deprecation and security updates.

What I wanted to address immediately is the lack of strong cryptographic locking. I am glad for the move to zip modules, but disagree that proxies and consistent version choices are sufficient to provide reproducible builds. I might be convinced that proxies are sufficient to address the availability concern, even if most users won't run them, leaving the Go ecosystem open to bit rot and left-pad issues.

However, what I think is unacceptable in a modern package management system is delegating integrity to the dependencies' repositories and their host. A repository owner is allowed to change the meaning of a git tag, and a malicious host might even just change the contents of zip files. Neither of these events should be allowed to change the behavior of a compiled program once it's locked and maybe reviewed by the developer.

Lock files got us from trusting all dependencies and all their hosting infrastructure at every build to only trusting them once, when the developer locks them. (Also limiting the potential of targeted attacks.) That's not a giant step forward, but it's an important one to build upon, and definitely not one we should take back.

Maybe the community will run a proxy that provides stronger availability guarantees, and that will get us the same stability of crates.io, but for such a solution to be possible, it's even more important that hashes are available to allow for the proxy not to be trusted. (The tooling will also have to know not to use the proxy to generate the hashes themselves, but I digress.)

2018-02-20 12:21 GMT-0500 Russ Cox <r...@golang.org>:

[Russ Cox]

On Tue, Feb 20, 2018 at 2:26 PM, Filippo Valsorda <fil...@ml.filippo.io> wrote:

What I wanted to address immediately is the lack of strong cryptographic locking.

We can do that too. It's only day 1. :-)

Russ

[Dave Cheney]

I love everything about this proposal! 👏

With one small exception

import "github.com/go-yaml/yaml/v2"

I don't think it's necessary to create v2 sub module (or whatever
they're called) to denote the api break if there is an existing v2.y.z
tag. For example I cannot use vgo to build a project that depends on
the correctly semver tagged k8s.io/client-go repo.

% cat go.mod
module "github.com/heptio/contour"

require (
"k8s.io/client-go" v6.0.0
)
% vgo build ./cmd/contour
vgo: errors parsing go.mod:
/Users/dfc/src/github.com/heptio/contour/go.mod:4: invalid module:
k8s.io/client-go should be v1, not v6 (v6.0.0)

https://github.com/golang/go/issues/23980

But apart from that, I think vgo is wonderful and I look forward to it
being included in Go 1.11.

Thanks

Dave

[Daniel Martí]

I share the concerns raised by Filippo and Dave. Looking forward to more
blog posts to expand in those areas.

There's one perhaps minor point that I'd like to be clarified too - the
decision to standardize around zip files. Not the fact that archives are
used instead of git/hg/etc, but rather the compression format.

If I understand zip correctly, as opposed to tar.* files, it has
advantages and disadvantages. For example, it compresses file by file,
which means that it's easier to extract or add single files. But I would
also guess that makes for worse overall compression of a module.

So, my question is - is the intent to standardize around exactly one
archive format? If so, is there a reason why it's zip? Has any thought
been given to allowing other formats, such as tar.*?

I can imagine that one reason to use zip is that hosting services like
GitHub and Bitbucket are more likely to support it, versus .tar.gz or
.tar.xz. And that extracting zip files tends to be easier across
platforms (especially Windows). If these are the reasons, I'd like them
clarified in the final proposal.

Some numbers for reference (tools-master.zip downloaded from
golang/tools on GitHub):

$ ls -lh tools-master.*
-rw-r--r-- 1 mvdan mvdan 2.2M Feb 21 12:03 tools-master.tar.gz
-rw-r--r-- 1 mvdan mvdan 1.7M Feb 21 12:03 tools-master.tar.xz
-rw-r--r-- 1 mvdan mvdan 2.5M Feb 21 12:02 tools-master.zip

I realise storage and transfer can be cheap these days, but it's still
something to keep in mind.

[samuel....@gmail.com]

It is necessary, unfortunately, because the MVS algorithm is only sound under the condition that all versions at a particular import path are backwards compatible. This is the import path compatibility rule.

This is one of the primary places the proposal offloads complexity.

[Russ Cox]

On Tue, Feb 20, 2018 at 10:49 PM, Dave Cheney <da...@cheney.net> wrote:

I love everything about this proposal! 👏

With one small exception

import "github.com/go-yaml/yaml/v2"

I don't think it's necessary to create v2 sub module (or whatever
they're called) to denote the api break if there is an existing v2.y.z
tag. For example I cannot use vgo to build a project that depends on
the correctly semver tagged k8s.io/client-go repo.

% cat go.mod
module "github.com/heptio/contour"

require (
        "k8s.io/client-go" v6.0.0
)
% vgo build ./cmd/contour
vgo: errors parsing go.mod:
/Users/dfc/src/github.com/heptio/contour/go.mod:4: invalid module:
k8s.io/client-go should be v1, not v6 (v6.0.0)

I'm sorry about that. If you look up the commit hash for v6.0.0 and put _that_ in the file, then vgo will take it from there. To start using the v6 the import paths will need to be updated (and k8s.io/client-go will need to add a go.mod with a module line, to declare explicitly that it's on board with /v6/ in its import paths).

I just posted a new post about this topic. research.swtch.com/vgo-import.

Best,

Russ

[roger peppe]

On 21 February 2018 at 20:43, Russ Cox <r...@golang.org> wrote:

I just posted a new post about this topic. research.swtch.com/vgo-import.

Thank you for articulating clearly some topics that have been real issues for us over the last few years. There was one issue in particular which ended up completely insoluble due to large repositories and diamond dependencies. I was planning to write an experience report of that, but you've made it unnecessary now. In quite a few places the fact that we were using gopkg.in (which uses the approach advocated in that post) has saved our bacon, and we've had several of the "same" package (different major versions) imported in the same binary. One of the less obvious implications of this approach is that if your API exposes any third party type, and you change to use a different major version of that third party repo, you'll need to change the major version of your package too. When you have a bunch of dependencies, that can result in you needing to change your major version more often than you might like, and it's not always obvious when you need to do that. It can also be a major pain to deal with the consequential changes, often involving long-winded coordination of PRs across multiple repositories.  

I'd like to see tooling that makes it easier to determine when an API has changed in a backwardly incompatible way and to make it easier to manage the knock-on changes required when you *do* change a major version. The go:fix idea is a lovely one, but I'm not sure how well it could work when the types you're returning are those of dependencies and not defined locally. 

[Dave Cheney]

Thanks for your reply Russ.

This is a bitter pill to swallow. It’s incompatible with the majority of big Go projects out there. When I pushed for semver tagging a few years ago, the question “should a tag start with a v” was hotly debated. In the end I believe the v prefix argument won out, in part because this was the style in use by all the major Go projects at the time. There were other styles in use, but the argument that adopting a tag format which invalidated the existing (and explicitly encouraged by GitHub) practice of adding a v prefix to the release tags for major Go projects like docker and k8s would be untenable.

I fear the same situation here. Right now, to a first order approximation, no projects include a /v2, /v3, etc top level directory in their repos (please don’t confuse this with kubernetes’ practice of adding a /v1 suffix to individual package paths). I think requiring an explicit prefix at the top level of the repository makes this proposal incompatible with the majority of established Go projects in the ecosystem today.

Is there no way to avoid duplicating the semver major version in the top level of the respository?

Thanks

Dave

[Russ Cox]

[Resending, sorry Daniel. Accidentally dropped golang-dev from previous reply.]

On Wed, Feb 21, 2018 at 7:12 AM, Daniel Martí <mv...@mvdan.cc> wrote:

There's one perhaps minor point that I'd like to be clarified too - the
decision to standardize around zip files. Not the fact that archives are
used instead of git/hg/etc, but rather the compression format.

Hi,

I guess there are two parts to your question: which container format, and which compression algorithm? I picked zip because it's so much better specified than tar (really!) and supports random access. I think both of those are very important.

As for compression efficiency, I don't think the system is going to be bottlenecked by file transfers. Most packages will be reused from cache. If zip supported better algorithms then it would be fine if we wanted to use them, but I'm not super worried. If xzip compression were an option I could enable in the zip writer, I would, of course. But I think the better container wins over compression efficiency.

Best,

Russ

[Henrik Johansson]

I agree very much with Dave about littering the repo with major version folders. Why can't this be inferred from tags, branches or releases to take the GitHub example?

--

[Russ Cox]

I think there are many ways to avoid the v2 in the import path, but they all end up being very very complex. I didn't just make up the -fclone and -fmultiverse flags in the blog post story. I actually had a design worked out last summer for inferring that kind of stuff automatically, and it became clear that, like in the post, when it did eventually break, it was going to be completely impossible to understand. Like the old saying that the better your 4-wheel-drive is, the further out you get stuck. So I think that v2 in the import path is almost certainly here to stay.

However, you asked about the repository, so I'm not sure if you meant import path or file system. It is not a requirement that a v2 repo have everything under a top-level v2 directory. There will be two supported modes: you can do that or not. I expect most projects won't, but some will. I have a post about this for tomorrow (not done yet or I'd just post it now). If you have a project that's at v2 right now and therefore doesn't work with vgo, all it needs to do is update all the import paths to put in the v2's and then also write a go.mod that says

    module "my/thing/v2"

(maybe along with some requirements). You can also make a v2/ subtree with the same go.mod file, and if you do, vgo will use that instead. It's up to you. But because the v2 or later behavior is different from expectations, vgo won't use a v2.0.0 or later tag unless the go.mod file exists and confirms that the developer is using the semantic import versioning convention.

Russ

[Russ Cox]

On Wed, Feb 21, 2018 at 4:59 PM, Henrik Johansson <dahan...@gmail.com> wrote:

I agree very much with Dave about littering the repo with major version folders. Why can't this be inferred from tags, branches or releases to take the GitHub example?

See my reply to Dave (was typing as your mail came in), but just to reemphasize: you do not have to put v2 code in a v2 directory. You can have a repo that's just v2 at the top level. You just still have to import it with a v2 import path.

One thing to add, which is why I'm writing this mail, is that if you are concerned about backwards compatibility with old "go get", then you can't just update your import paths without also changing the directory structure. That's part of why it's allowed to use a v2 subdir. But if your users are all using other versioning tools, they might all work fine configured to find "my/thing/v2" in github.com/you/mything v2.0.0's root directory.

Best,

Russ

[Bakul Shah]

On Wed, 21 Feb 2018 21:59:56 +0000 Henrik Johansson <dahan...@gmail.com> wrote:
>
> I agree very much with Dave about littering the repo with major version
> folders. Why can't this be inferred from tags, branches or releases to take
> the GitHub example?

This is another reason I like

import <path> [<version>]

syntax. People don't start a new repo when a new version is
created; they just tag a specific commit or release. By not
putting the version number in the path, it can be mapped to an
SCM tag. This may complicate what vgo has to do a bit more
but feels more right.

[Henrik Johansson]

Excellent, thx Russ!

[Dave Cheney]

Thanks a Russ. I look forward to tomorrow’s blog post. 

--

[roger peppe]

The big question in my mind is whether I can successfully import and
use a repo that has version tags >= 2.0.0 but isn't using the /vN
suffixes in its own source code.

[Aram Hăvărneanu]

> just to reemphasize: you do not have to put v2 code in a v2 directory.
> You can have a repo that's just v2 at the top level. You just still
> have to import it with a v2 import path.

That makes the mapping between import paths and the filesystem
structure on disk non-trivial, which can open its own can of worms.

I agree that different major versions need different names. But
perhaps there is a better solution than simply appending /vx? I
will think about this.

--
Aram Hăvărneanu

[Aram Hăvărneanu]

[+cc golang-nuts, it was lost at some point]

I find it unexpected that to compute a version you need information
not from one, but from two places that use very similar but ultimately
different syntax.

In vgo the major version has to be specified in Go import paths,
and the minor version needs to be specified in go.mod file, which
is not Go code but looks very similar to Go code.

Why not just use Go code throughout? We could extend Go syntax in
a backwards-compatible way to be able to express all the information
contained in the go.mod file. The bulk of the code would continue
to import old-style, "bare", imports, while a single, or perhaps a
small number of Go files would encode the information required for
version management using new-style imports. Perhaps this file, or
files could have standardized names, zmod.go, or whatever (please
don't bikeshed the name).

These files could be guarded by build-tags, so old compilers ignore
them, just like how old compilers ignore the go.mod file.

--
Aram Hăvărneanu

[Russ Cox]

> [+cc golang-nuts, it was lost at some point]

BCC golang-nuts. Please do NOT mix the two threads. I expect that the two different lists with different people will have different focuses for their discussions.

[Russ Cox]

I guess now it's day 2. Here's a start. research.swtch.com/vgo-repro

Russ

[Edward Muller]

Why can't (or shouldn't) vgo be able to do this?

Given

* go.mod

module "github.com/foo/bar"

require "github.com/nancy/drew" v2.0.0

* main.go

package main

import "github.com/nancy/drew"

func main() {

  println(drew.Author{})

}

Uses $GOPATH/src/v/github.com/nancy/drew(v2.0.0)

Furthermore...

Given:

* go.mod

module "github.com/library/books"

require "github.com/nancy/drew" v3.0.0

* books.go

package books

import "github.com/nancy/drew"

func NancyDrewAuthor() drew.Author {

  return drew.Author{}

}

use $GOPATH/src/v/github.com/nancy/drew(v3.0.0)

Which is then used in a later version of `github.com/foo/bar` like so:

* go.mod

module "github.com/foo/bar:

require (

  "github.com/nancy/drew" v2.0.0

  "github.com/library/books" v1.0.0 {

    "github.com/nancy/drew" v3.0.0

  }

)

  

* main.go

package main

import (

  "github.com/nancy/drew"

  "github.com/library/books"

)

func main() {

  println(drew.Author{})

  println(books.NancyDrewAuthor())

}

The type returned by books.NancyDrewAuthor() would be (could be?) `github.com/nancy/drew(v3.0.0).Author`, while the type returned by drew.Author{} is `github.com/nancy/drew(v2.0.0).Author`. These values wouldn't be equal, even if all their members were the same, because the types are different (at least unless the rules were changed).

This puts the definition of a dependency's version in one place: The nearest parental mod file relative to the source that imported it. The declarations in the mod files of dependencies (transitively) are pulled up into mine. This lays out for me, in my program's mod file which of my deps are using `github.com/nancy/drew v3.0.0". This helps me understand the cost of using `github.com/library/books` and highlights possible places where my direct and transitive deps may conflict. I can also then try to tweak them. Maybe "github.com/library/books v1.0.0" works with `github.com/nancy/drew v2.0.0" (it's likely that it won't, but hey; this is my program)? Errors, %T, panics and the like would contain the entire type info, including "(version)", so it won't be ambiguous what is what.

The biggest downside I can see if that a particular dep could be (is likely to be) repeated multiple times in my mod file, often times perhaps deeply nested. Although only a single copy of each {module,version} tuple would be utilized by my program (I'm not asking for npm style duplication). Tooling could help bump versions across all those dependencies eliminating search and replace errors. When updated versions of transitive deps are discovered in an updated direct dependency's mod file we could have the option of adopting one or more of them and running tests for the module. Essentially a dependency's go.mod file becomes advisory for consumers of that dependency.

This also helps highlight explosive dependencies (dependencies which require seemingly countless other dependencies). Things that require k8s / docker will have a rather large next within that require, but IME that's also the nature of those beasts. And honestly it probably means that they should be broken down into more modules over time.

This avoids the need to use '/v2' in import paths or for me, as a library author to decide if I need to make '/v2' directories or not. (PS: I'm still not sure when I would need to add or use '/v2', both as a library author or consumer. But I'm still not caught up on your blog posts/writtings. :-( ).

PPS: An engineer noted that this discussion is very similar to RESTful API versioning discussions which boil down to either: (a) declare the version of the resource you are requesting via the ACCEPT header; or (b) put the version in the path, thereby requesting different resources between versions.

--

[roger peppe]

On 21 February 2018 at 22:11, roger peppe <rogp...@gmail.com> wrote:
> The big question in my mind is whether I can successfully import and
> use a repo that has version tags >= 2.0.0 but isn't using the /vN
> suffixes in its own source code.

I've been thinking a bit more about this. Assuming this is a current
hard restriction, this essentially means we can't start using vgo for
any of our tools until all their dependencies with tagged versions >=
2 have updated to use versioned import paths internally. That's
probably not going to happen for a long time.

Another issue is that when a major version changes in a repo, all the
import statements in that repo need to be changed too. Although that
can be automated, the churn is non-trivial (for example, in one
project I work on, there are almost 12000 import statements that would
change).

I wonder if this is necessary.

It is already the case that import statements aren't absolute. An
import must be *resolved* to find out what the actual package path is,
taking into account vendor directories.

Also, it seems that there's a way of lexically determining the major
version that applies to any given import path.

How about this: when an import path within a repo does not contain an
explicit major version, that version is implied by looking at the
go.mod file for that repo?

One possible rule might be:

- select all modules from go.mod that match the repo name.
- if there's more than one, error with "ambiguous import path"
- otherwise select that major version.

ISTM that this could significantly reduce the burden of having major
versions in import paths, while still providing essentially the same
semantics as Russ's proposal.

[Henrik Johansson]

This is more or less how I envisioned it to work. The tuple {"Import path", "resolved version"} would essentially be what determines a package, i.e. no vN needed in any imports.

[roger peppe]

On 22 February 2018 at 14:35, Henrik Johansson <dahan...@gmail.com> wrote:
> This is more or less how I envisioned it to work. The tuple {"Import path",
> "resolved version"} would essentially be what determines a package, i.e. no
> vN needed in any imports.

You would of course still need vN when you want to import different
major versions of the same package,
but the way I see it now, most Go code would not need to change.

I think this is a way of avoiding Dave Cheney's concern on Twitter:

"i have a feeling that the more people start to look and apply tooling
to compute major / minor changes we'll find that, as titus said at
c++con this year, almost everything is a major breaking change.
There's gonna be a lot of /vNNNN's in import paths."

Having used gopkg.in for a few years now, I've made a large number of large
changes just to change major versions in import paths, and I'd love
to be able to avoid doing that in the future. (I've also greatly benefited
from having major versions in import paths FWIW).

[Henrik Johansson]

Would vN really be needed if there is a manifest that declares the version I want?

[roger peppe]

On 22 February 2018 at 15:44, Henrik Johansson <dahan...@gmail.com> wrote:
> Would vN really be needed if there is a manifest that declares the version I
> want?

Yes, because you might want to be able to use two versions at the same
time. One example is to provide a backwardly compatible upgrade path.
Another is that different dependencies might be using different major
versions. It's crucial to Russ's proposal that it be possible to use
different major versions of a dependency in the same binary.

[Henrik Johansson]

If my program want's to explicitly use 2 different versions I get that I need some explicit discriminator to tell that at a certain point I want v1 or v2.

But if a transitive dependency wants to use v1 and my program wants v2 then it can work without it right given that there are manifest entries available.

Right?

Either way, neither case should need directories in the source tree.

I might very well be missing something fundamental...

[Scott Cotton]

I am sorry I don't understand your argument.  Let me explain why:

To use two versions at the same time, the versions could, often enough or even usually, be 

resolved by defaulting to the manifest/mod file major version for an import  path without vN.

Then multiple versions could be in the same binary without /vN appended to the import paths, provided they are in different modules.

In the case that multiple versions need to be in the same module, they could be disambiguated _optionally_ with some import path syntax.  In this case the mod files could specify requirements and constraints for /vN suffixed imports.

There is nothing I see that prevents simply defaulting to using the major version in a resolved mod file, allowing the user to not append /vN unless it is necessary for disambiguation.  It is already defaulting to v0 or v1 -- why not extend the default and relieve the burden on all the projects to change their import paths that don't need to?

If there is something I'm not getting, please do explain!

[Scott Cotton]

Hi Russ,

First, great work on vgo! 

I wanted to chime in with a plead addressing your statement

"""

I think there are many ways to avoid the v2 in the import path, but they all end up being very very complex. I didn't just make up the -fclone and -fmultiverse flags in the blog post story. I actually had a design worked out last summer for inferring that kind of stuff automatically, and it became clear that, like in the post, when it did eventually break, it was going to be completely impossible to understand

"""

I think that many of us would likely prefer a) difficult or ambiguous and therefore impossible to understand conditions "when it breaks"  to b) putting "/vN" in the import path.

Problems with b) are everywhere, not just when it breaks.  I think their ubiquity bears reconsideration of the weight of the difficulties when it breaks, because even though errors happen regularly, I believe they would happen _much_ less frequently than dealing with putting "/vN" in the import path.

To my understanding -- please correct me if I'm wrong -- here are some problems:

1) non backward compatible import statements, imagine say "v8" in path referring to the beverage or javascript engine.

2) what if a project does not use the append "/vN" in its imports and is >= v2.0.0?  This seems likely to me given the plethora of projects and estimated long road to unification.

3) It's not DRY w.r.t. mod files.

4) What Dave Cheney said but I didn't follow because I wasn't paying attention to the original semver discussion he refers to when it happened.

5) losing the 1-1 mapping of imports to files

While I appreciate the elegance of the concept that one import path should represent backward compatible development, maybe celebrating it with such verbosity detracts from that?   Why can't we just define the entity with backward compatibility guarantees to be the import path in an import statement resolved to major version as per the mod file?  IE if the resolved  mod file exists and states v69.1.0 then implicitly assume that defines the unique import path, otherwise not.

Again, overall I'm very happy with and impressed by vgo, great work and incredible progress and thanks much!  It's just the nature of the feedback requested which leads this message to detail more the concerns than the high points, of which there are many.

Scott 

[roger peppe]

That's precisely what I was suggesting.

[roger peppe]

On 22 February 2018 at 16:22, Henrik Johansson <dahan...@gmail.com> wrote:
> If my program want's to explicitly use 2 different versions I get that I
> need some explicit discriminator to tell that at a certain point I want v1
> or v2.
> But if a transitive dependency wants to use v1 and my program wants v2 then
> it can work without it right given that there are manifest entries
> available.
>
> Right?

I'm not sure what you mean by "manifest entries" there, but yes I think
we're in agreement. All I'm saying is that if an import path doesn't
have a major version, the major version can be inferred from the
nearest go.mod file in an ancestor directory.

> Either way, neither case should need directories in the source tree.

I don't think I said anything about needing directories in the source tree.
They're not necessary now and I don't think they need to be.

[Henrik Johansson]

Great! 😀

[jimmy frasche]

I'm relieved to know that the /vN directories can be implicit, but I
still don't really understand how it works when they are explicit
since there are multiple major versions of the code present
simultaneously but only the major-est has minor/patch metadata.

Even ignoring the concerns of how to manage such a repo: What if, say,
I depend on some sequence of packages that selects foo v2.0.4 and foo
v3.1.0 but the v3.1.0 tag has the code for v2.7.24? Would it only be
loading that and implicitly selecting the maximum of the 2.x line that
exists with the minimum of the 3.x line? If so that seems at odds with
the rest of the design. Would it be loading the library twice like it
does with implicit import naming? If so that seems at odds with the
goal of allowing a module to have multiple versions interoperate.

It seems like it would be simpler to only have the implicit /vN
directories (and, at that point, since the import string is just a
string maybe a clearer representation like import "github.com/foo/bar
v2" that doesn't mix the tag and the import path, making it difficult
to copy/paste the directory).

I can see why it would nice be able to share code in some situations
but those situations seem rare and look like they'd cause more
problems than they solve, though I have been known to miss things.

I'm sorry if that's been covered elsewhere. Bit of an information
overload on this topic lately.

[Bakul Shah]

If pkg A & pkg B both use the same import path to pkg C but
actually use different major versions, you have to look into
their respective .mod files to see the difference. The Vn
disambiguation is needed only when both versions are used in
the same pkg. This is what you are saying, right? You want
this because you want to avoid having to update a very large
number of packages (i.e. backward compatibility).

I see two problems here. 1) it breaks, or makes it harder, to
use different names for incompatible behavior. If I just read
sourcefiles of A & B and see both of them use C.Foo() I would
assume they refer to the same Foo(). Having the version number
in the pkg sourcefile is more "in your face". Harder to ignore.
2) When disambiguation is used, it gets even more confusing.
C.Foo() in A is not the same as C.Foo() in B but Cv2.Foo() in
A is the same as C.Foo in B.

What this means is now I would be forced to look in the .mod
files all the time while reviewing code and *remember* the
binding used in .mod. I am more likely to just add a comment
in the .go file to remind me of this difference.

There is also the issue of what happens when you are the
maintainer of pkg C. How are you going to make both versions
simultaneously available?

[Zellyn]

Russ, after multiple attempts, I'm still having trouble understanding your last couple of posts here. It seems a bit like you're saying, “You don't need /v2/ in the path; just make sure you put a /v2/ in the path.” Evidently I'm not understanding the terminology enough to see the distinctions between terms.

Are you saying that because vgo is downloading zips, you can call something "github.com/foo/bar/v2/" as a module name even though it lives at "github.com/foo/bar"? Or am I missing something?

Thanks,

Zellyn

On Wednesday, February 21, 2018 at 5:07:00 PM UTC-5, rsc wrote:

On Wed, Feb 21, 2018 at 4:59 PM, Henrik Johansson <dahan...@gmail.com> wrote:

I agree very much with Dave about littering the repo with major version folders. Why can't this be inferred from tags, branches or releases to take the GitHub example?

See my reply to Dave (was typing as your mail came in), but just to reemphasize: you do not have to put v2 code in a v2 directory. You can have a repo that's just v2 at the top level. You just still have to import it with a v2 import path.

One thing to add, which is why I'm writing this mail, is that if you are concerned about backwards compatibility with old "go get", then you can't just update your import paths without also changing the directory structure. That's part of why it's allowed to use a v2 subdir. But if your users are all using other versioning tools, they might all work fine configured to find "my/thing/v2" in github.com/you/mything v2.0.0's root directory.

Best,

Russ

[Edward Muller]

The thing I'm obviously missing is the need to import two versions of the same package. IMNSHO doing so through the path isn't ideal (it's confusing and conflates directory structure with versions). Something like this though.....

import (

  "github.com/foo/bar(v2)" // usable as `bar.`

  v1subpkg "github.com/foo/bar(v1)/subpkg" // usable as `v1subpkg.` (named as such)

  v1bar "github.com/foo/bar(v1)" // usable as `v1bar.` (named as such)

)

Where `(vX)` is specified at the module level. Duplicate imports of the same versions or w/o versions would be a compile error. The common case (singular import from that module, no "(vX)") would use the local .mod file to lookup the appropriate version to use.

[Jan Mercl]

On Thu, Feb 22, 2018 at 10:19 PM Edward Muller <edwa...@interlix.com> wrote:

> The thing I'm obviously missing is the need to import two versions of the same package.

That should be IMHO avoided at all costs and as a consequence not supported by the vgo tool.

--

-j

[Henrik Johansson]

But hasn't Russ's posts been about this?

To actually support this.

--

[Edward Muller]

It seems to me it's already the case where you can import two versions of the same package this way with vgo. 

https://research.swtch.com/vgo-import

https://research.swtch.com/vgo-import#comment-3770258784

Ditto today, with packages that are published via gopkg.in (ex: gopkg.in/yaml.v1 vs gopkg.in/yaml.v2).

My earlier response, which I replied to in this last response, detailed my opinion/view on how to have vgo set things up so that different versions of the "same package" are actually different packages, ala gopkg.in, just w/o the trailing version (versions would be at the module level.

PS: I don't disagree, but perhaps am a little less adamant about not supporting it, just that it should be the exception, not the rule.

 

--

-j

[Peter Bourgon]

On Thu, Feb 22, 2018 at 1:02 PM, Zellyn <zel...@gmail.com> wrote:
> Russ, after multiple attempts, I'm still having trouble understanding your
> last couple of posts here. It seems a bit like you're saying, “You don't
> need /v2/ in the path; just make sure you put a /v2/ in the path.” Evidently
> I'm not understanding the terminology enough to see the distinctions between
> terms.

You don't need /v2/ in the ·code· path, just make sure you put a /v2/
(suffix) on the module identifier (as an author) and in the import
path (as a consumer).

Import path and physical code location (in a repo or on your disk)
were, until now, essentially equal to each other. But that fact was
always carefully caveated as an implementation detail leveraged by the
`go` tool, technically they didn't need to be identical. With `vgo`
they are broken apart in this circumstance.

> Are you saying that because vgo is downloading zips, you can call something
> "github.com/foo/bar/v2/" as a module name even though it lives at
> "github.com/foo/bar"? Or am I missing something?

That property is true, but not (AFAIU) because `vgo` is downloading zips.

[Zellyn]

On Thursday, February 22, 2018 at 4:37:42 PM UTC-5, Peter Bourgon wrote:

Import path and physical code location (in a repo or on your disk)
were, until now, essentially equal to each other. But that fact was
always carefully caveated as an implementation detail leveraged by the
`go` tool, technically they didn't need to be identical. With `vgo`
they are broken apart in this circumstance.

If it means giving up "import path == physical code location" I'd be happy to just deal with insisting that "/v2/" be in the path.

Zellyn 

[Russ Cox]

You're going to lose import path == file system code location no matter what. Your system needs to be able to deal with multiple versions of a given package being available, for different builds. They can't both be in the same place. Also replacements and the like change things.

Today you can run 'go list -f {{.Dir}} x/y/z' to find out what directory it is in, instead of assuming $GOPATH/src/x/y/z. That will keep working, and it will give a correct answer even as you cd between modules using different versions of x/y/z.

Best,

Russ

[Russ Cox]

On Wed, Feb 21, 2018 at 10:59 PM, Edward Muller <edwa...@interlix.com> wrote:

Why can't (or shouldn't) vgo be able to do this? 

... 

require (

  "github.com/nancy/drew" v2.0.0

  "github.com/library/books" v1.0.0 {

    "github.com/nancy/drew" v3.0.0

  }

)

... 

This avoids the need to use '/v2' in import paths or for me, as a library author to decide if I need to make '/v2' directories or not. (PS: I'm still not sure when I would need to add or use '/v2', both as a library author or consumer. But I'm still not caught up on your blog posts/writtings. :-( ).

This is not too far off from the proposal I passed around last summer. It gets complicated very fast if you have to start thinking about a single import path (nancy/drew in this case) meaning different versions in different packages. What happens when modules lower in the dependency graph suggest other mappings? And so on. What you've described is basically the first diagram of "Chapter 2" in https://research.swtch.com/vgo-import, but then I think it's very easy to end up at the second or later diagrams and just end up horribly confused. In the Go 1.10 prep work I made the Go command internals powerful enough to implement both -fclone and -fmultiverse from the post, but I realized that it always eventually breaks, and the more sophisticated the automatic workarounds, the harder it is understand the eventual breakage. 

On the other hand, if everything in the program is clear about importing nancy/drew/v2 and nancy/drew/v3, that's a little more work for code authors but it makes clearer the previously hidden costs of major version bumps, and the work that authors do naturally has the same effect as -fclone and -fmultiverse, but in a way that doesn't eventually break. I'm also completely convinced by Rich Hickey's point that redefining the meaning of names is the source of all this complexity, and that the answer is to not do that. 

Honestly, if you'd asked me 18 months ago if we should have major versions in import paths, I'd probably have said no, it's purely a matter a taste and it's ugly, so let's not do it. But the thought experiment laid out in the vgo-import post convinced me that's not so. There are sound technical reasons to prefer major versions in import paths, even though that wasn't my original preference. 

I was also highly skeptical of uppercase-for-export when it went in. Mainly I thought it was ugly, but I had no technical argument against it. After using it for a while, the clarity of seeing, at every use, whether this is something that can also be used from outside the package has made it one of my favorite features. It's one of the things I miss most when I go back to other languages. I suspect major versions in import paths will be like that. They'll become second-nature, and we'll come to expect the clarity and miss it elsewhere.

Best,

Russ

[Russ Cox]

On Wed, Feb 21, 2018 at 5:11 PM, roger peppe <rogp...@gmail.com> wrote:

The big question in my mind is whether I can successfully import and

use a repo that has version tags >= 2.0.0 but isn't using the /vN
suffixes in its own source code.

No, I don't think so. You can import it with a v0 pseudo-version though. I thought for a while about how to implement this, and I think it just gets out of hand very quickly. If we end up with very important cases where it's necessary, maybe we'll have to reconsider, but I hope not.

Russ

[Russ Cox]

On Thu, Feb 22, 2018 at 8:21 AM, roger peppe <rogp...@gmail.com> wrote:

On 21 February 2018 at 22:11, roger peppe <rogp...@gmail.com> wrote:
> The big question in my mind is whether I can successfully import and
> use a repo that has version tags >= 2.0.0 but isn't using the /vN
> suffixes in its own source code.

I've been thinking a bit more about this. Assuming this is a current
hard restriction, this essentially means we can't start using vgo for
any of our tools until all their dependencies with tagged versions >=
2 have updated to use versioned import paths internally. That's
probably not going to happen for a long time.

You can use v0 pseudo-versions until them. It's a little unstable but it gets things started. 

Another issue is that when a major version changes in a repo, all the
import statements in that repo need to be changed too. Although that
can be automated, the churn is non-trivial (for example, in one
project I work on, there are almost 12000 import statements that would
change).

This just reemphasizes what a big deal it is to make a new major version. It causes a lot of work for users. To the extent that making it non-trivial for authors helps avoid breaking users, that could be a win.

How about this: when an import path within a repo does not contain an
explicit major version, that version is implied by looking at the
go.mod file for that repo?

There are lots of rules you could imagine here, but they all end up breaking the property that within a given build identical import paths anywhere in the build refer to the same package. And if you do this then it makes it impossible for v2 to import v1, which I think will be important (see the end of vgo-import, about go fix). People who do bump major versions forward often will probably end up with tools to update import paths, which doesn't seem like a huge price to pay for consistency of meaning throughout the build.

Best,
Russ

[Russ Cox]

On Thu, Feb 22, 2018 at 4:02 PM, Zellyn <zel...@gmail.com> wrote:

Russ, after multiple attempts, I'm still having trouble understanding your last couple of posts here. It seems a bit like you're saying, “You don't need /v2/ in the path; just make sure you put a /v2/ in the path.” Evidently I'm not understanding the terminology enough to see the distinctions between terms.

Are you saying that because vgo is downloading zips, you can call something "github.com/foo/bar/v2/" as a module name even though it lives at "github.com/foo/bar"? Or am I missing something?

There are two paths: the import path and the file system path within the repo. You have to put /v2/ in the import path. You can put the v2 code at the root of the repo if you write a go.mod there saying it's for v2. I hope https://research.swtch.com/vgo-module makes that clearer. ^F for "major branch".

Best,

Russ

[land...@gmail.com]

Hey Russ, am I understanding things correctly? If you don't specify which version you want to pull in as a dependency, instead of pulling the latest version, it pulls in the latest v1?  I can't be reading that correctly.  I must be mistaken there.  So lets say you pulled in firefox as a dependency (assuming it was a go package).  Instead of pulling in firefox 56 (or whatever it is on now), it would pull in firefox 1?, unless you explicitly told it to use firefox 56?

- thanks.


On Tuesday, February 20, 2018 at 11:21:23 AM UTC-6, rsc wrote:

Hi,

For Go 1.11 I am thinking about doing this.

https://research.swtch.com/vgo-intro.

Feedback welcome.

:-)

Best,

Russ

[roger peppe]

On 22 February 2018 at 20:19, Bakul Shah <ba...@bitblocks.com> wrote:
> I see two problems here. 1) it breaks, or makes it harder, to
> use different names for incompatible behavior. If I just read
> sourcefiles of A & B and see both of them use C.Foo() I would
> assume they refer to the same Foo(). Having the version number
> in the pkg sourcefile is more "in your face". Harder to ignore.
> 2) When disambiguation is used, it gets even more confusing.
> C.Foo() in A is not the same as C.Foo() in B but Cv2.Foo() in
> A is the same as C.Foo in B.
>
> What this means is now I would be forced to look in the .mod
> files all the time while reviewing code and *remember* the
> binding used in .mod. I am more likely to just add a comment
> in the .go file to remind me of this difference.

Yes, this is the trade-off. Note that this is the case *right now*
with vendor directories, so this isn't something new.

Also, even without vendor directories, you have this situation right
now if you lock dependencies - even if all
paths within a build have consistent package paths, move to a
different build and the package paths mean something different.

I don't think the situation is as bad as you think. In any given
repository, we'll almost always be using a single major version of
each module dependency, and that's what the import paths will be
referring to. Most of the time you'll be able to relax and just use
the normal unversioned import paths and they'll mean what you expect
them to mean, I believe.

Also, tooling can help. Your editor could show the major version that
applies, jump-to-definition will take you to the right place, etc.

> There is also the issue of what happens when you are the
> maintainer of pkg C. How are you going to make both versions
> simultaneously available?

I don't see the problem here. How is it different from what's proposed
in vgo now in this respect?

[roger peppe]

On 23 February 2018 at 01:15, Russ Cox <r...@golang.org> wrote:
> On Thu, Feb 22, 2018 at 8:21 AM, roger peppe <rogp...@gmail.com> wrote:
>>
>> On 21 February 2018 at 22:11, roger peppe <rogp...@gmail.com> wrote:
>> > The big question in my mind is whether I can successfully import and
>> > use a repo that has version tags >= 2.0.0 but isn't using the /vN
>> > suffixes in its own source code.
>>
>> I've been thinking a bit more about this. Assuming this is a current
>> hard restriction, this essentially means we can't start using vgo for
>> any of our tools until all their dependencies with tagged versions >=
>> 2 have updated to use versioned import paths internally. That's
>> probably not going to happen for a long time.
>
>
> You can use v0 pseudo-versions until them. It's a little unstable but it
> gets things started.

OK, thanks. I thought I'd tried that but evidently not.

>> Another issue is that when a major version changes in a repo, all the
>> import statements in that repo need to be changed too. Although that
>> can be automated, the churn is non-trivial (for example, in one
>> project I work on, there are almost 12000 import statements that would
>> change).
>
>
> This just reemphasizes what a big deal it is to make a new major version. It
> causes a lot of work for users. To the extent that making it non-trivial for
> authors helps avoid breaking users, that could be a win.

Speaking as someone that has been maintaining code that uses import
paths with major versions for the last few years, I have to say I'm
not convinced it's a win. I know it's *meant* to be hard work, but
applying these changes feels like make-work. Ironically at the start,
I argued just as you did, but I've changed my mind.

A major version change happens when *anything* in a module changes in
a backwardly incompatible way. But we often won't be using the parts
that have changed, so even though none of the calling code needs to
change, all of the import paths do. This feels like unnecessary churn,
when a single line in a go.mod file could be sufficient.

I know that maintaining backward compatibility is a virtue, but it's a
virtue that comes with trade-offs. If there's a fast-moving module
that's imported by hardly anyone, there's a significant penalty to pay
for maintaining backward compatibility (and I'm not talking about
maintaining versioned import paths here). In practice, I believe that
maintainers of fast-moving modules *will* break backwards
compatibility anyway, and that if they need to change all the import
paths every time they do it, they just won't bother maintaining
semantic versions, and we'll be back in the bad old days of everything
being broken all the time.

I'd like the decision to change a major version to be based on
potential impact to the ecosystem, not just the sheer hassle of
changing all the import paths.

One final point: when there are many major versions of everything,
it's easy to forget which version number of which module implies what.
Just because you've got the major version in the import path doesn't
mean you can actually remember what API that implies, so it's less
helpful than you might think.

In summary: I've had quite a bit of experience of using major versions
in import paths and it's a pain. I think that module-implied major
versions could provide the best of both worlds: a well versioned
ecosystem which still strongly incentivises people to maintain
compatibility, and nice import paths that most of the time don't
burden the user with the need to know about major version numbers.

>> How about this: when an import path within a repo does not contain an
>> explicit major version, that version is implied by looking at the
>> go.mod file for that repo?
>
>
> There are lots of rules you could imagine here, but they all end up breaking
> the property that within a given build identical import paths anywhere in
> the build refer to the same package.

I don't think that this property is as important as you think it is.
It's already not the case.

> People who do bump major versions forward
> often will probably end up with tools to update import paths, which doesn't
> seem like a huge price to pay for consistency of meaning throughout the
> build.

Yes, I've written those tools and I'm still not happy with the result.
I think consistency within a module is good enough - consistency
throughout the build seems like an unnecessary burden.

cheers,
rog.

[Henrik Johansson]

`I think that module-implied major

versions could provide the best of both worlds: a well versioned
ecosystem which still strongly incentivises people to maintain
compatibility, and nice import paths that most of the time don't

burden the user with the need to know about major version numbers.`

This seems like a very strong and nice point and it would be super nice!

[Nic Pottier]

On Friday, February 23, 2018 at 4:26:12 AM UTC-5, rog wrote:

I know that maintaining backward compatibility is a virtue, but it's a
virtue that comes with trade-offs. If there's a fast-moving module
that's imported by hardly anyone, there's a significant penalty to pay
for maintaining backward compatibility (and I'm not talking about
maintaining versioned import paths here). In practice, I believe that
maintainers of fast-moving modules *will* break backwards
compatibility anyway, and that if they need to change all the import
paths every time they do it, they just won't bother maintaining
semantic versions, and we'll be back in the bad old days of everything
being broken all the time.

I think this is a key and salient point. If we make it hurt that bad to increment major version numbers then people just won't do it. (while still breaking APIs) Yes, super stable APIs are great, and are certainly one reason Go as a language as done so well, but I'm not sure that library authors will be so nice about it if we make it this painful. (both for them and their users, the culture will just become to to stay v1 forever)

On another note, already having tried to play with vgo a bit in my projects I can already see how painful this import path system is going to be, especially for existing libraries. What is the migration path for a library that is at v3 already? If they add a mod.go and change their import paths then they break everybody else not using vgo. That seems like that is really going to stall adoption.

A proposal like Rog's allows existing libraries to introduce a go.mod and therefore provides a smooth transition from go->vgo.

I do agree there is a downside of not being able to glance at a package and know what major version it is using, but this is kind of the state of the world already and it seems editors/tooling could help on that front if it was deemed a critical deficiency. But having this kind of "nearest" mod.go system to resolve dependencies seems like it maintains the clear and simple rules that Russ is proposing (which I agree are great) while not putting an undue burden on library authors and users.

-Nic

[Zellyn]

On Thursday, February 22, 2018 at 7:55:46 PM UTC-5, rsc wrote:

You're going to lose import path == file system code location no matter what. Your system needs to be able to deal with multiple versions of a given package being available, for different builds. They can't both be in the same place. Also replacements and the like change things.

I'm fine with replacements changing things - that's what they're for.

But it would be nice not to break the rule that if you see github.com/zellyn/goapple2/v2 in an import statement, and load that URL in your browser, you can expect to find something there. That seems worth forcing everyone to use “Go module using major subdirectories” rather than “Go module using major branches” (from part 6). It also seems like a logical application of what seems to me to be the (or one of the) fundamental core philosophical ideas behind vgo: two different things should not have the same name.”

I understand the desire not to dictate how people organize their code… and that there are tradeoffs I haven't considered. Also, who knows what actual version of the repo you clicked through to (probably the latest on master). But usually it would show you something useful, and it might be worth being more prescriptive on this one. vgo is already pretty opinionated; might as well go whole hog :-)

Zellyn

[Russ Cox]

On Fri, Feb 23, 2018 at 9:58 AM, Zellyn <zel...@gmail.com> wrote:

On Thursday, February 22, 2018 at 7:55:46 PM UTC-5, rsc wrote:

You're going to lose import path == file system code location no matter what. Your system needs to be able to deal with multiple versions of a given package being available, for different builds. They can't both be in the same place. Also replacements and the like change things.

I'm fine with replacements changing things - that's what they're for.

But it would be nice not to break the rule that if you see github.com/zellyn/goapple2/v2 in an import statement, and load that URL in your browser, you can expect to find something there.

That doesn't work today. I can import github.com/zellyn/goapple2/shiny but the same URL is a 404.

Russ

[Zellyn Hunter]

On Fri, Feb 23, 2018 at 10:08 AM Russ Cox <r...@golang.org> wrote:

That doesn't work today. I can import github.com/zellyn/goapple2/shiny but the same URL is a 404.

True :-)  Github should fix that, but point taken: it's not much harder to click around to look for "shiny" than it would be to click around and find the v2 branch, if they did it that way. Maybe a little trickier for newcomers to understand.

Zellyn

[Josh Bleecher Snyder]

But it would be nice not to break the rule that if you see github.com/zellyn/goapple2/v2 in an import statement, and load that URL in your browser, you can expect to find something there.

That doesn't work today. I can import github.com/zellyn/goapple2/shiny but the same URL is a 404.

But phrased as a feature request, it’d be nice if vgo could tell you about the upstream/source for a given module—things like where you should contribute changes, file bugs, ask questions. That’d probably mean adding a field to the download protocol, probably free text, since it is meant for humans. For example, for major code hosts, it would probably tell you the repo url, the relevant tag or branch, and the file path within that repo. For commercial offerings perhaps it would provide a support email address.

Speaking of which, out of curiosity, how does vgo play with binary-only packages? Is there a well-defined place in the zip file for them? 

-josh

[yiyus]

I have not used vgo yet, but it looks great. Good job! (both to Russ and everyone involved in dep, etc). However, I wonder if it may be more consistent in how to name versioned packages.

The current one-to-one correspondence between import path, argument to go get and source directory inside GOPATH is a very nice feature. However, if I understood correctly from the documents, there are at least three different ways to refer to the same package in vgo:

  "rsc.io/quote" 1.2.0   // in the go.mod file

    $ vgo get "rsc.io/qu...@1.2.0"  # in the command line

    $ cd /Users/rsc/src/v/rsc.io/quote(v1.5.2)  # source dir

I think it would be better to always use the same. For example, I may want to copy the string “rsc.io/qu...@1.2.0” from the go.mod file and paste it in my terminal to run a go get command after increasing the version number, and then change to the corresponding directory to make a quick inspection.

I actually think it may be convenient to always use a "versioned import path" everywhere, including logs and error messages, and even to encourage its usage when talking about package versions in bug reports, mailing lists and so on.

Moreover, there is no reason versioned import paths could not be allowed in go source files. Minimal version selection would do the right thing anyway. But that is another topic...

[roger peppe]

Other possibilities for module-inferred major versions:

- explicitly marked default in go.mod 

- use latest version. 

I think I prefer the first of those, as it's less sensitive to arbitrary changes made to go.mod by the go tool. It also perhaps makes it reasonable to *require* that importing the default major version of a module uses a versionless path, thus ensuring consistency of import paths within a module.

[Bakul Shah]

On Fri, 23 Feb 2018 08:22:14 +0000 roger peppe <rogp...@gmail.com> wrote:
roger peppe writes:
> On 22 February 2018 at 20:19, Bakul Shah <ba...@bitblocks.com> wrote:
> > I see two problems here. 1) it breaks, or makes it harder, to
> > use different names for incompatible behavior. If I just read
> > sourcefiles of A & B and see both of them use C.Foo() I would
> > assume they refer to the same Foo(). Having the version number
> > in the pkg sourcefile is more "in your face". Harder to ignore.
> > 2) When disambiguation is used, it gets even more confusing.
> > C.Foo() in A is not the same as C.Foo() in B but Cv2.Foo() in
> > A is the same as C.Foo in B.
> >
> > What this means is now I would be forced to look in the .mod
> > files all the time while reviewing code and *remember* the
> > binding used in .mod. I am more likely to just add a comment
> > in the .go file to remind me of this difference.
>
> Yes, this is the trade-off. Note that this is the case *right now*
> with vendor directories, so this isn't something new.

But not something worth maintaining (IMHO).

> I don't think the situation is as bad as you think. In any given
> repository, we'll almost always be using a single major version of
> each module dependency, and that's what the import paths will be
> referring to. Most of the time you'll be able to relax and just use
> the normal unversioned import paths and they'll mean what you expect
> them to mean, I believe.

Yes, for your own code you'd rely on a single major version
but a third party pkg you import may be importing a different
version of the pkg you use. For example, your pkg a may have this:

import "b"
import "c" // v1

c.Foo(1)

and pkg b may have this:

import "c" // v2

c.Foo("one")

Given this, what should "go doc c.Foo" return?

Another issue. Now pkg a compiles but doesn't work right and
you suspect c may be the cause but you don't know whether it
is the version b uses or you. You want to debug this issue and
need access to both.

"Magic" paths complicate things. Ultimately they have to map
to some real paths. Vendor directories achieve this one way.
vgo has to do this differently as there is no local vendor/
to stash packages in.

> Also, tooling can help. Your editor could show the major version that
> applies, jump-to-definition will take you to the right place, etc.

So the editor has to now know about go.mod? I use acme & nvi
(not even vim)! And grep & go doc (I'd love to see go grep!).

Given the right abstractions simple tools can usually suffice.

This is one reason I shy away from autoconf/automake & their
kind that hide complexity (something breaks and you have no
choice but to learn the hidden complexity). Or eclipse & other
IDEs. In a GUI it is impossible to compose just the right
command you want out of existing commands so IDEs end up being
kitchen sinks.

> > There is also the issue of what happens when you are the
> > maintainer of pkg C. How are you going to make both versions
> > simultaneously available?
>
> I don't see the problem here. How is it different from what's proposed
> in vgo now in this respect?

Consider the case where you are maintaining pkg c (from the
above example) and helping solve the issue with pkg a. I
think you will have to maintain separate git repos for both
versions of c and once the issue is resolved upstream changes
to c's master repo.

[this is probably equally painful right now]

[Aram Hăvărneanu]

Another problem with requiring major versions in import paths is that
it makes it significantly more difficult to backport changes between
major versions. Patches won't apply cleanly because of all the churn
in import paths. Roger's proposal fixes that too.

--
Aram Hăvărneanu

[roger peppe]

I think this fits fine with the currently documented rules of go doc.
That is, it depends which directory you run the "go doc" command in.
If you run it in pkg a's directory, you'll get v1; in pkg b's
directory you'll get v2. If you're not under a module's directory,
it's arguable what it should return - perhaps the latest version
that's available, or perhaps it could refuse to work at all for
non-stdlib packages.

> Another issue. Now pkg a compiles but doesn't work right and
> you suspect c may be the cause but you don't know whether it
> is the version b uses or you. You want to debug this issue and
> need access to both.
>
> "Magic" paths complicate things. Ultimately they have to map
> to some real paths. Vendor directories achieve this one way.
> vgo has to do this differently as there is no local vendor/
> to stash packages in.

This is already the case - paths are already "magic". Try using `vgo
get` and using "vgo list -e -f '{{.Dir}}'" on one of the names of the
dependent packages. You might be surprised.

>> Also, tooling can help. Your editor could show the major version that
>> applies, jump-to-definition will take you to the right place, etc.
>
> So the editor has to now know about go.mod? I use acme & nvi
> (not even vim)! And grep & go doc (I'd love to see go grep!).

No, the editor doesn't *have* to know about go.mod. But "jump to
definition" (a.k.a. godef or guru) is already clever enough to use
local context to go to the right place. This doesn't seem too
different to me.

> Given the right abstractions simple tools can usually suffice.
>
> This is one reason I shy away from autoconf/automake & their
> kind that hide complexity (something breaks and you have no
> choice but to learn the hidden complexity). Or eclipse & other
> IDEs. In a GUI it is impossible to compose just the right
> command you want out of existing commands so IDEs end up being
> kitchen sinks.
>
>> > There is also the issue of what happens when you are the
>> > maintainer of pkg C. How are you going to make both versions
>> > simultaneously available?
>>
>> I don't see the problem here. How is it different from what's proposed
>> in vgo now in this respect?
>
> Consider the case where you are maintaining pkg c (from the
> above example) and helping solve the issue with pkg a. I
> think you will have to maintain separate git repos for both
> versions of c and once the issue is resolved upstream changes
> to c's master repo.
>
> [this is probably equally painful right now]

Yes, this is no different. In fact nothing is different in my proposal
apart from the fact that you would be able to omit major version
numbers from import paths.

cheers,
rog.

[roger peppe]

A few more thoughts about module-inferred major versions. One thing I
hadn't considered was what happens when there's an unversioned import
and no available information in the go.mod file. This happens when
we're resolving dependencies when using vgo for the first time on a
module or when a dependency is added, and also in "legacy" modules -
i.e. modules which have no go.mod file.

Please note that none of the below applies when there are go.mod files
available - this is *only* necessary when the necessary version
information isn't available. None of the rules below would need to be
considered after go.mod has been updated and there are no legacy
module dependencies.

The "top level module" below is the module that we're running the vgo
command in.

When a module imports a package within itself, it should use itself:

Rule 1 (self import). For unversioned imports in any module, then we
infer the major version of the legacy module itself.

When a legacy module has external dependencies, things become a little
less obvious. I can see a few possibilities:

1) fail - refuse to resolve and demand an explicit resolution
2) choose the latest available major version
3) choose the major version used in the top level module if there is
one; otherwise fall back to one of the other rules.
4) assume major version of v0 or v1

1 is possible but isn't very helpful - vgo already does better than this today.
2 and 3 perhaps sound attractive, but break import path determinism -
I think the major versions should always mean the same thing
regardless of what context it's being imported in.

That leaves 4 as the least-bad option, which is what vgo does right
now AIUI. The user is free to explicitly resolve legacy import paths
to v0.0.0-date-commit versions. This has the implication that
unversioned imports within all legacy modules always resolve to the
same major version, which corresponds nicely to Go's current rules:

Rule 2 (legacy import): Within a legacy module, an unversioned import
of an external module, use version 1 (if available) or 0 (if not).

That leaves the case of what happens when you try to do a build in the
top level module where information is missing. I tried to come up with
an intuitive rule that would use the latest major version by default,
but it turns out to be tricky and potentially order-dependent. So, I
think it should probably be an error if there's an unversioned import
of a module when the go.mod file doesn't mention that module.

Rule 3 (unresolvable): If there's an unversioned import that has no
corresponding entry in go.mod, the build fails.

All that's left is the originally proposed rule for inferring major
versions. Currently I think it's probably best to have some mechanism
for explicitly marking a default major version for a module in go.mod.
To resolve an error from rule 3, the user can run "go get" explicitly
(e.g. go get github.com/foo/bar@v2) to choose the default major
version for a module.

Rule 4 (resolvable): An unversioned import that has a corresponding
default entry in go.mod uses the major version from that entry.

To avoid the need to use `go get`, it would always be possible to use
a versioned import path the first time you import a given module.
External tools like goimports are free to choose whatever heuristics
they like to decide what major version to use, as they do today.

[Henrik Johansson]

I would very much like to see inferred versions as well.

[Russ Cox]

On Wed, Feb 21, 2018 at 5:31 PM, Aram Hăvărneanu <ara...@mgk.ro> wrote:

Why not just use Go code throughout? We could extend Go syntax in
a backwards-compatible way to be able to express all the information
contained in the go.mod file. The bulk of the code would continue
to import old-style, "bare", imports, while a single, or perhaps a
small number of Go files would encode the information required for
version management using new-style imports. Perhaps this file, or
files could have standardized names, zmod.go, or whatever (please
don't bikeshed the name).

I think the key point is that go.mod applies to a whole tree of packages, not just one package. When people say "why not use Go syntax" then you have to start worrying about what it means when different files say different things. What if x.go says v1.5.2 and y.go says v1.6.3? Why allow a system where you have to answer that question?

Another reason is that go.mod is rewritten by the vgo command. That's fundamental to a smooth user experience. The go command does not today rewrite any Go files as an ordinary side effect of builds ('go fmt' is different, because it's explicit). 

The information in go.mod is really not Go code. Every attempt I've seen to shoehorn it into Go code adds more rough edges than it removes.

Russ

[Sam Whited]

On Tue, Feb 20, 2018, at 11:21, Russ Cox wrote:
> For Go 1.11 I am thinking about doing this.
> https://research.swtch.com/vgo-intro.
>
> Feedback welcome.

After reading your posts on the topic, asking a number of (possibly foolish; thanks for your patience) questions, and experimenting a bit with the current prototype, I'm still a bit confused about versioning. A few things about it that stand out to me:

- Package versioning is stored "out of band" in either the Git tags or the zip file name. When fetching a package we no longer have to use Git, but servers or package release software that build zip files will still need to understand it. What happens when I'm using <future VCS>? Does the release build software need to support that too? What if it doesn't support tags?
It seems to me like tags in Git should only be for backwards compatibility with repos that are being imported "go get"-style.
- If we introduce signing of package zips later, the fact that the version is only in the filename means it's not signed (probably) unless we introduce some other sort of "info" file that gets signed with the zip. This isn't a big deal, but it seems like the version could just be somewhere that it gets signed along with the other files in the repo.
- Putting the major version in two places by convention feels very "un-go-like" because it gives me so many different and confusing ways to handle it (it could be in the physical path, it could be only in the module name, it could even be the package name). While I suspect that it only really makes sense to change the module name, this still feels confusing.

The "obvious" thing to me would be to include the version in the mod file and only use tags for backwards compatibility, but I suspect this has been thought of; what would the downside be to doing something like this, for example:

module "example.com/mypackage" v2.5.0

require (
"example.org/anotherpackage" v1.0.0
)

Instead of putting the major version in the path name and in git, if it is in the mod file we can show it in errors ("v2 oauth.Token is not of type v3 oauth.Token" or similar), and we don't need to set the version in two places, the mod file becomes the only source of truth (though of course, if we want to set it in Git we still can).
This does lose you the ability to see what major version you're getting at import time, but if that's desirable we could also introduce this syntax there or special case the /v2 type import paths (though I don't especially like this or see why it would be necessary).

import "example.com/mypackage" v2

—Sam

[Paul Jolly]

I will have to play with the API promise and minimal version selection in my head for a bit, in particular around what they will encourage in terms of deprecation and security updates.

Not sure whether your point about security updates was specifically about known problems, declared vulnerabilities etc, but here's a drive-by thought on how that could be achieved.

Building on the idea of exclusions, security problems (of varying degrees?) could be described in terms of exclusions. Particularly if exclusions could be defined with <= in addition to ==.

i.e. as the author of package X, I could declare that <= 1.5.6 is critically insecure because of problem ABC. A security expert might additionally declare that a version 1.5.8 has a problem. And as the program author (that ultimately uses X) I want to exclude 1.5.9. You get the idea.

I think this keeps vgo as a minimal version problem, overlaid with various exclusions (that could come from multiple sources with the net effect being applied at build time)

[roger peppe]

On 26 February 2018 at 15:53, Paul Jolly <pa...@myitcv.io> wrote:
>> I will have to play with the API promise and minimal version selection in
>> my head for a bit, in particular around what they will encourage in terms of
>> deprecation and security updates.
>
>
> Not sure whether your point about security updates was specifically about
> known problems, declared vulnerabilities etc, but here's a drive-by thought
> on how that could be achieved.
>
> Building on the idea of exclusions, security problems (of varying degrees?)
> could be described in terms of exclusions. Particularly if exclusions could
> be defined with <= in addition to ==.
>
> i.e. as the author of package X, I could declare that <= 1.5.6 is critically
> insecure because of problem ABC. A security expert might additionally
> declare that a version 1.5.8 has a problem. And as the program author (that
> ultimately uses X) I want to exclude 1.5.9. You get the idea.

Yes. I had a similar thought. If we have some way of cryptographically
verifying version to hash correspondence (via some online equivalent
of go.modverify), then that could also serve as a conduit for signed
security updates.

I can imagine a scheme where if you can prove you control a repo and a
private key (by adding a specifically named branch containing some
signed info, for example), then you'd be able to publish signed
statements to a verifier service to the effect that a given version or
set of versions should be excluded.

Any builder could include those exclusions when calculating the
versions to use. Note that when publishing a given version, you'd
probably want to publish the exclusions at the time of publication
too, otherwise you'd lose the current nice property that a given
version will deterministically produce a given build artifact.

[Paul Jolly]

Yes. I had a similar thought. If we have some way of cryptographically
verifying version to hash correspondence (via some online equivalent
of go.modverify), then that could also serve as a conduit for signed
security updates.

I can imagine a scheme where if you can prove you control a repo and a
private key (by adding a specifically named branch containing some
signed info, for example), then you'd be able to publish signed
statements to a verifier service to the effect that a given version or
set of versions should be excluded.

A related point came up recently in a Twitter thread (unrelated to vgo). How much of this would we get from having signed commits and tags? 

The master branch of a repo could contain all the latest security details from the package owner's perspective. And would have the added benefit of not adding to an existing workflow.

Other sources of truth for security issues could of course co-exist and would probably require a scheme like you describe.

[roger peppe]

On 26 February 2018 at 16:47, Paul Jolly <pa...@myitcv.io> wrote:
>> Yes. I had a similar thought. If we have some way of cryptographically
>> verifying version to hash correspondence (via some online equivalent
>> of go.modverify), then that could also serve as a conduit for signed
>> security updates.
>>
>> I can imagine a scheme where if you can prove you control a repo and a
>> private key (by adding a specifically named branch containing some
>> signed info, for example), then you'd be able to publish signed
>> statements to a verifier service to the effect that a given version or
>> set of versions should be excluded.
>
>
> A related point came up recently in a Twitter thread (unrelated to vgo). How
> much of this would we get from having signed commits and tags?

A tag of the form "insecure-v1.2.3" might be sufficient to mark
version 1.2.3 as insecure. I'm not sure that signing it gives that
much more security unless we insist that all other commits and tags
are signed too.

[Russ Cox]

On Thu, Feb 22, 2018 at 11:16 PM, <land...@gmail.com> wrote:

Hey Russ, am I understanding things correctly? If you don't specify which version you want to pull in as a dependency, instead of pulling the latest version, it pulls in the latest v1?  I can't be reading that correctly.  I must be mistaken there.  So lets say you pulled in firefox as a dependency (assuming it was a go package).  Instead of pulling in firefox 56 (or whatever it is on now), it would pull in firefox 1?, unless you explicitly told it to use firefox 56?

Yes but the hypothetical is misleading. The major version is part of the name. A better example is github.com/go-yaml/yaml, which has two different APIs, v1 and v2. v1 would be "github.com/go-yaml/yaml" and v2 would be "github.com/go-yaml/yaml/v2". You write the import for the API you want, and then you get the latest of that major version.

Russ

[Russ Cox]

On Fri, Feb 23, 2018 at 10:26 AM, Josh Bleecher Snyder <josh...@gmail.com> wrote:

But it would be nice not to break the rule that if you see github.com/zellyn/goapple2/v2 in an import statement, and load that URL in your browser, you can expect to find something there.

That doesn't work today. I can import github.com/zellyn/goapple2/shiny but the same URL is a 404.

But phrased as a feature request, it’d be nice if vgo could tell you about the upstream/source for a given module—things like where you should contribute changes, file bugs, ask questions. That’d probably mean adding a field to the download protocol, probably free text, since it is meant for humans. For example, for major code hosts, it would probably tell you the repo url, the relevant tag or branch, and the file path within that repo. For commercial offerings perhaps it would provide a support email address.

I agree it would be nice to get this information. It seems like a standard file in the root of the module tree would suffice. I don't think vgo needs to be involved except to tell you where the root of the module is cached.

 

Speaking of which, out of curiosity, how does vgo play with binary-only packages? Is there a well-defined place in the zip file for them? 

A binary-only package is just a directory with slightly different files in it than a regular Go package. Vgo has no special case for them.

Honestly I think very few people using binary-only packages, and I don't go out of my way to make it easy. I did consider just disallowing them in modules. But I didn't go out of my way to do that either. If you know of compelling use cases of binary-only packages I'd be interested to hear them.

Russ

[Russ Cox]

On Fri, Feb 23, 2018 at 3:05 PM, yiyus <yiyu...@gmail.com> wrote:

However, if I understood correctly from the documents, there are at least three different ways to refer to the same package in vgo:

  "rsc.io/quote" 1.2.0   // in the go.mod file

    $ vgo get "rsc.io/qu...@1.2.0"  # in the command line

    $ cd /Users/rsc/src/v/rsc.io/quote(v1.5.2)  # source dir

I think it would be better to always use the same. For example, I may want to copy the string “rsc.io/qu...@1.2.0” from the go.mod file and paste it in my terminal to run a go get command after increasing the version number, and then change to the corresponding directory to make a quick inspection.

The command line and the directory names are the same module@version. I dropped module(version) in favor of module@version only a day or two before posting everything, so a few stale references got missed. This is also why people like Ed and Peter have been using module(version) - they saw and digested an earlier version of the posts.

It's less clear to me that go.mod should say module@version, because we expect the version to vary while the module is held fixed. Also, the use of the quoted string makes sure we can handle future expansions in the path syntax nicely. (On the command-line, quoting is some other program's problem.)

Moreover, there is no reason versioned import paths could not be allowed in go source files. Minimal version selection would do the right thing anyway. But that is another topic...

It would just make the source files very misleading, since one might say import "rsc.io/qu...@v1.5.2" but really it's getting rsc.io/quote v1.6.7. One nice thing about go.mod in the current module is that vgo rewrites it, so it's never misleading like that.

Russ

[Russ Cox]

On Mon, Feb 26, 2018 at 10:53 AM, Sam Whited <s...@samwhited.com> wrote:

After reading your posts on the topic, asking a number of (possibly foolish; thanks for your patience) questions, and experimenting a bit with the current prototype, I'm still a bit confused about versioning. A few things about it that stand out to me:

- Package versioning is stored "out of band" in either the Git tags or the zip file name. When fetching a package we no longer have to use Git, but servers or package release software that build zip files will still need to understand it. What happens when I'm using <future VCS>? Does the release build software need to support that too? What if it doesn't support tags?
It seems to me like tags in Git should only be for backwards compatibility with repos that are being imported "go get"-style.

To be clear, this is really really not specific to Git. Every version control system I'm aware of, back to CVS and probably earlier, has a concept of introducing a human-chosen name for a particular revision. Personally, I am comfortable assuming that any future VCS will also have that kind of functionality in some form. And if not, then we'll look at what idioms it does have and figure out how to map them. This seems like a non-issue.

 

- If we introduce signing of package zips later, the fact that the version is only in the filename means it's not signed (probably) unless we introduce some other sort of "info" file that gets signed with the zip. This isn't a big deal, but it seems like the version could just be somewhere that it gets signed along with the other files in the repo.

The directory hashes in research.swtch.com/vcs-repro already include the version number, because the zip contains a single top-level directory module@version, and the hash includes the full file names within the zip file.

- Putting the major version in two places by convention feels very "un-go-like" because it gives me so many different and confusing ways to handle it (it could be in the physical path, it could be only in the module name, it could even be the package name). While I suspect that it only really makes sense to change the module name, this still feels confusing.

The major version is not the package name. If you import rsc.io/quote/v2, the package name should be expected to be quote, not v2.

I'm not sure I understood the rest of your point, or at least how it differs from arguing I should be able to write import "yaml" instead of "github.com/go-yaml/yaml", as long as the latter is in the mod file for disambiguation.

The "obvious" thing to me would be to include the version in the mod file and only use tags for backwards compatibility, but I suspect this has been thought of; what would the downside be to doing something like this, for example:

    module "example.com/mypackage" v2.5.0

One downside is that it's very difficult to walk up to a version control system and say "find me the one commit in which go.mod says v2.5.0", whereas "find me the commit with tag v2.5.0" is very efficient. Another downside is that there's very likely to be more than one commit in which go.mod says v2.5.0, unless you're super-disciplined and always follow the commit adding a version number with an immediate commit removing that version number. If you're not that disciplined, which one is it? The first one? The last one?

Instead of putting the major version in the path name and in git, if it is in the mod file we can show it in errors ("v2 oauth.Token is not of type v3 oauth.Token" or similar),

The compiler already knows to do this for differing import paths. There's no need for a new special case. 

 

and we don't need to set the version in two places, the mod file becomes the only source of truth (though of course, if we want to set it in Git we still can).

As I said, this justifies import "yaml" too, and I think we agree that's not a good idea.

 

Russ

[Sam Whited]

Thanks, all of those points make sense and cleared up some of my confusion, with one exception:

On Mon, Feb 26, 2018, at 12:36, Russ Cox wrote:
> Personally, I am comfortable assuming that any future VCS will also have
> that kind of functionality in some form.

That assumption seems fair.
Does this mean that whatever release tool or server we come up with still has to support all current VCS' and possibly future ones to know how to create zip files though?

> The directory hashes in research.swtch.com/vcs-repro already include the
> version number, because the zip contains a single top-level directory
> module@version, and the hash includes the full file names within the zip
> file.

Ah, I see, this directory exists in the zip file regardless of whether it exists on the original filesystem; this also was confusing me but makes perfect sense. Thanks!

—Sam

[Russ Cox]

On Mon, Feb 26, 2018 at 1:54 PM, Sam Whited <s...@samwhited.com> wrote:

Thanks, all of those points make sense and cleared up some of my confusion, with one exception:

On Mon, Feb 26, 2018, at 12:36, Russ Cox wrote:
> Personally, I am comfortable assuming that any future VCS will also have
> that kind of functionality in some form.

That assumption seems fair.
Does this mean that whatever release tool or server we come up with still has to support all current VCS' and possibly future ones to know how to create zip files though?

The as-yet hypothetical 'go release', run by authors, would need to know how to invoke the author's local version control tool. That seems OK to me. Servers do not have to know - they just work with zip files. I mean, someone could choose to write a different server as a VCS bridge, and that would be fine. But in general servers don't need to talk to VCS anymore.

Russ

[Bakul Shah]

On Mon, 26 Feb 2018 12:31:50 -0500 Russ Cox <r...@golang.org> wrote:
>
> It would just make the source files very misleading, since one might say
> import "rsc.io/qu...@v1.5.2" but really it's getting rsc.io/quote v1.6.7.
> One nice thing about go.mod in the current module is that vgo rewrites it,
> so it's never misleading like that.

Suggestion: How about

import "rsc.io/quote@v1" in .go,
"rsc.io/qu...@v1.2.3" in go.mod, and
vgo get rsc.io/qu...@v1.5.2 on commandline.

where the cmdline fetches the specified version, if so told.
go.mod can be updated by vgo so its contents show the result
of the latest vgo use. The import path just specifies the
major version as now. Basically similar syntax for similar
purpose as that is easier to remember. Semantics remain
the same as now.

[yiyus]

I am not sure having a file that asks for 1.5.2 but gets 1.6.7 is much more misleading than having a dependency that requires 1.5.2 but gets 1.6.7. Anyway, I think minor versions should generally remain in the go.mod file for convention. But I thought it would be nice to increase the version directly in the source file when a new feature is required, then let go build update go.mod, and run a tool that removes the minor versions from import paths before changes are committed.

I do not have enough experience with vgo to know if this is a great or a terrible idea.

[roger peppe]

On 26 February 2018 at 21:19, Bakul Shah <ba...@bitblocks.com> wrote:
> On Mon, 26 Feb 2018 12:31:50 -0500 Russ Cox <r...@golang.org> wrote:
>>
>> It would just make the source files very misleading, since one might say
>> import "rsc.io/qu...@v1.5.2" but really it's getting rsc.io/quote v1.6.7.
>> One nice thing about go.mod in the current module is that vgo rewrites it,
>> so it's never misleading like that.
>
> Suggestion: How about
>
> import "rsc.io/quote@v1" in .go,

This one is https://github.com/golang/go/issues/24119 FWIW.

[matthe...@gmail.com]

.mod breaks this to me:

An explicit goal for Go from the beginning was to be able to build Go code using only the information found in the source itself, not needing to write a makefile or one of the many modern replacements for makefiles. If Go needed a configuration file to explain how to build your program, then Go would have failed.

From https://golang.org/doc/articles/go_command.html

Matt

[Brian Hatfield]

I disagree with that assertion.

Please read Russ' original blog post at https://research.swtch.com/vgo-intro - especially the section on "minimal version selection".

--

You received this message because you are subscribed to the Google Groups "golang-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+unsubscribe@googlegroups.com.