vgo & semantic import versioning

[Russ Cox]

The other major response besides "don't drop vendoring!" was various exploration of ways to avoid putting the major version in the import path. I'm sympathetic to the general desire, having spent months off and on trying to figure out a way to do that myself. But it seems to me that the benefits outweigh the costs here. There are at least three very important benefits.

First, go.mod does not define the general semantics of programs. It changes exactly how the program gets built, but it does not control the semantic meaning of an import at the level of "which module are we talking about?" That is, today it's the case that if you zero go.mod and run vgo get -u, you should end up with the same answer as not zeroing go.mod and running vgo get -u, assuming that vgo can determine the right module path for your code. The requirement lists have minor effects on the build but no major effects like selecting "is this go-yaml v1 or v2?" The very limited damage of removing go.mod means that all the data for interpreting Go programs is really still in the Go programs themselves. Not so if we let go.mod specify the major version of every import.

Second, import path equality means package equality. We lost this when we introduced vendoring, and it has caused no end of interesting problems in systems processing Go source code. I would really like to avoid giving this up a second time. One important benefit of this property is that you can easily copy code from one package to another without worrying about the imports changing their meaning in the new context. The proposals to let go.mod redefine the meaning of imports at the major version level end up reintroducing a new, hidden mapping just like vendor directories did, at significant complexity. I guarantee, for example, that people will be confused about why code imported as "x/y/z" returns "x/y/v2/z" in its reflect data.

Third, import paths remain fully qualified. Back before goinstall, people shared code by git cloning into the "right" directory in their GOPATH. After goinstall, the code at github.com/jacobsa/igo/set was imported with that URL, whereas before no one, not even the code's author, had expected anything but "igo/set" or maybe even just "set". Many people objected to the longer names, but in hindsight it was clearly right to give different packages different names. We don't really notice the prefixes anymore, and they're incredibly valuable. They fully qualify the name to keep separate things separate. The arguments for dropping v2/ from the import path seem to apply equally to dropping github.com/jacoabsa/igo/, and I know we don't want to do that.

All three of these are really just different ways of saying the same thing: import paths stand alone; they don't require extra interpretation. I think this clarity helps programmers. The story in research.swtch.com/vgo-import goes so wrong exactly because the major version is being hidden from Moe, who then releases a new Moauth that ends up being incompatible with the old version, but invisibly so (unless you looked in the equivalent of the go.mod files).

The "import paths stand alone without extra interpretation" is a fundamental property of Go programs that we would very much like to preserve. Not at all costs, but certainly very nearly so. 

The part of this balance that I don't understand very well is the costs. As I said in https://blog.golang.org/toward-go2:

Convincing others that a problem is significant is an essential step. When a problem appears insignificant, almost every solution will seem too expensive. But for a significant problem, there are usually many solutions of reasonable cost. When we disagree about whether to adopt a particular solution, we're often actually disagreeing about the significance of the problem being solved.

I don't understand the development models where changing major versions so often makes sense. It seems to me that probably such churn is a mistake. But I would like to understand these situations better.

If you have experience with a project that bumps major version number frequently, can you please explain to my why that's essential to the way the project operates and why that's not impacting users?

For example, if you bump the major version when you delete a function, why not just mark the function deprecated, implement it in terms of whatever new functionality users should reach for instead, and leave it in? 

Why would you automatically issue a new major version every N months? How do you do that without losing all your users, repeatedly?

Thanks.

Russ

[land...@gmail.com]

Then you end up with the nonsense in python where you have urllib, urllib2, urllib3 apis, and then everyone just switches to the request library.   Often too many depreciated apis can be as bad of a problem as just making small breaks.  Of course large breaking changes can be detrimental to a project (look at the python2->python3 problem), if python had spread out the breaking changes, and made tools to make transitions easier, it would have been better,

Lets say you have a long running product, where you have been depreciating api surface regularly.  Eventually, you want to remove those calls, because the upkeep on them becomes a pretty serious hassle (or can pose security risks, etc). The kubernetes project is a pretty good example of this.  Their API does not follow semvar, they break far more often than semvar would suggest,  they treat minor releases as if they were major ones for purposes of their API.  For example: the kubectl for 1.9.x doesn't work with a kubernetes cluster set up with kubernetes 1.7.x

Also, often very long-running projects break APIs regularly after they have been depreciated for a very long time (often years).  This is more common with older packages.    They then have to bump major releases as those apis are removed.

Lastly, you don't have to do that.  With proper version management  (like cargo, etc) you can make breaking changes all day long, and people can continue to use your older APIs without any problem, but it pushes people to use the latest version for new projects.

Example:  Every 6 months big project issues a list of depreciations, then 3 years later, they remove those depreciated APIs, well, what happens in another 6 months?  They have to make breaking changes again, to remove the next batch of depreciated APIs from 3 years ago.  So, while for 3 years they weren't issuing a lot of major version bumps, they are now.  This is pretty common with, very large apps, like web browsers.

Why would you automatically issue a new major version every N months? How do you do that without losing all your users, repeatedly?

 
Possibilities 1)  It is not unusual projects that are mostly internal to an org, especially for medium size organizations.
2) If you break things in small ways, the improvements to the project can outweigh the damage, and if you hadn't made some of the changes another competitor project would have, and then stole mind-share or market share from you.  This is pretty common in languages that have good package management and move very quickly, and have a lot of transitive deps.  Javascript sees this behavior a lot.


One this I think needs changing in vgo is that a naked version means v0-v1.  This is a problem if you already have tags out for v2 and above.  A naked version should try to pull the latest possible thing, and if you want to freeze to v1, you should do so explicitly.  v1 being "magical" is a bad idea, and goes against the principle of least surprise.  Its different enough from how every single other package manager does things, that it will cause problems.

[Henrik Johansson]

I just think I have gotten very used to github.com/lib/pq to also refer to that path in GOPATH and attaching /v2 at the end comes with strange connotations of "versioning by copying directories" which I have always seen as an anti pattern but admittedly done it myself once or twice.

The missing go.mod file reproducibility problem can be the same even with inferred versions right? So would the Moauth problem.

The tools (compiler, editors etc) just need to know the version to use, it doesn't have to be in the path.

If we are anyway going to write tools to rewrites major path bits then why not externalize it to go.mod in the first place since there won't be any real resistance to bumping.

I guess you have read these arguments and I guess the "import path stands alone" (IPSA) property has been very powerful but I am fairly certain that it's value is greatly diminished in light of vgo and versioning even to the point as to devalue both the IPSA and vgo itself because it splinters the version definitions into several places. Before it used to be one and which exact version was determined by tools and vendoring. Now it will be several places and possibly determined by vendoring as well. In keeping with the Go spirit it is my opinion that inferring major version from go.mod is the more Go'ish way than keeping versions scattered and possibly conflicting.

2¢

--
You received this message because you are subscribed to the Google Groups "golang-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Henrik Johansson]

Actually, I am not sure now...

The Unity example I guess needs to not just allow for the existence of two versions of moauth but also usage of these in the main program. Perhaps creating config for the two versions of moauth and then passing these on to the respective AWS lib.

If the go.mod had a name classifier such as the import statement itself then the inferred version would also work but perhaps it's bang is lost then.

Maybe I just need to digest the problem space more. At least it feels better now as long as tooling would help with version up- or downgrading.

[skinn...@gmail.com]

I just want to add some viewpoints from a consumer of packages.  Breaking changes are bad. I hate them. Starting a week where I discover that I need to update to a breaking version of a new library to get the bug-fix I want sucks all enjoyment out of being a programmer, quite simply because usually I get nothing out of the breaking change. Sure, the API might look nicer. Do I get anything out of it? Usually not.

The node ecosystem is horrible at this. One of the main reasons I use Go is precisely because of the lack of breaking changes. The StdLib is sound, and the packages I've relied on haven't broken BC yet (been using Go for about a year). This, to me, is Go's greatest feature.

Jorge: You seem to suggest that switching from urlib2 -> urlib3 -> request is a bad thing. It isn't. That's the preferred way. Use urlib2 as long as it does what you want, if it doesn't, maybe urlib3 does, or maybe a competing library? The way JS works is horrible. JS-fatigue is not a compliment to the ecosystem, it's its worst feature.

So from a consumer standpoint, changing the import path every once in a while (like, at most every 2 years) is fine. When BC is broken anyway, changing a few import statements is not where one spends time. Also, making it more difficult to issue major breaking changes (even if it's just a little bit) is good as well. Breaking changes are not to be made lightly, and in several other languages, they are made too lightly.

I hope this wasn't too much off topic. The direction vgo is going is really exciting, and I just wanted to my thumbs up.

[Aram Hăvărneanu]

> The part of this balance that I don't understand very well is the
> costs. As I said in https://blog.golang.org/toward-go2:
>
>> Convincing others that a problem is significant is an essential
>> step. When a problem appears insignificant, almost every solution
>> will seem too expensive. But for a significant problem, there are
>> usually many solutions of reasonable cost. When we disagree about
>> whether to adopt a particular solution, we're often actually
>> disagreeing about the significance of the problem being solved.
>
> I don't understand the development models where changing major
> versions so often makes sense. It seems to me that probably such
> churn is a mistake. But I would like to understand these situations
> better.
>
> If you have experience with a project that bumps major version
> number frequently, can you please explain to my why that's essential
> to the way the project operates and why that's not impacting users?

It's common in organizations that primarily deliver an executable
product (not a library) and have different components developed by
different, independent teams. The producers and consumers of these
components belong to the same organization. These components may,
or may not be exposed publicly, but even if they are publicly
exposed, the development focus is towards providing support for the
other team in order to achieve a specific common goal, not to support
any other 3rd party consumer.

> If you have experience with a project that bumps major version
> number frequently, can you please explain to my why that's essential
> to the way the project operates and why that's not impacting users?

It is impacting users, but it's impacting mostly (or solely) internal
users, and the cost of this is offset by whatever is gained by the
change in the first place.

> For example, if you bump the major version when you delete a function,
> why not just mark the function deprecated, implement it in terms
> of whatever new functionality users should reach for instead, and
> leave it in?

That's possible, and probably the right choice when you are developing
a library that is supposed to be used by arbitrary people for
arbitrary reasons. However, in the case described above libraries
are only (or mostly) used internally, and the expectation is that
all code will migrate to the new API, so there won't be any more
consumers left that still use the old API. In this case maintaining
the old API is a cost that doesn't provide any benefit.

Someone might ask: "if there's no desire to maintain old APIs
anymore, why not update both the producers and the consumers
simultaneously?"

The answer is pretty obvious, it's the same reason we got type
aliases. Different teams need to move forward at their own pace.
They might not have the right, or the desire, to commit to each
other's source tree.

There's another reason why organizations might chose to use versioned
libraries internally, one that's more subtle. In big projects it's
normal for different teams to work on mutually-dependent seperate
modules, that like before, are more or less independently developed.
Go packages can't form cycles, so such organizations need to create
a 3rd, common package that both teams will import and both will
have to modify as time passes by. This requires collaboration between
teams because now they both have to work on the same code. This
creates friction between teams as their own different schedules
might not be ideal for these kind of modifications.

With versioned Go modules, this problem is much aleviated. Modules
can be mutually dependent. Of course there is still a need for
intermediary packages, but there is less of a need for coordination
when making changes to these packages. Versioning allows for a
"two-phase-commit" type of development where each team can make the
changes they deem necessary, and it is understood that the other
team will need to pick up. Preferabily as soon as possible, but
ultimately they can pick it up at their own pace.

--
Aram Hăvărneanu

[Nic Pottier]

If you have experience with a project that bumps major version number frequently, can you please explain to my why that's essential to the way the project operates and why that's not impacting users?

We use go-chi and it moves pretty quick while also using semantic versioning. I'd say that does indeed impact users and yes it could probably be minimized with more thought but some authors have a bias towards removing all cruft and bumping major versions instead of keeping backwards compatibility. I think that is rational behavior if you believe your future users will be greater than your present users, breaking changes are ok because you believe they will allow you to move faster and gain more users. (as opposed to being weighed down supporting old users) 

It does become MORE onerous if we also now have to go rewrite all our import paths, though I've now totally come around to import paths including major versions. Could be nice to have that built into `go` somehow (maybe have a go get @ major version rewrite direct imports if called with the appropriate flag?) but I'm sure the community will build something soon enough if not.

In any case, after playing around with vgo more over the past week and moving a few projects and libraries over, I'm now totally sold on the semantic versioning approach and paths as versions. It feels slightly weird at first and import rewrites will be painful the first time around, but the benefits outweigh the costs I think. I think there will be a go cultural impact to major versioning changes being more painful downstream, but I think that'll actually be a good thing or at worst we don't know and it'll be a fun experiment.

Would still like to see an easy way to have guaranteed reproducible offline builds but I see there's another thread on that.

-Nic

[Ian Lance Taylor]

On Mon, Feb 26, 2018 at 12:28 PM, Russ Cox <r...@golang.org> wrote:
>
> If you have experience with a project that bumps major version number
> frequently, can you please explain to my why that's essential to the way the
> project operates and why that's not impacting users?

I think that the examples that people are giving show that the answer
is fairly simple: some packages are willing to break their API and,
for one reason or another, don't care about the effect on their users.
Their users are simply expected to upgrade to the new version.

For a package following this development strategy, semver is just a
fig leaf, a way of recording the breaking change but not a signal of
any intent to avoid breaking changes.

This means that there is a different approach that such a package can
take. Don't change the major version. Don't change the import path.
Treat minor version changes as breaking changes. When a new minor
version comes out, every user will update their go.mod files to block
that version (or they will simply avoid updating their dependencies).
They will then update on their own time.

The point is, for packages that make regular breaking changes, users
have to regularly adjust somehow. We can make them adjust by
regularly changing their import path, or we can make them adjust by
regularly changing their go.mod file. vgo supports both models. Each
package can decide what is best for their users.

We could make this simpler by having the go.mod file have a way to
lock to a minor version. I don't know if that would cause any trouble
for minimal version selection. I can't think of a problem other than
the fact that there will be more possibilities for unresolvable
conflicts, but that is in practice unavoidable for a package that
makes regular breaking changes.

Ian

[Ian Lance Taylor]

On Mon, Feb 26, 2018 at 12:28 PM, Russ Cox <r...@golang.org> wrote:
>

> The other major response besides "don't drop vendoring!" was various
> exploration of ways to avoid putting the major version in the import path.

This was probably pointed out already, but one oddity of the current
proposal is that the final element of the import path no longer
matches the package name. Since this final element is not necessarily
intended to match a path name in the repository, did you consider
alternate syntax to make the major version look less like a path name?
For example, the language spec already permits implementations to
reject import paths containing '#', so instead of
"github.com/ianlancetaylor/demangle/v2" we could write
"github.com/ianlancetaylor/demangle#v2". That would keep all the
desirable properties of import paths while clarifying that the major
version is not literally a path name.

Ian

[Ian Lance Taylor]

I see now that this is https://golang.org/issue/24119. Sorry for the noise.

Ian

[Axel Wagner]

Just responding to one specific question:

On Mon, Feb 26, 2018 at 9:28 PM Russ Cox <r...@golang.org> wrote:

For example, if you bump the major version when you delete a function, why not just mark the function deprecated, implement it in terms of whatever new functionality users should reach for instead, and leave it in?

My 2¢ is, that if you want to remove a declaration, you want to stop supporting it, slim down the API, remove unnecessary code and just in general for people to stop using it. I think it'd be fine to leave them rot in a dedicated deprecated.go file that is understood to contain deprecated stuff, but it should come with more of a push from the tools to not use them. Examples of something like that would be

* Not showing deprecated declarations in godoc

* Showing a compiler-warning when a deprecated declaration is used

* Making go-vet fail when a deprecated declaration is used (though given that go-vet will be a test-failure in the future, that might be too close to an actual removel)

* Making vgo error out (or more mild: warn), if a deprecated declaration is used, when it upgrades to a newer version (either on the initial addition or when using vgo get -u)

There should at least be a very clear incentive to not use deprecated APIs. Currently there is none, deprecation is effect-free.

[Chris Hines]

Go has basic support for marking things deprecated. It's not used much in the community for various reasons, one being that it's not prominently documented so not many people know about it.

The following is quoted from https://blog.golang.org/godoc-documenting-go-code

Sometimes a struct field, function, type, or even a whole package becomes redundant or unnecessary, but must be kept for compatibility with existing programs. To signal that an identifier should not be used, add a paragraph to its doc comment that begins with "Deprecated:" followed by some information about the deprecation.

The idea of go vet or golint reporting use of deprecated things was discussed and considered inappropriate or out of scope almost two years ago: https://github.com/golang/lint/issues/238 and https://groups.google.com/d/msg/golang-dev/xok4FvP3WEM/OueVwEWrAwAJ.

But staticheck does implement the check: https://staticcheck.io/docs/staticcheck

Chris  

[Axel Wagner]

On Tue, Feb 27, 2018 at 7:00 PM Chris Hines <ggr...@cs-guy.com> wrote:

There should at least be a very clear incentive to not use deprecated APIs. Currently there is none, deprecation is effect-free.

Go has basic support for marking things deprecated. It's not used much in the community for various reasons, one being that it's not prominently documented so not many people know about it.

The following is quoted from https://blog.golang.org/godoc-documenting-go-code

Sometimes a struct field, function, type, or even a whole package becomes redundant or unnecessary, but must be kept for compatibility with existing programs. To signal that an identifier should not be used, add a paragraph to its doc comment that begins with "Deprecated:" followed by some information about the deprecation.

The idea of go vet or golint reporting use of deprecated things was discussed and considered inappropriate or out of scope almost two years ago: https://github.com/golang/lint/issues/238 and https://groups.google.com/d/msg/golang-dev/xok4FvP3WEM/OueVwEWrAwAJ.

But staticheck does implement the check: https://staticcheck.io/docs/staticcheck

The problem isn't marking things as deprecated - it is making that deprecation having an actual effect. As long as people have to go out of their way to find out that they are using something that shouldn't exist anymore, I personally don't consider deprecation an effective tool.

[jimmy frasche]

https://github.com/golang/go/issues/17056 is for hiding deprecated by
default items in godoc

[Chris Hines]

Agreed. I thought it was worthwhile to point out the current level of support for deprecating things in Go packages and provide references to prior discussion for anyone that was curious.

[Chris Hines]

I am most worried about the migration path between a pre-vgo world and a vgo world going badly. I think we risk inflicting major pain on the Go community if there isn't a smooth migration path. Clearly the migration cannot be atomic across the whole community, but if I've understood all that you've written about vgo so far, there may be some situations where existing widely used packages will not be usable by both pre-vgo tools and post-vgo tools.

Specifically, I believe that existing packages that already have tagged releases with major versions >= 2 will not work with vgo until they have a go.mod file and also are imported with a /vN augmented import path. However, once those changes are made to the repository it will break pre-vgo uses of the package.

This seems to create a different kind of diamond import problem in which the two sibling packages in the middle of the diamond import a common v2+ package. I'm concerned that the sibling packages must adopt vgo import paths atomically to prevent the package at the top of the diamond from being in an unbuildable state whether it's using vgo or pre-vgo tools.

I haven't seen anything yet that explains the migration path in this scenario.

Thanks,

Chris

[Nazri Ramliy]

On Tue, Feb 27, 2018 at 4:28 AM, Russ Cox <r...@golang.org> wrote:
> The other major response besides "don't drop vendoring!" was various
> exploration of ways to avoid putting the major version in the import path.
> I'm sympathetic to the general desire, having spent months off and on trying
> to figure out a way to do that myself. But it seems to me that the benefits
> outweigh the costs here. There are at least three very important benefits.

Why limit the version in the import path to only the major version?
Consider the benefit of having the exact version there. This way there
is no ambiguity as to what version will be used:

package frob

import "foo"
// Use "foo", as is, from stdlib or $GOPATH

import "github.com/foo/bar"
// If it isn't in $GOPATH, Get it from github, use the latest one

import "github.com/foo/bar#1.2.3"
// Get it from github if necessary, and use the version tagged as "1.2.3"

import "example.com/fub/blu#34-rc2"
// Get it from github if necessary, and use the version tagged as
// "34-rc2" (it doesn't have to be semantic)

import oldbar "github.com/foo/bar#1.2.3"
import newbar "github.com/foo/bar#2.0.1"

import "example.com/foo/bar#d74aa80c579822078a33d9c37f74f2600a63c440"
// Get it from github if necessary, and use the revision at commit
d74aa80c579822078a33d9c37f74f2600a63c440
// Yes the import path is going to look "ugly", but I don't see
// any downside to this. I litte harder to write, yes, but tooling
// can help.

This would obviate go.mod, wouldn't it? Also at the source code level
there's no concern of whether the imported package uses semantic
versioning.

nazri

[Robin Heggelund Hansen]

Because go packages are supposed to be backwards compatible unless there’s a major version, having the entire version number in the import path isn’t necessary.

Really, the way to look at it is that it’s not allowed to break backwards compatibility unless one creates a new package. Adding a V2 to the import path is just a very easy way of doing just that.

> --
> You received this message because you are subscribed to a topic in the Google Groups "golang-dev" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/golang-dev/Plc42fslQEk/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to golang-dev+...@googlegroups.com.

[Josh Bleecher Snyder]

> If you have experience with a project that bumps major version number
> frequently, can you please explain to my why that's essential to the way the
> project operates and why that's not impacting users?

One case is Go projects that wrap or track an upstream project that
regularly bumps major version numbers. Two examples that come to mind
are go-clang and git2go. (And interestingly, git2go wraps libgit2, and
the libgit2 readme explains that they break backwards compatibility
because they themselves track git.)

-josh