How to release a new major version of a library?

[Ross Light]

Hi Package Management Working Group,

Jaana and I have been contemplating releasing a major version 2 to golang.org/x/oauth2.  It has many mistakes in its public API, to the point where maintaining two separate versions is acceptable to both of us.  We'd like to hear your thoughts on how the proposed dep tool can handle this, or whether its design should be amended to assist in this case.

To give background, the oauth2 package is a quite widely used package throughout the Go ecosystem.  In particular, one of its types is particularly pervasive in exported APIs: oauth2.TokenSource.  It looks like this:

type TokenSource interface {

  Token() (*Token, error)

}

For various reasons, we'd like to change the API in a trivial but backward incompatible way (simplifying a bit):

type TokenSource interface {

  Token(context.Context) (*Token, error)

}

Understandably, we do not want to mandate that every package that depends on oauth2 to make this conversion immediately.  So in the meantime, we'd like to allow binaries to have both types of TokenSource available.  The package does not have global mutable state concerns.  And we've structured the new API such that converting between these types can be written as functions:

func TokenSourceToV2(TokenSource) v2.TokenSource

func TokenSourceFromV2(v2.TokenSource) TokenSource

Now comes our problem: where do we put this new v2?  of which I see three options: 1) a subdirectory of the existing repository 2) a new branch or 3) a new repository

Placing v2 in a subdirectory seems not ideal since the SemVer tags on the repo would also include the v2 directory, so they would be somewhat meaningless.  I'm not going to explore this option in any great depth.

Placing v2 on a new branch and creating a new SemVer tag seems to be the preferred option for the dep tool.  However, IIUC both packages would have the same import path, so only one would be allowed in an application.  This would be an exceedingly frustrating experience for all users of oauth2, because an application is now stuck in all-or-nothing scenario.  The application would have to ask all of their dependencies to either upgrade or downgrade their dependency on oauth2 to v1 or v2.  AFAICT, there would be no way to write the two conversion functions above, since only one version of a particular repository is allowed at one time.  This is unfortunate, because we as library maintainers would be happy to provide utility code to allow the two to peacefully coexist.

So let's examine the new repository option.  (For most purposes, this is identical to the gopkg.in approach.)  This would allow us as library maintainers to have v1 and v2 coexist, and add utility functions in a new minor version of v1 that can up-convert to v2.  This way, an application can gradually upgrade their v1 and v2 usage without requiring other dependencies to be rewritten to use v2.  However, what the library maintainer loses in this approach is that they have to push a new URL to developers learning about the package. (This is what I've done for go-capnproto vs go-capnproto2, for instance.)  It feels a bit clumsy, since branches ought to be able to solve this.  However, this is an approach that works today, even in the current `go get` model.

I think this is an important use case for the dep tool, and I want to make sure the oauth2 package is setting the right precedent here.  If I were to ask for a feature, it would be for major semantic versions to be treated as wholly separate packages.  Along with type aliases, this would allow library maintainers to assist their users in gradually moving from one major semantic version to another.

Thanks,

-Ross

[Nathan Youngman]

Hi Ross,

If everyone was already using dep, I'd personally prefer to see only a single version of the oauth package used in a given project without conversion functions and what-not. A release-branch.oauth1.0 could be maintained for backporting fixes for users still locked to version 1, but master would be the latest and greatest. Yes, that does mean all intermediate dependencies would need to upgrade to provide compatibility with v2, and be versioned themselves. Code would need to be written and pull requests opened, but in my mind that is a good thing. Everyone contributes and the ecosystem moves on.

In reality, dep is still very new, and it's hard to even say if gophers are all using the vendor folder (one would hope). Introducing breaking changes on master would be unfortunate for anyone using the go get mechanism. So for now, my suggestion would be to fall back on the past approaches.

https://golang.org/doc/faq#get_version

That is, start a new repo with a new import path, or avoid breaking changes in the first place. Maybe you can add the desired functionality in another way (e.g. TokenSourceWithContext).

Good luck,

Nathan.

P.S. These are just my opinions, you would do well to hear from the dep team too.

[Jaana Burcu Dogan]

On Thu, May 18, 2017 at 12:48 PM, Nathan Youngman <he...@nathany.com> wrote:

Hi Ross,

If everyone was already using dep, I'd personally prefer to see only a single version of the oauth package used in a given project without conversion functions and what-not. A release-branch.oauth1.0 could be maintained for backporting fixes for users still locked to version 1, but master would be the latest and greatest. Yes, that does mean all intermediate dependencies would need to upgrade to provide compatibility with v2, and be versioned themselves. Code would need to be written and pull requests opened, but in my mind that is a good thing. Everyone contributes and the ecosystem moves on.

In reality, dep is still very new, and it's hard to even say if gophers are all using the vendor folder (one would hope). Introducing breaking changes on master would be unfortunate for anyone using the go get mechanism. So for now, my suggestion would be to fall back on the past approaches.

We can keep the master branch as it is and start the development of v2 on a new branch. The only downside is that godoc.org currently doesn't support branches, and we don't have a clear story when to replace the master branch with the latest version branch in the future once everyone is off of go get.

It would be so nice to start thinking about a migration plan for repos that are willing to use dep. Not everyone is automagically switch to dep overnight, there will be a migration period and we would like to know what are the best practices.

[Nathan Youngman]

Indeed, we do need to start thinking about a plan to transition repos to dep. 

What this post brings to mind is the motivation -- when people decide to transition to dep, whether as a maintainer or a consumer of packages.

Dave Cheney and myself have been proclaiming for some time that we need to start tagging our repos with versions -- so the meta data is there for tools to work with. That's a good first step.

https://github.com/golang/oauth2/tags

https://dave.cheney.net/2016/06/24/gophers-please-tag-your-releases

https://nathany.com/os-release/

Transitioning from a stable master approach to an approach with stable tagged versions and ever-changing master is a big deal and I can't say I've given it much thought.

I did this once for fsnotify with the help of gopkg.in, but I forked the repository to a new location before doing so. That was using tags and an unstable master, which should work perfectly well with dep. I've seen a number of Canonical projects with v2 and v3 branches (also using gopkg.in) but I found it a bit confusing as a user, and as you say, how does one transition back to using master? I don't know what the solution is.

If it's possible to defer the breaking changes in oauth2 until a later cleanup phase, perhaps dep will be more wildly adopted by then?

--

Nathan Youngman 
Email: he...@nathany.com
Web: https://nathany.com

[Ross Light]

That's why I'm trying to understand, how will dep help with a new v2?  One part here troubles me, replying inline:

On Thu, May 18, 2017 at 12:48 PM, Nathan Youngman <he...@nathany.com> wrote:

Yes, that does mean all intermediate dependencies would need to upgrade to provide compatibility with v2, and be versioned themselves.

This seems untenable.  It seems like a disincentive to update to the new version, since once you have a major change to library foo, you need to have a major version rev to every library that depends on foo.  There are many ways that a dependent library bar could update in backwards-compatible ways to use a new foo version, and it seems unfair to force bar to increment major version.

-Ross

[Nathan Youngman]

you need to have a major version rev to every library that depends on foo

That's not necessarily true. A library that depends on foo may not itself have any API changes when it upgrades to support foo v2. That could be a minor update with only internal changes. It is the job of dep to look at the whole graph and determine whether or not foo v2 or v1 is appropriate, or whether there is a conflict where both versions are being requested.

For this to work, the libraries do need to participate in the dep ecosystem (unless dep happily handles libraries with no version at all... which it may? I'm not sure).

That could be an issue in terms of transition. If project owners are only looking at dep and starting to tag versions when they have breaking changes, dep is stuck trying to get unversioned libraries to play in the versioned ecosystem.

Sam or someone else from the dep team will have a better answer to how that works or will work...

--
You received this message because you are subscribed to the Google Groups "Go Package Management" group.
To unsubscribe from this group and stop receiving emails from it, send an email to go-package-management+unsub...@googlegroups.com.
To post to this group, send email to go-package-management@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Andrew Gerrand]

I think a new repository with a different path is the way to go.

It's effectively a new package.

Andrew

--

[Sarah Adams]

To feed the point that we need multiple versions in a binary, here is an example:

Let's use this oauth2 pkg as an example. The old API is crusty, awful and broken. So as an application author, I want to move my code to use v2 of oauth2. 

Many of my dependencies also use oauth2. Thus, I have to wait for each of my dependencies' authors to get up off their asses, make the change to incorporate oauth2 v2, and tag a new minor release before I can even think about moving my own code to oauth2. 

So logically/mathematically, yes I agree, we should be able to have a single version of any given pkg in a binary.

But practically, OSS maintainers are strapped for time. Some of them are going to be very slow to update. Even if users send PRs for all of these changes (which most won't), maintainers still have to merge. 

This set of slow package maintainers, however small, will hold everyone else back.

Steve Francia recently did a survey. There was a question around this - whether or not folks would like to have multiple versions of their pkg in a binary. Perhaps you've seen it already? The majority of responders wanted multiple versions in a single binary. I can dig it up if you haven't seen it.

So even if you disagree with my example, there is no denying that users want multiple versions of a package in a binary. If dep won't help them do it, they will work *around* dep as it is designed today, not with it. 

If the only way for a packages to co-exist with other versions of itself is to create a new pkg, then this will become the new idiom for a major release in Go.

I think that we can provide a better user experience than this.

- Sarah

On Thu, May 18, 2017 at 2:10 PM, Andrew Gerrand <a...@google.com> wrote:

I think a new repository with a different path is the way to go.

It's effectively a new package.

Andrew

--

To unsubscribe from this group and stop receiving emails from it, send an email to go-package-management+unsubscri...@googlegroups.com.

[Nathan Youngman]

Hi Sarah,

Thanks for the concrete example. 

I agree that maintainers being slow to update or even merge pull requests is an issue. I'm one of those bad maintainers, which is why I've tried to recruit other people to take over. 

The solution I'd like to see to help with the slow maintainer issue is to specify an in-place replacement in the dep manifest to use my own fork of the library that I've patched. (And opened a pull request for)

But note that I don't represent the dep team. This is just my opinion. 

Cheers,

Nathan. 

On Thu, May 18, 2017 at 4:34 PM 'Sarah Adams' via Go Package Management <go-package...@googlegroups.com> wrote:

To feed the point that we need multiple versions in a binary, here is an example:

Let's use this oauth2 pkg as an example. The old API is crusty, awful and broken. So as an application author, I want to move my code to use v2 of oauth2. 

Many of my dependencies also use oauth2. Thus, I have to wait for each of my dependencies' authors to get up off their asses, make the change to incorporate oauth2 v2, and tag a new minor release before I can even think about moving my own code to oauth2. 

So logically/mathematically, yes I agree, we should be able to have a single version of any given pkg in a binary.

But practically, OSS maintainers are strapped for time. Some of them are going to be very slow to update. Even if users send PRs for all of these changes (which most won't), maintainers still have to merge. 

This set of slow package maintainers, however small, will hold everyone else back.

Steve Francia recently did a survey. There was a question around this - whether or not folks would like to have multiple versions of their pkg in a binary. Perhaps you've seen it already? The majority of responders wanted multiple versions in a single binary. I can dig it up if you haven't seen it.

So even if you disagree with my example, there is no denying that users want multiple versions of a package in a binary. If dep won't help them do it, they will work *around* dep as it is designed today, not with it. 

If the only way for a packages to co-exist with other versions of itself is to create a new pkg, then this will become the new idiom for a major release in Go.

I think that we can provide a better user experience than this.

- Sarah

On Thu, May 18, 2017 at 2:10 PM, Andrew Gerrand <a...@google.com> wrote:

I think a new repository with a different path is the way to go.

It's effectively a new package.

Andrew

On 19 May 2017 at 05:14, 'Ross Light' via Go Package Management <go-package...@googlegroups.com> wrote:

Hi Package Management Working Group,

Jaana and I have been contemplating releasing a major version 2 to golang.org/x/oauth2.  It has many mistakes in its public API, to the point where maintaining two separate versions is acceptable to both of us.  We'd like to hear your thoughts on how the proposed dep tool can handle this, or whether its design should be amended to assist in this case.

To give background, the oauth2 package is a quite widely used package throughout the Go ecosystem.  In particular, one of its types is particularly pervasive in exported APIs: oauth2.TokenSource.  It looks like this:

type TokenSource interface {

  Token() (*Token, error)

}

For various reasons, we'd like to change the API in a trivial but backward incompatible way (simplifying a bit):

type TokenSource interface {

  Token(context.Context) (*Token, error)

}

Understandably, we do not want to mandate that every package that depends on oauth2 to make this conversion immediately.  So in the meantime, we'd like to allow binaries to have both types of TokenSource available.  The package does not have global mutable state concerns.  And we've structured the new API such that converting between these types can be written as functions:

func TokenSourceToV2(TokenSource) v2.TokenSource

func TokenSourceFromV2(v2.TokenSource) TokenSource

Now comes our problem: where do we put this new v2?  of which I see three options: 1) a subdirectory of the existing repository 2) a new branch or 3) a new repository

Placing v2 in a subdirectory seems not ideal since the SemVer tags on the repo would also include the v2 directory, so they would be somewhat meaningless.  I'm not going to explore this option in any great depth.

Placing v2 on a new branch and creating a new SemVer tag seems to be the preferred option for the dep tool.  However, IIUC both packages would have the same import path, so only one would be allowed in an application.  This would be an exceedingly frustrating experience for all users of oauth2, because an application is now stuck in all-or-nothing scenario.  The application would have to ask all of their dependencies to either upgrade or downgrade their dependency on oauth2 to v1 or v2.  AFAICT, there would be no way to write the two conversion functions above, since only one version of a particular repository is allowed at one time.  This is unfortunate, because we as library maintainers would be happy to provide utility code to allow the two to peacefully coexist.

So let's examine the new repository option.  (For most purposes, this is identical to the gopkg.in approach.)  This would allow us as library maintainers to have v1 and v2 coexist, and add utility functions in a new minor version of v1 that can up-convert to v2.  This way, an application can gradually upgrade their v1 and v2 usage without requiring other dependencies to be rewritten to use v2.  However, what the library maintainer loses in this approach is that they have to push a new URL to developers learning about the package. (This is what I've done for go-capnproto vs go-capnproto2, for instance.)  It feels a bit clumsy, since branches ought to be able to solve this.  However, this is an approach that works today, even in the current `go get` model.

I think this is an important use case for the dep tool, and I want to make sure the oauth2 package is setting the right precedent here.  If I were to ask for a feature, it would be for major semantic versions to be treated as wholly separate packages.  Along with type aliases, this would allow library maintainers to assist their users in gradually moving from one major semantic version to another.

Thanks,

-Ross

--

You received this message because you are subscribed to the Google Groups "Go Package Management" group.

To unsubscribe from this group and stop receiving emails from it, send an email to go-package-manag...@googlegroups.com.
To post to this group, send email to go-package...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Go Package Management" group.

To unsubscribe from this group and stop receiving emails from it, send an email to go-package-manag...@googlegroups.com.
To post to this group, send email to go-package...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Dave Cheney]

I'd be keen to see those results. I'm concerned that people who said "yes, I want [to allow] multiple copies of a package in my binary" do not understand what they are asking for, or are applying an interpretation for how they think it​ might work in Go based on prior experience with a language with a different linking model. 

On Fri, 19 May 2017, 08:34 'Sarah Adams' via Go Package Management <go-package...@googlegroups.com> wrote:

To feed the point that we need multiple versions in a binary, here is an example:

Let's use this oauth2 pkg as an example. The old API is crusty, awful and broken. So as an application author, I want to move my code to use v2 of oauth2. 

Many of my dependencies also use oauth2. Thus, I have to wait for each of my dependencies' authors to get up off their asses, make the change to incorporate oauth2 v2, and tag a new minor release before I can even think about moving my own code to oauth2. 

So logically/mathematically, yes I agree, we should be able to have a single version of any given pkg in a binary.

But practically, OSS maintainers are strapped for time. Some of them are going to be very slow to update. Even if users send PRs for all of these changes (which most won't), maintainers still have to merge. 

This set of slow package maintainers, however small, will hold everyone else back.

Steve Francia recently did a survey. There was a question around this - whether or not folks would like to have multiple versions of their pkg in a binary. Perhaps you've seen it already? The majority of responders wanted multiple versions in a single binary. I can dig it up if you haven't seen it.

So even if you disagree with my example, there is no denying that users want multiple versions of a package in a binary. If dep won't help them do it, they will work *around* dep as it is designed today, not with it. 

If the only way for a packages to co-exist with other versions of itself is to create a new pkg, then this will become the new idiom for a major release in Go.

I think that we can provide a better user experience than this.

- Sarah

On Thu, May 18, 2017 at 2:10 PM, Andrew Gerrand <a...@google.com> wrote:

I think a new repository with a different path is the way to go.

It's effectively a new package.

Andrew

On 19 May 2017 at 05:14, 'Ross Light' via Go Package Management <go-package...@googlegroups.com> wrote:

Hi Package Management Working Group,

Jaana and I have been contemplating releasing a major version 2 to golang.org/x/oauth2.  It has many mistakes in its public API, to the point where maintaining two separate versions is acceptable to both of us.  We'd like to hear your thoughts on how the proposed dep tool can handle this, or whether its design should be amended to assist in this case.

To give background, the oauth2 package is a quite widely used package throughout the Go ecosystem.  In particular, one of its types is particularly pervasive in exported APIs: oauth2.TokenSource.  It looks like this:

type TokenSource interface {

  Token() (*Token, error)

}

For various reasons, we'd like to change the API in a trivial but backward incompatible way (simplifying a bit):

type TokenSource interface {

  Token(context.Context) (*Token, error)

}

Understandably, we do not want to mandate that every package that depends on oauth2 to make this conversion immediately.  So in the meantime, we'd like to allow binaries to have both types of TokenSource available.  The package does not have global mutable state concerns.  And we've structured the new API such that converting between these types can be written as functions:

func TokenSourceToV2(TokenSource) v2.TokenSource

func TokenSourceFromV2(v2.TokenSource) TokenSource

Now comes our problem: where do we put this new v2?  of which I see three options: 1) a subdirectory of the existing repository 2) a new branch or 3) a new repository

Placing v2 in a subdirectory seems not ideal since the SemVer tags on the repo would also include the v2 directory, so they would be somewhat meaningless.  I'm not going to explore this option in any great depth.

Placing v2 on a new branch and creating a new SemVer tag seems to be the preferred option for the dep tool.  However, IIUC both packages would have the same import path, so only one would be allowed in an application.  This would be an exceedingly frustrating experience for all users of oauth2, because an application is now stuck in all-or-nothing scenario.  The application would have to ask all of their dependencies to either upgrade or downgrade their dependency on oauth2 to v1 or v2.  AFAICT, there would be no way to write the two conversion functions above, since only one version of a particular repository is allowed at one time.  This is unfortunate, because we as library maintainers would be happy to provide utility code to allow the two to peacefully coexist.

So let's examine the new repository option.  (For most purposes, this is identical to the gopkg.in approach.)  This would allow us as library maintainers to have v1 and v2 coexist, and add utility functions in a new minor version of v1 that can up-convert to v2.  This way, an application can gradually upgrade their v1 and v2 usage without requiring other dependencies to be rewritten to use v2.  However, what the library maintainer loses in this approach is that they have to push a new URL to developers learning about the package. (This is what I've done for go-capnproto vs go-capnproto2, for instance.)  It feels a bit clumsy, since branches ought to be able to solve this.  However, this is an approach that works today, even in the current `go get` model.

I think this is an important use case for the dep tool, and I want to make sure the oauth2 package is setting the right precedent here.  If I were to ask for a feature, it would be for major semantic versions to be treated as wholly separate packages.  Along with type aliases, this would allow library maintainers to assist their users in gradually moving from one major semantic version to another.

Thanks,

-Ross

--

You received this message because you are subscribed to the Google Groups "Go Package Management" group.

To unsubscribe from this group and stop receiving emails from it, send an email to go-package-manag...@googlegroups.com.
To post to this group, send email to go-package...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Go Package Management" group.

To unsubscribe from this group and stop receiving emails from it, send an email to go-package-manag...@googlegroups.com.
To post to this group, send email to go-package...@googlegroups.com.

[Sarah Adams]

+Steve 

On Thu, May 18, 2017 at 4:11 PM, Dave Cheney <da...@cheney.net> wrote:

I'd be keen to see those results. I'm concerned that people who said "yes, I want [to allow] multiple copies of a package in my binary" do not understand what they are asking for, or are applying an interpretation for how they think it​ might work in Go based on prior experience with a language with a different linking model. 

On Fri, 19 May 2017, 08:34 'Sarah Adams' via Go Package Management <go-package-management@googlegroups.com> wrote:

To feed the point that we need multiple versions in a binary, here is an example:

Let's use this oauth2 pkg as an example. The old API is crusty, awful and broken. So as an application author, I want to move my code to use v2 of oauth2. 

Many of my dependencies also use oauth2. Thus, I have to wait for each of my dependencies' authors to get up off their asses, make the change to incorporate oauth2 v2, and tag a new minor release before I can even think about moving my own code to oauth2. 

So logically/mathematically, yes I agree, we should be able to have a single version of any given pkg in a binary.

But practically, OSS maintainers are strapped for time. Some of them are going to be very slow to update. Even if users send PRs for all of these changes (which most won't), maintainers still have to merge. 

This set of slow package maintainers, however small, will hold everyone else back.

Steve Francia recently did a survey. There was a question around this - whether or not folks would like to have multiple versions of their pkg in a binary. Perhaps you've seen it already? The majority of responders wanted multiple versions in a single binary. I can dig it up if you haven't seen it.

So even if you disagree with my example, there is no denying that users want multiple versions of a package in a binary. If dep won't help them do it, they will work *around* dep as it is designed today, not with it. 

If the only way for a packages to co-exist with other versions of itself is to create a new pkg, then this will become the new idiom for a major release in Go.

I think that we can provide a better user experience than this.

- Sarah

On Thu, May 18, 2017 at 2:10 PM, Andrew Gerrand <a...@google.com> wrote:

I think a new repository with a different path is the way to go.

It's effectively a new package.

Andrew

On 19 May 2017 at 05:14, 'Ross Light' via Go Package Management <go-package-management@googlegroups.com> wrote:

Hi Package Management Working Group,

Jaana and I have been contemplating releasing a major version 2 to golang.org/x/oauth2.  It has many mistakes in its public API, to the point where maintaining two separate versions is acceptable to both of us.  We'd like to hear your thoughts on how the proposed dep tool can handle this, or whether its design should be amended to assist in this case.

To give background, the oauth2 package is a quite widely used package throughout the Go ecosystem.  In particular, one of its types is particularly pervasive in exported APIs: oauth2.TokenSource.  It looks like this:

type TokenSource interface {

  Token() (*Token, error)

}

For various reasons, we'd like to change the API in a trivial but backward incompatible way (simplifying a bit):

type TokenSource interface {

  Token(context.Context) (*Token, error)

}

Understandably, we do not want to mandate that every package that depends on oauth2 to make this conversion immediately.  So in the meantime, we'd like to allow binaries to have both types of TokenSource available.  The package does not have global mutable state concerns.  And we've structured the new API such that converting between these types can be written as functions:

func TokenSourceToV2(TokenSource) v2.TokenSource

func TokenSourceFromV2(v2.TokenSource) TokenSource

Now comes our problem: where do we put this new v2?  of which I see three options: 1) a subdirectory of the existing repository 2) a new branch or 3) a new repository

Placing v2 in a subdirectory seems not ideal since the SemVer tags on the repo would also include the v2 directory, so they would be somewhat meaningless.  I'm not going to explore this option in any great depth.

Placing v2 on a new branch and creating a new SemVer tag seems to be the preferred option for the dep tool.  However, IIUC both packages would have the same import path, so only one would be allowed in an application.  This would be an exceedingly frustrating experience for all users of oauth2, because an application is now stuck in all-or-nothing scenario.  The application would have to ask all of their dependencies to either upgrade or downgrade their dependency on oauth2 to v1 or v2.  AFAICT, there would be no way to write the two conversion functions above, since only one version of a particular repository is allowed at one time.  This is unfortunate, because we as library maintainers would be happy to provide utility code to allow the two to peacefully coexist.

So let's examine the new repository option.  (For most purposes, this is identical to the gopkg.in approach.)  This would allow us as library maintainers to have v1 and v2 coexist, and add utility functions in a new minor version of v1 that can up-convert to v2.  This way, an application can gradually upgrade their v1 and v2 usage without requiring other dependencies to be rewritten to use v2.  However, what the library maintainer loses in this approach is that they have to push a new URL to developers learning about the package. (This is what I've done for go-capnproto vs go-capnproto2, for instance.)  It feels a bit clumsy, since branches ought to be able to solve this.  However, this is an approach that works today, even in the current `go get` model.

I think this is an important use case for the dep tool, and I want to make sure the oauth2 package is setting the right precedent here.  If I were to ask for a feature, it would be for major semantic versions to be treated as wholly separate packages.  Along with type aliases, this would allow library maintainers to assist their users in gradually moving from one major semantic version to another.

Thanks,

-Ross

--

You received this message because you are subscribed to the Google Groups "Go Package Management" group.

To unsubscribe from this group and stop receiving emails from it, send an email to go-package-management+unsub...@googlegroups.com.
To post to this group, send email to go-package-management@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Go Package Management" group.

To unsubscribe from this group and stop receiving emails from it, send an email to go-package-management+unsub...@googlegroups.com.
To post to this group, send email to go-package-management@googlegroups.com.

[Sarah Adams]

But Dave, even if you are correct about the survey results, I still think my example holds.

+Steve 

--

To unsubscribe from this group and stop receiving emails from it, send an email to go-package-management+unsubscri...@googlegroups.com.

To post to this group, send email to go-package-management@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Go Package Management" group.

To unsubscribe from this group and stop receiving emails from it, send an email to go-package-management+unsubscri...@googlegroups.com.

[Steve Francia]

The survey was clear and so were the results.  People would rather have things just work. The vast majority preferred including two different versions of a package over having to resolve conflicts.

While they are effectively two different packages as Andrew said, there are significant reasons to not make them separate repos. A short list would include: retaining history, not having to recreate all of the maintainers and permissions, retaining popularity, not supporting the old version indefinitely, wanting people to only adopt the newest one, minimizing version confusion. I'm sure there are many many more.

I'm stuck on a train but I can post the survey results tomorrow.

+Steve 

--

To unsubscribe from this group and stop receiving emails from it, send an email to go-package-manag...@googlegroups.com.
To post to this group, send email to go-package...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Go Package Management" group.

To unsubscribe from this group and stop receiving emails from it, send an email to go-package-manag...@googlegroups.com.
To post to this group, send email to go-package...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Steve Francia  |  TPM Go Lang |  sfra...@google.com  |  Google Inc.

[Bradley Falzon]

Note, you obviously can have multiple packages per repo, although I agree with the ideology of having multiple versions per binary, oauth2 package could also have a oauth2/v2 package (for lack of a better name) in the same repository.

+Steve 

--

To unsubscribe from this group and stop receiving emails from it, send an email to go-package-management+unsub...@googlegroups.com.
To post to this group, send email to go-package-management@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Go Package Management" group.

To unsubscribe from this group and stop receiving emails from it, send an email to go-package-management+unsub...@googlegroups.com.
To post to this group, send email to go-package-management@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Steve Francia  |  TPM Go Lang |  sfra...@google.com  |  Google Inc.

--

You received this message because you are subscribed to the Google Groups "Go Package Management" group.

To unsubscribe from this group and stop receiving emails from it, send an email to go-package-management+unsub...@googlegroups.com.
To post to this group, send email to go-package-management@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Bradley Falzon
br...@teambrad.net

[Sarah Adams]

Yes, perhaps.

But is this the best idiom to force upon our community? One pkg per version?

Maybe? But if so, our tooling should work well with this.

To your point about two versions in the same repo - I believe we're converging on semver?

So if the two versions are two packages in the same repo, you'd have to bump minor semver version tags for both pkg v1 and v2 at the same time. This wouldn't work.

On Thu, May 18, 2017 at 5:12 PM, Bradley Falzon <br...@teambrad.net> wrote:

Note, you obviously can have multiple packages per repo, although I agree with the ideology of having multiple versions per binary, oauth2 package could also have a oauth2/v2 package (for lack of a better name) in the same repository.

+Steve 

--

To unsubscribe from this group and stop receiving emails from it, send an email to go-package-management+unsubscri...@googlegroups.com.

To post to this group, send email to go-package-management@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Go Package Management" group.

To unsubscribe from this group and stop receiving emails from it, send an email to go-package-management+unsubscri...@googlegroups.com.

To post to this group, send email to go-package-management@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Steve Francia  |  TPM Go Lang |  sfra...@google.com  |  Google Inc.

--
You received this message because you are subscribed to the Google Groups "Go Package Management" group.

To unsubscribe from this group and stop receiving emails from it, send an email to go-package-management+unsubscri...@googlegroups.com.

To post to this group, send email to go-package-management@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Bradley Falzon
br...@teambrad.net

[Jaana Burcu Dogan]

On Thu, May 18, 2017 at 4:11 PM, Dave Cheney <da...@cheney.net> wrote:

I'd be keen to see those results. I'm concerned that people who said "yes, I want [to allow] multiple copies of a package in my binary" do not understand what they are asking for, or are applying an interpretation for how they think it​ might work in Go based on prior experience with a language with a different linking model. 

I cannot follow this. It is not technically incorrect to have two different versions in the same binary as soon as they don't interop via the common dependency's symbols. The reason why dep doesn't support multiple versions is totally a different story and a tradeoff rather than being a limitation of Go's linking model.

I can design a Go dependency management tool where each version has their own namespace, and can reject the bad interops. The only argument against this model is that it disables users who are overriding their dependencies' dependencies without having to patch someone else's code. We briefly brainstormed about versions being different namespaces with Russ weeks ago. He might have more some notes.

On Fri, 19 May 2017, 08:34 'Sarah Adams' via Go Package Management <go-package-management@googlegroups.com> wrote:

To feed the point that we need multiple versions in a binary, here is an example:

Let's use this oauth2 pkg as an example. The old API is crusty, awful and broken. So as an application author, I want to move my code to use v2 of oauth2. 

Many of my dependencies also use oauth2. Thus, I have to wait for each of my dependencies' authors to get up off their asses, make the change to incorporate oauth2 v2, and tag a new minor release before I can even think about moving my own code to oauth2. 

So logically/mathematically, yes I agree, we should be able to have a single version of any given pkg in a binary.

But practically, OSS maintainers are strapped for time. Some of them are going to be very slow to update. Even if users send PRs for all of these changes (which most won't), maintainers still have to merge. 

This set of slow package maintainers, however small, will hold everyone else back.

Steve Francia recently did a survey. There was a question around this - whether or not folks would like to have multiple versions of their pkg in a binary. Perhaps you've seen it already? The majority of responders wanted multiple versions in a single binary. I can dig it up if you haven't seen it.

So even if you disagree with my example, there is no denying that users want multiple versions of a package in a binary. If dep won't help them do it, they will work *around* dep as it is designed today, not with it. 

If the only way for a packages to co-exist with other versions of itself is to create a new pkg, then this will become the new idiom for a major release in Go.

I think that we can provide a better user experience than this.

- Sarah

On Thu, May 18, 2017 at 2:10 PM, Andrew Gerrand <a...@google.com> wrote:

I think a new repository with a different path is the way to go.

It's effectively a new package.

Andrew

On 19 May 2017 at 05:14, 'Ross Light' via Go Package Management <go-package-management@googlegroups.com> wrote:

Hi Package Management Working Group,

Jaana and I have been contemplating releasing a major version 2 to golang.org/x/oauth2.  It has many mistakes in its public API, to the point where maintaining two separate versions is acceptable to both of us.  We'd like to hear your thoughts on how the proposed dep tool can handle this, or whether its design should be amended to assist in this case.

To give background, the oauth2 package is a quite widely used package throughout the Go ecosystem.  In particular, one of its types is particularly pervasive in exported APIs: oauth2.TokenSource.  It looks like this:

type TokenSource interface {

  Token() (*Token, error)

}

For various reasons, we'd like to change the API in a trivial but backward incompatible way (simplifying a bit):

type TokenSource interface {

  Token(context.Context) (*Token, error)

}

Understandably, we do not want to mandate that every package that depends on oauth2 to make this conversion immediately.  So in the meantime, we'd like to allow binaries to have both types of TokenSource available.  The package does not have global mutable state concerns.  And we've structured the new API such that converting between these types can be written as functions:

func TokenSourceToV2(TokenSource) v2.TokenSource

func TokenSourceFromV2(v2.TokenSource) TokenSource

Now comes our problem: where do we put this new v2?  of which I see three options: 1) a subdirectory of the existing repository 2) a new branch or 3) a new repository

Placing v2 in a subdirectory seems not ideal since the SemVer tags on the repo would also include the v2 directory, so they would be somewhat meaningless.  I'm not going to explore this option in any great depth.

Placing v2 on a new branch and creating a new SemVer tag seems to be the preferred option for the dep tool.  However, IIUC both packages would have the same import path, so only one would be allowed in an application.  This would be an exceedingly frustrating experience for all users of oauth2, because an application is now stuck in all-or-nothing scenario.  The application would have to ask all of their dependencies to either upgrade or downgrade their dependency on oauth2 to v1 or v2.  AFAICT, there would be no way to write the two conversion functions above, since only one version of a particular repository is allowed at one time.  This is unfortunate, because we as library maintainers would be happy to provide utility code to allow the two to peacefully coexist.

So let's examine the new repository option.  (For most purposes, this is identical to the gopkg.in approach.)  This would allow us as library maintainers to have v1 and v2 coexist, and add utility functions in a new minor version of v1 that can up-convert to v2.  This way, an application can gradually upgrade their v1 and v2 usage without requiring other dependencies to be rewritten to use v2.  However, what the library maintainer loses in this approach is that they have to push a new URL to developers learning about the package. (This is what I've done for go-capnproto vs go-capnproto2, for instance.)  It feels a bit clumsy, since branches ought to be able to solve this.  However, this is an approach that works today, even in the current `go get` model.

I think this is an important use case for the dep tool, and I want to make sure the oauth2 package is setting the right precedent here.  If I were to ask for a feature, it would be for major semantic versions to be treated as wholly separate packages.  Along with type aliases, this would allow library maintainers to assist their users in gradually moving from one major semantic version to another.

Thanks,

-Ross

--

You received this message because you are subscribed to the Google Groups "Go Package Management" group.

To unsubscribe from this group and stop receiving emails from it, send an email to go-package-management+unsub...@googlegroups.com.
To post to this group, send email to go-package-management@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Go Package Management" group.

To unsubscribe from this group and stop receiving emails from it, send an email to go-package-management+unsub...@googlegroups.com.
To post to this group, send email to go-package-management@googlegroups.com.

[Sam Boyer]

hi Ross -

Interesting case! Thanks for sharing it. As dep is about to pass over the milestone into "stable files," having established answers to questions like this is going to be really important. I'd love it if we could assemble some people just to write docs/guides on how the migration of the ecosystem should proceed. Frankly, I think we may all have a bad time if we don't do that - and soon.

In the general case, it should be fine to have a series of semver-conforming tags reflecting each of the different branches, split across a major version: v1.x.x, v2.x.x. If you're willing to maintain both branches, then that presumably entails a branch for each release line. (I won't get into the naming of those, as that's a) a bikeshed and b) irrelevant to how dep will handle the situation). As you point out, though, both versions share the same import path, which the current semantics of vendor make impossible.

Allowing multiple versions of the same package is something the committee has specifically discussed with rsc, and something that looks like it's likely we'll want to support eventually - to quote him, from a go-nuts thread:

- Multiple packages. At scale, I think allowing v1 and v2 of a single package to be linked into the final binary is unavoidable. It's also not possible to express in the "top-level vendor directory" model. Another reason to eliminate the "translate lock file to vendor directory" step.

I opened up https://github.com/golang/dep/issues/507 a couple weeks ago, which would be all we need from dep's metadata to be able to support this situation. However, there's still the limitations imposed by vendor. Addressing that would take a lot more work, including toolchain (linker) changes. I wrote a sketch of a partial solution to this several months ago: https://gist.github.com/sdboyer/def9b138af448f39300cb078b0e94cc3 . Another consideration is the point you made about global state, which Dave Cheney has often raised in these discussions; Steve Francia suggested way back last September that we might simply have a property in (what is now) Gopkg.toml where a project can declare itself stateless/encapsulable.

Of course, none of this is really feasible to pursue prior to dep's absorption into the toolchain.

However, even if all of this were in place, you have this particular requirement that I think may put your approach in a different category altogether:

func TokenSourceToV2(TokenSource) v2.TokenSource

func TokenSourceFromV2(v2.TokenSource) TokenSource

These suggest a bidirectional relationship between your different APIs - one that, in order to exploit it, the old and new forms of the identifier must coexist within the scope of a single file. Even when we get around to allowing multiple versions of the same import path to be used within a single binary, each individual project (and thus certainly each package, and each file) will still be restricted to relying on just one of those versions. That's a necessary consequence of the choice to orient dependencies towards projects (trees of packages), not individual packages. Though, even if we were package-oriented, this likely still wouldn't be possible, as it would ultimately end up necessitating the same import path to appear twice in a file, but then somehow refer to these two different versions. As you note, this is sorta gopkg.in-style (though that really only works with single-package libs).

In short, if it's crucial that you provide this kind of bidirectional relationship, I don't think there's anything dep really could do to help. You'll ultimately have to create a whole new project/repository...which, yes, sucks for all the reasons you gave. (IIRC I gave those same basic reasons to Jaana on twitter a while ago :D)

cheers

s

[Sam Boyer]

Sorry - ultimately you'll have to create a separate repository, OR move them into a new package within the existing project. Point is, this has to be accomplished at the namespace level, not the version level :D

[Jaana Burcu Dogan]

On Thu, May 18, 2017 at 5:42 PM, Sam Boyer <samuel....@gmail.com> wrote:

Sorry - ultimately you'll have to create a separate repository, OR move them into a new package within the existing project. Point is, this has to be accomplished at the namespace level, not the version level :D

Is versioned import paths a broader best practice right now? "Each time you start a repo, add a top-level major version sub directory."

[Andrew Gerrand]

I don't see why it should be in the same repository.

In fact, it seems strictly worse to put the new package in the same repository.

It just creates more potential versioning issues down the road.

If they're separate they should be separate.

[Bradley Falzon]

On 19 May 2017 at 10:43, 'Andrew Gerrand' via Go Package Management <go-package...@googlegroups.com> wrote:

I don't see why it should be in the same repository.

Are these not valid points from Steve Francia:

While they are effectively two different packages as Andrew said, there are significant reasons to not make them separate repos. A short list would include: retaining history, not having to recreate all of the maintainers and permissions, retaining popularity, not supporting the old version indefinitely, wanting people to only adopt the newest one, minimizing version confusion. I'm sure there are many many more.

To add one more, when creating an issue which affects multiple versions (a bug or feature), the issue may need to be duplicated, and the PR/commit may be more difficult to manage as well (duplication).

 

In fact, it seems strictly worse to put the new package in the same repository.

It just creates more potential versioning issues down the road.

I don't see how it would create more versioning issues down the road as it's still a new package, but just happens to be in the same repository. If the concern is having to tie both versions to a single repository meaning you can't update one without updating the other - then I agree, that is a real problem with a single repository. An edge case, but someone may have been depending on some behaviour even if it wasn't a breaking change and may not want to update both versions.

But the common case of day to day management (which repo does this issue belong in, how can I cherry pick these commits), is being optimised in the single repository/mulitple packages case.

 

If they're separate they should be separate.

To unsubscribe from this group and stop receiving emails from it, send an email to go-package-management+unsubscri...@googlegroups.com.

To post to this group, send email to go-package-management@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Go Package Management" group.
To unsubscribe from this group and stop receiving emails from it, send an email to go-package-management+unsub...@googlegroups.com.
To post to this group, send email to go-package-management@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Bradley Falzon
br...@teambrad.net

[Andrew Gerrand]

Apologies for this style of nitpicky reply, but I feel it is necessary to be clear on each of these points.

On 19 May 2017 at 11:36, Bradley Falzon <br...@teambrad.net> wrote:

retaining history

The history is still there in the old repository. Or if you really care, start the new repository by cloning the old one and committing a change that deletes all the old files. But if the new package is really a new package, why does the history matter?

not having to recreate all of the maintainers and permissions, retaining popularity

Neither of these seem like relevant concerns for the oauth2 package. Copying maintainers+permissions for a Go repo is just cloning the empty repo. The new package will become popular inasmuch as it'll be better than the existing one. Plus we control godoc.org, right? We should hide deprecated packages from its search results.

not supporting the old version indefinitely

I don't think the old package should be *supported* indefinitely, but having it *deprecated but available* is a feature, not a bug. The old package works fine for many users and they should be able to continue to use it indefinitely.

wanting people to only adopt the newest one

minimizing version confusion.

I don't see how either of these issues are better or worse, whether or not the new package is in a new repository.

To add one more, when creating an issue which affects multiple versions (a bug or feature), the issue may need to be duplicated, and the PR/commit may be more difficult to manage as well (duplication).

This is IMO a minor burden for the package maintainers, not for the package's users. We should optimize for the users. But also, the old package would be deprecated, right? So no new features or non-critical bug fixes would be applied to it anyway.

In summary, I think if we are going to create a new package with a new API then it should be in a new repository with a new import path. I am happy to field further questions on my perspective on this, if I have been too pithy here.

Cheers,

Andrew

[Dave Cheney]

I'm deeply concerned about enabling this behaviour for the reasons outlined in my April 2015 presentation. http://go-talks.appspot.com/github.com/davecheney/presentations/reproducible-builds.slide#8

+Steve 

--

To unsubscribe from this group and stop receiving emails from it, send an email to go-package-manag...@googlegroups.com.
To post to this group, send email to go-package...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Go Package Management" group.

To unsubscribe from this group and stop receiving emails from it, send an email to go-package-manag...@googlegroups.com.
To post to this group, send email to go-package...@googlegroups.com.

[Sam Boyer]

IMO versioned import paths, at least in the style of gopkg.in, should become a thing of the past.

Again, I think the key is that, once the requirement is explicit logical interaction between symbols from the old and new systems, then it's no longer "versioning" in the same sense that dep is deals with. It's better modeled as two projects that serve a similar purpose, with a well-defined transformation between them.

Also...on further reflection, I realized it is entirely possible to have types from both versions of the lib present in a single file's scope, as it's entirely possible for symbols to entire a file, e.g. via return values, without there being an import statement for those symbols' package. So, in a post-vendor world, it would be possible to facilitate this situation. But the code would be quite abstruse, with breakages that are subtle to trigger but spectacular in effect. I'm sorta loathe to even explore it much further.

[Jaana Burcu Dogan]

On Thu, May 18, 2017 at 7:10 PM, Andrew Gerrand <a...@google.com> wrote:

Apologies for this style of nitpicky reply, but I feel it is necessary to be clear on each of these points.

On 19 May 2017 at 11:36, Bradley Falzon <br...@teambrad.net> wrote:

retaining history

The history is still there in the old repository. Or if you really care, start the new repository by cloning the old one and committing a change that deletes all the old files. But if the new package is really a new package, why does the history matter?

not having to recreate all of the maintainers and permissions, retaining popularity

Neither of these seem like relevant concerns for the oauth2 package. Copying maintainers+permissions for a Go repo is just cloning the empty repo. The new package will become popular inasmuch as it'll be better than the existing one. Plus we control godoc.org, right? We should hide deprecated packages from its search results.

The question was not for the oauth2 case but broad. I maintain a lot of packages that are not as privileged as the oauth2 package.

I worry that a new import path will hurt a package's value/visibility. Why to devalue a package's brand value if we are releasing a new major version? Many widely used libraries from other language ecosystems went through several major rewrites without name changes, that's how top packages persistently keep being the best and making sure that users move on to the new from the legacy. I am a little worried that we will promote an ecosystem with full of zombie packages by actively promoting the existence of two repos.

To be clear, these are just my personal opinions. Maybe dep can reach out to some library maintainers to receive some feedback around these issues.

[Andrew Gerrand]

Okay, I definitely had the wrong frame of reference here. I thought this discussion was specific to oauth2. Mea culpa. I'm sorry for replying without fully absorbing the context.

Assuming that the world starts using dep, here's an approach that would work, based on the approach that rsc has been advocating.

- start with the old API in branch v1 (for example)

- add new API to the existing package while preserving the functionality of the old API (v1.2),

- at the same time, deprecate the old API by adding "Deprecated:" to its godoc comments,

- cut a new major version branch (v2.0) which is identical to 1.2 but without the old API.

That way the world can stay at v1.2 while they gradually update code to use the v2 API.

Projects that only use the v2 API can specify a constraint of "v1.2 OR v2".

Once uses of the old API are gone, people can update their constraints to just specify v2.

You will likely have worse names, though. Is it worth it? Probably.

[Steve Francia]

I went to find the results and realized we've never published them and there current are comments in the Google slides that I'm not sure are ok for public viewing. Historically we only shared them with the Dep WG. I do think we should definitely publish the aggregate results publicly. I'll work on preparing them for that. 

The relevant data is that 81% of users said they would usually or sometimes want "if two of my dependencies have transitive dependency on different versions of the same package and that package can safely co-exist with other versions of itself it should work without any user interaction"

You could say that our users didn't understand the question and that's possible but I also conducted a lot (30+) in person interviews with people from a wide set of backgrounds and roles and this was a very common sentiment across those interviews as well and due to the interactive interview nature I can say will full confidence that the people knew exactly what we were talking about. 

[Dave Cheney]

I'm astonished that after two years of watching Go developers
struggled with the problems created by nested vendor/ directories
allow multiple copies of a package to exist with different names in
the same binary -- this is raison d'etre for developing tools to
flattening the vendor directory.

I feel that either you don't understand the concerns I've been talking
about since April 2015, or you're choosing some form of popularity
driven design philosophy.

Dave

[Peter Bourgon]

> if two of my dependencies have transitive dependency on different
> versions of the same package and that package can safely co-exist
> with other versions of itself

To the best of my knowledge, this last clause is unsatisfiable,
uncomputeable, unknowable, etc., which renders this whole discussion
moot. Is this not the case? If not, can someone please propose how to
support the following?

// package foo, version 1
var Version int = 1

// package foo, version 2
package foo
var Version int = 2

// my user code
import "foo"
println(foo.Version)



On Fri, May 19, 2017 at 4:02 PM, 'Steve Francia' via Go Package

[Sam Boyer]

It is possible - this is what I was saying over on a different subthread.

// will refer to foo v1, the only one we explicitly reference

import "foo"

// foo.Foo v1

var foov1 foo.Foo

// bar depends on foo at v2 and returns a foo.Foo at v2 from its bar.Foo() func

foov2 := bar.Foo()

>>>>>>>> send an email to go-package-management+unsub...@googlegroups.com.

>>>>>>>> To post to this group, send email to
>>>>>>>> go-package...@googlegroups.com.
>>>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>> You received this message because you are subscribed to the Google
>>>>>> Groups "Go Package Management" group.
>>>>>> To unsubscribe from this group and stop receiving emails from it, send

>>>>>> an email to go-package-management+unsub...@googlegroups.com.

>>>>>> To post to this group, send email to
>>>>>> go-package...@googlegroups.com.
>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>
>>>>
>>>
> --
>
> Steve Francia  |  TPM Go Lang |  sfra...@google.com  |  Google Inc.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Go Package Management" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to go-package-management+unsub...@googlegroups.com.

[Sam Boyer]

FWIW, I am aware of these issues, and have no plans to let any sort of dependency duplication into dep unless/until we have demonstrable ways of preventing them from happening.

+Steve 

--

To unsubscribe from this group and stop receiving emails from it, send an email to go-package-management+unsub...@googlegroups.com.

To post to this group, send email to go-package...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Go Package Management" group.

To unsubscribe from this group and stop receiving emails from it, send an email to go-package-management+unsub...@googlegroups.com.

[Ross Light]

It seems simpler to me to provide a mechanism (a la gopkg.in) that allows major semantic version releases to have different import paths, like:

github.com/foo/bar@v1/baz

github.com/foo/bar@v2/baz

That way, you could ensure there's only one version of each major semantic version, and then the two major ones could coexist comfortably at different namespaces.  I see this as complementary to the dep tool, which would focus on the reproducible builds aspect of the problem.

(Credit goes to @rakyll and @neild for the idea.)

To unsubscribe from this group and stop receiving emails from it, send an email to go-package-manag...@googlegroups.com.

To post to this group, send email to go-package...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Go Package Management" group.

To unsubscribe from this group and stop receiving emails from it, send an email to go-package-manag...@googlegroups.com.

[Sam Boyer]

This is not a good idea for the general case. It may be merited for the particular situation you've presented, but I've called attention to the issue that makes your problem both harder, and atypical. Any proposal to solve your problem should be restricted to that subset of cases.

To unsubscribe from this group and stop receiving emails from it, send an email to go-package-management+unsub...@googlegroups.com.

To post to this group, send email to go-package...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Go Package Management" group.

To unsubscribe from this group and stop receiving emails from it, send an email to go-package-management+unsub...@googlegroups.com.

[Ross Light]

Sorry, I must have missed the objection.  What do you see is the subset and what do you see is the problem?  It seems like any major semantic version can (and should) be treated as a separate package, as it breaks compatibility.  Allowing coexistence can allow the older semantic versions to shim to the newer version where there is large compatibility -- especially once type aliases lands.

To unsubscribe from this group and stop receiving emails from it, send an email to go-package-manag...@googlegroups.com.

To post to this group, send email to go-package...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Go Package Management" group.

To unsubscribe from this group and stop receiving emails from it, send an email to go-package-manag...@googlegroups.com.

To post to this group, send email to go-package...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "Go Package Management" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/go-package-management/fp2uBMf6kq4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to go-package-manag...@googlegroups.com.

[Sam Boyer]

Original note is here - https://groups.google.com/d/msg/go-package-management/fp2uBMf6kq4/sX96SZ7DBQAJ

i've gotta run, will try to respond in more detail later, but basically - yes, the subcase is shimming, which is def adjacent to type alias territory.

To unsubscribe from this group and stop receiving emails from it, send an email to go-package-management+unsub...@googlegroups.com.

To post to this group, send email to go-package...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Go Package Management" group.

To unsubscribe from this group and stop receiving emails from it, send an email to go-package-management+unsub...@googlegroups.com.

To post to this group, send email to go-package...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "Go Package Management" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/go-package-management/fp2uBMf6kq4/unsubscribe.

To unsubscribe from this group and all its topics, send an email to go-package-management+unsub...@googlegroups.com.

[Ross Light]

Thanks for the clarification.

You say in that message: "As you note, this is sorta gopkg.in-style (though that really only works with single-package libs)."  That's not true: if you give different branches of the repository different import paths, then the branches can use their designated import path.  gopkg.in definitely works for multi-package libraries.

I would be disappointed as a library maintainer that even if I wanted to put in effort to give my users functions to convert between my different version's types that it would be impossible for me to do with dep without starting a new repository.  I agree with your earlier sentiment, writing up guides on each of these scenarios would help get feedback on pain points from the community.

For oauth2, sounds like the advice I'm getting is to start a new repository, not create a v2 subdirectory, since that would muddle the semantic version tags.  Effectively, this is a new branch (retaining prior history) but contained at a separate physical location.  Is that accurate?

To unsubscribe from this group and stop receiving emails from it, send an email to go-package-manag...@googlegroups.com.

To post to this group, send email to go-package...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Go Package Management" group.

To unsubscribe from this group and stop receiving emails from it, send an email to go-package-manag...@googlegroups.com.

To post to this group, send email to go-package...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "Go Package Management" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/go-package-management/fp2uBMf6kq4/unsubscribe.

To unsubscribe from this group and all its topics, send an email to go-package-manag...@googlegroups.com.

To post to this group, send email to go-package...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "Go Package Management" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/go-package-management/fp2uBMf6kq4/unsubscribe.

To unsubscribe from this group and all its topics, send an email to go-package-manag...@googlegroups.com.

[Nathan Youngman]

Hi Ross,

I've worked with gopkg.in in the past, and while it was better than nothing, the approach wasn't without issues.

In my initial reply, I said "for now, my suggestion would be to fall back on the past approaches." After more thought, I need to take back that advice.

Consider this. The oauth library has the limelight of being a golang.org/x package. The dep tool is being built as the hopeful versioning strategy for the Go community.

If the oauth library needs to make breaking changes, to use anything but dep, doesn't send a very good message. This is a situation where best practices can be established and shown through example to the rest of the community. Whatever you do is likely to be copied by others.

This is going to take some patience. 

It is also going to involve rethinking your initial plans, because as of now, the dep tool and the way it utilizes the vendor folder doesn't support the strategy of multiple versions per binary.

~

The first step is to start tagging the v1.0 releases and suggest users vendor oauth2 in their projects. Give enough warning before breaking things, if indeed that is necessary.

~

Sarah Adams pointed out the issue with slow maintainers not merging pull requests of intermediary packages that may be required for an end-user of oauth to upgrade. That is an annoyance, but as it turns out, it isn't a real problem using dep today.

Firstly, the end-user can either lock into v1 and be patient. There's absolutely nothing wrong with v2 adoption having a slow ramp up.

The more ambitious can fork the intermediaries and use the source attribute in dep to utilize their own fork. Hopefully opening a pull request / CL will result in that fork being a very temporary measure, but if not, it's a solution that works.

~

Personally I don't think you should create a subdirectory or create a new repository. I don't think you should try to provide helpers to convert from one version's type to another. It's unnecessary complexity.

Use versioning to allow for forward momentum in the face of breaking changes, don't try to straddle multiple versions, at least not until such time as that has official support from the tooling -- official support that has been well considered and deemed worth the tradeoffs, if indeed it is.

All the best,

Nathan.

To unsubscribe from this group and stop receiving emails from it, send an email to go-package-management+unsub...@googlegroups.com.

To post to this group, send email to go-package...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Go Package Management" group.

To unsubscribe from this group and stop receiving emails from it, send an email to go-package-management+unsub...@googlegroups.com.

To post to this group, send email to go-package...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Go Package Management" group.

To unsubscribe from this group and stop receiving emails from it, send an email to go-package-management+unsub...@googlegroups.com.
To post to this group, send email to go-package-management@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Nathan Youngman 
Email: he...@nathany.com
Web: https://nathany.com

[Jaana Burcu Dogan]

This flow aligns well with what I was looking for!

Currently, we are still blocked to use this approach given some ecosystem tools (e.g. godoc.org) doesn't support version branches but fixing them are not a big deal.

 

[Ross Light]

So in that flow, we would have a oauth2.TokenSource symbol and a oauth2.TokenSourceV2 symbol?  And then v2 will perpetually have all of its symbols suffixed with V2, even once the unsuffixed ones are gone?

Yeah, that does solve the set of constraints.  It seems somewhat unfortunate, but if that's the direction we want to go, I guess we can be guinea pigs.

--

You received this message because you are subscribed to a topic in the Google Groups "Go Package Management" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/go-package-management/fp2uBMf6kq4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to go-package-manag...@googlegroups.com.

To post to this group, send email to go-package...@googlegroups.com.

[Ross Light]

Hang on.  Actually, you could do the same thing with a v2 subpackage, then have your v2 branch cut out the original package.  Still seems unfortunate, but is more inline with our original thinking.

[Jaana Burcu Dogan]

On Fri, May 19, 2017 at 9:28 AM, Nathan Youngman <he...@nathany.com> wrote:

Hi Ross,

I've worked with gopkg.in in the past, and while it was better than nothing, the approach wasn't without issues.

In my initial reply, I said "for now, my suggestion would be to fall back on the past approaches." After more thought, I need to take back that advice.

Consider this. The oauth library has the limelight of being a golang.org/x package. The dep tool is being built as the hopeful versioning strategy for the Go community.

AFAIK, this is also Ross' thinking. He suggested that the rewrite of oauth2 can dogfood the dep tool and we can discover/document some dep best practices.

 

I would optimally wish that we can use this rewrite as a pilot study for dep and learn from it. So, rather than suggesting pragmatic solutions that fits the current state, I would prefer others to guide us on what the optimal flow should be to rewrite a package if they are already not doing so :)

To unsubscribe from this group and stop receiving emails from it, send an email to go-package-management+unsubscri...@googlegroups.com.

To post to this group, send email to go-package...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Go Package Management" group.

To unsubscribe from this group and stop receiving emails from it, send an email to go-package-management+unsubscri...@googlegroups.com.

To post to this group, send email to go-package...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Go Package Management" group.

To unsubscribe from this group and stop receiving emails from it, send an email to go-package-management+unsubscri...@googlegroups.com.

To post to this group, send email to go-package-management@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Sam Boyer]

Yep, this approach works well because it doesn't *change* any symbols. It only adds, and then at certain infrequent moments, removes them. It's basically the "accretion-only" model Rich Hickey argued for last fall: https://www.youtube.com/watch?v=oyLBGkS5ICk 

As long as a given symbol has only one definition throughout the entire version history of a namespace, things are much easier; later versions can keep earlier versions around in order to facilitate shimming. I'm glad you're amenable to this approach; it's probably the safest policy we could define for how to handle major version transitions.

To unsubscribe from this group and all its topics, send an email to go-package-management+unsub...@googlegroups.com.

[Sam Boyer]

On Friday, May 19, 2017 at 12:06:11 PM UTC-4, Ross Light wrote:

Thanks for the clarification.

You say in that message: "As you note, this is sorta gopkg.in-style (though that really only works with single-package libs)."  That's not true: if you give different branches of the repository different import paths, then the branches can use their designated import path.  gopkg.in definitely works for multi-package libraries.

Yeah, I was eliding details with "really". It works in a basic case, but it does not handle complexity well, and as such is a poor foundation to build on.

Say there's a project - github.com/go-foo/foo; it can be referenced at gopkg.in/foo. In all versions, there is a subpackage, bar, and root package foo imports subpackage bar.

The operant question is how foo references bar for import (with relative imports being a non-starter).

On the branch called v1, then foo could import bar via gopkg.in/foo.v1/bar. This is mostly fine, as long as there's only ever really a branch (which was the world for which gopkg.in was really designed). But if a release is tagged - say v1.0.0 - then *both* the v1 branch and the v1.0.0 tag are valid patterns that can fulfill gopkg.in/foo.v1. 

If I now want to import from a second project, and I import gopkg.in/foo.v1.0, then I get the tag...but if the maintainer didn't change the import paths for just the 1.0.0 release, then I'll end up with two packages from "separate" repositories: gopkg.in/foo.v1.0.0 and gopkg.in/foo.v1/bar, with the latter being from a different version.

Or, let's assume the maintainer did the "right" thing and changed the import paths for the v1.0.0 release - well, let's hope they didn't ever push that revision to the v1 branch, because otherwise I might get that version when importing gopkg.in/foo.v1, and end up with gopkg.in/foo.v1.0.0/bar. If the import direction is from bar -> root, rather than root -> bar, then the particulars of the problem change again, and we descend further into hell :)

One basic issue here is that import paths are not a property of the package, but the filesystem layout. Using import path package comments kinda helps with this (though it doesn't change the fundamental relationship - it just provides validation). Pushing version information - which we treat as a container-level property - down into the code ends up conflating name with version in a way that creates a fractal of failure modes, and a ton of bookkeeping tasks for users.

 

I would be disappointed as a library maintainer that even if I wanted to put in effort to give my users functions to convert between my different version's types that it would be impossible for me to do with dep without starting a new repository.

As would I; my follow-up note indicated that I do think it's possible, though the gymnastics are likely to be considerable. The solution Andrew suggested is far preferable, though something's tugging at the back of my mind with it; some gymnastics may still be necessary.

 

  I agree with your earlier sentiment, writing up guides on each of these scenarios would help get feedback on pain points from the community.

Awesome :D There's really a lot of this, and we need to start moving on it. I also very much like the idea (as you/Jaana put forward elsewhere) of using this as a very public test case.

 

To unsubscribe from this group and stop receiving emails from it, send an email to go-package-management+unsub...@googlegroups.com.

To post to this group, send email to go-package...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Go Package Management" group.

To unsubscribe from this group and stop receiving emails from it, send an email to go-package-management+unsub...@googlegroups.com.

To post to this group, send email to go-package...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "Go Package Management" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/go-package-management/fp2uBMf6kq4/unsubscribe.

To unsubscribe from this group and all its topics, send an email to go-package-management+unsub...@googlegroups.com.

To post to this group, send email to go-package...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "Go Package Management" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/go-package-management/fp2uBMf6kq4/unsubscribe.

To unsubscribe from this group and all its topics, send an email to go-package-management+unsub...@googlegroups.com.

[Ross Light]

There are definitely some gymnastics to Andrew's approach.  It's unclear to me at what point you could actually delete your v1 code from your master branch.  And +1 to Jaana's remarks: the reason we're asking here is that we want to set a good example for how folks can use dep in the future.

The example that you give for gopkg.in is wrong, though.  gopkg.in does not put minor versions into the import path, nor am I proposing that.  There are two entangled problems that folks want versioned dependencies to solve:

1) Code that expects one API surface, but receives another (incompatible) API surface by accident

2) Not being able to reproduce builds

gopkg.in (and this thread) is touching on problem #1, but says nothing about #2.  dep seems to be great at fixing #2 (yay!), but I don't see much support for #1.  The reason I raise this is because even in the approach that Andrew outlined, we're establishing a very loose convention that already introduces "major" version number into the import path.  I think codifying that convention (a branch/major semantic version acts as a separate import path) makes Go align with how developers already use version control tools.

I'm still not exactly sure what to do for x/oauth2.  Seems like creating a subdirectory or a separate repository are still the only options even in post-dep-world, where what I'd love to be able to do is just start a new branch and give it a new import path.

-Ross

To unsubscribe from this group and stop receiving emails from it, send an email to go-package-manag...@googlegroups.com.

To post to this group, send email to go-package...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Go Package Management" group.

To unsubscribe from this group and stop receiving emails from it, send an email to go-package-manag...@googlegroups.com.

To post to this group, send email to go-package...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "Go Package Management" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/go-package-management/fp2uBMf6kq4/unsubscribe.

To unsubscribe from this group and all its topics, send an email to go-package-manag...@googlegroups.com.

To post to this group, send email to go-package...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "Go Package Management" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/go-package-management/fp2uBMf6kq4/unsubscribe.

To unsubscribe from this group and all its topics, send an email to go-package-manag...@googlegroups.com.

To post to this group, send email to go-package...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "Go Package Management" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/go-package-management/fp2uBMf6kq4/unsubscribe.

To unsubscribe from this group and all its topics, send an email to go-package-manag...@googlegroups.com.

[Sam Boyer]

On Fri, May 19, 2017 at 4:54 PM, Ross Light <li...@google.com> wrote:

There are definitely some gymnastics to Andrew's approach.  It's unclear to me at what point you could actually delete your v1 code from your master branch.  And +1 to Jaana's remarks: the reason we're asking here is that we want to set a good example for how folks can use dep in the future.

The example that you give for gopkg.in is wrong, though.  gopkg.in does not put minor versions into the import path, nor am I proposing that. 

Hmm, looks like I did misread the docs there...funny, given that I did an implementation of gopkg.in within dep. i guess that's what i get for trying to venture into new, complex argument territory from the passenger seat of the car.

So yes, i'm wrong about that particular problem. But i'm not wrong about the more fundamental issue, of which that was merely a symptom: placing constraint information in import paths takes a system that is neatly orthogonal, and conflates parts together in level-busting, duplicative ways. In the current design, constraints are declared in, and only in, Gopkg.toml. Versions are declared by, and only by, meta-repository structures (e.g. branches and tags). Duplicating that information in import statements (external/other-project imports ultimately behave like constraints; internal imports ultimately behave like versions) can create subtle synchronization requirements. If these duplicative declarations drift out of sync, how does the system behave? It will happen, because humans are humans, and the possibility space for mismatches is huge. 

Putting version information into import paths breaks through abstractions in the system in a way that introduces new complexity, and new failure modes. It's a giant footgun. I think it's fair to say that our design, to this point, has tried very hard to make choices that prevent such situations from even being possible - minimizing absurdity, gaining safety, at the expense of some flexibility.

Also, some related aspects of the idea were discussed in this forum a year-ish ago - i think a bit here https://groups.google.com/forum/#!topic/go-package-management/WfE5QnKlgZ8 and more here https://groups.google.com/forum/#!topic/go-package-management/PyQte5OXvYs

HOWEVER...all this gopkg.in stuff is mostly a red herring.

 

There are two entangled problems that folks want versioned dependencies to solve:

1) Code that expects one API surface, but receives another (incompatible) API surface by accident

2) Not being able to reproduce builds

gopkg.in (and this thread) is touching on problem #1, but says nothing about #2.  dep seems to be great at fixing #2 (yay!), but I don't see much support for #1.  The reason I raise this is because even in the approach that Andrew outlined, we're establishing a very loose convention that already introduces "major" version number into the import path.  I think codifying that convention (a branch/major semantic version acts as a separate import path) makes Go align with how developers already use version control tools.

OK, at this point, I think this discussion needs to back up.

There are quite a number of entangled problems that we have to solve - more than just these two. But the assertion that dep doesn't deal with #1 is incorrect. To say otherwise suggests to me that you have either not spent much time with dep, and/or that you are confusing the sufficiency of the system as a whole with dep's immediate capability for dealing with your currently-devised solution for this particular use case.

To me, at least, you were signaling this from the OP, by suggesting very design of dep may have a fault. Now, it's of course possible that any one thing we haven't considered might seriously undermine our current design. (I live in constant, albeit gradually subsiding fear of that happening). But opening up a discussion within that frame, and following it on with assertions that dep is essentially not fit for purpose...well, right now, this discussion feels to me like harmfully overeager tugging at a loose thread on a sweater. I'd prefer a narrower focus on solving the specific problem at hand - shims.

Let me try to summarize some relevant but sorta disconnected points, before I check out for the weekend:

1. Shims are a real use case, and it's important we have a plan for how to address them.
2. Shims are not the only use case, and must be considered in the wider context of all the tradeoffs we need to make in the tool.
3. While i do not currently have a fully-formed solution in my mind for shims, I have a latent one based on this observation https://groups.google.com/d/msg/go-package-management/fp2uBMf6kq4/OOKZLqbxBQAJ that my instincts tell me, when coupled with some smart use of interfaces, should get shims at least 80% of the way there.
4. The accretion-only, with occasional deprecation cleanup on major version bumps, is probably the sanest overall versioning model we could advocate.
5. Keeping names and versions as orthogonal properties contributes significantly to system simplicity and regularity, and should probably remain as the foundation on which we build. 

I'm still not exactly sure what to do for x/oauth2.  Seems like creating a subdirectory or a separate repository are still the only options even in post-dep-world, where what I'd love to be able to do is just start a new branch and give it a new import path.

This does seem like the right goal to set, when we go about the business of weighing tradeoffs.

 

-Ross

To unsubscribe from this group and stop receiving emails from it, send an email to go-package-management+unsub...@googlegroups.com.

To post to this group, send email to go-package...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Go Package Management" group.

To unsubscribe from this group and stop receiving emails from it, send an email to go-package-management+unsub...@googlegroups.com.

To post to this group, send email to go-package...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "Go Package Management" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/go-package-management/fp2uBMf6kq4/unsubscribe.

To unsubscribe from this group and all its topics, send an email to go-package-management+unsub...@googlegroups.com.

To post to this group, send email to go-package...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "Go Package Management" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/go-package-management/fp2uBMf6kq4/unsubscribe.

To unsubscribe from this group and all its topics, send an email to go-package-management+unsub...@googlegroups.com.

To post to this group, send email to go-package...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "Go Package Management" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/go-package-management/fp2uBMf6kq4/unsubscribe.

To unsubscribe from this group and all its topics, send an email to go-package-management+unsub...@googlegroups.com.

To post to this group, send email to go-package-management@googlegroups.com.

[Ross Light]

Replies inline.

On Fri, May 19, 2017 at 8:17 PM Sam Boyer <samuel....@gmail.com> wrote:

There are quite a number of entangled problems that we have to solve - more than just these two. But the assertion that dep doesn't deal with #1 is incorrect. To say otherwise suggests to me that you have either not spent much time with dep, and/or that you are confusing the sufficiency of the system as a whole with dep's immediate capability for dealing with your currently-devised solution for this particular use case.

A totally fair assessment.  I have tried using dep for several (small) projects, but since there are few docs/guides on how to use dep from a library maintainer's standpoint, I'm wanting to know more about best practices around usage.  "Use semantic version tags" appears to be the popular advice, but because I don't understand how these tags will be used by importers of the library, it's hard for me to make a judgement call.  I didn't want to file an issue in case I was "holding it wrong". :)

 

To me, at least, you were signaling this from the OP, by suggesting very design of dep may have a fault. Now, it's of course possible that any one thing we haven't considered might seriously undermine our current design. (I live in constant, albeit gradually subsiding fear of that happening). But opening up a discussion within that frame, and following it on with assertions that dep is essentially not fit for purpose...well, right now, this discussion feels to me like harmfully overeager tugging at a loose thread on a sweater. I'd prefer a narrower focus on solving the specific problem at hand - shims.

I apologize that my post came across that way; it was not my intent.  My post was intended as a request for support. But IIUC dep is meant to be a tool we can rapidly iterate on, I wanted to point this out as a use-case/feedback in case further refinement is desirable.

 

Let me try to summarize some relevant but sorta disconnected points, before I check out for the weekend:

1. Shims are a real use case, and it's important we have a plan for how to address them.
2. Shims are not the only use case, and must be considered in the wider context of all the tradeoffs we need to make in the tool.
3. While i do not currently have a fully-formed solution in my mind for shims, I have a latent one based on this observation https://groups.google.com/d/msg/go-package-management/fp2uBMf6kq4/OOKZLqbxBQAJ that my instincts tell me, when coupled with some smart use of interfaces, should get shims at least 80% of the way there.
4. The accretion-only, with occasional deprecation cleanup on major version bumps, is probably the sanest overall versioning model we could advocate.
5. Keeping names and versions as orthogonal properties contributes significantly to system simplicity and regularity, and should probably remain as the foundation on which we build. 

Makes sense. I'm glad this is under consideration!  Any threads/issues I can follow?

 

I'm still not exactly sure what to do for x/oauth2.  Seems like creating a subdirectory or a separate repository are still the only options even in post-dep-world, where what I'd love to be able to do is just start a new branch and give it a new import path.

This does seem like the right goal to set, when we go about the business of weighing tradeoffs.

In hindsight, I should have done more to highlight this point a bit more in my OP.  This is the tradeoff that I'm struggling with now, since we'd like to be able to improve people's experience with oauth2 as soon as we can (since not all of our users are going to be using dep).

-Ross

[Sam Boyer]

hi Ross -

i appreciate you handling my rather cranky response with grace. it was the end of a long day, and i feel like i could have handled that better - i apologize.

i think you (also) make a totally fair assessment about the state of our docs in this area. we really haven't done a decent job of documenting it the expected interaction between systems in the wild, at all. the most real reference to it i've made publicly is pointing at e.g. Dave's & Nathan's posts about needing to tag releases, and handwavily pointing at the work Bradley Falzon's done on an api compatibility checker (and corresponding semver bump suggestor). beyond that, i think i/we/many of us have just been engaging in magical thinking that people will look at the affordances of the system and work out how they're supposed to use it for the management of all their software. that is, of course, absurd!

our goal has been to announce metadata file-level stability sometime in this past/current week, with the idea that that would mean we could start pushing people to use the tool seriously - as in, commit and release code with Gopkg.toml and Gopkg.lock. however, i've had increasingly vertiginous feelings about this as it's become clearer that our lack of docs covering workflows and release practices could result in folks to not only have a difficult time with their own projects, but also to make choices that impact the ecosystem in a way where the confusion and poor practices become self-reinforcing.

this discussion confirms to me that we would be ill-advised to roll even an alpha release without docs, guides, and real-world examples (like, say, oauth2!) for people to follow after. and no, there are no open issues for most/all of these things - at most, we have some general docs-related issues.

unfortunately, i am so swamped with other maintainership responsibilities (also, yknow, dayjob, family) that while i can contribute and guide ad hoc, i just won't have the time to organize and drive the pulling together of such docs. the most context i think i can provide right now is a quick rundown of areas that i think our docs probably ought to cover:

• How to reinterpret longstanding conventional Go package management wisdom in the world of dep (compare to e.g. "git for svn users")
• How to use branches and tags in the post-dep world
• How to roll a release, and how to pick the right version number
• How to accommodate your dependers that aren't using dep
• How to make breaking changes (so, the discussion in this thread)
• Best practices for constraining dependencies - whether they're already using dep or not
• Everyday tool usage (these may need to be delayed until https://github.com/golang/dep/issues/277, as the CLI commands are going to change significantly)

this is by no means exhaustive, but i think it would be a very solid foundation. i'd be perfectly happy to see individual issues posted for each of these - but mostly i'd just be thrilled to see someone step up to this. maybe that can be y'all/oauth2 - if so, fantastic!

cheers

s

On Monday, May 22, 2017 at 11:55:15 AM UTC-4, Ross Light wrote:

Replies inline.

On Fri, May 19, 2017 at 8:17 PM Sam Boyer <samuel....@gmail.com> wrote:

There are quite a number of entangled problems that we have to solve - more than just these two. But the assertion that dep doesn't deal with #1 is incorrect. To say otherwise suggests to me that you have either not spent much time with dep, and/or that you are confusing the sufficiency of the system as a whole with dep's immediate capability for dealing with your currently-devised solution for this particular use case.

A totally fair assessment.  I have tried using dep for several (small) projects, but since there are few docs/guides on how to use dep from a library maintainer's standpoint, I'm wanting to know more about best practices around usage.  "Use semantic version tags" appears to be the popular advice, but because I don't understand how these tags will be used by importers of the library, it's hard for me to make a judgement call.  I didn't want to file an issue in case I was "holding it wrong". :) 

 

To me, at least, you were signaling this from the OP, by suggesting very design of dep may have a fault. Now, it's of course possible that any one thing we haven't considered might seriously undermine our current design. (I live in constant, albeit gradually subsiding fear of that happening). But opening up a discussion within that frame, and following it on with assertions that dep is essentially not fit for purpose...well, right now, this discussion feels to me like harmfully overeager tugging at a loose thread on a sweater. I'd prefer a narrower focus on solving the specific problem at hand - shims.

I apologize that my post came across that way; it was not my intent.  My post was intended as a request for support. But IIUC dep is meant to be a tool we can rapidly iterate on, I wanted to point this out as a use-case/feedback in case further refinement is desirable.

It's OK, i understand now that was your intent. yes, this is something that has been lacking in the discussion, and we do need to bring it in.

[Sarah Adams]

I feel like we can crowdsource these docs.

There are so many people in the ecosystem who are so eager to contribute to Go in *any* way. I feel like this is a perfect way to a. get these docs started/done, b. get more folks toying around with dep, and c. make people feel included and awesome for contributing.

I'd be happy to help organize. 

Contributors could play with dep for each of ^ these scenarios, write docs, and folks on the dep working group could code review.

LMK

[Sam Boyer]

This sounds great to me - exactly my kind of thinking! If we can provide folks a basic framework in which to experiment and provide their feedback, I absolutely expect we'll get a ton of participation. I know there are people hungering for more opportunities to contribute - the pace of PRs is already increasing basically as fast as I can find issues that would be good for new contributors to work on.

Please just lmk what you need from me to make this go. I'll definitely promote it in either this or next week's dep status update.

[David Collier-Brown]

I noticed this rather late, but I went through this once before with David J. Brown, when he was getting Solaris versioning into a sane state.

We reinvented an approach from Multics, and added semi-invisible labels to individual interfaces, so that one could distinguish between old and new versions.  The Linux glibc folkes picked it up for the same reasom, so you could have, for example, the  old memcpy@GLIBC_2.2.5 that does overlapping source and destination, and the new memcpy@GLIBC_2.3 that will blow up if you overlap source and destination.  New programs get 2.3 by default, ones developed back in the 2.2 era get 2.2.5.  The "@GLIBC_2.2.5" isn't visible by default, but is there to allow us to distinguish old from new.

I have a short note at https://leaflessca.wordpress.com/2017/02/12/dll-hell-and-avoiding-an-np-complete-problem/ that has links to David J's work, and a histortical paper from Paul Stachour documenting the Multics approach.

It's an open question if this is (a) elegant, and (b) what Go should do, but it's probably time to look at it once more.

--dave (DRBrow...@Hi-Multics.ARPA) c-b

[Sam Boyer]

I've gotten started with a number of these in the FAQ - https://github.com/golang/dep/pull/656

Do you think you could join #vendor in slack, so that we can coordinate a bit more synchronously about what organizing this effort would look like?

cheers

s

On Monday, May 22, 2017 at 6:00:27 PM UTC-4, Sarah Adams wrote: