Versioning of libraries and applications -- not packages or repositories

[Nathan Youngman]

At this point, I think the community is generally agreed to use some sort of version numbers?

For years I was silently hoping that someone brighter than I would come up with a better solution that focused on the code and APIs rather than using the crude proxy that versions are. Static analysis, Sourcegraph-style smarts for API comparisons, time-based release cycles, gradual refactoring and deprecation mechanisms, rainbows and fluffy bunnies.

But it would seem versions are here to stay.

I still have a lot of questions -- even at the most basic level. So this post isn't about specifying constraints, just specifying the version of things in the first place.

What is the thing that is versioned and where is that version of that thing indicated?

Right now I have a few libraries. Each library has its own repository but may contain several packages. I use git tags with v-prefixed SemVer and GitHub's Releases for release notes. It seems altogether too tedious and unnecessary to version individual packages -- these things are a unit. Or are they?

If and when code is vendored, those tags are no longer visible. What version was it? Having a version in a manifest file that is copied somewhere below that vendor/ folder works better. Maybe tags are at the discretion of the library author and not required? Maybe both are required?

Git tags aren't usable in a monorepo situation. Not just in a corporate setting -- today there are those situations where there is a really useful package buried deep under the docker/docker repo.

So does there need to be a 1:1 relation between repos and libraries or is library a separate concept from repository or package? A repository may contain zero or more libraries and zero or more applications (cmds), and a library may contain one or more packages. This could be a simple matter of where the manifest file is rooted, defining a library out of packages below.

If a manifest declares dependencies for a library or application, what is the use of the application's meta data (description, version #)? Does it become an alternative to -ldflags="-X ..."?

I think I like where this is going. Transplanting libraries from other repos into your vendor/ folder including their manifests. Checkin that manifest and it contains the version # and other metadata in your repo -- alongside the source code.

Nathan.

[Nathan Youngman]

The subject of this thread should be "What are we versioning?" I'm certainly not prescribing this idea -- the subject is just a summary of the idea.

[Dave Cheney]

> What is the thing that is versioned and where is that version of that thing indicated?

In my mind, the thing that is versioned is the import path _prefix_.

The where is up for debate, by my money is on a tag on the repo.

So, the tag v1.2.3 on the repo https://github.com/davecheney/project,
says that import path with the prefix "github.com/davecheney/project"
refers to version 1.2.3.

> --
> You received this message because you are subscribed to the Google Groups
> "Go Package Management" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to go-package-manag...@googlegroups.com.
> To post to this group, send email to go-package...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Sam Boyer]

I'll try to keep my responses to my sense of the general state of the ecosystem and opinion as I've observed it, and keep my editorializing to a minimum...

On Thursday, July 28, 2016 at 12:37:28 AM UTC-4, Nathan Youngman wrote:

At this point, I think the community is generally agreed to use some sort of version numbers?

Pretty much, yes, with most everyone in basic agreement on semver.

 

For years I was silently hoping that someone brighter than I would come up with a better solution that focused on the code and APIs rather than using the crude proxy that versions are. Static analysis, Sourcegraph-style smarts for API comparisons, time-based release cycles, gradual refactoring and deprecation mechanisms, rainbows and fluffy bunnies.

IMO, these things aren't mutually exclusive. gps already includes some static analysis for validation, and type compatibility validation is a not-too-distant (couple months?) TODO.

Also, it's a much longer discussion, but I'll say that, IMO, the fact that version numbers are a crude proxy is both a blessing and a curse.

 

But it would seem versions are here to stay.

I still have a lot of questions -- even at the most basic level. So this post isn't about specifying constraints, just specifying the version of things in the first place.

What is the thing that is versioned and where is that version of that thing indicated?

I don't think there are any serious proposals for anything other than relying on versions derived from the underlying VCS. Most other possibilities would necessitate a central package system, which is something that's been pretty soundly rejected (at least for now).

 

Right now I have a few libraries. Each library has its own repository but may contain several packages. I use git tags with v-prefixed SemVer and GitHub's Releases for release notes. It seems altogether too tedious and unnecessary to version individual packages -- these things are a unit. Or are they?

Per the above, I think most people say "yes, they're a unit." There are some cases where that's awkward (kit repos), and there is some dispute over whether/how possible it should be to use packages from a single repository at different versions.

 

If and when code is vendored, those tags are no longer visible. What version was it? Having a version in a manifest file that is copied somewhere below that vendor/ folder works better.

In general, discussion has been focused around a vendor-spec (or alternately, "lock") file that would contain such information. In general, "manifest" is used to refer to a separate concept - the constraints, not a record of what's in vendor

 

Maybe tags are at the discretion of the library author and not required? Maybe both are required?

Not quite sure what you mean by this part? Whether the lock contains tags, or just revisions?

 

Git tags aren't usable in a monorepo situation. Not just in a corporate setting -- today there are those situations where there is a really useful package buried deep under the docker/docker repo.

So does there need to be a 1:1 relation between repos and libraries or is library a separate concept from repository or package?

this is tricky to answer directly, because it depends on what you're doing - specifying a constraint on a package, vs. repository? trying to use a repo or package as the root of a project? trying to get rid of packages you don't need from vendor/?

- glide is mostly repo-level right now (though it avoids the big wastes that can come in that way), and plugins can help do further pruning

- in this respect, i think godep's basically the same as glide

- govendor is very much package-level focused, and will let you use pkgs from different versions in a given repo

- sorry, can't speak to gb :(

- gps operates on both levels, and has the information necessary to prune unnecessary bits out, and in the future may allow breaking a repo up to use different pkgs at different versions

 

A repository may contain zero or more libraries and zero or more applications (cmds), and a library may contain one or more packages. This could be a simple matter of where the manifest file is rooted, defining a library out of packages below.

i said (jokingly-but-wouldn't-it-be-nice) recently that acknowledging this should be one of the oaths you swear on speaking in #vendor. so, +1 :)

 

If a manifest declares dependencies for a library or application, what is the use of the application's meta data (description, version #)? Does it become an alternative to -ldflags="-X ..."?

complicated question.

* no real discussion has happened in #vendor around the use of the information at build time yet; we're still focused on just getting the information in place in vendor

* in general, i'm quite against holding version numbers in the manifest. (longer discussion that we can have, if you want)

* my view, at least, on the use of dependency metadata is that "every project gets to express version constraints, and the root project gets some extra powers atop of that."

 

I think I like where this is going. Transplanting libraries from other repos into your vendor/ folder including their manifests. Checkin that manifest and it contains the version # and other metadata in your repo -- alongside the source code.

trying to run a package manager out of what's already in vendor/ can (again, IMO here) make things rather tricky. treating vendor/ as volatile and disposable, even if it's checked in, is one good way of carving out one source of complexity in implementation.

 

Nathan.

[roger peppe]

On 28 July 2016 at 05:56, Dave Cheney <da...@cheney.net> wrote:
>> What is the thing that is versioned and where is that version of that thing indicated?
>
> In my mind, the thing that is versioned is the import path _prefix_.
>
> The where is up for debate, by my money is on a tag on the repo.
>
> So, the tag v1.2.3 on the repo https://github.com/davecheney/project,
> says that import path with the prefix "github.com/davecheney/project"
> refers to version 1.2.3.

+1

[Matt Farina]

Here are some quick thoughts...

On Thursday, July 28, 2016 at 12:37:28 AM UTC-4, Nathan Youngman wrote:

At this point, I think the community is generally agreed to use some sort of version numbers?

I think this is right and SemVer is the generally agreed to method for those.

 

For years I was silently hoping that someone brighter than I would come up with a better solution that focused on the code and APIs rather than using the crude proxy that versions are. Static analysis, Sourcegraph-style smarts for API comparisons, time-based release cycles, gradual refactoring and deprecation mechanisms, rainbows and fluffy bunnies.

API comparisons don't tell you what's going on inside an API. Was a feature added? Did something change? Is there a bug? There are some tools that do API analysis today and they are great to have. But, they are no substitute for broadcasting intent.

Imagine you had a parser that found things A, B, and C is some text. A new feature was added to the code that now finds D but the API didn't change. How do you share that form of intent?

Static analysis is quite helpful and we should use it. But, SemVer provides a level of intent and information we can't do with source analysis, yet.

 

But it would seem versions are here to stay.

I still have a lot of questions -- even at the most basic level. So this post isn't about specifying constraints, just specifying the version of things in the first place.

What is the thing that is versioned and where is that version of that thing indicated?

The common methods are tags on repos or metadata in a central system (e.g. crates.io for Rust). Since Go doesn't have a central system the common method is tags. Is there a reason not to use that?

You'll see me refer to common methods. Other languages have done a lot of work on package management and they have a very common style for some things. That means programmers coming from other languages will be familiar with these common points. If many or all the languages solved the problem the same way we have to consider why that is as well. If we want to do something different we should ask what it is that we know they don't or how Go is so different as it does not apply. This is just my philosophy.

 

Right now I have a few libraries. Each library has its own repository but may contain several packages. I use git tags with v-prefixed SemVer and GitHub's Releases for release notes. It seems altogether too tedious and unnecessary to version individual packages -- these things are a unit. Or are they?

Versioning at the package level within a repo has some drawbacks:

• We typically tend to develop on a source tree. What happens if you use two different packages at two different versions in the same repo? Was that ever tested? Will there be unknown bugs?
• Other languages have the concept of separate packages but they use different names. For example, in PHP it's namespace. In these other languages they version at the repo level. What is it that makes Go so special as to be different?
• We want to help users not get into trouble. Sure, power users might want to do some crazy tricks but the default should be for the non-power user. Versioning at the repo level works with CI/CD, is typical for those coming from other languages, and works with existing tools today. What reason is there to do something different?

 

If and when code is vendored, those tags are no longer visible. What version was it? Having a version in a manifest file that is copied somewhere below that vendor/ folder works better. Maybe tags are at the discretion of the library author and not required? Maybe both are required?

Small alteration... the locked to version should be in a lock file. A manifest file can hold intent which may be version ranges. For example, "^1.2.3" (shorthand for ">= 1.2.3, < 2.0.0").

 

Git tags aren't usable in a monorepo situation. Not just in a corporate setting -- today there are those situations where there is a really useful package buried deep under the docker/docker repo.

This is a tough one. In programming at large monorepos are in the 20%. Designing for the 80% means they aren't the target. Yet, in Go there are some fairly major projects that are monorepos and Google uses a monorepo. This puts it more squarely in our view and decision making process.

I can tell you what we do in Glide for something like this.

1. Glide only resolves the import tree. That means just the import tree for any subpackages you import in your code will be resolved. This helps limit fetching all the possible things (though there is a flag for that).
2. There's a plugin (glide-vc) to strip out unused packages from vendor/. Some other package managers do this by default. We don't so you can make the right decision based on the license of the source and your lawyers stance.

Importing monorepos to happen and we try to make it easier.

Matt Butcher, who started Glide, leads the development of Kubernetes/helm. This imports Kubernetes and we've been trying to build something that makes that work as well as it can.

 

So does there need to be a 1:1 relation between repos and libraries or is library a separate concept from repository or package? A repository may contain zero or more libraries and zero or more applications (cmds), and a library may contain one or more packages. This could be a simple matter of where the manifest file is rooted, defining a library out of packages below.

I think you'll find there are repos with just libraries, repos with just applications, and repos with both. We generally need to make both work. You can do both in at least some other languages and we have both today.

 

If a manifest declares dependencies for a library or application, what is the use of the application's meta data (description, version #)? Does it become an alternative to -ldflags="-X ..."?

I think I like where this is going. Transplanting libraries from other repos into your vendor/ folder including their manifests. Checkin that manifest and it contains the version # and other metadata in your repo -- alongside the source code.

The manifests being in the vendor folder with the source is important. It should have supported version ranges for each dependency. Then the dependency manager can work out all the supported ranges and figure out the right versions to have everywhere. This happens in other systems today and is fairly common practice.

 

Nathan.

[zbobe...@gmail.com]

On Wednesday, July 27, 2016 at 10:37:28 PM UTC-6, Nathan Youngman wrote:
> At this point, I think the community is generally agreed to use some sort of version numbers?
>
>
> For years I was silently hoping that someone brighter than I would come up with a better solution that focused on the code and APIs rather than using the crude proxy that versions are. Static analysis, Sourcegraph-style smarts for API comparisons, time-based release cycles, gradual refactoring and deprecation mechanisms, rainbows and fluffy bunnies.

While we can make tools that help rule out _some_ cases of version incompatibility, we can not make tools that rule them all out. Even a minor change, that breaks no ones apis, could still cause some programs to never halt, and thus we have the halting problem.

So we must ask the developers to give us there intention. Breaking.feature.minor/fix has served many well.

> In my mind, the thing that is versioned is the import path _prefix_.

+1 here as well. I went through three iterations of "what are we versioning" when writing my package manager for go and this proved to be both the most robust and simplest to for users to understand.

Anything less invariably left someone messaging me out of confusion. People _get_ prefixs almost naturally. I can explain to any laymen that all names beginning with a are under a in the phone book.

[Nathan Youngman]

Thanks for all the responses.

From what I gather...

We want versions (for better or worse).

The thing being versioned is a repository via VCS tags on the repository, which is simple to understand for VCS users.

Matt Butcher, who started Glide, leads the development of Kubernetes/helm. This imports Kubernetes and we've been trying to build something that makes that work as well as it can.

I don't see how this would work since Kubernetes isn't a library afaik. That is why I was considering one or more libraries in a single repository where the manifest files (one per library) indicates the version rather than VCS tags.

It seems I didn't present that idea clearly as I see several comments suggesting that I was confusing manifest files with lock files containing sha hashes.

An alternative would be to extract a library in its own repo -- likely with a manifest at that repo root and VCS tags to indicate the version of the library. As far as I can tell, vendoring code from any repository that isn't purpose built as a library wouldn't work with this scheme. Which may not be a bad thing -- just a tradeoff. And of course there are tradeoffs to splitting libraries out into their own repos in terms of issue management (on GitHub at least) and atomic changes.

~

Once we form some sort of committee, I like the idea of soliciting feedback on even the most basic assumptions. We are introducing versions for things -- the pros and cons written up. The thing we want to version is VCS repositories. We want to indicate the versions of libraries with VCS tags, etc, etc.

Even if we plow ahead in implementing a cargo-clone -- and I'd be happy enough if we did -- I would like to take the time to know the tradeoffs we're accepting for those choices.

Cheers,
Nathan.

[Dave Cheney]

On Fri, Jul 29, 2016 at 1:37 PM, Nathan Youngman <he...@nathany.com> wrote:
> Thanks for all the responses.
>
> From what I gather...
>
> We want versions (for better or worse).

Why do you say "or worse", do you think there is an alternative that
hasn't been discussed?

Who is we ? I don't say this to be argumentative, I have a concern
that "we" somehow stands for "the entire Go community", and if it does
not, then who is we?

[Nathan Youngman]

I meant "we" as the general direction of the people talking on this one thread --- before a committee is even formed. Once there is a committee, and once they solicit feedback from "the entire Go community" then "we" will mean that.

I say "for better or worse" because everything comes with tradeoffs.

Chances are everything has been discussed a million times in the past 5 years. Yet here we are. :-(

--

Nathan Youngman 
Email: he...@nathany.com
Web: https://nathany.com

[Dave Cheney]

It sounds like you are disappointed that versioning is a fait
accompli. From my POV it is the only sensible choice, to the extent
that there is no other choice. What would you like to see instead?

[John Souvestre]

> What is the thing that is versioned ...

 

My first reaction is per package.  I understand that there are some advantages to per repo, but I think that there are also some disadvantages.  What if the same package is used in repos for multiple projects?  Or, if I want to review the changes between versions of a single package?

 

> ... and where is that version of that thing indicated?

 

> I don't think there are any serious proposals for anything other

 > than relying on versions derived from the underlying VCS.

 

A VCS comment is the authoritative source for the version number of code sitting outside of it?  There's no way to see the version by looking at code I'm editing?  This seems like a priority inversion to me.  Should the VCS be issuing the version number or just recording it?

 

How about including a VERSION const string in the code.  This way the version number and the code are less likely to get out of sync.  Tools can find it easily.  Perhaps it avoids the mondorepo problem?  And it can be checked/used by runtime code.

 

John

    John Souvestre - New Orleans LA

 

[Eric Johnson]

Hi John,

On 7/29/16 5:13 AM, John Souvestre wrote:
>
> How about including a VERSION const string in the code. This way the
> version number and the code are less likely to get out of sync. Tools
> can find it easily. Perhaps it avoids the mondorepo problem? And it
> can be checked/used by runtime code.
>

Perhaps putting the VERSION in the code is desirable in some cases. As I
understand it, the Go linker already has a way to (re)set a global value
during link. So you can achieve this end.

However, I don't think this is something you want to /require/ over
time. At my company both release and QA teams like it when, for example,
a "release candidate" can be promoted to a final build without changes
to the binary. At least, whenever possible.

Eric.

[Matt Farina]

Nathan, Kubernetes is a great case to discuss because it's one of the more difficult to work with. It's large, has a large dependency tree, and is difficult to work with. If we can solve for a complicated case like Kubernetes we can solve for most cases. So, I like using that example.

Kubernetes is both an application and can be imported as a library. Some projects, like Helm, do that today.

To be honest, if some parts were pulled out into separate libraries, versioned, and imported by others it would make those who are not Kubernetes lives easier. The internals of Kubernetes sometimes change and that can create issues for developers importing parts. I've heard plenty of complaints about this. Kubernetes may gain from a change like that.

But, it doesn't have to. We should be able to solve for this complicated and large case. That's one of my goals with Glide.

[Sam Boyer]

Putting a VERSION number in the code itself means that you've stuck temporal information inside the spatial dimension. That is, is that VERSION applicable only to the first commit in which it appears? The last? All of them? Only the first one where some other criteria (e.g., tests pass, code compiles) passes? Who's making that decision?

Having that information directly in the code itself can be very useful in a paradigm where version control isn't involved, or if a central registry can do work on it. For now, though, our domain is mostly circumscribed by version control, which means such information is basically a poor man's branch.

(Those with knowledge of hg should rightfully infer from this that hg's tagging model makes me slightly twitchy :P)

[Jessica Frazelle]

FWIW I opened a PR to switch kubernetes to govendor
(https://github.com/kubernetes/kubernetes/pull/29625#issuecomment-235644472)
and it was pretty simple to switch but obviously this is not the ideal
solution but it is good to keep in mind the tools that work well
today. That being said I don't know if it's worth merging since we are
in the midst of working through a common solution today. But yes, I
agree larger projects have a pretty different vendoring problem. I
still sadly love the pure bash script we use in the docker repo ;)

> --
> You received this message because you are subscribed to the Google Groups
> "Go Package Management" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to go-package-manag...@googlegroups.com.
> To post to this group, send email to go-package...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.



--

Jessie Frazelle
4096R / D4C4 DD60 0D66 F65A 8EFC 511E 18F3 685C 0022 BFF3
pgp.mit.edu

[Matt Farina]

Jessie, I was talking with Tim Hockin a little bit ago about moving k8s to Glide. With a couple more changes we'll have addressed all his issues and made lifecycle management of the deps far easier. The lifecycle mgmt part isn't such a small deal for that codebase from what I've learned. I'd love to know more though.

> email to go-package-management+unsub...@googlegroups.com.

[Jessica Frazelle]

Oh yes I reviewed that proposal before making the counter PR ;) I have
nothing against glide per say I more just wanted to get something that
is easier than Godep, I've tried just about every vendoring tool, and
for some reason that is the one I have never gotten to work.

>> > email to go-package-manag...@googlegroups.com.

>> > To post to this group, send email to go-package...@googlegroups.com.
>> > For more options, visit https://groups.google.com/d/optout.
>>
>>
>>
>> --
>>
>>
>> Jessie Frazelle
>> 4096R / D4C4 DD60 0D66 F65A 8EFC 511E 18F3 685C 0022 BFF3
>> pgp.mit.edu
>

> --
> You received this message because you are subscribed to the Google Groups
> "Go Package Management" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to go-package-manag...@googlegroups.com.

[Sam Boyer]

yeah, there were a couple blockers Tim had. (at this point, i believe,
all will be resolved once we get gps merged in to glide). doesn't
necessarily mean it would've resolved the issues you ultimately had, of
course :)

[John Souvestre]

Hi Eric.

> However, I don't think this is something you want to
> /require/ over time. At my company both release and QA
> teams like it when, for example, a "release candidate"
> can be promoted to a final build without changes to the
> binary. At least, whenever possible.

I can certainly see the advantage to this. I'm thinking that it would only apply to the project's version, not to the versions of the individual packages which it consists of. So I don't see this as being a problem.

Out of curiosity, if you want the project to be able to display its version to the user, how do you handle that? Read it from an external file?

John

[John Souvestre]

Hi Sam.

 

Ø  Putting a VERSION number in the code itself means that you've stuck temporal information inside the spatial dimension.

 

I’m not sure I understand your point.  The VERSION number is no more temporal than the code itself.  They both change at the same time.

 

Ø  That is, is that VERSION applicable only to the first commit in which it appears? The last? All of them?

 

All commits for which the code is unchanged.

 

Ø  Who's making that decision?

 

The programmer making the changes to the code.  However, a tool could ensure that when it detects code changes that the VERSION is also changed.

 

Ø  Having that information directly in the code itself can be very useful in a paradigm where version control isn't involved, or if a central registry can do work on it. For now, though, our domain is mostly circumscribed by version control, which means such information is basically a poor man's branch.

 

Yes.  It also means your PM is less dependent on what PM the author of an import used, since the version number is self-contained.

 

Does doing it this way interfere with the version control method?  I was expecting that a PM getting the version number this way would use that info as it does now, including for VCS tags.

 

John

    John Souvestre - New Orleans LA

 

[Sam Boyer]

I may have missed some context - if so, sorry. It is, IMO, not crazy to
have something like this, so long as the relationship of versions
expressed this way, versus versions expressed through VCS, is
well-defined. (defining it well is something that i'd put as an
intermediate-ish term goal)

That said, it would be a whole new dimension of problems to
intentionally create an expressive capacity that allows "reasonable
tools to disagree" about what version string corresponds to which
literal set of files.

Bottom line - if a version number cannot be reasonably relied on to
correspond to a snapshot of a codebase in time, then it loses a lot of
its utility. We'd end up having to just recreate that information in
some other way, which amounts to writing our own pseudo-VCS.

It feels like this is veering back into that complexity fractal I
cautioned against at the end of last week.

On 7/29/16 15:03, John Souvestre wrote:
> Hi Sam.
>
>
>
> Ø Putting a VERSION number in the code itself means that you've stuck
> temporal information inside the spatial dimension.
>
>
>
> I’m not sure I understand your point. The VERSION number is no more
> temporal than the code itself. They both change at the same time.
>
>
>
> Ø That is, is that VERSION applicable only to the first commit in which
> it appears? The last? All of them?
>
>
>
> All commits for which the code is unchanged.
>
>
>
> Ø Who's making that decision?
>
>
>
> The programmer making the changes to the code. However, a tool could
> ensure that when it detects code changes that the VERSION is also changed.
>
>
>
> Ø Having that information directly in the code itself can be very
> useful in a paradigm where version control isn't involved, or if a
> central registry can do work on it. For now, though, our domain is
> mostly circumscribed by version control, which means such information is
> basically a poor man's branch.
>
>
>
> Yes. It also means your PM is less dependent on what PM the author of
> an import used, since the version number is self-contained.
>
>
>
> Does doing it this way interfere with the version control method? I was
> expecting that a PM getting the version number this way would use that
> info as it does now, including for VCS tags.
>
>
>
> John
>
> John Souvestre - New Orleans LA
>
>
>

> *From:*go-package...@googlegroups.com
> [mailto:go-package...@googlegroups.com] *On Behalf Of *Sam Boyer
> *Sent:* 2016 July 29, Fri 12:00
> *To:* Go Package Management
> *Subject:* Re: [go-pm] Re: Versioning of libraries and applications --

> not packages or repositories
>
>
>
> Putting a VERSION number in the code itself means that you've stuck
> temporal information inside the spatial dimension. That is, is that
> VERSION applicable only to the first commit in which it appears? The
> last? All of them? Only the first one where some other criteria (e.g.,
> tests pass, code compiles) passes? Who's making that decision?
>
>
>
> Having that information directly in the code itself can be very useful
> in a paradigm where version control isn't involved, or if a central
> registry can do work on it. For now, though, our domain is mostly
> circumscribed by version control, which means such information is
> basically a poor man's branch.
>
>
>
> (Those with knowledge of hg should rightfully infer from this that hg's
> tagging model makes me slightly twitchy :P)
>
>
>

> --
> You received this message because you are subscribed to a topic in the

> Google Groups "Go Package Management" group.

> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/go-package-management/qDZax8mSUGM/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> go-package-manag...@googlegroups.com
> <mailto:go-package-manag...@googlegroups.com>.

> To post to this group, send email to
> go-package...@googlegroups.com

> <mailto:go-package...@googlegroups.com>.

[John Souvestre]

Hi Sam.

> It is, IMO, not crazy to have something like this, so long as the relationship of versions expressed this way, versus versions expressed through VCS, is well-defined.

The version numbers would be the same in both cases.

> That said, it would be a whole new dimension of problems to intentionally create an expressive capacity that allows "reasonable tools to disagree" about what version string corresponds to which literal set of files.

The version number would apply to the package (or repo) it's in (only one per package/repo!).

> Bottom line - if a version number cannot be reasonably relied on to correspond to a snapshot of a codebase in time, then it loses a lot of its utility.

If unchanged, doesn't a given package/repo's unchanged version number (and the unchanged code) appear in snapshots taken at different times? Isn't the goal that when (and only when) the code changes that the version number should change too?

btw - Although I don't see them often in Go, I'm also used to seeing a History file listing the versions with descriptions of changes of interest to the user of the package. It's unrelated to this discussion, but it something that I often wonder about.

> It feels like this is veering back into that complexity fractal I cautioned against at the end of last week.

Where does a package's version number come from now?

[Eric Johnson]

Hi John,

On 7/29/16 11:37 AM, John Souvestre wrote:
> Out of curiosity, if you want the project to be able to display its version to the user, how do you handle that? Read it from an external file?

In some ideal future, an executable will be composed of multiple
libraries, all versioned. Yes, you may want the final executable to have
a version number baked in. Everything else, though, likely not. And even
in the case of the executable, maybe it is undesirable to bake in the
version.

Which is my entire point - it can't be mandated without breaking some
use cases.

Eric.

[Nathan Youngman]

The golang.org/x/ libraries are another interesting case to look at.

This repository holds supplementary Go networking libraries.

 - https://github.com/golang/net

With VCS tagging for versioning they would be treated as a single unit, which may be perfectly fine. There really shouldn't be a need to use a version of net/html that is a different commit from net/http2. Plus vendor tools already can do smart pruning to pull in just the necessary packages from a larger repo like this.

Alternatively, repositories like this could be spit up into smaller repositories and then tagged with VCS versions.

Either way, that keeps things simple vs. specifying versions in some other (more flexible) way. Simplicity is valuable -- and one of the attractions to Go for myself and others.

Nathan.

[Chris Hines]

The golang.org/x/sys repo would be annoying to deal with as far as SemVer tagging goes. It includes code specific to mulitple OSes and the backwards compatibility of the code is somewhat dependent on changes in the various OS APIs the packages wrap.

[Sam Whited]

On Fri, Aug 5, 2016 at 9:42 AM, Chris Hines <chris....@gmail.com> wrote:
> The golang.org/x/sys repo would be annoying to deal with as far as SemVer
> tagging goes. It includes code specific to mulitple OSes and the backwards
> compatibility of the code is somewhat dependent on changes in the various OS
> APIs the packages wrap.

That's the same as any other library, no? SemVer merely tells you that
an API changed, not necessarily an that the API is one that you use.
If a BSD specific syscall changes and breaks the API, it's still a
major version change even if it doesn't affect Linux users just as if
we change an API in text/unicode it would still require a version bump
even if it were a little used feature that would only break something
for 1 or two people.

(Although I sort of know what you mean, it does feel different somehow
if it's only breaking or changing an API used by all projects on a
specific OS as opposed to used by a specific project across all OSs,
but I'm not sure that it actually is)

Similarly, it doesn't matter *why* the API changed (because we changed
it, or because an upstream change forced us to change it). We just
have to tag every time this happens, and users will decide if they
need to update or not.

—Sam



--
Sam Whited
pub 4096R/54083AE104EA7AD3

[Ulrich Kunitz]

I'm not comfortable with speaking of a version of an import path prefix for two reasons:

1) Not all import path prefixes refer to repositories.

2) Import path prefixes will remain constant for almost all versions. (It might even be required that a prefix for a repo should never change.)

So I suggest to speak of full versions (major.minor.patch) being names for revisions of a repo. A repo is uniquely identified by an import path prefix and contains revisions of one or multiple Go packages.

The challenge is now that the Go language knows only dependencies between Go packages, but we have to handle them on repo level. While dependencies cannot be cyclic between Go packages, they might be between repos. Repo 1 might have package A dependent on B in repo 2 and package D of repo 2 dependent on C in repo 1 at the same time. That might be outlawed, but in the go-get model it is absolutely possible.

[Dave Cheney]

On Sat, Aug 6, 2016 at 8:05 AM, Ulrich Kunitz <uli.k...@gmail.com> wrote:
> I'm not comfortable with speaking of a version of an import path prefix for
> two reasons:
>
> 1) Not all import path prefixes refer to repositories.
> 2) Import path prefixes will remain constant for almost all versions. (It
> might even be required that a prefix for a repo should never change.)
>
> So I suggest to speak of full versions (major.minor.patch) being names for
> revisions of a repo. A repo is uniquely identified by an import path prefix
> and contains revisions of one or multiple Go packages.

Yes. Thank you, this is what I meant but your definition is more accurate.

>
> The challenge is now that the Go language knows only dependencies between Go
> packages, but we have to handle them on repo level. While dependencies
> cannot be cyclic between Go packages, they might be between repos. Repo 1
> might have package A dependent on B in repo 2 and package D of repo 2
> dependent on C in repo 1 at the same time. That might be outlawed, but in
> the go-get model it is absolutely possible.

Indeed, there's already one of these metastasising in the
golang.org/x/ sub repos today.

>
> --
> You received this message because you are subscribed to the Google Groups
> "Go Package Management" group.

> To unsubscribe from this group and stop receiving emails from it, send an
> email to go-package-manag...@googlegroups.com.

> To post to this group, send email to go-package...@googlegroups.com.

[atd...@gmail.com]

On Saturday, August 6, 2016 at 12:05:50 AM UTC+2, Ulrich Kunitz wrote:

The challenge is now that the Go language knows only dependencies between Go packages, but we have to handle them on repo level. While dependencies cannot be cyclic between Go packages, they might be between repos. Repo 1 might have package A dependent on B in repo 2 and package D of repo 2 dependent on C in repo 1 at the same time. That might be outlawed, but in the go-get model it is absolutely possible.

What is the issue with that? 

[Matt Farina]

As long as the needed versions of repo1 and repo2 work for each others supported version ranges this works. Layering in versions only causes a problem when repo1 needs a version of repo2 that doesn't support the version of repo1. In practice this would break today due to API differences anyway. 

[atd...@gmail.com]

Thanks. I'm still not quite sure I understand everything though.

What is the meaning of repo here? A synonym of project? Or something that exists in a different $GOPATH?

What do you mean by version? API-breaking? If the packages are stored in a different location, they will be considered different anyway at the language level, nowadays. API-breaking or not.

[Edward Muller]

On Wed, Aug 10, 2016 at 8:48 AM <atd...@gmail.com> wrote:

Thanks. I'm still not quite sure I understand everything though.

What is the meaning of repo here? A synonym of project? Or something that exists in a different $GOPATH?

github.com/foo/bar is the repo. It's also a package and the base for all other packages in that namespace. They would all be versioned together as an atomic unit. So changing github.com/foo/bar/baz would necessitate a bump in version for every package with a prefix of github.com/foo/bar. At least if I'm not misunderstanding anything. 

 

What do you mean by version? API-breaking? If the packages are stored in a different location, they will be considered different anyway at the language level, nowadays. API-breaking or not.

A version would be a semver tag applied to a commit (or equivalent for other VCS).

 

On Wednesday, August 10, 2016 at 3:28:44 PM UTC+2, Matt Farina wrote:

On Saturday, August 6, 2016 at 3:51:44 AM UTC-4, atd...@gmail.com wrote:

On Saturday, August 6, 2016 at 12:05:50 AM UTC+2, Ulrich Kunitz wrote:

The challenge is now that the Go language knows only dependencies between Go packages, but we have to handle them on repo level. While dependencies cannot be cyclic between Go packages, they might be between repos. Repo 1 might have package A dependent on B in repo 2 and package D of repo 2 dependent on C in repo 1 at the same time. That might be outlawed, but in the go-get model it is absolutely possible.

What is the issue with that? 

As long as the needed versions of repo1 and repo2 work for each others supported version ranges this works. Layering in versions only causes a problem when repo1 needs a version of repo2 that doesn't support the version of repo1. In practice this would break today due to API differences anyway. 

--

[atd...@gmail.com]

Thank you. My main issue with this is that it is particular to a given dvcs (github here).

It's not clear what it corresponds to in Go's vernacular, if it does even correspond to anything.

On Wednesday, August 10, 2016 at 6:49:12 PM UTC+2, Edward Muller wrote:

On Wed, Aug 10, 2016 at 8:48 AM <atd...@gmail.com> wrote:

Thanks. I'm still not quite sure I understand everything though.

What is the meaning of repo here? A synonym of project? Or something that exists in a different $GOPATH?

github.com/foo/bar is the repo. It's also a package and the base for all other packages in that namespace. They would all be versioned together as an atomic unit. So changing github.com/foo/bar/baz would necessitate a bump in version for every package with a prefix of github.com/foo/bar. At least if I'm not misunderstanding anything. 

For instance here, you may have a git subtree. But Go does not care about the intricacies of Github's versioning (nor should it).

Especially since, one can have development branches, release branches etc...

 

 

What do you mean by version? API-breaking? If the packages are stored in a different location, they will be considered different anyway at the language level, nowadays. API-breaking or not.

A version would be a semver tag applied to a commit (or equivalent for other VCS).

That is not specific enough because you never know where HEAD is pointing at. The dev branch or the release branch?

Go does not even care about this (nor should it).

What is a version in Go is different than what a version is in git. The constraints entailed by each are not even necessarily the same either.

I hope that we can start by defining exactly the semantics without relying on a given dvcs.

 

 

On Wednesday, August 10, 2016 at 3:28:44 PM UTC+2, Matt Farina wrote:

On Saturday, August 6, 2016 at 3:51:44 AM UTC-4, atd...@gmail.com wrote:

On Saturday, August 6, 2016 at 12:05:50 AM UTC+2, Ulrich Kunitz wrote:

The challenge is now that the Go language knows only dependencies between Go packages, but we have to handle them on repo level. While dependencies cannot be cyclic between Go packages, they might be between repos. Repo 1 might have package A dependent on B in repo 2 and package D of repo 2 dependent on C in repo 1 at the same time. That might be outlawed, but in the go-get model it is absolutely possible.

What is the issue with that? 

As long as the needed versions of repo1 and repo2 work for each others supported version ranges this works. Layering in versions only causes a problem when repo1 needs a version of repo2 that doesn't support the version of repo1. In practice this would break today due to API differences anyway. 

--
You received this message because you are subscribed to the Google Groups "Go Package Management" group.

To unsubscribe from this group and stop receiving emails from it, send an email to go-package-management+unsub...@googlegroups.com.

[Eric Johnson]

On 8/10/16 10:31 AM, atd...@gmail.com wrote:

>
> What is a version in Go is different than what a version is in git.
> The constraints entailed by each are not even necessarily the same either.
> I hope that we can start by defining exactly the semantics without
> relying on a given dvcs.

On this point, I want to give a +1. My company still uses Subversion.
Subversion repositories generally host many projects, all versioned
separately.

I also anticipate scenarios (and have already seen them on GitHub) where
a bigger project has several utility libraries (Go packages) that end up
being more broadly useful than the containing project. Using Git, those
sub-projects can be tagged separately, just as an example. If a
dependency resolution mechanism forces those packages to spin off into a
separate repository, that seems like we're using tooling to force a
particular source code layout and DVCS usage. Maybe that is a good idea?
(After all, Go is opinionated about things, and mostly that makes the
language better.) However, maybe that really isn't a good idea?

Eric.

[Konstantin Shaposhnikov]

What do you mean by version? API-breaking? If the packages are stored in a different location, they will be considered different anyway at the language level, nowadays. API-breaking or not.

A version would be a semver tag applied to a commit (or equivalent for other VCS).

That is not specific enough because you never know where HEAD is pointing at. The dev branch or the release branch?

Go does not even care about this (nor should it).

Why is it important to know where HEAD is pointing at? A tag is esentially a pointer for a particular state of the source code (revision), the branch is implicitly included.

What is a version in Go is different than what a version is in git. The constraints entailed by each are not even necessarily the same either.

I hope that we can start by defining exactly the semantics without relying on a given dvcs.

Without the central repo the only place to store versions is VCS. Tags are universally supported by all VCSes so it seems logical to use them to store versions.

> 
> What is a version in Go is different than what a version is in git. 
> The constraints entailed by each are not even necessarily the same either. 
> I hope that we can start by defining exactly the semantics without 
> relying on a given dvcs. 

On this point, I want to give a +1. My company still uses Subversion. 
Subversion repositories generally host many projects, all versioned 
separately. 

I also anticipate scenarios (and have already seen them on GitHub) where 
a bigger project has several utility libraries (Go packages) that end up 
being more broadly useful than the containing project. Using Git, those 
sub-projects can be tagged separately, just as an example. If a 
dependency resolution mechanism forces those packages to spin off into a 
separate repository, that seems like we're using tooling to force a 
particular source code layout and DVCS usage. Maybe that is a good idea? 
(After all, Go is opinionated about things, and mostly that makes the 
language better.) However, maybe that really isn't a good idea? 

+1. One versionable artefact per repo feels like artifical limitation that is too Git-specific.

[atd...@gmail.com]

On Wednesday, August 10, 2016 at 9:03:08 PM UTC+2, Konstantin Shaposhnikov wrote:

What do you mean by version? API-breaking? If the packages are stored in a different location, they will be considered different anyway at the language level, nowadays. API-breaking or not.

A version would be a semver tag applied to a commit (or equivalent for other VCS).

That is not specific enough because you never know where HEAD is pointing at. The dev branch or the release branch?

Go does not even care about this (nor should it).

Why is it important to know where HEAD is pointing at? A tag is esentially a pointer for a particular state of the source code (revision), the branch is implicitly included.

Because of backward-compatibility concerns. A revision is not a version. That's exactly the reason why I asked what Matt meant by "version". There is a big conflation between git terminology and go semantics. Even in pure git terms, some people will use the term "version" for what is a "revision".

 

What is a version in Go is different than what a version is in git. The constraints entailed by each are not even necessarily the same either.

I hope that we can start by defining exactly the semantics without relying on a given dvcs.

Without the central repo the only place to store versions is VCS. Tags are universally supported by all VCSes so it seems logical to use them to store versions.

Different VCS have different tagging schemes with different semantics. Even worse is that there are not necessarily always reliable (how do you keep someone from using `git reset` ?)

The point is to use the hosting services as pure source hosting without relying on them as the wearers of the release information. That will also allow for mirror hosting solutions that do not have tagging info. Or to switch easily between svn, github, whichever comes next... 

A file per package could testify that the given package is effectively a stable release and hold metadata that guarantees the system resilience. 

[Konstantin Shaposhnikov]

On 10 August 2016 at 20:34, <atd...@gmail.com> wrote:
>
>
> On Wednesday, August 10, 2016 at 9:03:08 PM UTC+2, Konstantin Shaposhnikov
> wrote:
>>>>>
>>>>> What do you mean by version? API-breaking? If the packages are stored
>>>>> in a different location, they will be considered different anyway at the
>>>>> language level, nowadays. API-breaking or not.
>>>>
>>>>
>>>> A version would be a semver tag applied to a commit (or equivalent for
>>>> other VCS).
>>>
>>>
>>> That is not specific enough because you never know where HEAD is pointing
>>> at. The dev branch or the release branch?
>>> Go does not even care about this (nor should it).
>>
>>
>> Why is it important to know where HEAD is pointing at? A tag is esentially
>> a pointer for a particular state of the source code (revision), the branch
>> is implicitly included.
>
>
> Because of backward-compatibility concerns. A revision is not a version.
> That's exactly the reason why I asked what Matt meant by "version". There is
> a big conflation between git terminology and go semantics. Even in pure git
> terms, some people will use the term "version" for what is a "revision".

A package manager needs a package version to reproducibly retreive the
source code of the package. So in the most general sense a version is
anything that allows a package manager to do it. It could be a
revision, a symbolic name that corresponds to a tag or even a branch
name (in case if I am happy to always use master for example).

>>> What is a version in Go is different than what a version is in git. The
>>> constraints entailed by each are not even necessarily the same either.
>>> I hope that we can start by defining exactly the semantics without
>>> relying on a given dvcs.
>>
>>
>> Without the central repo the only place to store versions is VCS. Tags are
>> universally supported by all VCSes so it seems logical to use them to store
>> versions.
>
>
> Different VCS have different tagging schemes with different semantics. Even
> worse is that there are not necessarily always reliable (how do you keep
> someone from using `git reset` ?)

This is real problem. Unfortunately in almost all VCSes tags are
mutable. I do not know how to solve this.

> The point is to use the hosting services as pure source hosting without
> relying on them as the wearers of the release information. That will also
> allow for mirror hosting solutions that do not have tagging info. Or to
> switch easily between svn, github, whichever comes next...
>
> A file per package could testify that the given package is effectively a
> stable release and hold metadata that guarantees the system resilience.

Are you suggesting to have a file that effectively maps a package
version to a VCS revision? E.g.

v1.0.1 = ea809983..
v1.0.2 = 98haeg79...
etc

[atd...@gmail.com]

No, not really. this coupling would be to unreliable.

Using or not using a vcs should not impact the release of your source code. These concerns should remain separated.

Especially since different packages may have been developed using different vcs and reconciliiating the different package vcs versions would be too cumbersome.

My point is, we probably need a formal package release authentication that would provide legitimacy to a given package "version". (And we really need to define what a version is in Go)

For example to me, a "version" in Go would stem from making backward-compatible changes to a package AND releasing it (creation of a file describing a pkg which is a missing step today)  AND having a canonical PkgPath() registered for any $GOPATH.

[John Souvestre]

Ø  Using or not using a vcs should not impact the release of your source code.

 

+1

 

John

    John Souvestre - New Orleans LA

 

--

You received this message because you are subscribed to the Google Groups "Go Package Management" group.

To unsubscribe from this group and stop receiving emails from it, send an email to go-package-manag...@googlegroups.com.

[Sam Boyer]

I'm going to, probably in vain, try to save people a headache, because this discussion is now turning in an exceedingly well-trod direction.

1. Please, keep in mind that, while these points may be new to some folks here, none of the issues, concerns, or constraints raised in today's discussion of version control are new to those of us who've been around a while.
   
2. There is a massive gap - one that folks very frequently miss - between requiring that dependencies be accessed from an SCM, and requiring that your final app/project be packaged up in a particular SCM. Conflating the two is harmful.
3. The general thrust of consensus so far is that it is not feasible to include as part of the first solution round. Some feel it will never be feasible, or even worthwhile, to do so.
   
4. Definition of any properties that a release must have is predicated on having some kind of central mechanism to validate those properties. Like, a central registry. Which, again, is pretty much off the table for round 1.
   
5. So, yes, we're reliant on direct SCM interaction, at least for now, to source our packages. This need not be the end of the world, as long as we carefully define the semantics of our interaction with them.
6. Import paths, source repositories, the scope of packages to which a given version applies, have complex interdependencies. Try to talk through them on a mailing list, and we will all have a bad time.
7. If you really, really want to go down the source/SCM semantics road now, ping me in slack with your questions, because I'm fairly sure I'm already at the end of it. (Or, follow https://github.com/sdboyer/gps/pull/83)
   

[atd...@gmail.com]

If we do not clearly define terminology outside of the vcs realm, how do you expect to finish solving the few issues we have with vendoring?

Even, if that question is left unanswered, everyone needs to have the same understanding of terminology (which is the reason why the OP started this thread).

A mailing-list is a channel of discussion like any other.