feedback requested on pkg spec

[Matt Farina]

I started on, and just updated, a proposal for a package spec. The intent is to be in a similar vein to the specs for Rust, PHP, Ruby, JavaScript, .NET, and many others. This is my request for feedback on the spec. The spec is designed to provide enough meta-data for any tooling to solve a series of use cases.

I do realize that vendor-spec does exist and this is a competing spec.

Please note, my hope is for a spec that can support solving the use cases. I'm more tied to solving those than to any particular format.

Since this is inspired by package managers for other languages I wrote up an overview of them for anyone not already familiar with them and why paying attention to them is important.

Any feedback is welcome. Thanks.

[mhhcbon]

I like this very much.

I can only hope something will happen in the way you are promoting and realizing by the work already done.

Thank you very much for *all* the sharing.

[Konstantin Khomoutov]

On Mon, 9 Nov 2015 11:45:52 -0800 (PST)
Matt Farina <matt....@gmail.com> wrote:

> I started on, and just updated, a proposal for a package spec

> <https://github.com/mattfarina/pkg>. The intent is to be in a similar

> vein to the specs for Rust, PHP, Ruby, JavaScript, .NET, and many
> others. This is my request for feedback on the spec. The spec is
> designed to provide enough meta-data for any tooling to solve a
> series of use cases

> <https://github.com/mattfarina/pkg/tree/master/use_cases>.
[...]

> Since this is inspired by package managers for other languages I
> wrote up an overview of them

> <http://engineeredweb.com/blog/2015/pkg-mgr-overview/> for anyone not

> already familiar with them and why paying attention to them is
> important.
>
> Any feedback is welcome. Thanks.

I've skimmed through the documents at the 3rd and 1st links, and failed
to see the explanation of how the spec really deals with the transitive
dependencies (what you call "recursion").

Say, I take your manifest file and declare I want package
A at version 42 and package B at version 24;
Now A imports package C at version 172 and
B imports the same package at version 271.
How the proposed solution should deal with the case like this?
The real problem -- as I understand this -- is that the format of
packages used by Go (at least the `gc` toolchain, that is), has no
notion of versioning, so 1) it must all happen outside of the realm of
the `go` tool; 2) the approach taken by `npm` cannot work.

[mhhcbon]

Hi,

> Now A imports package C at version 172 and B imports the same package at version 271.

Do you have an example of strongly typed language which handles this ? Could be interesting to test it.

[Konstantin Khomoutov]

On Tue, 28 Jun 2016 04:15:24 -0700 (PDT)
mhhcbon <cpasmabo...@gmail.com> wrote:

> > Now A imports package C at version 172 and B imports the same
> > package at
> version 271.
>
> Do you have an example of strongly typed language which handles
> this ? Could be interesting to test it.

Maven of Java comes to mind [4].

I'm not sure it's idea of "the dependency with the shortest dependency
path wins" has any sense in it. I don't know of other
languages/package systems which have this problem solved.

[...]

> > failed to see the explanation of how the spec really deals with the
> > transitive dependencies (what you call "recursion").
> >
> > Say, I take your manifest file and declare I want package
> > A at version 42 and package B at version 24;
> > Now A imports package C at version 172 and
> > B imports the same package at version 271.
> > How the proposed solution should deal with the case like this?
> > The real problem -- as I understand this -- is that the format of
> > packages used by Go (at least the `gc` toolchain, that is), has no
> > notion of versioning, so 1) it must all happen outside of the realm
> > of the `go` tool; 2) the approach taken by `npm` cannot work.

4. https://maven.apache.org/guides/introduction/introduction-to-dependency-mechanism.html#Transitive_Dependencies

[Henrik Johansson]

Perhaps go for the "newest" version?

With semantic versions in place it should be easy especially if there is a way to pin/choose a particular version.

--
You received this message because you are subscribed to the Google Groups "Go Package Management" group.
To unsubscribe from this group and stop receiving emails from it, send an email to go-package-manag...@googlegroups.com.
To post to this group, send email to go-package...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Dave Cheney]

> With >>>>semantic versions<<<<< in place it should be easy especially if there is a way to pin/choose a particular version.

Yes! People are finally starting to catch on!

[Henrik Johansson]

http://www.well-typed.com/blog/9/

Pretty neat how Cabal did/does do it.

Simple warning on conflicts and chooses a default (newest?) version for compile.

 

[mhhcbon]

Thanks, indeed, it s interesting,

• "nearest definition" means that the version used will be the closest one to your project in the tree of dependencies, eg. if dependencies for A, B, and C are defined as A -> B -> C -> D 2.0 and A -> E -> D 1.0, then D 1.0 will be used when building A because the path from A to D through E is shorter. You could explicitly add a dependency to D 2.0 in A to force the use of D 2.0

"nearest definition" does not make sense when semver is involved, best version is chosen given the dependency rules expressed as semver ranges.
Which is not necessarily the latest.
They may conflict in which case we probably don t want to build a program upon them for the very same reasons explained in http://www.well-typed.com/blog/9/.

[Matt Farina]

You are right that transitive dependencies need to be handled.

The spec is just one part of the picture and doesn't take into account how transitive dependencies should be handled. This isn't a list of requirements for a package manager rather the manifest and lock files with the information they should have.

You are right that handling transitive dependencies should happen outside the `go` tool. `npm` works outside execution machine. `npm` fetches the packages, puts them in the right place, and separately the application is executed which picks up the packages from those places. The `vendor/` folder in Go can be used in a similar manner.

In Glide we scan the source to know all the transitive dependencies. For example, Glide scans the imports in use and pulls them in. So, if you import a package within Kubernetes it will just fetch the few dependencies that package (and it's tree) have rather than all of kubernetes (though there is a flag to fetch all of the things).  Sam Boyer is working on a smarter resolver right now as the next piece of iterative improvement.

[Matt Farina]

The diamond dependency problem (what you're noting with A and B importing C at different versions) is an issue solved by dependency managers for static and dynamic languages.

If you want to see a static language that does this look at Rust.

The simple solution is that libraries release versions (the popular versioning is Semantic Versioning which the go programming language itself uses). Then, those that import an outside dependency specify a range of API compatible versions (e.g, `>= 1.2.3, < 2`). The dependency manager resolver walks the complete tree and works out what version of each thing to use.

You can see Rust, Python, JavaScript, and even PHP have package managers that do this today.

It's a solved problem space we just need to have working in Go.

[Henrik Johansson]

Great! Is this settled then? SemVer with or without vendoring can be supported by a tool.

If gb and glide does this then maybe there can be an easier transition into go for people very used to, for better or worse, working with versions.

Personally I really like vendoring but proper versions would be a great help. Things like querying for possible updates and selecting them possibly even interactively would be really nice.

--

[Nathan Youngman]

Hi Matt,

How does this spec compare to https://github.com/kardianos/vendor-spec? Are there other similar specs in the Go community?

One thing I feel could be important is tracking of (git) remotes for dependencies. This may not be so important if using the `vendor` approach, but could be really valuable if some people choose to not check in the source code of deps. The idea being that you could fork a dependency and use that fork in place of the standard/package/name while waiting for upstream to merge a patch (with the import path of the original).

Nathan.

[Nathan Youngman]

Hi Matt,

How does this spec compare to https://github.com/kardianos/vendor-spec? Are there other similar specs in the Go community?

One thing I feel could be important is tracking of (git) remotes for dependencies. This may not be so important if using the `vendor` approach, but could be really valuable if some people choose to not check in the source code of deps. The idea being that you could fork a dependency and use that fork in place of the standard/package/name while waiting for upstream to merge a patch (with the import path of the original).

Nathan.

On Monday, 9 November 2015 12:45:52 UTC-7, Matt Farina wrote:

[Matt Farina]

Nathan,

A few things...

1. Go works with Bzr, Hg, and Svn in addition to Git. They are all in use and we need a system that plays well with all of them. Just something to keep in mind.
2. The spec at https://github.com/kardianos/vendor-spec is a lock file while the proposal I made was for both a lock and manifest file. Along with those there are use cases the information in the two files is needed to fill.
3. Your fork approach is one of the reasons we have the `repo` field in the spec I proposed and the `glide.yaml` file for Glide. Forks are a noted use case. This setup works for Git, Hg, Bzr, and Svn in practice.

Does that help?

- Matt

[Nathan Youngman]

Thanks. I musta missed that (repo).

Another thing that could be interesting to capture in a lock file is a checksum of the vendored files -- similar to the git sha (or other vcs) -- but after any pruning done by the tool.

The idea being that the update process could warn if those vendored files were "tampered" with (patched) before overwriting them.

I think little details like this could make it a pleasant experience even if the tools are otherwise fairly similar to other language communities. Of course there are much larger things to figure out and agree on first -- starting with who's on the committee.

Nathan.

[Dave Cheney]

On Fri, Jul 29, 2016 at 12:54 PM, Nathan Youngman <he...@nathany.com> wrote:
>
> Thanks. I musta missed that (repo).
>
> Another thing that could be interesting to capture in a lock file is a
> checksum of the vendored files -- similar to the git sha (or other vcs) --
> but after any pruning done by the tool.
>
> The idea being that the update process could warn if those vendored files
> were "tampered" with (patched) before overwriting them.

why ? They're being checked into source control, can't that track
changes for you?

[Nathan Youngman]

On 28 July 2016 at 21:45, Dave Cheney <da...@cheney.net> wrote:

On Fri, Jul 29, 2016 at 12:54 PM, Nathan Youngman <he...@nathany.com> wrote:
>
> Thanks. I musta missed that (repo).
>
> Another thing that could be interesting to capture in a lock file is a
> checksum of the vendored files -- similar to the git sha (or other vcs) --
> but after any pruning done by the tool.
>
> The idea being that the update process could warn if those vendored files
> were "tampered" with (patched) before overwriting them.

why ? They're being checked into source control, can't that track
changes for you?

Of course source control tracks changes. The point is that if you run `go vendor update some/repo` and it says "hey, you made some local changes -- might want double check before I overwrite all that".

Perhaps some fancy VCS commands could determine that without needing to store anything in the lock file. Implementation details TBD -- if it's an idea that ends up happening at all.

Nathan.

--

Nathan Youngman 
Email: he...@nathany.com
Web: https://nathany.com

[Sam Boyer]

Yeah, I've waffled on doing something like this, but haven't thus far in gps. It's a different part of the problem that currently seems less costly to solve with a norm - "you don't mess with vendor/ - that's tooling's job."

People still will, of course, but if we can establish the expectation that that's a bad practice and tooling will blow away any such tampering with wild abandon, then that's a much smaller solution space for us to cover, with far fewer failure modes, than trying to still operate correctly when under muddied conditions. (I have a whole diatribe on how I think *that* mentality is deeply idiomatic Go, but I'll save that for elsewhere)