Go vendor file specification

[Daniel Theophanes]

Are people actively working on a specification for the vendor file? Is there a living document? I am not aware of one currently.

If there is no existing draft, I would like to propose we use the following as a base:

https://github.com/kardianos/vendor-spec

I'm aware that many go users don't want to vendor packages. That's fine. Please don't exhaust yourself to try to convince others of your opinion in this case. All sides have been well stated.

If you have comments or are a tool owner that copies packages, please reply or open a pull request.

-Daniel

[Matt Silverlock]

Take a look at https://github.com/tools/godep#file-format as well.

Note that there seems to be a strong preference to retain the FQDN for the imports (i.e. $YOURPACKAGE/internal/github.com/$SOMEPERSON/$PACKAGENAME, etc) rather than just $PACKAGENAME because it makes imports much more "obvious" (i.e. you can tell where they came from).

[roger peppe]

In the thread in golang-dev, I proposed this JSON format, taking
on board some of the feedback there and my experience
with the godeps tool. It has had some positive responses and no
objections so far.
Please consider it.

// Dependencies holds information on all dependencies of some set of
// packages.
type Dependencies struct {
// Repos holds information on all the VCS repositories used by
// all dependencies, keyed by import path pattern. The key must
// be of the form path/..., indicating that the repository holds
// all the packages inside path/... .
Repos map[string]Repo

// Pkgs holds information on the source code by used by all
// dependencies, keyed by package path. It will hold only
// packages which are actually used as dependencies, so will not
// include every package found in the above repositories.
Pkgs map[string]Pkg
}

// Repo holds information on a repository.
type Repo struct {
// VCS holds the type of VCS that the repository uses, such as
"git", "hg", "bzr".
VCS string

// Rev holds the VCS-specific revision of the dependency.
Rev string

// Revno holds the revision number of the revision. It will be
// empty for VCSs that do not support revision numbers.
Revno *int `json:",omitempty"`

// Date holds the commit time of the revision.
Date time.Time
}

// Pkg holds information on a package dependency.
type Pkg struct {
// SHA256 holds the SHA256 sum of all the non-test source files in
// the package's directory ordered lexically by lower-cased file
// name. If a package contains two names that differ only in
// case, it will be an error.
SHA256 string

> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Daniel Theophanes]

It's true, I started to glaze over on that thread after a bit so I must have missed your message. I found your message on page 8 of 9 in that thread.

I like how you separated out repositories and packages where repositories have vcs info. I'd like to think more on that. I like the date too. I think it would be a mistake to include the pkg hash though. The copied files will need to be re-written and thus the file content of the non-test packages will change between the version checked in and the version copied.

You seem to argue against import path re-writing in the rest of your original post. I thought that was the entire point here, that we were NOT going to mess with the existing GOPATH, either in content or in definition. Could you comment on that?

Regarding having both a Import field and a Local field, where the actual place on disk is different then fully qualified import path. I have seen times where cockroachdb needed to vendor etcd which vendored golang.org/x/net/context. That import path was fun. Having both "Import" and "Local" would allow moving golang.org/x/net/context to the root of the import directory while still retaining the fact it came from the etcd repository. This would also allow the tool (and end user) to decide if they want super long paths with fully qualified names or if they are alright shortening them according to some define rule set.

Thanks,

-Daniel

[roger peppe]

On 10 April 2015 at 14:58, Daniel Theophanes <kard...@gmail.com> wrote:
> It's true, I started to glaze over on that thread after a bit so I must have
> missed your message. I found your message on page 8 of 9 in that thread.
>
> I like how you separated out repositories and packages where repositories
> have vcs info. I'd like to think more on that. I like the date too. I think
> it would be a mistake to include the pkg hash though. The copied files will
> need to be re-written and thus the file content of the non-test packages
> will change between the version checked in and the version copied.

The dependencies file is all about dependencies and not about
how those dependencies are stored. There are a quite a few
possibilities there, and the vendor file format discussion is
supposed to be orthogonal to those.

Even if import paths are rewritten (something that I don't think should
be mandatory) there are ways in which the source code hash
can still be useful. For example, the vendoring tool could rewrite the
import paths but keep the diffs. Also just because the hashes is there
doesn't mean they *need* to be checked.

> You seem to argue against import path re-writing in the rest of your
> original post. I thought that was the entire point here, that we were NOT
> going to mess with the existing GOPATH, either in content or in definition.
> Could you comment on that?

As I said above, that's a decision that I don't think is being made right now
and doesn't need to be. The important thing is that we have a standard way
to record dependencies

> Regarding having both a Import field and a Local field, where the actual
> place on disk is different then fully qualified import path. I have seen
> times where cockroachdb needed to vendor etcd which vendored
> golang.org/x/net/context. That import path was fun. Having both "Import" and
> "Local" would allow moving golang.org/x/net/context to the root of the
> import directory while still retaining the fact it came from the etcd
> repository. This would also allow the tool (and end user) to decide if they
> want super long paths with fully qualified names or if they are alright
> shortening them according to some define rule set.

The "fun" you have with nested vendoring above is exactly what I'd very
much like to avoid. I hope we can standardise on one-package-path to
one-implementation. The dependencies file can specify dependency
information for package paths, then it's up to other tools (as yet unspecified)
to determine how to map those package paths to the actual source code
that the dependencies imply.

cheers,
rog.

[Daniel Theophanes]

Thank you for your reply. I suspect we have two different end goals in mind.

 * My goal is to be able to "go get" a package with all vendor's packages in a build-able state. This *requires* import path re-writing.[1]

 * Your goal appears to be to describe the dependencies that are possibly include with the source for posterity. But I could be misunderstanding.

While you say it isn't in scope to decide how the tool behaves, the minimum required fields for the file will strongly influence the possible behaviors of a tool. Without a "Local" field an entire possible tool behavior is eliminated. So to counter, the tool behavior, to a degree, is very much in scope.

I've updated my spec with a VersionTime field and an FAQ:

https://github.com/kardianos/vendor-spec#faq

-Daniel

[1] Vendoring "bitbucket.org/kardianos/rdb" and "bitbucket.org/kardianos/rdb/ms" would need to re-write the imports to work correctly if "go get"-able is a desired trait. 

[Luna Duclos]

Your assertion this requires import path re-writing is false. It merely requires the source to be in the repo, and some mechanism for go get to use that source. One of those mechanisms is import path re-writing, but it certainly isn't the only possibility.

--

[roger peppe]

On 10 April 2015 at 16:17, Daniel Theophanes <kard...@gmail.com> wrote:
> Thank you for your reply. I suspect we have two different end goals in mind.
> * My goal is to be able to "go get" a package with all vendor's packages in
> a build-able state. This *requires* import path re-writing.[1]
> * Your goal appears to be to describe the dependencies that are possibly
> include with the source for posterity. But I could be misunderstanding.

If your only goal is the former, then you don't need a dependencies
description file at all AFAICS. An author can vendor a package's dependencies,
rewrite the import paths and make it available.

There's more to it than that though. Many of the issues were
discussed at length in the golang-dev thread. Specifically it's
useful to have a way of describing the dependencies for all
the tasks *other* than "go get" involved with maintaining
a package.

- Updating dependencies.
- Editing dependencies.
- Synchronising dependencies between different packages.

> While you say it isn't in scope to decide how the tool behaves, the minimum
> required fields for the file will strongly influence the possible behaviors
> of a tool. Without a "Local" field an entire possible tool behavior is
> eliminated.

That's not true. For any given tool, it's entirely possible to have
other metadata for that tool. For example, for your vendoring tool example,
another file might contain a mapping from package path to local path,
independent of the dependencies file. I see the mapping from
import path to actual source code as orthogonal to the recorded
dependency information.

cheers,
rog.

[Daniel Theophanes]

Hi, This is true if you allow want to run another command after "go get" or "git clone" but before building. If this is within the design parameters, then yeah, you're correct. I thought it was out of the design parameters.

-Daniel

[Daniel Theophanes]

The original reason the vendor file was proposed was to facilitate import path re-writing.

I strongly suspect we are trying to solve different problems, or perhaps the same problem in very different ways.

From the initial proposal:

Our proposal is that the Go project,

1. officially recommends vendoring into an “internal” directory with import rewriting (not GOPATH modifications) as the canonical way to pin dependencies.

2. defines a common config file format for dependencies & vendoring

and ends with:

Note that we have rejected non-vendoring approaches that require modifications to GOPATH or new semantics inside the go command and toolchain. We believe it is important that the solution not require additional effort on the part of all the tools that already understand how to build, analyze, or modify code in the standard GOPATH hierarchy.

I think if we aren't talking about import rewriting, then we are talking about different vendoring strategies and thus probably different config files.

-Daniel

[roger peppe]

On 10 April 2015 at 17:02, Daniel Theophanes <kard...@gmail.com> wrote:
> The original reason the vendor file was proposed was to facilitate import
> path re-writing.
>
> I strongly suspect we are trying to solve different problems, or perhaps the
> same problem in very different ways.
>
> From the initial proposal:
>
> Our proposal is that the Go project,
>
>

> 1. officially recommends vendoring into an “internal” directory with import

> rewriting (not GOPATH modifications) as the canonical way to pin
> dependencies.
>

> 2. defines a common config file format for dependencies & vendoring

>
>
> and ends with:
>
> Note that we have rejected non-vendoring approaches that require
> modifications to GOPATH or new semantics inside the go command and
> toolchain. We believe it is important that the solution not require
> additional effort on the part of all the tools that already understand how
> to build, analyze, or modify code in the standard GOPATH hierarchy.
>
> I think if we aren't talking about import rewriting, then we are talking
> about different vendoring strategies and thus probably different config
> files.

I believe that it is possible to address proposal 2 without necessarily making
a final decision about proposal 1.

It was my understanding that the opinion in golang-dev was that
proposal 1 was fairly controversial (there are a number of advantages
and disadvantages of vendoring, and there are issues which don't
currently have a good answer), so we can concentrate entirely
on proposal 2 for the moment, as we need that *at least*.

I think that's entirely possible. I don't see why the choice
of a common file format for dependencies needs to preclude
non-vendoring approaches.

FWIW at least one successful vendoring tool (godep) has a dependency
file format that doesn't include any information about local
import paths.

cheers,
rog.

[Daniel Theophanes]

Hi Rog,

I guess I'll disagree with you about defining a config file without agreeing on proposal (1).

I'm a current user of godep, but it was first used for GOPATH re-writing. Import path re-writing was added later (and currently broken on Windows and plan 9). I think support for re-writing paths in the configuration file could be enhanced. I also have problems with various aspects of the configuration file: From https://github.com/kardianos/vendor-spec#faq

The godeps meta-data file includes the revision number in with the revision hash. It includes the go version number which is detrimental in a team, especially when one team member is tasked with getting a product ready for the next Go version before that Go version is released. It also lacks a Local field for adequate re-write support and doesn't have a time of revision.

While we do disagree, I think I understand the root of the difference. Thank you.

-Daniel

[roger peppe]

On 10 April 2015 at 17:40, Daniel Theophanes <kard...@gmail.com> wrote:
> Hi Rog,
>
> I guess I'll disagree with you about defining a config file without agreeing
> on proposal (1).
>
> I'm a current user of godep, but it was first used for GOPATH re-writing.
> Import path re-writing was added later (and currently broken on Windows and
> plan 9). I think support for re-writing paths in the configuration file
> could be enhanced. I also have problems with various aspects of the
> configuration file: From https://github.com/kardianos/vendor-spec#faq
>
> The godeps meta-data file includes the revision number in with the revision
> hash. It includes the go version number which is detrimental in a team,
> especially when one team member is tasked with getting a product ready for
> the next Go version before that Go version is released. It also lacks a
> Local field for adequate re-write support and doesn't have a time of
> revision.

AFAICS the Local field is there entirely to make something
that should never be happening (nested vendoring) more palatable.

If this *is* happening, I think it's reasonable that the vendored import paths
are even uglier than the usual rewritten paths. At least the
import paths represent the reality of the situation.

cheers,
rog.

[bak...@gmail.com]

For dependencies and versions, I wish it was as easy or simple as:

import "package-name optional-version"

No JSONs, no vendor dep files, etc. all in go files. Make it optional.

E.g.:

import "github.com/example/test"         //go get will import latest version of test
import "github.com/example/test 1.2.3"   //go get will import version "1.2.3" of test
import "github.com/example/test 1ceb8dc" //go get will import test with commit id "1ceb8dc"

or longer version 

import "github.com/example/test 1ceb8dcb0f108eb98cb28401c66f743c3abb4620"  

For naming, I second "vendor" name instead of "internal".

[Daniel Theophanes]

Hi,
Thank you for reading it. It has been discussed to include the version information in comments in go files. They can't go in the import string because that would break compatibility.

Specifically, "vendor" refers to taking a dependent package from an external source and putting in a under a local directory.

The "internal" folder has already been decided upon, as has import re-writing. The missing piece is what the config file should look like.

-Daniel

--
You received this message because you are subscribed to a topic in the Google Groups "golang-nuts" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/golang-nuts/-8xRooqqyJo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to golang-nuts+unsubscribe@googlegroups.com.

[Uli Kunitz]

How would a version extension of the import string break compatibility? Old import paths would still work. 

The only issue that I see is the issue that multiple versions of the same package might be included by different packages or even a single package. While this may break Go tool conventions it wouldn't break the language as such.

From my perspective this extension would be from a user's perspective the most simple solution and would get rid of additional files with a specific format.

To unsubscribe from this group and all its topics, send an email to golang-nuts...@googlegroups.com.

[Daniel Theophanes]

Regardless of if it breaks or works, locking an import to a specific version isn't the only goal.
https://github.com/kardianos/vendor-spec#import-and-local-fields

Also read the example given.

-Daniel

[Uli Kunitz]

I don't the support the idea to allow random remappings of import strings. The consequence of such a feature would be that importing the fmt package could mean anything.

I think the agreement must be on the problems to solve. I see currently two problems.

1) I cannot fix an import to a specific version.

2) The imported path might disappear for whatever reason.

The first problem can be solved by the extension of the import path. The second problem can only be solved by vendoring. If you are intending to solve the second problem then I agree you have to develop mechanisms to support vendoring and Brad's proposal tries to standardize the data format to support vendoring.

[Daniel Theophanes]

Hi, Thanks for reading. I'd like to correct one aspect of what you read. If "Local" = "fmt", then that means that the "fmt" package (wherever it's source) was copied to the "internal" directory. So to import that "fmt" package you would import "path/to/mypkg/internal/fmt", which is very distinct from just "fmt". I've added a paragraph to try to make sure this is clear:

https://github.com/kardianos/vendor-spec#import-and-local-fields

This is an attempt by the community to try to standardize the meta-data file to support vendoring. If there are others ongoing efforts I'd like to know of them.

-Daniel

[timothy....@gmail.com]

Could you explain, where the different versions would live when 2 packages have 'vendored' a different version?

In your example, you are using github.com/example/test, so what does it become on disk?

I think the obvious approach is some form of import path rewriting, so this does not get away from that.

When go1.5 enables the internal packages, it might be possible for the discussion to coalesce, but right now it seemed stuck on wanting to vendor w/ import path rewriting and not having agreement on the import path rewriting. 

automatic vendoring if the import path was prefixed with 'internal' might be nice... but more people would have to think it through.  It solves the problem i raised in your example since it confines deps just to your project, and doesn't pollute the GOPATH in general.

So that is to say

import "github.com/example/test"               // non-vendored dep

import "internal/github.com/example/test"      // package private internal dep, latest code, pulled from github.com/example/test

import "internal/github.com/example/test 1.2.3"// package private internal dep, specific version

Of course, looking at the packaging solutions in other langs, they often have funny characters to say what level of version (semver) they want.  I'm not sure i'd go that far, and putting the version strings in the code is still a bit odd to me.

I think i'd prefer:

1. A package file to specify dependencies

2. using the  'internal' prefix to trigger that dependences local to package.

'internal' proposal:

https://docs.google.com/document/d/1e8kOo3r51b2BWtTs_1uADIA5djfXhPT36s6eHVRIvaU/edit

[Daniel Theophanes]

Hi Timothy,

Copying packages into the internal directory will not be done automatically, ever. You can also use "internal" directories right now. The go tool just won't enforce it in user packages until go1.5.

To answer your question about two different versions of the same package, it would be up to the user and the vendor tool to choose one to use. If you actually wanted both (a possibility) you should just vendor them into different directories.

To be clear, if I wanted to vendor the package osext, the package would be copied into "path/to/mypkg/internal/github.com/kardianos/osext". Your import path would never start with "internal/".

To state it again, from the example file:

--

[Daniel Theophanes]

From the example file: https://github.com/kardianos/vendor-spec#example

These would be the import paths:
"github.com/kardianos/mypkg/internal/rsc.io/pdf"
"github.com/kardianos/mypkg/internal/crypto/tls"

"github.com/kardianos/mypkg/internal/github.com/coreos/etcd/raft"

"github.com/kardianos/mypkg/internal/golang.org/x/net/context"

Hope that makes more sense,

-Daniel

[Dan Kortschak]

I does however break the separation of concerns between the build tools and the compiler. If you really want to do this, a comment after the import string would do just as well.

[Медер Бакиров]

суббота, 11 апреля 2015 г., 22:18:47 UTC+6 пользователь timothy....@gmail.com написал:

Could you explain, where the different versions would live when 2 packages have 'vendored' a different version?

In your example, you are using github.com/example/test, so what does it become on disk?

plain

 

"import github.com/example/test"

will be as it is now, i.e. non-vendored and become on disk: "$GOPATH/.../github.com/example/test"

imports with versions can go into "$GOPATH/.../[a-z]+/github.com/example/test", e.g.:

"import github.com/example/test 1.2.3"

would go into "$GOPATH/.../a/github.com/example/test"

"import github.com/example/test 1ds34f"

would go into "$GOPATH/.../b/github.com/example/test"

and so on...

[Daniel Theophanes]

Hi,

If you vendor a package, the imports will be rewritten and they will be copied to your package's "/internal/" directory. Please read the full example section here: https://github.com/kardianos/vendor-spec#example

If the specific tool used to copy imports wanted to, it could allow putting them in "a" and "b" folders. But then it would also need to re-write the import paths to refer to them. It should also be noted that it would be a mistaken in every case I can think of to import the exact same path, but with different version, more then once. Tools should disallow this, though I don't think it would be a hard requirement to do so.

-Daniel

--

[Медер Бакиров]

BTW, different versions of the same package with optional version part can be stored on disk like this:

$GOPATH/.../{hash of the import path}/github.com/example/test

A hash can be a non-cryptographic Fowler–Noll–Vo hash function: FNV-1 or FNV-1a, which is supported by Go. So, an import path:

import "github.com/example/test 1.2.3"

will be stored on disk like "$GOPATH/.../f6d67a1b34a00719/github.com/example/test/", 

where f6d67a1b34a00719 is a FNV_1a hash of this import string: "github.com/example/test 1.2.3"

воскресенье, 12 апреля 2015 г., 15:17:53 UTC+6 пользователь Медер Бакиров написал: