--
Aram Hăvărneanu

--

You received this message because you are subscribed to the Google Groups "golang-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

On Monday, March 2, 2015 at 12:38:02 PM UTC-5, Brad Fitzpatrick wrote:

In Google’s internal source tree, we vendor (copy) all our dependencies into our source tree and have at most one copy of any given external library. We have the equivalent of only one GOPATH and rewrite our imports to refer to our vendored copy. For example, Go code inside Google wanting to use “golang.org/x/crypto/openpgp” would instead import it as something like “google/third_party/golang.org/x/crypto/openpgp”. This has worked out very well for us and we get reproducible builds. If we ever want to update openpgp in our vendored directory, we update it, verify that all the affected code in the world still compiles and tests pass (making code changes as necessary), and then check it in. The world compiles and passes at all points in the version control system’s history, and if we bisect in time, we see which version of an external library was in use at any point.

>> Also, will library packages also have dependencies specified like that or
>> only application/main packages?
>
> The mechanism should work for libraries too, but I think we'll discourage
> overuse of it for libraries. We imagine it'll be mostly used for package
> main.

On Mon, Mar 2, 2015 at 12:10 PM, Axel Wagner
<axel.wa...@googlemail.com> wrote:
> Reproducible build only requires all the version information of
> all transitive dependencies used to build a binary plus the version of
> the go toolchain and stdlib used

> (btw: Does godep or nut address that?
> Just out of curiosity).

On Mon, 2 Mar 2015, at 18:46, Aram Hăvărneanu wrote:
> One of the nicest things about Go is that all the information required
> to build Go programs is in the source code, a.i. in .go files, and not
> in any other files (Makefiles, .cfg files, etc).

It's good to see such discussion going on. I'm the author of nut so my comments might be bias :)

It sounds like there're two things nut does differently from Brad's suggestions:

1. the format of the config file
2. the file structure for vendored dependencies

For 1), I'm open to suggestions as long as the format is clear and concise. Currently nut adopts Tomlas the config format. An example of declaring depnednecies:

[dependencies]

"rsc.io/arm/armasm" = "616aea947362"
"rsc.io/x86/x86asm" = "af2970a7819d"

I personally think this is very clear and less noisy than JSON. But I understand the preference of using something Go standard library can parse. It wouldn't be hard for nut to support JSON as the config file. But I think the proposed JSON schema is a bit verbose: the keys of "ImportPath" and "Rev" seem unecessary. How about removing the keys?

{
    "Deps": {
        "rsc.io/arm/armasm": "616aea947362",
        "rsc.io/x86/x86asm": "af2970a7819d"
    }
}

For 2), I don't have a problem of renaming "vendor" as in nut to "internal": making vendored dependencies only accessible to current package makes sense.

> Vendoring should be used only for the dependencies of binaries, not for the dependencies of packages.

> officially recommends vendoring into an “internal” directory with import rewriting (not GOPATH modifications) as the canonical way to pin dependencies.

One advantage to Brad's proposed syntax is that we can add extra fields (if need be) while maintaining backward compatibility with older tools.

On 3 March 2015 at 08:46, Owen Ou <jing...@gmail.com> wrote:

[...]I think the proposed JSON schema is a bit verbose: the keys of "ImportPath" and "Rev" seem unecessary. How about removing the keys?

One advantage to Brad's proposed syntax is that we can add extra fields (if need be) while maintaining backward compatibility with older tools.

On Mon, Mar 2, 2015 at 1:57 PM, Andrew Gerrand <a...@golang.org> wrote:
> On 3 March 2015 at 08:46, Owen Ou <jing...@gmail.com> wrote:

>> the proposed JSON schema is a bit verbose: the keys of "ImportPath" and
>> "Rev" seem unecessary. How about removing the keys?
>
> One advantage to Brad's proposed syntax is that we can add extra fields (if
> need be) while maintaining backward compatibility with older tools.

> Maybe my answer to your first question helps?

On Monday, March 2, 2015 at 10:54:11 PM UTC+1, Andrew Gerrand wrote:

On 3 March 2015 at 06:34, <ben.d...@gmail.com> wrote:

Granted, this situation is a bit unusual since etcd is primarily intended to be an end-user executable rather than a reusable go library, and we intend to break out the parts that are used by cockroachdb (the 'raft' subpackage) into a separate package/repo, but that just moves the problem around. The new raft package would either have to vendor its dependencies and rewrite its imports, or use canonical import paths for everything with no way to control dependency versions even for its own tests. 

Vendoring should be used only for the dependencies of binaries, not for the dependencies of packages.

On Monday, March 2, 2015 at 10:38:02 AM UTC-7, Brad Fitzpatrick wrote:

We currently maintain those copies by hand. Instead, we want to write a file (filename and syntax to be determined), such as:

     src/cmd/internal/TBDCONFIG.CFG:

             “rsc.io/x86/x86asm” with revision af2970a7819d

             “rsc.io/arm/armasm” with revision 616aea947362

On Tue, Mar 03, 2015 at 10:56:55AM +0000, Matthew Sackman wrote:
> I do not understand why libraries should be treated differently to
> "programs".

> defines a common config file format for dependencies & vendoring

On Monday, 2 March 2015 17:24:41 UTC-5, Ben Darnell wrote:

But if we had this version-aware infrastructure (which I admit is a harder problem), would we still want to rewrite imports? I claim that for purposes of reproducible builds it's better to place the entire GOPATH under version control (as long as you can have per-project GOPATHs) than to use an internal/vendor directory to ensure that all your dependencies share a common prefix deep in the source tree.

On 3 March 2015 at 15:26, Brad Fitzpatrick <brad...@golang.org> wrote:

> Good point. That's probably the strongest argument in favor of Go instead of
> JSON. I'm always annoyed by the lack of comments in JSON.

Is there a use-case for vendoring some dependencies, but pulling others from your GOPATH?   Otherwise, I don't see the advantage of import rewriting over pointing GOPATH at your vendored libraries.

On Tue, Mar 03, 2015 at 02:48:27PM +0000, Matthew Sackman wrote:
> I'm writing package X. It depends on some other package Y that I have no
> control over. The most sensible and responsible thing I can do is to
> specify that X can only use the exact version of Y that I've built it to
> use. I have no ability to control what the author of Y does in the
> future.

On Tuesday, 3 March 2015 17:59:49 UTC+1, Giacomo Tartari wrote:

On 3Mar, 2015, at 17:49 , Michael Schuett <michae...@gmail.com> wrote:

I feel like allowing every import to specify a version ends up with people in "version hell". You would also have to specify the version everytime you wanted to use that package. If you changed one would an error be thrown until you changed it in the various other places that you are specifying the old version? 

I would assume so, but that is another easily automated task.

Could even be made interactive:

$go vendor

$go vendor found mismatching version for package X: v1, v2, v3.

do you wish to unify the versions or something? y,n

version [1], [2], [3]?

And it would be cool it tags are supported

//go:vendor rev:616aea947362

//go:vendor tag:616aea947362

On Tuesday, March 3, 2015 at 6:29:36 PM UTC+1, Govert Versluis wrote: 

I'm against storing this data in .go files, especially if it's in some form of magic comment.

The need for structure in directives such as these stands completely opposed to comments (which can be anything).

> On 3Mar, 2015, at 18:29 , Govert Versluis <gov...@ver.slu.is> wrote:
>
> If one uses tags or commit hashes in the .go source file, how are conflicting versions stored in the gopath?

Russ

On Tue, Mar 03, 2015 at 08:15:25PM +0000, Ian Davis wrote:
>
>
>
>
> On Tue, Mar 3, 2015, at 07:20 PM, shado...@gmail.com wrote:
> > I think a better solution is this:
> >
> > import "import/path/that/has/some/slashes#commit-like"
>
> This cant work in general. Does the commit hash have to be on every
> import in every file that uses the package? What happens if two files in
> the same project specify different commits?

On 3 March 2015 at 15:26, Brad Fitzpatrick <brad...@golang.org> wrote:

> Good point. That's probably the strongest argument in favor of Go instead of
> JSON. I'm always annoyed by the lack of comments in JSON.

{"comment": "this is a comment", "otherField": 1066, ...}

{"comment": ["lacking multiline strings", "use an array"]}

Would that not suffice despite its clunkiness?

Chris

--
Chris "or is it 'clunkyness'?" Dollin

On Tue, Mar 3, 2015, at 08:48 PM, Sam Dodrill wrote:
> > On Tue, Mar 3, 2015, at 07:20 PM, shado...@gmail.com wrote:
> > > I think a better solution is this:
> > >
> > > import "import/path/that/has/some/slashes#commit-like"
> >
> > This cant work in general. Does the commit hash have to be on every
> > import in every file that uses the package? What happens if two files in
> > the same project specify different commits?
>
> It's a commit-like so you get the advantage of specifying git tags, git
> commits, svn commits, bzr commits/tags etc.
>

Why not instead just add support to Go for an optional tag or commit ID to the import string, along the lines of what Joshua Marsh is suggesting:

    import "github.com/johndoe/mylib 1.0"
    import "bitbucket.org/janedoe/another fe3b56a01ccde"

If it's not there, use the "master" branch.

    import "github.com/old/mylib" // uses master branch

I believe that's where tooling comes in.  Also, see Russ's comment earlier in the thread.  From what I guess, vendoring would look like this:

On Tuesday, March 3, 2015 at 10:31:58 AM UTC-5, rsc wrote:

I like the idea of using a Go file, but note that it cannot be one that actually builds, because it needs to record the original import paths, not the vendored ones. I would expect that a Go config file would have a standard .go suffix (so that gofmt etc apply) but a build tag keeping it from building, a standard name (say, vendor.go), and a required package name (say, vendor):

---

// +build ignore

package vendor

import (

    // These are for the disassemblers.

    "rsc.io/arm/armasm" // 616aea947362

    "rsc.io/x86/x86asm" // af2970a7819d

)

---

Russ

On Tuesday, 3 March 2015 08:31:58 UTC-7, rsc wrote:

I like the idea of using a Go file, but note that it cannot be one that actually builds, because it needs to record the original import paths, not the vendored ones. I would expect that a Go config file would have a standard .go suffix (so that gofmt etc apply) but a build tag keeping it from building, a standard name (say, vendor.go), and a required package name (say, vendor):

---

// +build ignore

package vendor

import (

    // These are for the disassemblers.

    "rsc.io/arm/armasm" // 616aea947362

    "rsc.io/x86/x86asm" // af2970a7819d

)

---

Russ

> But the point is, I'm not mucking about with SHA commit hashes, that's what we have tools for

My understanding is that we're primarily discussing the format of the file the vendor information is stored in and the basic details of how vendoring will work. It's likely that tools will sprout up around this that will allow for what you describe, regardless of what format is chosen.

On Tuesday, 3 March 2015 23:40:39 UTC-7, Brian Picciano wrote:

> But the point is, I'm not mucking about with SHA commit hashes, that's what we have tools for

My understanding is that we're primarily discussing the format of the file the vendor information is stored in and the basic details of how vendoring will work. It's likely that tools will sprout up around this that will allow for what you describe, regardless of what format is chosen.

--

Nathan Youngman 
Email: he...@nathany.com
Web: http://www.nathany.com

           "ImportPath": "rsc.io/arm/armasm",

           "Rev": "616aea947362"

       },

       {

           "ImportPath": "rsc.io/x86/x86asm",

           "Rev": "af2970a7819d"

       }

   ]

}

We can start with that for discussion.

Note that we have rejected non-vendoring approaches that require modifications to GOPATH or new semantics inside the go command and toolchain. We believe it is important that the solution not require additional effort on the part of all the tools that already understand how to build, analyze, or modify code in the standard GOPATH hierarchy.

Vendoring should be used only for the dependencies of binaries, not for the dependencies of packages

On 3 March 2015 at 06:34, <ben.d...@gmail.com> wrote:

Granted, this situation is a bit unusual since etcd is primarily intended to be an end-user executable rather than a reusable go library, and we intend to break out the parts that are used by cockroachdb (the 'raft' subpackage) into a separate package/repo, but that just moves the problem around. The new raft package would either have to vendor its dependencies and rewrite its imports, or use canonical import paths for everything with no way to control dependency versions even for its own tests. 

Vendoring should be used only for the dependencies of binaries, not for the dependencies of packages. This particular problem could be solved by:

a) a policy of API stability adhered to by the raft subpackage and its dependent packages, 

b) testing infrastructure that is version-aware.

Personally, while I see the value of import-rewriting for corporate environments where you have one large meta-project, I think the better solution for the open-source world is to avoid rewriting imports and instead improve tooling support for a per-project GOPATH (I now let the emacs package go-projectile manage my GOPATH, which works very well for my workflow) and to use something like gpm or goop to pin dependency versions.

I think we can develop the tools to make it easier to test packages with different versions of their dependencies, and to make it easier for the ultimate consumer (a program binary) to vendor the correct dependencies (or provide diagnostics in the rare case of incompatible dependenct versions). Those tools may be [based on] projects like gpm or goop.

Right now I'm working on an API stability policy similar to the Go 1 compatibility promise for the gokit project and its dependencies. If that goes well, maybe the greater Go community can adopt the policy for their projects.

Andrew

[...]

The subject of this thread should likely have been "standardizing on a vendoring configuration file" instead.

On Wednesday, 4 March 2015 00:40:26 UTC-7, Alex Smith wrote:

While I like this idea (it is useful, especially for alpha/beta) how would backwards comparability work?  Is it a tag, branch, etc. with standardized naming?  This is *exactly* why I started using that weird go alias, it allowed me to "go get" a given library at a given time.

On Wed, Mar 4, 2015 at 1:31 AM, Nathan Youngman <he...@nathany.com> wrote:

I would lean towards the more verbose key/value pairs like {"repo":rsc.io/arm/armasm", "commit": "616aea947362"}. It's ugly and seems wasteful, but it provides more flexibilty down the road.

Since we already have all the code in our repo, I assume the reason why we want this metadata at all is to restore the world outside our hermetic bubble to match up?

Say I submit a patch but it hasn't been merged upstream yet. Maybe a tool would like to store the fact that I had these other "remotes": ["github.com/libgo/arm"] when I took my snapshot. Then if said tool can't find "616aea947362" on a colleagues computer, it can add the remotes and see if that helps. 

That's just an example. The point is simplicity and extensibility.

Nathan.

P.S. I have add comments for dependencies before. Usually that I can't use some new thing until such and such. It's fine to put those comments in the code alongside the import statement though. No need to stuff comments in the JSON.

Whatever comes of this discussion: Distancing any proposed model from how Google works internally is going to be difficult, but necessary.

The open source community as a whole has been burned before by the behaviors that emerge from a company culture that attempts open source but maintains things only in their personal vendorized ecosystem: remember the release of thrift.... followed by the release of thrift?  This is -- and there's no shame in this, but I remember a number of central gophers admitting their workplace basically means they never use 'go get' -- a large part of the reason 'go get' is in such a dilapidated state compared to the rest of the otherwise charming and excellent go tool.

I don't want to see go getting hung up on root misunderstandings of how a needs of a global collaboration culture may differ from those of a single company with a single dictatorial source tree with a single snapshot view available for a single point in time.  The latter is easier, to be sure.  Let's focus on moving beyond that.  Planet Earth does not have a single shared source tree.

Solutions need to account for usage *without prior coordination* between package authors.

... And that's where a whole bunch of the proposed stuff fails.  Badly.

This whole discussion started with vendoring presumed.  (Why is it even in the thread title?)  Import rewriting is then proposed as a workaround.

There's still huge trouble with this when extended to more than *literally one* project, so it's proposed (with *colossal* handwaving) that libraries and programs should be different.

These are both ridiculous duct tape workarounds.  No: import rewriting is a ridiculous ducktape workaround, and claiming that libraries and programs are clearly distinct is just lying back and thinking of England.

I propose we take a moment and clearly define the goalposts again: everyone's concerns are oriented around isolation, repeatability, and composability.  Are there better ways we can serve these goals, especially with scaling across many repositories and many uncoordinated authors?


### isolation

Notice how I didn't say "import rewriting".  What everyone wants is isolation.  We should discuss how to implement that concept.  "import rewriting" is one possible way to implement isolation; let's phrase the conversation around the concept first.

I second every single person (there have been many) who suggest either using per-project GOPATH settings, or -- if this hasn't been suggested outloud yet, I'd like to make the proposal -- creating a new system with similar semantics.  Having wholey separate directory trees for each project is, in fact, absolutely the semantic that I want.  Global variables are bad; global library heaps are equally bad, particularly when shared between many different programs of wildly unrelated authorship.  (Anyone from the maven world -- remember how it feels to `rm -rf ~/.m2`?  Tension just flows out of the body.  Tension that never should have existed.)

Much like we gophers value our statically linked single file binaries for their ease of shipping, so too do I value a project folder that is clearly that project, only that project, and all of that project.  This is what happens when I set `GOPATH=$project/.gopath/`.  Much like the whole truth and nothing but the truth, it's great!  This means I can sync severable parts of my work to various computers easily, etc, and is super empowering.

I regret to say I cannot buy suggestions that this would be an overwhelmingly disruptive chance to the go ecosystem.  Currently doing so with every single go library and program I've ever touched in fact seems like rather concrete proof that this is doable, since it is in fact, done.

Import rewriting makes sense in the situation that I have a diamond dependency (A -> B -> D.v1 & A -> C -> D.v2) and I'm willing to duplicate the two different versions of that transitively required library in order to isolate them while retaining both.  This is by far the exception case, not the norm.  Imagine if every package that used something as common as "fmt" from the standard library re-wrote and re-vendored it!

Note that I have little care for the size of my working tree in an active development environment -- duplicated libraries on disk there per project is fine; disk is cheap.  Duplication that ends up in version control is where concern lies, because this duplication has troublesome impacts over deep time, and the trouble can be pushed to other people who are then powerless to address it.


### repeatability

Notice how I didn't say "vendoring".  What everyone wants is repeatability.  "vendoring" is one possible way to implement repeatability; let's phrase the conversation around the concept first.

As pointed out earlier in the threat, repeatability is also maintained by systems such as Nix: hashes do wonders.  Similarly, I've been using git submodules to get perfectly reproducible builds.  Bower can pin dependencies by hashes.  There's a huge list of concrete examples of repeatability without vendoring.  Hash-based resource references solve repeatability.

Vendoring may score points in the immediate offline mode.  And I'm deeply pleased with importance placed on offline operation -- I share this priority the extent that I label my business cards "sneakernet advocate" -- but offline operation is orthogonal to vendoring and also available through other choices.  It's good to have a command after which the build is guaranted not to need network; that command need not be `git clone`, and could just as easily be some other subcommand of the go tool.  I think folk coming from other languages would be perfectly unsurprised by such a feature in the go tool, and we could have very satisfying outcomes from a command that does all resource acquisition and then stops.

Vendoring has gained traction largely because it works without additional tooling, and that makes it the most contributor-friendly thing in the current landscape where the go tool has no chosen stance and thus the community as a whole is uncommitted.  In discussing additions to the go tool, this impetus is made irrelevant.

Vendoring has a variety of serious drawbacks.  It gets repeatability right and *everything else* wrong.  It's not transparent to libraries; it results in globally multiple histories for files; and it results in permanent bloat to a project source repo.  (This was discussed before at length in a particular golang project forum and this table of comparisons was made back in 2013, which remains fully accurate today: https://gist.github.com/heavenlyhash/6343783 )

The most major threat to repeatability when using hash-based resource references is network available in the deep time sense: what happens when the organization behind mycompany.net goes out of business and drops their domain, or library bananapancakes is renamed to something more user-centric, etc.  In tracking recent changes, this is not frequently an issue, but doing a bisect across a well-aged repo can become troublesome.  Fortunately, this is solvable: The system must have a way to replace old network addresses with updated aliases.  We can do this.


### composability

I'm not sure this is in the discussion as such yet, but it's been mentioned in passing, and so I figured I might as well give it a name and a heading.

As a user of go, when examining a new library for potential use, I regularly want to see if it builds.  Thereafter, I want to see if its tests pass.  This requires that the library specify it's dependencies; specifically, it means the library must have repeatability just like a "real" "program".  

Bundler and rubygems actually got this fairly right: specifying a semver dependency range in one file, and specifying the "locked", pre-resolved versions of everything depended on (including transitively) is a great example of unifying both composability and repeatability goals.  Bundler fell short in that it still only resolves down to a precise semver string, where it should really resolve to a hash.  In the javascript world, Bower did similar things, and also carried the resolve results all the way to a hash: this truly satisfies repeatability.  We should take a page from these books.

While it is possible to ship executables at the end of the day without tackling this (and that is the most important thing), as Matthew Sackman has pointed out, it's a severe cramp to developer usability if we ignore this.

On the plus side, this is also possible to do later, since isolation and repeatability are solvable without a solution to composability -- but it may be worth thinking about it now, when discussing data formats.


### aside: this "libraries and programs are different" thing *doesn't work* in the field

Treating libraries and programs differently has a hidden presumption: that I have any control over other authors.

Regrettably, I lack total control over all programmers.  (Imagine!  We could forgo this entire discussion!  World conquest by friday!  Etc.)  And even if everyone in the world reoriented their views to ask all present and future users whether a package should be a library or not, there's no guarantee we'd agree.  And a time dimension is indeed a part of that discussion, unless we break free of VCS repository layouts entirely, which has not yet come to pass.

As one example, Docker has some code that I'd like to use as a library.  But they don't consider themselves a library, and as a result, they've already vendored about 5 megs of other sources (and if including history, a much larger amount even after compression).  This situation is understandable -- they consider themselves a "program", and rightly so -- but nonetheless, it's become very difficult for me to reuse their code without forking (and I mean manually, by copy-paste, a highly undesirable outcome).  And then AFTER copy-paste forking the sections of code I want, I have to rewrite all their imports again to fit in my project, since I'm already using several of the same libraries at the same versions.  This entire process would be nonsensical made-work that actively distracts from real development, and with all due respect to all the authors in this scenario who are all understandably acting in their best interests, I want no part of this to be in my future for other golang proje
cts.

The reason for storing the git hash is that it documents which version of the particular dependency you are using.

In the case of vendoring it helps the tool maintain the right version of files in the local directories. In the absence of vendoring, the toolchain can check out the matching versions of dependencies rather than whatever happens to be on the master branch that day.

In the case of conflicts, the toolchain can choose to select or pin particular dependencies to resolve them.

Am Dienstag, 3. März 2015 00:47:41 UTC+1 schrieb rsc:

People seem to agree that libraries should not vendor other libraries without a good reason. That's actually beyond the scope here. There is some question about what happens if libraries *do* vendor other libraries, and that gets to the heart of the proposal. 

If there is an agreed-upon vendoring approach (import path rewriting) and data format that describes what vendoring did happen, then no matter what vendoring helper did it, another tool (or perhaps the same one) can come along and analyze the situation and either just fix it or help the programmer fix it with minimal interactions.

On the other hand, if the vendoring tools use different semantics or even just different config files, this kind of metatool becomes much more difficult.

The goal here is to (1) agree on import path rewriting (as opposed to dynamic GOPATH tweaking or other complications), and (2) agree on a config file format that records the rewriting that happened, so that different projects can use different tools and still interoperate, both for just building things and for using meta tools that provide things like deduplication and diamond crushing.

Russ

+1

> The thing that we as a community need to figure out is the recommended
> configuration file format.
>
> We’d prefer something that the Go standard library can already parse
> easily. That includes XML, JSON, and Go.  Nobody likes XML, so that
> leaves JSON and Go.
>
> godep already uses JSON, as do Go tools themselves (e.g. “go list
> -json fmt”), so we’ll probably want something like godep’s JSON
> format:
>
> {
>     "Deps": [
>         {
>             "ImportPath": "rsc.io/arm/armasm",
>             "Rev": "616aea947362"
>         },
>         {
>             "ImportPath": "rsc.io/x86/x86asm",
>             "Rev": "af2970a7819d"
>         }
>     ]
> }

The thing that we as a community need to figure out is the recommended configuration file format.

We’d prefer something that the Go standard library can already parse easily. That includes XML, JSON, and Go.  Nobody likes XML, so that leaves JSON and Go.

godep already uses JSON, as do Go tools themselves (e.g. “go list -json fmt”), so we’ll probably want something like godep’s JSON format:

{

   "Deps": [

       {

           "ImportPath": "rsc.io/arm/armasm",

           "Rev": "616aea947362"

       },

       {

           "ImportPath": "rsc.io/x86/x86asm",

           "Rev": "af2970a7819d"

       }

   ]

}

We can start with that for discussion.

Note that we have rejected non-vendoring approaches that require modifications to GOPATH or new semantics inside the go command and toolchain. We believe it is important that the solution not require additional effort on the part of all the tools that already understand how to build, analyze, or modify code in the standard GOPATH hierarchy.

Salman Aljammaz <s...@0x65.net> writes:
> go get would just work.

mat....@coull.com writes:
> We currently use a single vendor folder for managing package dependancies
> across a few production apps, and frankly have had no issues with it.

> It's a simple way of managing a production build state, that's all.
> I can see how a complex app with ever changing external packages could get
> annoying, and in that sense a config file is a pretty simple solution - i
> mean who can't read nicely printed json.
>
> I find I mostly work out of my GOPATH and only update vendor folders when
> I'm completely happy with a current package version - they are in effect a
> barrier to unknown change, you have to physically do something to it to
> make it different - a good step to reproducible builds. They're not meant
> to give you multiple GOPATH folders that you have to switch to everytime
> you move to tweak something in a new app. That's how I see it, and seem to
> work anyway.

Andrew Gerrand <a...@golang.org> writes:
> Thats' what Mat is saying; they use vendoring for managing their apps'
> dependencies. There's nobody downstream from there.

On 5 March 2015 at 17:17, Andrew Gerrand <a...@golang.org> wrote:

> Thats' what Mat is saying; they use vendoring for managing their apps'
> dependencies. There's nobody downstream from there.

Except distribution packagers, who are constrained to packaging each
library separately and thus have to re-rewrite all the import paths
back to where they started. :/

On 03/05/2015 06:17 PM, Andrew Gerrand wrote:
> On 6 March 2015 at 11:14, Axel Wagner <axel.wa...@googlemail.com> wrote:
>
>> vendoring doesn't create issues for the people who use it :) The claimed
>> problems with vendorings are for the users of a project, not the
>> developers. One of my main grudges with vendoring is, that it passes the
>> buck downstream, to package maintainers and users (who want to re-use
>> your stuff and can't because you vendor a different/custom/too old/too
>> new version of the library and they first have to go through version
>> hell). So yes, I acknowledge that it probably makes the life easier for
>> the author of a package, but that's not the issue :)
>>
>

> You shouldn't be vendoring the depdencies of libraries, but rather the
> dependencies of binaries.
>

> Thats' what Mat is saying; they use vendoring for managing their apps'
> dependencies. There's nobody downstream from there.
>

On 5 March 2015 at 18:20, Eric Myhre <ha...@exultant.us> wrote:
> Speaking of!  Hey Tianon, would you be so kind as to rewrite docker//pkg/system's utimes syscall stuff out into a separate library, just for me?
>
> You see, docker rightly considers itself a program, and so vendored things, but I really want to use that package as a library ;)

On 5 March 2015 at 19:57, Tianon Gravi <admw...@gmail.com> wrote:
> This is actually one of the reasons we vendor, but don't rewrite import paths.

If only it were just about unstable; API incompatible is also a factor.

♥,
- Tianon

> officially recommends vendoring into an “internal” directory
> with import rewriting (not GOPATH modifications) as the canonical
> way to pin dependencies.

On Friday, 6 March 2015 04:19:57 UTC+1, Alex Smith wrote:

On this note what if git repos had some sort of tag based system eg:

go get example.org/repo-tag

or

go get example.org/repo-1.0

 

...

If you're concerned about the remote changing, you could still maintain your own fork so you control the changes. This seems like a reasonable compromise.

I'd also like to add a vote to the "vendoring is a bad idea" camp. Reproducible builds and composability are critical, but vendoring isn't the only way to achieve that, it's just the only one that fills the peculiar shape created by what "go get" does and doesn't do. The go tool does both too much and too little to solve these problems: too little because it provides no mechanism for reproducible builds, and too much because its conventions, like using a repo path (but no version!) as an import identifier, make it impossible to properly solve them and maintain full compatibility without breaking the conventions.

Rather than rehashing all the problems with vendoring (well-summarized in Eric's post), I'll just jump straight to the direction I'd rather see things go:

The solution I've created for go builds at my workplace is a tool called begot[1]. You record your dependencies in a yaml file that points to git repos, optionally pinning to a branch or tag or commit (sorta like a Gemfile). It fetches them for you into a private cache, which you're not supposed to look at. Commit ids are recorded in another file (sorta like a Gemfile.lock). When building, it ensures that all the cached repos are at the right version, rewrites imports as necessary (but the rewrites are only local and never pushed anywhere!), constructs a temporary workspace with symlinks pointing to the right places (sorta like Blaze), then sets GOPATH and calls the go command to do the build. When you declare a dependency on another repo that uses begot, it checks that all the versions match.

This decouples import paths used in code from the origin of the code, so you can feel free to use short and convenient names for your imports, with the actual origin recorded in the dependency file. It also means it's easy to switch a whole dependency to a private fork (e.g. while you're waiting for a patch to get merged upstream) with a just one-line change to the dependency file. (You can even do this for indirect dependencies without forking all the intermediate repos.)

It also allows for extra metadata to be included with dependencies, such as which git transport to use, so private repos with ssh just work.

Here's what I think this approach does right:

- Fully deterministic and reproducible builds.

- Continues the go tradition of using scm tools as a "package system" without binary packages (even more than the vendoring approach). 

- Decouples import paths from code origin.

- Allows more metadata than can fit in an import path (versions, aliases, transports, etc.).

- Always collapses multiple copies of a library.

- Doesn't bloat repos with full copies of their dependencies.

Here's where it needs some more work:

- It requires that all versions of a repo referenced in a project's dependency tree be locked to the same commit id. This is probably too strict for fast-moving projects. The approach could be extended to be more flexible, e.g. with semantic versions in git tags.

- In order to depend on a library that uses begot, you have to use begot too. Obviously, if an approach like this becomes the standard and is built into the go tool, this isn't a concern anymore.

- It doesn't provide quite as strong of a reproducibility guarantee as vendoring: if you refer to a third-party's repo on say, GitHub, and they delete it or rewrite their history such that a commit id no longer exists, you might not be able to reconstruct it. My plan for solving this is a separate tool (not yet written) that makes it easy to mirror git/other scm repos so that you have a copy under your organization's control. This could integrate with the build tool so that using a mirror in place of the origin is completely transparent.

---

The go tool and vendoring conventions have created a sort of local maximum that we can't break out of without sacrificing something, but is far from the ideal. I suggest we think about how to sacrifice some compatibility with the present world so we can get to a better one in the future.

[1] https://github.com/solanolabs/begot

Our proposal is that the Go project,

officially recommends vendoring into an “internal” directory with import rewriting (not GOPATH modifications) as the canonical way to pin dependencies.


1. defines a common config file format for dependencies & vendoring
   
2. <snip>
   

On 03/06/2015 08:23 AM, roger peppe wrote:
> I think the decision whether to to vendor
> dependencies is largely orthogonal to how to specify what those dependencies
> are.
>

>> That's why I agree with Russ Cox that this conversation should be about
> agreeing on a configuration file format, not standardising on a single tool
> that interprets that format.
>

> That's also part of why I am opposed to import path rewriting, because
> import path rewriting constrains the available options. In my view
> a vendored location is just *one* place from which the dependency path can be

> satisfied. It's the job of a tool to combine the Go source code with a
> set of dependencies
> to produce a final binary. We don't need to decide on that tool right now.

On 03/06/2015 08:07 AM, Ian Davis wrote:>
> The problem with this is that it doesn't guarantee reproducible builds

> repositories have moved location or even vcs type.
>

On 03/06/2015 07:03 PM, David Reiss wrote:
> What we actually want is two dependency metadata files, not one. The first
> file (the "request file") is where a human writes a request for a package,
> which obviously includes a repository identifier and may also include some
> sort of version specifier. Often this will just be "master"/"go1",
> sometimes another branch, and rarely a specific commit id. Or maybe some
> sort of semantic version specifier. The second file (the "lock file") is
> where a tool will record the actual commit ids that it got when resolving
> the version specifiers at a specific time. The request file is meant to be
> human editable and the lock file is not. The lock file contains all the
> metadata needed to reproduce a build (not the data, which could either be
> embedded in the project repo or not, depending on your vendoring
> preference).

...

> next we can solve the vendoring problems by vendoring everything into one huge repo.

>> For more options, visit
>>
>> ...
>

> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-dev+...@googlegroups.com.

On 7 March 2015 at 01:03, David Reiss <dav...@gmail.com> wrote:
> What we actually want is two dependency metadata files, not one. The first
> file (the "request file") is where a human writes a request for a package,
> which obviously includes a repository identifier and may also include some
> sort of version specifier. Often this will just be "master"/"go1", sometimes
> another branch, and rarely a specific commit id. Or maybe some sort of
> semantic version specifier. The second file (the "lock file") is where a
> tool will record the actual commit ids that it got when resolving the
> version specifiers at a specific time. The request file is meant to be human
> editable and the lock file is not. The lock file contains all the metadata
> needed to reproduce a build (not the data, which could either be embedded in
> the project repo or not, depending on your vendoring preference).
>
> In the status quo, the request file isn't actually a separate file but is
> spread across the imports of all .go files, and it doesn't support version
> specifiers.

On Mon, Mar 2, 2015 at 12:10 PM, Axel Wagner
<axel.wa...@googlemail.com> wrote:
> Reproducible build only requires all the version information of
> all transitive dependencies used to build a binary plus the version of
> the go toolchain and stdlib used

Unfortunately, experience shows this is not true.

A reproducible build requires the source code needed to make
the build. With vendoring, you have the source code. Without it,
you need not only the version information but also a *means* to
acquire the source code—a network that is functional and fast
enough. You might be surprised how often that requirement is
not met.

> (btw: Does godep or nut address that?
> Just out of curiosity).

Godep records the output of "go version" when it generates
the file Godeps.json.

On 9 March 2015 at 22:44,  <murillo...@gmail.com> wrote:
> If all you need to a reproducible build is the source code, then storing
> commit's identifiers is pointless, just save the code in the vendored
> folder.

> The go way of vendoring looks like this for me:
>
> go vendor bitbucket.com/lunargoddess/superimportantproject (it is a super
> important project, so I need vendoring...)
>
> no need for config files, nor stuff like that. Just like you do go get, you
> should do go vendor.
>
> then the go-vendor command scans superimportantproject imports list, copies
> the sources to
> $GOPATH/src/bitbucket.com/lunargoddess/superimportantproject/vendored/ and
> rewrites the imports list from "github.com/someone/something" to
> "bitbucket.com/lunargoddess/superimportantproject/vendored/github.com/someone/something"