Draft spec for package management tool

[Peter Bourgon]

Hi Gophers,

The package management committee[0] has now completed a draft spec for
a dependency management tool. It is available here:

https://docs.google.com/document/d/1qnmjwfMmvSCDaY4jxPmLAccaaUI5FfySNE90gB0pTKQ/edit

Feedback is greatly appreciated. As before, interested Gophers can
comment on the document inline, and everyone can of course discuss
here or on Gophers' Slack #vendor. Due to the limits of Google Docs'
commenting system, I'd like to ask everyone to limit in-doc comments
to specific questions and clarifications, and to direct broader
questions and feedback to this email thread (or Slack).

As stated in the introduction, "once the spec has been reviewed by the
interested parties it will be converted into a proposal document and
submitted to the Go issue tracker."

The committee has already begun experimenting with a reference
implementation, tentatively called `dep`, which will be open-sourced
after the spec becomes a bit more stable. Logistically, it will begin
as a `go get`-able binary in the golang organization on GitHub, and
the committee aims for it to become a subcommand of the `go` tool by
the time of the 1.9 release. Contributors are extremely welcome; stay
tuned for more on that front in the coming weeks.

Finally, for completeness, I am personally interested in writing
separate docs which map each of the user stories[1], features[2], and
design space issues[3] to `dep` workflows and decisions, including
documenting the user stories and features that aren't presently
supported. I'll send another email when that's underway.

Thank you for your patience and interest!

Cheers,
Andrew, Ed, Jess, Peter, and Sam.

[0] https://docs.google.com/document/d/18tNd8r5DV0yluCR7tPvkMTsWD_lYcRO7NhpNSDymRr8/edit
[1] https://docs.google.com/document/d/1wT8e8wBHMrSRHY4UF_60GCgyWGqvYye4THvaDARPySs/edit
[2] https://docs.google.com/document/d/1JNP6DgSK-c6KqveIhQk-n_HAw3hsZkL-okoleM43NgA/edit
[3] https://docs.google.com/document/d/1TpQlQYovCoX9FkpgsoxzdvZplghudHAiQOame30A-v8/edit

[tvm...@gmail.com]

Would there be an easy way to do a dry-run on an update of a dependency?

Say for instance, if the newer version of the dependency breaks something, you want to test out the new version but easily be able to roll back to the previous version.

Best regards,

Ty

[mikegl...@gmail.com]

Great! Minor suggestion: would use "dep get" instead of "dep ensure". It does more things but is somewhat parallel to the "go get" command. It gets the package, get updates or get a specific version.

okay okay.. the real reason is that I would save 3 keystrokes.

[Jessica Frazelle]

-n is dry run, it's in the doc

--
You received this message because you are subscribed to the Google Groups "Go Package Management" group.
To unsubscribe from this group and stop receiving emails from it, send an email to go-package-management+unsub...@googlegroups.com.
To post to this group, send email to go-package-management@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Jessica Frazelle]

Can we keep comments limited to only two venues, right now it's slack, google doc comments and this thread. Let's consider it like the CAP theorem and you can only choose two ;)
What do you think can we cut off gdoc comments and just do the other two?

On Oct 31, 2016 10:34, "Jessica Frazelle" <m...@jessfraz.com> wrote:

-n is dry run, it's in the doc

On Oct 31, 2016 10:25, <tvm...@gmail.com> wrote:

Would there be an easy way to do a dry-run on an update of a dependency?

Say for instance, if the newer version of the dependency breaks something, you want to test out the new version but easily be able to roll back to the previous version.

Best regards,

Ty

--
You received this message because you are subscribed to the Google Groups "Go Package Management" group.

To unsubscribe from this group and stop receiving emails from it, send an email to go-package-management+unsubscri...@googlegroups.com.

[Sam Whited]

On Mon, Oct 31, 2016 at 10:33 AM, 'Jessica Frazelle' via Go Package
Management <go-package...@googlegroups.com> wrote:
> Can we keep comments limited to only two venues, right now it's slack,
> google doc comments and this thread. Let's consider it like the CAP theorem
> and you can only choose two ;)
> What do you think can we cut off gdoc comments and just do the other two?

Please, *please* don't make Slack an official discussion venue. I do
not want to have to log in to yet another web UI that I can't use my
own clients with or sign up for an account with yet another company
that could have done things in a standards compatible manner, but
instead opted to force me to use their clients and services if I want
to participate in a discussion which should be kept open.

Thanks,
Sam



--
Sam Whited
pub 4096R/54083AE104EA7AD3

[Jessica Frazelle]

Which is why I said "choose two" :)

[Sam Whited]

On Mon, Oct 31, 2016 at 10:50 AM, Jessica Frazelle <m...@jessfraz.com> wrote:
> Which is why I said "choose two" :)

I was specifically replying to:

> What do you think can we cut off gdoc comments and just do the other two?

Sorry, should have clipped the rest. I suppose that does mean that at
least partial discussion is still here, but it still wrankles that I'd
miss half the discussion just because I don't have a Slack account
(also that the archive would be split between both places).

—Sam

[Jessica Frazelle]

We really can't stop people from discussing on slack :/

--
You received this message because you are subscribed to the Google Groups "Go Package Management" group.

To unsubscribe from this group and stop receiving emails from it, send an email to go-package-management+unsub...@googlegroups.com.

[Sam Whited]

On Mon, Oct 31, 2016 at 10:56 AM, Jessica Frazelle <m...@jessfraz.com> wrote:
> We really can't stop people from discussing on slack :/

Of course, but it would be nice if it weren't encouraged in an
official capacity, or if any feedback that was going to be
incorporated was copied here for further discussion, though perhapse
this isn't realistically doable.

[kevin.c...@gmail.com]

This is great! Fantastic even.

Very happy to see references to the gps project and explicit manifest files, lock files and SemVer.

My first hope is that the `ensure` command (funny name, isn't it?) is capable of smoothly updating a single dep's version as described. This comes up a lot with rapidly evolving projects and is one of the most frustrating aspects of current dep management tools. I'm glad it's being addressed here.

My second hope is that some attention is paid to how and when test dependencies should be included or excluded from the manifest or vendor directory. My opinion is that test code is as important as non-test code, so test deps should always be vendored along with everything else. Or at least that should be the default.

Looking forward to the first incarnation of this tool :thumbsup:

-Kevin

[Pavan Kumar Sunkara]

I haven't seen any example about how use a different branch or revision of a repository. I understand that the spec intends to support them but example would help hash out any potential problems that may arise.

I am assuming that this tool also manipulates the GOPATH such that the dependencies can be stored in vendor directory and can be used from there?

Thanks.

[Jessica Frazelle]

The vendor directory was added to Go in 1.5 as experiemental and
exposed in 1.6, it's been there since then. It is merely using that
support, the tool does not manipulate GOPATH in any way.

> --
> You received this message because you are subscribed to the Google Groups
> "Go Package Management" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to go-package-manag...@googlegroups.com.
> To post to this group, send email to go-package...@googlegroups.com.

> For more options, visit https://groups.google.com/d/optout.

--


Jessie Frazelle
4096R / D4C4 DD60 0D66 F65A 8EFC 511E 18F3 685C 0022 BFF3
pgp.mit.edu

[Peter Bourgon]

The doc is open for comments to the interested Gophers, an opt-in list
for which registration closed in July. Everyone is free to comment in
this email thread or on Gophers Slack #vendor. The committee is
monitoring all of those channels.

On Mon, Oct 31, 2016 at 7:57 PM, <i...@uber.com> wrote:

> Could you open up the up for comments/suggestions?

[i...@uber.com]

On Sunday, October 30, 2016 at 9:28:40 PM UTC-7, Peter Bourgon wrote:

[Eric Johnson]

Awesome to see this moving forward!

Thanks for all the hard work.

On 10/30/16 9:28 PM, Peter Bourgon wrote:
> Hi Gophers,
>
> The package management committee[0] has now completed a draft spec for
> a dependency management tool. It is available here:
>
> https://docs.google.com/document/d/1qnmjwfMmvSCDaY4jxPmLAccaaUI5FfySNE90gB0pTKQ/edit

Minor item: the document has a few question marks throughout (I counted
three) that aren't captured in open questions.

The glossary says: "the unit of dependency is the repository, not the
package." This totally makes sense to me for DVCS, but is confusing to
me with respect to centralized servers (i.e. Subversion) that can
combine many projects - versioned in a unrelated way - into a single
repository. With centralized servers, this definition doesn't really
help. I suspect there's common understanding of this situation, but
perhaps the glossary just isn't clear? I checked the use-cases and
didn't see anything that covered this.

Manifest:
" alternate network source location" - Seems like this could be an
optional field for dependencies and overrides, but that's not written
here. That is, the import name + version may be sufficient to identify
network location, so "alternate network source" is optional. Also, long
term, it strikes me that there's some value in a layer of indirection. A
company might want to set up an internal repository of external
projects, so that reproducible builds use a corporate controlled
repository (US14). Do you intend here that the manifest identify the
direct location to the internal clone of external repository, or does
this mean a "root" location under which the package can be found.

License information is mentioned in the features document, but isn't
called out in the manifest. Licensing also not mentioned in use cases.
Is this over-reach in the features document, or an oversight for the spec?

Under "Init":
The assumption: "The project builds and works as intended with the
current GOPATH and/or vendored dependencies" - this fits the use case of
building a main application, but not so well for a library. In order for
libraries to build /test against multiple versions of dependencies,
vendoring dependencies is awkward at best, and in the alternate, cannot
possibly cannot all live on the same GOPATH. I've puzzled over this, and
think I've hit on a possible solution, outlined below.

Another assumption: "A VCS is initialized for the project (eg "git
init")." I think this means that the tool must assume that the source of
the project is already under version control, not that dep init will
also run "git init". The wording here confused me.

Answered questions:
" Look for a VCS metadata directory, from the current directory upwards"
- does this work for all known VCSes? I seem to recall Perforce storing
data about project roots somewhere other than the project directory, but
maybe they do it differently now? Been a long time since I used Perforce
regularly.

Status, fetch/update, remove
I suspect it would be extremely useful for downstream tooling (IDEs,
etc.) if dep status et. al. took an option to return machine readable
format - something like "-json". Probably something that can be punted
for future versions. Perhaps worth calling out for "not addressed" section?

Thanks so much for all the hard work trying to get this resolved!

Eric.

---------------------
Proposed concept for not vendoring dependencies, but enabling multiple
versions.

Assumption stated in "init" process is that everything builds with
current GOPATH and/or vendored dependencies. I don't see how I can make
this work if I want to test my library against multiple versions of a
dependency. (Example, two different versions of github.com/foobar/baz.)
I can't vendor both versions in my repo, and I can't put both versions
on the GOPATH either (since the whole point is that they share the same
name import name, but are of different versions). I see only two
solutions - either forcing developers to use alternate GOPATHs for this
case (a unique GOPATH for each project, but also a unique GOPATH for
each testing scenario), or generating a GOPATH based on some other
context. Either way, it isn't appropriate to put the full paths to these
scenarios in the lock file, because the location will be specific to a
machine and/or user. So this has to be done relative to a user-specific
configuration.

What if the dependency tool supports a special "deps" folder? In the
"lock" file, in addition to specifying the network location from which
the project is fetched, it also (optionally) contains a path, relative
to this "deps" folder, pointed to by an environment variable
(GO_DEP_PATH?), where specific versions of a dependencies can be found.
Fetching a dependency would store it in the GO_DEP_PATH, but under a
path that is specific to a version, so that different versions can exist
simultaneously on my machine. And then, enhance the go tooling to look
at the lock file, and add those relative locations within the
GO_DEP_PATH to the GOPATH for a build? (Probably, adding a tooling
option to generate the effective GOPATH, so all the tooling need not
recreate this logic?)

Now, if the tooling detects a GO_DEP_PATH, instead of vendoring the
dependencies into the project, it saves them into the GO_DEP_PATH, and
saves the path relative to GO_DEP_PATH in the lock file.

I can see potential challenges & limitations to this approach, but
perhaps this is somewhere to start?

Eric.

[Dave Cheney]

Hi Peter,

Would you please ask the committee to post a status report. It's been a month since the last update and I would like to stay current with your progress.

Thanks

Dave

[Peter Bourgon]

I've been posting [admittedly brief] updates to the original process
document as things have been progressing.

https://docs.google.com/document/d/18tNd8r5DV0yluCR7tPvkMTsWD_lYcRO7NhpNSDymRr8/edit#

The committee is actively working on a prototype implementation based
on the spec. My current plan/hope is to make the repo public at the
new year.

I've cc:'d the committee members individually if they want to provide
more detail.

Hope this helps!
Peter.

[Dave Cheney]

Thanks Peter.