dep: Roadmap for merging into the toolchain

4944 views
Skip to first unread message

Sam Boyer

unread,
Mar 7, 2017, 6:47:57 AM3/7/17
to golang-nuts
Hi folks,

Since we opened up the dep project in late January, there's been flurry of contributions, which has been great. However, the plans and end goal of dep haven't been entirely clear to everyone.

To address that, and help guide contributors, we've put out a roadmap.

We're hoping to have dep ready to merge into the toolchain at the beginning of the 1.10 cycle. But that'll only happen if the community steps up even more to help!

cheers
sam

Russ Cox

unread,
Mar 13, 2017, 5:09:19 PM3/13/17
to Sam Boyer, golang-nuts
Hi Sam,

You make it sound like dep is just going to become 'go dep' in Go 1.10. That's not the plan I thought we discussed.

As I wrote to you and the package management group in January:

I think it's very important instead to weave package and dependency management into the go command so that you don't notice it as a separate thing in day-to-day work. As a result, I see dep as an important improvement over current practice that's usable immediately, an important step toward a final solution, and also an experiment that helps us learn more about what structure works for people, but I don't see it, for example, as a prototype of the eventual go command integration (since it's not integrated at all).

In particular, I agree strongly with all of these decisions in dep:

- Manifest file listing version requirements.
- Lock file listing exact versions for a reproducible build.
- Version control repo as project granularity is good.
- Calling the group of packages released as a unit a "project" is a good name to establish.
- Implementing coordination with the build by populating vendor is clearly the right implementation.

For eventual integration with the go command and the broader go toolchain, some of the decisions are likely not optimal. In particular:

- go get vs not. Dep creates a parallel world separate from 'go get'. Having two is not great. Ideally, I'd like 'go get' to improve instead of adding a second mechanism.
- Subcommand complexity. As mentioned in #25, if dep foo became 'go dep foo', then it would be the first sub-sub-commands and likely indicates a less than clean integration.
- Vendor directories. As an implementation mechanism today, these make sense. They stop making sense if we can change the toolchain. Instead of translating the lock file into an ephemeral vendor directory, the tooling can just read the lock file directly.
- Multiple packages. At scale, I think allowing v1 and v2 of a single package to be linked into the final binary is unavoidable. It's also not possible to express in the "top-level vendor directory" model. Another reason to eliminate the "translate lock file to vendor directory" step.
- GOPATH vs not. As I understand it, dep continues to assume GOPATH, but in a multiversion world that may not be appropriate. We need to understand the implications here. GOPATH has to keep working well, but I think we probably want 'git clone foo && cd foo && do work' outside GOPATH to work equally well, and package management has an important role to work out here.

I understand that not changing the go command was a constraint from the discussions at Gophercon and also from the point of view of making something people can use today. That's absolutely appropriate and correct for dep. But for long-term integration, I think we do need to rethink these aspects of the go command itself. 

By far the most important things to do with dep right now are (1) build something people can use and get benefit from today and eliminate blockers to adoption and (2) based on experience with that usage, learn what the eventual design and go command integration should look like. 

It's important to note that I do not expect projects using dep to automatically work with the new go command integration too. In particular, I expect that the lock and manifest files will change in syntax and therefore file name during integration into the go command, so that it is possible for projects to work with both dep and the go command during a transition. I also expect that we need to have a discussion about supported version constraints, to avoid forcing our tools to solve NP-complete problems (it's not just crazy hard, it's crazy hard to explain to users when it breaks). The best input to that discussion will be examination of what dep users use.

I also was hoping to have more thoughts fleshed out more than a month ago, but I've had a sequence of higher-priority interrupts and haven't had a chance to do that yet. That's on me, and I apologize for that. (I'm not going to be around a lot for the next three weeks either, so likely not much will happen on my end until April.)

Dep is another important step - just as vendor dirs were - in the effort toward proper package management within the go command. It's a way for us to learn more about what works and what doesn't. There will be iteration and revision and almost certainly rethinking on the way into the go command proper. My concern is that the roadmap and some recent Reddit comments do not reflect that level of uncertainty. I believe we're still very much in the "build and experiment and learn" phase, not the "polish and ship" phase.

Russ

Sam Boyer

unread,
Mar 14, 2017, 12:36:35 AM3/14/17
to golang-nuts, samuel....@gmail.com
Hi Russ -

thanks, this is great. I think all of this will be much easier with these discussions happening in the open.


On Monday, March 13, 2017 at 5:09:19 PM UTC-4, Russ Cox wrote:
On Tue, Mar 7, 2017 at 6:47 AM, Sam Boyer <samuel....@gmail.com> wrote:
Hi folks,

Since we opened up the dep project in late January, there's been flurry of contributions, which has been great. However, the plans and end goal of dep haven't been entirely clear to everyone.

To address that, and help guide contributors, we've put out a roadmap.

We're hoping to have dep ready to merge into the toolchain at the beginning of the 1.10 cycle. But that'll only happen if the community steps up even more to help!

Hi Sam,

You make it sound like dep is just going to become 'go dep' in Go 1.10. That's not the plan I thought we discussed.

Ahh, if that's the way it's coming across, I apologize! I've tried to adhere as closely to the plans we've discussed as possible. At times, in some places (particularly high-level docs like roadmaps), I've omitted some detail, in favor of making the project approachable for new contributors.

For example, I can see how the wording I used in the OP here could be read as suggesting that it would become `go dep` in 1.10, which would violate your subcommand complexity requirement. That wasn't my intent; I most recently reiterated that requirement earlier today.

My guess is that my general enthusiasm about all of this may be obscuring some of the uncertainty. I mean...I'm enthusiastic about this effort, what can I say :) More on this at the bottom.

 

As I wrote to you and the package management group in January:

I think it's very important instead to weave package and dependency management into the go command so that you don't notice it as a separate thing in day-to-day work. As a result, I see dep as an important improvement over current practice that's usable immediately, an important step toward a final solution, and also an experiment that helps us learn more about what structure works for people, but I don't see it, for example, as a prototype of the eventual go command integration (since it's not integrated at all).

In particular, I agree strongly with all of these decisions in dep:

- Manifest file listing version requirements.
- Lock file listing exact versions for a reproducible build.
- Version control repo as project granularity is good.
- Calling the group of packages released as a unit a "project" is a good name to establish.
- Implementing coordination with the build by populating vendor is clearly the right implementation.

For eventual integration with the go command and the broader go toolchain, some of the decisions are likely not optimal. In particular:

- go get vs not. Dep creates a parallel world separate from 'go get'. Having two is not great. Ideally, I'd like 'go get' to improve instead of adding a second mechanism.
- Subcommand complexity. As mentioned in #25, if dep foo became 'go dep foo', then it would be the first sub-sub-commands and likely indicates a less than clean integration.
- Vendor directories. As an implementation mechanism today, these make sense. They stop making sense if we can change the toolchain. Instead of translating the lock file into an ephemeral vendor directory, the tooling can just read the lock file directly.
- Multiple packages. At scale, I think allowing v1 and v2 of a single package to be linked into the final binary is unavoidable. It's also not possible to express in the "top-level vendor directory" model. Another reason to eliminate the "translate lock file to vendor directory" step.
- GOPATH vs not. As I understand it, dep continues to assume GOPATH, but in a multiversion world that may not be appropriate. We need to understand the implications here. GOPATH has to keep working well, but I think we probably want 'git clone foo && cd foo && do work' outside GOPATH to work equally well, and package management has an important role to work out here.

I understand that not changing the go command was a constraint from the discussions at Gophercon and also from the point of view of making something people can use today. That's absolutely appropriate and correct for dep. But for long-term integration, I think we do need to rethink these aspects of the go command itself. 

I realize we've discussed these a bit further since that email, but for everyone's benefit:

All of these are things that, as you note, can only really be addressed once we're seriously looking at moving directly into the toolchain. It's felt to me like there's a catch-22 in bringing up such things up publicly; doing so would provoke (reasonable) responses of "we're getting ahead of ourselves," but not bringing them up leaves us open to being blindsided when we start seriously looking at toolchain integration.

(Subcommand complexity is sort of an exception; we're at least trying to design for minimizing changes when integrating into the toolchain. This seems tricky to me in general, as lessons learned in dep about tool ergonomics may not be ones that transfer well directly into the toolchain...unless we're explicitly thinking ahead to that now.)

In any case, I think the proto-proposal I wrote addresses everything (except subcommands) in this list. The only thing we differ on there is the future of `go get`, which I think makes sense to focus on as a tool for users, not developers. In general, that proto-proposal is the loosely-held goal I'm keeping in mind, and the capabilities described therein are a pretty straight shot from our current trajectory.
 

By far the most important things to do with dep right now are (1) build something people can use and get benefit from today and eliminate blockers to adoption and (2) based on experience with that usage, learn what the eventual design and go command integration should look like. 

I think it's certainly the case that we're doing 2) right now, as well as the second half of 1) - eliminating adoption blockers.

I did not realize from our discussions, though, that your intent was for dep to live for some time as a standalone tool on which the ecosystem operates. I see it in retrospect, thinking over our discussions, and it does seem like a less risky way of going about this whole transition. Happily, I don't think it necessarliy changes much in terms of our short or medium term priorities, as we've already been moving in that general direction.
 

It's important to note that I do not expect projects using dep to automatically work with the new go command integration too. In particular, I expect that the lock and manifest files will change in syntax and therefore file name during integration into the go command, so that it is possible for projects to work with both dep and the go command during a transition.

This is not something I recall from earlier discussions. It would be very helpful to hear more about how you imagine that process going. Are there changes you're imagining outside of the multi-version lock (above) and more restricted constraint operations (below)? I had nascent plans to add the former already, but it seems worthwhile to me to at least _try_ to minimize pain during the transition into the toolchain, if we can. Either way, it seems I overreached with the idea of a into-toolchain compatibility guarantee. Sorry. We can walk that back.
 
I also expect that we need to have a discussion about supported version constraints, to avoid forcing our tools to solve NP-complete problems (it's not just crazy hard, it's crazy hard to explain to users when it breaks). The best input to that discussion will be examination of what dep users use. 

I also was hoping to have more thoughts fleshed out more than a month ago, but I've had a sequence of higher-priority interrupts and haven't had a chance to do that yet. That's on me, and I apologize for that. (I'm not going to be around a lot for the next three weeks either, so likely not much will happen on my end until April.)

Not a problem! While this is a major question and certainly at the heart of how the system works/people will use it, in terms of practical engineering, it's quite contained. The only user-facing effect is, as you noted, the version constraints we allow people to declare.

Beyond that, I'm gonna leave this topic for another discussion elsewhere :)
 

Dep is another important step - just as vendor dirs were - in the effort toward proper package management within the go command. It's a way for us to learn more about what works and what doesn't. There will be iteration and revision and almost certainly rethinking on the way into the go command proper.

Of course - I'd have to imagine that everyone expects toolchain integration to be an iterative process. For me at least, the big question here is, "how much of that iteration and revision should we try to do now?" This highlights the tricky position that dep is in.

If we limit the scope of our planning with dep to problems that can be solved when outside the toolchain, then we might create a tool that works well for that problem, but makes poor assumptions that will ultimately make transition into the toolchain more difficult (subcommand + flag design is probably the greatest risk here), and painful for those who've begun using dep for real.

OTOH, if we just experiment with dep as a proto-toolchain tool and never really stabilize it, then we won't ever get that crucial feedback coming from people using dep in anger. There'd also be very little hope of getting data to inform the decision about constraint expressiveness.

These are all tensions reflected in your comments - we're in the experiment phase, but also needing to build something that helps, today. 

The tightrope I've been trying to walk is based on the goal of create two things: 1) a working tool that people have used, designed in a considered way that makes it mostly ready for absorption into the toolchain, and 2) a list of TODOs to take care of once absorption begins.
 
My concern is that the roadmap and some recent Reddit comments do not reflect that level of uncertainty. I believe we're still very much in the "build and experiment and learn" phase, not the "polish and ship" phase.

If I'm understanding your overall take correctly, then I'm guessing "candidate" was really the problem, as it suggests a direct merge of dep into the toolchain. So again, mea culpa - "the official experiment" may have been better.

But, I suspect your concerns may have less to do with any specific wording, and more from the rising tidal wave of effort around this right now. I think that's just peoples' excitement about seeing this problem solved, combined with even a whiff of officialness. And maintaining that momentum seems essential to making this experimental period an effective one - we want people on board, we want people kicking the tires, because otherwise, we're not getting the information we need. But none of that means we're not maintaining an experimental outlook. I feel like my recent response to Francesc's PR is a good illustration of that.

Like I said, I think we're walking a tightrope. I can certainly be more circumspect in my discussions of the future, but on the whole, IMO we're balancing on this tightrope pretty well so far.

cheers
s
 

Russ

Albert Tedja

unread,
Mar 14, 2017, 2:22:54 AM3/14/17
to golang-nuts, samuel....@gmail.com
I was using godeps before and now govendor, and never felt 100% happy with either one.  I have some feedback of how I think the package management tool should work from the developer perspective, and here some features that I think is crucial.

1.  Standardize the package file.  With govendor, this is vendor/vendor.json.  This is great, but I want to commit vendor/vendor.json to repo, but not the vendor folder.  I had to do some `git add -f`, a bit yucky.  Just standardize it, something like dep.json, or perhaps Depfile.  Doesn't matter the filename, just standardize it, and please don't put it under the vendor folder.

2. I want to specify which branch or which tag or which sha revision of my dependencies.  Do not let this process to be automatic unless a version is not specified in which case it should be the latest dep can acquire.  It is frustrating when it pulls the wrong one.

3. I want to update my vendor folder, and it should use the Depfile as the authoritative source of what versions should be downloaded.  Not the $GOPATH!  Package management tool should move away from relying $GOPATH/src as its authoritative source.  My $GOPATH/src contains versions of code that could be in development, months old, and what not.  Let me control my $GOPATH with `go get`, while dep can use other ways to pull code directly to the vendor folder without messing with the $GOPATH.  $GOPATH is the development workspace.  vendor folder should have the stable versions for that particular project.  I had a time when govendor pulled a version from my GOPATH. Ugh.

4. Packages/libraries should be allowed to have its own Depfile.  I have heard/read somewhere that somehow this is discouraged by the Go community.  I am not sure why.  I thought it would simplify the process by a whole lot from development perspective that you reuse other people code.  Put the responsibility to the library maintainers to maintain up-to-date Depfile.  And from dependency management, you don't have to scan the entire source files to find the dependency tree.  In case of version conflict, such as package A depends on C.v1, and package B depends on C.v2, just pick one (latest) version but WARNS the user.  If there is a compile error, it won't build anyway.  I feel like the only way to resolve this is to actually overwrite the import statements.

5. Since packages is allowed to have its own Depfile, then each project should only be concerned with its direct dependencies, not the complete tree.

6. Let Depfile be user specified, not something that dep automatically generates. I kinda like the way Gemfile and Gemfile.lock works.  Gemfile is user specified, while Gemfile.lock is automatically generated and contains the complete dependency tree. You commit Depfile to repo, but you do not commit Depfile.lock to repo.

Having said that, I normally approach problems from end-user perspective and start there. Assuming `dep` is the command here, the following commands are possibly some that I would use:
`dep init` - Read Depfile, checks out direct dependencies to vendor folder, figure out the entire tree, create Depfile.lock, and fill the vendor folder with the complete dependency tree, while also stripping out unnecessary files like tests, .gitignores, etc from them.
`dep update %package_name%` - Updates Depfile.lock for that particular package only to use the latest version (and of course its dependencies), iff the package is specified on Depfile with an unspecified version.  If not specified on Depfile, just throw the error.
`dep build` - Build the project using the vendor folder ONLY. This is to make sure that your app can be shipped to somewhere else with a different environment.  I can totally see somebody has github.com/foo somewhere in their GOPATH, but not specified on Depfile, and just says 'it works on my computer?'

And...that's it.  I don't understand why all the dependency tools have these dozens of commands that 99% of time you don't end up using.

I would rather have tools that is plain but consistent and has expected behaviors, than attempt to be smart but failed.

Something nice:
Allow zipping up the packages in the vendor folder so I can actually check the zips into the repo.  While this can be counterintuitive, but recently I have had a problem where I had to do a complete pull of my old project, and one of the dependencies had breaking changes.  I did not want to update my code to follow the breaking changes, and the package maintainer is bad at keeping track of branches and tags that there is virtually no way for me to go back in time and find the version that is compatible.  To make matters worst, this library depends on 6 other libraries that all depend on each other, and all have breaking changes.  Yes, NP-complete problem.  I ended up writing my own.

This also would be nice for deploys.  If you can check in your entire dependency, instead of redownloading all code from github or wherever, in case of situations where internet connection isn't available.

I hope I don't sound too demanding. I like Go, but the reason why I haven't committed to it is frustration working with the dependency tools, something that other languages seem to have no problem with.  And other tools like godeps and govendor seem to manage to create their own problems.

Konstantin Khomoutov

unread,
Mar 14, 2017, 3:01:16 AM3/14/17
to Albert Tedja, golang-nuts, samuel....@gmail.com
On Mon, 13 Mar 2017 23:22:54 -0700 (PDT)
Albert Tedja <nicho...@gmail.com> wrote:

> I was using godeps before and now govendor, and never felt 100% happy
> with either one. I have some feedback of how I think the package
> management tool should work from the developer perspective, and here
> some features that I think is crucial.
[...]
> I hope I don't sound too demanding. I like Go, but the reason why I
> haven't committed to it is frustration working with the dependency
> tools, something that other languages seem to have no problem with.
> And other tools like godeps and govendor seem to manage to create
> their own problems.

Please don't miss the monumental heaps of discussions about package
management in Go [1]. The solution dubbed `dep` here is an attempt to
finally converge on something which is a) a part of the stock Go
toolchain; b) serves most of needs the way most devs are comfortable
with. Please consider revieweing what's been discussed on [1] --
especially about this new attempt. Note that it even has its formal
proposal -- see links at [2].

1. https://groups.google.com/d/forum/go-package-management
2. https://groups.google.com/d/topic/go-package-management/P8TehVoFLjg/discussion

Jan Mercl

unread,
Mar 14, 2017, 3:22:22 AM3/14/17
to Konstantin Khomoutov, Albert Tedja, golang-nuts, samuel....@gmail.com
On Tue, Mar 14, 2017 at 8:01 AM Konstantin Khomoutov <flat...@users.sourceforge.net> wrote:

> Please don't miss the monumental heaps of discussions about package
management in Go [1].

Executive summary: Solving the Go's package dependency resolution problem can be either automatic or correct. Pick one.

--

-j

Sam Boyer

unread,
Mar 14, 2017, 7:15:27 AM3/14/17
to golang-nuts, flat...@users.sourceforge.net, nicho...@gmail.com, samuel....@gmail.com
I'm don't think that's an accurate summary, at all.

Jan Mercl

unread,
Mar 14, 2017, 7:28:25 AM3/14/17
to Sam Boyer, golang-nuts, flat...@users.sourceforge.net, nicho...@gmail.com
On Tue, Mar 14, 2017 at 12:15 PM Sam Boyer <samuel....@gmail.com> wrote:

> I'm don't think that's an accurate summary, at all.

The proof is easy ;-)

--

-j

Russ Cox

unread,
Mar 27, 2017, 9:14:57 PM3/27/17
to Sam Boyer, golang-nuts
On Mon, Mar 13, 2017 at 5:09 PM, Russ Cox <r...@golang.org> wrote:
By far the most important things to do with dep right now are (1) build something people can use and get benefit from today and eliminate blockers to adoption and (2) based on experience with that usage, learn what the eventual design and go command integration should look like. 

I've been away and am not fully back yet, but I want to reemphasize these two lines, for everyone on this list and anyone else reading. I've heard a few people in various forums saying things like "I won't use dep until it's official". 

Please don't wait. Use it now. Make it possible for dep users to depend on specific versions of your packages: convert your packages to have manifest and lock files, and add version tags to your repos. Use dep for your own projects, to find out how well the approach works. Let us know.

This is all critical. If you wait to try any of this until it's in the go command, then your feedback basically won't have any impact, because things will be so much harder to change at that point.

(The flip side of this is that dep should freeze these basic details so that users aren't chasing a moving target. They don't need to be perfect. We know that JSON is not perfect, for example, but it's fine for this trial. The goal is to learn about the overall experience and semantics offered, not the file format.)

Russ

Peter Bourgon

unread,
Mar 28, 2017, 4:39:06 AM3/28/17
to Russ Cox, Sam Boyer, golang-nuts
On Tue, Mar 28, 2017 at 3:14 AM, Russ Cox <r...@golang.org> wrote:
> On Mon, Mar 13, 2017 at 5:09 PM, Russ Cox <r...@golang.org> wrote:
>>
>> By far the most important things to do with dep right now are (1) build
>> something people can use and get benefit from today and eliminate blockers
>> to adoption and (2) based on experience with that usage, learn what the
>> eventual design and go command integration should look like.
>
> I've been away and am not fully back yet, but I want to reemphasize these
> two lines, for everyone on this list and anyone else reading. I've heard a
> few people in various forums saying things like "I won't use dep until it's
> official".
>
> Please don't wait. Use it now. Make it possible for dep users to depend on
> specific versions of your packages: convert your packages to have manifest
> and lock files, and add version tags to your repos. Use dep for your own
> projects, to find out how well the approach works. Let us know.

Agreed! But one important addendum: please wait until we've merged in
the manifest/lock file format changes[0][1] :) I'll post another
update to this thread when that's complete, as well as update dep's
README to remove the warnings. Thanks!

[0] https://github.com/golang/dep/issues/119
[1] https://github.com/golang/dep/issues/168


> This is all critical. If you wait to try any of this until it's in the go
> command, then your feedback basically won't have any impact, because things
> will be so much harder to change at that point.
>
> (The flip side of this is that dep should freeze these basic details so that
> users aren't chasing a moving target. They don't need to be perfect. We know
> that JSON is not perfect, for example, but it's fine for this trial. The
> goal is to learn about the overall experience and semantics offered, not the
> file format.)
>
> Russ
>
> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

Kiyoshi Murata

unread,
Mar 30, 2017, 8:42:01 PM3/30/17
to golang-nuts, r...@golang.org, samuel....@gmail.com
I think Russ' feedback may imply dep needing to be versioned, as in the project itself publishing releases. Specially considering Peter Bourgon's lastest response, there's no way of users to know if dep is usable yet or not (roadmap is too abstract in that regard), and after that, what features made in. Is that a real concern or am I too off the mark here?

Peter Bourgon

unread,
Apr 2, 2017, 1:46:16 PM4/2/17
to Kiyoshi Murata, golang-nuts, Russ Cox, Sam Boyer
The dep project is in an alpha state; we'll leave alpha and enter a
versioned releases phase as soon as the manifest/lock file format
changes are in.

Peter Bourgon

unread,
Apr 13, 2017, 3:25:33 AM4/13/17
to Russ Cox, Sam Boyer, golang-nuts
And an update: thanks to the excellent work of contributor Carolyn Van
Slyck, we have the manifest and lock file format changes complete and
merged[0] to dep. From this point forward the file formats should be
stable enough to start committing. So, as Russ originally noted:
please do give it a try!

[0] https://github.com/golang/dep/pull/342

Peter Bourgon

unread,
Apr 13, 2017, 8:10:59 AM4/13/17
to Russ Cox, Sam Boyer, golang-nuts
I should clarify one thing: there are two open issues regarding the
·semantics· of the files: how the default versions are parsed re: the
caret[0], and how OS/arch/build tags are parameterized[1]. Until these
issues are closed, dep may change its interpretation of the files. If
you're interested in helping us get to a place of stability faster,
please volunteer in those tickets :)

[0] https://github.com/golang/dep/issues/225
[1] https://github.com/golang/dep/issues/291

Cheers,
Peter.
Reply all
Reply to author
Forward
0 new messages