[ANN] GLock - Dependency management w/out vendoring

[robfig]

Announcing a vendor-less dependency management tool.

https://github.com/robfig/glock

## Overview

This one is very simple.  It provides 2 commands and a version control hook:

• "glock save project" writes the transitive root package[1] dependencies of all packages under "project/..." to a GLOCKFILE
• "glock sync project" updates all packages listed in project/GLOCKFILE to the listed version. 
• "glock install project" installs a version control hook that watches for changes to project/GLOCKFILE and incrementally applies them.

Glock is similar to "godep -copy=false"

GLOCKFILEs are simple text files that record a root package's version, e.g.

bitbucket.org/tebeka/selenium 02df1758050f

code.google.com/p/cascadia 4f03c71bc42b

code.google.com/p/go-uuid 7dda39b2e7d5

...

## Workfow 

Here's the expected flow for working with a Glock-managed repo:

1. Clone the repo
   
2. "glock sync" your GOPATH
3. "glock install" the hook
4. "glock save" when you add/update a dependency

## Use case

At work, we keep our Go code in one repo (rather than many small ones) and use a single GOPATH.  This tool allows us to gain reproducible builds, with version updates automatically propagated to the team via the hook, with the following advantages:

• We still use the normal Go toolchain / dev process (e.g. not having to run everything in a godep sandbox).  We can more easily contribute to 3rd party libraries, since they are not in a vendor sandbox or have rewritten import paths.
• We avoid the repo bloat of checking in our dependencies (> 100 MB), in addition to the extra churn.  Updating a dependency involves a change to one line of a text file instead of thousand-line diffs.
• Much easier and less error-prone than manually checking in dependencies.  Developers don't have to fight git because git wants to make the project a submodule instead of just checking in the files.  Running glock on your CI server or as a pre-commit hook can ensure that any new dependencies have been recorded.

Hope it's useful for others as well.  There is some more information in the README

Questions / comments? 

-Rob

[1] "root package" refers to the base package in a repository.  For example, although code.google.com/p/go.net/websocket is a Go package, code.google.com/p/go.net is the "root package", and any dependencies on non-root packages roll up to the root.

[Jan Mercl]

On Tue, Aug 5, 2014 at 3:48 PM, robfig <rob...@gmail.com> wrote:
> Questions / comments?

What happens when some user go gets a project which developer
published at, say github and the developer uses this tool to make the
project dependent on locked down versions of other publicly accessible
(ie. go get accessible) repositories/packages which have been
meanwhile updated?

What will such user end with at his local machine after plain old go
get github.com/dev/proj?

-j

[Peter Bourgon]

Seems like a reasonable solution for anyone who wants reproducible
builds without vendoring dependencies. But I'm not sure I buy any of
your downsides of vendoring... I'm also very curious what your
dependency tree looks like, if you have over 100MB of Go source in it!

(snip)

> We avoid the repo bloat of checking in our dependencies (> 100 MB), in
> addition to the extra churn. Updating a dependency involves a change to one
> line of a text file instead of thousand-line diffs.
> Much easier and less error-prone than manually checking in dependencies.

Until the upstream disappears. —OK, OK, snark mode off :)

> Developers don't have to fight git because git wants to make the project a
> submodule instead of just checking in the files.

Also solvable with subtrees, or (of course) other tools like Godep.

[Rob Figueiredo]

What happens when some user go gets a project which developer
published at, say github and the developer uses this tool to make the
project dependent on locked down versions of other publicly accessible
(ie. go get accessible) repositories/packages which have been
meanwhile updated?

Hah, I knew you would find a way sneak in the "but resolving versions is undecidable" comment. 

This is just a tool for recording and updating packages in your GOPATH.  Its use case is not for the small open source projects, it is for the large (probably) closed source projects.  It also does not address publishing libraries that are consumed by other parties -- that can be done as usual using the hope-for-the-best policy.

If the build works on your local machine with the current versions of everything in your GOPATH, then you can record that and allow other developers to have the same environment.  

 

What will such user end with at his local machine after plain old go
get github.com/dev/proj?

The same thing that happens you "go get" anything else.  They would end up with that project in their GOPATH, using whatever cocktail of other packages happen to also be there.

[Rob Figueiredo]

 

Seems like a reasonable solution for anyone who wants reproducible
builds without vendoring dependencies. But I'm not sure I buy any of
your downsides of vendoring... I'm also very curious what your
dependency tree looks like, if you have over 100MB of Go source in it!

(snip)

https://gist.github.com/robfig/93f21860883b5ff969d8

We have about 50k lines of Go (ignoring generated protocol buffers).  

 

> We avoid the repo bloat of checking in our dependencies (> 100 MB), in
> addition to the extra churn.  Updating a dependency involves a change to one
> line of a text file instead of thousand-line diffs.
> Much easier and less error-prone than manually checking in dependencies.

Until the upstream disappears. —OK, OK, snark mode off :)

It's true, that is the one risk of this approach.  If that happens with any regularity then this would not be a good approach.  Luckily, it does not seem to happen in practice, and if it does you can either update to the latest revision (if it's just that they rewrote history), or upload a fork from an existing developer's machine.

On balance it doesn't seem like it could be more than a small and rare inconvenience.

 

> Developers don't have to fight git because git wants to make the project a
> submodule instead of just checking in the files.

Also solvable with subtrees, or (of course) other tools like Godep.

I did look at using subtrees instead.  I didn't because it doesn't work with all dependencies (only git), and I don't believe that updates to the subtree revision get applied to working directories on regular "git pull".  I see the auto-updating as a big feature of glock, to avoid the developers having to somehow know that they need to re-sync, or having to remember to always use git pull --recursive (or however you do it).

IMO the one-time setup of glock followed by auto-updates without having to change the normal workflow (use git as usual, use go as usual) as a big feature.

[wkharold]

> We avoid the repo bloat of checking in our dependencies (> 100 MB), in
> addition to the extra churn.  Updating a dependency involves a change to one
> line of a text file instead of thousand-line diffs.
> Much easier and less error-prone than manually checking in dependencies.

Until the upstream disappears. —OK, OK, snark mode off :)

It's true, that is the one risk of this approach.  If that happens with any regularity then this would not be a good approach.  Luckily, it does not seem to happen in practice, and if it does you can either update to the latest revision (if it's just that they rewrote history), or upload a fork from an existing developer's machine.

On balance it doesn't seem like it could be more than a small and rare inconvenience.

It only happens when you least expect, and can least afford, it to happen. Like when you're trying to cut a release under pressure from product marketing. That leaves a mark. 

[Jan Mercl]

On Tue, Aug 5, 2014 at 5:18 PM, Rob Figueiredo <rob...@gmail.com> wrote:
>>
>> What happens when some user go gets a project which developer
>> published at, say github and the developer uses this tool to make the
>> project dependent on locked down versions of other publicly accessible
>> (ie. go get accessible) repositories/packages which have been
>> meanwhile updated?
>>
>
> Hah, I knew you would find a way sneak in the "but resolving versions is
> undecidable" comment.

No, that one is still in queue ;-)

> This is just a tool for recording and updating packages in your GOPATH. Its
> use case is not for the small open source projects, it is for the large
> (probably) closed source projects. It also does not address publishing
> libraries that are consumed by other parties -- that can be done as usual
> using the hope-for-the-best policy.
>
> If the build works on your local machine with the current versions of
> everything in your GOPATH, then you can record that and allow other
> developers to have the same environment.
>
>
>>
>> What will such user end with at his local machine after plain old go
>> get github.com/dev/proj?
>>
>
> The same thing that happens you "go get" anything else. They would end up
> with that project in their GOPATH, using whatever cocktail of other packages
> happen to also be there.

Even though there are cases where it might work, I'm glad that we now
agree that it cannot work in the general case. And while talking about
general cases: It cannot work also for the (big, proprietary etc.)
case when (internal, proprietary, closed) project C depends on other
internal projects A and B if A and B use GLock and both depend on
whatever D, but in different/incompatible versions of D. Then glock
sync will attempt to checkout conflicting versions of D in different
moments. It can probably even sometimes silently change the semantics
of C's operations.

And finally, here it goes: It's because resolving version conflicts
can be either automatic or correct. Pick one.

-j

[Rob Figueiredo]

> This is just a tool for recording and updating packages in your GOPATH.  Its
> use case is not for the small open source projects, it is for the large
> (probably) closed source projects.  It also does not address publishing
> libraries that are consumed by other parties -- that can be done as usual
> using the hope-for-the-best policy.
>
> If the build works on your local machine with the current versions of
> everything in your GOPATH, then you can record that and allow other
> developers to have the same environment.
>
>
>>
>> What will such user end with at his local machine after plain old go
>> get github.com/dev/proj?
>>
>
> The same thing that happens you "go get" anything else.  They would end up
> with that project in their GOPATH, using whatever cocktail of other packages
> happen to also be there.

Even though there are cases where it might work, I'm glad that we now
agree that it cannot work in the general case. And while talking about
general cases: It cannot work also for the (big, proprietary etc.)
case when (internal, proprietary, closed) project C depends on other
internal projects A and B if A and B use GLock and both depend on
whatever D, but in different/incompatible versions of D. Then glock
sync will attempt to checkout conflicting versions of D in different
moments. It can probably even sometimes silently change the semantics
of C's operations.

And finally, here it goes: It's because resolving version conflicts
can be either automatic or correct. Pick one.

Correct.  Glock does not offer automatic resolving of version conflicts.

Glock is made for proprietary projects A, B, and C, which are all checked in to a single repository, and which all depend on a single set of dependency versions.  Resolving version conflicts with Glock is manual, not automatic.  

I tried to express that in the README -- perhaps it's not clear enough

[Jan Mercl]

On Wed, Aug 6, 2014 at 3:51 PM, Rob Figueiredo <rob...@gmail.com> wrote:
> Correct. Glock does not offer automatic resolving of version conflicts.
>
> Glock is made for proprietary projects A, B, and C, which are all checked in
> to a single repository, and which all depend on a single set of dependency
> versions. Resolving version conflicts with Glock is manual, not automatic.

I was careful to properly define a specific setup, which is routinely
happening all over commercial projects (of say, non trivial
complexity), but the above is not that setup and no one claimed that
the above setup cannot work. The above setup is a special case and it
will work.

I think we now agree that no version resolution system can work in the
general, which in no way means rare, case.

> I tried to express that in the README -- perhaps it's not clear enough

If the tool has limits within which it works/is safe to use/etc. -
it's always a great advantage to the user to know about them long
before she decides to use the tool. The particular limitation of the
GLock tool is that a GLocked project cannot depend (safely) on
another, separately GLocked project (in the first approximation).

-j

[Dmitri Shuralyov]

GLOCKFILEs are simple text files that record a root package's version, e.g.

Since you used the word "package", what happens if there is no root package? Meaning the repo only contains Go packages in its subdirectories.

An example of that is https://github.com/google/go-github repo. Note that there is no "github.com/google/go-github" Go package, but there are "github.com/google/go-github/github" and others.

My question is what would the GLOCKFILE contain (the answer is probably not a root package, but the root repo)?

[Dmitri Shuralyov]

I've just noticed that my exact question is already answered in the footnotes. Sorry for missing it.

In my defense, the footnote reference is not mentioned in the sentence I was looking at.

Also, may I suggest you use the term "repository root" instead of "root package" as that is a more accurate name/description and doesn't require a footnote explanation?

[Sean Russell]

On Tuesday, August 5, 2014 9:48:17 AM UTC-4, robfig wrote:

Announcing a vendor-less dependency management tool.

https://github.com/robfig/glock

...

At work, we keep our Go code in one repo (rather than many small ones) and use a single GOPATH.  This tool allows us to gain reproducible builds, with version updates automatically propagated to the team via the hook, with the following advantages:

Thanks for this package Rob.  It's an excellent execution of a solution to some dependency issues with which we've been struggling.  Some of the other non-vendoring tools have come close, but most either interfere more with the go toolchain or introduce more complex configuration DSLs than glock.

Cheers, and thanks for giving back.

--- SER

[exp...@gmail.com]

Hey Rob -

Thanks for writing this. I'm very interested in this tool and tools like it, and it sounds like your development workflow matches ours. We have one large polyglot repository rather than many small repositories (as Google and Facebook seem to do), and we'd like all of our code to have the same dependencies.

We have a large repository that contains multiple languages (Go, Java, etc). The root of our git repo is like this:

/

/ops/...

/java/...

/php/...

/go/   <--- GOPATH

/go/{bin,pkg}

/go/src/mycorp.com/  <-- our code

/go/src/{github,bitbucket,etc}.com <-- .gitignore'd

Unfortunately, glock seems to require that the package being locked (in our case, mycorp.com) be the root of the repository. Is there any reason that is a hard requirement? It seems reasonable to let it drop the glockfile in $GOPATH.

Thanks.

[robfig]

@Dmitri, thanks for the terminology clarification.  I updated the README to reflect that

@Sean Thank you for the kind words.  Please let me know if you have any issues or questions with the usage.

Unfortunately, I just realized that the git hook installed by glock was only being activated by "git pull", not "git pull --rebase".  I've pushed a fix, so any users should update glock and re-install the hook: 

go get -u github.com/robfig/glock

glock install project

Cheers,

Rob

[robfig]

Hi,

It wouldn't be hard to support GLOCKFILEs not in the repo root - the original motivation for that requirement was: 

- Avoiding multiple (presumably accidental) GLOCKFILEs in the same repo

- Knowing the filename relative to repository root makes it easy to filter the git log to look for changes

I would be supportive of such a change, as I suspect your setup is not uncommon.  Note that the GLOCKFILE would still go in /go/src/mycorp.com/GLOCKFILE, not $GOPATH (which would require bigger changes)

Potentially I'll have a chance to look at it this weekend, or feel free to submit a PR

Cheers,

Rob