How do library dependencies work today?

[Ben McCann]

Hi,

I'll start off by saying I've never written a line of Go code. I started reading some Go code and trying to figure out how dependencies work and it's a completely baffling to me even after skimming the docs. I'm guessing the answer here for some of this stuff is just that Go is a newer language and hasn't figured all of this out, but I'm really curious how people use Go to build larger projects right now.

Things I'm confused about:

• You depend on source code instead of binaries?
• You always depend on head and there's no way to depend on a version of a library?
• How does a library get updated? Does the build process essentially do a git pull on every transitive library dependency each time you compile? Or once you download the code it doesn't update? The former seems not very performant. The latter seems like a nightmare in working with a team where everyone ends up with a different dependency versions depending on when they downloaded the code.
• This seems awfully brittle. What happens when someone deletes a GitHub repo you depended on? Coming from Java this is not a worry because once you publish a package to the Maven central repository it is always there. Also, if someone changes where they host their source code you have to update it in every single file that uses that package? That seems rather strange to me as well compared to having a list of packages configured one place in your project (like a build file) and imports that do not rely on the source code location. This has happened several times with dependencies I've used in Java and there it requires absolutely no changes, but seems really hard in Go. E.g. someone moves their project from Sourceforge to GitHub or from their personal GitHub to a GitHub organization. In Go that seems like a complete nightmare because you have to not only update every file in your project to point to the new location, but you also have to get all the libraries you depend on transitively that may depend on the same library to also make the same change.

Appreciate the help. It looks like this group is trying to come up with better solutions for some of these problems, so I'm guessing these are all things you've thought about.

Thanks,

Ben

[William Kennedy]

I'll start off by saying I've never written a line of Go code. I started reading some Go code and trying to figure out how dependencies work and it's a completely baffling to me even after skimming the docs. I'm guessing the answer here for some of this stuff is just that Go is a newer language and hasn't figured all of this out, but I'm really curious how people use Go to build larger projects right now.

Things I'm confused about:

• You depend on source code instead of binaries?

Yes

• You always depend on head and there's no way to depend on a version of a library?

By default the Master branch is used. You can tag the Master branch with "go1" and the tool will use the latest revision with that tag.

• How does a library get updated? Does the build process essentially do a git pull on every transitive library dependency each time you compile? Or once you download the code it doesn't update? The former seems not very performant. The latter seems like a nightmare in working with a team where everyone ends up with a different dependency versions depending on when they downloaded the code.

When you use "go get" the dependencies are pulled down and then the project is built and installed. Using the -u option will update all the code for an existing repo, instead of performing the clone.

ex.  go get github.com/goinggo/revel-mgo   then use    go get -u github.com/goinggo/revel-mgo

The "go get" tool already support Mercurial, Bazaar, Github, Subversion and maybe a few more.

---

It is recommended to Vendor your projects. You have several tools such as Godep (https://github.com/kr/godep) and Vendorize (https://github.com/kisielk/vendorize) to choose from.

The godep tool does this very well. By Vendoring your "go get" will only need to pull down one project and the dependencies are consistent for each and every build. Godep makes it easy to update those dependencies when necessary. Using godep you can issue a "godep go get" to bring the project down and build it. Subsequent builds can be done using the tool as well "godep go build".

Blog Post On Using Godep: http://www.goinggo.net/2013/10/manage-dependencies-with-godep.html

Others create their own script files to help Vendor their code and place the code in the right place inside the workspace. Then the go tools can be used directly.

If you don't want to use Vendoring, Godep also support pulling down specific revisions. This however does require multiple download calls for each of the dependencies when using "godep go get". There are a few other tools that support this option as well, such as Johnny Deps (https://github.com/VividCortex/johnny-deps) and RX (http://godoc.org/kylelemons.net/go/rx).

• This seems awfully brittle. What happens when someone deletes a GitHub repo you depended on? Coming from Java this is not a worry because once you publish a package to the Maven central repository it is always there. Also, if someone changes where they host their source code you have to update it in every single file that uses that package? That seems rather strange to me as well compared to having a list of packages configured one place in your project (like a build file) and imports that do not rely on the source code location. This has happened several times with dependencies I've used in Java and there it requires absolutely no changes, but seems really hard in Go. E.g. someone moves their project from Sourceforge to GitHub or from their personal GitHub to a GitHub organization. In Go that seems like a complete nightmare because you have to not only update every file in your project to point to the new location, but you also have to get all the libraries you depend on transitively that may depend on the same library to also make the same change.

Vendoring solves the problems you are describing. This is also what the Go team recommends. It is how Google internally handles their projects.

Appreciate the help. It looks like this group is trying to come up with better solutions for some of these problems, so I'm guessing these are all things you've thought about.

Yes, we are trying to identify the best practices and provide the community a tool to make this as easy and functional as possible. I recommend trying Vendoring first using Godep. I use this tool and it works great.

Thanks,

Ben

--
You received this message because you are subscribed to the Google Groups "Go Package Management" group.
To unsubscribe from this group and stop receiving emails from it, send an email to go-package-manag...@googlegroups.com.
To post to this group, send email to go-package...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Islan Dberry]

• You depend on source code instead of binaries?

Yes. 

• You always depend on head and there's no way to depend on a version of a library?

The standard Go tools do not provide dependency management. There are third party tools for managing dependencies. It's simple to implement a dependency management solution for a project using a shell script and version control system commands.

The standard 'go get' command does fetch head. This command is useful for experimentation. It's not intended to be used as a dependency management tool.  

• How does a library get updated?

This depends on the dependency management tool used.

The command 'go get -u' updates dependencies to head recursively.

• This seems awfully brittle.

With the exception of the 'go get' command, the standard Go tools do not map a package import path to a network location. For example, if I fork http://github.com/gorllla/mux to http://github.com/islandberry/mux, modify my fork and clone the fork to $GOPATH/src/github/gorrilla/mux, then all of the standard Go tools will happily use my version of the mux package.

Some of the third party dependency management tools use vendoring to isolate projects from changes to their upstream dependencies.

The path used to publish a package does not need to include the version control service used to publish the package. See the use of meta tags at http://golang.org/cmd/go/#hdr-Remote_import_path_syntax.

All that said, it is disruptive if people move their projects around.

[Ben McCann]

Thanks for the replies. This is becoming much clearer to me now. One follow up... I'm not familiar with the term "vendoring". Could you describe what this means or point me at a resource for it?

Thanks!

-Ben

--
You received this message because you are subscribed to a topic in the Google Groups "Go Package Management" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/go-package-management/bTWi4KanW8M/unsubscribe.
To unsubscribe from this group and all its topics, send an email to go-package-manag...@googlegroups.com.

To post to this group, send email to go-package...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
about.me/benmccann

[Islan Dberry]

On Wednesday, November 20, 2013 9:51:20 PM UTC-8, Ben McCann wrote:

Thanks for the replies. This is becoming much clearer to me now. One follow up... I'm not familiar with the term "vendoring". Could you describe what this means or point me at a resource for it?

The term comes from the Ruby On Rails vendor directory.

In the Go world, "vendoring" refers to copying a project's dependencies to a location owned by the project.

[chris....@gmail.com]

At the risk of side tracking the conversation, I think the concept of vendoring predates Ruby On Rails. According to Wikipedia[1] Ruby On Rails as an open source project dates back to July 2004.

I have found descriptions of the concept of vendoring, however, as old as 1998 in "Streamed Lines: Branching Patterns for Parallel Software Development"[2]. Although this source calls the pattern "Third Party Codeline" the word "vendor" appears 18 times in the description and the illustrative diagram uses "vendor" as the name of the branch.

In addition, the Subversion Book for Subversion v1.0 (2002) has a section describing the practice of maintaining vendor branches.[3]

These last two references also provide a very detailed explanation of the reasons for vendoring that should easily suffice to answer the question, "What is vendoring?". So hopefully I have added value beyond the etymological debate.

[1] http://en.wikipedia.org/wiki/Ruby_on_Rails#History
[2] http://www.bradapp.com/acme/branching/branch-structs.html#ThirdPartyLine
[3] http://svnbook.red-bean.com/en/1.0/svn-book.html#svn-ch-7-sect-4

[Nicolas Grilly]

On Thursday, November 21, 2013 7:15:29 AM UTC+1, Islan Dberry wrote:

The term comes from the Ruby On Rails vendor directory.

The term finds its real origin, not in computer science, but in logistics :)

According to Wikipedia:
A vendor, or a supplier, in a supply chain is an enterprise that contributes goods or services in a supply chain. Generally, a supply chain vendor manufactures inventory/stock items and sells them to the next link in the chain.

http://en.wikipedia.org/wiki/Vendor

[Islan Dberry]

Thank you for the historical references. I did not know that the vendor directory name predated Rails. As far as I now, the use of "vendor" as a verb emerged in the Ruby community.

[Dave Cheney]

I cannot find any references offhand, but the term may extend back to CVS. I have a foggy revolution that CVS provided a way to overlay your revisions on top of a 3rd parties, the vendor, in a way that allows you to update the vendor code *underneath* your source tree. 

I'm probably remembering this wrong, I don't think this feature was very popular. 

On 23 Nov 2013, at 10:18, Islan Dberry <island...@gmail.com> wrote:

Thank you for the historical references. I did not know that the vendor directory name predated Rails. As far as I now, the use of "vendor" as a verb emerged in the Ruby community.

--

You received this message because you are subscribed to the Google Groups "Go Package Management" group.

To unsubscribe from this group and stop receiving emails from it, send an email to go-package-manag...@googlegroups.com.