You might have seen early Go talks in which Rob jokes that the idea
for Go arose while waiting for a large Google server to compile. That
really was the motivation for Go: to build a language that worked well
for building the large software that Google writes and runs. It was
clear from the start that such a language must provide a way to
express dependencies between code libraries clearly, hence the package
grouping and the explicit import blocks. It was also clear from the
start that you might want arbitrary syntax for describing the code
being imported; this is why import paths are string literals.
An explicit goal for Go from the beginning was to be able to build Go
code using only the information found in the source itself, not
needing to write a makefile or one of the many modern replacements for
makefiles. If Go needed a configuration file to explain how to build
your program, then Go would have failed.
At first, there was no Go compiler, and the initial development
focused on building one and then building libraries for it. For
expedience, we postponed the automation of building Go code by using
make and writing makefiles. When compiling a single package involved
multiple invocations of the Go compiler, we even used a program to
write the makefiles for us. You can find it if you dig through the
The purpose of the new go command is our return to this ideal, that Go
programs should compile without configuration or additional effort on
the part of the developer beyond writing the necessary import
Configuration versus convention
The way to achieve the simplicity of a configuration-free system is to
establish conventions. The system works only to the extent that the
convention is followed. When we first launched Go, many people
published packages that had to be installed in certain places, under
certain names, using certain build tools, in order to be used. That's
understandable: that's the way it works in most other languages. Over
the last few years we have consistently reminded people about the
goinstall command and its conventions: first, that the import path is
derived in a known way from the URL of the source code; second, that
that the place to store the sources in the local file system is
derived in a known way from the import path; third, that each
directory in a source tree corresponds to a single package; and
fourth, that the package is built using only information in the source
code. Today, the vast majority of packages I see mentioned are
'goinstallable': they follow the conventions. The Go ecosystem is
simpler and more powerful for it.
We received many requests to allow a makefile in a package directory
to override goinstall's defaults, to provide just a little extra
configuration beyond what's in the source code. But that would have
introduced new rules. Because we did not accede to such requests, we
were able to write the new go command and eliminate our use of make or
any other build system.
It is important to understand that the go command is not a general
build tool. It cannot be configured and it does not attempt to build
anything but Go packages. These are important simplifying
assumptions: they simplify not only the implementation but also, more
important, the use of the tool itself.
The go command retains the conventions established by the goinstall command.
First, the import path is derived in an known way from the URL of the
source code. For Bitbucket, GitHub, Google Code, and Launchpad, the
root directory of the repository is identified by the repository's
main URL, without the http:// prefix. Subdirectories are named by
adding to that path. For example, the supplemental networking
libraries for Go are obtained by running
hg clone http://code.google.com/p/go.net
and thus the import path for the root directory of that repository is
"code.google.com/p/go.net". The websocket package is stored in a
subdirectory, so its import path is
These paths are on the long side, but in exchange we get an
automatically managed name space for import paths and the ability for
a tool like the go command to look at an unfamiliar import path and
deduce where to obtain the source code.
Second, the place to store sources in the local file system is derived
in an known way from the import path. Specifically, the first choice
is $GOPATH/src/<import-path>. If $GOPATH is unset, the go command
will fall back to storing source code alongside the standard Go
packages, in $GOROOT/src/pkg/<import-path>. If $GOPATH is set to a
list of paths, the go command tries <dir>/src/<import-path> for each
of the directories in that list.
Each of those trees contains, by convention, a top-level directory
named "bin", for holding compiled executables, and a top-level
directory named "pkg", for holding compiled packages that can be
imported, and the "src" directory, for holding package source files.
Imposing this structure lets us keep each of these directory trees
self-contained: the compiled form and the sources are always near each
These naming conventions also let us work in the reverse direction,
from a directory name to its import path. This mapping is important
for many of the go command's commands, as we'll see below.
Third, each directory in a source tree corresponds to a single
package. By restricting a directory to a single package, we don't have
to create hybrid import paths that specify first the directory and
then the package within that directory. Also, most file management
tools and UIs work on directories as fundamental units. Tying the
fundamental Go unit - the package - to file system structure means
that file system tools become Go package tools. Copying, moving, or
deleting a package corresponds to copying, moving, or deleting a
Fourth, each package is built using only the information present in
the source files. This makes it much more likely that the tool will
be able to adapt to changing build environments and conditions. For
example, if we allowed extra configuration like compiler flags or
command line recipes, then that configuration would need to be updated
each time the build tools changed; it would also be inherently tied
to the use of a specific toolchain.
Getting started with the go command
Finally, a quick tour of how to use the go command. Assuming you want
to keep your source code separate from the Go distribution source
tree, the first step is to set $GOPATH, the one piece of global
configuration that the go command needs. The $GOPATH can be a list of
directories, but by far the most common usage should be to set it to a
single directory. In particular, you do not need a separate entry in
$GOPATH for each of your projects. One $GOPATH can support many
Here’s an example. Let’s say we decide to keep our Go code in the
directory $HOME/mygo. We need to create that directory and set
$ mkdir $HOME/mygo
$ export GOPATH=$HOME/mygo
Into this directory, we now add some source code. Suppose we want to
use the indexing library from the codesearch project along with a
left-leaning red-black tree. We can install both with the “go get”
Both of these projects are now downloaded and installed into our
$GOPATH directory. The one tree now contains the two directories
src/github.com/petar/GoLLRB/llrb/, along with the compiled packages
(in pkg/) for those libraries and their dependencies.
Because we used version control systems (Mercurial and Git) to check
out the sources, the source tree also contains the other files in the
corresponding repositories, such as related packages. The “go list”
subcommand lists the import paths corresponding to its arguments, and
the pattern “./...” means start in the current directory (“./”) and
find all packages below that directory (“...”):
$ go list ./...
We can also test those packages:
$ go test ./...
? code.google.com/p/codesearch/cmd/cgrep [no test files]
? code.google.com/p/codesearch/cmd/cindex [no test files]
? code.google.com/p/codesearch/cmd/csearch [no test files]
ok code.google.com/p/codesearch/index 0.239s
ok code.google.com/p/codesearch/regexp 0.021s
? code.google.com/p/codesearch/sparse [no test files]
? github.com/petar/GoLLRB/example [no test files]
ok github.com/petar/GoLLRB/llrb 0.231s
If a go subcommand is invoked with no paths listed, it operates on the
$ cd $GOPATH/src/code.google.com/p/codesearch/regexp
$ go list
$ go test -v
=== RUN TestNstateEnc
--- PASS: TestNstateEnc (0.00 seconds)
=== RUN TestMatch
--- PASS: TestMatch (0.01 seconds)
=== RUN TestGrep
--- PASS: TestGrep (0.00 seconds)
ok code.google.com/p/codesearch/regexp 0.021s
$ go install
That “go install” subcommand installs the latest copy of the package
into the pkg directory. Because the go command can analyze the
dependency graph, “go install” also installs any packages that this
package imports but that are out of date, recursively.
Notice that “go install” was able to determine the name of the import
path for the package in the current directory, because of the
convention for directory naming. It would be a little more convenient
if we could pick the name of the directory where we kept source code,
and we probably wouldn't pick such a long name, but that ability would
require additional configuration and complexity in the tool. Typing an
extra directory name or two is a small price to pay for the increased
simplicity and power.
As the example shows, it’s fine to work with packages from many
different projects at once within a single $GOPATH root directory.
As mentioned above, the go command is not a general-purpose build
tool. In particular, it does not have any facility for generating Go
source files during a build. Instead, if you want to use a tool like
yacc or the protocol buffer compiler, you will need to write a
makefile (or a configuration file for the build tool of your choice)
to generate the Go files and then check those generated source files
into your repository. This is more work for you, the package author,
but it is significantly less work for your users, who can use “go get”
without needing don't need to obtain and build any additional tools.
There is not yet a full manual for the go command. For more
information, run “go help” or visit http://tip.golang.org/cmd/go/.
> Here’s an example. Let’s say we decide to keep our Go code in the
> directory $HOME/mygo.
The implication here is that "my Go code" is somehow distinct from "my
C code" (or whatever). This implication is false: I don't draw
partitions in this way. I keep "my Go code" with all my other code:
cloned into ~/src. I expect Go packages and binaries to be built "in
place" in those cloned subdirectories, just like all my other
software/language stacks. This expectation is reinforced by the fact
that the Go toolchain produces [nearly] pure, statically-linked
binaries, requiring no runtime framework. And, "gb" manages to satisfy
it. (It does so by resolving import dependencies into the $GOROOT
tree, which is fine by me.)
It could be that I'm being stubborn, or my use-case is atypical. If
so, I hope some people on this list will point it out and (gently)
correct me :) But, at least for now, I'm not sure that I'm all that
weird in having these expectations. The "go" tool feels more like the
weird element in the equation.
Thanks for the great write up Russ.
I'll take this chance to highlight one minor point:
The new "goinstall" is "go get".
The "install" subcommand of the "go" tool will build and install a
local package only.
-- I'm not absolutely sure of anything.
I use ~/src as well, and I have GOPATH=$HOME. Works fine.
> It could be that I'm being stubborn, or my use-case is atypical. If
> so, I hope some people on this list will point it out and (gently)
> correct me :)
I don't think your use case is atypical, and I don't see why you might
be called stubborn in this regard. Just use ~/src, and set up the go
tool so it uses it.
> But, at least for now, I'm not sure that I'm all that
> weird in having these expectations. The "go" tool feels more like the
> weird element in the equation.
I disagree here, though. You seem to like gb. Have you realized that
one of the key reasons why you can use another build tool which is
non-standard with pretty much every single package out there is
because those packages don't depend on a build tool at all? Try to use
an autoconf package without autoconf, or a make-based package without
make, or an ant-based package without ant. The concept behind the go
tool is what allows you to easily use go packages without the tool
Except (as mentioned in IRC) this populates my ~/bin and ~/pkg subdirs
with artifacts, rather than dropping them in place. In my view, this
> I disagree here, though. You seem to like gb. Have you realized that
> one of the key reasons why you can use another build tool which is
> non-standard with pretty much every single package out there is
> because those packages don't depend on a build tool at all? Try to use
> an autoconf package without autoconf, or a make-based package without
> make, or an ant-based package without ant. The concept behind the go
> tool is what allows you to easily use go packages without the tool
Oh, don't get me wrong: I'm totally in favor of convention over
configuration. And for the most part, I'm totally fine with the
conventions that goinstall (and/or the go tool) established.
My problem is in the details: specifically, that the go tool both
requires extra convention ($GOPATH) and imposes extra restrictions
(mandating a home for "my Go code"; binaries necessarily to
$GOPATH/bin; packages necessarily to $GOPATH/pkg), both of which gb
manages to avoid.
I'm trying to figure out the cost that gb pays to achieve this (in my
view) clear usability win, but I'm not seeing it.
It's just a different taste. I personally appreciate the fact that the
source tree remains clean after a build, and also that I can drop
binaries onto my ~/bin with "go get" or "go install". To me that's a
win, but it may be a different convention than the one you prefer. The
only way for both camps to be happy would be with configuration, and
we both seem to agree that convention wins over configuration.
All else being equal, the solution which requires no extra environment
variables should be preferred.
Me too, I already had all my code, Go or not, in ~/src. GOPATH=$HOME
is perfect for me.
Peter Bourgon wrote:
> Except (as mentioned in IRC) this populates my ~/bin and ~/pkg subdirs
> with artifacts, rather than dropping them in place. In my view, this
> is undesirable.
Go putting stuff in ~/bin is a very nice bonus to me, I already had
~/bin added to my $PATH, so I didn't have to do anything extra to
accommodate myself to the new tool . ~/pkg wasn't there, but it's just
a directory I usually ignore, why is this an issue? It has to go
somewhere, I don't mind it in my home.
Thanks. This really helps.
> into your repository. This is more work for you, the package author,
> but it is significantly less work for your users, who can use “go get”
> without needing don't need to obtain and build any additional tools.
I will be watching the src/cmd/goyacc directory to see how that will
actually work in that case :)
> I use ~/src as well, and I have GOPATH=$HOME. Works fine.
Beyond populating ~/bin and ~/pkg, this has the additional restriction
of making the "natural" home for my projects the tail end of a rather
long chain of subdirectories. For example, my project "goop" moves
from the logical home ~/src/goop to
~/src/github.com/peterbourgon/goop. That's totally unnatural, and not
something I'm keen to abide.
I remain open to arguments that the go tool's way is better. What I'm
less enthusiastic about is the notion that it's somehow no more
difficult, or not meaningfully different, than gb's way. The
convention of the go tool—ie. its mandated structure—imposes
significant cognitive load on the developer when compared to "the
competition" (viz. gb), and for no concrete reason or benefit that I
(At least, this is my opinion.)
I'm trying to figure out the cost that gb pays to achieve this (in my
view) clear usability win, but I'm not seeing it.
On Tue, Feb 7, 2012 at 16:40, Stefan Nilsson
> Sure, you could put your test code in the same package. But common practice
> seems to be to put the test code for package x in package x_test. For
Indeed. The go tool continues to support the common practice of having
a test package with the prefix _test within the same directory as the
actual package. Nothing changed in that regard.
If you put them in the same directory, it seems that you need to give the full path for x within the x_test package. That's inconvenient if you want to move your code.
go install -v
See where it gets installed.
If you put them in the same directory, it seems that you need to give the full path for x within the x_test package. That's inconvenient if you want to move your code.The _test package is only intended to break dependency cycles incurred by test code. Since the _test files are ignored, they don't count as another package normally, and go test understands them.
How are library versions handled? I would like to pin remote libraries in github at a particular version for example.
No, you're quite right. From "go help get":
The -u flag instructs get to use the network to update the named packages
and their dependencies. By default, get uses the network to check out
missing packages but does not use it to look for updates to existing packages.