the go command
Russ Cox
automates the downloading, building, installation, and testing of Go
packages and commands. It replaces both goinstall and make. I
thought it would help to say a little bit about why we wrote a new
command, what it is, what it's not, and how to use it.
Motivation
You might have seen early Go talks in which Rob jokes that the idea
for Go arose while waiting for a large Google server to compile. That
really was the motivation for Go: to build a language that worked well
for building the large software that Google writes and runs. It was
clear from the start that such a language must provide a way to
express dependencies between code libraries clearly, hence the package
grouping and the explicit import blocks. It was also clear from the
start that you might want arbitrary syntax for describing the code
being imported; this is why import paths are string literals.
An explicit goal for Go from the beginning was to be able to build Go
code using only the information found in the source itself, not
needing to write a makefile or one of the many modern replacements for
makefiles. If Go needed a configuration file to explain how to build
your program, then Go would have failed.
At first, there was no Go compiler, and the initial development
focused on building one and then building libraries for it. For
expedience, we postponed the automation of building Go code by using
make and writing makefiles. When compiling a single package involved
multiple invocations of the Go compiler, we even used a program to
write the makefiles for us. You can find it if you dig through the
repository history.
The purpose of the new go command is our return to this ideal, that Go
programs should compile without configuration or additional effort on
the part of the developer beyond writing the necessary import
statements.
Configuration versus convention
The way to achieve the simplicity of a configuration-free system is to
establish conventions. The system works only to the extent that the
convention is followed. When we first launched Go, many people
published packages that had to be installed in certain places, under
certain names, using certain build tools, in order to be used. That's
understandable: that's the way it works in most other languages. Over
the last few years we have consistently reminded people about the
goinstall command and its conventions: first, that the import path is
derived in a known way from the URL of the source code; second, that
that the place to store the sources in the local file system is
derived in a known way from the import path; third, that each
directory in a source tree corresponds to a single package; and
fourth, that the package is built using only information in the source
code. Today, the vast majority of packages I see mentioned are
'goinstallable': they follow the conventions. The Go ecosystem is
simpler and more powerful for it.
We received many requests to allow a makefile in a package directory
to override goinstall's defaults, to provide just a little extra
configuration beyond what's in the source code. But that would have
introduced new rules. Because we did not accede to such requests, we
were able to write the new go command and eliminate our use of make or
any other build system.
It is important to understand that the go command is not a general
build tool. It cannot be configured and it does not attempt to build
anything but Go packages. These are important simplifying
assumptions: they simplify not only the implementation but also, more
important, the use of the tool itself.
Go's conventions
The go command retains the conventions established by the goinstall command.
First, the import path is derived in an known way from the URL of the
source code. For Bitbucket, GitHub, Google Code, and Launchpad, the
root directory of the repository is identified by the repository's
main URL, without the http:// prefix. Subdirectories are named by
adding to that path. For example, the supplemental networking
libraries for Go are obtained by running
hg clone http://code.google.com/p/go.net
and thus the import path for the root directory of that repository is
"code.google.com/p/go.net". The websocket package is stored in a
subdirectory, so its import path is
"code.google.com/p/go.net/websocket".
These paths are on the long side, but in exchange we get an
automatically managed name space for import paths and the ability for
a tool like the go command to look at an unfamiliar import path and
deduce where to obtain the source code.
Second, the place to store sources in the local file system is derived
in an known way from the import path. Specifically, the first choice
is $GOPATH/src/<import-path>. If $GOPATH is unset, the go command
will fall back to storing source code alongside the standard Go
packages, in $GOROOT/src/pkg/<import-path>. If $GOPATH is set to a
list of paths, the go command tries <dir>/src/<import-path> for each
of the directories in that list.
Each of those trees contains, by convention, a top-level directory
named "bin", for holding compiled executables, and a top-level
directory named "pkg", for holding compiled packages that can be
imported, and the "src" directory, for holding package source files.
Imposing this structure lets us keep each of these directory trees
self-contained: the compiled form and the sources are always near each
other.
These naming conventions also let us work in the reverse direction,
from a directory name to its import path. This mapping is important
for many of the go command's commands, as we'll see below.
Third, each directory in a source tree corresponds to a single
package. By restricting a directory to a single package, we don't have
to create hybrid import paths that specify first the directory and
then the package within that directory. Also, most file management
tools and UIs work on directories as fundamental units. Tying the
fundamental Go unit - the package - to file system structure means
that file system tools become Go package tools. Copying, moving, or
deleting a package corresponds to copying, moving, or deleting a
directory.
Fourth, each package is built using only the information present in
the source files. This makes it much more likely that the tool will
be able to adapt to changing build environments and conditions. For
example, if we allowed extra configuration like compiler flags or
command line recipes, then that configuration would need to be updated
each time the build tools changed; it would also be inherently tied
to the use of a specific toolchain.
Getting started with the go command
Finally, a quick tour of how to use the go command. Assuming you want
to keep your source code separate from the Go distribution source
tree, the first step is to set $GOPATH, the one piece of global
configuration that the go command needs. The $GOPATH can be a list of
directories, but by far the most common usage should be to set it to a
single directory. In particular, you do not need a separate entry in
$GOPATH for each of your projects. One $GOPATH can support many
projects.
Here’s an example. Let’s say we decide to keep our Go code in the
directory $HOME/mygo. We need to create that directory and set
$GOPATH accordingly.
$ mkdir $HOME/mygo
$ export GOPATH=$HOME/mygo
$
Into this directory, we now add some source code. Suppose we want to
use the indexing library from the codesearch project along with a
left-leaning red-black tree. We can install both with the “go get”
subcommand:
$ go get code.google.com/p/codesearch/index
$ go get github.com/petar/GoLLRB/llrb
$
Both of these projects are now downloaded and installed into our
$GOPATH directory. The one tree now contains the two directories
src/code.google.com/p/codesearch/index/ and
src/github.com/petar/GoLLRB/llrb/, along with the compiled packages
(in pkg/) for those libraries and their dependencies.
Because we used version control systems (Mercurial and Git) to check
out the sources, the source tree also contains the other files in the
corresponding repositories, such as related packages. The “go list”
subcommand lists the import paths corresponding to its arguments, and
the pattern “./...” means start in the current directory (“./”) and
find all packages below that directory (“...”):
$ go list ./...
code.google.com/p/codesearch/cmd/cgrep
code.google.com/p/codesearch/cmd/cindex
code.google.com/p/codesearch/cmd/csearch
code.google.com/p/codesearch/index
code.google.com/p/codesearch/regexp
code.google.com/p/codesearch/sparse
github.com/petar/GoLLRB/example
github.com/petar/GoLLRB/llrb
$
We can also test those packages:
$ go test ./...
? code.google.com/p/codesearch/cmd/cgrep [no test files]
? code.google.com/p/codesearch/cmd/cindex [no test files]
? code.google.com/p/codesearch/cmd/csearch [no test files]
ok code.google.com/p/codesearch/index 0.239s
ok code.google.com/p/codesearch/regexp 0.021s
? code.google.com/p/codesearch/sparse [no test files]
? github.com/petar/GoLLRB/example [no test files]
ok github.com/petar/GoLLRB/llrb 0.231s
$
If a go subcommand is invoked with no paths listed, it operates on the
current directory:
$ cd $GOPATH/src/code.google.com/p/codesearch/regexp
$ go list
code.google.com/p/codesearch/regexp
$ go test -v
=== RUN TestNstateEnc
--- PASS: TestNstateEnc (0.00 seconds)
=== RUN TestMatch
--- PASS: TestMatch (0.01 seconds)
=== RUN TestGrep
--- PASS: TestGrep (0.00 seconds)
PASS
ok code.google.com/p/codesearch/regexp 0.021s
$ go install
$
That “go install” subcommand installs the latest copy of the package
into the pkg directory. Because the go command can analyze the
dependency graph, “go install” also installs any packages that this
package imports but that are out of date, recursively.
Notice that “go install” was able to determine the name of the import
path for the package in the current directory, because of the
convention for directory naming. It would be a little more convenient
if we could pick the name of the directory where we kept source code,
and we probably wouldn't pick such a long name, but that ability would
require additional configuration and complexity in the tool. Typing an
extra directory name or two is a small price to pay for the increased
simplicity and power.
As the example shows, it’s fine to work with packages from many
different projects at once within a single $GOPATH root directory.
Limitations
As mentioned above, the go command is not a general-purpose build
tool. In particular, it does not have any facility for generating Go
source files during a build. Instead, if you want to use a tool like
yacc or the protocol buffer compiler, you will need to write a
makefile (or a configuration file for the build tool of your choice)
to generate the Go files and then check those generated source files
into your repository. This is more work for you, the package author,
but it is significantly less work for your users, who can use “go get”
without needing don't need to obtain and build any additional tools.
More information
There is not yet a full manual for the go command. For more
information, run “go help” or visit http://tip.golang.org/cmd/go/.
Peter Bourgon
are helpful. One point, however:
> Here’s an example. Let’s say we decide to keep our Go code in the
> directory $HOME/mygo.
The implication here is that "my Go code" is somehow distinct from "my
C code" (or whatever). This implication is false: I don't draw
partitions in this way. I keep "my Go code" with all my other code:
cloned into ~/src. I expect Go packages and binaries to be built "in
place" in those cloned subdirectories, just like all my other
software/language stacks. This expectation is reinforced by the fact
that the Go toolchain produces [nearly] pure, statically-linked
binaries, requiring no runtime framework. And, "gb" manages to satisfy
it. (It does so by resolving import dependencies into the $GOROOT
tree, which is fine by me.)
It could be that I'm being stubborn, or my use-case is atypical. If
so, I hope some people on this list will point it out and (gently)
correct me :) But, at least for now, I'm not sure that I'm all that
weird in having these expectations. The "go" tool feels more like the
weird element in the equation.
Gustavo Niemeyer
> thought it would help to say a little bit about why we wrote a new
> command, what it is, what it's not, and how to use it.
Thanks for the great write up Russ.
I'll take this chance to highlight one minor point:
The new "goinstall" is "go get".
The "install" subcommand of the "go" tool will build and install a
local package only.
--
Gustavo Niemeyer
http://niemeyer.net
http://niemeyer.net/plus
http://niemeyer.net/twitter
http://niemeyer.net/blog
-- I'm not absolutely sure of anything.
Gustavo Niemeyer
> The implication here is that "my Go code" is somehow distinct from "my
> C code" (or whatever). This implication is false: I don't draw
> partitions in this way. I keep "my Go code" with all my other code:
> cloned into ~/src. I expect Go packages and binaries to be built "in
> place" in those cloned subdirectories, just like all my other
I use ~/src as well, and I have GOPATH=$HOME. Works fine.
> It could be that I'm being stubborn, or my use-case is atypical. If
> so, I hope some people on this list will point it out and (gently)
> correct me :)
I don't think your use case is atypical, and I don't see why you might
be called stubborn in this regard. Just use ~/src, and set up the go
tool so it uses it.
> But, at least for now, I'm not sure that I'm all that
> weird in having these expectations. The "go" tool feels more like the
> weird element in the equation.
I disagree here, though. You seem to like gb. Have you realized that
one of the key reasons why you can use another build tool which is
non-standard with pretty much every single package out there is
because those packages don't depend on a build tool at all? Try to use
an autoconf package without autoconf, or a make-based package without
make, or an ant-based package without ant. The concept behind the go
tool is what allows you to easily use go packages without the tool
itself.
Peter Bourgon
> On Tue, Feb 7, 2012 at 15:13, Peter Bourgon <pe...@bourgon.org> wrote:
>> The implication here is that "my Go code" is somehow distinct from "my
>> C code" (or whatever). This implication is false: I don't draw
>> partitions in this way. I keep "my Go code" with all my other code:
>> cloned into ~/src. I expect Go packages and binaries to be built "in
>> place" in those cloned subdirectories, just like all my other
>
> I use ~/src as well, and I have GOPATH=$HOME. Works fine.
Except (as mentioned in IRC) this populates my ~/bin and ~/pkg subdirs
with artifacts, rather than dropping them in place. In my view, this
is undesirable.
> I disagree here, though. You seem to like gb. Have you realized that
> one of the key reasons why you can use another build tool which is
> non-standard with pretty much every single package out there is
> because those packages don't depend on a build tool at all? Try to use
> an autoconf package without autoconf, or a make-based package without
> make, or an ant-based package without ant. The concept behind the go
> tool is what allows you to easily use go packages without the tool
> itself.
Oh, don't get me wrong: I'm totally in favor of convention over
configuration. And for the most part, I'm totally fine with the
conventions that goinstall (and/or the go tool) established.
My problem is in the details: specifically, that the go tool both
requires extra convention ($GOPATH) and imposes extra restrictions
(mandating a home for "my Go code"; binaries necessarily to
$GOPATH/bin; packages necessarily to $GOPATH/pkg), both of which gb
manages to avoid.
I'm trying to figure out the cost that gb pays to achieve this (in my
view) clear usability win, but I'm not seeing it.
Gustavo Niemeyer
> I'm trying to figure out the cost that gb pays to achieve this (in my
> view) clear usability win, but I'm not seeing it.
It's just a different taste. I personally appreciate the fact that the
source tree remains clean after a build, and also that I can drop
binaries onto my ~/bin with "go get" or "go install". To me that's a
win, but it may be a different convention than the one you prefer. The
only way for both camps to be happy would be with configuration, and
we both seem to agree that convention wins over configuration.
Peter Bourgon
> source tree remains clean after a build, and also that I can drop
> binaries onto my ~/bin with "go get" or "go install". To me that's a
> win, but it may be a different convention than the one you prefer. The
> only way for both camps to be happy would be with configuration, and
> we both seem to agree that convention wins over configuration.
All else being equal, the solution which requires no extra environment
variables should be preferred.
Aram Hăvărneanu
> I use ~/src as well, and I have GOPATH=$HOME. Works fine.
Me too, I already had all my code, Go or not, in ~/src. GOPATH=$HOME
is perfect for me.
Peter Bourgon wrote:
> Except (as mentioned in IRC) this populates my ~/bin and ~/pkg subdirs
> with artifacts, rather than dropping them in place. In my view, this
> is undesirable.
Go putting stuff in ~/bin is a very nice bonus to me, I already had
~/bin added to my $PATH, so I didn't have to do anything extra to
accommodate myself to the new tool . ~/pkg wasn't there, but it's just
a directory I usually ignore, why is this an issue? It has to go
somewhere, I don't mind it in my home.
--
Aram Hăvărneanu
Miek Gieben
> Recent weekly snapshots have a new command, called “go”, that
> automates the downloading, building, installation, and testing of Go
> packages and commands. It replaces both goinstall and make. I
> thought it would help to say a little bit about why we wrote a new
> command, what it is, what it's not, and how to use it.
Thanks. This really helps.
> Limitations
<SNIP>
> into your repository. This is more work for you, the package author,
> but it is significantly less work for your users, who can use “go get”
> without needing don't need to obtain and build any additional tools.
I will be watching the src/cmd/goyacc directory to see how that will
actually work in that case :)
grtz Miek
Stefan Nilsson
Paul Borman
Paul Borman
Stefan Nilsson
Peter Bourgon
> I use ~/src as well, and I have GOPATH=$HOME. Works fine.
Beyond populating ~/bin and ~/pkg, this has the additional restriction
of making the "natural" home for my projects the tail end of a rather
long chain of subdirectories. For example, my project "goop" moves
from the logical home ~/src/goop to
~/src/github.com/peterbourgon/goop. That's totally unnatural, and not
something I'm keen to abide.
I remain open to arguments that the go tool's way is better. What I'm
less enthusiastic about is the notion that it's somehow no more
difficult, or not meaningfully different, than gb's way. The
convention of the go tool—ie. its mandated structure—imposes
significant cognitive load on the developer when compared to "the
competition" (viz. gb), and for no concrete reason or benefit that I
can discern.
(At least, this is my opinion.)
John Asmuth
On Tuesday, February 7, 2012 1:02:28 PM UTC-5, Peter Bourgon wrote:
I'm trying to figure out the cost that gb pays to achieve this (in my
view) clear usability win, but I'm not seeing it.
Steve McCoy
Paul Borman
dhconnelly
Paul Borman
Gustavo Niemeyer
On Tue, Feb 7, 2012 at 16:40, Stefan Nilsson
<trollerip...@gmail.com> wrote:
> Sure, you could put your test code in the same package. But common practice
> seems to be to put the test code for package x in package x_test. For
> example:
Indeed. The go tool continues to support the common practice of having
a test package with the prefix _test within the same directory as the
actual package. Nothing changed in that regard.
Kyle Lemons
If you put them in the same directory, it seems that you need to give the full path for x within the x_test package. That's inconvenient if you want to move your code.
Yohann Coppel
Russ Cox
> I may be wrong, but I don't think you *have* to use the
> src/github.com/peterbourgon/goop directory to write your code.
>
> A first option is put your code is ~/src/goop/, then symlink
> ~/src/github.com/peterbourgon/goop to ~/src/goop, and use "github.com/...."
> as import path. That would solve everything.
cd ~/src/goop
go install -v
See where it gets installed.
Russ
Yohann Coppel
Yohann Coppel
Paul Ruane
As there are conventions that need to be followed for the tooling to
operate as desired, do you think it would be useful (obviously not for
Go 1) for the 'go' command to have a facility for spitting out a basic
project structure that follows these conventions (a la Maven
archetypes or the Ruby on Rails 'rails new')?
E.g.
> export GOPATH=$HOME/mygo
> go create helloworld
>
> read more »
Stefan Nilsson
If you put them in the same directory, it seems that you need to give the full path for x within the x_test package. That's inconvenient if you want to move your code.The _test package is only intended to break dependency cycles incurred by test code. Since the _test files are ignored, they don't count as another package normally, and go test understands them.
Kyle Lemons
Adam Logghe
Kyle Lemons
How are library versions handled? I would like to pin remote libraries in github at a particular version for example.
John Asmuth
Adam Logghe
John Asmuth
Andrew Gerrand
> I'm probably wrong about this, from reading other posts. Perhaps I should
> experiment before talking.
No, you're quite right. From "go help get":
The -u flag instructs get to use the network to update the named packages
and their dependencies. By default, get uses the network to check out
missing packages but does not use it to look for updates to existing packages.
Andrew
Nickos Ventouras
> > I use ~/src as well, and I have GOPATH=$HOME. Works fine.
>
> with artifacts, rather than dropping them in place. In my view, this
> is undesirable.
And also no thoughts at all about version management? It would be a
disappointment to have to use yet another tool like "rvm" and/or
"virtualenv" in the future because "go" wouldn't cut it.
Jessta
> And also no thoughts at all about version management? It would be a
> disappointment to have to use yet another tool like "rvm" and/or
> "virtualenv" in the future because "go" wouldn't cut it.
Static linking means that this is only an issue if you happen to be
developing multiple applications that require different versions of a
library, which is unlikely.
virtualenv and rvm are symptoms of the problems that dynamic linking creates.
--
=====================
http://jessta.id.au
Gustavo Niemeyer
> And also no thoughts at all about version management? It would be a
> disappointment to have to use yet another tool like "rvm" and/or
> "virtualenv" in the future because "go" wouldn't cut it.
The foundation has been put in place 10 minutes ago. Let's wait until
it dries so we can build on it.
Nikos Ventouras
> On Wed, Feb 8, 2012 at 1:31 PM, Nickos Ventouras <mit...@gmail.com> wrote:
>> And also no thoughts at all about version management? It would be a
>> disappointment to have to use yet another tool like "rvm" and/or
>> "virtualenv" in the future because "go" wouldn't cut it.
>
> Static linking means that this is only an issue if you happen to be
> developing multiple applications that require different versions of a
> library, which is unlikely.
With libraries in a state of flux, as they especially are for a new language, that's actually a quite possible scenario.
A common case is one wanting to keep working on an older version of his app that uses an older version of a lib
(say, to just add bugfixes, etc). Consider supporting the 1.x and 2.x versions of a shipped product. Is it far fetched to
consider such uses?
Besides, that's just about linking. What about how the binaries are installed? Can someone use both version 1.x
and version 2.x of the same "go" installable app?
@Gustavo Niemeyer <