goinstall: an experiment in package installation

1630 views
Skip to first unread message

Russ Cox

unread,
Feb 24, 2010, 6:41:31 PM2/24/10
to golang-nuts
CL 224043, not yet sent for review, would add a new command
goinstall that can download packages into the local file system.
Rather than establish a central naming authority for Go packages
(witness the multiple mysqlgo packages), it uses import paths
that contain the names of common hosting sites. The full docs
are below.

Goinstall already works with standalone packages that
don't depend on other packages. For example, I can run:

goinstall github.com/hoisie/web.go.git
goinstall github.com/jacobsa/igo.git/set
goinstall go-avltree.googlecode.com/svn/trunk
goinstall mimeparse.googlecode.com/svn/trunk/go
goinstall gomatrix.googlecode.com/hg
goinstall golang.googlecode.com/hg/src/pkg/scanner

And then import any of those using the same paths.
If those had referred to other import paths of the same
form, goinstall would have downloaded the dependencies,
recursively. So in your own package, you can put
a line like import "gomatrix.googlecode.com/hg" and
then just run

goinstall my/package

which will notice the import line and download and install
the remote package. The first time goinstall downloads
and installs a public package, it pings the Go dashboard
to update a list of what people are using.
http://godashboard.appspot.com/package.

This is a request for comments on the general approach.
Please comment here, on the mailing list, rather than in the
codereview site. It will be easier to have a discussion in an
actual discussion forum.


-- documentation from goinstall/doc.go

Goinstall is an experiment in automatic package installation.
It installs packages, possibly downloading them from public version
control systems, and it maintains a list of public Go packages at
http://godashboard.appspot.com/packages.

Usage:
goinstall [flags] importpath...

Flags and default settings:
-dashboard=true tally public packages on godashboard.appspot.com
-update=false update already-downloaded packages
-v=false verbose operation

Goinstall installs each of the packages identified by import path on
the command line. It installs a package's prerequisites before
trying to install the package itself.

The source code for a package with import path foo/bar is expected
to be in the directory $GOROOT/src/pkg/foo/bar/. If the import
path refers to a code hosting site, goinstall will download it if necessary.
The recognized code hosting sites are:

GitHub (Git)

import "github.com/user/project.git"
import "github.com/user/project.git/sub/directory"

Google Code Project Hosting (Mercurial, Subversion)

import "project.googlecode.com/hg"
import "project.googlecode.com/hg/sub/directory"

import "project.googlecode.com/svn/trunk"
import "project.googlecode.com/svn/trunk/sub/directory"


If the directory (e.g., $GOROOT/src/pkg/bitbucket.org/user/project)
already exists and contains an appropriate checkout, goinstall will not
attempt to fetch updates. The -update flag changes this behavior,
causing goinstall to update all remote packages encountered during
the installation.

When downloading or updating, goinstall first looks for a tag or branch
named "release". If there is one, it uses that version of the code.
Otherwise it uses the default version selected by the version control
system (typically HEAD for git, tip for Mercurial).

After a successful download and installation of a publicly accessible
remote package, goinstall reports the installation to godashboard.appspot.com,
which increments a count associated with the package and the time
of its most recent installation. This mechanism powers the package list
at http://godashboard.appspot.com/packages, allowing Go programmers
to learn about popular packages that might be worth looking at.
The -dashboard=false flag disables this reporting.

By default, goinstall prints output only when it encounters an error.
The -v flag causes goinstall to print information about packages
being considered and installed.

Goinstall does not attempt to be a replacement for make.
Instead, it invokes "make install" after locating the package sources.
For local packages without a Makefile and all remote packages,
goinstall creates and uses a temporary Makefile constructed from
the import path and the list of Go files in the package.


FAQ:

• Why not use full URLs instead of hard-coding hosting sites?

Goinstall is an experiment, and it is easier if the experiment does
not involve changes to the compilers and godoc. The paths used
above double as file system paths, while full URLs do not.
Also, the import path must also convey both the location of
the data and the choice of version control system, so a full URL
would not suffice.

Using this notation does not precluding changing to other
notation once we have more experience or in other contexts.
For example, if Go were to run inside a browser it would make
sense for import lines to be URLs of individual files rather than
version control repositories.

• Why isn't my favorite hosting site supported?

Feel free to add it and send us a CL.
See http://golang.org/doc/contribute.html

• How can I use goinstall with a package I commit changes to?

Instead of having goinstall do the checkout automatically,
check out your package yourself into the same location where
goinstall would put it. You can use that as your working copy
and push to the world as appropriate. For example, if you
were working on github.com/hoisie/web.go, you check it out
into $GOROOT/src/pkg/github.com/hoisie/web.go.git
and then edit the files in that directory, pushing back to github
as needed. Goinstall will use the checked-out git repository
instead of trying to download a fresh one.

• What information is sent to and logged by the dashboard?

The source code for both goinstall and the dashboard are
in the Go repository:

$GOROOT/src/cmd/goinstall/download.go
$GOROOT/misc/dashboard/godashboard/package.py

Goinstall sends the package import path.
The dashboard increments a counter and sets the "most recently installed" time.
There is no other reporting or logging. The data recorded by the dashboard
is publicly visible at http://godashboard.appspot.com/package.

-- end documentation


Again, comments welcome, but please post them here rather
than on the code review site.

Thanks.
Russ

John Asmuth

unread,
Feb 24, 2010, 9:46:12 PM2/24/10
to golang-nuts
Pretty cool - what about the package name?

For instance (and I like that you used my package as one of your
examples)
import "gomatrix.googlecode.com/hg"

The package name is matrix, which is not reflected in the import
statement. Now, I understand that, when goinstall downloads it, it
will examine what it downloads and know the correct package name for
use in the code.

But consider the case when someone asks how to use matrices in go, and
someone in the IRC channel says, "oh, just use <import statement> at
the top of your file", referring to the import atatement above.

Then this hapless coder has no immediate way to see what the name of
the package is. Maybe "gomatrix"? But no, it's "matrix", which you can
only know by examining the gomatrix source.

This problem doesn't appear so much normally, because it is customary
to name the directory containing a package after that package.

Of course, if this hapless coder is slightly less hapless, he or she
can name the import. This obviously isn't an insurmountable problem.
Just one that can cause some confusion at first for those coders who
have trouble looking stuff up on their own (for whatever reason).

But this seems like a really useful idea, and a very slick way to keep
code-versions up-to-date.

Which makes me think... is there a way to refer to a certain version
of, for example, a googlecode hg project? What if your code works with
an older version, and breaks with a newer? For instance, it is
possible that some clever compiler optimizations would make some of
the methods in gomatrix pointless, and they could be removed. But what
about projects that used those now-removed methods? Being able to
refer to a specific and immutable version would be sweet, and the
default code be the most up-to-date.


- John

> to update a list of what people are using.http://godashboard.appspot.com/package.


>
> This is a request for comments on the general approach.
> Please comment here, on the mailing list, rather than in the
> codereview site.  It will be easier to have a discussion in an
> actual discussion forum.
>
> -- documentation from goinstall/doc.go
>
> Goinstall is an experiment in automatic package installation.
> It installs packages, possibly downloading them from public version

> control systems, and it maintains a list of public Go packages athttp://godashboard.appspot.com/packages.

> athttp://godashboard.appspot.com/packages, allowing Go programmers

> Seehttp://golang.org/doc/contribute.html


>
> • How can I use goinstall with a package I commit changes to?
>
> Instead of having goinstall do the checkout automatically,
> check out your package yourself into the same location where
> goinstall would put it.  You can use that as your working copy
> and push to the world as appropriate.  For example, if you
> were working on github.com/hoisie/web.go, you check it out
> into $GOROOT/src/pkg/github.com/hoisie/web.go.git
> and then edit the files in that directory, pushing back to github
> as needed.  Goinstall will use the checked-out git repository
> instead of trying to download a fresh one.
>
> • What information is sent to and logged by the dashboard?
>
> The source code for both goinstall and the dashboard are
> in the Go repository:
>
>     $GOROOT/src/cmd/goinstall/download.go
>     $GOROOT/misc/dashboard/godashboard/package.py
>
> Goinstall sends the package import path.
> The dashboard increments a counter and sets the "most recently installed" time.
> There is no other reporting or logging.  The data recorded by the dashboard

> is publicly visible athttp://godashboard.appspot.com/package.

Russ Cox

unread,
Feb 25, 2010, 12:58:55 AM2/25/10
to John Asmuth, golang-nuts
Thanks for the feedback.

> For instance (and I like that you used my package as one of your
> examples)
>  import "gomatrix.googlecode.com/hg"
>
> The package name is matrix, which is not reflected in the import
> statement. Now, I understand that, when goinstall downloads it, it
> will examine what it downloads and know the correct package name for
> use in the code.
>
> But consider the case when someone asks how to use matrices in go, and
> someone in the IRC channel says, "oh, just use <import statement> at
> the top of your file", referring to the import atatement above.
>
> Then this hapless coder has no immediate way to see what the name of
> the package is. Maybe "gomatrix"? But no, it's "matrix", which you can
> only know by examining the gomatrix source.

It's true; this is a problem. I think this is going to end up
being solved by common sense and convention. Maybe
gomatrix.googlecode.com/hg should call itself package gomatrix,
or maybe it should be in a subdirectory named matrix.
The subdirectory = package name rule is a good general
convention, but there are reasonable exceptions too.
For example, the mimeparse project has source code for
many different languages, and the Go code is in the go
subdirectory. It doesn't bother me that I write import
"mimeparse.googlecode.com/svn/trunk/go" to get package
mimeparse, and I'd much rather get package mimeparse
than package go. It's an open issue, but I think conventions
will go a long way.

> Which makes me think... is there a way to refer to a certain version
> of, for example, a googlecode hg project? What if your code works with
> an older version, and breaks with a newer? For instance, it is
> possible that some clever compiler optimizations would make some of
> the methods in gomatrix pointless, and they could be removed. But what
> about projects that used those now-removed methods? Being able to
> refer to a specific and immutable version would be sweet, and the
> default code be the most up-to-date.

Subversion encodes this info in the path, so you can always
import something other than trunk. In hg and git, there's no
direct support for specific versions but goinstall does prefer to
use the branch or tag named "release" over using git HEAD/hg tip.

In general the versioning problem is pretty tricky. If you must use
version 1 of a package and Michael must use version 2 and I want
to use both your packages, then I end up with both copies in my
binary. 6g can handle that; gccgo might need a little help.
Either way it's suboptimal to be linking every version in.
Goinstall nods at the versioning problem by preferring "release"
but it's by no means solved, and I don't know that there's
a magic bullet.

Russ

chris dollin

unread,
Feb 25, 2010, 2:47:18 AM2/25/10
to r...@golang.org, golang-nuts
On 24 February 2010 23:41, Russ Cox <r...@golang.org> wrote:

(WRT goinstall)


of its most recent installation.  This mechanism powers the package list
at http://godashboard.appspot.com/packages, allowing Go programmers
to learn about popular packages that might be worth looking at.

That link currently serves an empty page, no text, no HTML, nothing, not
even a Coming Soon; is that as expected?

--
Chris "allusive" Dollin

Russ Cox

unread,
Feb 25, 2010, 3:23:42 AM2/25/10
to chris dollin, golang-nuts
>> of its most recent installation.  This mechanism powers the package list
>> at http://godashboard.appspot.com/packages, allowing Go programmers
>> to learn about popular packages that might be worth looking at.
>
> That link currently serves an empty page, no text, no HTML, nothing, not
> even a Coming Soon; is that as expected?

Sorry, typo. The correct URL is http://godashboard.appspot.com/package.

Russ

Esko Luontola

unread,
Feb 25, 2010, 5:19:20 AM2/25/10
to golang-nuts
What about libraries which consist of multiple packages? For example
I'll split GoSpec (http://github.com/orfjackal/gospec) into multiple
packages as soon as I have a build tool which can handle building and
testing it easily (probably gobbler or igo). Then the users of GoSpec
will need to have the following imports in their test sources:

import "gospec"
import . "gospec/matchers"

And the gospec package in turn will depend on multiple internal
packages: "gospec/reporting", "gospec/foo" etc. All of those packages
are needed for the library to work.

andrey mirtchovski

unread,
Feb 25, 2010, 10:22:03 AM2/25/10
to golang-nuts
Two questions:

- How are you handling inclusions in the Makefile (and the NOTEST
option), or would those packages not be recompiled during "all.bash"?

- bitbucket is mentioned in your email, but not explicitly as working.
is it supported?

thanks

Russ Cox

unread,
Feb 25, 2010, 11:32:55 AM2/25/10
to andrey mirtchovski, golang-nuts
> - How are you handling inclusions in the Makefile (and the NOTEST
> option), or would those packages not be recompiled during "all.bash"?

downloaded Makefiles are ignored.
the package gets built using a standard-form makefile
generated from the list of files in the directory
(must only be one non-main package in the directory).

the packages are not recompiled by all.bash.

> - bitbucket is mentioned in your email, but not explicitly as working.
> is it supported?

i intended to add it but haven't for lack of a test case.

russ

Russ Cox

unread,
Feb 25, 2010, 11:35:36 AM2/25/10
to Esko Luontola, golang-nuts

you'd make those

import "github.com/orfjackal/gospec.git"
import "github.com/ofjackal/gospec.git/matchers"

and then the gospec package would say, internally

import "github.com/orfjackal/gospec.git/reporting"
import "github.com/orfjackal/gospec.git/foo"

when goinstall downloaded gospec for the first import,
it would see the others and build those subpackages
first. when working on the package locally, you'd just
work in the $GOROOT/src/pkg/github.com/orfjackal/gospec.git
directory, which would be a standard git checkout.

russ

Esko Luontola

unread,
Feb 25, 2010, 12:59:11 PM2/25/10
to golang-nuts
On Feb 25, 6:35 pm, Russ Cox <r...@golang.org> wrote:
> when goinstall downloaded gospec for the first import,
> it would see the others and build those subpackages
> first.  when working on the package locally, you'd just
> work in the $GOROOT/src/pkg/github.com/orfjackal/gospec.git
> directory, which would be a standard git checkout.

What about handling version dependencies? Let's say that I'm
developing a library and I have N projects using that library.

When working on the library itself, I edit the development version of
the library. But at the same time, when working on those N other
projects, they must use the release version of the library. And when
there are incompatible changes to the library, each of the N projects
must use a specific version of the library.

So there is need for being able to specify the version number of the
library. If the version number is part of the import statement, then
when a library is upgraded, one must change the version number in
possibly hundreds of files. This smells like http://en.wikipedia.org/wiki/Shotgun_surgery

And anyways I don't like being forced to put all the sources under
$GOROOT. Also having to install all packages there (as it's done
today) feels like a bad idea. I think that a long-term solution would
be a better build tool, where you can configure the dependencies and
versions in one file per project (similar to Maven et al).

Russ Cox

unread,
Feb 25, 2010, 1:06:38 PM2/25/10
to Esko Luontola, golang-nuts
> What about handling version dependencies? Let's say that I'm
> developing a library and I have N projects using that library.
>
> When working on the library itself, I edit the development version of
> the library. But at the same time, when working on those N other
> projects, they must use the release version of the library. And when
> there are incompatible changes to the library, each of the N projects
> must use a specific version of the library.

So don't do that. I answered this earlier on the thread:
versioning is a tough problem and we're not trying to solve it today.

> So there is need for being able to specify the version number of the
> library. If the version number is part of the import statement, then
> when a library is upgraded, one must change the version number in
> possibly hundreds of files. This smells like http://en.wikipedia.org/wiki/Shotgun_surgery

Do you have a proposal for how to specify the version number
that a particular file needs without maintaining that information?

> And anyways I don't like being forced to put all the sources under
> $GOROOT. Also having to install all packages there (as it's done
> today) feels like a bad idea. I think that a long-term solution would
> be a better build tool, where you can configure the dependencies and
> versions in one file per project (similar to Maven et al).

Configuration = complexity.
Some may be warranted, but I don't want to just take what everyone
else does, because it's too much manual effort. In contrast, goinstall
is automatic if you're willing to live within the conventions.

I do think there will be a sort of unified go path eventually (today,
godoc, 6g, and 6l each have their own idea of what that means),
but we're not there yet. One step at a time.

Russ

Michael Hoisie

unread,
Feb 25, 2010, 1:28:08 PM2/25/10
to golang-nuts
I think this is a great start but I'm worried this won't scale well.

First of all, as someone pointed out, there will probably be naming
conflicts. Github doesn't enforce any kind of unique naming, so you
could have packages overwrite each other. Neither does Google Code.

The second issue is dependency management. Right now most Go packages
are simple and only depend on the core library. However, this is
likely to change as libraries stabilize and become more useful. There
isn't any meta-data on Github or Google code about dependencies.

Why not model the system more closely to rubygems (http://
www.rubygems.org). Rubygems has a structured package format with a
small metadata file, so it can automatically handle versioning and
dependencies.

There could be some central repo of packages on http://godashboard.appspot.com/packages,
so global naming could be enforced. To submit a package to this site,
you would have to conform to the go package format. It would be more
strict, but I feel the system would be more extendable than just using
github or google code.

- Mike

On Feb 25, 10:06 am, Russ Cox <r...@golang.org> wrote:
> > What about handling version dependencies? Let's say that I'm
> > developing a library and I have N projects using that library.
>
> > When working on the library itself, I edit the development version of
> > the library. But at the same time, when working on those N other
> > projects, they must use the release version of the library. And when
> > there are incompatible changes to the library, each of the N projects
> > must use a specific version of the library.
>
> So don't do that.  I answered this earlier on the thread:
> versioning is a tough problem and we're not trying to solve it today.
>
> > So there is need for being able to specify the version number of the
> > library. If the version number is part of the import statement, then
> > when a library is upgraded, one must change the version number in

> > possibly hundreds of files. This smells likehttp://en.wikipedia.org/wiki/Shotgun_surgery

Russ Cox

unread,
Feb 25, 2010, 1:40:05 PM2/25/10
to Michael Hoisie, golang-nuts
Thanks for your comments. I think both issues you raised are
already address by the system:

> First of all, as someone pointed out, there will probably be naming
> conflicts. Github doesn't enforce any kind of unique naming, so you
> could have packages overwrite each other. Neither does Google Code.

But they do. I can't go to Github and create a package named
github.com/hoisie/web.go.git. That's what the package gets
installed and imported as. There might be packages with two
different import paths that both claim to be package web.
That's fine: Go explicitly allows that, and knows enough to keep
them separate.

> The second issue is dependency management. Right now most Go packages
> are simple and only depend on the core library. However, this is
> likely to change as libraries stabilize and become more useful. There
> isn't any meta-data on Github or Google code about dependencies.

But there is: the source files contain import statements.
Goinstall looks at them.

For example, goinstall github.com/jacobsa/igo.git/parse fails today,
because parse/parse.go tries to import "igo/set", which doesn't exist.
But if Aaron changed his program to use the goinstall conventions
(and I hope people will), it would import "github.com/jacobsa/igo.git/set",
and goinstall would gladly download and install that first.

Yes, we could set up a CPAN or a RubyGems or whatever,
but that requires a centralized authority giving out import paths
and is more mechanism. Goinstall is zero configuration
without any need for central coordination.

Russ

Bob Cunningham

unread,
Feb 25, 2010, 4:26:54 PM2/25/10
to r...@golang.org, golang-nuts
1. "goinstall"? I think it should have been called "goget" from the
get-go. ;^)

2. I'm concerned that package source trees would need to include a new
non-Go file whenever external packages are used: The "goinstall" script.

Would the build system be extended to automatically run such scripts?
Not ideal...

Instead, could goinstall be trivially internalized by the language, to
extend "import" to run "goinstall" when the package either is not
present, or is in the form of a goinstall path? (import error -> attempt
goinstall -> retry import) This would permit go programs and packages
to be distributed without an additional goinstall script file.

Another alternative would be to permit goinstall itself to recognize
goinstall commands in the comments at the top of a Go source file, and
execute them in sequence. Then the build system could call goinstall
either on every file, or do the parsing and call it only when goinstall
comments are present.

I'd like to see the default generated Makefile system not be broken by
goinstall dependencies.


-BobC

Russ Cox

unread,
Feb 25, 2010, 4:31:55 PM2/25/10
to Bob Cunningham, golang-nuts
> 2.  I'm concerned that package source trees would need to include a new
> non-Go file whenever external packages are used: The "goinstall" script.

I'm not sure what you're asking.
How is needing goinstall different from needing 6g?
Goinstall is a single program to be installed with the main distribution.
Packages do not distribute their own copies of goinstall,
and there is no goinstall configuration file.

> Instead, could goinstall be trivially internalized by the language, to
> extend "import" to run "goinstall" when the package either is not present,
> or is in the form of a goinstall path? (import error -> attempt goinstall ->
> retry import)  This would permit go programs and packages to be distributed
> without an additional goinstall script file.

From the original post:

Goinstall is an experiment, and it is easier if the experiment does
not involve changes to the compilers and godoc. The paths used
above double as file system paths, while full URLs do not.
Also, the import path must also convey both the location of
the data and the choice of version control system, so a full URL
would not suffice.

> I'd like to see the default generated Makefile system not be broken by
> goinstall dependencies.

We're not going to introduce dependencies on
external packages into the main tree, so I don't
think it will be.

Russ

Michael Hoisie

unread,
Feb 25, 2010, 7:18:43 PM2/25/10
to golang-nuts
I'm trying to think how this would work in a production environment.

Let's say you are deploying some server which depends on mysqlgo. You
want to make sure it's a stable interface, so ideally you'd want to
reference a specific version, like say 0.5.

Are you planning to support tags, like: "import github.com/mysqlgo/
mysqlgo.git#0.5" ?

Now let's say mysqlgo depends on goyaml. Would the author of mysqlgo
need to import specific version of goyaml for each of his release
tags?

- Mike

Ryanne Dolan

unread,
Feb 25, 2010, 7:36:52 PM2/25/10
to Michael Hoisie, golang-nuts

It seems to me that the release version of a package should only be dl'd once, after which it is stored on your local machine. Since only one version will be dl'd, changes in version obviously won't be an immediate problem for local builds.  Now, if there is a new release version of a package, you should be able to run goinstall again to get the new release, at which point you'd need to worry about changes in the API. But that isn't any different than what we have now, right? We'd still rely on documentation, change logs etc to guide us.

If goinstall were to run everytime I build, it would be a bad thing cuz my code could suddenly break after a new release, but I don't think it does work that way.

For _releasing_ source code for use with goinstall, we might want to reference a specific version number. But like others have said, this would require a complicated dependency relationship among various versions of various packages.

I think a better solutions is for published source code to remain living and changing, so that a change to a dependency would encourage updates to the dependents. So long as the code stays alive and and doesn't depend on rapidly changing packages, it shouldn't be a big problem. And if you happen to be the first person to notice that a package change breaks a dependent package, then you can report the bug or fix it, and in the meantime use hg, svn, etc to revert to a previous release.

In sum, we don't lose anything bc of goinstall, and we gain a lot of convenience when using stable packages.

Ryanne

- from my phone -

On Feb 25, 2010 6:18 PM, "Michael Hoisie" <hoi...@gmail.com> wrote:

I'm trying to think how this would work in a production environment.

Let's say you are deploying some server which depends on mysqlgo. You
want to make sure it's a stable interface, so ideally you'd want to
reference a specific version, like say 0.5.

Are you planning to support tags, like: "import github.com/mysqlgo/
mysqlgo.git#0.5
" ?

Now let's say mysqlgo depends on goyaml. Would the author of mysqlgo
need to import specific version of goyaml for each of his release
tags?

- Mike


On Feb 25, 1:31 pm, Russ Cox <r...@golang.org> wrote:

> > 2.  I'm concerned that package source tre...

Russ Cox

unread,
Feb 25, 2010, 7:43:05 PM2/25/10
to Michael Hoisie, golang-nuts
All the problems about versioning are orthogonal to goinstall.
They are fundamental to any system in which you're using
multiple code bases that progress independently.
Solving that problem is explicitly not a goal for goinstall today.
I'd be more than happy for goinstall to solve the versioning
problem later, if we can figure out how.

I am not dismissing the problem. I just think it is difficult and
not any different for goinstall than it is for any other software
system.

Russ

Marcelo Cantos

unread,
Feb 26, 2010, 4:27:46 AM2/26/10
to golang-nuts
I'm not particularly knowledgeable in this area, but I suggest having
a look at ZeroInstall for ideas. It has a remarkably robust deployment
model that handles architectures, versioning, multi-version co-
existence and source selection (e.g., a local repo. for controlled
deployments) with the greatest of ease. I think all package management
systems should be based on its ideas.

You don't actually deploy anything with ZeroInstall; you just
"execute" a URL. ZeroInstall grabs and caches the package, recursively
following its dependencies. Dependencies denote versioning and
architecture as well. Two users on the same system can fetch different
versions of the same product by using slightly different URL's, and
ZeroInstall fetches any shared dependencies just once (That's the
theory, anyway; this wasn't implemented when I last looked.) The whole
system is based on strong hashing, so naming conflicts are basically
impossible.


Cheers,
Marcelo

P.S.: +1 vote for "goget"

Sverre Rabbelier

unread,
Feb 26, 2010, 4:36:32 AM2/26/10
to r...@golang.org, Michael Hoisie, golang-nuts
Heya,

On Thu, Feb 25, 2010 at 19:40, Russ Cox <r...@golang.org> wrote:
> But they do.  I can't go to Github and create a package named
> github.com/hoisie/web.go.git.  That's what the package gets
> installed and imported as.  There might be packages with two
> different import paths that both claim to be package web.
> That's fine: Go explicitly allows that, and knows enough to keep
> them separate.

> But there is: the source files contain import statements.


> Goinstall looks at them.
>
> For example, goinstall github.com/jacobsa/igo.git/parse fails today,
> because parse/parse.go tries to import "igo/set", which doesn't exist.
> But if Aaron changed his program to use the goinstall conventions
> (and I hope people will), it would import "github.com/jacobsa/igo.git/set",
> and goinstall would gladly download and install that first.
>
> Yes, we could set up a CPAN or a RubyGems or whatever,
> but that requires a centralized authority giving out import paths
> and is more mechanism.  Goinstall is zero configuration
> without any need for central coordination.

Wow, this is truly awesome. You've taken distributed version control
and extended it to code dependencies in a very intuitive way, very
neat.

--
Cheers,

Sverre Rabbelier

Russ Cox

unread,
Feb 26, 2010, 11:36:19 AM2/26/10
to Marcelo Cantos, golang-nuts
> You don't actually deploy anything with ZeroInstall; you just
> "execute" a URL. ZeroInstall grabs and caches the package, recursively
> following its dependencies. Dependencies denote versioning and
> architecture as well. Two users on the same system can fetch different
> versions of the same product by using slightly different URL's, and
> ZeroInstall fetches any shared dependencies just once (That's the
> theory, anyway; this wasn't implemented when I last looked.) The whole
> system is based on strong hashing, so naming conflicts are basically
> impossible.

Thanks for the suggestion. I'll take a look at it.
Hash-based naming can solve the "keep things separate" part,
but Go can already do that itself without hashes.
The real issue, it seems, is whether you want a system that
leads to having ten different versions of the same library linked
into a single binary, because as soon as you can say
"I need this specific version", that's where you're heading.

> P.S.: +1 vote for "goget"

The name goget is great. It's certainly catchier than goinstall.
I'm a little hesitant though, because the meaning is right
for my examples but not the day-to-day use I envision.
I've been writing examples in this thread like

goinstall mimeparse.googlecode.com/svn/trunk/go

to illustrate what the import paths look like, but I don't
expect that to be a common usage pattern. Instead,
I hope people will write their own packages in some local
directory my/pkg with import statements like
import "mimeparse.googlecode.com/svn/trunk/go"
and then run

goinstall my/pkg

which will grab the dependencies as needed.
This way avoids the repetition of having to write the same
string both in an import statement and on a command line.
For that example,

goget my/pkg

doesn't look quite right: the command's not getting my/pkg
at all, because I wrote my/pkg and it's already here.

Russ

John Asmuth

unread,
Feb 26, 2010, 12:44:56 PM2/26/10
to golang-nuts
I suggest we adopt a convention to address this issue.

In the source control directory, we could add a directory whose name
means something. One directory would be the name of the package (to
make it obvious how to refer to it in code). It can then be prefixed
with version information, or not in the case of edge.

Requires slightly more work by the developer, but only about 30
seconds per release. Pretty reasonable.

gomatrix.googlecode.com/hg/matrix //the "edge" version
gomatrix.googlecode.com/hg/release/matrix //latest stable version,
happens to be v0.6
gomatrix.googlecode.com/hg/v0.6/matrix //previous release that happens
to be release
gomatrix.googlecode.com/hg/v0.5/matrix //previous release that is now
out-of-date

For source controls that support version information in the url, we
can continue using that. But for gomatrix, which uses mercurial, we
can do the above.

Thoughts?

- John

David Roundy

unread,
Feb 26, 2010, 5:56:58 PM2/26/10
to r...@golang.org, golang-nuts
Hi Russ,

Looks exciting!

Is there a reason why you don't do the build from within goinstall
itself, but instead generate a Makefile? I guess the advantage of
creating a Makefile and passing it off to make means that if the
included makefiles change, then you benefit from those changes for
"free" (since they are included). But on the other hand, you have a
dependency on make, and you have the complexity of both generating
correct input for make and running it properly.

I'm particularly curious about directions of the building back side
because I'm interested in making gotit template packages easy to
build. Teaching goinstall how to build .got files would be great, but
looks particularly complicated if it means both modifying the standard
Makefiles *and* modifying goinstall itself. The gotit approach to
templating (implemented most clearly in gotimports) is very close to
goinstall's approach to downloading: it recognizes special import
formats and builds the appropriate package for you. I think this is
compatible with goinstall, but using the two simultaneously would
obviously require some hacking. gotit isn't ready for prime time
(since noone but me has tried it), but I'd like to make it useful
enough that people *do* try it and try using packages built with it.

David

joseph.wecker

unread,
Feb 27, 2010, 9:53:55 PM2/27/10
to golang-nuts

> So there is need for being able to specify the version number of the
> library. If the version number is part of the import statement, then
> when a library is upgraded, one must change the version number in
> possibly hundreds of files. This smells likehttp://en.wikipedia.org/wiki/Shotgun_surgery

Someone performing shotgun surgery like that wouldn't be able to blame
goinstall, but instead the fragile dependencies they've been forced to
import.

I think this is great overall- it allows you to be as granular as you
need to be or as generic as you can be with your imports, including
versioning if/when the url-scheme + vcs allows it.

Marcelo Cantos

unread,
Feb 28, 2010, 12:04:59 AM2/28/10
to golang-nuts
On Feb 27, 3:36 am, Russ Cox <r...@golang.org> wrote:
> Thanks for the suggestion.  I'll take a look at it.
> Hash-based naming can solve the "keep things separate" part,
> but Go can already do that itself without hashes.
> The real issue, it seems, is whether you want a system that
> leads to having ten different versions of the same library linked
> into a single binary, because as soon as you can say
> "I need this specific version", that's where you're heading.

ZeroInstall operates at the dynamic binding level. At load time, it
binds environment variables such as LD_LIBRARY_PATH and PYTHONPATH so
that package A's executable can find package B's shared library. If B
gets an upgrade, the bindings change. If it's a static library you
want to "share", then the ideas in ZeroInstall may not suit unless you
can somehow bring them to bear on the build process.

chris dollin

unread,
Mar 16, 2010, 5:30:43 AM3/16/10
to r...@golang.org, golang-nuts
goinstall is now documented in yesterday's release (hooray). The
introductory paragraph of  http://golang.org/cmd/goinstall/ says:

  It maintains a list of public Go packages at http://godashboard.appspot.com/packages.

(a) Oops. "packages" should be "package".

(b) the URL should surely be a link. Mr Click is your friend!

Should I raise an issue?

Chris

--
Chris "allusive" Dollin

Andrew Gerrand

unread,
Mar 16, 2010, 5:34:51 AM3/16/10
to chris dollin, r...@golang.org, golang-nuts
Please do. I'll fix it tomorrow (if somebody else doesn't beat me to it. =)

chris dollin

unread,
Mar 16, 2010, 5:47:33 AM3/16/10
to Andrew Gerrand, r...@golang.org, golang-nuts
On 16 March 2010 09:34, Andrew Gerrand <a...@google.com> wrote:
Please do. I'll fix it tomorrow (if somebody else doesn't beat me to it. =)

OK, issue 668.

--
Chris "allusive" Dollin

chris dollin

unread,
Mar 16, 2010, 9:03:59 AM3/16/10
to r...@golang.org, golang-nuts
I've just done a (successful) install of web.go using goinstall. Why did I pick web.go?
Because when I looked at http://godashboard.appspot.com/package there was no
package-supplied summary line telling me what the package was about, web.go
had "web" in it, and it was vaguely near the top. (There's no way I would have guessed
that mustache.go was a templating package.)

Could we establish conventions for one-liner and short-summary texts to be
provided by the package?

Also I think times like 23:29:55.492225 might provide a little too much in the
way of precision, at least for the human reader. (For the machine, content
negotiation for json or rdf forms of the data, and all that yummy precision,
might be useful.)

David Roundy

unread,
Mar 16, 2010, 2:40:42 PM3/16/10
to Andrew Gerrand, golang-nuts
On Sun, Feb 28, 2010 at 5:50 PM, Andrew Gerrand <a...@google.com> wrote:

> On 27 February 2010 09:56, David Roundy <rou...@physics.oregonstate.edu> wrote:
>> Hi Russ,
>>
>> Looks exciting!
>>
>> Is there a reason why you don't do the build from within goinstall
>> itself, but instead generate a Makefile? I guess the advantage of
>> creating a Makefile and passing it off to make means that if the
>> included makefiles change, then you benefit from those changes for
>> "free" (since they are included).  But on the other hand, you have a
>> dependency on make, and you have the complexity of both generating
>> correct input for make and running it properly.
>
> The purpose of goinstall is to install a package and its dependencies.
> Having goinstall perform builds would complicate the tool. Having
> goinstall create makefiles just makes the external packages part of
> your existing build. If you change the way you build (eg,
> architecture), you need only rebuild as normal without involving
> goinstall.

I see. It's a subtle distinction. Then I will ask why install a
Makefile at all? Why not rely on the Makefile provided with the
package?

>> I'm particularly curious about directions of the building back side
>> because I'm interested in making gotit template packages easy to
>> build.  Teaching goinstall how to build .got files would be great, but
>> looks particularly complicated if it means both modifying the standard
>> Makefiles *and* modifying goinstall itself.  The gotit approach to
>> templating (implemented most clearly in gotimports) is very close to
>> goinstall's approach to downloading:  it recognizes special import
>> formats and builds the appropriate package for you.  I think this is
>> compatible with goinstall, but using the two simultaneously would
>> obviously require some hacking.  gotit isn't ready for prime time
>> (since noone but me has tried it), but I'd like to make it useful
>> enough that people *do* try it and try using packages built with it.
>

> Would making gotit compatible with the standard Makefile format solve
> this issue?

I'm not sure what you mean by that, unless you mean to modify the
"standard Makefile"?
--
David Roundy

Russ Cox

unread,
Mar 16, 2010, 4:10:27 PM3/16/10
to David Roundy, Andrew Gerrand, golang-nuts
>> Having goinstall perform builds would complicate the tool. Having
>> goinstall create makefiles just makes the external packages part of
>> your existing build. If you change the way you build (eg,
>> architecture), you need only rebuild as normal without involving
>> goinstall.
>
> I see.  It's a subtle distinction.  Then I will ask why install a
> Makefile at all? Why not rely on the Makefile provided with the
> package?

The main reason is to force the target install path to
be the one that goinstall thinks it should be. This lets
packages written without knowing about goinstall
or without using a Makefile still be installed with goinstall.

Now that goinstall is known, I hope people will change
their makefiles to install under the goinstall target path.
If the Makefile has a reasonable TARG= then it is
probably okay to use the distributed Makefile.

Russ

Andrew Gerrand

unread,
Mar 16, 2010, 7:17:53 PM3/16/10
to chris dollin, r...@golang.org, golang-nuts
On 17 March 2010 00:03, chris dollin <ehog....@googlemail.com> wrote:
> I've just done a (successful) install of web.go using goinstall. Why did I
> pick web.go?
> Because when I looked at http://godashboard.appspot.com/package there was no
> package-supplied summary line telling me what the package was about, web.go
> had "web" in it, and it was vaguely near the top. (There's no way I would
> have guessed
> that mustache.go was a templating package.)
>
> Could we establish conventions for one-liner and short-summary texts to be
> provided by the package?

I have been thinking about this. The information could be stored in a
special file included in the distribution, eg 'PACKAGE'. It would
contain the package name, description, website URL, logo URL, etc.

I'm also working on a rating system, so people can log into
godashboard and rate packages as good/bad. I'm not sure of the best
rating system though - it's a complex problem. Do you rate a
well-designed package badly if it's buggy? What if it's good but
totally unmaintained? Should individual ratings become less
influential on the overall rating as they age (so the more current
ratings have a greater influence)?

Andrew

Brian Slesinsky

unread,
Mar 17, 2010, 12:53:58 AM3/17/10
to golang-nuts
This looks elegant and useful. However, goinstall is nondeterministic
due to depending on the current date, and I suspect that could lead to
confusion. Within a single organization you'd want to check in
everything under $GOROOT into your own source control system after
downloading it, so I don't see that as much of a problem. But for
people at different organizations who want to