goinstall: an experiment in package installation

1531 views
Skip to first unread message

Russ Cox

unread,
Feb 24, 2010, 6:41:31 PM2/24/10
to golang-nuts
CL 224043, not yet sent for review, would add a new command
goinstall that can download packages into the local file system.
Rather than establish a central naming authority for Go packages
(witness the multiple mysqlgo packages), it uses import paths
that contain the names of common hosting sites. The full docs
are below.

Goinstall already works with standalone packages that
don't depend on other packages. For example, I can run:

goinstall github.com/hoisie/web.go.git
goinstall github.com/jacobsa/igo.git/set
goinstall go-avltree.googlecode.com/svn/trunk
goinstall mimeparse.googlecode.com/svn/trunk/go
goinstall gomatrix.googlecode.com/hg
goinstall golang.googlecode.com/hg/src/pkg/scanner

And then import any of those using the same paths.
If those had referred to other import paths of the same
form, goinstall would have downloaded the dependencies,
recursively. So in your own package, you can put
a line like import "gomatrix.googlecode.com/hg" and
then just run

goinstall my/package

which will notice the import line and download and install
the remote package. The first time goinstall downloads
and installs a public package, it pings the Go dashboard
to update a list of what people are using.
http://godashboard.appspot.com/package.

This is a request for comments on the general approach.
Please comment here, on the mailing list, rather than in the
codereview site. It will be easier to have a discussion in an
actual discussion forum.


-- documentation from goinstall/doc.go

Goinstall is an experiment in automatic package installation.
It installs packages, possibly downloading them from public version
control systems, and it maintains a list of public Go packages at
http://godashboard.appspot.com/packages.

Usage:
goinstall [flags] importpath...

Flags and default settings:
-dashboard=true tally public packages on godashboard.appspot.com
-update=false update already-downloaded packages
-v=false verbose operation

Goinstall installs each of the packages identified by import path on
the command line. It installs a package's prerequisites before
trying to install the package itself.

The source code for a package with import path foo/bar is expected
to be in the directory $GOROOT/src/pkg/foo/bar/. If the import
path refers to a code hosting site, goinstall will download it if necessary.
The recognized code hosting sites are:

GitHub (Git)

import "github.com/user/project.git"
import "github.com/user/project.git/sub/directory"

Google Code Project Hosting (Mercurial, Subversion)

import "project.googlecode.com/hg"
import "project.googlecode.com/hg/sub/directory"

import "project.googlecode.com/svn/trunk"
import "project.googlecode.com/svn/trunk/sub/directory"


If the directory (e.g., $GOROOT/src/pkg/bitbucket.org/user/project)
already exists and contains an appropriate checkout, goinstall will not
attempt to fetch updates. The -update flag changes this behavior,
causing goinstall to update all remote packages encountered during
the installation.

When downloading or updating, goinstall first looks for a tag or branch
named "release". If there is one, it uses that version of the code.
Otherwise it uses the default version selected by the version control
system (typically HEAD for git, tip for Mercurial).

After a successful download and installation of a publicly accessible
remote package, goinstall reports the installation to godashboard.appspot.com,
which increments a count associated with the package and the time
of its most recent installation. This mechanism powers the package list
at http://godashboard.appspot.com/packages, allowing Go programmers
to learn about popular packages that might be worth looking at.
The -dashboard=false flag disables this reporting.

By default, goinstall prints output only when it encounters an error.
The -v flag causes goinstall to print information about packages
being considered and installed.

Goinstall does not attempt to be a replacement for make.
Instead, it invokes "make install" after locating the package sources.
For local packages without a Makefile and all remote packages,
goinstall creates and uses a temporary Makefile constructed from
the import path and the list of Go files in the package.


FAQ:

• Why not use full URLs instead of hard-coding hosting sites?

Goinstall is an experiment, and it is easier if the experiment does
not involve changes to the compilers and godoc. The paths used
above double as file system paths, while full URLs do not.
Also, the import path must also convey both the location of
the data and the choice of version control system, so a full URL
would not suffice.

Using this notation does not precluding changing to other
notation once we have more experience or in other contexts.
For example, if Go were to run inside a browser it would make
sense for import lines to be URLs of individual files rather than
version control repositories.

• Why isn't my favorite hosting site supported?

Feel free to add it and send us a CL.
See http://golang.org/doc/contribute.html

• How can I use goinstall with a package I commit changes to?

Instead of having goinstall do the checkout automatically,
check out your package yourself into the same location where
goinstall would put it. You can use that as your working copy
and push to the world as appropriate. For example, if you
were working on github.com/hoisie/web.go, you check it out
into $GOROOT/src/pkg/github.com/hoisie/web.go.git
and then edit the files in that directory, pushing back to github
as needed. Goinstall will use the checked-out git repository
instead of trying to download a fresh one.

• What information is sent to and logged by the dashboard?

The source code for both goinstall and the dashboard are
in the Go repository:

$GOROOT/src/cmd/goinstall/download.go
$GOROOT/misc/dashboard/godashboard/package.py

Goinstall sends the package import path.
The dashboard increments a counter and sets the "most recently installed" time.
There is no other reporting or logging. The data recorded by the dashboard
is publicly visible at http://godashboard.appspot.com/package.

-- end documentation


Again, comments welcome, but please post them here rather
than on the code review site.

Thanks.
Russ

John Asmuth

unread,
Feb 24, 2010, 9:46:12 PM2/24/10
to golang-nuts
Pretty cool - what about the package name?

For instance (and I like that you used my package as one of your
examples)
import "gomatrix.googlecode.com/hg"

The package name is matrix, which is not reflected in the import
statement. Now, I understand that, when goinstall downloads it, it
will examine what it downloads and know the correct package name for
use in the code.

But consider the case when someone asks how to use matrices in go, and
someone in the IRC channel says, "oh, just use <import statement> at
the top of your file", referring to the import atatement above.

Then this hapless coder has no immediate way to see what the name of
the package is. Maybe "gomatrix"? But no, it's "matrix", which you can
only know by examining the gomatrix source.

This problem doesn't appear so much normally, because it is customary
to name the directory containing a package after that package.

Of course, if this hapless coder is slightly less hapless, he or she
can name the import. This obviously isn't an insurmountable problem.
Just one that can cause some confusion at first for those coders who
have trouble looking stuff up on their own (for whatever reason).

But this seems like a really useful idea, and a very slick way to keep
code-versions up-to-date.

Which makes me think... is there a way to refer to a certain version
of, for example, a googlecode hg project? What if your code works with
an older version, and breaks with a newer? For instance, it is
possible that some clever compiler optimizations would make some of
the methods in gomatrix pointless, and they could be removed. But what
about projects that used those now-removed methods? Being able to
refer to a specific and immutable version would be sweet, and the
default code be the most up-to-date.


- John

> to update a list of what people are using.http://godashboard.appspot.com/package.


>
> This is a request for comments on the general approach.
> Please comment here, on the mailing list, rather than in the
> codereview site.  It will be easier to have a discussion in an
> actual discussion forum.
>
> -- documentation from goinstall/doc.go
>
> Goinstall is an experiment in automatic package installation.
> It installs packages, possibly downloading them from public version

> control systems, and it maintains a list of public Go packages athttp://godashboard.appspot.com/packages.

> athttp://godashboard.appspot.com/packages, allowing Go programmers

> Seehttp://golang.org/doc/contribute.html


>
> • How can I use goinstall with a package I commit changes to?
>
> Instead of having goinstall do the checkout automatically,
> check out your package yourself into the same location where
> goinstall would put it.  You can use that as your working copy
> and push to the world as appropriate.  For example, if you
> were working on github.com/hoisie/web.go, you check it out
> into $GOROOT/src/pkg/github.com/hoisie/web.go.git
> and then edit the files in that directory, pushing back to github
> as needed.  Goinstall will use the checked-out git repository
> instead of trying to download a fresh one.
>
> • What information is sent to and logged by the dashboard?
>
> The source code for both goinstall and the dashboard are
> in the Go repository:
>
>     $GOROOT/src/cmd/goinstall/download.go
>     $GOROOT/misc/dashboard/godashboard/package.py
>
> Goinstall sends the package import path.
> The dashboard increments a counter and sets the "most recently installed" time.
> There is no other reporting or logging.  The data recorded by the dashboard

> is publicly visible athttp://godashboard.appspot.com/package.

Russ Cox

unread,
Feb 25, 2010, 12:58:55 AM2/25/10
to John Asmuth, golang-nuts
Thanks for the feedback.

> For instance (and I like that you used my package as one of your
> examples)
>  import "gomatrix.googlecode.com/hg"
>
> The package name is matrix, which is not reflected in the import
> statement. Now, I understand that, when goinstall downloads it, it
> will examine what it downloads and know the correct package name for
> use in the code.
>
> But consider the case when someone asks how to use matrices in go, and
> someone in the IRC channel says, "oh, just use <import statement> at
> the top of your file", referring to the import atatement above.
>
> Then this hapless coder has no immediate way to see what the name of
> the package is. Maybe "gomatrix"? But no, it's "matrix", which you can
> only know by examining the gomatrix source.

It's true; this is a problem. I think this is going to end up
being solved by common sense and convention. Maybe
gomatrix.googlecode.com/hg should call itself package gomatrix,
or maybe it should be in a subdirectory named matrix.
The subdirectory = package name rule is a good general
convention, but there are reasonable exceptions too.
For example, the mimeparse project has source code for
many different languages, and the Go code is in the go
subdirectory. It doesn't bother me that I write import
"mimeparse.googlecode.com/svn/trunk/go" to get package
mimeparse, and I'd much rather get package mimeparse
than package go. It's an open issue, but I think conventions
will go a long way.

> Which makes me think... is there a way to refer to a certain version
> of, for example, a googlecode hg project? What if your code works with
> an older version, and breaks with a newer? For instance, it is
> possible that some clever compiler optimizations would make some of
> the methods in gomatrix pointless, and they could be removed. But what
> about projects that used those now-removed methods? Being able to
> refer to a specific and immutable version would be sweet, and the
> default code be the most up-to-date.

Subversion encodes this info in the path, so you can always
import something other than trunk. In hg and git, there's no
direct support for specific versions but goinstall does prefer to
use the branch or tag named "release" over using git HEAD/hg tip.

In general the versioning problem is pretty tricky. If you must use
version 1 of a package and Michael must use version 2 and I want
to use both your packages, then I end up with both copies in my
binary. 6g can handle that; gccgo might need a little help.
Either way it's suboptimal to be linking every version in.
Goinstall nods at the versioning problem by preferring "release"
but it's by no means solved, and I don't know that there's
a magic bullet.

Russ

chris dollin

unread,
Feb 25, 2010, 2:47:18 AM2/25/10
to r...@golang.org, golang-nuts
On 24 February 2010 23:41, Russ Cox <r...@golang.org> wrote:

(WRT goinstall)


of its most recent installation.  This mechanism powers the package list
at http://godashboard.appspot.com/packages, allowing Go programmers
to learn about popular packages that might be worth looking at.

That link currently serves an empty page, no text, no HTML, nothing, not
even a Coming Soon; is that as expected?

--
Chris "allusive" Dollin

Russ Cox

unread,
Feb 25, 2010, 3:23:42 AM2/25/10
to chris dollin, golang-nuts
>> of its most recent installation.  This mechanism powers the package list
>> at http://godashboard.appspot.com/packages, allowing Go programmers
>> to learn about popular packages that might be worth looking at.
>
> That link currently serves an empty page, no text, no HTML, nothing, not
> even a Coming Soon; is that as expected?

Sorry, typo. The correct URL is http://godashboard.appspot.com/package.

Russ

Esko Luontola

unread,
Feb 25, 2010, 5:19:20 AM2/25/10
to golang-nuts
What about libraries which consist of multiple packages? For example
I'll split GoSpec (http://github.com/orfjackal/gospec) into multiple
packages as soon as I have a build tool which can handle building and
testing it easily (probably gobbler or igo). Then the users of GoSpec
will need to have the following imports in their test sources:

import "gospec"
import . "gospec/matchers"

And the gospec package in turn will depend on multiple internal
packages: "gospec/reporting", "gospec/foo" etc. All of those packages
are needed for the library to work.

andrey mirtchovski

unread,
Feb 25, 2010, 10:22:03 AM2/25/10
to golang-nuts
Two questions:

- How are you handling inclusions in the Makefile (and the NOTEST
option), or would those packages not be recompiled during "all.bash"?

- bitbucket is mentioned in your email, but not explicitly as working.
is it supported?

thanks

Russ Cox

unread,
Feb 25, 2010, 11:32:55 AM2/25/10
to andrey mirtchovski, golang-nuts
> - How are you handling inclusions in the Makefile (and the NOTEST
> option), or would those packages not be recompiled during "all.bash"?

downloaded Makefiles are ignored.
the package gets built using a standard-form makefile
generated from the list of files in the directory
(must only be one non-main package in the directory).

the packages are not recompiled by all.bash.

> - bitbucket is mentioned in your email, but not explicitly as working.
> is it supported?

i intended to add it but haven't for lack of a test case.

russ

Russ Cox

unread,
Feb 25, 2010, 11:35:36 AM2/25/10
to Esko Luontola, golang-nuts

you'd make those

import "github.com/orfjackal/gospec.git"
import "github.com/ofjackal/gospec.git/matchers"

and then the gospec package would say, internally

import "github.com/orfjackal/gospec.git/reporting"
import "github.com/orfjackal/gospec.git/foo"

when goinstall downloaded gospec for the first import,
it would see the others and build those subpackages
first. when working on the package locally, you'd just
work in the $GOROOT/src/pkg/github.com/orfjackal/gospec.git
directory, which would be a standard git checkout.

russ

Esko Luontola

unread,
Feb 25, 2010, 12:59:11 PM2/25/10
to golang-nuts
On Feb 25, 6:35 pm, Russ Cox <r...@golang.org> wrote:
> when goinstall downloaded gospec for the first import,
> it would see the others and build those subpackages
> first.  when working on the package locally, you'd just
> work in the $GOROOT/src/pkg/github.com/orfjackal/gospec.git
> directory, which would be a standard git checkout.

What about handling version dependencies? Let's say that I'm
developing a library and I have N projects using that library.

When working on the library itself, I edit the development version of
the library. But at the same time, when working on those N other
projects, they must use the release version of the library. And when
there are incompatible changes to the library, each of the N projects
must use a specific version of the library.

So there is need for being able to specify the version number of the
library. If the version number is part of the import statement, then
when a library is upgraded, one must change the version number in
possibly hundreds of files. This smells like http://en.wikipedia.org/wiki/Shotgun_surgery

And anyways I don't like being forced to put all the sources under
$GOROOT. Also having to install all packages there (as it's done
today) feels like a bad idea. I think that a long-term solution would
be a better build tool, where you can configure the dependencies and
versions in one file per project (similar to Maven et al).

Russ Cox

unread,
Feb 25, 2010, 1:06:38 PM2/25/10
to Esko Luontola, golang-nuts
> What about handling version dependencies? Let's say that I'm
> developing a library and I have N projects using that library.
>
> When working on the library itself, I edit the development version of
> the library. But at the same time, when working on those N other
> projects, they must use the release version of the library. And when
> there are incompatible changes to the library, each of the N projects
> must use a specific version of the library.

So don't do that. I answered this earlier on the thread:
versioning is a tough problem and we're not trying to solve it today.

> So there is need for being able to specify the version number of the
> library. If the version number is part of the import statement, then
> when a library is upgraded, one must change the version number in
> possibly hundreds of files. This smells like http://en.wikipedia.org/wiki/Shotgun_surgery

Do you have a proposal for how to specify the version number
that a particular file needs without maintaining that information?

> And anyways I don't like being forced to put all the sources under
> $GOROOT. Also having to install all packages there (as it's done
> today) feels like a bad idea. I think that a long-term solution would
> be a better build tool, where you can configure the dependencies and
> versions in one file per project (similar to Maven et al).

Configuration = complexity.
Some may be warranted, but I don't want to just take what everyone
else does, because it's too much manual effort. In contrast, goinstall
is automatic if you're willing to live within the conventions.

I do think there will be a sort of unified go path eventually (today,
godoc, 6g, and 6l each have their own idea of what that means),
but we're not there yet. One step at a time.

Russ

Michael Hoisie

unread,
Feb 25, 2010, 1:28:08 PM2/25/10
to golang-nuts
I think this is a great start but I'm worried this won't scale well.

First of all, as someone pointed out, there will probably be naming
conflicts. Github doesn't enforce any kind of unique naming, so you
could have packages overwrite each other. Neither does Google Code.

The second issue is dependency management. Right now most Go packages
are simple and only depend on the core library. However, this is
likely to change as libraries stabilize and become more useful. There
isn't any meta-data on Github or Google code about dependencies.

Why not model the system more closely to rubygems (http://
www.rubygems.org). Rubygems has a structured package format with a
small metadata file, so it can automatically handle versioning and
dependencies.

There could be some central repo of packages on http://godashboard.appspot.com/packages,
so global naming could be enforced. To submit a package to this site,
you would have to conform to the go package format. It would be more
strict, but I feel the system would be more extendable than just using
github or google code.

- Mike

On Feb 25, 10:06 am, Russ Cox <r...@golang.org> wrote:
> > What about handling version dependencies? Let's say that I'm
> > developing a library and I have N projects using that library.
>
> > When working on the library itself, I edit the development version of
> > the library. But at the same time, when working on those N other
> > projects, they must use the release version of the library. And when
> > there are incompatible changes to the library, each of the N projects
> > must use a specific version of the library.
>
> So don't do that.  I answered this earlier on the thread:
> versioning is a tough problem and we're not trying to solve it today.
>
> > So there is need for being able to specify the version number of the
> > library. If the version number is part of the import statement, then
> > when a library is upgraded, one must change the version number in

> > possibly hundreds of files. This smells likehttp://en.wikipedia.org/wiki/Shotgun_surgery

Russ Cox

unread,
Feb 25, 2010, 1:40:05 PM2/25/10
to Michael Hoisie, golang-nuts
Thanks for your comments. I think both issues you raised are
already address by the system:

> First of all, as someone pointed out, there will probably be naming
> conflicts. Github doesn't enforce any kind of unique naming, so you
> could have packages overwrite each other. Neither does Google Code.

But they do. I can't go to Github and create a package named
github.com/hoisie/web.go.git. That's what the package gets
installed and imported as. There might be packages with two
different import paths that both claim to be package web.
That's fine: Go explicitly allows that, and knows enough to keep
them separate.

> The second issue is dependency management. Right now most Go packages
> are simple and only depend on the core library. However, this is
> likely to change as libraries stabilize and become more useful. There
> isn't any meta-data on Github or Google code about dependencies.

But there is: the source files contain import statements.
Goinstall looks at them.

For example, goinstall github.com/jacobsa/igo.git/parse fails today,
because parse/parse.go tries to import "igo/set", which doesn't exist.
But if Aaron changed his program to use the goinstall conventions
(and I hope people will), it would import "github.com/jacobsa/igo.git/set",
and goinstall would gladly download and install that first.

Yes, we could set up a CPAN or a RubyGems or whatever,
but that requires a centralized authority giving out import paths
and is more mechanism. Goinstall is zero configuration
without any need for central coordination.

Russ

Bob Cunningham

unread,
Feb 25, 2010, 4:26:54 PM2/25/10
to r...@golang.org, golang-nuts
1. "goinstall"? I think it should have been called "goget" from the
get-go. ;^)

2. I'm concerned that package source trees would need to include a new
non-Go file whenever external packages are used: The "goinstall" script.

Would the build system be extended to automatically run such scripts?
Not ideal...

Instead, could goinstall be trivially internalized by the language, to
extend "import" to run "goinstall" when the package either is not
present, or is in the form of a goinstall path? (import error -> attempt
goinstall -> retry import) This would permit go programs and packages
to be distributed without an additional goinstall script file.

Another alternative would be to permit goinstall itself to recognize
goinstall commands in the comments at the top of a Go source file, and
execute them in sequence. Then the build system could call goinstall
either on every file, or do the parsing and call it only when goinstall
comments are present.

I'd like to see the default generated Makefile system not be broken by
goinstall dependencies.


-BobC

Russ Cox

unread,
Feb 25, 2010, 4:31:55 PM2/25/10
to Bob Cunningham, golang-nuts
> 2.  I'm concerned that package source trees would need to include a new
> non-Go file whenever external packages are used: The "goinstall" script.

I'm not sure what you're asking.
How is needing goinstall different from needing 6g?
Goinstall is a single program to be installed with the main distribution.
Packages do not distribute their own copies of goinstall,
and there is no goinstall configuration file.

> Instead, could goinstall be trivially internalized by the language, to
> extend "import" to run "goinstall" when the package either is not present,
> or is in the form of a goinstall path? (import error -> attempt goinstall ->
> retry import)  This would permit go programs and packages to be distributed
> without an additional goinstall script file.

From the original post:

Goinstall is an experiment, and it is easier if the experiment does
not involve changes to the compilers and godoc. The paths used
above double as file system paths, while full URLs do not.
Also, the import path must also convey both the location of
the data and the choice of version control system, so a full URL
would not suffice.

> I'd like to see the default generated Makefile system not be broken by
> goinstall dependencies.

We're not going to introduce dependencies on
external packages into the main tree, so I don't
think it will be.

Russ

Michael Hoisie

unread,
Feb 25, 2010, 7:18:43 PM2/25/10
to golang-nuts
I'm trying to think how this would work in a production environment.

Let's say you are deploying some server which depends on mysqlgo. You
want to make sure it's a stable interface, so ideally you'd want to
reference a specific version, like say 0.5.

Are you planning to support tags, like: "import github.com/mysqlgo/
mysqlgo.git#0.5" ?

Now let's say mysqlgo depends on goyaml. Would the author of mysqlgo
need to import specific version of goyaml for each of his release
tags?

- Mike

Ryanne Dolan

unread,
Feb 25, 2010, 7:36:52 PM2/25/10
to Michael Hoisie, golang-nuts

It seems to me that the release version of a package should only be dl'd once, after which it is stored on your local machine. Since only one version will be dl'd, changes in version obviously won't be an immediate problem for local builds.  Now, if there is a new release version of a package, you should be able to run goinstall again to get the new release, at which point you'd need to worry about changes in the API. But that isn't any different than what we have now, right? We'd still rely on documentation, change logs etc to guide us.

If goinstall were to run everytime I build, it would be a bad thing cuz my code could suddenly break after a new release, but I don't think it does work that way.

For _releasing_ source code for use with goinstall, we might want to reference a specific version number. But like others have said, this would require a complicated dependency relationship among various versions of various packages.

I think a better solutions is for published source code to remain living and changing, so that a change to a dependency would encourage updates to the dependents. So long as the code stays alive and and doesn't depend on rapidly changing packages, it shouldn't be a big problem. And if you happen to be the first person to notice that a package change breaks a dependent package, then you can report the bug or fix it, and in the meantime use hg, svn, etc to revert to a previous release.

In sum, we don't lose anything bc of goinstall, and we gain a lot of convenience when using stable packages.

Ryanne

- from my phone -

On Feb 25, 2010 6:18 PM, "Michael Hoisie" <hoi...@gmail.com> wrote:

I'm trying to think how this would work in a production environment.

Let's say you are deploying some server which depends on mysqlgo. You
want to make sure it's a stable interface, so ideally you'd want to
reference a specific version, like say 0.5.

Are you planning to support tags, like: "import github.com/mysqlgo/
mysqlgo.git#0.5
" ?

Now let's say mysqlgo depends on goyaml. Would the author of mysqlgo
need to import specific version of goyaml for each of his release
tags?

- Mike


On Feb 25, 1:31 pm, Russ Cox <r...@golang.org> wrote:

> > 2.  I'm concerned that package source tre...

Russ Cox

unread,
Feb 25, 2010, 7:43:05 PM2/25/10
to Michael Hoisie, golang-nuts
All the problems about versioning are orthogonal to goinstall.
They are fundamental to any system in which you're using
multiple code bases that progress independently.
Solving that problem is explicitly not a goal for goinstall today.
I'd be more than happy for goinstall to solve the versioning
problem later, if we can figure out how.

I am not dismissing the problem. I just think it is difficult and
not any different for goinstall than it is for any other software
system.

Russ

Marcelo Cantos

unread,
Feb 26, 2010, 4:27:46 AM2/26/10
to golang-nuts
I'm not particularly knowledgeable in this area, but I suggest having
a look at ZeroInstall for ideas. It has a remarkably robust deployment
model that handles architectures, versioning, multi-version co-
existence and source selection (e.g., a local repo. for controlled
deployments) with the greatest of ease. I think all package management
systems should be based on its ideas.

You don't actually deploy anything with ZeroInstall; you just
"execute" a URL. ZeroInstall grabs and caches the package, recursively
following its dependencies. Dependencies denote versioning and
architecture as well. Two users on the same system can fetch different
versions of the same product by using slightly different URL's, and
ZeroInstall fetches any shared dependencies just once (That's the
theory, anyway; this wasn't implemented when I last looked.) The whole
system is based on strong hashing, so naming conflicts are basically
impossible.


Cheers,
Marcelo

P.S.: +1 vote for "goget"

Sverre Rabbelier

unread,
Feb 26, 2010, 4:36:32 AM2/26/10
to r...@golang.org, Michael Hoisie, golang-nuts
Heya,

On Thu, Feb 25, 2010 at 19:40, Russ Cox <r...@golang.org> wrote:
> But they do.  I can't go to Github and create a package named
> github.com/hoisie/web.go.git.  That's what the package gets
> installed and imported as.  There might be packages with two
> different import paths that both claim to be package web.
> That's fine: Go explicitly allows that, and knows enough to keep
> them separate.

> But there is: the source files contain import statements.


> Goinstall looks at them.
>
> For example, goinstall github.com/jacobsa/igo.git/parse fails today,
> because parse/parse.go tries to import "igo/set", which doesn't exist.
> But if Aaron changed his program to use the goinstall conventions
> (and I hope people will), it would import "github.com/jacobsa/igo.git/set",
> and goinstall would gladly download and install that first.
>
> Yes, we could set up a CPAN or a RubyGems or whatever,
> but that requires a centralized authority giving out import paths
> and is more mechanism.  Goinstall is zero configuration
> without any need for central coordination.

Wow, this is truly awesome. You've taken distributed version control
and extended it to code dependencies in a very intuitive way, very
neat.

--
Cheers,

Sverre Rabbelier

Russ Cox

unread,
Feb 26, 2010, 11:36:19 AM2/26/10
to Marcelo Cantos, golang-nuts
> You don't actually deploy anything with ZeroInstall; you just
> "execute" a URL. ZeroInstall grabs and caches the package, recursively
> following its dependencies. Dependencies denote versioning and
> architecture as well. Two users on the same system can fetch different
> versions of the same product by using slightly different URL's, and
> ZeroInstall fetches any shared dependencies just once (That's the
> theory, anyway; this wasn't implemented when I last looked.) The whole
> system is based on strong hashing, so naming conflicts are basically
> impossible.

Thanks for the suggestion. I'll take a look at it.
Hash-based naming can solve the "keep things separate" part,
but Go can already do that itself without hashes.
The real issue, it seems, is whether you want a system that
leads to having ten different versions of the same library linked
into a single binary, because as soon as you can say
"I need this specific version", that's where you're heading.

> P.S.: +1 vote for "goget"

The name goget is great. It's certainly catchier than goinstall.
I'm a little hesitant though, because the meaning is right
for my examples but not the day-to-day use I envision.
I've been writing examples in this thread like

goinstall mimeparse.googlecode.com/svn/trunk/go

to illustrate what the import paths look like, but I don't
expect that to be a common usage pattern. Instead,
I hope people will write their own packages in some local
directory my/pkg with import statements like
import "mimeparse.googlecode.com/svn/trunk/go"
and then run

goinstall my/pkg

which will grab the dependencies as needed.
This way avoids the repetition of having to write the same
string both in an import statement and on a command line.
For that example,

goget my/pkg

doesn't look quite right: the command's not getting my/pkg
at all, because I wrote my/pkg and it's already here.

Russ

John Asmuth

unread,
Feb 26, 2010, 12:44:56 PM2/26/10
to golang-nuts
I suggest we adopt a convention to address this issue.

In the source control directory, we could add a directory whose name
means something. One directory would be the name of the package (to
make it obvious how to refer to it in code). It can then be prefixed
with version information, or not in the case of edge.

Requires slightly more work by the developer, but only about 30
seconds per release. Pretty reasonable.

gomatrix.googlecode.com/hg/matrix //the "edge" version
gomatrix.googlecode.com/hg/release/matrix //latest stable version,
happens to be v0.6
gomatrix.googlecode.com/hg/v0.6/matrix //previous release that happens
to be release
gomatrix.googlecode.com/hg/v0.5/matrix //previous release that is now
out-of-date

For source controls that support version information in the url, we
can continue using that. But for gomatrix, which uses mercurial, we
can do the above.

Thoughts?

- John

David Roundy

unread,
Feb 26, 2010, 5:56:58 PM2/26/10
to r...@golang.org, golang-nuts
Hi Russ,

Looks exciting!

Is there a reason why you don't do the build from within goinstall
itself, but instead generate a Makefile? I guess the advantage of
creating a Makefile and passing it off to make means that if the
included makefiles change, then you benefit from those changes for
"free" (since they are included). But on the other hand, you have a
dependency on make, and you have the complexity of both generating
correct input for make and running it properly.

I'm particularly curious about directions of the building back side
because I'm interested in making gotit template packages easy to
build. Teaching goinstall how to build .got files would be great, but
looks particularly complicated if it means both modifying the standard
Makefiles *and* modifying goinstall itself. The gotit approach to
templating (implemented most clearly in gotimports) is very close to
goinstall's approach to downloading: it recognizes special import
formats and builds the appropriate package for you. I think this is
compatible with goinstall, but using the two simultaneously would
obviously require some hacking. gotit isn't ready for prime time
(since noone but me has tried it), but I'd like to make it useful
enough that people *do* try it and try using packages built with it.

David

joseph.wecker

unread,
Feb 27, 2010, 9:53:55 PM2/27/10
to golang-nuts

> So there is need for being able to specify the version number of the
> library. If the version number is part of the import statement, then
> when a library is upgraded, one must change the version number in
> possibly hundreds of files. This smells likehttp://en.wikipedia.org/wiki/Shotgun_surgery

Someone performing shotgun surgery like that wouldn't be able to blame
goinstall, but instead the fragile dependencies they've been forced to
import.

I think this is great overall- it allows you to be as granular as you
need to be or as generic as you can be with your imports, including
versioning if/when the url-scheme + vcs allows it.

Marcelo Cantos

unread,
Feb 28, 2010, 12:04:59 AM2/28/10
to golang-nuts
On Feb 27, 3:36 am, Russ Cox <r...@golang.org> wrote:
> Thanks for the suggestion.  I'll take a look at it.
> Hash-based naming can solve the "keep things separate" part,
> but Go can already do that itself without hashes.
> The real issue, it seems, is whether you want a system that
> leads to having ten different versions of the same library linked
> into a single binary, because as soon as you can say
> "I need this specific version", that's where you're heading.

ZeroInstall operates at the dynamic binding level. At load time, it
binds environment variables such as LD_LIBRARY_PATH and PYTHONPATH so
that package A's executable can find package B's shared library. If B
gets an upgrade, the bindings change. If it's a static library you
want to "share", then the ideas in ZeroInstall may not suit unless you
can somehow bring them to bear on the build process.

chris dollin

unread,
Mar 16, 2010, 5:30:43 AM3/16/10
to r...@golang.org, golang-nuts
goinstall is now documented in yesterday's release (hooray). The
introductory paragraph of  http://golang.org/cmd/goinstall/ says:

  It maintains a list of public Go packages at http://godashboard.appspot.com/packages.

(a) Oops. "packages" should be "package".

(b) the URL should surely be a link. Mr Click is your friend!

Should I raise an issue?

Chris

--
Chris "allusive" Dollin

Andrew Gerrand

unread,
Mar 16, 2010, 5:34:51 AM3/16/10
to chris dollin, r...@golang.org, golang-nuts
Please do. I'll fix it tomorrow (if somebody else doesn't beat me to it. =)

chris dollin

unread,
Mar 16, 2010, 5:47:33 AM3/16/10
to Andrew Gerrand, r...@golang.org, golang-nuts
On 16 March 2010 09:34, Andrew Gerrand <a...@google.com> wrote:
Please do. I'll fix it tomorrow (if somebody else doesn't beat me to it. =)

OK, issue 668.

--
Chris "allusive" Dollin

chris dollin

unread,
Mar 16, 2010, 9:03:59 AM3/16/10
to r...@golang.org, golang-nuts
I've just done a (successful) install of web.go using goinstall. Why did I pick web.go?
Because when I looked at http://godashboard.appspot.com/package there was no
package-supplied summary line telling me what the package was about, web.go
had "web" in it, and it was vaguely near the top. (There's no way I would have guessed
that mustache.go was a templating package.)

Could we establish conventions for one-liner and short-summary texts to be
provided by the package?

Also I think times like 23:29:55.492225 might provide a little too much in the
way of precision, at least for the human reader. (For the machine, content
negotiation for json or rdf forms of the data, and all that yummy precision,
might be useful.)

David Roundy

unread,
Mar 16, 2010, 2:40:42 PM3/16/10
to Andrew Gerrand, golang-nuts
On Sun, Feb 28, 2010 at 5:50 PM, Andrew Gerrand <a...@google.com> wrote:

> On 27 February 2010 09:56, David Roundy <rou...@physics.oregonstate.edu> wrote:
>> Hi Russ,
>>
>> Looks exciting!
>>
>> Is there a reason why you don't do the build from within goinstall
>> itself, but instead generate a Makefile? I guess the advantage of
>> creating a Makefile and passing it off to make means that if the
>> included makefiles change, then you benefit from those changes for
>> "free" (since they are included).  But on the other hand, you have a
>> dependency on make, and you have the complexity of both generating
>> correct input for make and running it properly.
>
> The purpose of goinstall is to install a package and its dependencies.
> Having goinstall perform builds would complicate the tool. Having
> goinstall create makefiles just makes the external packages part of
> your existing build. If you change the way you build (eg,
> architecture), you need only rebuild as normal without involving
> goinstall.

I see. It's a subtle distinction. Then I will ask why install a
Makefile at all? Why not rely on the Makefile provided with the
package?

>> I'm particularly curious about directions of the building back side
>> because I'm interested in making gotit template packages easy to
>> build.  Teaching goinstall how to build .got files would be great, but
>> looks particularly complicated if it means both modifying the standard
>> Makefiles *and* modifying goinstall itself.  The gotit approach to
>> templating (implemented most clearly in gotimports) is very close to
>> goinstall's approach to downloading:  it recognizes special import
>> formats and builds the appropriate package for you.  I think this is
>> compatible with goinstall, but using the two simultaneously would
>> obviously require some hacking.  gotit isn't ready for prime time
>> (since noone but me has tried it), but I'd like to make it useful
>> enough that people *do* try it and try using packages built with it.
>

> Would making gotit compatible with the standard Makefile format solve
> this issue?

I'm not sure what you mean by that, unless you mean to modify the
"standard Makefile"?
--
David Roundy

Russ Cox

unread,
Mar 16, 2010, 4:10:27 PM3/16/10
to David Roundy, Andrew Gerrand, golang-nuts
>> Having goinstall perform builds would complicate the tool. Having
>> goinstall create makefiles just makes the external packages part of
>> your existing build. If you change the way you build (eg,
>> architecture), you need only rebuild as normal without involving
>> goinstall.
>
> I see.  It's a subtle distinction.  Then I will ask why install a
> Makefile at all? Why not rely on the Makefile provided with the
> package?

The main reason is to force the target install path to
be the one that goinstall thinks it should be. This lets
packages written without knowing about goinstall
or without using a Makefile still be installed with goinstall.

Now that goinstall is known, I hope people will change
their makefiles to install under the goinstall target path.
If the Makefile has a reasonable TARG= then it is
probably okay to use the distributed Makefile.

Russ

Andrew Gerrand

unread,
Mar 16, 2010, 7:17:53 PM3/16/10
to chris dollin, r...@golang.org, golang-nuts
On 17 March 2010 00:03, chris dollin <ehog....@googlemail.com> wrote:
> I've just done a (successful) install of web.go using goinstall. Why did I
> pick web.go?
> Because when I looked at http://godashboard.appspot.com/package there was no
> package-supplied summary line telling me what the package was about, web.go
> had "web" in it, and it was vaguely near the top. (There's no way I would
> have guessed
> that mustache.go was a templating package.)
>
> Could we establish conventions for one-liner and short-summary texts to be
> provided by the package?

I have been thinking about this. The information could be stored in a
special file included in the distribution, eg 'PACKAGE'. It would
contain the package name, description, website URL, logo URL, etc.

I'm also working on a rating system, so people can log into
godashboard and rate packages as good/bad. I'm not sure of the best
rating system though - it's a complex problem. Do you rate a
well-designed package badly if it's buggy? What if it's good but
totally unmaintained? Should individual ratings become less
influential on the overall rating as they age (so the more current
ratings have a greater influence)?

Andrew

Brian Slesinsky

unread,
Mar 17, 2010, 12:53:58 AM3/17/10
to golang-nuts
This looks elegant and useful. However, goinstall is nondeterministic
due to depending on the current date, and I suspect that could lead to
confusion. Within a single organization you'd want to check in
everything under $GOROOT into your own source control system after
downloading it, so I don't see that as much of a problem. But for
people at different organizations who want to reproduce each other's
results, perhaps it should take a date argument?

goinstall -date='2010-03-15 01:02' package1 package2

This should ignore any commits after the specified date and time (GMT
presumably), so that anyone running goinstall with a -date argument
downloads exactly the same version of each file. (Presuming that the
remote source control systems are sane.)

- Brian

> to update a list of what people are using.http://godashboard.appspot.com/package.


>
> This is a request for comments on the general approach.
> Please comment here, on the mailing list, rather than in the
> codereview site.  It will be easier to have a discussion in an
> actual discussion forum.
>
> -- documentation from goinstall/doc.go
>
> Goinstall is an experiment in automatic package installation.
> It installs packages, possibly downloading them from public version

> control systems, and it maintains a list of public Go packages athttp://godashboard.appspot.com/packages.

> of its most recent installation.  This mechanism powers the package list

> athttp://godashboard.appspot.com/packages, allowing Go programmers


> to learn about popular packages that might be worth looking at.

> The -dashboard=false flag disables this reporting.
>
> By default, goinstall prints output only when it encounters an error.
> The -v flag causes goinstall to print information about packages
> being considered and installed.
>
> Goinstall does not attempt to be a replacement for make.
> Instead, it invokes "make install" after locating the package sources.
> For local packages without a Makefile and all remote packages,
> goinstall creates and uses a temporary Makefile constructed from
> the import path and the list of Go files in the package.
>
> FAQ:
>
> • Why not use full URLs instead of hard-coding hosting sites?
>

> Goinstall is an experiment, and it is easier if the experiment does
> not involve changes to the compilers and godoc.  The paths used
> above double as file system paths, while full URLs do not.
> Also, the import path must also convey both the location of
> the data and the choice of version control system, so a full URL
> would not suffice.
>

> Using this notation does not precluding changing to other
> notation once we have more experience or in other contexts.
> For example, if Go were to run inside a browser it would make
> sense for import lines to be URLs of individual files rather than
> version control repositories.
>
> • Why isn't my favorite hosting site supported?
>
> Feel free to add it and send us a CL.

> Seehttp://golang.org/doc/contribute.html


>
> • How can I use goinstall with a package I commit changes to?
>
> Instead of having goinstall do the checkout automatically,
> check out your package yourself into the same location where
> goinstall would put it.  You can use that as your working copy
> and push to the world as appropriate.  For example, if you
> were working on github.com/hoisie/web.go, you check it out
> into $GOROOT/src/pkg/github.com/hoisie/web.go.git
> and then edit the files in that directory, pushing back to github
> as needed.  Goinstall will use the checked-out git repository
> instead of trying to download a fresh one.
>
> • What information is sent to and logged by the dashboard?
>
> The source code for both goinstall and the dashboard are
> in the Go repository:
>
>     $GOROOT/src/cmd/goinstall/download.go
>     $GOROOT/misc/dashboard/godashboard/package.py
>
> Goinstall sends the package import path.
> The dashboard increments a counter and sets the "most recently installed" time.
> There is no other reporting or logging.  The data recorded by the dashboard

> is publicly visible athttp://godashboard.appspot.com/package.

Russ Cox

unread,
Mar 17, 2010, 1:30:46 AM3/17/10
to Brian Slesinsky, golang-nuts
On Tue, Mar 16, 2010 at 21:53, Brian Slesinsky <skyb...@google.com> wrote:
> This looks elegant and useful. However, goinstall is nondeterministic
> due to depending on the current date, and I suspect that could lead to
> confusion. Within a single organization you'd want to check in
> everything under $GOROOT into your own source control system after
> downloading it, so I don't see that as much of a problem. But for
> people at different organizations who want to reproduce each other's
> results, perhaps it should take a date argument?
>
>  goinstall -date='2010-03-15 01:02' package1 package2
>
> This should ignore any commits after the specified date and time (GMT
> presumably), so that anyone running goinstall with a -date argument
> downloads exactly the same version of each file. (Presuming that the
> remote source control systems are sane.)

Versioning is a very hard problem, one that goinstall is not
attempt to solve right now. The command you gave may
be fine for one or two packages, but what if package1 needs
an older copy of package2 while package3 needs a newer one?
It gets complicated quickly.

Goinstall does have two decisions meant to help avoid
some versioning pain. The first is that by default goinstall
only downloads new packages; it does not update existing
packages unless you run it with -u. So if you're happy with
what you have, running goinstall to get one more package
won't break the ones you have. The second is that if a
package has a tag or branch named "release", then goinstall
will choose to use that one instead of tip or HEAD. So the
package author can check in as needed and occasionally
change the meaning of "release" when things are stable,
like we do with the Go releases.

These are clearly not a complete solution, but I don't know
what a complete solution would look like. I know that I don't
want to have N copies of every package just to get specific
versions for different dependencies.

Russ

Brian Slesinsky

unread,
Mar 17, 2010, 2:16:59 AM3/17/10
to r...@golang.org, golang-nuts
On Tue, Mar 16, 2010 at 10:30 PM, Russ Cox <r...@golang.org> wrote:
> Versioning is a very hard problem, one that goinstall is not
> attempt to solve right now.

Completely agreed and I'm not attempting to solve it.

> The command you gave may
> be fine for one or two packages, but what if package1 needs
> an older copy of package2 while package3 needs a newer one?

Don't do that. For consistency, use the same -date parameter for every
package you download.

The idea is that if you do a clean install in a new directory using
goinstall, you're compiling at "head" of all the Go packages on the
Internet (where "head" is the latest released version of each
package). Normally that should build, but in case it doesn't, we can
temporarily go back in time to get a working build until "head" is
fixed.

This isn't so different from what Google does internally. It's not
intended to be a complete solution to the versioning problem but
should make the current situation more livable.

(That said, I suppose we should wait until the actual problem comes up.)

- Brian

Hans Stimer

unread,
Mar 18, 2010, 12:29:12 AM3/18/10
to golang-nuts
In defense of a versionless goinstall:

1) Most of the time you just want the latest version
2) Most of the time, the imported package's imports will work with the
latest version
3) If you need to freeze you can just not goinstall
4) If you need older versions you can clone/fork pretty trivially and
just import the clone

I can live with this. I would rather have something under featured
than over featured like CORBA, Java, C++, and J2EE, or any number of
other committee designed abominations.

btw - goinstall updating http://godashboard.appspot.com/packages is a
clever idea. However I think the web site is broken; I'm getting a
blank page.

Andrew Gerrand

unread,
Mar 18, 2010, 12:34:52 AM3/18/10
to Hans Stimer, golang-nuts
This is a typo in the docs. The correct url is
http://godashboard.appspot.com/package

Andrew

Hans Stimer

unread,
Mar 18, 2010, 4:10:01 PM3/18/10
to golang-nuts
Suggestions for http://godashboard.appspot.com/package
---
1) list the package name -- it isn't always apparent from the path
e.g. golang.googlecode.com
2) list the actual include path -- the paths listed don't always work;
sometimes the actual package is buried e.g. golang.googlecode.com
3) don't clump multiple packages into a single listing if they are all
at root level within the same repository e.g. golang.googlecode.com
4) show clones and their relationship to the base package -- helps you
to decide if you want the main package or one of it's clones
5) mark packages busted if they don't build -- lets not waste people's
time trying to install busted packages
6) show which versions of go and dependent packages are known to build
the package

I love the realtime/realworld reporting on packages at godashboard.
Excellent idea. At least until some idiot starts posting false info.


Suggestions for goinstall
---
1) add option for showing clones of installed packages -- helps you to
decide if you should be using a clone instead
2) add option for package searching, and showing clones -- maybe do a
text search of the readmes, or maybe even standardize on the readmes
(markdown anyone?), or standardize embedded package docs in the code
3) add option for showing when clones are pulled back into the main
trunk -- to help decide when you can abandon a bug fix clone and go
back to the main branch

David Roundy

unread,
Mar 19, 2010, 1:11:24 PM3/19/10
to r...@golang.org, Andrew Gerrand, golang-nuts
On Tue, Mar 16, 2010 at 1:10 PM, Russ Cox <r...@golang.org> wrote:
>> Makefile at all? Why not rely on the Makefile provided with the
>> package?
>
> The main reason is to force the target install path to
> be the one that goinstall thinks it should be.  This lets
> packages written without knowing about goinstall
> or without using a Makefile still be installed with goinstall.
>
> Now that goinstall is known, I hope people will change
> their makefiles to install under the goinstall target path.
> If the Makefile has a reasonable TARG= then it is
> probably okay to use the distributed Makefile.

What do you mean by a reasonable TARG=? It seems like it'd be bad to
mention the source in the TARG line, since it means that temporary
forks would have to change the Makefile. e.g. if I make a
github-hosted clone of web.go, I shouldn't have to modify the Makefile
in order for it to goinstall into a different directory.

To be honest, I'm still a little unclear as to how goinstall is
working. Where does it install the Makefile that it generates? When I
run

goinstall github.com/hoisie/web.go

The package is installed in $GOROOT/pkg/github.com/hoisie/web.go.a

But then if I do

cd $GOROOT/src/pkg/github.com/hoisie/web.go
make install

It gets installed as $GOROOT/pkg/web.a

I guess this is because I am now using the package's included
makefile. But where did the generated one go?

One appealing (to me) possibility would be to pass the installation
path to make as an environment variable. You could then check if the
existing Makefile mentions that variable, and if it does, you could
use it, otherwise try generating your own? (and if you keep the check
simple, then the Makefile could mention the variable name in a
comment, in case the build system is more complicated and doesn't
actually use the variable directly in the Makefile (which might itself
call something else that *does* check the install target).
--
David Roundy

Russ Cox

unread,
Mar 19, 2010, 1:16:57 PM3/19/10
to David Roundy, golang-nuts
> What do you mean by a reasonable TARG=? It seems like it'd be bad to
> mention the source in the TARG line, since it means that temporary
> forks would have to change the Makefile. e.g. if I make a
> github-hosted clone of web.go, I shouldn't have to modify the Makefile
> in order for it to goinstall into a different directory.

That's an interesting argument, but it's only a single line change.
And right now goinstall does behave as you say:
you don't need to include a Makefile at all.

> I guess this is because I am now using the package's included
> makefile.  But where did the generated one go?

It went to make's standard input.

> One appealing (to me) possibility would be to pass the installation
> path to make as an environment variable.

This fails when goinstall recurses to get dependencies.
There really needs to be a standard algorithm to translate
source location to install path.

Russ

David Roundy

unread,
Mar 19, 2010, 5:02:21 PM3/19/10
to r...@golang.org, golang-nuts
On Fri, Mar 19, 2010 at 10:16 AM, Russ Cox <r...@golang.org> wrote:
>> What do  you mean by a reasonable TARG=? It seems like it'd be bad to
>> mention the source in the TARG line, since it means that temporary
>> forks would have to change the Makefile.  e.g. if I make a
>> github-hosted clone of web.go, I shouldn't have to modify the Makefile
>> in order for it to goinstall into a different directory.
>
> That's an interesting argument, but it's only a single line change.
> And right now goinstall does behave as you say:
> you don't need to include a Makefile at all.

It also won't install any package that I've written, so this question
does seem relevant to me. It looks like in order to allow goinstall
to work, I would need to give up any use of local private packages
(imported with "./foo/bar"). I don't want these packages to be
installed globally, and that seems to be the only option when using
the "standard" makefile (and hence goinstall).

goinstall seems to be going the direction of cabal (see
http://www.haskell.org/cabal/), which is to create an easy
installation tool, which embeds a simplistic build system. The result
(with cabal, but hopefully not with goinstall in the end) is that you
have a tool that works for the most simple of programs, but fails to
work properly for more complicated programs. (On other aspects of why
cabal is bad, you could talk with John Meacham, who is now employed at
Google...)

>> One appealing (to me) possibility would be to pass the installation
>> path to make as an environment variable.
>
> This fails when goinstall recurses to get dependencies.
> There really needs to be a standard algorithm to translate
> source location to install path.

I don't quite follow. I didn't mean to suggest anything to the
contrary, just that by passing the install path as an environment
variable you could allow developers to create our own Makefiles that
would work with goinstall.

David

Russ Cox

unread,
Mar 19, 2010, 5:10:20 PM3/19/10
to David Roundy, golang-nuts
> It also won't install any package that I've written, so this question
> does seem relevant to me.  It looks like in order to allow goinstall
> to work, I would need to give up any use of local private packages
> (imported with "./foo/bar").  I don't want these packages to be
> installed globally, and that seems to be the only option when using
> the "standard" makefile (and hence goinstall).

There's no such thing as a local private package.
You can pretend that, but 6l needs to find all the
object files when it links everything, so you'd have
to keep those object files around, both for yourself
and any clients of the package. It's much clearer
to install them in the standard place than have
things grubbing around in the build directories.
In general imports of ./anything are for special cases,
not for general use. They're not as private as you think.

> goinstall seems to be going the direction of cabal (see
> http://www.haskell.org/cabal/), which is to create an easy
> installation tool, which embeds a simplistic build system.  The result
> (with cabal, but hopefully not with goinstall in the end) is that you
> have a tool that works for the most simple of programs, but fails to
> work properly for more complicated programs.  (On other aspects of why
> cabal is bad, you could talk with John Meacham, who is now employed at
> Google...)

We're certainly trying to build something that works for
complicated programs.

> I don't quite follow.  I didn't mean to suggest anything to the
> contrary, just that by passing the install path as an environment
> variable you could allow developers to create our own Makefiles that
> would work with goinstall.

If you goinstall a package that has import lines that
refer to other packages, goinstall builds those first,
downloading them if necessary. The environment
variable could only possibly apply to the top level
of the recursion.

Russ

David Roundy

unread,
Mar 20, 2010, 10:47:05 AM3/20/10
to r...@golang.org, golang-nuts
On Fri, Mar 19, 2010 at 2:10 PM, Russ Cox <r...@golang.org> wrote:
>> It also won't install any package that I've written, so this question
>> does seem relevant to me.  It looks like in order to allow goinstall
>> to work, I would need to give up any use of local private packages
>> (imported with "./foo/bar").  I don't want these packages to be
>> installed globally, and that seems to be the only option when using
>> the "standard" makefile (and hence goinstall).
>
> There's no such thing as a local private package.
> You can pretend that, but 6l needs to find all the
> object files when it links everything, so you'd have
> to keep those object files around, both for yourself
> and any clients of the package.  It's much clearer
> to install them in the standard place than have
> things grubbing around in the build directories.
> In general imports of ./anything are for special cases,
> not for general use.  They're not as private as you think.

Okay, I hadn't realized this. You're right, that's ugly, and I'll
change my code to avoid doing that.

For applications, rather than packages, is there a problem with local
imports of the ./foo form? I don't like my program build process to
install anything until the build is complete, and would prefer that it
doesn't install anything but the application itself. I guess I could
also use -I and -L to avoid installing packages. Or, of course, I
could put everything in package main, but that doesn't work so nicely
if I want to use shared code for several programs.

I guess the next question is how to compile a multi-package package
(i.e. one that depends on a sub-package) without installing anything.
I guess the answer must be to use -I?

> We're certainly trying to build something that works for
> complicated programs.

Good! (Then I hope I'm not annoying you by trying to get my
complicated programs to work...)

At the moment, I've modified all my code to use gotgo for their slice
handling. I store the generated packages in each repository, so users
shouldn't need to have gotgo itself installed, and the Makefile
(hopefully) works with this. But currently the generated packages
aren't always suitable for public consumption, since they may refer to
data types by qualified import names, which could clash with another
package's use.

Currently, gotgo's import syntax looks like:

import errorslice "gotgo/slice(os.Error)"

which will import a package for concatenating, importing, filtering,
etc on []os.Error. The ugliness shows up due to issues with package
naming. I don't want template expansion to require looking into the
contents of other imported packages (which potentially haven't yet
been built or installed, and might thus be hard to find), so I require
that "os" above is imported via:

import os "os"

so that I know that os.Error refers to the package imported as "os".
But at this point, I have no guarantee that you didn't write something
like

import os "io"

in which case we couldn't share this package with anyone else, because
the qualified import name isn't unique. So I guess maybe what I need
to do is use a syntax that directly uses import strings, which is much
uglier, and would look something like:

import fooslice "github.com/droundy/gotgo/slice(github.com/droundy/foo.Type)"

Does this sound like a reasonable sort of approach to you? Then the
build process would involve gotgo first looking through the imports
and creating any of those packages, and *then* goinstall looks through
and builds the remainder. And if gotgo were to be accepted into the
core, then goinstall could do both steps.

>> I don't quite follow.  I didn't mean to suggest anything to the
>> contrary, just that by passing the install path as an environment
>> variable you could allow developers to create our own Makefiles that
>> would work with goinstall.
>
> If you goinstall a package that has import lines that
> refer to other packages, goinstall builds those first,
> downloading them if necessary.  The environment
> variable could only possibly apply to the top level
> of the recursion.

Hmmm. I'd have thought that the recursive goinstall would simply
redefine that environment variable, since it knows it's building a
different package.
--
David Roundy

Russ Cox

unread,
Mar 20, 2010, 1:32:26 PM3/20/10
to David Roundy, golang-nuts
> For applications, rather than packages, is there a problem with local
> imports of the ./foo form? I don't like my program build process to
> install anything until the build is complete, and would prefer that it
> doesn't install anything but the application itself.  I guess I could
> also use -I and -L to avoid installing packages.  Or, of course, I
> could put everything in package main, but that doesn't work so nicely
> if I want to use shared code for several programs.

It really sounds like you want to install the packages.
At some point we'll have to grapple with having multiple
places where packages can live, and then maybe you
won't be so opposed to installing them if they can go
in some private place?

> how to compile a multi-package package
> (i.e. one that depends on a sub-package) without installing anything.
> I guess the answer must be to use -I?

Just install them.

> import fooslice "github.com/droundy/gotgo/slice(github.com/droundy/foo.Type)"
>
> Does this sound like a reasonable sort of approach to you?

I don't think it makes sense to change the design of imports or
installation to adapt to gotgo. gotgo is a fun experiment, and I hope
it leads to a nice system, but if so that system can't be import-based,
because:

* I have to write a different import line for every kind of vector I want.
* I can't use a vector of type T inside the package that defines type T,
because the imports would create a cycle.
* I can't use a vector of type t ever, because the imported package
cannot talk about my non-exported type t.

Those all seem like show stoppers to me, and they're fundamental to
any import-based approach to generics. Again, I hope that experience
with gotgo will help us learn more about the eventual solution, but I don't
think it's close enough to being that solution to adjust the import or
install mechanisms to accommodate it.

Russ

David Roundy

unread,
Mar 20, 2010, 3:35:23 PM3/20/10
to r...@golang.org, golang-nuts
On Sat, Mar 20, 2010 at 10:32 AM, Russ Cox <r...@golang.org> wrote:
>> For applications, rather than packages, is there a problem with local
>> imports of the ./foo form? I don't like my program build process to
>> install anything until the build is complete, and would prefer that it
>> doesn't install anything but the application itself.  I guess I could
>> also use -I and -L to avoid installing packages.  Or, of course, I
>> could put everything in package main, but that doesn't work so nicely
>> if I want to use shared code for several programs.
>
> It really sounds like you want to install the packages.
> At some point we'll have to grapple with having multiple
> places where packages can live, and then maybe you
> won't be so opposed to installing them if they can go
> in some private place?

The problem is that then I can't easily have two versions of my code
on the same machine. If I compile the experimental one, then it
installs the experimental packages. Then if I compile the bugfix
code, it links with the experimental packages. The only way I see to
solve this is to put all my code into the main package, which doesn't
seem too fit well with the idea of minimal recompiles.

>> import fooslice "github.com/droundy/gotgo/slice(github.com/droundy/foo.Type)"
>>
>> Does this sound like a reasonable sort of approach to you?
>
> I don't think it makes sense to change the design of imports or
> installation to adapt to gotgo.  gotgo is a fun experiment, and I hope
> it leads to a nice system, but if so that system can't be import-based,
> because:
>
>  * I have to write a different import line for every kind of vector I want.
>  * I can't use a vector of type T inside the package that defines type T,
>    because the imports would create a cycle.
>  * I can't use a vector of type t ever, because the imported package
>    cannot talk about my non-exported type t.

Yay, the first substantive feedback I've gotten on gotgo! :) Thanks!

> Those all seem like show stoppers to me, and they're fundamental to
> any import-based approach to generics.  Again, I hope that experience
> with gotgo will help us learn more about the eventual solution, but I don't
> think it's close enough to being that solution to adjust the import or
> install mechanisms to accommodate it.

I agree that I need to have a non-import way to use gotgo, and have
already implemented such a mechanism (not yet committed or pushed).
But it also seems important to include an import-based approach,
otherwise any generic library needs to be recompiled every time
anything that uses it is compiled, which brings us down to C++-level
compile time, so far as I can see.
--
David Roundy

Esko Luontola

unread,
Mar 20, 2010, 5:32:08 PM3/20/10
to golang-nuts
On Mar 20, 9:35 pm, David Roundy <roun...@physics.oregonstate.edu>
wrote:

> On Sat, Mar 20, 2010 at 10:32 AM, Russ Cox <r...@golang.org> wrote:
> >> For applications, rather than packages, is there a problem with local
> >> imports of the ./foo form? I don't like my program build process to
> >> install anything until the build is complete, and would prefer that it
> >> doesn't install anything but the application itself.  I guess I could
> >> also use -I and -L to avoid installing packages.  Or, of course, I
> >> could put everything in package main, but that doesn't work so nicely
> >> if I want to use shared code for several programs.
>
> > It really sounds like you want to install the packages.
> > At some point we'll have to grapple with having multiple
> > places where packages can live, and then maybe you
> > won't be so opposed to installing them if they can go
> > in some private place?
>
> The problem is that then I can't easily have two versions of my code
> on the same machine.  If I compile the experimental one, then it
> installs the experimental packages.  Then if I compile the bugfix
> code, it links with the experimental packages.  The only way I see to
> solve this is to put all my code into the main package, which doesn't
> seem too fit well with the idea of minimal recompiles.

I'm (very slowly) working on a build tool whose goal is to allow just
that: allow building and testing projects without requiring to install
them. Makefiles are just too cumbersome when you've gotten used to the
tools that for example the Java and Ruby communities have.

My biggest problem with Makefiles (which use "$(GOROOT)/src/Make.*")
was that in a multi-package project it's not possible to run tests
(with gotest) which use more than one of the current project's
packages, unless all of them are first installed. This in turn means
that during development, you are required to install broken code (code
which does not pass the tests) hundreds of times per day (when using
TDD), which in turn means that developing a library and using it in
other projects at the same time is very cumbersome, because those
other projects can not depend on a stable version of the library.

The second problem with Makefiles is that you need to manually
maintain a list of all the files which belong to the project, when the
build tool could find them automatically.

Giles Lean

unread,
Mar 20, 2010, 8:46:27 PM3/20/10
to David Roundy, golang-nuts

David Roundy <rou...@physics.oregonstate.edu> wrote:

> The problem is that then I can't easily have two versions of my code
> on the same machine. If I compile the experimental one, then it
> installs the experimental packages. Then if I compile the bugfix
> code, it links with the experimental packages. The only way I see to
> solve this is to put all my code into the main package, which doesn't
> seem too fit well with the idea of minimal recompiles.

I have multiple installations (last release, tip, feature I'm
working on, local package(s)) and switch $PATH by sourcing a
script, e.g.:

$ . ./set_env

when I switch from one to another. set_env says something like:

$ cat set_env
export GOARCH=amd64
export GOOS=darwin
export GOROOT=/Users/giles/Projects/go-release/go
export GOBIN=/Users/giles/Projects/go-release/bin

. "$HOME/.path"
PATH="$GOBIN:$PATH"

I agree with you that this doesn't reduce the number of
compiles a lot, but with the speed of the 6g/8g compilers we
have to do /something/ to restore coffee drinking and mailing
list reading time so it's not too bad in practice. :-/

What is annoying of course is building a package with the
"wrong" environment variables set; for that I really have no
solution except to wrap 'make' in a function that checks $PWD
aginst $GOROOT, which I might yet do but haven't got around to
yet.

Keeping seperate trees around also has the advantage of
letting me use mercurial queues (or branches if I wanted).
The only trees that need the codereview extension in them are
ones I plan to submit code from.

Your milage may vary ....

Giles

Reply all
Reply to author
Forward
0 new messages