Re: [go-nuts] Re: goinstall: an experiment in package installation
Russ Cox
> "execute" a URL. ZeroInstall grabs and caches the package, recursively
> following its dependencies. Dependencies denote versioning and
> architecture as well. Two users on the same system can fetch different
> versions of the same product by using slightly different URL's, and
> ZeroInstall fetches any shared dependencies just once (That's the
> theory, anyway; this wasn't implemented when I last looked.) The whole
> system is based on strong hashing, so naming conflicts are basically
> impossible.
Thanks for the suggestion. I'll take a look at it.
Hash-based naming can solve the "keep things separate" part,
but Go can already do that itself without hashes.
The real issue, it seems, is whether you want a system that
leads to having ten different versions of the same library linked
into a single binary, because as soon as you can say
"I need this specific version", that's where you're heading.
> P.S.: +1 vote for "goget"
The name goget is great. It's certainly catchier than goinstall.
I'm a little hesitant though, because the meaning is right
for my examples but not the day-to-day use I envision.
I've been writing examples in this thread like
goinstall mimeparse.googlecode.com/svn/trunk/go
to illustrate what the import paths look like, but I don't
expect that to be a common usage pattern. Instead,
I hope people will write their own packages in some local
directory my/pkg with import statements like
import "mimeparse.googlecode.com/svn/trunk/go"
and then run
goinstall my/pkg
which will grab the dependencies as needed.
This way avoids the repetition of having to write the same
string both in an import statement and on a command line.
For that example,
goget my/pkg
doesn't look quite right: the command's not getting my/pkg
at all, because I wrote my/pkg and it's already here.
Russ
John Asmuth
In the source control directory, we could add a directory whose name
means something. One directory would be the name of the package (to
make it obvious how to refer to it in code). It can then be prefixed
with version information, or not in the case of edge.
Requires slightly more work by the developer, but only about 30
seconds per release. Pretty reasonable.
gomatrix.googlecode.com/hg/matrix //the "edge" version
gomatrix.googlecode.com/hg/release/matrix //latest stable version,
happens to be v0.6
gomatrix.googlecode.com/hg/v0.6/matrix //previous release that happens
to be release
gomatrix.googlecode.com/hg/v0.5/matrix //previous release that is now
out-of-date
For source controls that support version information in the url, we
can continue using that. But for gomatrix, which uses mercurial, we
can do the above.
Thoughts?
- John
On Feb 25, 7:43 pm, Russ Cox <r...@golang.org> wrote:
> All the problems about versioning are orthogonal to goinstall.
> They are fundamental to any system in which you're using
> multiple code bases that progress independently.
> Solving that problem is explicitly not a goal for goinstall today.
> I'd be more than happy for goinstall to solve the versioning
> problem later, if we can figure out how.
>
> I am not dismissing the problem. I just think it is difficult and
> not any different for goinstall than it is for any other software
> system.
>
> Russ
David Roundy
Looks exciting!
Is there a reason why you don't do the build from within goinstall
itself, but instead generate a Makefile? I guess the advantage of
creating a Makefile and passing it off to make means that if the
included makefiles change, then you benefit from those changes for
"free" (since they are included). But on the other hand, you have a
dependency on make, and you have the complexity of both generating
correct input for make and running it properly.
I'm particularly curious about directions of the building back side
because I'm interested in making gotit template packages easy to
build. Teaching goinstall how to build .got files would be great, but
looks particularly complicated if it means both modifying the standard
Makefiles *and* modifying goinstall itself. The gotit approach to
templating (implemented most clearly in gotimports) is very close to
goinstall's approach to downloading: it recognizes special import
formats and builds the appropriate package for you. I think this is
compatible with goinstall, but using the two simultaneously would
obviously require some hacking. gotit isn't ready for prime time
(since noone but me has tried it), but I'd like to make it useful
enough that people *do* try it and try using packages built with it.
David
joseph.wecker
> So there is need for being able to specify the version number of the
> library. If the version number is part of the import statement, then
> when a library is upgraded, one must change the version number in
> possibly hundreds of files. This smells likehttp://en.wikipedia.org/wiki/Shotgun_surgery
Someone performing shotgun surgery like that wouldn't be able to blame
goinstall, but instead the fragile dependencies they've been forced to
import.
I think this is great overall- it allows you to be as granular as you
need to be or as generic as you can be with your imports, including
versioning if/when the url-scheme + vcs allows it.
Marcelo Cantos
> Thanks for the suggestion. I'll take a look at it.
> Hash-based naming can solve the "keep things separate" part,
> but Go can already do that itself without hashes.
> The real issue, it seems, is whether you want a system that
> leads to having ten different versions of the same library linked
> into a single binary, because as soon as you can say
> "I need this specific version", that's where you're heading.
ZeroInstall operates at the dynamic binding level. At load time, it
binds environment variables such as LD_LIBRARY_PATH and PYTHONPATH so
that package A's executable can find package B's shared library. If B
gets an upgrade, the bindings change. If it's a static library you
want to "share", then the ideas in ZeroInstall may not suit unless you
can somehow bring them to bear on the build process.
chris dollin
>> of its most recent installation. This mechanism powers the package listSorry, typo. The correct URL is http://godashboard.appspot.com/package.
>> at http://godashboard.appspot.com/packages, allowing Go programmers
>> to learn about popular packages that might be worth looking at.
>
> That link currently serves an empty page, no text, no HTML, nothing, not
> even a Coming Soon; is that as expected?
introductory paragraph of http://golang.org/cmd/goinstall/ says:
It maintains a list of public Go packages at http://godashboard.appspot.com/packages.
(a) Oops. "packages" should be "package".
(b) the URL should surely be a link. Mr Click is your friend!
Should I raise an issue?
Chris
--
Chris "allusive" Dollin
Andrew Gerrand
chris dollin
Please do. I'll fix it tomorrow (if somebody else doesn't beat me to it. =)
OK, issue 668.
--
Chris "allusive" Dollin
chris dollin
Because when I looked at http://godashboard.appspot.com/package there was no
package-supplied summary line telling me what the package was about, web.go
had "web" in it, and it was vaguely near the top. (There's no way I would have guessed
that mustache.go was a templating package.)
Could we establish conventions for one-liner and short-summary texts to be
provided by the package?
Also I think times like 23:29:55.492225 might provide a little too much in the
way of precision, at least for the human reader. (For the machine, content
negotiation for json or rdf forms of the data, and all that yummy precision,
might be useful.)
David Roundy
> On 27 February 2010 09:56, David Roundy <rou...@physics.oregonstate.edu> wrote:
>> Hi Russ,
>>
>> Looks exciting!
>>
>> Is there a reason why you don't do the build from within goinstall
>> itself, but instead generate a Makefile? I guess the advantage of
>> creating a Makefile and passing it off to make means that if the
>> included makefiles change, then you benefit from those changes for
>> "free" (since they are included). But on the other hand, you have a
>> dependency on make, and you have the complexity of both generating
>> correct input for make and running it properly.
>
> Having goinstall perform builds would complicate the tool. Having
> goinstall create makefiles just makes the external packages part of
> your existing build. If you change the way you build (eg,
> architecture), you need only rebuild as normal without involving
> goinstall.
I see. It's a subtle distinction. Then I will ask why install a
Makefile at all? Why not rely on the Makefile provided with the
package?
>> I'm particularly curious about directions of the building back side
>> because I'm interested in making gotit template packages easy to
>> build. Teaching goinstall how to build .got files would be great, but
>> looks particularly complicated if it means both modifying the standard
>> Makefiles *and* modifying goinstall itself. The gotit approach to
>> templating (implemented most clearly in gotimports) is very close to
>> goinstall's approach to downloading: it recognizes special import
>> formats and builds the appropriate package for you. I think this is
>> compatible with goinstall, but using the two simultaneously would
>> obviously require some hacking. gotit isn't ready for prime time
>> (since noone but me has tried it), but I'd like to make it useful
>> enough that people *do* try it and try using packages built with it.
>
> Would making gotit compatible with the standard Makefile format solve
> this issue?
I'm not sure what you mean by that, unless you mean to modify the
"standard Makefile"?
--
David Roundy
Russ Cox
>> goinstall create makefiles just makes the external packages part of
>> your existing build. If you change the way you build (eg,
>> architecture), you need only rebuild as normal without involving
>> goinstall.
>
> I see. It's a subtle distinction. Then I will ask why install a
> Makefile at all? Why not rely on the Makefile provided with the
> package?
The main reason is to force the target install path to
be the one that goinstall thinks it should be. This lets
packages written without knowing about goinstall
or without using a Makefile still be installed with goinstall.
Now that goinstall is known, I hope people will change
their makefiles to install under the goinstall target path.
If the Makefile has a reasonable TARG= then it is
probably okay to use the distributed Makefile.
Russ
Andrew Gerrand
> I've just done a (successful) install of web.go using goinstall. Why did I
> pick web.go?
> Because when I looked at http://godashboard.appspot.com/package there was no
> package-supplied summary line telling me what the package was about, web.go
> had "web" in it, and it was vaguely near the top. (There's no way I would
> have guessed
> that mustache.go was a templating package.)
>
> Could we establish conventions for one-liner and short-summary texts to be
> provided by the package?
I have been thinking about this. The information could be stored in a
special file included in the distribution, eg 'PACKAGE'. It would
contain the package name, description, website URL, logo URL, etc.
I'm also working on a rating system, so people can log into
godashboard and rate packages as good/bad. I'm not sure of the best
rating system though - it's a complex problem. Do you rate a
well-designed package badly if it's buggy? What if it's good but
totally unmaintained? Should individual ratings become less
influential on the overall rating as they age (so the more current
ratings have a greater influence)?
Andrew
Brian Slesinsky
due to depending on the current date, and I suspect that could lead to
confusion. Within a single organization you'd want to check in
everything under $GOROOT into your own source control system after
downloading it, so I don't see that as much of a problem. But for
people at different organizations who want to reproduce each other's
results, perhaps it should take a date argument?
goinstall -date='2010-03-15 01:02' package1 package2
This should ignore any commits after the specified date and time (GMT
presumably), so that anyone running goinstall with a -date argument
downloads exactly the same version of each file. (Presuming that the
remote source control systems are sane.)
- Brian
On Feb 24, 4:41 pm, Russ Cox <r...@golang.org> wrote:
> CL 224043, not yet sent for review, would add a new command
> goinstall that can download packages into the local file system.
> Rather than establish a central naming authority for Go packages
> (witness the multiple mysqlgo packages), it uses import paths
> that contain the names of common hosting sites. The full docs
> are below.
>
> Goinstall already works with standalone packages that
> don't depend on other packages. For example, I can run:
>
> goinstall github.com/hoisie/web.go.git
> goinstall github.com/jacobsa/igo.git/set
> goinstall go-avltree.googlecode.com/svn/trunk
> goinstall mimeparse.googlecode.com/svn/trunk/go
> goinstall gomatrix.googlecode.com/hg
> goinstall golang.googlecode.com/hg/src/pkg/scanner
>
> And then import any of those using the same paths.
> If those had referred to other import paths of the same
> form, goinstall would have downloaded the dependencies,
> recursively. So in your own package, you can put
> a line like import "gomatrix.googlecode.com/hg" and
> then just run
>
> goinstall my/package
>
> which will notice the import line and download and install
> the remote package. The first time goinstall downloads
> and installs a public package, it pings the Go dashboard
> to update a list of what people are using.http://godashboard.appspot.com/package.
>
> This is a request for comments on the general approach.
> Please comment here, on the mailing list, rather than in the
> codereview site. It will be easier to have a discussion in an
> actual discussion forum.
>
> -- documentation from goinstall/doc.go
>
> Goinstall is an experiment in automatic package installation.
> It installs packages, possibly downloading them from public version
> control systems, and it maintains a list of public Go packages athttp://godashboard.appspot.com/packages.
>
> Usage:
> goinstall [flags] importpath...
>
> Flags and default settings:
> -dashboard=true tally public packages on godashboard.appspot.com
> -update=false update already-downloaded packages
> -v=false verbose operation
>
> Goinstall installs each of the packages identified by import path on
> the command line. It installs a package's prerequisites before
> trying to install the package itself.
>
> The source code for a package with import path foo/bar is expected
> to be in the directory $GOROOT/src/pkg/foo/bar/. If the import
> path refers to a code hosting site, goinstall will download it if necessary.
> The recognized code hosting sites are:
>
> GitHub (Git)
>
> import "github.com/user/project.git"
> import "github.com/user/project.git/sub/directory"
>
> Google Code Project Hosting (Mercurial, Subversion)
>
> import "project.googlecode.com/hg"
> import "project.googlecode.com/hg/sub/directory"
>
> import "project.googlecode.com/svn/trunk"
> import "project.googlecode.com/svn/trunk/sub/directory"
>
> If the directory (e.g., $GOROOT/src/pkg/bitbucket.org/user/project)
> already exists and contains an appropriate checkout, goinstall will not
> attempt to fetch updates. The -update flag changes this behavior,
> causing goinstall to update all remote packages encountered during
> the installation.
>
> When downloading or updating, goinstall first looks for a tag or branch
> named "release". If there is one, it uses that version of the code.
> Otherwise it uses the default version selected by the version control
> system (typically HEAD for git, tip for Mercurial).
>
> After a successful download and installation of a publicly accessible
> remote package, goinstall reports the installation to godashboard.appspot.com,
> which increments a count associated with the package and the time
> of its most recent installation. This mechanism powers the package list
> athttp://godashboard.appspot.com/packages, allowing Go programmers
> to learn about popular packages that might be worth looking at.
> The -dashboard=false flag disables this reporting.
>
> By default, goinstall prints output only when it encounters an error.
> The -v flag causes goinstall to print information about packages
> being considered and installed.
>
> Goinstall does not attempt to be a replacement for make.
> Instead, it invokes "make install" after locating the package sources.
> For local packages without a Makefile and all remote packages,
> goinstall creates and uses a temporary Makefile constructed from
> the import path and the list of Go files in the package.
>
> FAQ:
>
> • Why not use full URLs instead of hard-coding hosting sites?
>
> Goinstall is an experiment, and it is easier if the experiment does
> not involve changes to the compilers and godoc. The paths used
> above double as file system paths, while full URLs do not.
> Also, the import path must also convey both the location of
> the data and the choice of version control system, so a full URL
> would not suffice.
>
> Using this notation does not precluding changing to other
> notation once we have more experience or in other contexts.
> For example, if Go were to run inside a browser it would make
> sense for import lines to be URLs of individual files rather than
> version control repositories.
>
> • Why isn't my favorite hosting site supported?
>
> Feel free to add it and send us a CL.
> Seehttp://golang.org/doc/contribute.html
>
> • How can I use goinstall with a package I commit changes to?
>
> Instead of having goinstall do the checkout automatically,
> check out your package yourself into the same location where
> goinstall would put it. You can use that as your working copy
> and push to the world as appropriate. For example, if you
> were working on github.com/hoisie/web.go, you check it out
> into $GOROOT/src/pkg/github.com/hoisie/web.go.git
> and then edit the files in that directory, pushing back to github
> as needed. Goinstall will use the checked-out git repository
> instead of trying to download a fresh one.
>
> • What information is sent to and logged by the dashboard?
>
> The source code for both goinstall and the dashboard are
> in the Go repository:
>
> $GOROOT/src/cmd/goinstall/download.go
> $GOROOT/misc/dashboard/godashboard/package.py
>
> Goinstall sends the package import path.
> The dashboard increments a counter and sets the "most recently installed" time.
> There is no other reporting or logging. The data recorded by the dashboard
> is publicly visible athttp://godashboard.appspot.com/package.
>
> -- end documentation
>
> Again, comments welcome, but please post them here rather
> than on the code review site.
>
> Thanks.
> Russ
Russ Cox
> This looks elegant and useful. However, goinstall is nondeterministic
> due to depending on the current date, and I suspect that could lead to
> confusion. Within a single organization you'd want to check in
> everything under $GOROOT into your own source control system after
> downloading it, so I don't see that as much of a problem. But for
> people at different organizations who want to reproduce each other's
> results, perhaps it should take a date argument?
>
> goinstall -date='2010-03-15 01:02' package1 package2
>
> This should ignore any commits after the specified date and time (GMT
> presumably), so that anyone running goinstall with a -date argument
> downloads exactly the same version of each file. (Presuming that the
> remote source control systems are sane.)
Versioning is a very hard problem, one that goinstall is not
attempt to solve right now. The command you gave may
be fine for one or two packages, but what if package1 needs
an older copy of package2 while package3 needs a newer one?
It gets complicated quickly.
Goinstall does have two decisions meant to help avoid
some versioning pain. The first is that by default goinstall
only downloads new packages; it does not update existing
packages unless you run it with -u. So if you're happy with
what you have, running goinstall to get one more package
won't break the ones you have. The second is that if a
package has a tag or branch named "release", then goinstall
will choose to use that one instead of tip or HEAD. So the
package author can check in as needed and occasionally
change the meaning of "release" when things are stable,
like we do with the Go releases.
These are clearly not a complete solution, but I don't know
what a complete solution would look like. I know that I don't
want to have N copies of every package just to get specific
versions for different dependencies.
Russ
Brian Slesinsky
> Versioning is a very hard problem, one that goinstall is not
> attempt to solve right now.
Completely agreed and I'm not attempting to solve it.
> The command you gave may
> be fine for one or two packages, but what if package1 needs
> an older copy of package2 while package3 needs a newer one?
Don't do that. For consistency, use the same -date parameter for every
package you download.
The idea is that if you do a clean install in a new directory using
goinstall, you're compiling at "head" of all the Go packages on the
Internet (where "head" is the latest released version of each
package). Normally that should build, but in case it doesn't, we can
temporarily go back in time to get a working build until "head" is
fixed.
This isn't so different from what Google does internally. It's not
intended to be a complete solution to the versioning problem but
should make the current situation more livable.
(That said, I suppose we should wait until the actual problem comes up.)
- Brian
Hans Stimer
1) Most of the time you just want the latest version
2) Most of the time, the imported package's imports will work with the
latest version
3) If you need to freeze you can just not goinstall
4) If you need older versions you can clone/fork pretty trivially and
just import the clone
I can live with this. I would rather have something under featured
than over featured like CORBA, Java, C++, and J2EE, or any number of
other committee designed abominations.
btw - goinstall updating http://godashboard.appspot.com/packages is a
clever idea. However I think the web site is broken; I'm getting a
blank page.
Andrew Gerrand
Hans Stimer
---
1) list the package name -- it isn't always apparent from the path
e.g. golang.googlecode.com
2) list the actual include path -- the paths listed don't always work;
sometimes the actual package is buried e.g. golang.googlecode.com
3) don't clump multiple packages into a single listing if they are all
at root level within the same repository e.g. golang.googlecode.com
4) show clones and their relationship to the base package -- helps you
to decide if you want the main package or one of it's clones
5) mark packages busted if they don't build -- lets not waste people's
time trying to install busted packages
6) show which versions of go and dependent packages are known to build
the package
I love the realtime/realworld reporting on packages at godashboard.
Excellent idea. At least until some idiot starts posting false info.
Suggestions for goinstall
---
1) add option for showing clones of installed packages -- helps you to
decide if you should be using a clone instead
2) add option for package searching, and showing clones -- maybe do a
text search of the readmes, or maybe even standardize on the readmes
(markdown anyone?), or standardize embedded package docs in the code
3) add option for showing when clones are pulled back into the main
trunk -- to help decide when you can abandon a bug fix clone and go
back to the main branch
David Roundy
>> Makefile at all? Why not rely on the Makefile provided with the
>> package?
>
> The main reason is to force the target install path to
> be the one that goinstall thinks it should be. This lets
> packages written without knowing about goinstall
> or without using a Makefile still be installed with goinstall.
>
> Now that goinstall is known, I hope people will change
> their makefiles to install under the goinstall target path.
> If the Makefile has a reasonable TARG= then it is
> probably okay to use the distributed Makefile.
What do you mean by a reasonable TARG=? It seems like it'd be bad to
mention the source in the TARG line, since it means that temporary
forks would have to change the Makefile. e.g. if I make a
github-hosted clone of web.go, I shouldn't have to modify the Makefile
in order for it to goinstall into a different directory.
To be honest, I'm still a little unclear as to how goinstall is
working. Where does it install the Makefile that it generates? When I
run
goinstall github.com/hoisie/web.go
The package is installed in $GOROOT/pkg/github.com/hoisie/web.go.a
But then if I do
cd $GOROOT/src/pkg/github.com/hoisie/web.go
make install
It gets installed as $GOROOT/pkg/web.a
I guess this is because I am now using the package's included
makefile. But where did the generated one go?
One appealing (to me) possibility would be to pass the installation
path to make as an environment variable. You could then check if the
existing Makefile mentions that variable, and if it does, you could
use it, otherwise try generating your own? (and if you keep the check
simple, then the Makefile could mention the variable name in a
comment, in case the build system is more complicated and doesn't
actually use the variable directly in the Makefile (which might itself
call something else that *does* check the install target).
--
David Roundy
Russ Cox
> mention the source in the TARG line, since it means that temporary
> forks would have to change the Makefile. e.g. if I make a
> github-hosted clone of web.go, I shouldn't have to modify the Makefile
> in order for it to goinstall into a different directory.
That's an interesting argument, but it's only a single line change.
And right now goinstall does behave as you say:
you don't need to include a Makefile at all.
> I guess this is because I am now using the package's included
> makefile. But where did the generated one go?
It went to make's standard input.
> One appealing (to me) possibility would be to pass the installation
> path to make as an environment variable.
This fails when goinstall recurses to get dependencies.
There really needs to be a standard algorithm to translate
source location to install path.
Russ
David Roundy
>> What do you mean by a reasonable TARG=? It seems like it'd be bad to
>> mention the source in the TARG line, since it means that temporary
>> forks would have to change the Makefile. e.g. if I make a
>> github-hosted clone of web.go, I shouldn't have to modify the Makefile
>> in order for it to goinstall into a different directory.
>
> That's an interesting argument, but it's only a single line change.
> And right now goinstall does behave as you say:
> you don't need to include a Makefile at all.
It also won't install any package that I've written, so this question
does seem relevant to me. It looks like in order to allow goinstall
to work, I would need to give up any use of local private packages
(imported with "./foo/bar"). I don't want these packages to be
installed globally, and that seems to be the only option when using
the "standard" makefile (and hence goinstall).
goinstall seems to be going the direction of cabal (see
http://www.haskell.org/cabal/), which is to create an easy
installation tool, which embeds a simplistic build system. The result
(with cabal, but hopefully not with goinstall in the end) is that you
have a tool that works for the most simple of programs, but fails to
work properly for more complicated programs. (On other aspects of why
cabal is bad, you could talk with John Meacham, who is now employed at
Google...)
>> One appealing (to me) possibility would be to pass the installation
>> path to make as an environment variable.
>
> This fails when goinstall recurses to get dependencies.
> There really needs to be a standard algorithm to translate
> source location to install path.
I don't quite follow. I didn't mean to suggest anything to the
contrary, just that by passing the install path as an environment
variable you could allow developers to create our own Makefiles that
would work with goinstall.
David
Russ Cox
> does seem relevant to me. It looks like in order to allow goinstall
> to work, I would need to give up any use of local private packages
> (imported with "./foo/bar"). I don't want these packages to be
> installed globally, and that seems to be the only option when using
> the "standard" makefile (and hence goinstall).
There's no such thing as a local private package.
You can pretend that, but 6l needs to find all the
object files when it links everything, so you'd have
to keep those object files around, both for yourself
and any clients of the package. It's much clearer
to install them in the standard place than have
things grubbing around in the build directories.
In general imports of ./anything are for special cases,
not for general use. They're not as private as you think.
> goinstall seems to be going the direction of cabal (see
> http://www.haskell.org/cabal/), which is to create an easy
> installation tool, which embeds a simplistic build system. The result
> (with cabal, but hopefully not with goinstall in the end) is that you
> have a tool that works for the most simple of programs, but fails to
> work properly for more complicated programs. (On other aspects of why
> cabal is bad, you could talk with John Meacham, who is now employed at
> Google...)
We're certainly trying to build something that works for
complicated programs.
> I don't quite follow. I didn't mean to suggest anything to the
> contrary, just that by passing the install path as an environment
> variable you could allow developers to create our own Makefiles that
> would work with goinstall.
If you goinstall a package that has import lines that
refer to other packages, goinstall builds those first,
downloading them if necessary. The environment
variable could only possibly apply to the top level
of the recursion.
Russ
David Roundy
>> It also won't install any package that I've written, so this question
>> does seem relevant to me. It looks like in order to allow goinstall
>> to work, I would need to give up any use of local private packages
>> (imported with "./foo/bar"). I don't want these packages to be
>> installed globally, and that seems to be the only option when using
>> the "standard" makefile (and hence goinstall).
>
> There's no such thing as a local private package.
> You can pretend that, but 6l needs to find all the
> object files when it links everything, so you'd have
> to keep those object files around, both for yourself
> and any clients of the package. It's much clearer
> to install them in the standard place than have
> things grubbing around in the build directories.
> In general imports of ./anything are for special cases,
> not for general use. They're not as private as you think.
Okay, I hadn't realized this. You're right, that's ugly, and I'll
change my code to avoid doing that.
For applications, rather than packages, is there a problem with local
imports of the ./foo form? I don't like my program build process to
install anything until the build is complete, and would prefer that it
doesn't install anything but the application itself. I guess I could
also use -I and -L to avoid installing packages. Or, of course, I
could put everything in package main, but that doesn't work so nicely
if I want to use shared code for several programs.
I guess the next question is how to compile a multi-package package
(i.e. one that depends on a sub-package) without installing anything.
I guess the answer must be to use -I?
> We're certainly trying to build something that works for
> complicated programs.
Good! (Then I hope I'm not annoying you by trying to get my
complicated programs to work...)
At the moment, I've modified all my code to use gotgo for their slice
handling. I store the generated packages in each repository, so users
shouldn't need to have gotgo itself installed, and the Makefile
(hopefully) works with this. But currently the generated packages
aren't always suitable for public consumption, since they may refer to
data types by qualified import names, which could clash with another
package's use.
Currently, gotgo's import syntax looks like:
import errorslice "gotgo/slice(os.Error)"
which will import a package for concatenating, importing, filtering,
etc on []os.Error. The ugliness shows up due to issues with package
naming. I don't want template expansion to require looking into the
contents of other imported packages (which potentially haven't yet
been built or installed, and might thus be hard to find), so I require
that "os" above is imported via:
import os "os"
so that I know that os.Error refers to the package imported as "os".
But at this point, I have no guarantee that you didn't write something
like
import os "io"
in which case we couldn't share this package with anyone else, because
the qualified import name isn't unique. So I guess maybe what I need
to do is use a syntax that directly uses import strings, which is much
uglier, and would look something like:
import fooslice "github.com/droundy/gotgo/slice(github.com/droundy/foo.Type)"
Does this sound like a reasonable sort of approach to you? Then the
build process would involve gotgo first looking through the imports
and creating any of those packages, and *then* goinstall looks through
and builds the remainder. And if gotgo were to be accepted into the
core, then goinstall could do both steps.
>> I don't quite follow. I didn't mean to suggest anything to the
>> contrary, just that by passing the install path as an environment
>> variable you could allow developers to create our own Makefiles that
>> would work with goinstall.
>
> If you goinstall a package that has import lines that
> refer to other packages, goinstall builds those first,
> downloading them if necessary. The environment
> variable could only possibly apply to the top level
> of the recursion.
Hmmm. I'd have thought that the recursive goinstall would simply
redefine that environment variable, since it knows it's building a
different package.
--
David Roundy
Russ Cox
> imports of the ./foo form? I don't like my program build process to
> install anything until the build is complete, and would prefer that it
> doesn't install anything but the application itself. I guess I could
> also use -I and -L to avoid installing packages. Or, of course, I
> could put everything in package main, but that doesn't work so nicely
> if I want to use shared code for several programs.
It really sounds like you want to install the packages.
At some point we'll have to grapple with having multiple
places where packages can live, and then maybe you
won't be so opposed to installing them if they can go
in some private place?
> how to compile a multi-package package
> (i.e. one that depends on a sub-package) without installing anything.
> I guess the answer must be to use -I?
Just install them.
> import fooslice "github.com/droundy/gotgo/slice(github.com/droundy/foo.Type)"
>
> Does this sound like a reasonable sort of approach to you?
I don't think it makes sense to change the design of imports or
installation to adapt to gotgo. gotgo is a fun experiment, and I hope
it leads to a nice system, but if so that system can't be import-based,
because:
* I have to write a different import line for every kind of vector I want.
* I can't use a vector of type T inside the package that defines type T,
because the imports would create a cycle.
* I can't use a vector of type t ever, because the imported package
cannot talk about my non-exported type t.
Those all seem like show stoppers to me, and they're fundamental to
any import-based approach to generics. Again, I hope that experience
with gotgo will help us learn more about the eventual solution, but I don't
think it's close enough to being that solution to adjust the import or
install mechanisms to accommodate it.
Russ
David Roundy
>> For applications, rather than packages, is there a problem with local
>> imports of the ./foo form? I don't like my program build process to
>> install anything until the build is complete, and would prefer that it
>> doesn't install anything but the application itself. I guess I could
>> also use -I and -L to avoid installing packages. Or, of course, I
>> could put everything in package main, but that doesn't work so nicely
>> if I want to use shared code for several programs.
>
> It really sounds like you want to install the packages.
> At some point we'll have to grapple with having multiple
> places where packages can live, and then maybe you
> won't be so opposed to installing them if they can go
> in some private place?
The problem is that then I can't easily have two versions of my code
on the same machine. If I compile the experimental one, then it
installs the experimental packages. Then if I compile the bugfix
code, it links with the experimental packages. The only way I see to
solve this is to put all my code into the main package, which doesn't
seem too fit well with the idea of minimal recompiles.
>> import fooslice "github.com/droundy/gotgo/slice(github.com/droundy/foo.Type)"
>>
>> Does this sound like a reasonable sort of approach to you?
>
> I don't think it makes sense to change the design of imports or
> installation to adapt to gotgo. gotgo is a fun experiment, and I hope
> it leads to a nice system, but if so that system can't be import-based,
> because:
>
> * I have to write a different import line for every kind of vector I want.
> * I can't use a vector of type T inside the package that defines type T,
> because the imports would create a cycle.
> * I can't use a vector of type t ever, because the imported package
> cannot talk about my non-exported type t.
Yay, the first substantive feedback I've gotten on gotgo! :) Thanks!
> Those all seem like show stoppers to me, and they're fundamental to
> any import-based approach to generics. Again, I hope that experience
> with gotgo will help us learn more about the eventual solution, but I don't
> think it's close enough to being that solution to adjust the import or
> install mechanisms to accommodate it.
I agree that I need to have a non-import way to use gotgo, and have
already implemented such a mechanism (not yet committed or pushed).
But it also seems important to include an import-based approach,
otherwise any generic library needs to be recompiled every time
anything that uses it is compiled, which brings us down to C++-level
compile time, so far as I can see.
--
David Roundy
Esko Luontola
wrote:
> On Sat, Mar 20, 2010 at 10:32 AM, Russ Cox <r...@golang.org> wrote:
> >> For applications, rather than packages, is there a problem with local
> >> imports of the ./foo form? I don't like my program build process to
> >> install anything until the build is complete, and would prefer that it
> >> doesn't install anything but the application itself. I guess I could
> >> also use -I and -L to avoid installing packages. Or, of course, I
> >> could put everything in package main, but that doesn't work so nicely
> >> if I want to use shared code for several programs.
>
> > It really sounds like you want to install the packages.
> > At some point we'll have to grapple with having multiple
> > places where packages can live, and then maybe you
> > won't be so opposed to installing them if they can go
> > in some private place?
>
> The problem is that then I can't easily have two versions of my code
> on the same machine. If I compile the experimental one, then it
> installs the experimental packages. Then if I compile the bugfix
> code, it links with the experimental packages. The only way I see to
> solve this is to put all my code into the main package, which doesn't
> seem too fit well with the idea of minimal recompiles.
I'm (very slowly) working on a build tool whose goal is to allow just
that: allow building and testing projects without requiring to install
them. Makefiles are just too cumbersome when you've gotten used to the
tools that for example the Java and Ruby communities have.
My biggest problem with Makefiles (which use "$(GOROOT)/src/Make.*")
was that in a multi-package project it's not possible to run tests
(with gotest) which use more than one of the current project's
packages, unless all of them are first installed. This in turn means
that during development, you are required to install broken code (code
which does not pass the tests) hundreds of times per day (when using
TDD), which in turn means that developing a library and using it in
other projects at the same time is very cumbersome, because those
other projects can not depend on a stable version of the library.
The second problem with Makefiles is that you need to manually
maintain a list of all the files which belong to the project, when the
build tool could find them automatically.
Giles Lean
David Roundy <rou...@physics.oregonstate.edu> wrote:
> The problem is that then I can't easily have two versions of my code
> on the same machine. If I compile the experimental one, then it
> installs the experimental packages. Then if I compile the bugfix
> code, it links with the experimental packages. The only way I see to
> solve this is to put all my code into the main package, which doesn't
> seem too fit well with the idea of minimal recompiles.
I have multiple installations (last release, tip, feature I'm
working on, local package(s)) and switch $PATH by sourcing a
script, e.g.:
$ . ./set_env
when I switch from one to another. set_env says something like:
$ cat set_env
export GOARCH=amd64
export GOOS=darwin
export GOROOT=/Users/giles/Projects/go-release/go
export GOBIN=/Users/giles/Projects/go-release/bin
. "$HOME/.path"
PATH="$GOBIN:$PATH"
I agree with you that this doesn't reduce the number of
compiles a lot, but with the speed of the 6g/8g compilers we
have to do /something/ to restore coffee drinking and mailing
list reading time so it's not too bad in practice. :-/
What is annoying of course is building a package with the
"wrong" environment variables set; for that I really have no
solution except to wrap 'make' in a function that checks $PWD
aginst $GOROOT, which I might yet do but haven't got around to
yet.
Keeping seperate trees around also has the advantage of
letting me use mercurial queues (or branches if I wanted).
The only trees that need the codereview extension in them are
ones I plan to submit code from.
Your milage may vary ....
Giles
Russ Cox
goinstall my/package
-- documentation from goinstall/doc.go
Usage:
goinstall [flags] importpath...
GitHub (Git)
import "github.com/user/project.git"
import "github.com/user/project.git/sub/directory"
import "project.googlecode.com/hg"
import "project.googlecode.com/hg/sub/directory"
import "project.googlecode.com/svn/trunk"
import "project.googlecode.com/svn/trunk/sub/directory"
at http://godashboard.appspot.com/packages, allowing Go programmers
FAQ:
See http://golang.org/doc/contribute.html
• How can I use goinstall with a package I commit changes to?
Instead of having goinstall do the checkout automatically,
check out your package yourself into the same location where
goinstall would put it. You can use that as your working copy
and push to the world as appropriate. For example, if you
were working on github.com/hoisie/web.go, you check it out
into $GOROOT/src/pkg/github.com/hoisie/web.go.git
and then edit the files in that directory, pushing back to github
as needed. Goinstall will use the checked-out git repository
instead of trying to download a fresh one.
• What information is sent to and logged by the dashboard?
The source code for both goinstall and the dashboard are
in the Go repository:
$GOROOT/src/cmd/goinstall/download.go
$GOROOT/misc/dashboard/godashboard/package.py
Goinstall sends the package import path.
The dashboard increments a counter and sets the "most recently installed" time.
There is no other reporting or logging. The data recorded by the dashboard
is publicly visible at http://godashboard.appspot.com/package.
John Asmuth
For instance (and I like that you used my package as one of your
examples)
import "gomatrix.googlecode.com/hg"
The package name is matrix, which is not reflected in the import
statement. Now, I understand that, when goinstall downloads it, it
will examine what it downloads and know the correct package name for
use in the code.
But consider the case when someone asks how to use matrices in go, and
someone in the IRC channel says, "oh, just use <import statement> at
the top of your file", referring to the import atatement above.
Then this hapless coder has no immediate way to see what the name of
the package is. Maybe "gomatrix"? But no, it's "matrix", which you can
only know by examining the gomatrix source.
This problem doesn't appear so much normally, because it is customary
to name the directory containing a package after that package.
Of course, if this hapless coder is slightly less hapless, he or she
can name the import. This obviously isn't an insurmountable problem.
Just one that can cause some confusion at first for those coders who
have trouble looking stuff up on their own (for whatever reason).
But this seems like a really useful idea, and a very slick way to keep
code-versions up-to-date.
Which makes me think... is there a way to refer to a certain version
of, for example, a googlecode hg project? What if your code works with
an older version, and breaks with a newer? For instance, it is
possible that some clever compiler optimizations would make some of
the methods in gomatrix pointless, and they could be removed. But what
about projects that used those now-removed methods? Being able to
refer to a specific and immutable version would be sweet, and the
default code be the most up-to-date.
- John
> to update a list of what people are using.http://godashboard.appspot.com/package.
>
> This is a request for comments on the general approach.
> Please comment here, on the mailing list, rather than in the
> codereview site. It will be easier to have a discussion in an
> actual discussion forum.
>
> -- documentation from goinstall/doc.go
>
> Goinstall is an experiment in automatic package installation.
> It installs packages, possibly downloading them from public version
> control systems, and it maintains a list of public Go packages athttp://godashboard.appspot.com/packages.
> athttp://godashboard.appspot.com/packages, allowing Go programmers
> Seehttp://golang.org/doc/contribute.html
>
> • How can I use goinstall with a package I commit changes to?
>
> Instead of having goinstall do the checkout automatically,
> check out your package yourself into the same location where
> goinstall would put it. You can use that as your working copy
> and push to the world as appropriate. For example, if you
> were working on github.com/hoisie/web.go, you check it out
> into $GOROOT/src/pkg/github.com/hoisie/web.go.git
> and then edit the files in that directory, pushing back to github
> as needed. Goinstall will use the checked-out git repository
> instead of trying to download a fresh one.
>
> • What information is sent to and logged by the dashboard?
>
> The source code for both goinstall and the dashboard are
> in the Go repository:
>
> $GOROOT/src/cmd/goinstall/download.go
> $GOROOT/misc/dashboard/godashboard/package.py
>
> Goinstall sends the package import path.
> The dashboard increments a counter and sets the "most recently installed" time.
> There is no other reporting or logging. The data recorded by the dashboard
> is publicly visible athttp://godashboard.appspot.com/package.
Russ Cox
> For instance (and I like that you used my package as one of your
> examples)
> import "gomatrix.googlecode.com/hg"
>
> The package name is matrix, which is not reflected in the import
> statement. Now, I understand that, when goinstall downloads it, it
> will examine what it downloads and know the correct package name for
> use in the code.
>
> But consider the case when someone asks how to use matrices in go, and
> someone in the IRC channel says, "oh, just use <import statement> at
> the top of your file", referring to the import atatement above.
>
> Then this hapless coder has no immediate way to see what the name of
> the package is. Maybe "gomatrix"? But no, it's "matrix", which you can
> only know by examining the gomatrix source.
It's true; this is a problem. I think this is going to end up
being solved by common sense and convention. Maybe
gomatrix.googlecode.com/hg should call itself package gomatrix,
or maybe it should be in a subdirectory named matrix.
The subdirectory = package name rule is a good general
convention, but there are reasonable exceptions too.
For example, the mimeparse project has source code for
many different languages, and the Go code is in the go
subdirectory. It doesn't bother me that I write import
"mimeparse.googlecode.com/svn/trunk/go" to get package
mimeparse, and I'd much rather get package mimeparse
than package go. It's an open issue, but I think conventions
will go a long way.
> Which makes me think... is there a way to refer to a certain version
> of, for example, a googlecode hg project? What if your code works with
> an older version, and breaks with a newer? For instance, it is
> possible that some clever compiler optimizations would make some of
> the methods in gomatrix pointless, and they could be removed. But what
> about projects that used those now-removed methods? Being able to
> refer to a specific and immutable version would be sweet, and the
> default code be the most up-to-date.
Subversion encodes this info in the path, so you can always
import something other than trunk. In hg and git, there's no
direct support for specific versions but goinstall does prefer to
use the branch or tag named "release" over using git HEAD/hg tip.
In general the versioning problem is pretty tricky. If you must use
version 1 of a package and Michael must use version 2 and I want
to use both your packages, then I end up with both copies in my
binary. 6g can handle that; gccgo might need a little help.
Either way it's suboptimal to be linking every version in.
Goinstall nods at the versioning problem by preferring "release"
but it's by no means solved, and I don't know that there's
a magic bullet.
Russ
chris dollin
of its most recent installation. This mechanism powers the package list
at http://godashboard.appspot.com/packages, allowing Go programmers
to learn about popular packages that might be worth looking at.
even a Coming Soon; is that as expected?
Chris "allusive" Dollin
Russ Cox
>> at http://godashboard.appspot.com/packages, allowing Go programmers
>> to learn about popular packages that might be worth looking at.
>
> That link currently serves an empty page, no text, no HTML, nothing, not
> even a Coming Soon; is that as expected?
Sorry, typo. The correct URL is http://godashboard.appspot.com/package.
Russ
Esko Luontola
I'll spli