Re: [go-nuts] Proposal for import/package simplification

358 views
Skip to first unread message

Dave Cheney

unread,
Jun 6, 2013, 9:03:25 AM6/6/13
to Morten Siebuhr, golang-nuts
Hello,

Thank you for your proposal, I agree that something needs to be done
to help folks who have more demanding needs for package dependency
versioning.

Responding to your proposal, the go tool is just a wrapper around the
{5,6,8}g compilers, that is to say if you have an import line

import "code.google.com/p/go/downloads/vet.zip"

The gc compilers themselves would have to know how to fetch that
dependency, and the linker would have to know how to read out of a zip
file. This is a non trivial amount of work.

I apologise if I have misunderstood your proposal, could you please
provide some worked examples of go code that uses your proposed
change.

Cheers

Dave

On Thu, Jun 6, 2013 at 8:07 PM, Morten Siebuhr <sb...@sbhr.dk> wrote:
> Hi all,
>
> There seem to be broad consensus that something needs to be done about
> imports in go, and a lot of discussion about how this should be solved.
>
> I would like to propose to add .zip, .tar and the like as pseudo-VCS'es in
> line with git, bzr, hg & svn:
>
> - Straight away, it would (at least) work with Github, as they permit
> downloads of the form
> https://github.com/{username}/{project}/archive/{commitish}.zip.
> - It significantly lowers the barrier for creating an online package
> system. Create a URL-scheme with whatever version/tag/release-scheme feels
> appropriate and expose .zip-files.
> - Using <meta>-tags as described in
> http://golang.org/cmd/go/#hdr-Remote_import_path_syntax, proxies to ex.
> github would be trivial to write. Godoc.org could trivially be upgraded to
> be a package-proxy.
> - Allows hosting packages as static files on a webserver (yeah, at least
> git allows that too, but here you can trivially have different versions).
>
> Downsides:
>
> - The VCS code
> (https://code.google.com/p/go/source/browse/src/cmd/go/vcs.go) does seem
> very ... VCS-specific, so there might be a considerable impedance mismatch.
> - Would probably spawn a NIH-fueled package-manager deathmatch instead of
> The One True Go Package Manager (but that path did seem to work out quite
> well for Node.js)
> - Will break the fact that go get currently tries (at least on git) to
> check out something that runs in your current go environment. (IMHO,
> packages change a lot more often than Go itself, and people do expect some
> breakage when updating Go.)
>
> My €.02, heavily inspired by the NPM (Node.js Package Manager).
>
> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

Jan Mercl

unread,
Jun 6, 2013, 9:04:00 AM6/6/13
to Morten Siebuhr, golang-nuts
On Thu, Jun 6, 2013 at 12:07 PM, Morten Siebuhr <sb...@sbhr.dk> wrote:
> There seem to be broad consensus that something needs to be done about
> imports in go, and a lot of discussion about how this should be solved.

I'm not participating in this consensus. Actually I don't even think
the tag 'broad' is fact based.

> I would like to propose to add .zip, .tar and the like as pseudo-VCS'es in
> line with git, bzr, hg & svn:
>
> - Straight away, it would (at least) work with Github, as they permit
> downloads of the form
> https://github.com/{username}/{project}/archive/{commitish}.zip.
> - It significantly lowers the barrier for creating an online package
> system. Create a URL-scheme with whatever version/tag/release-scheme feels
> appropriate and expose .zip-files.
> - Using <meta>-tags as described in
> http://golang.org/cmd/go/#hdr-Remote_import_path_syntax, proxies to ex.
> github would be trivial to write. Godoc.org could trivially be upgraded to
> be a package-proxy.
> - Allows hosting packages as static files on a webserver (yeah, at least
> git allows that too, but here you can trivially have different versions).

I wonder what a package is. I think it's a compilation unit. Hosting
packages is already done by publishing the source code of the package.
It seems that you're talking about something different, even though
you're using the same term, 'package'. Please clarify your intent
and/or terminology, I'm confused.

-j

Morten Siebuhr

unread,
Jun 6, 2013, 9:26:40 AM6/6/13
to golan...@googlegroups.com, Morten Siebuhr


Den torsdag den 6. juni 2013 15.03.25 UTC+2 skrev Dave Cheney:
Hello,

Thank you for your proposal, I agree that something needs to be done
to help folks who have more demanding needs for package dependency
versioning.

Responding to your proposal, the go tool is just a wrapper around the
{5,6,8}g compilers, that is to say if you have an import line

import "code.google.com/p/go/downloads/vet.zip"

The gc compilers themselves would have to know how to fetch that
dependency, and the linker would have to know how to read out of a zip
file. This is a non trivial amount of work.

I apologise if I have misunderstood your proposal, could you please
provide some worked examples of go code that uses your proposed
change.

Just like the VCS-backed stuff is go-get'ed into a directory matching is URL, the .zip (or whatever) should be downloaded & unpacked into such a directory.
 
(Please correct me if I've misunderstood the general workings of the VCS-backed imports.)

Cheers

Dave

[...]

Morten Siebuhr

unread,
Jun 6, 2013, 9:45:43 AM6/6/13
to golan...@googlegroups.com, Morten Siebuhr


Den torsdag den 6. juni 2013 15.04.00 UTC+2 skrev Jan Mercl:
On Thu, Jun 6, 2013 at 12:07 PM, Morten Siebuhr <sb...@sbhr.dk> wrote:
> There seem to be broad consensus that something needs to be done about
> imports in go, and a lot of discussion about how this should be solved.

I'm not participating in this consensus. Actually I don't even think
the tag 'broad' is fact based.

Sorry about that. No offence meant.
 
> I would like to propose to add .zip, .tar and the like as pseudo-VCS'es in
> line with git, bzr, hg & svn:
>
>  - Straight away, it would (at least) work with Github, as they permit
> downloads of the form
> https://github.com/{username}/{project}/archive/{commitish}.zip.
>  - It significantly lowers the barrier for creating an online package
> system. Create a URL-scheme with whatever version/tag/release-scheme feels
> appropriate and expose .zip-files.
>  - Using <meta>-tags as described in
> http://golang.org/cmd/go/#hdr-Remote_import_path_syntax, proxies to ex.
> github would be trivial to write. Godoc.org could trivially be upgraded to
> be a package-proxy.
>  - Allows hosting packages as static files on a webserver (yeah, at least
> git allows that too, but here you can trivially have different versions).

I wonder what a package is. I think it's a compilation unit. Hosting
packages is already done by publishing the source code of the package.
It seems that you're talking about something different, even though
you're using the same term, 'package'. Please clarify your intent
and/or terminology, I'm confused.

I'm trying to refer to packages as `go help packages` does.

I'm well aware that I can publish the source code for a package by publishing it in a VCS somewhere.

BUT I can only retrieve the most recent version of the package (or whatever is deemed the most recent for the version of Go you're running). The Node Package Manager (NPM) behaved largely in this way in the beginning, and it turned into a horrible mess. It's now switched to pinning to most recent minor release by default (they use MAJOR.MINOR.PATCH versioning, and by default allows PATCH-updates).

James Bardin

unread,
Jun 6, 2013, 9:48:07 AM6/6/13
to golan...@googlegroups.com, Morten Siebuhr
I think a common misconception is that the import path is the remote location of the package, which it is not.
The import path is the path within your GOPATH that contains the package. If the package can't be found in your GOPATH, the go tooling takes a guess at where to get the code based on the path, and tries some VCSs to retrieve it.

Morten Siebuhr

unread,
Jun 6, 2013, 9:54:46 AM6/6/13
to golan...@googlegroups.com, Morten Siebuhr


Den torsdag den 6. juni 2013 15.48.07 UTC+2 skrev James Bardin:
I think a common misconception is that the import path is the remote location of the package, which it is not.
The import path is the path within your GOPATH that contains the package. If the package can't be found in your GOPATH, the go tooling takes a guess at where to get the code based on the path, and tries some VCSs to retrieve it.

 Yes. The only change I propose is that it also takes some guesses at non-VCS remote resources, if it can't find it in GOPATH.

Kamil Kisiel

unread,
Jun 6, 2013, 12:06:51 PM6/6/13
to golan...@googlegroups.com
What would it look like if the package were in a subdirectory of the archive? Or would you expect a separate archive for each package?

For example if you get github.com/user/project/some/package and github.com/user/project/other/package2 the go tool only needs to retrieve github.com/user/project and then both packages, as well as any additional packages contained in the repo, are then available within GOPATH.

So in your proposal I assume you'd need something like https://github.com/user/project/archive/commit.zip/some/package ?

On Thursday, June 6, 2013 3:07:03 AM UTC-7, Morten Siebuhr wrote:
Hi all,

There seem to be broad consensus that something needs to be done about imports in go, and a lot of discussion about how this should be solved.

I would like to propose to add .zip, .tar and the like as pseudo-VCS'es in line with git, bzr, hg & svn:

 - Straight away, it would (at least) work with Github, as they permit downloads of the form https://github.com/{username}/{project}/archive/{commitish}.zip.
 - It significantly lowers the barrier for creating an online package system. Create a URL-scheme with whatever version/tag/release-scheme feels appropriate and expose .zip-files.
 - Using <meta>-tags as described in http://golang.org/cmd/go/#hdr-Remote_import_path_syntax, proxies to ex. github would be trivial to write. Godoc.org could trivially be upgraded to be a package-proxy.
 - Allows hosting packages as static files on a webserver (yeah, at least git allows that too, but here you can trivially have different versions).

Jesse McNelis

unread,
Jun 6, 2013, 9:19:09 PM6/6/13
to Kamil Kisiel, golang-nuts
On Fri, Jun 7, 2013 at 2:06 AM, Kamil Kisiel <kamil....@gmail.com> wrote:
> What would it look like if the package were in a subdirectory of the
> archive? Or would you expect a separate archive for each package?
> So in your proposal I assume you'd need something like
> https://github.com/user/project/archive/commit.zip/some/package ?

This would be consistent with the current 'go get' of a package in a repo.
eg. import "example.org/repo.git/foo/bar"




--
=====================
http://jessta.id.au

Peter Kleiweg

unread,
Jun 7, 2013, 5:58:44 AM6/7/13
to golan...@googlegroups.com, Morten Siebuhr
Op donderdag 6 juni 2013 15:04:00 UTC+2 schreef Jan Mercl het volgende:

On Thu, Jun 6, 2013 at 12:07 PM, Morten Siebuhr <sb...@sbhr.dk> wrote:
> There seem to be broad consensus that something needs to be done about
> imports in go, and a lot of discussion about how this should be solved.

I'm not participating in this consensus. Actually I don't even think
the tag 'broad' is fact based.

I think package developers should take backward compatibility serious. If you realy need to break that compatibility, you should change the import path from github.com/foo/v1/bar to github.com/foo/v2/bar or something like that.

Making imports in Go more complex only stimulates bad package development. 

Morten Siebuhr

unread,
Jun 7, 2013, 6:57:02 AM6/7/13
to golan...@googlegroups.com, Morten Siebuhr
Den fredag den 7. juni 2013 11.58.44 UTC+2 skrev Peter Kleiweg:
Op donderdag 6 juni 2013 15:04:00 UTC+2 schreef Jan Mercl het volgende:
On Thu, Jun 6, 2013 at 12:07 PM, Morten Siebuhr <sb...@sbhr.dk> wrote:
> There seem to be broad consensus that something needs to be done about
> imports in go, and a lot of discussion about how this should be solved.

I'm not participating in this consensus. Actually I don't even think
the tag 'broad' is fact based.

I think package developers should take backward compatibility serious. If you realy need to break that compatibility, you should change the import path from github.com/foo/v1/bar to github.com/foo/v2/bar or something like that.

That would be nice, but given that 98[1] packages of 7598[2] on http://godoc.org follow this convention (≃1,3%), I think that's a lost cause. 
 
Making imports in Go more complex only stimulates bad package development. 

I fail to see how downloading an archive file and unpacking it can be considered "more complex" than what is currently in place.

Granted, it would be nice if packages never broke backwards compatibility, introduced new bugs, had subtle changes in behavior, etc. But I just don't think that's realistic to expect.

[1] curl godoc.org/-/index | egrep 'v[[:digit:]]' | wc -l
[2] Noted in the bottom of godoc.org/-/index

Kamil Kisiel

unread,
Jun 7, 2013, 12:15:38 PM6/7/13
to golan...@googlegroups.com, Morten Siebuhr
Well, technically with this proposal the import path *is* changed because presumably you would be pointing to a different archive path for different version.

I'm still trying to decide how I feel about it overall, I probably would never use it for distribution myself but it doesn't seem terribly unreasonable.

Morten Siebuhr

unread,
Jun 8, 2013, 5:01:33 PM6/8/13
to golan...@googlegroups.com, Morten Siebuhr


Den torsdag den 6. juni 2013 15.03.25 UTC+2 skrev Dave Cheney:
Hello,

Thank you for your proposal, I agree that something needs to be done
to help folks who have more demanding needs for package dependency
versioning.

Responding to your proposal, the go tool is just a wrapper around the
{5,6,8}g compilers, that is to say if you have an import line

import "code.google.com/p/go/downloads/vet.zip"

The gc compilers themselves would have to know how to fetch that
dependency, and the linker would have to know how to read out of a zip
file. This is a non trivial amount of work.

I apologise if I have misunderstood your proposal, could you please
provide some worked examples of go code that uses your proposed
change.

I just spent a few hours trying to hack it directly into the VCS-code, but that didn't pan out.

I ended up with the following change to src/cmd/go/vcs.go:

diff -r 3378d2483995 src/cmd/go/vcs.go
--- a/src/cmd/go/vcs.go Fri Jun 07 19:01:07 2013 +0100
+++ b/src/cmd/go/vcs.go Sat Jun 08 22:48:55 2013 +0200
@@ -44,6 +44,7 @@
 
 // vcsList lists the known version control systems
 var vcsList = []*vcsCmd{
+ vcsCurl,
  vcsHg,
  vcsGit,
  vcsSvn,
@@ -61,6 +62,15 @@
  return nil
 }
 
+var vcsCurl = &vcsCmd{
+ name: "curlUnzip",
+ cmd:  "curlUnzip",
+
+ createCmd: "{repo} {dir}",
+
+ scheme: []string{"https", "http"},
+}
+
 // vcsHg describes how to use Mercurial.
 var vcsHg = &vcsCmd{
  name: "Mercurial",
@@ -572,6 +582,12 @@
  check:  launchpadVCS,
  },
 
+ // Zipfiles
+ {
+ re:  `^(?P<root>(?P<repo>([a-z0-9.\-]+\.)+[a-z0-9.\-]+(:[0-9]+)?/[A-Za-z0-9_.\-/]*?\.zip))(/[A-Za-z0-9_.\-]+)*$`,
+ vcs: "curlUnzip",
+ },
+
  // General syntax for any server.
  {
  re:   `^(?P<root>(?P<repo>([a-z0-9.\-]+\.)+[a-z0-9.\-]+(:[0-9]+)?/[A-Za-z0-9_.\-/]*?)\.(?P<vcs>bzr|git|hg|svn))(/[A-Za-z0-9_.\-]+)*$`,

And dropping the following into my path (as curlUnzip) as a pseudo-VCS:

#!/usr/bin/env sh
curl $1 > $2.zip
unzip -o $2.zip -d $2

That's it.

The code can be tested with this small program:

package main

import test "sbhr.dk/test.zip"

func main() {
test.Hello()
}

I've placed a zip-file at the URL containing a single file, test.go:

package main

import test "sbhr.dk/test.zip"

func main() {
test.Hello()
}

Building go with the patch above & curlUnzip in path, the following should get the program working

> go run test_zip_import.go
Hello World

There is currently no way around using explicit import paths, and I'll be the first to admit that the shell-script is extremely hacky.

André Moraes

unread,
Jun 9, 2013, 5:47:19 PM6/9/13
to Morten Siebuhr, golan...@googlegroups.com
On Thu, Jun 6, 2013 at 10:54 AM, Morten Siebuhr <sb...@sbhr.dk> wrote:
>
>
> Den torsdag den 6. juni 2013 15.48.07 UTC+2 skrev James Bardin:
>>
>> I think a common misconception is that the import path is the remote
>> location of the package, which it is not.
>> The import path is the path within your GOPATH that contains the package.
>> If the package can't be found in your GOPATH, the go tooling takes a guess
>> at where to get the code based on the path, and tries some VCSs to retrieve
>> it.
>
>
> Yes. The only change I propose is that it also takes some guesses at
> non-VCS remote resources, if it can't find it in GOPATH.
>

This can be solved using a little bit of DNS records + HTML + hg

How?

HG can be used just to store repository data and serve it using http
(this could even be done using http/httputil.ReverserProxy). No need
to use it as a vcs just push when you want a new version

DNS if you want to make that available over a local network, if you
expose your server on internet you already have this part done.

HTML to provide a landing page used by "go get" to discover the real
location of your code and also the actual VCS to use


--
André Moraes
http://amoraes.info
Reply all
Reply to author
Forward
0 new messages