Improving go tools for multi-langauge development.

[aco...@redhat.com]

I'm adding Go support to a multi-language project. The go tools are almost great but a little inflexible. I think it would be easy to fix, I have very specific problems and suggestions.

The project has C and Java libraries and language bindings for the C library: python, ruby, php, javascript...  and now go. For testing purposes it is not reasonable to separate each binding into it's own repo. In any case it's not my repo, I can argue for change but "go tools can't hack it" isn't a strong argument.

I want my packages to have sensible names e.g. qpid.apache.org/proton/events, so I set up <meta> tags at qpid.apache.org.

PROBLEMS:

1. go get <meta> tags only allow you to alias a repository *root*. My packages would be qpid.apache.org/proton/proton-c/bindings/go/events. That sucks. I created a "go" symlink so my packages are qpid.apache.org/proton/go/events. Still sucks but less.

2. My work was on a branch called "go". 'go get' will only get branches master or "go1" So I renamed my branch "go1". Now I can do 'go get', hurray!

3. go doc -http doesn't follow symlinks, but it does follow every other path. So it displays nonsense empty packages e.g. "proton/examples/go" It displays my packages under the long path. I moved the go directory to repo root and the symlink to where it should be. This is significant repo surgery to facilitate a tool. Not cool. I'm still on a branch so nobody cares, yet. Now go doc works for my packages but still produces nonsense for the examples.

4. I develop in a lot of languages. I cannot have every language dictating where I put my clones (git is the first language that has tried). Symlinks from the go workspace would be fine but see problem 3.

SUGGESTIONS (implementation suggests are just ideas, ignore at will if there's a better way)

1. All go tools should handle symlinks in the workspace. Even windows has symlinks now.

2. <meta> tags should be able to alias a subdirectory and a branch in a repository. master is not even required by git it's just a (very common) convention.

The import path syntax already almost allows this (no branch) so you could kill two birds with one stone, extend the import path syntax and use it in the meta tags for a consistent package-locating experience. Here's a backward compatible suggestion:

    <meta name="go-import" content="qpid.apache.org/proton git https://git-wip-us.apache.org/repos/asf/qpid-proton.git/proton-c/bindings/go?branch=go1">

I don't care about the syntax as long as I can specify repo, path and branch.

When 'go get' sees that tag it could check out the repo but not directly in the src tree:

     $GOPATH/repos/qpid.apache.org/proton

and create a symlink:

     $GOPATH/src/qpid.apache.org/proton -> ../../repos/qpid.apache.org/proton/proton-c/bindings/go

now all the other go tools work unmodified (except for symlink support) and do exactly what I want.


If I have missed anything and it is possible to solve my problems with the existing go tools then apologies and please tell me how. I did ask on go-nuts, the situation above is as far as I could get.

[Nigel Tao]

On Fri, May 15, 2015 at 7:48 AM, <aco...@redhat.com> wrote:
> I'm adding Go support to a multi-language project. The go tools are almost
> great but a little inflexible. I think it would be easy to fix, I have very
> specific problems and suggestions.

I'd ask adg or rsc for go get branchy symlinky meta-y dir-layouty questions.

[Andrew Gerrand]

On 15 May 2015 at 07:48, <aco...@redhat.com> wrote:

1. go get <meta> tags only allow you to alias a repository *root*. My packages would be qpid.apache.org/proton/proton-c/bindings/go/events. That sucks. I created a "go" symlink so my packages are qpid.apache.org/proton/go/events. Still sucks but less.

At the risk of seeming flippant, you could just redefine your definition of "suck". I work on projects with import paths of similar lengths, and it's not a problem for me. Have you used the "goimports" tool? With it installed, I seldom actually type import paths.

 

SUGGESTIONS (implementation suggests are just ideas, ignore at will if there's a better way)

1. All go tools should handle symlinks in the workspace. Even windows has symlinks now.

Handling symlinks correctly in all cases is difficult. Ignoring symlinks has the nice property that the tools only ever see the same file in one place. That makes the tools much more predictable in use. That's why we've avoided supporting them until now.

Maybe Russ can say more about this.

 

2. <meta> tags should be able to alias a subdirectory and a branch in a repository. master is not even required by git it's just a (very common) convention. 

The import path syntax already almost allows this (no branch) so you could kill two birds with one stone, extend the import path syntax and use it in the meta tags for a consistent package-locating experience. Here's a backward compatible suggestion:

    <meta name="go-import" content="qpid.apache.org/proton git https://git-wip-us.apache.org/repos/asf/qpid-proton.git/proton-c/bindings/go?branch=go1">

I don't care about the syntax as long as I can specify repo, path and branch.  

When 'go get' sees that tag it could check out the repo but not directly in the src tree:

     $GOPATH/repos/qpid.apache.org/proton

and create a symlink:

     $GOPATH/src/qpid.apache.org/proton -> ../../repos/qpid.apache.org/proton/proton-c/bindings/go

now all the other go tools work unmodified (except for symlink support) and do exactly what I want.

What does git do if you try to use it inside the symlinked directory? Does it correctly traverse the tree back to the actual repository root? Or does it get confused?

What happens if you use the go tool inside the $GOPATH/repos/... hierarchy? Nothing?

 

Specifying the branch in the meta tag does seem feasible. Care to create an issue for that specific feature request? https://golang.org/issue/new

[Russ Cox]

On Sun, May 17, 2015 at 9:44 PM, Andrew Gerrand <a...@golang.org> wrote:

SUGGESTIONS (implementation suggests are just ideas, ignore at will if there's a better way)

1. All go tools should handle symlinks in the workspace. Even windows has symlinks now.

Handling symlinks correctly in all cases is difficult. Ignoring symlinks has the nice property that the tools only ever see the same file in one place. That makes the tools much more predictable in use. That's why we've avoided supporting them until now.

Maybe Russ can say more about this.

Andrew is right. Symlinks make a lot of trouble in a variety of ways. They're not a good idea.

 

2. <meta> tags should be able to alias a subdirectory and a branch in a repository. master is not even required by git it's just a (very common) convention. 

The import path syntax already almost allows this (no branch) so you could kill two birds with one stone, extend the import path syntax and use it in the meta tags for a consistent package-locating experience. Here's a backward compatible suggestion:

    <meta name="go-import" content="qpid.apache.org/proton git https://git-wip-us.apache.org/repos/asf/qpid-proton.git/proton-c/bindings/go?branch=go1">

I don't care about the syntax as long as I can specify repo, path and branch.  

When 'go get' sees that tag it could check out the repo but not directly in the src tree:

     $GOPATH/repos/qpid.apache.org/proton

and create a symlink:

     $GOPATH/src/qpid.apache.org/proton -> ../../repos/qpid.apache.org/proton/proton-c/bindings/go

now all the other go tools work unmodified (except for symlink support) and do exactly what I want.

What does git do if you try to use it inside the symlinked directory? Does it correctly traverse the tree back to the actual repository root? Or does it get confused?

What happens if you use the go tool inside the $GOPATH/repos/... hierarchy? Nothing?

I agree again with Andrew: the symlinks here are going to cause many more problems than they solve. 

 

 Specifying the branch in the meta tag does seem feasible. Care to create an issue for that specific feature request? https://golang.org/issue/new

Also agree here: we should let the meta tag specify a branch. That said, for Go 1.5 we've removed the assumption about 'master' from 'go get'. It will use whatever branch is checked out by git clone of the given URL. In the example that motivated it, if you've set your default branch on GitHub to something other than master, 'go get' will use that branch now. If your git server supports ?branch=go1 then it should just work. If you were asking for the go command to interpret ?branch= itself, then no we can't do that, because what if that was meant as a directive to the Git server to set the default branch?

If we still need a way to set the branch for the clone in 'go get' itself, then I would suggest we write

<meta name="go-import" content="qpid.apache.org/proton git https://git-wip-us.apache.org/repos/asf/qpid-proton.git/proton-c/bindings/go branch=go1">

That will not be backwards compatible with earlier versions of 'go get', but neither would any other way of specifying the branch: earlier versions wouldn't know what it meant and would get the wrong branch.

Russ

[aco...@redhat.com]

If we still need a way to set the branch for the clone in 'go get' itself, then I would suggest we write

<meta name="go-import" content="qpid.apache.org/proton git https://git-wip-us.apache.org/repos/asf/qpid-proton.git/proton-c/bindings/go branch=go1">

That's perfect for the branch, I raised  https://github.com/golang/go/issues/10913

So agreed that symlinks are not a good thing. Let me re-state my problem without trying to suggest a solution and see if there is something that can be done.

[aco...@redhat.com]

On Sunday, May 17, 2015 at 9:45:15 PM UTC-4, Andrew Gerrand wrote:

On 15 May 2015 at 07:48, <aco...@redhat.com> wrote:

1. go get <meta> tags only allow you to alias a repository *root*. My packages would be qpid.apache.org/proton/proton-c/bindings/go/events. That sucks. I created a "go" symlink so my packages are qpid.apache.org/proton/go/events. Still sucks but less.

At the risk of seeming flippant, you could just redefine your definition of "suck". I work on projects with import paths of similar lengths, and it's not a problem for me. Have you used the "goimports" tool? With it installed, I seldom actually type import paths.

Flippant is fine by me, "suck" was pretty flippant :) Let me be more concrete. 3 problems, 2 ugly one fatal:

- the path is long, I like short. Go design generally seems to agree with that sentiment.
- "proton-c" and "binding" are meaningless to my users, and having "go" in a Go package name is redundant. I believe the Go naming guidelines call that "stutter"? ;)
 
Here's the big problem: the name is not durable. The "real" path to the code today is "https://git-wip-us.apache.org/repos/asf/qpid-proton.git/proton-c/bindings"

"git-wip-us" is a "temporary" anomaly. I don't have to tell anyone at go-dev what happens when your hosting site is renamed or disappears. That's why you added <meta> tags - which is great but only solves half my problem.

qpid-proton is an open source project that that is not just about Go. I don't control the directory layout. I can lay out the Go directory any way I want, but I can't guarantee the path to it. I need a stable alias, not just for a pretty package name but so "go get" can be relied on into the future. Meta tags only solve the problem up to the repo root, if the path changes inside the repo then the meta tags are useless. This is not an imaginary problem, the parent Qpid project just hit critical mass and is splitting into multiple repos with path changes everywhere as we speak. I don't see it happening to proton tomorrow but I definitely don't want to ship libraries with package names based on the happenstance layout of proton today.

I know that I can just ignore 'go get' and use cmake install, RPM packages and all that traditional stuff to install the packages under whatever name I want. But I think go get is *reallly cool* so I want it to work dammit!

 

SUGGESTIONS (implementation suggests are just ideas, ignore at will if there's a better way)

1. All go tools should handle symlinks in the workspace. Even windows has symlinks now.

Handling symlinks correctly in all cases is difficult. Ignoring symlinks has the nice property that the tools only ever see the same file in one place. That makes the tools much more predictable in use. That's why we've avoided supporting them until now.

 

I get that, symlinks are a pain in many ways. So far they are the only tool I can think of but there must be a better solution.
 

aybe Russ can say more about this.

 

2. <meta> tags should be able to alias a subdirectory and a branch in a repository. master is not even required by git it's just a (very common) convention. 

The import path syntax already almost allows this (no branch) so you could kill two birds with one stone, extend the import path syntax and use it in the meta tags for a consistent package-locating experience. Here's a backward compatible suggestion:

    <meta name="go-import" content="qpid.apache.org/proton git https://git-wip-us.apache.org/repos/asf/qpid-proton.git/proton-c/bindings/go?branch=go1">

I don't care about the syntax as long as I can specify repo, path and branch.  

When 'go get' sees that tag it could check out the repo but not directly in the src tree:

     $GOPATH/repos/qpid.apache.org/proton

and create a symlink:

     $GOPATH/src/qpid.apache.org/proton -> ../../repos/qpid.apache.org/proton/proton-c/bindings/go

now all the other go tools work unmodified (except for symlink support) and do exactly what I want.

What does git do if you try to use it inside the symlinked directory? Does it correctly traverse the tree back to the actual repository root? Or does it get confused?

What happens if you use the go tool inside the $GOPATH/repos/... hierarchy? Nothing?

Didn't try this out extensively. I was trying to come up with a way to say you can still check out the whole repo but just present part of it as being in the go tree. Git appears to work fine. The go tools I have tried (imports and go doc) follow symlinks no problem if you specify a complete package name (e.g. import in source code, "go doc foo/bar/package") but in "browse" mode (e.g. generating tree for go doc -http) go doc ignores symlinks and follows everything else which isn't what I want either if my whole repo is in the source tree.
 

[aco...@redhat.com]

I have a non-symlink suggestion that may suck less: How about we drop some metadata files in there? E.g. a file called ".go-packages" to specify re-mapping from the actual file layout to the package namespaces that the tools use. E.g.

.go-packages

# Go tools ignore everything below $PWD except as directed by "package" directives below.
.    proton-c/bindings/go # "." means act as if proton-c/bindings/go/* were in the current directory.
foo  proton-c/foobar # Act as if proton-c/foobar/* were in $PWD/foo.

So the idea is you can drop one of these anywhere in the src tree. It specifies a re-mapping of files relative to the .go-packages file onto package names relative to the package directory containing the .go-packages file. So if you have a repo with go stuff buried in it, then you use the existing <meta> mechanism to alias the repo, and the repo itself contains a .go-packages file to say how to remap the directory names. No symlinks, you can still clone the whole repo into the tree. Minor point of confusion for developers that the packages paths aren't direct anymore you can get over that. Some complication for the tools but not huge, have to build a "virtual" tree or path taking account of the .go-package files you find along the way, and use that instead of the real directory tree.

This is just a sketch, syntax is not important, it could be made more flexible