directory structures: gorename, buildutil.ContainingPackage, symlinks and .godir

[andy...@gmail.com]

This is probably relevant to Alan Donovan, though I don't have his email handy.

Background:

In an issue related to vim-go and gorename ( https://github.com/fatih/vim-go/issues/221 ), we came to the hypothesis that part of our issue was that buildutils.ContainingPackage ( https://github.com/golang/tools/blob/master/go/buildutil/util.go#L57 ) was resolving symlinks rather than using their logical paths when creating an abspath for a relative filename, resulting in gorename ( https://github.com/golang/tools/blob/master/refactor/rename/spec.go#L202 ) being unable to find the correct package and failing.

I found a relatively platform specific workaround ( https://github.com/fatih/vim-go/issues/221#issuecomment-68802671 ) that works as long as the logical working directory for Vim is under the GOPATH. 

Problem:

The problem is a significant portion of developers are trying really hard (and failing) to develop golang from a path relative to their projects. They want to do that partially because it is the most convenient, most easily silos their projects and deps, and is a common practice for many other popular programming languages.

The most common workaround I've seen and used is to keep your golang code in the root with subpackages, then create a relative gopath and symlink your root directory into a subdirectory thereof (example of this process: https://github.com/coreos/etcd/blob/master/build ). This allows you to easily produce an "example.com/foo/bar" import path for other projects and still allows your working directory to be your root directory. 

Unfortunately, as evidenced above, this isn't really supported by many golang support tools.

Solutions:

A first step towards improving the ecosystem would seem to be having buildutil.ContainingPackage handle the logical symlink path in a way that is compatible with that workflow, though that still requires somebody to cd into the GOPATH/example.com/foo/bar directory to do their work (which is part of what they were trying to avoid in the first place).

A second step would be looking for a more general solution for this development model. As mentioned in the linked issue, a few projects use a ".godir" file to tell consuming applications where this code should be tucked into the GOPATH, and some other languages have conceptually similar ideas (.pth files, in the case of python). I think it might solve a variety of developer questions if golang tools would allow a .godir file to mean "treat this path as if it existed at the described path under GOPATH. Some of these questions include:

 - first run experience ("Where is my GOPATH supposed to be?")

 - global vs relative GOPATHs ("Wait, I have to install this globally to test it? Oh, I can do it locally but I have to write a script to adjust my GOPATH every time I work on this project?")

 - nested directory structures ("Does my project root need to be under my GOPATH? Then how do I have separate GOPATH's for each project?")

This may have been brought up before and I am just using the wrong search terms, but these issues are similar to ones I've heard brought up time and time again by new-to-golang developers and I still run into some of them as a not-so-new-to-golang developer.

[adon...@google.com]

On Monday, 5 January 2015 19:42:53 UTC-5, Andy Smith wrote:

This is probably relevant to Alan Donovan, though I don't have his email handy.

Hi.  (It's adon...@google.com, for the record.)

 

Background:

In an issue related to vim-go and gorename ( https://github.com/fatih/vim-go/issues/221 ), we came to the hypothesis that part of our issue was that buildutils.ContainingPackage ( https://github.com/golang/tools/blob/master/go/buildutil/util.go#L57 ) was resolving symlinks rather than using their logical paths when creating an abspath for a relative filename, resulting in gorename ( https://github.com/golang/tools/blob/master/refactor/rename/spec.go#L202 ) being unable to find the correct package and failing.

Can you give a specific example?

ContainingPackage does file name manipulations, but has no logic that treats symlinks specially nor finds the canonical path.   The only two file names it cares about are the one specified directly by -offset (or indirectly by -from, by way of $GOROOT/$GOPATH), and iff this path is relative, the process's working directory.  Both of them are directly controlled by the user.  I don't see how ContainingPackage could make fewer assumptions about your workspace and still attempt to compute this function at all.

 

By the way, it has been my experience that for every problem symlinks solve, they create another two.  (The way to avoid one of those is to make sure that programs interpret file names to the smallest extent possible, treating them mostly as opaque byte strings to be plumbed through from the user interface to the kernel and back out to the user interface.)

Problem:

The problem is a significant portion of developers are trying really hard (and failing) to develop golang from a path relative to their projects.

What do you mean by "develop golang from a path relative to their projects"?  [I've added my guess below]

The most common workaround I've seen and used is to keep your golang code in the root with subpackages, then create a relative gopath and symlink your root directory into a subdirectory thereof (example of this process: https://github.com/coreos/etcd/blob/master/build ). This allows you to easily produce an "example.com/foo/bar" import path for other projects and still allows your working directory to be your root directory. 

Are you saying, for example, that if your project is golang.org/x/tools, you check it out at /foo and make $GOROOT/src/golang.org/x/tools a symlink to /foo?  I see how this would confuse gorename: it has no way to know, short of exhaustively enumerating all packages' symlinks, that /foo/cmd/gorename/main.go is an alias for $GOROOT/src/golang.org/x/tools/cmd/gorename/main.go, and thus belongs to package golang.org/x/tools/cmd/gorename (and perhaps others as well, in a pathological case).

By the way, the behavior of the 'go' tool on symlinks directly beneath $GOPATH/src is already not very consistent; for example, 'go build' works; 'go list' does not.  (See https://github.com/golang/go/issues/9054.)

 

A first step towards improving the ecosystem would seem to be having buildutil.ContainingPackage handle the logical symlink path in a way that is compatible with that workflow, though that still requires somebody to cd into the GOPATH/example.com/foo/bar directory to do their work (which is part of what they were trying to avoid in the first place).

How could ContainingPackage do this without enumerating all the packages in the workspace?  (gorename does enumerate all packages, but not until later, when it has determined that this relatively expensive step is actually necessary because an exported package-level identifier is being renamed.)

 

A second step would be looking for a more general solution for this development model. As mentioned in the linked issue, a few projects use a ".godir" file to tell consuming applications where this code should be tucked into the GOPATH, and some other languages have conceptually similar ideas (.pth files, in the case of python). I think it might solve a variety of developer questions if golang tools would allow a .godir file to mean "treat this path as if it existed at the described path under GOPATH. Some of these questions include:

 - first run experience ("Where is my GOPATH supposed to be?")

 - global vs relative GOPATHs ("Wait, I have to install this globally to test it? Oh, I can do it locally but I have to write a script to adjust my GOPATH every time I work on this project?")

 - nested directory structures ("Does my project root need to be under my GOPATH? Then how do I have separate GOPATH's for each project?")

This may have been brought up before and I am just using the wrong search terms, but these issues are similar to ones I've heard brought up time and time again by new-to-golang developers and I still run into some of them as a not-so-new-to-golang developer.

All of these tricks and hacks would require systematic, complex, and ill-tested changes to every single tool that must load or process Go source code.  I think the general solution for this development model is: don't do that.  By all means use a hack like a shell alias to make switching between directories with long names easier, but don't insert those hacks into the workflow used by all tools and all tool users.

cheers

alan

[Andy Smith]

Background:

In an issue related to vim-go and gorename ( https://github.com/fatih/vim-go/issues/221 ), we came to the hypothesis that part of our issue was that buildutils.ContainingPackage ( https://github.com/golang/tools/blob/master/go/buildutil/util.go#L57 ) was resolving symlinks rather than using their logical paths when creating an abspath for a relative filename, resulting in gorename ( https://github.com/golang/tools/blob/master/refactor/rename/spec.go#L202 ) being unable to find the correct package and failing.

Can you give a specific example?

Sure, I've added it as a gist: https://gist.github.com/termie/be603974a8b36e97efcf 

 

ContainingPackage does file name manipulations, but has no logic that treats symlinks specially nor finds the canonical path.   The only two file names it cares about are the one specified directly by -offset (or indirectly by -from, by way of $GOROOT/$GOPATH), and iff this path is relative, the process's working directory.  Both of them are directly controlled by the user.  I don't see how ContainingPackage could make fewer assumptions about your workspace and still attempt to compute this function at al.l

AFAICT, the issue occurs when deciding what the process's working directory is, as you can see in the example:

  gorename: can't find package containing /Users/termie/dev/wercker/sentcli/sentcli.go

References a different (resolved symlinks) path than pwd:

  /Users/termie/p/wercker/sentcli/gopath/src/github.com/wercker/sentcli

 

 

By the way, it has been my experience that for every problem symlinks solve, they create another two.  (The way to avoid one of those is to make sure that programs interpret file names to the smallest extent possible, treating them mostly as opaque byte strings to be plumbed through from the user interface to the kernel and back out to the user interface.)

Aye, they certainly create more hassle for library developers while providing a large amount of utility for end-users.

 

Problem:

The problem is a significant portion of developers are trying really hard (and failing) to develop golang from a path relative to their projects.

What do you mean by "develop golang from a path relative to their projects"?  [I've added my guess below]

Your guess is correct. A further example would be, I develop my project under $HOME/p/bar and link that directory as $GOPATH/foo/bar and run ``go build foo/bar`` to build my project.

Further example (from the issue linked in original post) is CoreOS's build script: https://github.com/coreos/etcd/blob/master/build 

A reason people want their project to be the root of their development path is that an environment belongs to a project, not vice-versa. People work on a variety of different projects with different dependencies, it is a common practice to isolate all those projects from each other such that each project has its own GOPATH.

The most common workaround I've seen and used is to keep your golang code in the root with subpackages, then create a relative gopath and symlink your root directory into a subdirectory thereof (example of this process: https://github.com/coreos/etcd/blob/master/build ). This allows you to easily produce an "example.com/foo/bar" import path for other projects and still allows your working directory to be your root directory. 

Are you saying, for example, that if your project is golang.org/x/tools, you check it out at /foo and make $GOROOT/src/golang.org/x/tools a symlink to /foo?  I see how this would confuse gorename: it has no way to know, short of exhaustively enumerating all packages' symlinks, that /foo/cmd/gorename/main.go is an alias for $GOROOT/src/golang.org/x/tools/cmd/gorename/main.go, and thus belongs to package golang.org/x/tools/cmd/gorename (and perhaps others as well, in a pathological case).

I am not disagreeing with you here, this part was not specific to gorename and is addressed in the second part of the "Solutions" section. The big issue ("second step") being addressed, specifically, is that tools have no way of knowing where in the GOPATH they would expect to find the current directory unless the working directory is already under the GOPATH. The smaller issue ("first step") is that it seems symlinks throw buildutils.ContainingPackage (or a library _it_ depends on) off even if your working directory _is_ under the GOPATH.

 

By the way, the behavior of the 'go' tool on symlinks directly beneath $GOPATH/src is already not very consistent; for example, 'go build' works; 'go list' does not.  (See https://github.com/golang/go/issues/9054.)

Would love to have that resolved as well :)

 

 

A first step towards improving the ecosystem would seem to be having buildutil.ContainingPackage handle the logical symlink path in a way that is compatible with that workflow, though that still requires somebody to cd into the GOPATH/example.com/foo/bar directory to do their work (which is part of what they were trying to avoid in the first place).

How could ContainingPackage do this without enumerating all the packages in the workspace?  (gorename does enumerate all packages, but not until later, when it has determined that this relatively expensive step is actually necessary because an exported package-level identifier is being renamed.)

I think I wasn't clear enough with the "cd into the GOPATH..." above. Hopefully the additional examples have cleared it up: I am running gorename from _within_ a directory (logically) under the GOPATH such that `pwd` clearly puts me under the GOPATH but the symlinks get resolved to a path out of the GOPATH and the command fails.

 

 

A second step would be looking for a more general solution for this development model. As mentioned in the linked issue, a few projects use a ".godir" file to tell consuming applications where this code should be tucked into the GOPATH, and some other languages have conceptually similar ideas (.pth files, in the case of python). I think it might solve a variety of developer questions if golang tools would allow a .godir file to mean "treat this path as if it existed at the described path under GOPATH. Some of these questions include:

 - first run experience ("Where is my GOPATH supposed to be?")

 - global vs relative GOPATHs ("Wait, I have to install this globally to test it? Oh, I can do it locally but I have to write a script to adjust my GOPATH every time I work on this project?")

 - nested directory structures ("Does my project root need to be under my GOPATH? Then how do I have separate GOPATH's for each project?")

This may have been brought up before and I am just using the wrong search terms, but these issues are similar to ones I've heard brought up time and time again by new-to-golang developers and I still run into some of them as a not-so-new-to-golang developer.

All of these tricks and hacks would require systematic, complex, and ill-tested changes to every single tool that must load or process Go source code.  I think the general solution for this development model is: don't do that.  By all means use a hack like a shell alias to make switching between directories with long names easier, but don't insert those hacks into the workflow used by all tools and all tool users.

All due respect, I don't see "all of these tricks and hacks" as being an appropriate term for the addition of a single piece of information at the root of a package. Golang's "don't do that" mindset is lovely in many ways but is not without its faults when it comes to user experience.

As for the "hack," you are surely more informed than I, but I wonder how many places rely upon their own lookups to determine what package some file is in vs a more re-usable method like buildutil.ContainingPackage. That inconsistency is the root of plenty of issues on its own, and if a library or process exists that fixes that inconsistency then this additional information would have a perfect home within it.

--andy