locking down vanity import paths

[Russ Cox]

I have been playing with 'vanity' import paths and cmd/go. Obviously they work. But I am a little worried about the same code showing up under two possible import paths. To be concrete, as an experiment in both vanity import paths and github, I set up 'rsc.io' pointing at 'github.com/rsc', and I dropped a PDF-reading package there as 'rsc.io/pdf'.

'go get rsc.io/pdf' works fine. But so does 'go get github.com/rsc/pdf'. I'd like to somehow encourage (if not force) people to use the first path, not the second, to avoid having the code under two canonical names. One of the features of the vanity import path is that if I get tired of github and move to, say, bitbucket, I just fix the redirect and people who re-go get rsc.io/pdf will keep working. But people who have found and used the direct github import path will break.

Can we do something about this? Should we? The best I have come up with is to drop a file in the root of the repo named 'go.meta' that contains JSON:

{

    "ImportRoot": "rsc.io/pdf"

}

Then 'go get github.com/rsc/pdf' would find this go.meta file, try to resolve 'rsc.io/pdf', find that it maps back to 'github.com/rsc/pdf', and say 'sorry, you're supposed to import rsc.io/pdf instead'. The re-resolving is important: if, say, Brad forks the pdf package as 'github.com/bradfitz/pdf', then he can leave the go.meta file alone and 'go get github.com/bradfitz/pdf' will still work, because 'rsc.io/pdf' does not resolve to 'github.com/bradfitz/pdf' so there is no collision.

One concern about this is that other information will creep into the go.meta file over time.

Russ

[Brad Fitzpatrick]

I'm not sure this is a real problem.

Camlistore's been using vanity imports from day one and despite many emails from users with build/usage questions, this has never come up as an issue.  And Camlistore is also on Github, so the theoretical problem is there... it just hasn't happened.

I would be reluctant to add new mechanisms to solve problems I'm not seeing.

[Ian Lance Taylor]

On Tue, Feb 25, 2014 at 7:56 AM, Russ Cox <r...@golang.org> wrote:
>

> 'go get rsc.io/pdf' works fine. But so does 'go get github.com/rsc/pdf'. I'd
> like to somehow encourage (if not force) people to use the first path, not
> the second, to avoid having the code under two canonical names.

It's easy to encourage people to use the first path by writing a
README file, so all we are really discussing is whether there should
be a way to force it.

Besides the idea of a magic file, we could force it by adding a magic
comment that the go tool would recognize.

// +pkgpath "rsc.io/pdf"

If the go tool sees that comment in a package with a different name,
it gives an error.

But as Brad says I don't know that it matters.

Ian

[Gustavo Niemeyer]

On Tue, Feb 25, 2014 at 2:08 PM, Ian Lance Taylor <ia...@golang.org> wrote:
> Besides the idea of a magic file, we could force it by adding a magic
> comment that the go tool would recognize.
>
> // +pkgpath "rsc.io/pdf"
>
> If the go tool sees that comment in a package with a different name,
> it gives an error.

That would be nice.

> But as Brad says I don't know that it matters.

I have seen the described behavior creating issues before, mainly due
to self-imports: let's say rsc.io/pdf imports rsc.io/pdf/fonts, and
then a user imports that under github.com/rsc/pdf. The contained
import will bring the intended path in as well, effectively importing
the same package twice under different paths. Then, the developer
managed to leave the two packages out of sync, and didn't realize why
killing github.com/rsc/pdf and re-installing had no effect.


gustavo @ http://niemeyer.net

[Russ Cox]

FWIW, the advantage of the go.meta file over package comments is that there can be one per repo instead of one per package (one per directory in the repo).

Another possibility is to recognize a line in README (or README.md) like

GoImportRoot: rsc.io/pdf

That avoids needing a separate file and limits the amount people will try to stuff in there.

[Brad Fitzpatrick]

If we do anything, I would prefer a new JSON file (which has a known syntax) over ambiguity of which file(s) to put some magic line in, and how. 

[Russ Cox]

On Tue, Feb 25, 2014 at 2:58 PM, Matt Reiferson <sna...@gmail.com> wrote:

This might have a negative effect on binaries which like to “vendor” dependencies and re-write dependency import paths, but perhaps this was always a terrible idea and they should be using the alternative approach of building a temporary GOPATH workspace with pinned dependencies.

I don't see the negative effect. Can you elaborate? In the proposal in my mail, in order for the directive to trigger, you have to have 

(1) import "x/y/z" maps to a repo with metadata saying "my name is a/b/c".

(2) looking up the repo "a/b/c" redirects to get the code from "x/y/z".

If you vendor x/y/z into my/x/y/z, then step 1 will still be satisfied (the metadata file will exist) but step 2 will not (you changed the import path), so the restriction does not trigger. In this sense vendoring is not much different than forking.

Russ

[Daniel Theophanes]

If you placed the "go.meta" file inside the repository, you'd still have to fetch the repository to read it. That seems less then optimal and might be confusing over time (fetched the wrong import path but didn't delete it). Perhaps it would clone it, read the meta file, then move it to the ImportRoot location if it didn't exist and inform the user, or remove it if the ImportRoot location already existed.

I like the idea of "ImportRoot", although I don't feel your pain for vanity urls, I would like to know if a public fork was made to add a patch or is truly a fork.

It seems like the idea should wait and perhaps filled out a bit before adding a first of a kind meta file; what is the long term intention for this file? I suspect that people would start asking for "lets add versions to dependencies" or "lets allow central aliases for package dependencies".

-Daniel

[Gary Burd]

The proposed go.meta file can be used to improve godoc.org. Here's how I propose to use the file:

If go.meta exists and ImportRoot is not equal to the root of the current repo then 

     If ImportRoot is a vanity path for the current repo then

         Redirect requests for the current repo to the vanity path

     else 

         Do not include the current repo in search results.

         Include a note on the package pages with a link back

            to ImportRoot and an explanation of why the package

            is not indexed.

Requests fo github.com/rsc/pdf will redirect to rsc.io/pdf. The repo github.com/bradfitz/pdf will not be included in search results.

On Tuesday, February 25, 2014 7:56:42 AM UTC-8, rsc wrote:

[Matt Reiferson]

This might have a negative effect on binaries which like to “vendor” dependencies and re-write dependency import paths, but perhaps this was always a terrible idea and they should be using the alternative approach of building a temporary GOPATH workspace with pinned dependencies.

Worth considering at least…

I guess the question is where is the line drawn?  Is this just for the go tool or does it actually effect lower level semantics?

--
 
---
You received this message because you are subscribed to the Google Groups "golang-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Nathan Youngman]

Gary,

Doesn't the badge/token idea work equally well for godoc.org in this situation?

https://github.com/garyburd/gddo/issues/107#issuecomment-23953268

If the badge includes rsc.io/pdf then the search results would include http://godoc.org/rsc.io/pdf but treat https://github.com/rsc/pdf like a fork that is ignored.

That doesn't prevent people from importing https://github.com/rsc/pdf as with the original proposal, but the combination of the README and fork-free search results should point people in the right direction.

Nathan.

[Andrew Gerrand]

On 26 February 2014 03:03, Brad Fitzpatrick <brad...@golang.org> wrote:

I would be reluctant to add new mechanisms to solve problems I'm not seeing.

I feel the same.

I don't have any specific reservations about this particular mechanism, except that it is more mechanism. Do we need it?

Of all the things people have been asking for in this domain, I haven't heard this one before.

Andrew

[Russ Cox]

I've spent a while thinking about this, including the responses both on and off list, and I think this is something we need to fix for Go 1.3. Aliasing is one of those "programming in the large" issues that Go explicitly tries to tackle: a package needs to have just one import path, not two. We disallow relative imports in GOPATH code for the same reason, but this aliasing situation is actually worse, because aliases take away the ability to move the code without breaking people.

I see two solutions on the table today:

1) go.meta file in the repo root.

2) some kind of directive line in the README

Does anyone else want to suggest something? I don't believe "do nothing" is a viable option.

Thanks.

Russ

[Brad Fitzpatrick]

If we have to do something, I argue for a go-meta.json file in the root, or something ending in ".json" and not ".meta".

[Andrew Gerrand]

I think the magic line in README is probably the wrong approach.

Presuming we go with the json meta file, what about third parties putting stuff in there? Is that sanctioned? Forbidden? Are we apathetic to it?

[Russ Cox]

On Wed, Feb 26, 2014 at 11:48 PM, Andrew Gerrand <a...@golang.org> wrote:

I think the magic line in README is probably the wrong approach.

Presuming we go with the json meta file, what about third parties putting stuff in there? Is that sanctioned? Forbidden? Are we apathetic to it?

i'm definitely worried about other stuff going into go.json. that's part of why i like the README approach. it's ugly, which will limit how much more it gets used.

[Nathan Youngman]

Would ImportRoot also be helpful to situations where a repository has moved?

Example 1

github.com/howeyc/fsnotify moved to code.google.com/p/go.exp/fsnotify

{

    "ImportRoot": "code.google.com/p/go.exp/fsnotify"

}

Example 2

https://github.com/gophertown/gat renamed to https://github.com/gophertown/looper

Moves/renames within the GitHub ecosystem will be redirected, but go get will still download to the gat folder.

Nathan.

[Dan Kortschak]

On Thu, 2014-02-27 at 15:48 +1100, Andrew Gerrand wrote:
> Presuming we go with the json meta file, what about third parties
> putting stuff in there? Is that sanctioned? Forbidden? Are we
> apathetic to it?

Hopefully it will not be allowed to evolve (degenerate?) into something
like pom.xml.

[Gary Burd]

Or this: https://www.npmjs.org/doc/files/package.json.html 

[minux]

why we can't have a special line in the root package (shortest import path of all packages

from a single repository) of a repository? e.g.

// +import path: rsc.io/pdf

package pdf

and when go getting the package, check the top level Go file for this annotation, and

reject if it's different from the path used to import this.

[Nathan Youngman]

On Wednesday, 26 February 2014 21:42:40 UTC-7, rsc wrote:

I've spent a while thinking about this, including the responses both on and off list, and I think this is something we need to fix for Go 1.3. Aliasing is one of those "programming in the large" issues that Go explicitly tries to tackle: a package needs to have just one import path, not two.

3) What about an independent linting tool that warns for import paths that represent the same repository?

Import for "github.com/rsc/pdf' found, primary is "rsc.io/pdf".

Imports for labix.org/v2/mgo and launchpad.net/mgo/v1 found. (Yikes!)

[minux]

I also have this fear. What about the // +import path: magic comment?

I have revised that idea.

we have // +build line, how about we have

// +import

line that checks the import path?

[Nathan Youngman]

On Tuesday, 25 February 2014 10:40:23 UTC-7, Gustavo Niemeyer wrote:

I have seen the described behavior creating issues before, mainly due
to self-imports: let's say rsc.io/pdf imports rsc.io/pdf/fonts, and
then a user imports that under github.com/rsc/pdf. The contained
import will bring the intended path in as well, effectively importing
the same package twice under different paths.

What if the go tool could detect this case... if importing github.com/rsc/pdf imports rsc.io/pdf/fonts which is a vanity url for github.com/rsc/pdf/fonts?

If that produced an error, then you could actually just use self-imports to prevent people from using github.com/rsc/pdf. :-)

(not really a good solution)

On 26 February 2014 03:03, Brad Fitzpatrick <brad...@golang.org> wrote:

I would be reluctant to add new mechanisms to solve problems I'm not seeing.

To be honest, I wonder how many Gophers use vanity imports, or even know about them?

Nathan.

[roger peppe]

The problem with this is that no-one will remember to run the tool.

If it could be made automatic, it would be great, but I think the
only way that's possible is to make go get store the final URL
for a repo somewhere, so it can be checked when compiling.

Unfortunately, there's no place to put that info - we don't want
to add a file in the source directory, and if we put it in
$GOPATH/pkg, the info will be lost when the pkg directory
is removed.

Also it wouldn't work when repositories are fetched
manually without go get.

So on balance, I think I support Russ's metadata file
suggestion. One way to avoid arbitrary expansion
of its scope is to store just a simple string holding the
canonical import path. Parse by reading the file and stripping
white space.

cheers,
rog.

[David Symonds]

On 27 February 2014 15:53, Russ Cox <r...@golang.org> wrote:

> i'm definitely worried about other stuff going into go.json. that's part of
> why i like the README approach. it's ugly, which will limit how much more it
> gets used.

I have no doubts that if someone would want to cram something in
go.json then they would not hesitate to cram it in a README. I see
README-based metadata worse than a go.json file.

[minux]

Why not store them in the Go source files? e.g.

// +import canonical/import/path

and make cmd/go check them at go get time.

I really don't like the idea of additional meta data (however simple it's, it

will get (ab)used.)

[Jan Mercl]

On Tue, Feb 25, 2014 at 4:56 PM, Russ Cox <r...@golang.org> wrote:
> I have been playing with 'vanity' import paths and cmd/go.

Technical note (not advocating anything): The symptom is rooted in the
magic redirection capability. One way to get rid of the problem is to
not provide the magic. I don't have statistic data about how often it
is used in the wild, but I have never seen it in any Go package I, or
my employer, have ever used - all of them are directly in
code.google.com or github.com, AFAIR.

Wrt metadata format and placement: One approach to non-extendability
is perhaps to make the metadata content zero size. What about having
an empty file named 'rsc.io%2fpdf.import' (or any other variant of
that [url encoded] scheme) in the root (should it be actually
package?) directory? 'go get' would consider the file iff there's
exactly one such and iff it's of zero size. That hopefully makes other
legitimate uses conflict surface negligible, I guess. Alternatively it
could be a "dot" file: '.rsc.io%2fpdf'.

> Then 'go get github.com/rsc/pdf' would find this go.meta file, try to resolve 'rsc.io/pdf',
> find that it maps back to 'github.com/rsc/pdf', and say 'sorry, you're supposed to import
> rsc.io/pdf instead'. The re-resolving is important: if, say, Brad forks the pdf package
> as 'github.com/bradfitz/pdf', then he can leave the go.meta file alone and 'go get
> github.com/bradfitz/pdf' will still work, because 'rsc.io/pdf' does not resolve to
> 'github.com/bradfitz/pdf' so there is no collision.

IIUC, Brad's fork is effectively not go gettable. Trying to go get it
would be redirected to go getting the pointed-to repository
github.com/rsc/pdf. I'm not sure what Brad's fork is good for then?
Also, what should be the correct outcome if Brad's fork and the
redirection target are not bit-by-bit the same? This hints me, and now
I am advocating, that maybe the whole redirection magic should be
reconsidered. In the extreme sense, it's also an attack vector, isn't
it?

-j

[David Symonds]

On 27 February 2014 20:27, Jan Mercl <0xj...@gmail.com> wrote:

> Technical note (not advocating anything): The symptom is rooted in the
> magic redirection capability. One way to get rid of the problem is to
> not provide the magic. I don't have statistic data about how often it
> is used in the wild, but I have never seen it in any Go package I, or
> my employer, have ever used - all of them are directly in
> code.google.com or github.com, AFAIR.

To be fair, there's a bit of a bootstrapping problem here: the support
for vanity import paths isn't great, which might be discouraging
people from using it. We won't know whether people really want to use
it in the wild until it's well-supported.

[Dan Kortschak]

On Thu, 2014-02-27 at 03:35 -0500, minux wrote:
> Why not store them in the Go source files? e.g.
> // +import canonical/import/path
> and make cmd/go check them at go get time.

How do you deal with multi-file packages? Do they all need an +import
line or only one, or strictly one? If more that one how to deal with
conflicts?

[Robin Eklind]

> IIUC, Brad's fork is effectively not go gettable. Trying to go get it
> would be redirected to go getting the pointed-to repository
> github.com/rsc/pdf. I'm not sure what Brad's fork is good for then?
> Also, what should be the correct outcome if Brad's fork and the
> redirection target are not bit-by-bit the same? This hints me, and now
> I am advocating, that maybe the whole redirection magic should be
> reconsidered. In the extreme sense, it's also an attack vector, isn't
> it?
>
> -j
>

This is also what worries me. If this suggestion would make it
impossible to go get a fork it could seriously hurt the open source
nature that Go currently encourages [1].

There are at least two valid reasons for creating a fork and using it
instead of the original repository:

1) An actual fork with custom modifications.
2) An identical copy of 3rd party code to lock the revision and make
sure it doesn't do any malicious updates.

The first case isn't affected by this since if you are making a true
fork than the canonical import path could also be removed.

The second case however would no longer be go getable, which in my
opinion is quite limiting.

Anyways, that's just my 2 cent. Not really trying to turn this into a
bikeshed discussion.

Regards /u

[1]: http://blog.natefinch.com/2013/01/go-is-for-open-source.html

[Damian Gryski]

On Thursday, February 27, 2014 10:27:26 AM UTC+1, Jan Mercl wrote:

On Tue, Feb 25, 2014 at 4:56 PM, Russ Cox <r...@golang.org> wrote:
> I have been playing with 'vanity' import paths and cmd/go.

Technical note (not advocating anything): The symptom is rooted in the
magic redirection capability. One way to get rid of the problem is to
not provide the magic. I don't have statistic data about how often it
is used in the wild, but I have never seen it in any Go package I, or
my employer, have ever used - all of them are directly in
code.google.com or github.com, AFAIR.

    A useful metric: the top domains hosting Go packages, as indexed by go-search: http://go-search.org/tops

    Damian

[minux]

any one file.

if there are more than one, with different import path, by definition, the check won't pass. then it's just a matter of better diagnostics.

[Gustavo Niemeyer]

On Thu, Feb 27, 2014 at 5:35 AM, minux <minu...@gmail.com> wrote:
> Why not store them in the Go source files? e.g.
> // +import canonical/import/path
> and make cmd/go check them at go get time.

This still sounds like the most natural extension from where we are. I
suppose Russ has been resisting this because in theory all the
subpackages would require an equivalent line, or we'll have to
introduce new logic that walks up the path and looks for such a line
in every go file in parent directories, which isn't ideal.


gustavo @ http://niemeyer.net

[minux]

go get needs to read every go source file, so that's not a problem.

we don't need to require the import line to be present in every file, just one will do. (in fact, i propose we just define go will check every line like that and if they don't all match, go get will always fail)

the problem i can think of is that every fork needs to change the import line, but i expect for non-trivial projects, people will need to change actual import lines anyway. however, as the check only happens at go get time, the usual git fetch/branch trick with already go get packages will still work.

[Rob Pike]

This seems like an issue that needs careful thought. I don't want to
rush the decision in order to get it into Go 1.3.

-rob

[Robert Griesemer]

I'm concerned about adding mechanism for a problem that few people he encountered (at the very least I haven't heard people complaining about it yet).

I'm very concerned about rushing an untried mechanism for 1.3.

- gri

--

[Gustavo Niemeyer]

On Thu, Feb 27, 2014 at 1:33 PM, minux <minu...@gmail.com> wrote:
>
> On Feb 27, 2014 11:19 AM, "Gustavo Niemeyer" <gus...@niemeyer.net> wrote:
>> On Thu, Feb 27, 2014 at 5:35 AM, minux <minu...@gmail.com> wrote:
>> > Why not store them in the Go source files? e.g.
>> > // +import canonical/import/path
>> > and make cmd/go check them at go get time.
>>
>> This still sounds like the most natural extension from where we are. I
>> suppose Russ has been resisting this because in theory all the
>> subpackages would require an equivalent line, or we'll have to
>> introduce new logic that walks up the path and looks for such a line
>> in every go file in parent directories, which isn't ideal.
>
> go get needs to read every go source file, so that's not a problem.
>
> we don't need to require the import line to be present in every file, just
> one will do. (in fact, i propose we just define go will check every line
> like that and if they don't all match, go get will always fail)

It needs to look for it in every file until it finds it, and in every
parent package, or every subpackage in the repository will need a line
on its own.

It's also not clear that checking only during "go get" and not "go
build" or "go install" is a good idea. It means people using the
package remotely might see errors that you don't see locally.


gustavo @ http://niemeyer.net

[gov...@ver.slu.is]

This is more out of my own curiosity than an actual suggestion, so please take it with a grain (or pound) of salt.

What would be against extending the package clause to allow setting an absolute package path, which can be checked by the compiler?

Eg. allow both:

package pdf

and:

package rsc.io/pdf

No compatibility promise is broken, syntax is merely extended for those who wish to use it and it is kept in line with how import statements look.

Also, it can be enforced across the package and doesn't result in any additional "magic comments" or metadata files with an unclear intent or future.

I could understand arguments against this being not wanting to further complicate the language, or wanting to have two similar but not identical ways of declaring a package.

I'm interested in the arguments against this that I haven't thought of.

[Jan Mercl]

On Fri, Feb 28, 2014 at 10:14 AM, <gov...@ver.slu.is> wrote:
> I'm interested in the arguments against this that I haven't thought of.

What would then be the package qualifier of a package currently
imported as, for example, import "github.com/skelterjohn/go.uik" or
import "foo.com/go-bar"?

package github.com/skelterjohn/go.uik

...

-j

PS: Another argument for considering using punctuation in an import
path basename a mistake.

[Rob Pike]

An idea we were chatting about yesterday was

package foo "github.com/gopherdelight/foo"

That is, somewhat in parallel with the import clause, providing an
optional package *path* to go with the package name. It would only
need to appear once in the package and obviously if it appeared more
than once it would need to be consistent. The compiler could verify
that during import of the package that the path is correct.

It has advantages and disadvantages but seems generally in alignment
with the language and ecosystems as it stands today, and is completely
backwards compatible.

If we do go that route, it should only be added in cases where there
is a risk of incorrect import path. The vast majority of packages
should never need the noise.

-rob

[Brad Fitzpatrick]

That sounds better than a new file.

[ron minnich]

That's a very nice idea.

[james4k]

Looks like a solid solution, but it seems like a concern that is more appropriate for the go tool to manage rather than the compiler.

[Daniel Theophanes]

I doubt I'd use this on any of our internal projects, but it is highly likely I would use this on *ALL* of my open source projects. It would allow others a way to determine if a fork is a divergent fork or if it is just a bug-fix fork with an intent to be merged back in (or just a locally controlled copy). Does that fit with your intent?

On Fri, Feb 28, 2014 at 8:23 AM, Rob Pike <r...@golang.org> wrote:

--

---
You received this message because you are subscribed to a topic in the Google Groups "golang-dev" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/golang-dev/RgHGBXROeAw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to golang-dev+...@googlegroups.com.

[jlmo...@gmail.com]

Significant Go projects have moved in the past:

  * postgres driver github.com/bmizerany/pq -> github.com/lib/pq

  * gorilla web toolkit code.google.com/p/gorilla -> github.com/gorilla

  * revel from github.com/robfig/revel -> github.com/revel/revel

As far as I know, this has not been a major problem.  Adding a package level metadata file is more like a slippery cliff than a slippery slope;  there's no chance it won't be "enhanced" by 3rd party tools, or encourage constant, insufficiently thoughtful requests for new inclusions.  It should definitely be a last resort to solve a major pain point people actually experience, rather than a hypothetical one.

-- Jason

[Brendan Tracey]

If this change were to occur, there needs to be a constraint on “excessive” vanity paths, such as

package graph “graph”   // as opposed to, say, “github.com/btracey/graph”

One of the benefits of the current standard is that there aren’t naming clashes; no one can “reserve” a name in the go ecosystem. Yes, it may be bad practice to do the above, but it benefits the user side (“If you use my graph package, you just need to say ‘import “graph” ‘ “) and it benefits the coder side due to the lock-in effects (If it’s established that I have the “graph” space, a competitor cannot, and then I gain an ‘ease of use’ argument). It doesn’t seem like this is a problem with the proposal, but I wanted to point out the concern so we can verify that it is not.

[Andrew Gerrand]

My understanding is that you can't take a vanity path that you don't actually own. That is, you can't have "graph" because you don't own the TLD "graph". The clause only works if you specify a canonical path and the code actually exists at that path. 

You received this message because you are subscribed to the Google Groups "golang-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+...@googlegroups.com.

[Andrew Gerrand]

By backward compatible do you mean that it would work with Go 1.2? Because I don't see how that is the case. Shouldn't the compiler complain of an unexpected string literal?

[minux]

On Feb 28, 2014 3:31 PM, "Andrew Gerrand" <a...@google.com> wrote:

> By backward compatible do you mean that it would work with Go 1.2? Because I don't see how that is the case. Shouldn't the compiler complain of an unexpected string literal?

yeah, that's a problem.
and it's the reason why i proposed the "// +import" line rather than changing the language.

i don't think this problem is big enough to change the language.

[Rob Pike]

It's compatible in the sense that a[i:j:k] was: only software that
uses the feature (or depends on etc.) is affected. No existing code
breaks.

-rob

[james4k]

It does mean the .go file with the extended package clause will need a // +build go1.3 tag as well.

http://play.golang.org/p/cJiWKJVAmB

[minux]

Yes, but I'd argue that this problem is different from a[i:j:k] fundamentally.

a[i:j:k] is impossible to do in Go 1.1 and earlier without unsafe, so using

it means deliberately breaking Go 1.1 and previous users (just like using

any newer APIs).

However, locking the vanity import path alone shouldn't render an otherwise

Go 1.2 clean package to be Go 1.3 only. I expect some users might not

start upgrading to Go 1.3 shortly after it is released, and any uses of vanity

import locked package will break their compilation, which is unfortunate.

The workaround is to add an empty source file like this:

// +build go1.3

package pdf "rsc.io/pdf"

to a package. I'd argue it's not better than adding this:

// +import "rsc.io/pdf"

package pdf

The compiler is already accepting commands in comments, so that's

not a problem too, if we want to check the import path at every build

even without the go command (this might pose problems for users of

custom build, though).

[Andrew Gerrand]

I like "// +import" because it solves the same problem (reporting information to the go tool) as build tags.

 

[minux]

On Fri, Feb 28, 2014 at 4:01 PM, minux <minu...@gmail.com> wrote:

On Fri, Feb 28, 2014 at 3:49 PM, Rob Pike <r...@golang.org> wrote:

It's compatible in the sense that a[i:j:k] was: only software that
uses the feature (or depends on etc.) is affected. No existing code
breaks.

Yes, but I'd argue that this problem is different from a[i:j:k] fundamentally.

a[i:j:k] is impossible to do in Go 1.1 and earlier without unsafe, so using

it means deliberately breaking Go 1.1 and previous users (just like using

any newer APIs).

However, locking the vanity import path alone shouldn't render an otherwise

Go 1.2 clean package to be Go 1.3 only. I expect some users might not

start upgrading to Go 1.3 shortly after it is released, and any uses of vanity

import locked package will break their compilation, which is unfortunate.

The workaround is to add an empty source file like this:

// +build go1.3

package pdf "rsc.io/pdf"

to a package. I'd argue it's not better than doing this:

// +import "rsc.io/pdf"

package pdf

The compiler is already accepting commands in comments, so that's

not a problem too, if we want to check the import path at every build

even without the go command (this might pose problems for users of

custom build, though).

To clarify, my position is that this should be handled by the go command

(e.g. go get and go install/build), not by the compiler. It's not a big problem

to change the language, at least not in a way to break Go 1.2 and early

versions.

[Lucio De Re]

Call me ungrateful, but I'd like to ask for a package method or even
constant that returns the "canonical" package path. And another for
the "actual" package path.

The rationale is that sometimes I expect additional information (the
most recent being templates) to be stored in the package's
neighbourhood.

Of course, this is a vote of confidence in the proposed idea.

Lucio.

> You received this message because you are subscribed to the Google Groups
> "golang-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an

[Nathan Youngman]

package foo "github.com/gopherdelight/foo"

This is very nice. 

Build tags may skip compilation of a single file, whereas this halts compilation of a package. I think it makes sense to use something different.

+ It's only verifying the path on disk rather than hitting the network as in the original proposal.

+ No chance of becoming a metadata file for every odd and end.

Would it be available via the ast package for rewriting package imports?

Does this apply to package main in command line tools?

If many packages do have this additional noise, are there any concerns beyond the noise?

Possible uses:

* Ensure vanity urls are used to handle future package moves, package pdf "rsc.io/pdf".

* Moved repositories, package main "code.google.com/p/go.tools/cmd/goimports" (at GitHub location).

* Renamed repositories (essentially the same, GitHub will redirect but "go get" puts code in the requested folder, this could prevent that). Example: go get github.com/gophertown/gat will get the code for looper (new name).

* Gary Burd has been looking for ways to remove temporary forks from search results on godoc.org. A canonical import path would help.

* It encourages cloning to the canonical location (for example, I've worked under the src/github.com/howeyc/fsnotify folder, not under src/github.com/gophertown/fsnotify based on my fork location). This keeps all my fsnotify import paths the same.

Possible issues:

* Another thing to fix when doing import rewriting for forks/vendor.

* No manual clone workaround if it's a compile-time check. (is that bad?)

* Unlike the original post, "go get github.com/bradfitz/pdf" won't work unless the canonical import path is changed in the fork. Instead one would "go get rsc.io/pdf" and then add an additional remote for bradfitz's fork to pull in his code. (I think this is a good thing, but it requires familiarity with adding remotes).

* Not able to compile under Go 1.2 or less. (without a +build go1.3 tag as per minux)

Nathan.

--
Nathan Youngman
Email: n...@nathany.com
Web: http://www.nathany.com

[minux]

On Sat, Mar 1, 2014 at 1:41 AM, Nathan Youngman <n...@nathany.com> wrote:

package foo "github.com/gopherdelight/foo"

This is very nice.

What's your opinion about the "// +import " proposal? I don't think it's any worse

than that and it maintains backward compatibility. 

Build tags may skip compilation of a single file, whereas this halts compilation of a package. I think it makes sense to use something different.

+ It's only verifying the path on disk rather than hitting the network as in the original proposal.

this benefit also present with // +import. 

+ No chance of becoming a metadata file for every odd and end.

same. 

Would it be available via the ast package for rewriting package imports?

as I argued before, it's not a problem that must be solved at language level,

changing cmd/go is enough. go/build should provide this info in Package struct. 

Does this apply to package main in command line tools?

no need to restrict this feature. if you set import path for a main package, it will

be checked. 

If many packages do have this additional noise, are there any concerns beyond the noise?

we probably should discourage this feature instead of encouraging its use (e.g. only recommended

for vanity import paths). It makes forking the code more difficult, among other issues.

Possible uses:

* Ensure vanity urls are used to handle future package moves, package pdf "rsc.io/pdf".

* Moved repositories, package main "code.google.com/p/go.tools/cmd/goimports" (at GitHub location).

* Renamed repositories (essentially the same, GitHub will redirect but "go get" puts code in the requested folder, this could prevent that). Example: go get github.com/gophertown/gat will get the code for looper (new name).

* Gary Burd has been looking for ways to remove temporary forks from search results on godoc.org. A canonical import path would help.

yes. it will definitely help eliminating temporary forks (those with wrong import constrains) from godoc.org.

* It encourages cloning to the canonical location (for example, I've worked under the src/github.com/howeyc/fsnotify folder, not under src/github.com/gophertown/fsnotify based on my fork location). This keeps all my fsnotify import paths the same.

Possible issues:

* Another thing to fix when doing import rewriting for forks/vendor.

any solution to this problem will have this issue. 

* No manual clone workaround if it's a compile-time check. (is that bad?)

again, this issue is present for all compile-time check solutions. 

* Unlike the original post, "go get github.com/bradfitz/pdf" won't work unless the canonical import path is changed in the fork. Instead one would "go get rsc.io/pdf" and then add an additional remote for bradfitz's fork to pull in his code. (I think this is a good thing, but it requires familiarity with adding remotes).

yes. Solving this issue essentially forces every fork to update the import lines, but forks

(if intended to be go getable) must modify the actual import statement anyway, so this

is a big deal. 

* Not able to compile under Go 1.2 or less. (without a +build go1.3 tag as per minux)

// +import doesn't have this problem. graceful degrade.

[Dan Kortschak]

Yes, this seems like a build tool domain problem, not a compiler domain problem.

[nate....@gmail.com]

This seems to be fixing a problem no one is complaining about. I get that's it's theoretically a problem, but if no one is complaining, it must not happen often.

As such, I suggest pursuing the least impactful change possible, if anything is done at all.

Adding any kind of configuration file is anathema to me. That's one of the things I like best about go - no extraneous files, just code.

[Nathan Youngman]

On Friday, 28 February 2014 23:56:17 UTC-7, minux wrote:

On Sat, Mar 1, 2014 at 1:41 AM, Nathan Youngman <n...@nathany.com> wrote:

package foo "github.com/gopherdelight/foo"

What's your opinion about the "// +import " proposal? I don't think it's any worse

than that and it maintains backward compatibility. 

If the "canonical import path" gains widespread adoption (say for the godoc.org use case) then I can see some benefit of not requiring Go 1.3 to compile packages that use it.

Is this important mainly for end-users compiling from source, with say Ubuntu 14.04 LTS?

For developers using Go, wouldn't we prefer that they upgrade to 1.3, even if it means working around this feature for fork/vendor workflows? (package pkg "you.com/external/original.com/pkg")

Compatibility with Go 1.2 will eventually be a non-issue (and you presented a workaround in the interim).

In a few years from now, which would you rather use?

Build tags may skip compilation of a single file, whereas this halts compilation of a package. I think it makes sense to use something different.

That, and I think having the canonical import path on the same line as the package is slightly cleaner (as a user reading the code). I have no strong preference as to whether the checks are implemented at compile-time or get/build time.

Both options address the issue Russ Cox mentioned and have a variety of other uses for those using go get and for godoc.org.

Nathan.

P.S. Being March 1st, I'm not sure if my references to Go 1.3 should really be 1.4.

[Robin Eklind]

> Build tags may skip compilation of a single file, whereas this halts
> compilation of a package. I think it makes sense to use something different.

I still haven't made up my mind if vanity imports actually solves a real
problem that many users of Go currently experiences.

I like the fact that with build tags you always know what is happening
by simply reading the code on the page. This is often quoted as: "In Go,
code does what it says on the page."

If a single file could contain `package foo "github.com/user/foo"`,
while all other files contain `package foo` this would no longer be
true. Say if there are 20 files in a package (like in the net package)
one would have to use grep to poll for this information, instead of
having it visible right before your eyes.

As much as I dislike a magic file, at least there is only one file per
repository. So if a repository has such a file, it would be obvious that
vanity imports are in use.

[Daniel Theophanes]

In a couple of years I'd rather use a formal annotation specification that is part of the formal spec. Maybe that just means adding "// +..." To the spec.  But it looks like there is a general need to add annotations to packages, files, and functions. If we break compatibility then let's design a complete formally parsed solution then just an ad-hoc change.

[nate....@gmail.com]

If we need a canonical place, why not import.go? Along the lines of doc.go.

[zom...@gmail.com]

Being able to pin a package to a specific version:

import "packagename" "version"

would seem more valuable IMHO than handling vanity imports.

Just my two cents,

Tom

[zom...@gmail.com]

[Nathan Youngman]

A canonical import path could be (ab)used by the author to force upgrades to a new version.

If at some point in time Gustavo modified the code at launchpad.net/mgo/v1 to include:

  package mgo "labix.org/v2/mgo"

That would prevent others from (unintentionally) using the old version or mixing v1 and v2 in a single application. I'm not sure if this is a good thing (or a real problem), but a potential "creative use" of a feature intended for locking down vanity paths.

@Tom, not at all what you were suggesting, I know. If you want to discuss versioning further I suggest starting a new discussion thread. My summary/thoughts on it are here: http://nathany.com/go-packages/ It's one of those topics that I think needs a lot more experience than I have, and a lot of hammock-time.

--

---
You received this message because you are subscribed to a topic in the Google Groups "golang-dev" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/golang-dev/RgHGBXROeAw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to golang-dev+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[James Gray]

To put it another way, this change would break the go1 compatibility
guidelines. Language changes that are not backwards compatible are not
to be introduced in go1.x

[minux]

On Sat, Mar 1, 2014 at 7:24 PM, James Gray <ja...@james4k.com> wrote:

To put it another way, this change would break the go1 compatibility
guidelines. Language changes that are not backwards compatible are not
to be introduced in go1.x

FYI, for the go1 compatibility point of view, this change is not forbidden,

as the compatibility promise is only about backward compatibility not

forward compatibility. (e.g. go 1.x program should run in go1.y environment

when y >= x; but go1.y might not be able run in go1.x environment when y > x)

Quote http://golang.org/doc/go1compat:

These same considerations apply to successive point releases. For instance, code that runs under Go 1.2 should be compatible with Go 1.2.1, Go 1.3, Go 1.4, etc., although not necessarily with Go 1.1 since it may use features added only in Go 1.2

[james4k]

I see. :(

That makes sense, I misunderstood that then. 

I still think this doesn't warrant a language change. If you have packages in your workspace, there should be no question what their import paths should be. They should continue to be implicitly defined through your directory structure, no strings attached. This change is essentially only about retrieving the correct package, which is where the go tool is responsible.

[David Symonds]

If we pursue the package clause option, there are two things I'd like:
- make it structured, like struct field tags (e.g.: package foo
`go:"github.com/gopherdelight/foo"`) so that such a significant change
permits more changes in the future without it needing to be a language
change.
- postpone it to Go 1.4, and have the Go 1.3 compiler accept but
ignore it. When Go 1.4 rolls around we can more reasonably assume that
most Go programmers will have upgraded to a version that won't break.

[Russ Cox]

Why make it structured? Import paths are not structured. We don't write:

import `go:"math/rand" foo:"bar"`

[David Symonds]

On 4 March 2014 03:47, Russ Cox <r...@golang.org> wrote:

> Why make it structured? Import paths are not structured. We don't write:
>
> import `go:"math/rand" foo:"bar"`

For exactly the same reason that we recommend (not require) struct
field tags to be structured: so things can be added later.

[Russ Cox]

I think you are missing the parallel with import paths:

package math "foo/math"

import math "foo/math"

Extensibility is explicitly not a design goal here. That's the whole reason not to do go.meta or some other thing. It will just grow and grow.

Russ

[Andrew Gerrand]

Are we converging on something here?

On the approach: Legitimate concerns have been raised about the special file, changing the language, and README files. That leaves "// +import", which AFAICT is the only unchallenged approach. I am in favor of this approach, if any.

On the timing: it seems too late for this to be in Go 1.3. If that's the case, should we file an issue and shelve this discussion until we've locked down most of the Go 1.3 issues?

[Rob Pike]

This will not happen in 1.3, so yes, let's mark it for 1.4 and come back to it.

-rob

[Russ Cox]

On Mon, Mar 3, 2014 at 7:46 PM, Andrew Gerrand <a...@golang.org> wrote:

Are we converging on something here?

I talked to Rob offline a few days ago and didn't update the thread. I agree with Rob and Robert that we should wait until Go 1.4. I created issue 7453. I don't mind if the discussion continues here but it's all moot for now.

Russ

[Gustavo Niemeyer]

Would it be worth at least ignoring a string used after the package
name, so if the feature is introduced in 1.4 it won't break people
using 1.3?

> --
>
> ---
> You received this message because you are subscribed to the Google Groups
> "golang-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to golang-dev+...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.

--

gustavo @ http://niemeyer.net

[minux]

On Mar 5, 2014 9:02 AM, "Gustavo Niemeyer" <gus...@niemeyer.net> wrote:
> Would it be worth at least ignoring a string used after the package
> name, so if the feature is introduced in 1.4 it won't break people
> using 1.3?

i don't think we've reached concensus to do this at language level.

[Gustavo Niemeyer]

I understand, and we don't have to reach consensus on the details. If
there's some basic agreement that it can work that way, it may be
worth doing at least the part of it that prevents 1.3 from rejecting
such files.


gustavo @ http://niemeyer.net

[Oleku Konko]

You have a valid point

[Andrew Gerrand]

It is unlikely that we will go down this route. Certainly not likely enough to warrant a defensive measure at this stage.

[James Gray]

Just another idea I thought I'd throw into the mix for future
discussion, although I don't expect it to be popular:

Documentation for package declarations are already defined in a single
place per-package. Perhaps if the "Package pkgname ..." convention is
followed, the canonical import path could be defined here as well.

// Package vani.ty/pkgname is an example package.
package pkgname

> --
>
> ---
> You received this message because you are subscribed to a topic in the
> Google Groups "golang-dev" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/golang-dev/RgHGBXROeAw/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to

[minux]

On Thu, Mar 6, 2014 at 1:10 AM, James Gray <ja...@james4k.com> wrote:

Just another idea I thought I'd throw into the mix for future
discussion, although I don't expect it to be popular:

Documentation for package declarations are already defined in a single
place per-package. Perhaps if the "Package pkgname ..." convention is
followed, the canonical import path could be defined here as well.

// Package vani.ty/pkgname is an example package.
package pkgname

This is actually against the established godoc convention.

also, what if people are using languages other than english for the package

document? how could we extract the vanity path out?

[Elazar Leibovich]

Just wanted to add, that we already have special file for package documentation. So one can check for the // +import comment there. Using already established standard to make searching for the tag easier.

On Friday, February 28, 2014 11:04:32 PM UTC+2, Andrew Gerrand wrote:

I like "// +import" because it solves the same problem (reporting information to the go tool) as build tags.

 

On 1 March 2014 08:01, minux <minu...@gmail.com> wrote:

On Fri, Feb 28, 2014 at 3:49 PM, Rob Pike <r...@golang.org> wrote:

It's compatible in the sense that a[i:j:k] was: only software that
uses the feature (or depends on etc.) is affected. No existing code
breaks.

Yes, but I'd argue that this problem is different from a[i:j:k] fundamentally.

a[i:j:k] is impossible to do in Go 1.1 and earlier without unsafe, so using

it means deliberately breaking Go 1.1 and previous users (just like using

any newer APIs).

However, locking the vanity import path alone shouldn't render an otherwise

Go 1.2 clean package to be Go 1.3 only. I expect some users might not

start upgrading to Go 1.3 shortly after it is released, and any uses of vanity

import locked package will break their compilation, which is unfortunate.

The workaround is to add an empty source file like this:

// +build go1.3

package pdf "rsc.io/pdf"

to a package. I'd argue it's not better than adding this:

// +import "rsc.io/pdf"

package pdf

The compiler is already accepting commands in comments, so that's

not a problem too, if we want to check the import path at every build

even without the go command (this might pose problems for users of

custom build, though).

[minux]

On Thu, Mar 6, 2014 at 3:51 AM, Elazar Leibovich <ela...@gmail.com> wrote:

Just wanted to add, that we already have special file for package documentation. So one can check for the // +import comment there. Using already established standard to make searching for the tag easier.

yeah, we can also come up with an empty import.go for this purpose.

also, even we don't use specific file name for the //+import line, it's

just a single grep away. 

[Nathan Youngman]

I wanted to reopen this discussion now that planning is underway for Go 1.4.

As yiyus pointed out, the recent "Internal" Packages proposal could be effective at locking down vanity import paths: 

https://groups.google.com/d/msg/golang-dev/_cAggq73yME/O52XOkRCcywJ

That doesn't provide the same level of information to tooling (like GoDoc.org), so it would be good to weigh the pros and cons. 

Also, is this something that can wait a year for Go 1.5, in order to coincide with the proposal for "Internal" Packages for GOPATH workspaces?

Nathan.

[Russ Cox]

I am going to post a proposal for this next week.

[Nathan Youngman]

Thanks Russ. That's awesome.

In addition to the original problem, I'd like to find a way for GoDoc.org to rank the canonical import path higher than the direct github path or the many forks (or even filter those ones out entirely).

https://github.com/golang/gddo/issues/172

Right now finding the distinction between a vanity import path and a fork seems difficult to me. 

Maybe there is a simple solution waiting to be found. Something to think about.

Nathan.

[Alberto García Hierro]

I'd really love to see a solution which does the exactly opposite of the initial proposal. Rather than allowing a single import path, I'd like to blacklist one. This will prevent users from installing the package directly from GitHub without the downside of breaking forks. We're currently using this at gopkgs.com, but we're doing it via a Go file which panics when the package is imported. See, for example http://gopkgs.com/download/gopkgs.go/magick/gopkgs.go (the file is auto generated). It would be really nice if we could emit an error at build time.

Regards,

Alberto

[Russ Cox]

On Fri, Jun 20, 2014 at 3:34 AM, Alberto García Hierro <alb...@garciahierro.com> wrote:

I'd really love to see a solution which does the exactly opposite of the initial proposal. Rather than allowing a single import path, I'd like to blacklist one. This will prevent users from installing the package directly from GitHub without the downside of breaking forks. We're currently using this at gopkgs.com, but we're doing it via a Go file which panics when the package is imported. See, for example http://gopkgs.com/download/gopkgs.go/magick/gopkgs.go (the file is auto generated). It would be really nice if we could emit an error at build time.

Forks of any complex package tree already need to rewrite import paths that are referring to the forked repository. Rewriting the canonical import is not a significant additional burden.

Russ

[Dominik Honnef]

On Tue, Feb 25, 2014 at 10:56:42AM -0500, Russ Cox wrote:
> One of the features of the vanity import path is that if I get tired
> of github and move to, say, bitbucket, I just fix the redirect and
> people who re-go get rsc.io/pdf will keep working. But people who
> have found and used the direct github import path will break.

For what it is worth, with the way `go get`, and `go get -u` in
particular, work today, even vanity imports have an issue here. If I
`go get rsc.io/pdf` while it points to github, I'll get a git
repository with a remote that points to github. Any future `go get -u
rsc.io/pdf` will just run `git pull` on that remote, even if you moved
to bitbucket in the meantime. That means I need to be aware that the
code moved and manually remove my copy and then re-go get.

Obviously this is less bad than someone importing the wrong path
directly, but it's still bad.

--
Dominik Honnef

[Alberto García Hierro]

It is. GitHub is full of very useful packages which use a single directory. Forking the repository to make some changes and then submitting a PR becomes way more painful, since I need to either checkout your package at its "original" place (e.g. clone github.com/fiam/pdf to $GOPATH/src/rsc.io/pdf) or remove the commit which disables the forced import path.

Regards,

Alberto

[Russ Cox]

I sent out the proposal to golang-dev with subject "proposal: custom import path checking".