proposal: internal packages

[Russ Cox]

golang.org/s/go14internal

comments welcome on this mail thread. thanks.

[John C.]

Perhaps I missed this in the doc.  If there are multiple paths in $GOPATH, can an internal package be imported across different $GOPATH entries?

i.e. if GOPATH has my/dir/cat and also my/dir/dog then can a package at my/dir/cat/abc/xyz import a package at my/dir/dog/abc/internal/?

[meta keule]

I am against the introduction of keywords in import paths.

Better take one character path elements, like /#/ or /!/ or /$/
they are less likely to break existing packages out in the wild
when this rule is imposed on $GOPATH

metakeule

[Russ Cox]

On Wed, Jun 18, 2014 at 2:46 PM, meta keule <marcre...@googlemail.com> wrote:

I am against the introduction of keywords in import paths.

Better take one character path elements, like /#/ or /!/ or /$/
they are less likely to break existing packages out in the wild
when this rule is imposed on $GOPATH

Interesting. We hadn't discussed those. Note that any choice must be possible to use as an actual file system directory name, so these are probably unfortunate. We did discuss Unicode pile of poo but decided against it.

Russ

[Dave Cheney]

From point 3:

"3. When the Go compiler is written in Go, we will need a way to write
library packages for it that live in the main repo but are not part of
the standard library."

This seems to be the main motivation for this proposal, which I agree
is valid. With reference to your C -> Go compilers talk you mentioned
that the go/ast packages were probably not going to be suitable to the
task of being directly useful to the compiler. This implies that
because we can't change them for Go 1.x, there will have to be a
second set of ast and parser packages inside the std library, and I
can understand the motivation for wanting to:

a. be able to iterate on those packages without the Go 1 guarantee
b. hide them to avoid confusion.

--

Counter proposal:

Establish an area of the distribution which is not part of the Go 1
contract. Sun did this with their "sun." namespace inside the JDK. I
think this solves what I think is the main driver for this proposal
without introducing this special case for import paths.

Dave

On Thu, Jun 19, 2014 at 5:42 AM, Russ Cox <r...@golang.org> wrote:
> golang.org/s/go14internal
> comments welcome on this mail thread. thanks.
>

> --
> You received this message because you are subscribed to the Google Groups
> "golang-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-dev+...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Caleb Spare]

What about backwards compatibility generally? Godoc.org shows a
variety of packages with 'internal' in their import paths[0]; it's
possible that some of these are imported by other packages (although I
didn't check; perhaps we can scrape the data and do that). These will
no longer build if this change is introduced.

[0] http://godoc.org/?q=internal

[Gustavo Niemeyer]

Very few of these use "internal" in the package path (it's in the
summary, instead), and of those that use it, all of them seem to imply
the desired semantics.

Feels like an argument towards using "internal", rather than against it.

--

gustavo @ http://niemeyer.net

[giacomo...@gmail.com]

I like the proposal, I found it elegant in a "it does what it says" way.

For the backward compatibility issue for $GOPATH I think it probably will not break any code for the same reason. If the import path is named internal it is probably meant for internal use.

In addition it could be done for $GOROOT and $GOPATH in a different way even if it is gross.

Giacomo

[Robert Griesemer]

While point 3 is a driver for the proposal, it does solve a more general problem not limited to the compiler (or just commands), but large libraries ("components") as well: It is can be beneficial to split up large packages into sub-packages. Often, we don't want to expose those sub-packages' APIs to other clients.

For instance, go/types in the go/tools repo makes use of the go/exact package. It's not a command, it's a very large library. I'd prefer not having other clients rely on go/exact. Right now there is no stability guarantee for go/tools APIs, but that could change over time. And even w/o such a guarantee, any API change is sure to cause pain in clients and complaints. It would be extremely useful to delineate what's publicly usable and what is off limits.

The same is true for godoc: There's no need for everything in it to be part of package main. At the same time, we don't want to expose factored out packages to the general public because they tend to be too specific, and because we don't want to restrict or complicate changes across package boundaries within godoc.

 "internal" is akin to a large-grain access control mechanism, albeit w/o the heavy machinery that sometimes is associated with such a mechanism.

- gri

[Caleb Spare]

On Wed, Jun 18, 2014 at 3:02 PM, Gustavo Niemeyer <gus...@niemeyer.net> wrote:
> Very few of these use "internal" in the package path (it's in the
> summary, instead), and of those that use it, all of them seem to imply
> the desired semantics.
>
> Feels like an argument towards using "internal", rather than against it.

Yes, that search includes those packages that use 'internal' in the
summary as well as the path. And yes, most of those with 'internal' in
the path seem to indicate the desired semantics.

My point is that there can exist packages in the world (and maybe not
even indexed by godoc.org) that will be broken by this change. So it
seems to me that this does break the Go 1 compatibility promise as far
as any compiler changes are concerned, albeit in a small way.

(I'm not even opposed to this change per se; just curious how it can
be reconciled with http://golang.org/doc/go1compat.)

[minux]

On Wed, Jun 18, 2014 at 6:31 PM, Caleb Spare <ces...@gmail.com> wrote:

(I'm not even opposed to this change per se; just curious how it can

be reconciled with http://golang.org/doc/go1compat.)

This is not a compiler change: it aim to change the go tool, and build tools are exempt from

the Go 1 contract.

http://golang.org/doc/go1compat#tools

[Caleb Spare]

Ah yes, indeed: "The rule will be enforced by the go command." I had
missed that. Thanks.

So I guess that internal packages will not be part of Go-the-language
at all (e.g., they cannot be mentioned in the spec).

[minux]

On Wed, Jun 18, 2014 at 6:43 PM, Caleb Spare <ces...@gmail.com> wrote:

Ah yes, indeed: "The rule will be enforced by the go command." I had
missed that. Thanks.

So I guess that internal packages will not be part of Go-the-language
at all (e.g., they cannot be mentioned in the spec).

no. it won't. similarly GOPATH isn't part of the language.

They're both part of the whole Go ecosystem, for sure.

[Dave Cheney]

Thanks for your reply Robert,

> For instance, go/types in the go/tools repo makes use of the go/exact
> package. It's not a command, it's a very large library. I'd prefer not
> having other clients rely on go/exact. Right now there is no stability
> guarantee for go/tools APIs, but that could change over time. And even w/o
> such a guarantee, any API change is sure to cause pain in clients and
> complaints. It would be extremely useful to delineate what's publicly usable
> and what is off limits.

We have examples in the std lib, regexp/syntax, crypto/subtle, and
sync/atomic come to mind, of packages that we don't intend for people
to import, and the documentation on that package says as much.

This feels to me like an argument for packages that are not covered by
a stability guarantee, not an argument for packages that are hidden or
have import restrictions.

> The same is true for godoc: There's no need for everything in it to be part
> of package main. At the same time, we don't want to expose factored out
> packages to the general public because they tend to be too specific, and
> because we don't want to restrict or complicate changes across package
> boundaries within godoc.
>
> "internal" is akin to a large-grain access control mechanism, albeit w/o
> the heavy machinery that sometimes is associated with such a mechanism.

I think my counter proposal of an prefix in the std lib that is not
covered by the go 1 guarantee achieves this equally well without
introducing another rule for import paths (we already have odd ones
like testdata and anything that starts with _).

The upside is that if we don't like how it turns out, the entire
prefix can be deleted and we can try this internal scheme.

Dave

[Russ Cox]

On Wed, Jun 18, 2014 at 2:52 PM, Dave Cheney <da...@cheney.net> wrote:

Establish an area of the distribution which is not part of the Go 1
contract.

That's what we're doing. It's any area named "internal".

I think you need to motivate the counter-proposal with a reason why this proposal is not good.

Russ

[Robert Griesemer]

On Wed, Jun 18, 2014 at 4:42 PM, Dave Cheney <da...@cheney.net> wrote:

Thanks for your reply Robert,

> For instance, go/types in the go/tools repo makes use of the go/exact
> package. It's not a command, it's a very large library. I'd prefer not
> having other clients rely on go/exact. Right now there is no stability
> guarantee for go/tools APIs, but that could change over time. And even w/o
> such a guarantee, any API change is sure to cause pain in clients and
> complaints. It would be extremely useful to delineate what's publicly usable
> and what is off limits.

We have examples in the std lib, regexp/syntax, crypto/subtle, and
sync/atomic come to mind, of packages that we don't intend for people
to import, and the documentation on that package says as much.

This feels to me like an argument for packages that are not covered by
a stability guarantee, not an argument for packages that are hidden or
have import restrictions.

Even if packages don't have a stability guarantee sometimes making such a package's interface suitable for public consumption (rather than one specific client) can be counter-productive. Compiler-internal packages are likely to export innards that cannot be reasonably used outside the compiler. The "internal" mechanism will prevent such use. The absence of a stability guarantee will not: In my experience, if there's something - however peculiar - one can depend on, even if strongly cautioned, somebody will!

> The same is true for godoc: There's no need for everything in it to be part
> of package main. At the same time, we don't want to expose factored out
> packages to the general public because they tend to be too specific, and
> because we don't want to restrict or complicate changes across package
> boundaries within godoc.
>
>  "internal" is akin to a large-grain access control mechanism, albeit w/o
> the heavy machinery that sometimes is associated with such a mechanism.

I think my counter proposal of an prefix in the std lib that is not
covered by the go 1 guarantee achieves this equally well without
introducing another rule for import paths (we already have odd ones
like testdata and anything that starts with _).

The "internal" proposal is very much the same, except that it's not per repo (or even just the std repo), but on the package/command level. I would definitively want such a mechanism not just in the std lib. Also, external developers are likely to want a similar scheme, we certainly don't want them to have to invent their special area for their respective repos. "internal" is general and works not just for the std lib.

[Dave Cheney]

On Thu, Jun 19, 2014 at 9:57 AM, Russ Cox <r...@golang.org> wrote:
> On Wed, Jun 18, 2014 at 2:52 PM, Dave Cheney <da...@cheney.net> wrote:
>>
>> Establish an area of the distribution which is not part of the Go 1
>> contract.
>
>
> That's what we're doing. It's any area named "internal".

I agree the results would be similar, please see the next paragraph
for my objections

> I think you need to motivate the counter-proposal with a reason why this
> proposal is not good.

I would prefer to see a single top level package path in the std lib
that is excluded from the go 1 contract because:

1. We don't need to change the tooling apart from the api tool.
Whatever prefix for the part of the standard library which is not
covered by the go1 guarantee, for arguments sake I'm going to call it
$GOROOT/src/private, needs no special cases in the build rules, it can
work today.

2. This proposal introduces the concept of a package hierarchy,
a/b/internal "belongs" to a/b, but not to a/c. For 4 years the
consistent message in all the documents, books, blog posts, irc, and
mailing list is that Go packages are not hierarchical. I don't want to
loose this simple consistent message when other options exist.

3. John C. pointed out earlier, how does the "internal" namespace
interact when a/b/internal and a/b come from different GOPATH
elements. I'm sure there can be a set of rules defined, but then we're
back to point 2, and having to explain and debate them on the mailing
list, etc.

Thanks

Dave

[Dave Cheney]

> Even if packages don't have a stability guarantee sometimes making such a
> package's interface suitable for public consumption (rather than one
> specific client) can be counter-productive. Compiler-internal packages are
> likely to export innards that cannot be reasonably used outside the
> compiler. The "internal" mechanism will prevent such use. The absence of a
> stability guarantee will not: In my experience, if there's something -
> however peculiar - one can depend on, even if strongly cautioned, somebody
> will!

I'm sure they will, and they will be wrong and the universe will
punish them. When a power supply has a big sticker that says "Warning
high voltages, no user serviceable parts inside", that means "don't
open it", not "we should reduce the standard domestic operating
voltage because people have screwdrivers and can open power supplies".

I feel strongly that if the package is documented as internal, and
clearly states it is not stable and should not be relied upon, then
that is all that needs to be said.

[Russ Cox]

On Wed, Jun 18, 2014 at 1:15 PM, John C. <jscroc...@gmail.com> wrote:

Perhaps I missed this in the doc.  If there are multiple paths in $GOPATH, can an internal package be imported across different $GOPATH entries?

No. The rule is worded carefully. Internal things are visible to packages in nearby directories, not packages in nearby import paths.

Russ

[Robert Griesemer]

1) The "internal" mechanism can work today as well. It could be looked at from a pure documentation point of view w/o enforcement. Perhaps that's a "soft" way to start. Enforcement could come later.

2) What's introduced is a concept that internal packages must not be used by packages above internal. Internally there may be many of them, in multiple subdirectories, and w/o hierarchy implied.

[Russ Cox]

On Wed, Jun 18, 2014 at 5:10 PM, Dave Cheney <da...@cheney.net> wrote:

1. We don't need to change the tooling apart from the api tool.

Whatever prefix for the part of the standard library which is not
covered by the go1 guarantee, for arguments sake I'm going to call it
$GOROOT/src/private, needs no special cases in the build rules, it can
work today.

This excludes people who want to use this outside the main repo. For example, a project like Camlistore might choose to place all vendored code under 'internal'. And it might have its own new internal packages too. I know Brad is particularly excited to use this.

2. This proposal introduces the concept of a package hierarchy,
a/b/internal "belongs" to a/b, but not to a/c. For 4 years the
consistent message in all the documents, books, blog posts, irc, and
mailing list is that Go packages are not hierarchical. I don't want to
loose this simple consistent message when other options exist.

I disagree that it introduces the concept of a package hierarchy. On the topic of package hierarchy I have said things like 'net/http is not a subpackage of net (whatever that would mean). The directory tree is purely for organization. It has no semantic meaning to the Go language.' This remains true.

This proposal defines an interpretation of the import paths, and that interpretation makes use of the existing directory hierarchy. It gives a semantic meaning for the go command, but the go command already imposes semantic meaning on import paths based on interpretation of the directory hierarchy. For example, 'go get' interprets github.com/foo/bar/baz/quux as meaning baz/quux in the bar repo by the foo user on github.com. Everything under github.com/foo/bar is required to fit that interpretation, not just baz/quux. The import aliases (like how rsc.io/pdf gets resolved) similarly depend on assigning semantics to directory paths.

3. John C. pointed out earlier, how does the "internal" namespace
interact when a/b/internal and a/b come from different GOPATH
elements. I'm sure there can be a set of rules defined, but then we're
back to point 2, and having to explain and debate them on the mailing
list, etc.

I answered this just now. I'm not sure how answering it brings us back to point 2.

Russ

[Dave Cheney]

Russ, Robert,

I was incorrect about my assertion about this proposal creating a
package hierarchy. Thank you for clarifying.

Dave

[Kevin Gillette]

I'd like to recommend, that instead of "internal", import paths whose final component begins with an underscore are internal. Thus `import "abc"` would have the traditional behavior, but `import "abc/_def"` would only be valid inside package abc or any of its descendants.

This allows namespaces to be a little flatter than they have to be with the original proposal. To make a internal package available to the whole stdlib, the first path component just needs an underscore, like `import "_itoa"`

[Kevin Gillette]

This is also likely non-breaking, as the godoc.org index has no search results for "/_"

[minux]

On Wed, Jun 18, 2014 at 8:59 PM, Kevin Gillette <extempor...@gmail.com> wrote:

This is also likely non-breaking, as the godoc.org index has no search results for "/_"

directories with underscore as prefix are ignored by the go tool, so this explains your finding.

Although this is a backward compatible change in that no existing packages will be treated

as an internal package, this still has the side effect that existing directories with the underscore

prefix will be treated as a Go package, which in certain cases, is also an incompatible change.

(for example, imagine, some people use a directory like _utils to store Go code not intended

to be built by the go tool. After this change, not only _utils will be treated as internal package,

it will also be built by go build ./...)

Besides, _itoa doesn't look better or more informative than "internal/itoa".

[Stephen Gutekanst]

I am for this proposal.

I would be excited to use this because it would allow me to insure various third-party packages in my library are not used by others when they should be using the, perhaps better, updated versions hosted elsewhere).

Stephen

[Gary Burd]

On Wednesday, June 18, 2014 5:59:17 PM UTC-7, Kevin Gillette wrote:

This is also likely non-breaking, as the godoc.org index has no search results for "/_"

There are no search results for "/_" because godoc.org ignores all directories with a leading "_".

 

[yiyus]

I would like to point out that this could also help to solve issue 7453, related to the discussion "locking down vanity import paths" in this list.

To lock a package import path, you just need to import some internal package (you can always add an empty one, for example, "rsc.io/pdf/internal/vendor"), and then the package won't be built if it is got from the wrong location.

[Kevin Gillette]

I was referring to a browser page-search of <http://godoc.org/-/index>. If your point still applies, then it serves as further justification, since it's much less likely that such packages will be found or used.

[Rui Ueyama]

I agree that `import "_iota"` doesn't look better. Since the leading underscore is a part of the package name, I'd have to write "_iota" instead of "iota" at all places I would've wrote "iota" if I make it an internal package. It doesn't look nice to me, and I don't think the leading underscore has such importance to be repeated every time I refer a package.

[Russ Cox]

It is more important to pick a good name than to pick a name that doesn't conflict with any possible current usage. Punctuation symbols are unclear. Names beginning with underscore are ugly. Underscore by itself is both. 

"internal" is clear and rarely used today, at least in the code I can see inside Google and on godoc.org. On godoc.org, I counted about 10 import paths containing 'internal', and the import for all but one of them were already compatible with the new rule. The only one I see is that github.com/gamingrobot/zephyr/events imports github.com/gamingrobot/steamgo/internal.

Strictly speaking, it is okay to break existing code for this. The go command and the meaning of import paths are not part of the Go 1 compatibility rules. Of course, we don't want to cause undue pain regardless, but it looks like it would affect very few packages. It would land in 1.4 at the very earliest, so people have at least five months to update the few uses that would break. And 1.5 seems more likely, which is eleven months away.

Getting a good name is worth this very minor inconvenience.

Russ

[Daniel Theophanes]

This feels very similar to the current convention of "third_party" that is ignored by tools such as godoc.org.

Would "third_party" also have the same root path build rule?

Would the "internal" paths not be listed in the package doc tools (like "third_party" currently in godoc.org) but still be available for for viewing if the path was entered in directly?

[gordon...@gmail.com]

On Thursday, June 19, 2014 2:29:00 AM UTC+2, gri wrote:

1) The "internal" mechanism can work today as well. It could be looked at from a pure documentation point of view w/o enforcement. Perhaps that's a "soft" way to start. Enforcement could come later.

This seems like a good incremental approach.  It solves a very specific problem: excluding code from compatibility guarantees (Go 1 or otherwise).  It is simpler, involves less work, and won't break anyone.  With time it should become clear whether or not visible internals are troublesome and hiding should be enforced.

[Taru Karttunen]

On 19.06 09:42, Dave Cheney wrote:
> We have examples in the std lib, regexp/syntax, crypto/subtle, and
> sync/atomic come to mind, of packages that we don't intend for people
> to import, and the documentation on that package says as much.
>
> This feels to me like an argument for packages that are not covered by
> a stability guarantee, not an argument for packages that are hidden or
> have import restrictions.
>

I think that documenting (and having godoc(.org) honor it) internal
packages would make much more sense.

E.g. if crypto/subtle was made an internal package lots of people
would either 1) fork it, 2) reinvent it poorly.

godoc.org tells us
"Package subtle is imported by 122 packages."

Would an annotation that tells the documentation tools (and the api
tool) that:
1) this package does not have a stable API and
2) is not tailored for external use
satisfy what is needed?

- Taru Karttunen

[Aram Hăvărneanu]

I don't understand why we need to write code, when we can just write
documentation. Yes, some people will ignore the warning, but there are
a million way users can break Go 1 contract today and we don't do
anything to prevent most of them.

The people who would ignore the warning are the same people who will
fork the package anyway (and then never update it again). The
end-result is worse, and there's extra work to do to achieve it.

The current design encourages people to think about API design at
*package* level. The proposed design will have the side effect of
making people think about API design at *program* or project level.
"Ah, I'll just hide everything and make one public package". I believe
this is a huge mistake. The current scope of a package as an API
boundary creates packages and APIs that are nice to use, and also
forces a certain discipline in program design that results in better
and easier to understand programs. At least that's what I feel when I
read and write Go code compared to other "modern" languages.

--
Aram Hăvărneanu

[giacomo...@gmail.com]

On Thursday, 19 June 2014 12:21:52 UTC+2, Aram Hăvărneanu wrote:

I don't understand why we need to write code, when we can just write
documentation. Yes, some people will ignore the warning, but there are
a million way users can break Go 1 contract today and we don't do
anything to prevent most of them.

The people who would ignore the warning are the same people who will
fork the package anyway (and then never update it again). The
end-result is worse, and there's extra work to do to achieve it.

 

I don't see  how it is worse, is just  a convention enforced by the standard tools. Make is always there if you don't like the conventions.

Also the import path should be meaningful, is part of the documentation as i see it. Having an "internal" import path tell quite clearly the intention for the package, even before reading the docs. 

The current design encourages people to think about API design at
*package* level. The proposed design will have the side effect of
making people think about API design at *program* or project level.
"Ah, I'll just hide everything and make one public package". I believe
this is a huge mistake. The current scope of a package as an API
boundary creates packages and APIs that are nice to use, and also
forces a certain discipline in program design that results in better
and easier to understand programs. At least that's what I feel when I
read and write Go code compared to other "modern" languages.

But you still have to import your internal package so you have to design its API. 

If you want a program interface you should just keep all in the main package; you can split it in many files, run tests etc.. 

If you want to refactor into package you have to write a package and figure out its API even if it's internal.

Just my 2 cents.

--
Aram Hăvărneanu

Giacomo 

[Ian Dawes]

I am very much for this suggestion. I've been looking for a clean way to split up a bunch of library code into exported API and internal API for a while now and this proposal would make that possible in what feels to me like a very clean way. Please consider extending it to GOPATH and not just GOROOT in 1.4.

ID

[Alberto García Hierro]

I totally agree with this, in fact we already took this approach months ago.

// Package internal contains low-level utilities

// used by several parts of Gondola.

//

// Users should not use this package nor any or

// this subpackages, since they're considered internal to

// Gondola, and only exported to avoid code repetition

// across several packages. As such, their interface

// might change at any time without prior notice.

package internal

And the problem is fixed!

Placing artificial constraints will just make most people work around them (and, as you have pointed out, it

will create even worse problems). No one will write their own package from scratch when there's already

a working implementation which can just be copied (or, even worse, symlinked!) elsewhere.

Regards,

Alberto

[Maxim Khitrov]

As the proposal stands, I agree with this as well. Documentation is
sufficient to label a package as "internal" and anyone who wants to
import it anyway will find a way.

The thing that would make sense to me (and I already know what the
response will be) is if this was a way of introducing true subpackages
that use paths relative to the parent package. If something is
internal and tightly bound to some parent, there is no reason for
giving it an absolute import path. I proposed this a long time ago as
the solution to the relative import problem, but there didn't seem to
be too much interest.

[Jan Mercl]

What about

SourceFile = [ "private" ] "package" ident ";" { ImportDecl  ";" } { TLD ";" } .

?

The private modifier disables import by packages not having an import path which is a prefix of the private package import path.

(phone)
-j

[Kevin Gillette]

On Thursday, June 19, 2014 4:56:33 AM UTC-6, giacomo...@gmail.com wrote:

I don't see  how it is worse, is just  a convention enforced by the standard tools.

Conventions are no longer conventions once they're enforced. 

[Kevin Gillette]

On Thursday, June 19, 2014 6:41:29 AM UTC-6, Jan Mercl wrote:

What about

SourceFile = [ "private" ] "package" ident ";" { ImportDecl  ";" } { TLD ";" } .

?

Adding a new keyword is a huge deal in Go. Further, if said keyword is "private", it'll give people all the wrong preconceptions about where it can be used (they'll think it can modify _anything_), and make visibility rules much harder to teach. If we were going to go vaguely in that direction, allowing the package decl to have the equivalent of a struct tag would be a better compromise, yet there's still no precedent for the tooling to treat tags specially.

[Jan Mercl]

Technically there's no need for it to be a keyword (meaning a reserved word).

(phone)
-j

[giacomo...@gmail.com]

OT: I respectfully disagree, to me a convention is just a less formal rule, but I am not a native english speaker so I will not push this.

Still using the go tool and setting up a $GOPATH is a (convenient) convention for me. 

[Ingo Oeser]

Actually what "internal" is intended to mean is not "please don't import this package" 

but "I will not promise any stability on semantics, content or even existence of this public symbol collection".

The same goal can be reached by inverting the problem and start using semantic versioning 

in one of the ways envisioned by the packaging discussions.

And then a simple marker like "0.0.0" can be used or defaulted on to provide "internal" packages.

This will solve both problems with a single change to the semantics of go import statement parsing.

Semantic versioning and it's promises are well understood and thus easy to communicate to peers.

Adding internal packages as proposed would mean to add another scoping layer instead with the added 

cognitive burden and continuous decision making that go with the "protected" keyword in classes.

[Ian Lance Taylor]

On Thu, Jun 19, 2014 at 3:21 AM, Aram Hăvărneanu <ara...@mgk.ro> wrote:
> I don't understand why we need to write code, when we can just write
> documentation. Yes, some people will ignore the warning, but there are
> a million way users can break Go 1 contract today and we don't do
> anything to prevent most of them.

The Go 1 contract is not something that most users of Go can break,
it's something that the Go developers can break. And the Go
developers try very hard not to break it. This mechanism, or
something like it, lets the Go developers have internal packages
without breaking the Go 1 contract.

Ian

[stuart...@gmail.com]

I like the word internal, as it has similar semantics to C#'s internal keyword. Specifically, it enables access to a given symbol only to other classes within the same assembly (aka .dll, package, module), but they are not visible to external users. .NET took it one step further and allows friend assemblies via the InternalsVisibleTo attribute, where one use case is creating a unit test assembly. I don't see this as feature at all necessary for Go.

The value to me is there are certainly scenarios where we might want to build internal support packages for a larger project. A nice side effect that internal modules still follow the same public / private semantics of regular packages in Go, however it's an explicit declaration by the engineers that it's not for public consumption. It is conceivable that at some later date, the package is moved out of the internal subtree and it's public / private API is already well defined.

Cheers,

Stu

[Robert Griesemer]

On Thu, Jun 19, 2014 at 3:21 AM, Aram Hăvărneanu <ara...@mgk.ro> wrote:

The current design encourages people to think about API design at
*package* level. The proposed design will have the side effect of
making people think about API design at *program* or project level.
"Ah, I'll just hide everything and make one public package". I believe
this is a huge mistake. The current scope of a package as an API
boundary creates packages and APIs that are nice to use, and also
forces a certain discipline in program design that results in better
and easier to understand programs. At least that's what I feel when I
read and write Go code compared to other "modern" languages.

I think this is an honorable sentiment, but the reality is that designing a good package API fit for public consumption is _very_ hard. There's a reason why the collective group of gophers has been slaving over the std lib for such a long time. In fact, most people (myself included) are not very good at designing great public APIs.

But more importantly, some libraries are (or going to be) very large, but really have a relatively small API that should be used publicly. One can either put everything into one huge package (not ideal), or split it up in sub-packages. Not being able to say those sub-packages are "internal" (via convention or enforcement) puts an undue burden on the implementor, but also the implementation: Some of those sub-packages will only work if used just the right way within the context of the bigger library. If one had to make them available for general consumption (per your suggestion), one may have to make them more general, possibly slower, perhaps with more abstract data types, and so on.

At a certain software size, packages are too small a building block for encapsulation. We need to be able to build larger things ("components") w/o exposing the internals, and at the same time we don't want to give up the encapsulation we get from packages. This is not a new problem. The Go team has discussed ideas how to create such components early on, but w/o experience with the language we didn't have much to work with. In the 1980's, Modula-2 had sub-modules (a Modula-2 module is almost identical to a Go package); but those had their own problems: they were too fine-grained as they had to be within the same source of a module (Modula-2 modules also served as monitors/protected regions, but that's orthogonal).

In Go, the encapsulation mechanisms are (from smallest to largest), blocks, functions, and then packages. Functions may be nested, so there's some scalability. One might argue that packages should be nestable too, but that poses language design challenges (see Modula-2). The "internal" mechanism is a language-external mechanism to achieve exactly that, without the need for language machinery: A way to encapsulate packages. It also scales because the internal mechanism can be nested.

That is, the "internal" mechanism is a scalable mechanism to abstract beyond a single package. It is extremely simple and provides significant "bang for the buck". And we can use it immediately, w/o enforcement, by simply stating it as a convention. I think enforcement will simply help prevent mistakes.

- gri

[Henrik Johansson]

Could it not be on a file basis? Some keyword after the package clause?

/Henrik

--

[Robert Griesemer]

A keyword (i.e., language change) is a much more significant change than a directory convention. Also, this is a package-level mechanism, so per-file basis would only complicate matters.

- gri

[Rodrigo Kochenburger]

I gotta say that as much as I love the "least keywords possible" mantra, a keyword modifier does feels like a better solution. It will avoid unexpected surprises to both existing developers and new developers learning go, and will be 100% backwards compatible.

Instead of using a private keyword, I'd go for a internal keyword:

internal package foo

[Dan Kortschak]

If this route were taken, wouldn't is be easier and less invasive to use a comment tagbakin to a build tag?

// +internal

package foo

This would keep the change as a tooling change rather than in the language. godoc could look for this and place a banner thing at the top of documentatio for such packages.

[Brendan Tracey]

Rodrigo -- your proposal is a language change. Russ has proposed a change to the compilation rules of the go toolchain (which has no language changes). As far as the language is concerned, packages are still just marked with a string.

I'd much rather see Russ's proposal than a build tag. Russ's proposal keeps the property that its obvious from the path where the package is (and in this case the behavior). With a build tag, one needs to read the source in order to see that the package should not be imported. Russ's proposal is clean and easy to understand and follow.

In addition, this concept is a function of the behavior between packages rather than the building of a particular package. The semantics of how one imports a package are in the import statement, and so it seems like such behavior should stay in the import statement.

--
You received this message because you are subscribed to a topic in the Google Groups "golang-dev" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/golang-dev/_cAggq73yME/unsubscribe.
To unsubscribe from this group and all its topics, send an email to golang-dev+...@googlegroups.com.

[Dan Kortschak]

To be clear, I did not suggest a build tag but rather a comment-based approach akin to the build tag system.

More broadly, I'm completely on the fence about the whole thing.

[John C.]

The proposal as-is addresses an issue I face and solves it well.  I think the word "internal" in the import path is an excellent flag for the reader.  I've read the pros and cons on the list and am still for the proposal as-is.  My opinion carries no special weight, but FWIW I am in favor of the proposal.

[Rodrigo Kochenburger]

Yeah, the more I think about it, the more I understand the proposal advantages. 

My only concern is somewhat related to the "principle of least surprise" since imports could simply break with no apparent reason, which could be a problem for existing apps and new Go developers learning the language. We can probably address that by documenting and presenting informative errors messages when an import fails.

- RK

[Henrik Johansson]

While I can definitely see the uses the ease of conventions can become obscuring at some point.

Hiding things can be very convenient and most if not all "normal" languages has some way to do it.

A language change is of course more invasive than a convention used by tools and it gives a way back if it leads wrong.

However a lesson from Java may be that today it seems many changes can not be made because it would break too many programs that rely on some or other coincidental behavior even if they were not supposed to do that. Often people rely on behavior rather than specifications in practice and I think it would be a mistake to think otherwise and ignore it.

Well generally I like the idea of being able to hide parts and maybe the "ignore" in the path is a safe trial. So +1 with some reservation.

/Henrik

You received this message because you are subscribed to the Google Groups "golang-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+...@googlegroups.com.

[Andrew Gerrand]

I'd like to do this but without the toolchain enforcement.

I think the proposed enforcement adds unnecessary friction for experimentation and hacking, but gives us little because people will fork repos anyway. (And we can enforce the rule where it matters, on App Engine for instance.)

I propose that we specify in our Go 1 compatibility guidelines that package with "internal" in the path should not be used and may change at any time.

Golint could be modified to report bad uses.

[Brad Fitzpatrick]

Why don't we also allow people to call lowercase functions cross-package while we're at it?

[Ian Lance Taylor]

On Thu, Jun 19, 2014 at 3:27 PM, Andrew Gerrand <a...@golang.org> wrote:
>

I don't understand the advantage of that. If people can import
internal packages (under whatever name is used for this), then they
will. And then we, as the standard library maintainers, will be put
into the position of breaking people's code. Yes, they did something
they weren't supposed to do, but it will still feel bad to break them.
If we're going to do this, let's make people have to work hard to
break themselves (for example, by forcing them to fork the go tool).
Then we won't put ourselves into the position of feeling bad in the
future.

Ian

[Andrew Gerrand]

On 20 June 2014 08:31, Brad Fitzpatrick <brad...@golang.org> wrote:

Why don't we also allow people to call lowercase functions cross-package while we're at it?

If you could still access unexported identifiers but they weren't present in godoc, I don't think we'd be in a very different situation today.

What does enforcement give us?

[Andrew Gerrand]

On 20 June 2014 08:34, Ian Lance Taylor <ia...@golang.org> wrote:

I don't understand the advantage of that.  If people can import
internal packages (under whatever name is used for this), then they
will.  And then we, as the standard library maintainers, will be put
into the position of breaking people's code.  Yes, they did something
they weren't supposed to do, but it will still feel bad to break them.
If we're going to do this, let's make people have to work hard to
break themselves (for example, by forcing them to fork the go tool).
Then we won't put ourselves into the position of feeling bad in the
future.

I wouldn't feel bad at all if someone relied on internal/foo and it changed out from under them. In fact I'd enjoy the schadenfreude.

[Andrew Gerrand]

On 20 June 2014 08:34, Andrew Gerrand <a...@golang.org> wrote:

If you could still access unexported identifiers but they weren't present in godoc, I don't think we'd be in a very different situation today.

Actually, having thought about this more, I take that back. But I still wonder what enforcing the internal rule really gives us.

[Brad Fitzpatrick]

*) scalability (Robert's earlier argument was well-written)

*) freedom from breakage guilt (Ian's latest argument), thus flexibility, just like renaming an internal method or type as a cleanup.

This will let us rearrange and cleanup big code base's packages and reduce the API surface as well.

[Dave Cheney]

I believe the driving force behind this proposal is to establish a
mechanism for the code for the C -> Go compiler transition to live in
the standard library without having to conform to the Go 1 guarantee.

The original proposal may be useful in a wider context, but I for one
want to see the Go compiler work separated from the more general
debate so I suggest a compromise proposal:

* Create a single 'internal' location at $GOROOT/src/internal [1]. If
I have understood the scoping rules correctly, the rest of the
packages in $GOROOT/src are siblings of $GOROOT/src/internal, or
children of siblings, like $GOROOT/src/cmd.

* The API tool is modified to ignore symbols inside the
$GOROOT/src/internal tree.

* The 1.4 release notes, and the package doc in
$GOROOT/src/internal/doc.go clearly state that packages in this path
are not stable and not part of the Go 1 guarantee.

I believe this proposal solves the immediate problem without closing
the door for adding full internal package support [2] later.

optional: the go tool could be modified to prevent import
"internal/$SOMETHING" across GOPATH/GOROOT boundaries as suggested in
the original proposal.

bonus: I think this counter proposal could provide a solution to the
question of adding an unstable fsnotify package to the stdlib. If the
package was $GOROOT/src/internal/fsnotify then cmd/go could use it for
the watching behaviour that rsc described (in a mail that I can't find
at the moment).

Dave

1. This assumes rsc's renaming src/pkg -> src proposal is accepted,
and there is no reason to believe it will not.

2. golang.org/s/go14internal

[Brad Fitzpatrick]

If people CAN import */internal/*, they will.

And then we'll feel guilty breaking them, even if they were at fault.

Ian and Robert and Russ and others said all this already.

This thread is looping.

[Dave Cheney]

On Fri, Jun 20, 2014 at 10:02 AM, Brad Fitzpatrick <brad...@golang.org> wrote:
> If people CAN import */internal/*, they will.
>
> And then we'll feel guilty breaking them, even if they were at fault.

I don't see that as a problem. Here are some examples that I'm aware
of where items which were not covered by the Go 1 spec were changed,
possibly breaking, or at least inconveniencing people

* In 1.1 the fallback to GOROOT if GOPATH was not set was removed.
This inconvenienced a number of people who were forced to make their
development workflow more 'standard', but on the whole it was a
positive change and removed an enormous barrier for new gophers.

* In 1.3 the flag parsing for the toolchain was changed, so anyone who
wasn't using the go tool would have had to adjust. We said it wasn't
covered under the Go 1 contract and that was the end of the matter.

* In 1.3 the implementation of the maps was changed from one
unspecified behaviour to another, breaking many people's assumptions
about how iteration over small maps operates (there was a similar
change in 1.2, but it only affected large maps and people didn't have
test fixtures that large so they didn't notice). Although it's early
days, people who's test code has broken because of the 1.3 map
iteration change have been overwhelmingly positive for pointing out a
mistake in their code.

My proposal continues along these lines, we say "internal" is private
and not stable, and that is the end of it. If someone imports it, and
then finds that it breaks their code on the next release, I have
demonstrated a clear precedent which has held since 1.1, that say's
it's their fault, not ours.

Dave

[Brad Fitzpatrick]

We have a choice here.

In those cases, we didn't.

I (and I think others) vote in favor of the conservative choice, not exposing things previously unexposed.

[Maxim Khitrov]

On Thu, Jun 19, 2014 at 8:02 PM, Brad Fitzpatrick <brad...@golang.org> wrote:
> If people CAN import */internal/*, they will.

There are legitimate reasons for doing so ("experimentation and
hacking" being one of them). You're looking at it from the perspective
of some long-lived external package importing something that it
shouldn't be and then breaking later on. I may want to write a
one-time-use program that needs access to some internal functionality
at this very moment and never again.

Suppose you have an application like mercurial or git written in Go.
It is bound to have some internal packages for managing the
repository, calculating deltas, handling network transfers, etc. There
have been cases in the past where I needed to access mercurial
internals and was quite glad that Python didn't get in my way of doing
so. Did I expect my code to be usable one year later? No, of course
not, but it's exactly what I needed at the moment. I don't see where
the guilt about changing internal packages comes from.

[Mikio Hara]

On Thu, Jun 19, 2014 at 4:42 AM, Russ Cox <r...@golang.org> wrote:

> golang.org/s/go14internal
> comments welcome on this mail thread. thanks.

This reminds me the day when I heard what's NAT and how it works. I'm
happy you guys pick the word "internal" instead of "private". Even if
there's some possibility that we may be need to traverse the
complicated internal/public namespace border, like a hairpinning NAT
router does; providing an internal->public->internal access, I'm fine
with this proposal as long as it is a simple package namespace
allocation or reservation stuff.

[Brad Fitzpatrick]

On Thu, Jun 19, 2014 at 5:39 PM, Maxim Khitrov <m...@mxcrypt.com> wrote:

On Thu, Jun 19, 2014 at 8:02 PM, Brad Fitzpatrick <brad...@golang.org> wrote:
> If people CAN import */internal/*, they will.

There are legitimate reasons for doing so ("experimentation and
hacking" being one of them). You're looking at it from the perspective
of some long-lived external package importing something that it
shouldn't be and then breaking later on. I may want to write a
one-time-use program that needs access to some internal functionality
at this very moment and never again.

But if I make it lowercase and unexported, you can't get at it anyway.

You weren't complaining about that.

 

[Robert Griesemer]

I am trying to understand your rationale. I think your proposal can be separated into two orthogonal pieces if I understand it correctly:

1) We permit only one "internal", at the top ($GOROOT/src/pkg/internal or $GOROOT/src/internal after removing pkg).

2) We don't enforce this convention (access restriction) with a tool.

I don't understand what you gain from #1 (which is an irregular approach) compared to the _regular_ and _scalable_ scheme proposed by rsc. The only difference (disregarding #2) is the restriction of the location. Unless you are convincingly clear beyond doubt why being irregular and non-scalable is better than the proposal, I think your suggestion is inferior.

2) We could start w/o enforcing the option - we have a choice. But since we have a choice it's better to be safe than sorry, for all the points brought up any times before in this thread by bradfitz, iant, rsc, and others.

As bradfitz has pointed out, we are going in circles. I have yet to see a strong counter-argument against the proposal.

- gri

[Robert Griesemer]

There are legitimate reasons for experimentation and hacking.

And with rsc's proposal you can still do that: You can always check out the source, and add an extra package parallel to internal and expose internal details. It's a bit more cumbersome, but that's ok, since this is likely a rare situation.

Also, the restriction is only enforced by the go tool. The compilers don't care. You can always hack.

- gri

On Thu, Jun 19, 2014 at 5:39 PM, Maxim Khitrov <m...@mxcrypt.com> wrote:

[Maxim Khitrov]

Well, now that you mention it... :)

My point was that a large library or application will be split into
internal packages along some logical lines. You may want to access the
higher-level functionality that is exported by the internal packages
in the same way that those packages are used by their non-internal
relatives. If I want to do this, I'm going to do it anyway by renaming
"internal" to something else and updating all affected import paths.
What for?

[Dave Cheney]

On Fri, Jun 20, 2014 at 10:55 AM, Robert Griesemer <g...@golang.org> wrote:
> I am trying to understand your rationale. I think your proposal can be
> separated into two orthogonal pieces if I understand it correctly:

My proposal seeks to address what I believe is the current requirement
-- a way to bring a package, or set of packages into the standard
library that support the C -> Go transition, that are not under
subject to the Go 1 guarantee.

Do you agree that this is the current requirement ?

> 1) We permit only one "internal", at the top ($GOROOT/src/pkg/internal or
> $GOROOT/src/internal after removing pkg).

I don't know about permit, it's just a directory like any other.

> 2) We don't enforce this convention (access restriction) with a tool.

Correctly, it's just a directory like any other.

> I don't understand what you gain from #1 (which is an irregular approach)
> compared to the _regular_ and _scalable_ scheme proposed by rsc. The only
> difference (disregarding #2) is the restriction of the location. Unless you
> are convincingly clear beyond doubt why being irregular and non-scalable is
> better than the proposal, I think your suggestion is inferior.

#1 is a compromise that I think incorporates rsc's ideas from the
original proposal _without_ having to enforce them. It also allows
rsc's proposal to be implemented later as it is compatible with the
scoping rules described in the document.

> 2) We could start w/o enforcing the option - we have a choice. But since we
> have a choice it's better to be safe than sorry, for all the points brought
> up any times before in this thread by bradfitz, iant, rsc, and others.
>
> As bradfitz has pointed out, we are going in circles. I have yet to see a
> strong counter-argument against the proposal.

It is the notion of enforcement in the tool, rather than by convention
or documentation, which I am against. This is my counter argument. I
don't want there to be a special case that the Go tool applies to
reject certain import paths. I don't want to have to blog about this,
to explain it over and over again on mailing lists, and irc. And I
don't see why rsc's original proposal needs to be adopted, with the
enforcing behaviour, to solve the current requirements of the C to Go
transition.

Dave

[andrewc...@gmail.com]

Special casing the most common build tool is almost the same as a change to the language.
Strong lint/doc warnings in my opinion would be better than blocking things entirely.

Perhaps just make the documentation of the internal packages *BIG RED UNDERLINED AND SCARY*. Then you shouldn't need to worry if you break some peoples code, because if they haven't understood the warnings, they are never going to listen anyway.

[Brad Fitzpatrick]

Rob doesn't like colored or stylized text.

--

[Erik St. Martin]

I feel like we have means for expressing visibility already, and import paths have always just been paths. It requires additional knowledge that needs to be documented to users about how the import paths and packages work. "imports are just paths, except in the case that they contain the word internal somewhere"

I can understand the need to have a part of the source tree that doesn't need to adhere to the go1 guarantee, but i'd also agree that i'm not sure that it's the go tool's job to enforce this. It should just be documented.

Is it any different than telling people it's not recommended to run your production builds against tip because it hasn't been fully tested? or a public git repo not recommending a package for production use.

If the issue this proposal stands to resolve is to provide a set of packages that do not have to adhere to the go1 guarantee, then I don't think the go tool should handle the path any differently. If this proposal stands to create packages with a somewhat "protected" visibility, that may be a reason to modify the go tool.

[andrewc...@gmail.com]

Or another possibility is that the tools emit an error which states something along the lines of:

"Using an internal package is highly likely to generate faulty software unless you have explicit understanding of the internals of the package. Internal packages of the standard library are not covered by the compatibility guarantees. If you are sure you want to do XXX please specify the -dangerous-imports flag."

So its easy to bypass if for example, it breaks your existing code.

[Carlos C]

I have read the thread, however I am not member of core Go team. If I am inserting myself in a debate I don't belong to, please just delete this message.

Two questions immediately arise for me for the proposal:

1. What does it mean when a package has a subpackage with internal. Like a/b/internal/c/d/internal/e? (which in matter of aesthetics, looks ugly for me, but I believe beauty is not in debate here)

2. how does this affect the ability of go.test of actually testing internals?

Particularly for 2., for me is rather important that the tests of the vendored packages are still executed, moreover my ability to validate vendored packages through testing should remain intact, IMHO.

Regards,
cc

[Kevin Gillette]

Yeah, it does sound like a good candidate for a +build flag. The Go tool would just automatically set that flag for anything importing their _own_ internal packages.

[Robert Griesemer]

On Thu, Jun 19, 2014 at 6:09 PM, Dave Cheney <da...@cheney.net> wrote:

On Fri, Jun 20, 2014 at 10:55 AM, Robert Griesemer <g...@golang.org> wrote:
> I am trying to understand your rationale. I think your proposal can be
> separated into two orthogonal pieces if I understand it correctly:

My proposal seeks to address what I believe is the current requirement
-- a way to bring a package, or set of packages into the standard
library that support the C -> Go transition, that are not under
subject to the Go 1 guarantee.

Do you agree that this is the current requirement ?

I agree that this may be the driver for the proposal, and that the proposal satisfies the requirement.

I disagree that we should introduce a specific solution for just this problem if we can do better, because the problem already exists elsewhere (godoc comes to mind, and some of the go/types packages), even if we left gc alone.

> 1) We permit only one "internal", at the top ($GOROOT/src/pkg/internal or
> $GOROOT/src/internal after removing pkg).

I don't know about permit, it's just a directory like any other.

sure - what I meant is that we would document "internal" accordingly (no Go 1 guarantees)

 

> 2) We don't enforce this convention (access restriction) with a tool.

Correctly, it's just a directory like any other.

> I don't understand what you gain from #1 (which is an irregular approach)
> compared to the _regular_ and _scalable_ scheme proposed by rsc. The only
> difference (disregarding #2) is the restriction of the location. Unless you
> are convincingly clear beyond doubt why being irregular and non-scalable is
> better than the proposal, I think your suggestion is inferior.

#1 is a compromise that I think incorporates rsc's ideas from the
original proposal _without_ having to enforce them. It also allows
rsc's proposal to be implemented later as it is compatible with the
scoping rules described in the document.

Can we agree that #1 (your proposal w/o enforcement) and rsc's proposal w/o enforcement is _one_ of the design axis? Enforcement is an other (orthogonal) one. Please let's stop mixing the two.

I am claiming that rsc's proposal (even w/o enforcement) is better than a special GOROOT-only approach because it scales. #1 solves an immediate need for gc, and perhaps for some other std lib packages. rsc's proposal (even w/o enforcement) solves a general need (godoc comes to mind, and some packages under go.tools, and I'm certain there's a multitude of other places in the wild where the internal convention would be useful.

For instance, it permits tool developers to use the convention now and split up their programs into internal packages. Of course they can just _document_ their packages as internal, but why should everybody come up with their own convention if there's a good convention that can work for everybody? This has been the spirit of Go from day one: The conventions enforced by the go tool, the formatting style enforced by gofmt, even some of the restrictions in the language all have been a boon for developers at large. I think the same will be the true here.

> 2) We could start w/o enforcing the option - we have a choice. But since we
> have a choice it's better to be safe than sorry, for all the points brought
> up any times before in this thread by bradfitz, iant, rsc, and others.
>
> As bradfitz has pointed out, we are going in circles. I have yet to see a
> strong counter-argument against the proposal.

It is the notion of enforcement in the tool, rather than by convention
or documentation, which I am against. This is my counter argument. I
don't want there to be a special case that the Go tool applies to
reject certain import paths. I don't want to have to blog about this,
to explain it over and over again on mailing lists, and irc. And I
don't see why rsc's original proposal needs to be adopted, with the
enforcing behaviour, to solve the current requirements of the C to Go
transition.

Ok, let me restate that: You are against the enforcement. You may be ok with rsc's proposal if it's not enforced. Is that correct?

(I think enforcement is debatable; though personally I think it would want it enforced. But that could be a separate discussion.)

Dave

[Robert Griesemer]

On Thu, Jun 19, 2014 at 6:43 PM, Erik St. Martin <alak...@gmail.com> wrote:

I feel like we have means for expressing visibility already, and import paths have always just been paths. It requires additional knowledge that needs to be documented to users about how the import paths and packages work. "imports are just paths, except in the case that they contain the word internal somewhere"

The spec deliberately attaches no semantics to import paths. The go tool already interprets them in very specific ways (for instance relative to $GOROOT, $GOPATH). The whole point of not attaching language-specific meaning to them is so that we can make them useful in unanticipated ways. This is one such way.

I can understand the need to have a part of the source tree that doesn't need to adhere to the go1 guarantee, but i'd also agree that i'm not sure that it's the go tool's job to enforce this. It should just be documented.

This is not only useful for the std lib. This is a general issue that affects developers at large, in a multitude of code bases. The Go development community would benefit tremendously from a simple, general, and scalable mechanism that is uniform. That way a developer doesn't have to learn different conventions used in different repos, by different people, in different companies. 

Is it any different than telling people it's not recommended to run your production builds against tip because it hasn't been fully tested? or a public git repo not recommending a package for production use.

See above. 

If the issue this proposal stands to resolve is to provide a set of packages that do not have to adhere to the go1 guarantee, then I don't think the go tool should handle the path any differently. If this proposal stands to create packages with a somewhat "protected" visibility, that may be a reason to modify the go tool.

The former is an immediate goal. The latter is a long-term benefit which we should aim for and which rsc's proposal satisfies.

 

On Thursday, June 19, 2014 9:33:13 PM UTC-4, andrewc...@gmail.com wrote:

Special casing the most common build tool is almost the same as a change to the language.
Strong lint/doc warnings in my opinion would be better than blocking things entirely.

Perhaps just make the documentation of the internal packages *BIG RED UNDERLINED AND SCARY*. Then you shouldn't need to worry if you break some peoples code, because if they haven't understood the warnings, they are never going to listen anyway.

--

[andrewc...@gmail.com]

If people agree to the proposal but are unsure about tooling enforcement:

No enforcement other than documentation.

Pros:
Won't break anything immediately.
Cons:
May break things in the future if people don't follow the convention.

Strict tool enforcement:

Pros:
No risk of people getting caught out unless they really try hard.
Cons:
May break things immediately.
People dislike being forced into conventions.

Tool enforcement (default on) with flag or environmental variable switches.

Pros:
People are not forced to change anything, but are given errors by default.
Cons:
More configuration.
Still some potential for people to screw themselves over.

[andrewc...@gmail.com]

Though I may retract that point about people being forced into conventions, since gofmt is a success.

[Dave Cheney]

> Can we agree that #1 (your proposal w/o enforcement) and rsc's proposal w/o
> enforcement is _one_ of the design axis? Enforcement is an other
> (orthogonal) one. Please let's stop mixing the two.

Agreed. I fear that I have wasted a lot of ink and people's time and
not been clear that the enforcement aspect of this proposal was the
source of my objection.

> I am claiming that rsc's proposal (even w/o enforcement) is better than a
> special GOROOT-only approach because it scales. #1 solves an immediate need
> for gc, and perhaps for some other std lib packages. rsc's proposal (even
> w/o enforcement) solves a general need (godoc comes to mind, and some
> packages under go.tools, and I'm certain there's a multitude of other places
> in the wild where the internal convention would be useful.
>
> For instance, it permits tool developers to use the convention now and split
> up their programs into internal packages. Of course they can just _document_
> their packages as internal, but why should everybody come up with their own
> convention if there's a good convention that can work for everybody? This
> has been the spirit of Go from day one: The conventions enforced by the go
> tool, the formatting style enforced by gofmt, even some of the restrictions
> in the language all have been a boon for developers at large. I think the
> same will be the true here.
>

> Ok, let me restate that: You are against the enforcement. You may be ok with
> rsc's proposal if it's not enforced. Is that correct?

Yes. If the go tool does not restrict who can import a package with
"internal" in its path, consider my objection withdrawn.

Thanks

Dave

[Gordon Klaus]

On Fri, Jun 20, 2014 at 2:37 AM, Brad Fitzpatrick <brad...@golang.org> wrote:

We have a choice here.

We do have a choice here.  But the nice thing about this situation is that we don't have to make the choice immediately based on our predictions of the outcome.  We can take one step forward (convention without enforcement) and gain some actual experience.  After some time, if we see that people repeatedly import internals and complain when they change, we can take another step and decide to enforce the hiding of internals.  Incrementally, carefully, lazily, safely.

I (and I think others) vote in favor of the conservative choice, not exposing things previously unexposed.

You almost convinced me by calling it "the conservative choice".  But it's really not that radical to expose something while waiving all guarantees about its usability -- it seems near enough to not exposing it at all.  But again, this is speculation -- let's let time tell us thetruthofthematter.

[Henrik Johansson]

I would ideally like a language mechanism for this which would be an even harder enforcement but given the compatibility guarantees it can be premature.

I vote for tool enforcement.

/Henrik

[roger peppe]

I've only just seen this proposal. Huge +1 from my point of view. We have
a large code base in Juju with many many packages which
could usefully be classified as internal in this way.
I think we may well move towards using this scheme even
if the Go tool does not yet support it, as Robert Griesemer suggests.

One aspect that I'm not sure is entirely addressed in the
proposal is the possibility of nested internal packages.

For example, in github.com/juju/juju, we have packages
which we would like to expose to the outside world,
but most of it should probably be considered internal.
But even *within* juju, there are packages which
would prefer to avoid exposing some implementation
detail packages to other parts of juju.

So I'd suggest that the rule should perhaps be:

An import of a path containing any elements named "internal" is disallowed
if the importing code is outside the tree rooted at the parent of
the last “internal”
directory.

How does that sound?

cheers,
rog.

[Ingo Oeser]

I would like to raise to major different points:

 * types defined in those "internal" packages used for arguments/return values of methods/functions

 * "internal" types embedded in types used for arguments/return values of methods/functions

What happens to the users of those structs?

What happens to reflection and serializations like XML/Gob/JSON/binary etc.pp. here?

How do I ensure that those will not suddenly break now?

What happens to map keys of those internal structures?

so this works: http://play.golang.org/p/nVA-Laluuw

change A and B Internal from string to []byte

so this stops compiling http://play.golang.org/p/hzFDiGrg4_

even if I used the stable Public struct

[David Symonds]

On 20 June 2014 21:40, Ingo Oeser <night...@googlemail.com> wrote:

> I would like to raise to major different points:
> * types defined in those "internal" packages used for arguments/return
> values of methods/functions
> * "internal" types embedded in types used for arguments/return values of
> methods/functions

They'll be in exactly the same situation as unexported types: they can
be returned or taken as arguments or embedded just fine. Packages
elsewhere won't be able to name the types, but they'd still be able to
access fields or declare vars (with :=) of those types.

[Maxim Khitrov]

I would prefer an overridable +build flag as well. The directory
hierarchy of Go projects is already too deep for my taste (almost as
bad as Java), no need to add an extra level.

Furthermore, you have to consider not just existing projects with an
internal/ directory, but also those that have documented internal
packages that are not under internal/, which would be the majority of
such projects. Adding a +build flag to existing packages is a lot
easier than having to move those packages to a new subdirectory. For
example:

http://godoc.org/gopkg.in/qml.v0/tref
http://godoc.org/github.com/openshift/geard/utils
http://godoc.org/github.com/smartystreets/goconvey/convey/reporting
http://godoc.org/github.com/smartystreets/goconvey/convey/gotest
http://godoc.org/github.com/mattrajca/GoEV3/utilities

[Jan Mercl]

And also able to call exported methods of such (internal) types, use
the instance to (statically) satisfy an interface, .... But all of
that is per the proposal unstable (method names, signatures,
existence, field names, ...), so AFAICT, we are back to "only
discipline as documented can prevent your code from possibly breaking
in the next release" and that discipline includes not transitively
exposing internal, but exported types having accessible (exported)
details.

-j

[Robert Griesemer]

On Fri, Jun 20, 2014 at 2:26 AM, roger peppe <rogp...@gmail.com> wrote:

    An import of a path containing any elements named "internal" is disallowed
    if the importing code is outside the tree rooted at the parent of
the last “internal”
    directory.

I believe this would already be supported by the proposal. This is what would make it scalable. Basically, a large component can contain inner components.

- gri

[Ian Lance Taylor]

On Thu, Jun 19, 2014 at 9:18 PM, Dave Cheney <da...@cheney.net> wrote:
>
> Yes. If the go tool does not restrict who can import a package with
> "internal" in its path, consider my objection withdrawn.

Good, I can't recall anybody objecting to the proposal as a concept.

If we accept that the proposal is conceptually OK, but only disagree
on enforcement in the go tool, I'd like to understand better the
argument against enforcement. To me, the arguments for enforcement
are that it prevents accidents and it saves us from getting into a
position where a major Go package imports an internal package, thus
making that internal package harder to change. I can see one argument
against enforcement, which is basically that it gives us something
else to document and something else for users to understand. Are
there other arguments against enforcement?

Ian

[Henrik Johansson]

I think enforcement is a must for this kind of thing.

If people want to fork, fine it can't be avoided but a simple "let's import ignored" will happen often.

Enforce!

/Henrik

[roger peppe]

That is excellent, thanks - it wasn't clear if that was the case or not to me.

[Kyle Lemons]

What about turning on enforcement at the outset with a link that can be clicked in the error message to raise objections (comments on an issue or something), and then we can reevaluate with real-world feedback as the freeze draws near?

[Brad Fitzpatrick]

One thing worth noting that I haven't seen mentioned:

Google's been effectively using this for our internal packages for ages, since our internal build system supports visibility.

So that's precedent that like this works well in large codebases.

On Wed, Jun 18, 2014 at 12:42 PM, Russ Cox <r...@golang.org> wrote:

golang.org/s/go14internal

comments welcome on this mail thread. thanks.

[Gordon Klaus]

My main argument against enforcement is that, at this point in time, it is not obviously necessary.  It is not clear that there is an actual problem that it solves, because we have no actual experience with internal packages yet.  Why expend effort and add complexity before we know it will really add value, especially when we can wait and see for ourselves if there is really a problem to solve?

Is there some harm in taking it slow, one step at a time?

--
You received this message because you are subscribed to a topic in the Google Groups "golang-dev" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/golang-dev/_cAggq73yME/unsubscribe.
To unsubscribe from this group and all its topics, send an email to golang-dev+...@googlegroups.com.